Thanks, Andy! (Community Facilitator for the JISC Flexible Service Delivery (FSD) Programme)

1. Event/calendar sharing
2. Public calendars
3. Epic source code clean up (and source code release)
4. Properly set up http://calendar.lincoln.ac.uk
5. Integrate library return dates
6. Open to everyone
(7. Rewrite OAuth/SSO to use SQL + Windows server)

Future desirables (in no specific order)

1) Mobile site
2) Improve IE compatibility
3) CalDav
4) Further integrations with other services – posters.lincoln.ac.uk, room bookings (likely to be our summer project), Jerome, etc
5) Revamp and bulk up our Nucleus APIs

Fun, fun, fun!

Total ReCal interface

Building on our experience from developing the Common Web Design (CWD), we present the CWDx 1.0, designed specifically for awesome web based applications.

CWDx is 100% HTML5 and CSS3, includes extensive WIA-ARIA mark-up for accessibility, is Ajax driven and is wonderfully minimalistic.

The application is build using the CodeIgniter 2.0 PHP framework, and uses MongoDB as the database. The front end uses jQuery, jQuery UI and a number of functions from PHP.js.

The calendar itself is an amazing jQuery plugin called FullCalendar which emulates a lot of Google Calendar’s functionality.

Currently students can view their academic timetable and if they have any, their Blackboard assignment deadlines. Both staff and students also have their own calendar which they can write to. Everyone can subscribe to their calendar on a mobile device or in another application such as Google Calendar.

In the next few weeks users will have access to their library book return dates, will be able to create new calendars and share events, and we’re currently developing a CalDav system that will allow true event syncing across multiple devices and applications.

First of all, Total ReCal helps staff by making any updates to calendar information replicate around the system as rapidly as possible. Since our Nucleus events platform serves as a single source of information any updates directly on Nucleus are instantaneously reflected in views on the data, and even changes to imported data such as timetables and assessments are shown a lot faster than before. A reduced lag time between making a change and systems updating means that it’s easier to make changes and cancellations to events, and that more people are likely to be informed of those changes. For some alterations we even have change detection which can be hooked into notification systems such as text, making cancelling a lecture and notifying students a simple operation.

There’s also a related benefit to the rapid updating of information in that Total ReCal (or, more accurately, Nucleus) represents a single location for calendaring data, meaning that it’s a lot easier to draw on collated data. This may initially seem like a somewhat ‘fluffy’ feature which will never be used, but with a little bit of thought it’s easy to see how it can help drive decisions. For example, we can draw pretty graphs of room usage over time and spot peaks and troughs, enabling smarter timetabling. We can detect collisions across disparate systems, reducing confusion over resource allocation. We can monitor assessment ‘pile-up’ to help spread the workload of students more evenly. In short we gain the ability to draw up reports on just about anything in real-time.

Finally – and most importantly – Total ReCal will make it easier to do things the ‘right’ way (ie by managing events centrally) rather than by groups or departments going off and doing their own thing. Where in the past things such as induction timetables were managed by departments and distributed (literally) as a gigantic Word document it’s now easier for everybody involved to just use Total ReCal and to distribute the result over My Calendar. Students can be forcibly added to calendars such as induction or exams, and our code takes care of all the hard work of making sure your events don’t collide with another. Even better, there’s no real deadline for content creation because there’s no publishing deadline. Change the event centrally and the change ripples out to all the users and other systems relying on calendar data within minutes.

We hope that within a year Total ReCal will have prompted students to demand that their departments use centralised timetabling and assessment deadline management, leading to a more unified, reliable, easy to use and just better looking life.

One of the main activities of academics is often teaching, be it lectures, seminars or smaller groups. At the moment we have a staff timetable which can be used to see where teaching happens, but My Calendar provides a much smoother interface to the same data. Once we’re sure the basic functionality is in place we can then go a step further, giving the academic access to information such as attendee lists directly from within the application. It’s a small thing, but we think it’s helpful.

Another big thing that we’ve heard time and time again from academics is that they very easily forget to return library books. In an ideal world everybody would jot down due dates in their calendar, but when you’re trying to manage everything else it just isn’t practical. My Calendar will solve this by making book due dates seamlessly available from the same place, automatically taking into account renewals. We’ll even put a ‘renew it now’ button straight in the calendar for the times when you just can’t be bothered walking to the library to return a text.

“But wait,” I hear you cry. “Academics don’t have time to visit yet another website to see things, otherwise these problems wouldn’t exist.” Well, we’ve solved that as well. My Calendar also sports industry-standard iCalendar outputs for everything, making it a breeze to load information straight into the calendaring tool of your choice and keep it updated with zero effort. Almost all smartphones support it, and the vast majority of desktop calendaring applications do as well. We’ll be providing in-depth instructions on how to get yourself synchronised at launch time.

In the past, students at Lincoln have used a wide variety of different systems to organise their lives. It’s not uncommon throughout the course of a year to have an induction/reinduction timetable, your academic timetable (which changes regularly), an exam timetable (or two), a list of assessment deadlines, a calendar of meetings with seminar groups or tutors, book due dates, club or society events you want to attend, guest lectures, skills training timetables, a list of gigs you want to see at the Engine Shed and a personal calendar on top of all that. With tens of different systems all trying to tell you when and where you need to be it’s far to easy to miss something or double book yourself. The consequences of mucking up your calendar can range from the mildly annoying through to putting your degree at risk.

Total ReCal (or, as we’re naming it for Lincoln, My Calendar) is the first step towards tidying up some of this mess by dragging as many of these systems as we can together into one. The benefits of this unification are readily apparent, when the University moved exam timetables from a single massive schedule into personal timetables the response was overwhelmingly positive. Even small reductions in the number of disparate systems which are required are welcomed with open arms by a student body who are increasingly overwhelmed by the sheer volume of information they are presented with. The reasoning why is simple – it’s a lot harder to miss something important if everything is kept in one place.

There’s another massive benefit to My Calendar as well: the ability to subscribe to your calendar and any changes using any method which supports the industry standard iCalendar format. This is used by almost everything which tries to share calendar information, and allows students to seamlessly and effortlessly import their University calendar into whatever they use to organise their personal lives. Windows Live Calendar, Google Calendar, Outlook, iCal and more desktop applications have support built right in. Even better, so do Android phones, Windows Phone 7 handsets, iPhones and many more smartphones. This means that straight out of the box there is a large portion of the student body who can keep their academic timetable, due dates and more right where they want them. Our initial user survey shows that a large portion of students own either a compatible smartphone or something such as an iPod Touch (which also supports the standard), and that a similarly sized chunk goes to the effort of transcribing their academic timetable into whatever they use to manage their time.

My Calendar also features our fanatical dedication to usability, and as a result is built to not only look easy on the eye but also to work happily alongside assistive technologies or on small screens. This is a distinct benefit over the present systems, which variously don’t work on some browsers, aren’t assistive technology friendly, require the user to be on a PC or are just plain ungainly to use. We’re also working alongside various groups and people from around the University to ensure we keep up to date with the latest in usability, and have some intensive testing lined up to make sure it’s as simple and friendly to use as possible. We hope that this ‘pick up and go’ approach will let students find their feet as far as timetabling goes a lot faster than at the moment, by presenting a more familiar interface to calendars. Once they’re happy with the system, it then goes a step further by allowing customisation of exactly what’s displayed and how.

That’s about it for the student benefits which we perceive, although it’s likely that more will appear as people find new and exciting ways to use the service. Stay tuned for how Total ReCal and My Calendar will benefit staff, both those using it for time management and those on the administrative side.

This post is mostly about Nucleus, our name for the storage layer which drives the Total ReCal components. The only way to communicate with Nucleus is over our RESTful API. This comes as somewhat of a shock to some people who believe that the way to move data around is a batch script with direct database access, but I digress…

What I’m going to try to do here is summarise just how epically confusing our permissions handling system for Nucleus is, mostly for the benefit of Alex and myself who (over the next week or so) will be trying to implement this layer without breaking anything important. It’s really, really essential that we get this done before we start promoting the service because of a few simple reasons:

• Data security is important, and we don’t want anybody being able to read everything without permission.
• Data security is important, and we don’t want anybody being able to write all over the place without permission.
• Changing this kind of thing on a live service is like trying to change the engine block on a Formula 1 car whilst it’s racing.
• We need to be able to guarantee the system can hold up to DoS attacks or runaway processes hammering the APIs.
• People are already asking for access to this data for important things, like their final year projects.

So, where to go from here? Let’s take a look at everything which will be going on in the finished version.

Server Rate Limiting

Even before the Nucleus code kicks in, the server is fine-tuned to avoid overloading from any IP address or hostname. Using a combination of the OS firewall and the web server configuration overall request rates and bandwidth usage is kept below thresholds to ensure that the server is never overloaded. Due to the RESTful nature of the API (in which each request must represent a complete transaction) we have no requirement to ensure server affinity, so if the load gets too heavy we can easily scale horizontally using pretty much any load balancer.

To keep the pipes clear for our ‘essential’ services we do maintain a whitelist of IPs which have higher (but still not uncapped) limits.

Key Based Access

The only way to access any data in Nucleus is with an access token, issued by our OAuth system. These come in two flavours, either a user token (which grants permission for a specific user), or an autonomous token (which is issued at an application level, and is ‘anonymous’). The very first thing that happens with any request is that the token it gives is validated. No token, no access. Invalid token, no access. Revoked token, no access. To keep things nice and fast we store the token lookup table in memory with a cache of a few minutes, since most requests occur in ‘bursts’.

Key Rate Limiting

Key Rate Limiting Rate limiting is an important part of keeping things going. Unlike the server-based rate limiting which affects IP addresses and hostnames, key based limitation prevents one application or user from going overboard with their requests. Each key is limited to a certain volume of requests per hour, the exact volume varying depending on the type of key. Individual user keys have relatively few, through to whitelisted internal applications which have extremely high caps.

Like Twitter we’re also implementing smart rate limiting, which will globally reduce API limits in times of unusually high demand such as freshers’ week. We’d rather that everybody can access some data than nobody can access any. To give people an idea what’s going on all responses will include information on how many requests per hour they’re allowed and how many they’ve used, letting applications ‘budget’ accordingly.

Scoping

Scoping is the next level of permissions, now that we’ve made sure the system making the request isn’t abusing the system and has a valid token. Scoping is quite hard to wrap your brain around, but once you’ve got it then it makes a lot of sense. In short, access tokens are given ‘scopes’ which define the type of data they’re allowed to ask for, but not necessarily what within that scope they are allowed to subsequently access. This means, for example, that a user can grant an application permission to look at their events but not other data such as their contact details. When an application is registered with us we decide which scopes are valid for its autonomous token, and when a user grants permission using OAuth they are clearly told which scopes the application is asking for. After all, you don’t want an automatic book renewing application being able to take a look at your home contact address.

Every request to Nucleus exists within a scope, and we make sure that the access token which is given actually has permission to access that scope of data. In the case of events we have a variety of scopes including public (just the name, start/end times and location of events in publicly timetabled spaces), basic (the same details, but including personal events as well) and full (The whole set of data, including things like attendee lists for lectures). There are also some scopes governing if a key is valid to write events, making sure that an application designed to give you a nice calendar view can’t instead fill your schedule with rubbish.

Object Permissions

Now we’re getting into the meat and potatoes of permissions, designed to simultaneously give the ability to set sweeping permissions at a high level and really, really accurate permissions at the object level. Permissions are similar to scopes in that there is a predefined list, but they differ in that they tell Nucleus how any given key can interact with any specific object. They are used to make sure that one user can’t go fiddling with another’s data, or to stop a runaway application from accidentally overwriting data it doesn’t own.

As an example, an application designed to show your upcoming events would have passed rate limiting, had its key validated, and been checked to have the ‘basic’ scope to read events. Object permissions say which of those events it can actually see, and this is where it gets really complicated.

Global Type Permissions

Some permissions are granted for every single object of a given type (such as every event), and these are known as global type permissions. An example is the ‘inherent’ permissions which are available to any valid key within the scope, such as the ability to read public data.

We can grant global type permissions either to all keys (a ‘universal’ permission), or to specific applications, groups or users. Whilst this is unlikely to occur in practice (and would be bad practice even if we did), it does allow us to build ‘god’ applications which are capable of performing actions across the board.

Universal Object Permissions

Alongside global type permissions we can approach the problem of mass permissions from the object end. Universal Object Permissions are stored along with the object, and are permissions which are available to any key. This type of permission finds itself suited to things such as public events, where we want to give anybody permission to see the details regardless of if they’ve been manually added.

Specific Object Permissions

The most restrictive type of permission is a specific object permission. These are a one-to-one relationship between an application, a group or a user which grants permissions for one object. An example would be a private event, which you wouldn’t want anybody else to be able to see (although if it’s occurring in a publicly timetabled space everybody will be able to see that an event is there, just not what it is). In this case the object would just have one specific permission granting you the ability to read and modify it.

Specific permissions also come in handy with shared events. The system would allow for you to grant specific users the ability to see an event, for example just inviting your study group to share a room booking.

Multiple Permissions

All permissions are standalone, there is no inherent hierarchy. They also stack neatly regardless of which route the permission is granted by, so it’s entirely possible that you may have a global read public data permission, a universal read details permission granted to your group, and a final specific permission allowing you to modify an event.

Wow…

That’s a high-level overview of what’s going on to make sure Nucleus is solid enough to stand up to a hammering whilst making sure that people can’t be naughty and at the same time being flexible enough to fine-tune permissions.

The API will prevent people from ‘orphaning’ events without permissions, as well as provide a route for reading and setting permissions (with permissions controlling who can set permissions…) and making sure that sensible permissions are set by default so that you can’t accidentally create an event you can’t then control.

Epic Permissions. Coming in the next couple of weeks.

After numerous code re-writes we’ve got a rock solid API for adding, updating and deleting events in our Nucleus data store. Our import code has also had many updates to support logging of changes to events which will be invaluable to students to keep them up to date. Once the main Total ReCal application has been developed we’re going to sit down and work out how we’re going to best make use of these logs.

When a lecturer calls in sick the central timetabling department isn’t informed (unless it will affect lecturers for a long period of time). Therefore based on our current nightly timetable imports we won’t find out about any changes. We’re going to develop a tool for faculty administration staff to make changes to events as they’re going to be more aware of what the situation is day to day. This means that we can then inform students of changes that day as soon as someone changes it.

In terms of the front end, I’ve forked our common web design, called it ‘common web design x’, made it fluid to adapt to browser size, made it completely semantic HTML5 based, and taken the concept of progressive enhancement to new levels. It will also make use of our new OAuth 2.0 based single sign on service that I’ve written and it will automatically adapt to mobile layouts.

Our next big job is to move away from bulk imports of data and instead start developing code that will go through and validate and verify events. So this could be looking for changes in the time of events or verifying that the right students are seeing the right events (in the event of a student changing course for example). With these changes logged we can then tackle one of the top requests that students have of the University and that is to be better informed of changes to their timetables.

The main timetables are produced by the Registry department however they aren’t informed if a lecturer is ill on a particular day, and in any case timetables aren’t updated currently until the following morning, so we’re planning on developing a tool for faculty offices to use so that they can make individual amendments to timetables when rooms need changing or lectures have been cancelled so that students can be informed sooner.

The logging of these changes will be important for Blackboard too. Certain schools and faculties like the idea of personalised assignment calendars however their own internal policies don’t allow staff to set deadlines inside Blackboard because deadlines may be changed by lecturers and senior staff aren’t informed. This is why the Computing School for example release a huge Excel spreadsheet of deadlines because it means only two people have access to change deadlines. We don’t want to be in a situation where we have to create individual departments their own tools to manage assignment deadlines, we’d prefer everyone used Blackboard and so with the ability to log changes to events what we could do is delay the update of the deadline in the student calendars for 24 or 48 hours giving senior staff a period in which to change it back to the original date or leave it (i.e. approve the change).

Our plan over the next few weeks is to perfect our API for querying events, give more students access to the their iCal feeds and also start developing the front end calendar application.