Explanation to the Blackboard failure from 25th - 28th November

The upgrade to Blackboard on the 25th of November caused some problems, resulting in the system being offline from Thursday 7.00 AM till Sunday 8.00 PM. We are aware that this has been an inconvenience to many people. Our apologies for this inconvenience. Below we will give further explanation on the nature and course of the problems. The upgrade to the current version of Blackboard in question, version 8, should have solved a number of security risks.

What was the exact problem that caused the failure?
After the upgrade on November 25th it was discovered that attachments, like sent-in assignments and uploaded documents, could not be opened. This problem did not present itself during the preliminary tests. Because this is an essential Blackboard-function, and because it was uncertain if this problem could be solved quickly, it was decided to roll-back the upgrade and place a back-up on the system. This action should have made Blackboard available again, only slightly later than the 3.00 PM which was initially announced. Upon testing the replaced back-up it was revealed that the attachment-issue was still present. In response we immediately established an emergency-team.

What action was taken during the emergency?

On Thursday afternoon, the emergency team brought in Blackboard's vendor. Thursday evening and Friday were spent working full-tilt to implement the suggested solutions. When these turned out to be ineffective, the vendor advised to perform a full reinstall of Blackboard late Friday afternoon. It cost a lot of time to restore all settings, programs and databases and we worked on this throughout the weekend. News about the failure was reported on the Maintenance and Failure report page of ICTS and on the Educational Applications site of S&O during office hours and on Sunday evening at 8.20 PM. Status updates were reported on at irregular intervals from Sunday afternoon around 1.00 PM till Sunday evening 8.30 PM.

The system was once again available on Sunday evening at 8.00 PM.

What was the cause of the problem?

The exact cause is not known yet and is being investigated by the emergency team. There are a number of ideas that are presently being investigated in a structured manner, in cooperation with Blackboard's vendor. First, we need clarity about the cause before we continue the implementation of the planned update.

How is a Blackboard upgrade tested?

Every Blackboard upgrade is subjected to extensive testing on separate Blackboard-servers that are used for this purpose only. The upgrade of the 25th of November (new service pack) was also tested prior to implementation, and the problem that happened on Thursday 25th of November did not occur during those tests. The Blackboard production environment is also tested by the administrators directly after the installation of a patch or service pack. It is then reviewed to verify the application is working as expected, before Blackboard is released to the public (students en teachers). This way we can ensure that possible unforeseen problems are spotted before the application is effectively put to use.

Why isn't Blackboard-maintenance done during the evenings or weekends?

When planning maintenance to Blackboard, we take the educational process into consideration to the fullest extent possible. This means that Blackboard-maintenance is never planned during:

  • the first week of a block (‘kwartiel’)
  • the week prior to the exam weeks (as well as the resits in August)
  • the exam weeks (as well as the resits in August).

We realise that it is preferable to perform maintenance to Blackboard during the evening hours or during weekends. Incidentally, this is currently performed based on voluntariness and availability. However, current policy and the existing employment agreements make it impossible to guarantee service outside of office hours.

Where is maintenance to Blackboard announced?

The announcement of maintenance is given a week in advance at the latest, and in this case was given two weeks in advance. The announcement was released in four locations:

In case of emergencies such as this, the homepage of the ICTS will report the situation. The students portal was insufficiently updated during the Saturday and Sunday when the failure took place. We have noticed this and regret any additional inconvenience this may have caused. We are aware that communication during an emergency needs to be improved, and we will handle this better in the future.