If you saw my recent Blog “When things get hot”, you will know the challenges faced by IT staff when server room cooling fails. Well, the air conditioning in that server room failed again on 8th December which was really unexpected because it had received a major overhaul. In the light of this further incident, we are now putting in place permanent 24×7 monitoring of the temperature in that room until we cease using it, and we have mobile cooling units on site that we can use should the need arise – these can keep the room sufficiently chilled in the event of any further failure of the main system. In fact, we will only be using this room for a few more months before moving all the kit into a new state of the art datacentre shared with Aberdeen University and Aberdeen College. The 24×7 monitoring will be in place in time for the Christmas break, and as we do every year, a number of IT Services staff are on call over the holiday.
As if that was not enough, on Tuesday 11th December we had a very unusual technical problem on our storage system, which hosts “home” and “shared” network drives for all staff and students. IT Services staff worked through the day, and through the night until 2am in conjunction with global support engineers from the manufacturer before a chap called Adam in their Australian support centre identified the problem and we got the “home” and “shared” drives back. Big sigh of relief! We will meet with the manufacturer early in January to carry out a review of what happened. Meantime, once again big thanks to a number of IT staff who worked well into the evening and night to sort this.
For staff and students, over these few days they would see a short outage of some services early Saturday morning, and the loss of network drives on Tuesday. Behind the scenes, however, staff from IT Services had a heavy programme of work to keep services running and secure for the whole of that week. With one of the server rooms operating at reduced capacity, they had to move some services to the other server room. Systems like e-mail, the web site, our Moodle Virtual Learning Environment, the Portal and many others kept operating throughout all of this period. A lot of the week was spent in conjunction with our Estates Department arranging for the cooling to be fixed – and I’m pleased to say that the faulty parts have been replaced and cooling is working again. IT Services staff also had to work for several days to re-establish the backup systems which had been significantly affected by the cooling and technical problem. All this is almost finished as I write. Staff and students don’t see that work, but it is essential to ensure that all our services are protected and properly backed up – certainly before the holiday period. Apart from the work to re-establish our backups systems, we have put a freeze on all other changes now until the University re-opens in January.