A Sound Investment

Some of you may have noticed some inclement weather in the North East of Scotland these past few weeks leading to flooding, blocked roads and power cuts. A lot of this peaked on December 30th in the Garthdee area of Aberdeen, where the RGU Campus is. We didn’t get flooded, but there was quite a big power cut across the Campus.

Over the past few years we have moved all our core IT infrastructure into two new datacentres – sharing with the University of Aberdeen and North East Scotland College. One of them is on the RGU Campus so this power cut was a good test. Both datacentres are well equipped with battery backup, plus generator backup – but of course as much as you can test these components in controlled circumstances, there is nothing like the real thing.

In the event, when it became clear that there had been a power cut, I received an anxious phone call from the duty manager asking me if the data centre and our IT systems would be OK? By the time I received the call, the automatic alerts had already notified technical staff in IT Services and the Head of IT Operations and Support, Richard Lynch, was on his way to the Garthdee Campus. There he found that both the battery backup and generator had kicked in automatically as they are designed to when the power was cut. All our IT services continued without interruption and when power was restored the generator powered itself down again and everything settled back to normal.

Many if not most of the major incidents I have encountered in IT have been down to a loss of cooling and/or power in the datacentre. Incidents such as these demonstrate the value of investing properly in datacentre resilience. What ended up being a non-event could otherwise have been a a much more protracted IT recovery operation.

Goodbye St Andrew Street!

Moving a data centre from one location to another without disrupting the organisation is a major undertaking. We moved one data centre to a shared facility with the University of Aberdeen and North East Scotland College last year successfully, and on the weekend of 24th / 25th May we moved our second data centre from St Andrew Street down to a new purpose built facility on our Garthdee Campus. This one will also be shared with the University of Aberdeen and North East Scotland College.

The St Andrew Street building has been the University’s IT hub pretty much since IT began. When the national JANET network came into being, its main entry point was into St Andrew Street, and for many years the University’s main computer room was there, with network links to the rest of the Campus. The St Andrew Street building will be sold at the end of June, and this data centre move is part of a wider set of tasks to decommission the whole building. This has involved re-routing our external fibre connections (sorry about the roadworks!), and significant changes to the University’s overall Campus network so that we can unplug St Andrew St.

Down at Garthdee, there was construction work to build the new data centre, and this also involved significant changes to the Campus network at Garthdee so that the new data centre was connected into our network. All this work has been going on quietly over the past several months stage by stage.

Prior to the weekend of 24th / 25th of May, all of our critical systems had to be switched so that they were running fully from the North East Shared Data centre at the University of Aberdeen. That allowed the IT team over the weekend in May to shut everything down at St Andrew Street, move it down the road, and bring it back up again without any disruption. Once everything was up and running, critical services had to be rebalanced to run across the two data centres.

The School of Computing and Digital Media also had to move their servers from St Andrew Street down to the new data centre and they had completed their physical moves ahead of the 24th and 25th of May.

In a couple of weeks, the three remaining University departments will leave the St Andrew Street building for ever. Once they are gone, IT Engineers will disconnect the building and decommission the internal networks – that truly will represent the end of an era for us, but we are looking forward to moving down to Garthdee!

Here is the St Andrew Street Data centre before the move:
062-01

And here it is looking pretty empty after everything was moved down to Garthdee:
062-02

And here is the new datacenter down at Garthdee:
062-03

Quick update on IT

Wow, where does the time go? I haven’t posted on the Blog since February – that’s bad, sorry about that. There has been lots happening since then, so here is a quick general update.

On Wireless, we still had a few problems with the new system but we have had great feedback from students through Facebook and face-to-face and this has really helped IT Services to identify and work through some of the problems. As I said in January, we can see from our logs that large numbers of people have been able to connect and use the system successfully, but we knew from feedback that quite a few were still having problems. We’ve worked with the manufacturer and have made a number of changes to the controllers and feedback from students now is that it is greatly improved. We still have to extend the coverage to ensure that all areas on the Campus are covered, and that is now planned for the summer.

We have been looking at the “old Library” area in the Aberdeen Business School and are right now finalising plans to put in more IT workstations, a help point, AV facilities, and better group work areas for students to use. IT Services along with Estates are just in the process of finalising cost estimates, and hopefully that will be approved and the work will commence in the near future.

Behind the scenes, IT Services are working hard on preparations to move our St Andrew Street datacentre. We moved one of our datacentres last year to the shared datacentre at University of Aberdeen and now we are moving the other one. It’s going into a new facility which we are also sharing with the University of Aberdeen so by the summer we will have all of our server kit in two new state of the art datacentres, each with environmentally friendly cooling and generator backup. We are moving ourselves too – once we get the datacentre out of St Andrew St the three remaining departments will move out of there and we will have to disconnect all the IT and remove the building from our Campus network. St Andrew St has been one of our main “hubs” for IT since IT began, so this is very much going to be the end of an era!

This is the time of year when Schools and Departments submit their planning statements for the forthcoming year and we are also looking at all of the IT related project requests so that we can update our forward plans. These are all going through the approval process as I write, and once things are finalised I’ll put a summary on the blog. Some of the key areas we are likely to be looking at are:

– A major investment in our storage and server infrastructure to replace end of life equipment and increase capacity and performance;
– Improvements to our Learning tools, including an upgrade to Moodle and improvements to access to Library e-resources
– Enhancements to our portfolio of communication tools to provide new video conference facilities across the Campus, more flexible access to telephony services and integrated access to Skype communications;
– Enhancements to the Staff and Student portal (RGyoU), particularly looking at the provision of more information for students in one place, and better document management and collaborative tools for staff;
– A new system for managing student placements
– Preparation for the opening of the new Scott Sutherland School of Architecture at Riverside East

These are just a few highlights – just as importantly we plan to spend some considerable time on preparatory work for initiatives which will start the following year, but more of that later.

Helping the Environment

I’ve put up a few posts here (Green ICT, How to move 160 Servers, Did You Notice?) about moving our datacentre and how amongst other things this will help the University reduce its environmental footprint. Well, we’ve set up the shared datacentre along with the University of Aberdeen and North East Scotland College and it’s great to see that our achievement has been recognised at national level.

EAUC (The Environmental Association of Universities and Colleges) have an annual Green Gown award ceremony, and the Shared Datacentre was a winner in the category of Modernisation: Effectiveness & Efficiency in the Estate. There’s a wee writeup of this in RGU’s “Green Times”.

The British Computer Society UK IT Industry Awards recognised our venture as a winner in their “datacentre of the year” category.

Computer Weekly has featured the project in their European User Awards as “as an exemplar of public-sector excellence and green efficiency”. It’s worth reading this article for some insight into the slightly trickier moments of the project – “. . . like a Bond movie.”

With this success under our belt, we are now actively pursuing a shared project to develop a backup datacentre – work should start in earnest in January. This one will be on RGU’s Campus and once again will be designed with the environment in mind and take advantage of the abundant supply of cool fresh air of the North East (yes, summer as well as winter) to keep our servers at the optimum temperature without spending a fortune on air conditioning. I’m definitely hoping for a white Christmas . . .

It’s not a side of IT that our staff and students often get to see, but just thought you’d like to know how much we are leading the field here and we’ve managed to achieve by sharing with other institutions. My thanks to RGU IT Staff and colleagues in University of Aberdeen and North East Scotland College!

Moving Datacentre – next two weeks

There have been a couple of postings about moving our servers into a new datacentre (Green ICT, How to Move 160 Servers). Well, now the time has come! As a reminder, we plan to move all of our servers out of the Faculty of Health and Social Care Server room into a refurbished datacentre shared with the University of Aberdeen and Aberdeen College. Over the past few months IT Services staff have been preparing the ground, making sure that the network connections are all working and that the services being moved are ready. Some of this work is highly specialised – one of the essential components to link the network had not been properly configured and had to be returned to Japan for further work, a round trip of 3 weeks. IT Services did also find that the configuration of some services had to be changed to allow them to operate in the new datacentre, and you will have seen a few small outages to allow these services to be updated.

The moves are going to happen over the next 2 weeks. ITS are not moving everything in one go, but will carry out the move in 2 or 3 stages. Some of the services ITS will be able to move without any downtime at all, some will require some downtime which is unavoidable.

The first batch are going to move this week, but the main move is planned to be the weekend of 27th July. That weekend in particular will be a substantial move and there is likely to be some downtime over the weekend, so do keep an eye on the information notices which will be issued and plan your work and studies around this.

Once this is complete, we will have removed a significant risk from our IT infrastructure by being able to decommission the old server room (see When Things Get Hot). We will also greatly improve our environmental credentials. One key measure for datacentres is “Power Usage Effectiveness”, or PUE. This is a measure of the total amount of power used by the datacentre, divided by the raw power used just by the servers – and the reason this figure is important is that older datacentres use a lot of extra power just to keep the servers cool. So, for example, if you have servers consuming 100kW of power, and if you need another 100kW of air conditioning to keep them cool, then the datacentre is using 200kW of power in total. The PUE is 200/100 = 2. We want a figure which is as close to 1 as possible – the lower the better.

In the shared datacentre we’re aiming for an average PUE of 1.2 or less, and have already reached figures as low as 1.08 at times which means we are using much less additional power to cool the servers. That sort of figure is close to Facebook’s big new datacentre in Sweden  and we anticipate we will reduce our carbon output by around 230,000 Kg per annum.

How to move 160 servers without moving 160 servers

What are some of the challenges faced by IT Services staff? Here is a guest contribution from “Bobby G” – one of our senior IT technical staff:

“It may not be the type of question we ask ourselves everyday but we have recently been in a position where we have been required to move 160 of the university’s key servers onto new computer hardware often in diverse locations, and we wondered how to carry this work out as quickly as possible and with as little impact to our customers as possible.

The servers are all real working servers providing many important roles for the University including Library, Teaching, Financial, Research, and Support services. The amount of data involved is also quite large with around 5TB of data being involved (For those of us familiar with 1.4MB Floppy Disks that’s around 3.5 Million Floppy disks worth of information).

There is a trick as I suppose there usually is with these types of questions, and the answer is to move most of the servers in a “virtual” manner. This still involves moving where the server really is in terms of all of its intelligence (CPU/Memory), but actually leaves the data with all of the information and disks exactly appearing to be where they always were. There are now a number of systems which allow us to carry out this type of work and the University has used a tool from VMware in this instance. This has allowed us to reduce the total number of real physical servers used by half from 20 to 10 servers while almost doubling the amount of computing power available.

This makes the system much greener as there are significant savings in electricity and room cooling costs, and makes it much easier to add additional servers at a very low cost.

With a little careful planning we were able to move all 160 servers in around 9 hours one weekend with most services being unaffected by the move and most of those that were affected only being shut down for around 10 minutes.

The new setup of the system has been automated in such a manner that as servers get busy or if a physical server fails the “virtual” server will now move around to find a comparatively quiet working server and no one will even know it has moved (unless they have access to the log files). So we currently know the room that your server runs in but not exactly where it is as it may have moved itself in the last 10 mins. One of the next challenges we are giving ourselves is to setup the system so that we don’t even know which room the server is running in to allow the systems to move between buildings for themselves when a service is busy or there is some form of problem (e.g. power outage) in a building.

So in answer to the question “how do you move 160 servers without moving 160 servers” – you only move the little bit of intelligence that runs the servers and leave the rest set up as it is. (i.e. move where the server thinks it is).”

A Busy Week!

If you saw my recent Blog “When things get hot”, you will know the challenges faced by IT staff when server room cooling fails. Well, the air conditioning in that server room failed again on 8th December which was really unexpected because it had received a major overhaul. In the light of this further incident, we are now putting in place permanent 24×7 monitoring of the temperature in that room until we cease using it, and we have mobile cooling units on site that we can use should the need arise – these can keep the room sufficiently chilled in the event of any further failure of the main system. In fact, we will only be using this room for a few more months before moving all the kit into a new state of the art datacentre shared with Aberdeen University and Aberdeen College. The 24×7 monitoring will be in place in time for the Christmas break, and as we do every year, a number of IT Services staff are on call over the holiday.

As if that was not enough, on Tuesday 11th December we had a very unusual technical problem on our storage system, which hosts “home” and “shared” network drives for all staff and students. IT Services staff worked through the day, and through the night until 2am in conjunction with global support engineers from the manufacturer before a chap called Adam in their Australian support centre identified the problem and we got the “home” and “shared” drives back. Big sigh of relief! We will meet with the manufacturer early in January to carry out a review of what happened. Meantime, once again big thanks to a number of IT staff who worked well into the evening and night to sort this.

For staff and students, over these few days they would see a short outage of some services early Saturday morning, and the loss of network drives on Tuesday. Behind the scenes, however, staff from IT Services had a heavy programme of work to keep services running and secure for the whole of that week. With one of the server rooms operating at reduced capacity, they had to move some services to the other server room. Systems like e-mail, the web site, our Moodle Virtual Learning Environment, the Portal and many others kept operating throughout all of this period. A lot of the week was spent in conjunction with our Estates Department arranging for the cooling to be fixed – and I’m pleased to say that the faulty parts have been replaced and cooling is working again. IT Services staff also had to work for several days to re-establish the backup systems which had been significantly affected by the cooling and technical problem. All this is almost finished as I write. Staff and students don’t see that work, but it is essential to ensure that all our services are protected and properly backed up – certainly before the holiday period. Apart from the work to re-establish our backups systems, we have put a freeze on all other changes now until the University re-opens in January.