We have just received this message from one of our suppliers. This only affects customers on uk1-pn, uk1-adj and a small number of dedicated server customers.

We need to perform some emergency network as soon as possible. This has been scheduled for tonight, which we appreciate is very short notice. We need to reboot a router to install new software, and this reboot will take up to 45 minutes. We will do everything we can to speed up the process as much as we can and reduce the maintenance time.

Date: 24/09/2009
Window: 23:00 for 2 hours
Duration: < 45 minutes.

The maintenance is to perform an emergency upgrade of Cisco software. We are using a Cisco VSS-1440 as part of our network core, and we have been experiencing some reduced performance with it today. There is no cause within our network configuration and set up of this, and it started to have a detrimental effect on some clients today. We escalated this to the Cisco TAC team, who have diagnosed a fault with the software on the router in the form of a memory leak. Cisco has supplied us with a new version of the software for the router which will fix the memory leak and slow performance.

The nature of this problem is that it will escalate as time goes on, which is why we want to apply the fix as soon as core business hours finish today. Please accept our apologies for the short notice, we hope our clients appreciate this problem was out of our control caused by Cisco software, and we are working as best we can to resolve it quickly.

We apologise for any inconvenience this may cause, please do not hesitate to contact us if you have any queries or questions regarding this maintenance window.

UPDATE 01:56: It looks like the network is coming back up now, it took longer than expected so I’ll post any updates in the morning.