Thursday, January 18, 2007
Power Outage
[Incident 12:50]
We have a complete power failure at the IFL2 data centre around 12:50 today.
All services including web, email, ADSL and DNS are affected.
[Update 15:33]
We have power, and some connectivity. First boxes back to life around 1335, although without connectivity from transit providers, and peering suppliers (many with routers in same installation).
We are bringing up services where we can. Some hardware has not come back from the hard off of a lack of power, but aim to return basic services as quickly as possible.
There has been no feedback from IFL as yet as to why or how their feeds/UPS/generators did not pick up the ball.
Please resist the temptation to call in - we will return services as quickly as we are able to diagnose and implement solutions or work arounds. It is currently impossible to deliver an ETA, as we do not know all of the factors involved.
[Update 16:20]
POP and IMAP are operational, inbound mail and relay are operational, primary web cluster is operational. There is currently no FTP service currently. The smaller clusters two and three are being worked on. A number of virtual server hosts need further attention.
Service -WILL- be slower while we catch up with any backlogs, and further interruptions are not out of the question as we finalise a short term solution (including an imminent restart of the main file storage machines).
The cause is still unknown - other than a power failure on our floor of the IFL2 installation.
[Update 21:03]
Nearly all services now up and running. Exceptions being a virtual server host machine and one web server.
Loads are running high in the absence of some SQL slaves, otherwise services are returning to normality.
IFL have informed us that a mains power tripped and went to battery which then presented a series of faults; this in turn caused a power failure on the second floor of the building. Individual rack trips then slowed a return to service as power was restored. Power is currently being supplied direct from mains until the UPS can be tested and recommissioned tomorrow.
We have a complete power failure at the IFL2 data centre around 12:50 today.
All services including web, email, ADSL and DNS are affected.
[Update 15:33]
We have power, and some connectivity. First boxes back to life around 1335, although without connectivity from transit providers, and peering suppliers (many with routers in same installation).
We are bringing up services where we can. Some hardware has not come back from the hard off of a lack of power, but aim to return basic services as quickly as possible.
There has been no feedback from IFL as yet as to why or how their feeds/UPS/generators did not pick up the ball.
Please resist the temptation to call in - we will return services as quickly as we are able to diagnose and implement solutions or work arounds. It is currently impossible to deliver an ETA, as we do not know all of the factors involved.
[Update 16:20]
POP and IMAP are operational, inbound mail and relay are operational, primary web cluster is operational. There is currently no FTP service currently. The smaller clusters two and three are being worked on. A number of virtual server hosts need further attention.
Service -WILL- be slower while we catch up with any backlogs, and further interruptions are not out of the question as we finalise a short term solution (including an imminent restart of the main file storage machines).
The cause is still unknown - other than a power failure on our floor of the IFL2 installation.
[Update 21:03]
Nearly all services now up and running. Exceptions being a virtual server host machine and one web server.
Loads are running high in the absence of some SQL slaves, otherwise services are returning to normality.
IFL have informed us that a mains power tripped and went to battery which then presented a series of faults; this in turn caused a power failure on the second floor of the building. Individual rack trips then slowed a return to service as power was restored. Power is currently being supplied direct from mains until the UPS can be tested and recommissioned tomorrow.
# posted by Netserve @ 1:25 PM 
