Wednesday, December 30, 2009
Rack B power trip
The power supply on server 32 the logs.nsnoc.com server failed at 2pm this afternoon.
The failed supply caused the trip switch to the rack to trip to protect the equipment. The UPS maintained service to the remaining servers for 15 minutes but it took longer than anticipated to identify the issue and gain access to the building.
An engineer was on site at 3pm to identify the issue and start the process of finding the server that caused the initial trip.
Power was restored to most servers by 3.30pm. We worked through all of the services to ensure that everything was working and by 5pm concluded that all remaining bugs had been resolved.
During the process we identified that one of the core switches had been incorrectly powered from the rack B UPS so the failure impacted on some services from other racks. This issue will be resolved on 4th January when the office reopens.
At this moment in time, the logs.nsnoc.com server is offline until a new power supply can be installed on 4th January. We will process the logs from today onwards once the server is working again.
All other services have been tested and confirmed as working. Please notify support @ nsnoc.com if you have any other issues.
The failed supply caused the trip switch to the rack to trip to protect the equipment. The UPS maintained service to the remaining servers for 15 minutes but it took longer than anticipated to identify the issue and gain access to the building.
An engineer was on site at 3pm to identify the issue and start the process of finding the server that caused the initial trip.
Power was restored to most servers by 3.30pm. We worked through all of the services to ensure that everything was working and by 5pm concluded that all remaining bugs had been resolved.
During the process we identified that one of the core switches had been incorrectly powered from the rack B UPS so the failure impacted on some services from other racks. This issue will be resolved on 4th January when the office reopens.
At this moment in time, the logs.nsnoc.com server is offline until a new power supply can be installed on 4th January. We will process the logs from today onwards once the server is working again.
All other services have been tested and confirmed as working. Please notify support @ nsnoc.com if you have any other issues.
Saturday, December 19, 2009
MySQL1 server
The Mysql1 server has crashed and failed to reboot this morning.
An engineer will be on site for 8.30am to investigate the issue further.
UPDATE
A second reboot of the server brought services up without errors. We will have standby hardware available for the server as a precaution over the Christmas holiday.
If customers are aware of any issues following the reboot please raise a support ticket in the usual way.
An engineer will be on site for 8.30am to investigate the issue further.
UPDATE
A second reboot of the server brought services up without errors. We will have standby hardware available for the server as a precaution over the Christmas holiday.
If customers are aware of any issues following the reboot please raise a support ticket in the usual way.
