Wednesday, May 13, 2009
Maintenance Window
We will be bringing our second router back online between 11am and 1pm on Thursday 14th May.
ADSL customers may experience a brief (60 second) outage while cables are re patched.
No other issues are expected as a result of the router replacement.
UPDATE 14:00
The work with the router replacement has taken longer than expected due to a faulty APC distribution switch causing our main load balancer to reboot. The result being that sites on clusters 150, 156 and 212 have suffered 3 outages in the last hour lasting up to 10 minutes.
Work is nearly completed to complete the router installation. Further updates will follow.
UPDATE 18:00
The work to replace router 2 was completed earlier this afternoon but is not yet in use. We will shedule out of hours work to enable the upstream connections on this router to reduce disruption during the settlment of our routing tables.
We have identified two issues during the work today.
An APC power switch which supports the load balancer for 150, 156 and 212 appears to have a loose connection which causes the load balancer, servers 29, 37 and 64 to reboot if the cables are moved. We will shedule another maintenance window to transfer the power cables to a new APC unit. As yet a date has not been set.
The second issue relates to a fault with a link to IFL1 which carries our ADSL customer traffic. The work on the new router today has highlighted a potential fault which we have raised with the link provider today. When the issue has been investigated and resolved we will shedule an out of hours maintenance window to transfer the connection from router 1 to router 2.
ADSL customers may experience a brief (60 second) outage while cables are re patched.
No other issues are expected as a result of the router replacement.
UPDATE 14:00
The work with the router replacement has taken longer than expected due to a faulty APC distribution switch causing our main load balancer to reboot. The result being that sites on clusters 150, 156 and 212 have suffered 3 outages in the last hour lasting up to 10 minutes.
Work is nearly completed to complete the router installation. Further updates will follow.
UPDATE 18:00
The work to replace router 2 was completed earlier this afternoon but is not yet in use. We will shedule out of hours work to enable the upstream connections on this router to reduce disruption during the settlment of our routing tables.
We have identified two issues during the work today.
An APC power switch which supports the load balancer for 150, 156 and 212 appears to have a loose connection which causes the load balancer, servers 29, 37 and 64 to reboot if the cables are moved. We will shedule another maintenance window to transfer the power cables to a new APC unit. As yet a date has not been set.
The second issue relates to a fault with a link to IFL1 which carries our ADSL customer traffic. The work on the new router today has highlighted a potential fault which we have raised with the link provider today. When the issue has been investigated and resolved we will shedule an out of hours maintenance window to transfer the connection from router 1 to router 2.
# posted by Netserve @ 8:51 PM 
