Monday, October 12, 2009
Mail Cluster Issue
INITIAL 9:45PM
We're currently repairing some file corruption on the main mail server file store.
The process should not take more than a few minutes. Until the repair is completed there is no access to POP, IMAP, or Webmail services.
UPDATE 10:15PM
The file system check and repair has completed however a server reboot is required to clear a number of Dead/Zombie processes on the server.
We're going to assess the issues and decide on a course of action shortly.
UPDATE 10:50PM
An engineer will be dispatched to IFL2 at 3am to recover the file server to our new racks. We hope to have the service restored shortly after 8am.
UPDATE 8.15AM
The overnight work to recover the file server to our new racks was completed on time and without incident.
UPDATE 11:45AM
Heavy storage loads have resulted in the need to pull the file system and force a tree rebuild - this is going to be a lengthy process, so we are working to restore a backup in the interim with an ETA of a fifth of the time before inboxes are present. AGAIN I would like to reiterate this is not preventing inbound mail from arriving, merely the ability to collect it, an important difference.
UPDATE 13:38
Working on a solution to deliver / present fresh email to the end users. While historical data will be missing, all new emails will be arriving and visible - allowing us to migrate these to the existing file system once rebuilt. Light at end of collective tunnels.
UPDATE 13:45
Temporary solution in place. POP users will now be seeing all their new email arriving. Those who are paying for the IMAP solution will notice that all their mail has gone. DO NOT adjust your set. Simply work with what you have for now, and your inbox will be reunited with old mail on completion of the rebuild over the next 12 to 24 hours.
This completes updates on this entry, work from this point on will continue in a new entry. Over the last 10 minutes, over 467Mbytes of inbound mail have been stored, and we can see you retrieving email.
NB, if you use webmail or Imap to collect mail, please avoid creating new directory structures until your old mail account has been merged with the new server.
UPDATE Wednesday 11.30am
Work is continuing on the old file server. We are currently running a snapshot of the data to backup before starting the merge with the new file server. Further updates will follow at 2.30pm today.
UPDATE Wednesday 2.30am
The snapshot of the data is now 1/3rd complete. It's likely to be early evening before we can start the process of merging the new mails with the old. Next update 5pm
UPDATE Wednesday 5pm
The snapshot of the data is now 3/4 complete. We're expecting to be able to start the data merge at 9pm which should take less than 1 hour to complete. Next update once data merge has been completed and all webmail/IMAP data has been restored.
UPDATE Thursday 1am
Our new file server is now live with a full copy of all mails, old and new. At no point has any mail been lost or rejected.
There is a small risk that the mail indexes will have changed for IMAP users resulting in a mis match between the From/Subject line presented by your mail client and the mail downloaded. The fix is to right click the folder concerned and select 'rebuild index'
We're currently repairing some file corruption on the main mail server file store.
The process should not take more than a few minutes. Until the repair is completed there is no access to POP, IMAP, or Webmail services.
UPDATE 10:15PM
The file system check and repair has completed however a server reboot is required to clear a number of Dead/Zombie processes on the server.
We're going to assess the issues and decide on a course of action shortly.
UPDATE 10:50PM
An engineer will be dispatched to IFL2 at 3am to recover the file server to our new racks. We hope to have the service restored shortly after 8am.
UPDATE 8.15AM
The overnight work to recover the file server to our new racks was completed on time and without incident.
UPDATE 11:45AM
Heavy storage loads have resulted in the need to pull the file system and force a tree rebuild - this is going to be a lengthy process, so we are working to restore a backup in the interim with an ETA of a fifth of the time before inboxes are present. AGAIN I would like to reiterate this is not preventing inbound mail from arriving, merely the ability to collect it, an important difference.
UPDATE 13:38
Working on a solution to deliver / present fresh email to the end users. While historical data will be missing, all new emails will be arriving and visible - allowing us to migrate these to the existing file system once rebuilt. Light at end of collective tunnels.
UPDATE 13:45
Temporary solution in place. POP users will now be seeing all their new email arriving. Those who are paying for the IMAP solution will notice that all their mail has gone. DO NOT adjust your set. Simply work with what you have for now, and your inbox will be reunited with old mail on completion of the rebuild over the next 12 to 24 hours.
This completes updates on this entry, work from this point on will continue in a new entry. Over the last 10 minutes, over 467Mbytes of inbound mail have been stored, and we can see you retrieving email.
NB, if you use webmail or Imap to collect mail, please avoid creating new directory structures until your old mail account has been merged with the new server.
UPDATE Wednesday 11.30am
Work is continuing on the old file server. We are currently running a snapshot of the data to backup before starting the merge with the new file server. Further updates will follow at 2.30pm today.
UPDATE Wednesday 2.30am
The snapshot of the data is now 1/3rd complete. It's likely to be early evening before we can start the process of merging the new mails with the old. Next update 5pm
UPDATE Wednesday 5pm
The snapshot of the data is now 3/4 complete. We're expecting to be able to start the data merge at 9pm which should take less than 1 hour to complete. Next update once data merge has been completed and all webmail/IMAP data has been restored.
UPDATE Thursday 1am
Our new file server is now live with a full copy of all mails, old and new. At no point has any mail been lost or rejected.
There is a small risk that the mail indexes will have changed for IMAP users resulting in a mis match between the From/Subject line presented by your mail client and the mail downloaded. The fix is to right click the folder concerned and select 'rebuild index'
# posted by Netserve @ 8:47 PM 
