Wednesday, February 28, 2007
Mail maintenance
As some of you may be aware we are currently dealing with a recurrent issue and working towards a long term fix.
While we appreciate that no one particularly likes being held up by road works, we are also aware that no one likes bad roads either - as a comparison.
Due to a combination of utilisation abuse and general increase in mail flow our projected capacity has been exceeded. This has resulted in the delivery outage of Sunday and Yesterday (Tuesday 10pm).
To repair strangeness on mail collection and webmail issues a cache repair is required on the IMAP and POP servers, this will induce a load, causing long mail collections to time out and performance to dip.
Once these have been replicated, steps will be taken for the long term fix of moving the mail from the existing platform to the disk pack which is already in place. The newer array delivering better redundancy and throughput in general.
While every effort will be made to ensure this doesn't impinge upon your mail service, interruption's will be necessary to make these improvements, and we apologise for any inconvenience caused.
[Update Friday 2nd March 4pm]
Steps have been taken to allow the move to take place, however the current demands on the mail and mail systems are resulting in a deadlock on the move.
In quantitative terms user email is burning through around 80mb in 15 minutes at this quiet time on a Friday afternoon.
We will be taking all precautions we can to ensure that the mail stays operative, but will be unable to complete a synchronisation until levels settle down.
While we appreciate that no one particularly likes being held up by road works, we are also aware that no one likes bad roads either - as a comparison.
Due to a combination of utilisation abuse and general increase in mail flow our projected capacity has been exceeded. This has resulted in the delivery outage of Sunday and Yesterday (Tuesday 10pm).
To repair strangeness on mail collection and webmail issues a cache repair is required on the IMAP and POP servers, this will induce a load, causing long mail collections to time out and performance to dip.
Once these have been replicated, steps will be taken for the long term fix of moving the mail from the existing platform to the disk pack which is already in place. The newer array delivering better redundancy and throughput in general.
While every effort will be made to ensure this doesn't impinge upon your mail service, interruption's will be necessary to make these improvements, and we apologise for any inconvenience caused.
[Update Friday 2nd March 4pm]
Steps have been taken to allow the move to take place, however the current demands on the mail and mail systems are resulting in a deadlock on the move.
In quantitative terms user email is burning through around 80mb in 15 minutes at this quiet time on a Friday afternoon.
We will be taking all precautions we can to ensure that the mail stays operative, but will be unable to complete a synchronisation until levels settle down.
Monday, February 26, 2007
Email storage
This weekend we suffered a delay to mail appearing in inboxes after a large influx of mail. This was not caused by the volume of mail since we allow for spikes, however by the lack of available space.
Users are reminded that mail inboxes are a free service supplied with hosting on domains. We have been contacting those who are using excessive space, and pointing out to those using inboxes without hosting that there are issues - some have been slow to act or respond - resulting in the lack of space and delay to perceived arrival.
Our mail mechanism has many servers ready to receive and process mail, and will happily sit on it until it is able to process this to inboxes. This issue was transient, and after alerts were issued, no longer current.
All delayed mail has been flushed to inboxes well before Monday morning.
Users are reminded that mail inboxes are a free service supplied with hosting on domains. We have been contacting those who are using excessive space, and pointing out to those using inboxes without hosting that there are issues - some have been slow to act or respond - resulting in the lack of space and delay to perceived arrival.
Our mail mechanism has many servers ready to receive and process mail, and will happily sit on it until it is able to process this to inboxes. This issue was transient, and after alerts were issued, no longer current.
All delayed mail has been flushed to inboxes well before Monday morning.
Wednesday, February 21, 2007
Advisory: Domain Registry of America
Please be advised, the DRoA are nothing at all to do with us, and thus should a mail sent to you requesting renewal or similar please keep this in mind before taking any further action. We are pointing this out since we are having a lot of queries from confused customers.
Mail relay
Our relay server is playing catch up after a hosting clients form was abused last night. While we have addressed the clients issue for them (please DO check your forms validate entries for escaped commands) this does mean we are currently experiencing delay on sent mail leaving the server.
Thursday, February 15, 2007
Transit updates
We have installed a new gigabit switch into the meet-me-room of the IFL2 installation. This should improve the service delivery to transit customers. This follows the new London link for shorter routes to and from London. Drop us a mail if you are looking for more information on competitive transit out of IFL, Reynolds House or Manchester Technopark.
Load balancer event
A failure event occurred this morning with the piece of equipment responsible for delivering content to each of the POP/IMAP/HTTP service clusters. On receipt of the alert the issue was fixed within minutes.
No inbound emails were harmed during the outage, and service was resumed on restart of the load balancer.
No inbound emails were harmed during the outage, and service was resumed on restart of the load balancer.
Saturday, February 10, 2007
Service events
Virtual Server Host 48 was power cycled today. It had ceased to respond. Remote power cycling failed to bring it back up. After a short delay on site operatives gave the unit an attended reboot bringing it back full service.
The host supplying egress filtering for the web clusters followed a similar demise, however did not return from a power cycle - querying PSU failure. In relation to this incident we have lost a primary resolving domain name server (not ns1 or ns2) - adding some latency to internal system and ADSL customer resolutions. This has resulted in a high load on cluster machines requesting data over HTTP from external sources.
However, a byproduct of this will be a lack of logging information for the immediate future. We don't envisage a loss of logging data - this is simply a loss of service delivery not data.
Changes will be affected across the clusters to remove this issue. In the interim pages that are reliant on external data will run slowly while the request times out, or appear not to run at all.
[1400 Update] Changes have been made to allow the cluster services to route their outbound HTTP requests via another egress filter, marking a return of service.
[Tuesday Update] We are on site working on the failed egress server.
The host supplying egress filtering for the web clusters followed a similar demise, however did not return from a power cycle - querying PSU failure. In relation to this incident we have lost a primary resolving domain name server (not ns1 or ns2) - adding some latency to internal system and ADSL customer resolutions. This has resulted in a high load on cluster machines requesting data over HTTP from external sources.
However, a byproduct of this will be a lack of logging information for the immediate future. We don't envisage a loss of logging data - this is simply a loss of service delivery not data.
Changes will be affected across the clusters to remove this issue. In the interim pages that are reliant on external data will run slowly while the request times out, or appear not to run at all.
[1400 Update] Changes have been made to allow the cluster services to route their outbound HTTP requests via another egress filter, marking a return of service.
[Tuesday Update] We are on site working on the failed egress server.
# posted by Netserve @ 1:12 PM
Friday, February 09, 2007
MSN & Hotmail 550 rejections
We are seeing a lot of mails in at the moment about hotmail and MSN rejecting mail. We are seeing clients mailing in headers that are not just from us, including the bigger players; Blueyonder are currently also sharing the pain. We are 'assuming' they are testing some new block listing (similar to the harshness that is AOL) or their servers are sick.
550 Your e-mail was rejected for policy reasons on this gateway. Reasons
for rejection may be related to content such as obscene language, graphics,
or spam-like characteristics (or) other reputation problems. For sender
troubleshooting information, please go to http://postmaster.msn.com. Please
note: if you are an end-user please contact your E-mail/Internet Service
Provider for assistance.
554 5.0.0 Service unavailable
Unfortunately they are the ones rejecting so we are unable to fix this for you. We have contacted MSN and are expecting an automated reply within the week. We should keep in mind that these are supplied as free services.
[update 12:26]
We are registered with Hotmail/MSN SDNS service. The information gathered from this suggests that clients with accounts set up to forward to Hotmail/MSN addresses have received spam that they have requested to be forwarded and marked it as junk - causing our server that delivered it to be blacklisted.
Currently our spam scanning servers are not listed, however the master relay server is.
The issue here with the Micro$oft system is that they are marking the last relay as opposed to the source of the spam (although all headers can be forged). This is a similar situation as is presented with AOL currently. Small solace can be taken from larger players currently also being blocked.
[Monday Update] We have started routing mail bound for to a dedicated IP for relay to Hotmail - since it is not on their block list. However, once it is blocked this situation will resume. Options include investing in scanning to relay to solve someone elses blocks, or preventing customers from setting AOL or Hotmail addresses as their destination for non scanned forwards or wild-card redirection.
Current situation mails via relay or from the web servers are being rejected. Systems mail is getting out.
IF YOU ARE A HOTMAIL USER - please contact Hotmail / MSN as we are delivering the service and they are rejecting it. Request that the block is removed for the subnets 195.69.95.0/24 and 94.0/24, or that you are at the very least able to whitelist the mail you require.
550 Your e-mail was rejected for policy reasons on this gateway. Reasons
for rejection may be related to content such as obscene language, graphics,
or spam-like characteristics (or) other reputation problems. For sender
troubleshooting information, please go to http://postmaster.msn.com. Please
note: if you are an end-user please contact your E-mail/Internet Service
Provider for assistance.
554 5.0.0 Service unavailable
Unfortunately they are the ones rejecting so we are unable to fix this for you. We have contacted MSN and are expecting an automated reply within the week. We should keep in mind that these are supplied as free services.
[update 12:26]
We are registered with Hotmail/MSN SDNS service. The information gathered from this suggests that clients with accounts set up to forward to Hotmail/MSN addresses have received spam that they have requested to be forwarded and marked it as junk - causing our server that delivered it to be blacklisted.
Currently our spam scanning servers are not listed, however the master relay server is.
The issue here with the Micro$oft system is that they are marking the last relay as opposed to the source of the spam (although all headers can be forged). This is a similar situation as is presented with AOL currently. Small solace can be taken from larger players currently also being blocked.
[Monday Update] We have started routing mail bound for to a dedicated IP for relay to Hotmail - since it is not on their block list. However, once it is blocked this situation will resume. Options include investing in scanning to relay to solve someone elses blocks, or preventing customers from setting AOL or Hotmail addresses as their destination for non scanned forwards or wild-card redirection.
Current situation mails via relay or from the web servers are being rejected. Systems mail is getting out.
IF YOU ARE A HOTMAIL USER - please contact Hotmail / MSN as we are delivering the service and they are rejecting it. Request that the block is removed for the subnets 195.69.95.0/24 and 94.0/24, or that you are at the very least able to whitelist the mail you require.
# posted by Netserve @ 11:23 AM
Thursday, February 08, 2007
Delays on inbound mail
We are currently experiencing a delay to the inboxes for inbound client mail. Mail is sitting on inbound servers, and waiting for the next retry to be passed to the concentrating server to be placed onto the mail backend. We envisage this being completed before 1700hrs today.
After testing a new build of the mail server software in house, going live showed index inconsistencies which in turn were generating high loads. In the interest of not bringing the mail collection system to its knees, common links were terminated with the inbound mail. This has since been reinstated - resolving the problem and closing the outage.
No mail will have been lost or rejected due to this outage.
After testing a new build of the mail server software in house, going live showed index inconsistencies which in turn were generating high loads. In the interest of not bringing the mail collection system to its knees, common links were terminated with the inbound mail. This has since been reinstated - resolving the problem and closing the outage.
No mail will have been lost or rejected due to this outage.
