Netserve Service Status

We are currently working around a poorly timed drive expiry - some users may be experiencing an issue with mail collection. We are aware of this. No inbound mail is being lost, however you will currently be unable to get to it until we have repaired the drive; this includes the webmail facility. For the users this involves the data has been set read only to prevent damage to the inboxes. We will update you as soon as we are able.
[Update 0934]
We have recovered the data, and are brining live mail collection within the next five minutes. Inbound mail will be live hopefully within the next 20 minutes. Differences between restores and live will hopefully be bought up to speed within the hour.
[update 1030]
The inboxes will commence updates from inbound mail now. You may still find historical mail arriving in an odd order for the time being as retries are serviced.
[update 1147]
Users who had PROCMAIL configurations will now find that these are functioning again.

We are experiencing an issue with the popping service. This is not effecting those who have paid for an IMAP service.

Earlier sending issues where down to a single user the relay service as a mass mail host. Their IP has been blocked, and further action is being taken currently.

We are currently unable to give an ETA on the POP issue, however we are both aware of it and working on it.

[update 1116]
The issue is fixed, however the server is playing catch up. Please ensure you are not checking your mail every minute; step back to every ten to allow the status quo to return. Thank you.

Yesterdays critical incident has been narrowed down to the activities of one server, and one VPS machine.

An apparent distributed denial of service attack took place, possibly with what also looks like a MAC flood against a connected switch device.

Other devices then shut off connectivity to the flooding switch presenting as a loss of device. Replacement of device did not resolve the issue. Realisation of the intended target and steps taken to minimise the impact reduced the overhead, however removal of the target machine resulted in an immediate return of service.

Fall back service provision will be removed during the course of the morning to return to full primary service. We do not envisage more than a momentary loss of service during this fail back most likely presenting as an error to email clients.

Thank you for resisting the temptation to call in during this period

A faulty authentication database has been repaired this morning. A subset of POP mail users will have had issues collecting mail for the duration - the issue having surfaced at around 2am this morning - and become more obvious as loads built up. The issue is now rectified.

We are currently experiencing issues with the scanning mail servers mxs1 and 2. We are working on a remedy to this currently, and will have it back to normal as quickly as we are able.
However in the meantime please resist the temptation to phone and request information on this issue, or worse still, mail in causing more of a backlog to clear. Many thanks
This in no way effects your ability to collect email.

Authentication is currently failing for mail collection. We are aware of this, and are trying to remedy the issue - please do not call in so we are able to progress.
What this means is that you are unable to collect email - however processing of inbound email is unaffected.

One of our 3 pop servers suffered an issue with its firewall which resulted in it blocking traffic from the LVS load balancer.

The issue has been resolved and should not cause any further error messages when customers try to download mail.

From Saturday 22nd December through to Tuesday 1st January we will be handling support via email only. Emergency support via email will be available on 25th and 26th December. Our offices reopen on Wednesday 2nd January 2008.

As normal on call engineers will be available at all times for service related issues.

I would like to take this opportunity to wish all of our customers a happy Christmas and a prosperous new year.

Keep your wits about you

We are seeing an increase in Spec Reg (speculative registrars) either offering to sell you a domain similar to yours, or even more amusing, reminding you of an expiry and offering to renew it. If its not in an email from us, then its not the real deal. People do fall for these otherwise they would not bother with them. As for people trying to sell you a domain - this is an insight into what happens when domains drop back into the public domain, and they will, from our experience try to sell you the most ridiculously cumbersome ones, so if you have a nice short name that draws traffic - be warned.

Here we have the DRoA (Domain Registry of America), they like to send letters, but here is an email as an example - they are nothing to do with us.

From: Transfer and Renewal Support [mailto:transfers@droa.com]
Sent: 27 November 2007
To: [deleted]
Subject: [deleted].com; Order # 1234567

Hello [deleted],

The transfer and renewal of your domain name, [deleted].com, is not yet
complete because your domain name is currently in a "REGISTRAR-LOCK" status
with your current registrar TOTAL WEB SOLUTIONS LIMITED TRADING AS
TOTALREGISTRATIONS

In order to complete the transfer and renewal, the "REGISTRAR-LOCK" status
needs to be removed and an EPP Key/Authorization code needs to be obtained.
Please see below for instructions on achieving this.

Solution:
- Log into your account with your current registrar, and change the status
of your domain, [deleted].com, from "locked" to "active".
- Then look for the EPP Key or Authorization Code.

- Alternatively you may call your current registrar, TOTAL WEB SOLUTIONS
LIMITED TRADING AS TOTALREGISTRATIONS (see phone number below) and ask them
to remove the lock status of your domain name, give you your EPP Key and
allow the transfer to Domain Registry of America.

- Once done please notify us that you have done so, by calling our toll-free
number below. We will then re-attempt the transfer and renewal of your
domain name.

As a convenience, we have supplied your current registrars phone number
below.

Domain: [deleted].com
Current Registrar: TOTAL WEB SOLUTIONS LIMITED TRADING AS TOTALREGISTRATIONS
Registrar Phone Number: Please visit their site to contact them

Transfer Department
Domain Registry of America
Toll free 1-866-434-0212 or for International Callers, dial +1(905)479-2533

.... and here we have a choice, yet more amusing example of why you should keep ontop of domains you like.

Recently, [deleted].com expired and we acquired it
in a domain name auction. Since you own the .net or .org
version of this domain name we wanted to provide you
with the opportunity to own the preferred .com version.

Our company specializes in recovering preferred expiring
domains and either selling them to individuals such as
yourself or building out our own web presence on those
valuable domains.

[deleted].com is a pretty darn good domain name and,
the truth is, the .COM is a far stronger version of the
name than then .NET is.

* .Com is the strongest brand on the internet.
When people think of a website, they intuitively
think ".com". Odds are people trying to get to
your website are inadvertently going to [deleted].com
because they assume that's where they can find you.

* .Com conveys Professionalism that .net & .org
cannot match.

If you'd like to own [deleted].com, you can buy it
now by covering our acquisition costs and a modest profit.

If you have any interest I encourage you to act quickly
because this domain name will only be offered for sale
for a limited time.

Click Here For Pricing On [deleted].com

https://secure.claimthisdomain.com/buy.php?[deleted].com


Please advise,

If you do wish to transfer to these people, or wish buy back a domain or whatever, just make sure you understand what their terms and conditions are, transfer limitations, costs and the like.

If you want to transfer away, give us a call, or drop us a mail and ask for our impartial advice; its always good to err on the side of caution.

We are dealing with an mail relay issue. We will have the service up and running as soon as we are able. We have had an influx from a spamming attempt, and are taking steps to remove these from the live queue and locate the source and take steps to prevent this flow before restarting the service.

[Update 1003]
The subsequent work has revealed an underlying file structure issue with a host machine serving data for a number of back office services. We have taken these offline temporarily to fix the issues before bringing those back online. We are sorry for the inconvenience - we should be back on within a few minutes.

[update 1030]
We have pulled the drive in question, and restored services from backup to other hosts. Primary services have been restored, and will be moving to tidy up any lose ends now until we can get new hardware instated.

We have had a few cases now where the nice people at BTinternet/Yahoo are delaying mail that is being forwarded to accounts with them.

Needless to say there is little we can do about this. If you are set up with a forwarder from your domain to a BT account, and mail is not getting through or is delayed, please contact them and ask for support on the matter.

In the meanwhile the sender may be receiving mails from our mail servers telling you that they have been unable to deliver so far; this is not our issue, they are just not accepting the mail.

Our LVS load balancer required a reboot this evening. Sadly our monitoring and sms alerting provided by Tmobile failed to notify the oncall engineer of the issue for over an hour. Once the alert had been issued we were able to restore service within 10 minutes.

Services with issues incude, hosting on service 150 and 212 as well as pop collection from pop1.nsnoc.com

A short outage was alerted and remedied this morning. This involved a load balancer, and for those users that were involved, connections to the cluster one and three and mail collection will have been noticed. This has been remedied for the effected inbound users traffic.

During this period inbound email was unaffected for all users.

We have experienced a complete power failure in one of our racks at the IFL data centre. This occured at the PDU breaker as opposed to in rack, and we are in the process of assessing damage and bringing up services as it is safe to do so.

This has taken out a number of primary services, however we are aware of the situation there is no need to mail in.

We will post progress as it occurs, and if you are experiencing issues above and beyond what is reported as fixed, then please do mail into support@support.nsnoc.com . Thank you.

[2209 Update] We have returned at this point domain name resolution and the primary of the three clusters. We are continuing to work on the situation both on site and remotely and will post more as we can. SSL server services have been returned.

[2240 Update] We have returned mail collection services and mail relay services. Inbound mail from external sources has not been effected. Clusters two and three are back in service delivering web traffic.

[2330 Update]
All core services have been returned to normal. We currently have 5 servers offline hosting about 30 customers which we have tracked back to a failed APC power distribution unit. It is unclear whether the unit failed causing the trip switch to cut power to the rack or whether the unit failed as a result of a power distribution fault. We are making arrangements for the unit to be tested and replaced if necessary as soon as possible.

One of the core SMTP relay servers relay.nsnoc.com is currently out of action and waiting on an egineer assisted reboot.

Further updates will be provided shortly.

UPDATE 19:54

Services have been restarted and will be monitored for the next 24 hours. The server had failed to reboot due to a some file system corruption.

We are aware of climbing loads on the mail cluster. This will not cause any issues with transit at all, however you may be experiencing slower responses than usual for those using the pop1.nsnoc pool this afternoon. This should continue to die off as the afternoon continues.

We are planning to move mysql1.nsnoc.com to from 4 to mySQL 5 on Wednesday 10th October (two weeks time).

We are currently testing the platform we are moving to, and will be releasing details on how you can test your dev / staging code against the installation before it goes live shortly.

Any questions on the matter please contact support.

We have experienced a fault with one of our load balancer. Taking it out of service users will have experienced sporadic connectivity to the mail server for a five minute period during the diagnosis and change over. Some web sites will also have been affected for this period.

We failed over to the stand in load balancer as soon as we were able, and will be looking to switch back to the main box as soon as we are confident the issue has been dealt with.

As an advisory to Plus.Net users - it would appear that for some reason they are classing our primary secondary and mail scanning mail clusters as "Dynamic Addresses". While this is not the case, it is not preventing them from currently rejecting mail forwarded to accounts on their servers from our clusters.

Feedback has been along the lines of:

[snip]I'm sorry to hear you have been adversely affected by changed we have made to our email delivery system. The changes were designed to reduce the amount of spam our network receives. This is a generally accepted practice in the industry.

No hosting service should be using dynamic IPs to forward mail, you might want to discuss this with them. I have however, passed on the info you have provided, to see if we can make allowances in this instance.[snip]

We are all behind the less spam option and use many restrictions on our own scanning and non scanning servers - however this misapprehension that we are on a dynamic range will cause users with this ISP who usually use our forwarding service issue until they get their block lists corrected.

We will update this with more information as we get it.

[Update 1605]
The following URL was supplied to us by a client for the PlusNet forum. This surely cannot be the case since this is not entirely practical especially with load balanced clustered systems. We await some more confirmation.

[Update Monday 1052]
Thank you to Bob Pullen from Plus.Net for getting in contact with us to remedy this issue so swiftly. Apparently their list was showing the block in question being largely dynamic, and this has now been corrected.

Our offices are closed today from 12 noon due to the marriage of our senior technician, Anthony. We will reopen for business 10am on Thursday 30th August.

In the mean time, limited email support will be provided for emergency cases only. On call engineers will still be available 24/7 for service related issues.

On behalf of all of the staff of Netserve and Designweb we would like to wish Anthony and Cally all the best wishes for their future together.

Webmail has been taken down for maintainence, we plan on returning it as soon we as we able.

This in no way effects the ability to collect mail from desktop clients, remote webmail clients, or the transit of mail in and out of the service.

We are experiencing a delay on some mail that has been passing over one of our inbound email servers. We are aware of the issue and working on it - due to the clustered nature of our setup this should effect a small percentage of mail only. Thank you for your patience.

We are currently having issue with a number of core systems, what appears to be a denial of service is effecting relay.nsnoc.com and one of the SSL servers. We are aware of the situation and working on it, please resist the temptation to phone in - we will return service as soon as we isolate the cause and make good.

[Update 1029] An automatic defensive shutdown occurred to a server that was being attacked from a massive influx of traffic. The issue was identified, and rectified. Service has resumed.

Customers have made us aware this morning of connectivity issues from selected ISP's within the UK, which appears to have started at 8am this morning.

Investigation into the matter has proved difficult but has now been traced to one of our upstream providers advertising network routes to our network but then failing to deliver the traffic.

Under normal circumstances where a router connection fails, the advertisement to that route is removed automatically. The new routing table should then be updated to other routers that peer with it thereby preventing traffic being passed off to routers which cannot handle it.

We have closed down the peer with the London provider concerned and have raised the issue with them. When we have an explanation for the failed routes we will post further information here.

We would like to apologise to customers affected by this partial routing outage and the length of time it has taken to identify the source of the issue.

*** UPDATE ***

Further investigation suggests that the issue was restricted to a single /25 subnet used for routing transit to three customers and may have started some time after 11pm last night.

At this stage it appears that a reset filter on a single interconnect may have been the cause of the issue.

we have encountered a partial failure of our backup load balancer this morning. So while services have been running without interruption, there has been no means to reach them.

The switch over has taken place back to the primary from the backup, and the opportunity has been taken to make some adjustments to the configuration. We are sorry for the inconvenience for those of you in the office too early this morning. This was rectified prior to 9am this morning.

In addition to the DRoA - we have a new kid on the block of thoughtfully reminding you that your domain is about to expire.

While you are free to move your domain over to whoever you wish, you are best reading all the print on these kinds of emails, and looking closely at their terms and conditions. We have recently had a number of clients who have accidentally moved away and lost control of their domain.

Please be sure that the mail you are dealing with is in plain text from either DomainCity, or DesignWeb - or that you have compared the pricings T&C and what you are agreeing to by clicking the link.

Here is an example of one such mail.



The Domain Registry of America and the Domain Renewal Online are in no way related to us.

We have a current issue with our primary load balancer, and are in the process of bringing our secondary service on line. This will be effecting both mail and web will be effected for some clients, others will see no impact what so ever. We will update this as we can.

[update 1255] - this has been resolved. The primary balancer has suffered a hardware failure, the secondary is now online and operating correctly.


[update 1307] Our previous month independent statistics are showing as 99.94% uptime, and so far this month 99.88% for the six days of August; we are quite pleased with these figures, and thought we would share them with you : )

Despite our requests to our AOL customers not to use the spam report features of their AOL mail accounts when dealing with mail sent through our network we have yet again had our AOL gateway server's IP address blocked. This is now the 5th block in as many months.

AOL have refused to whitelist the IP addresses of our mail servers.

AOL customers now have two options to ensure that they are able to have email services on their domain names.

1) Migrate your ADSL service away from AOL to ourselves. Our prices start at £11.75 per month so we're considerably cheaper than AOL.

2) Have your mail delivered to a POP box on our servers rather than forwarded to AOL.
Prices for mail accounts are available from our web site.

We will configure a new mail gateway today to handle mail bound for AOL customers and will contact all customers using AOL forwarders that the service will not continue to operate from the time AOL next blocks the new IP address.

If you are unhappy with the service provided by AOL please ask them to provide you with a MAC code which you can then forward to ourselves to migrate your DSL service away.

Labels: AOL DSL migration

Customers in IFL2 Manchester can now get 1Mbit of free PA address space with 8 IP's for 12 months. Simply pay the setup fee of £150 and arrange for IFL to install cable from your rack to our Meet-me-room switches to take advantage of this very special offer.

This special offer runs for August only and is limited to 10 new customers.

Additional transit is available at £30 per mbit. The contract can be upgraded to higher a CDR at any time which can reduce the rate to as low as £15 per Mbit.

Full details of the Manchester transit offer are available from our web site.

Labels: Manchester IFL transit

We have been advised by Protx that our payment gateway is currently down due to essential database maintenance and ugprades taking longer than expected.

While the service is unavailable we will be upgrading our own payment pages to support the new 3D-Secure service from Mastercard. We anticipate that the service will be unavailable for most of today.

UPDATE 02/08/07 : 15.00

Card processing services have now been restored. 3D Secure service will be available on Maestro cards within the hour.

Welcome on board to seven new transit clients this month, along with new branches to our internal infrastructure.

Supplying redundant and robust connectivity for transit in Reynolds House, Manchester Technopark, IFL2, for co-location, primary and secondary routes.

See Netserve website for more on our transit offerings.

In July we have added to our peering routes with NTL / Telewest / BlueYonder / Virgin Media group (approximately 250 routes) at gigabit speeds.

In simple terms this means shorter / faster paths to and from their networks - less congestion - more robustness.

In the spirit of reporting not just bad things - an advisory for you.
We are currently on an advisory that one of our peering providers is diagnosing a circuit in London. For this period there will be a loss of protection. We do not envisage this impacting delivery of transit or services beyond possible path should this fail - which is nice to be able to report.

We have noticed a number of emails arriving today from euidregistration.eu claiming they have a "obligation to reduce domain ownership disputes and brand name loss" and offering to register your .eu domain name.

Can customers please note this is simply spam mail and the company concerned has no obligation or position of authority to be contacting you. The email is best put in the spam folder!

If you're interested in seeing what this spammer has to offer, I'm sorry but this is all you'll see....

"Warning: include(index__en.html) [function.include]: failed to open stream: No such file or directory in /var/www/localhost/htdocs/mall/index.php on line 25"

If you wish to secure your .eu domain with ourselves, they're available as always at £8.50 + vat via our support site.

Labels: euidregistration.eu