Netserve Service Status

Routing Issue

2011-12-19T09:41:00.001Z

We've been investigating two reports of end users not able to reach our network. In both cases the points of failure appear to be well outside of our network. As yet we have been unable to identify the network at the heart of the issue.If anything develops we'll advise here, but at this point in time we're not seeing any changes in network usage or border routes.

Rack A5 power trip

2011-12-08T13:56:00.003Z

The power supply to rack A5 has tripped due to the power supply on server 64 failing.

Power has been restored, servers in A5 are all working as normal.

Services (POP/IMAP mail) from Server 64 have transferred to servers 65 and 66 via the cluster management system.

POP/IMAP/Webmail Servers

2011-12-04T22:02:00.008Z

The file server responsible for storing mail data has crashed three times in the last 3 days.

Investigations have shown one of the drives to be generating IO errors.

At midnight the failed drive will be replaced and re silvered

UPDATE 9am Monday

The drive replacement / resilvering is 50% completed. The file server is operating in a degraded state so is running with less performance than normal.

We expect the process to be completed by 6pm.

Until that time the mail servers will run with a higher than normal load and may be less responsive that usual.

UPDATE 1pm Monday

Work to bring the file server up to speed with a full disk set should now be completed by 4pm.

We have identified during the work that one of the POP cluster servers is suffering from a faulty network connection. Further investigation will continue this afternoon.

UPDATE 3 pm Monday

The process to re silver the missing drive is due to complete at 4.30pm.

During the monitoring of the server loads it has become increasingly apparent that there are a number of customers with POP accounts where mail is being stored for considerable periods of time. POP is not best suited to storing mail for longer periods and it is apparent that the loads on the POP servers are being caused by inappropriate use of the POP accounts.

We will be performing a review of POP usage over the next week and advising customers where their usage needs to be modified.

UPDATE 8pm Monday

File system completed the rebuilt at 4.30pm. Server loads all returned to normal within a few minutes of the work being completed.

We will monitor the situation over the next 24 hours to ensure there are no further issues.

File Server outage

2011-05-02T15:05:00.001Z

Server 67 our core file server has suffered a crash. It's not viable today to attempt a reboot so services are currently running from our backup file server.

We'll reboot 67 in the morning and run a file system check.

At the moment we've disabled the FTP upload service to avoid any possible conflicts between the live server and backup.

Update

2011-02-05T12:06:00.003Z

Bad weather in North Wales has caused some power outages on the St Asaph Business Park overnight.

Capital Court hasn't suffered any power issues but we have been indirectly affected by switches on our transit routes being power cycled between mains and generator backup. We installed a UPS at 7,30pm last night on a switch that had been suffering from power cycling issues.

This morning the transfer from generator to mains at a local POP appears to have caused one of our Capital Court switches to lock it's network facing uplink port. We've made changes to the switch so service to Row A racks 3,4 and 5 has been restored.

IFL Incident

2010-10-12T23:36:00.002Z

An incident in the main energy centre at IFL has resulted in services for area C1 being run on generator. Suite C1 houses our core switches and routers so our core network is currently at risk.

Further information will be provided as soon as we are made aware of the situation.

ADSL Service

2010-10-06T08:49:00.003Z

A network upgrade yesterday has caused a number of our cisco switches to lock up.
We've replaced 4 switches, but now need one more to complete the process and restore the ADSL services. The last switch should be with us this morning to install and configure.

4th October Outage

2010-10-04T23:27:00.003Z

The upgrade to the North Wales circuit appeared to have completed correctly however further investigation revealed that the two core Cisco switches in Manchester had blocked the connections for North Wales post upgrade.

Switch reboots failed to resolve the issue resulting in us having to transport two new switches to Manchester.

As yet we've not had time to pick over the bones of the issue. The Ethernet circuit upgrade should not have caused any issues but the timing of the two events would suggest that they are related.

We'll be meeting with the Fibre providers in a few hours to see if they can shed any light on the event from their side of things.

Maintenance Schedule - September/October

2010-09-17T10:50:00.000Z

Friday 17th September
Switch upgrade OpticGlyndwr St Asaph.
Customers at the site may see a short (3 seconds) loss of Ethernet circuit while the cables are patched between switches.

Sunday 19th September - IFL2
The Cisco switch located at C1-S5/A4 in IFL2 which supports colo equipment in Suite C4 at IFL2 will be replaced.
Patching of cables should take no more than 3 seconds for the Ethernet circuits to establish themselves on the new switch.

Monday 4th October 8pm
St Asaph, Bangor, Deeside, St Davids Park, Abergele
Our backhaul circuit over the Fibrespeed network will be upgraded between 8pm and 9pm. A brief outage is possible while the work is performed.

IFL Cable test

2010-08-16T10:09:00.003Z

IFL are testing one of our core network cables in IFL2 within the next few minutes to identify the cause of some packet loss hitting our St Asaph network.

It's likely that there will be a few seconds of instability on the network during the test.

UPDATE

IFL cable proved to be working fine.

UPDATE

Fault traced to local circuit in St Asaph. Media converters replaced at both the Fibrespeed pop and OpticTechnium appears to have rectified the issue.

New Office Move

2010-07-25T21:35:00.004Z

Please be aware that we have been moving to new offices over the weekend.

In order to finalise the move we will be splicing new fibre cable on Monday morning. We expect to have the work between 10am and 11am.

Until that time we'll have no telephone support available and email support will be restricted to high priority tickets only.

Our new office details are...

Netserve Consultants Ltd
Unit 44B
Capital Court
St Asaph Business Park
Denbighshire
LL17 0JG

Email storage reboot

2010-06-01T09:47:00.001Z

The storage system that supports POP/IMAP and webmail has been rebooted after locking up at approx 10:15.

All email appears to be working without issue now.

Host 8

2010-03-08T23:01:00.002Z

We have identified that a number of files on Host server 8 have been modified by a remote third party via a vulnerability within a client web site.

As a precaution we have disabled a number of additional PHP functions on the server and increased the settings on mod_security.

The modified files have been replaced with valid files from our backup.

Mysql1 Server

2010-01-07T03:54:00.005Z

Thursday 7th Jan 4am.

The sql server mysql1.nsnoc.com has crashed and requires a restart. It has failed to respond to two automated reboot attempts and will require manual intervention to restart it.

An engineer will be on site shortly to attempt to resolve the issue.

Next update due 7am.

UPDATE
The server has responded to an engineer being present at the reboot. All services are working normally.

It is not clear at this time why the server is not rebooting on the remote switch when required so we will take steps today to migrate the database to new hardware.

UPDATE 2PM
The cause of last nights issue has been traced to a faulty network card. We are in the process of replacing the card and will provide an update shortly.

UPDATE 2.45PM
The network card has been replaced and hte service appears to be working now without issue.

We will monitor the system over the next 24 hours to ensure stability.

Rack B power trip

2009-12-30T18:34:00.004Z

The power supply on server 32 the logs.nsnoc.com server failed at 2pm this afternoon.

The failed supply caused the trip switch to the rack to trip to protect the equipment. The UPS maintained service to the remaining servers for 15 minutes but it took longer than anticipated to identify the issue and gain access to the building.

An engineer was on site at 3pm to identify the issue and start the process of finding the server that caused the initial trip.

Power was restored to most servers by 3.30pm. We worked through all of the services to ensure that everything was working and by 5pm concluded that all remaining bugs had been resolved.

During the process we identified that one of the core switches had been incorrectly powered from the rack B UPS so the failure impacted on some services from other racks. This issue will be resolved on 4th January when the office reopens.

At this moment in time, the logs.nsnoc.com server is offline until a new power supply can be installed on 4th January. We will process the logs from today onwards once the server is working again.

All other services have been tested and confirmed as working. Please notify support @ nsnoc.com if you have any other issues.

MySQL1 server

2009-12-19T06:57:00.003Z

The Mysql1 server has crashed and failed to reboot this morning.

An engineer will be on site for 8.30am to investigate the issue further.

UPDATE

A second reboot of the server brought services up without errors. We will have standby hardware available for the server as a precaution over the Christmas holiday.

If customers are aware of any issues following the reboot please raise a support ticket in the usual way.

Network cable update

2009-11-19T17:12:00.002Z

We have a number of network cables which need to be re-patched over the course of Friday.

Network customers may notice a few seconds instability while the patch leads are moved.

Host Server 8

2009-11-16T14:35:00.004Z

Host server 8 has now been upgraded to PHP5 and Mysql 5.

We have migrated all databases and sites to the server and the service appears stable.

There are however some points :-

- If your hosting service has expired prior to June 2009 then your site has not been upgraded and is not live. Please contact us to renew the service and re enable your site.

- A small number of sites have databases with some locked/crashed database tables. There is nothing to suggest these are active tables or that the issue has only recently occurred but if you do have a table that requires a repair please let us know.

- The ftp server on the new host is being very 'Monday' and not authenticating requests at the moment. Assuming nothing major crops up after the migration we'll work on the fault over the remainder of the afternoon.

UPDATE 15:30

FTP accounts on the server are working without issue.

Support will run to 11pm via email this evening to resolve any remaining issues with users.

Web Cluster maintenance

2009-11-10T16:41:00.005Z

We will be performing mantenance on the web cluster from 5.30am tomorrow morning.

The following services will be taken offline, Cluster 150, 156 and 212.

We are anticipating that work will be completed and full services restored by 8am with a possible overrun to 8.30am at the latest.

While the work is being completed sites will show an under maintenance page.

The work is necessary to allow us to migrate to a new storage platform and add support for php5 to the web clusters

UPDATE 7AM
IFL have failed to renew the access pass for the engineer carrying out this mornings scheduled work. The pass expired 2 days ago thereby preventing us from completing the migration according to our original timetable.

We have no alternative dates to complete the work so it will commence at 7.30 am this morning with a target completion of 9.30am.

UPDATE 11AM
We're seeing high traffic levels at the file server caused by a single site. If services do not settle to an acceptable level within the next 5 to 10 minutes we will disable the site and look for stability to return to the servers.

UPDATE 1.30PM
We have been able to confirm the issues seen after the file server was returned to service this morning we caused by a single site. The site in question has been disabled until we can migrate it to alternative hardware.

We will be monitoring the servers for any further issues for the rest of today.

Phishing Emails

2009-10-20T13:45:00.002Z

We've seen a batch of fishing emails arrive with customers today asking them to update contact details as part of a security update. We remind customers to be wary about clicking links in emails even when they appear to come from known sources.

Web cluster stability

2009-10-19T14:23:00.003Z

We've been experiencing some instability with the web cluster 150 servers over the last hour. The cause has now been traced to a single site causing a the server to run out of memory.

The site in question has been disabled pending further investigations.

Mail Cluster Issue

2009-10-12T20:47:00.011Z

INITIAL 9:45PM

We're currently repairing some file corruption on the main mail server file store.

The process should not take more than a few minutes. Until the repair is completed there is no access to POP, IMAP, or Webmail services.

UPDATE 10:15PM
The file system check and repair has completed however a server reboot is required to clear a number of Dead/Zombie processes on the server.

We're going to assess the issues and decide on a course of action shortly.

UPDATE 10:50PM
An engineer will be dispatched to IFL2 at 3am to recover the file server to our new racks. We hope to have the service restored shortly after 8am.

UPDATE 8.15AM
The overnight work to recover the file server to our new racks was completed on time and without incident.

UPDATE 11:45AM
Heavy storage loads have resulted in the need to pull the file system and force a tree rebuild - this is going to be a lengthy process, so we are working to restore a backup in the interim with an ETA of a fifth of the time before inboxes are present. AGAIN I would like to reiterate this is not preventing inbound mail from arriving, merely the ability to collect it, an important difference.

UPDATE 13:38
Working on a solution to deliver / present fresh email to the end users. While historical data will be missing, all new emails will be arriving and visible - allowing us to migrate these to the existing file system once rebuilt. Light at end of collective tunnels.

UPDATE 13:45
Temporary solution in place. POP users will now be seeing all their new email arriving. Those who are paying for the IMAP solution will notice that all their mail has gone. DO NOT adjust your set. Simply work with what you have for now, and your inbox will be reunited with old mail on completion of the rebuild over the next 12 to 24 hours.

This completes updates on this entry, work from this point on will continue in a new entry. Over the last 10 minutes, over 467Mbytes of inbound mail have been stored, and we can see you retrieving email.

NB, if you use webmail or Imap to collect mail, please avoid creating new directory structures until your old mail account has been merged with the new server.

UPDATE Wednesday 11.30am
Work is continuing on the old file server. We are currently running a snapshot of the data to backup before starting the merge with the new file server. Further updates will follow at 2.30pm today.

UPDATE Wednesday 2.30am
The snapshot of the data is now 1/3rd complete. It's likely to be early evening before we can start the process of merging the new mails with the old. Next update 5pm

UPDATE Wednesday 5pm
The snapshot of the data is now 3/4 complete. We're expecting to be able to start the data merge at 9pm which should take less than 1 hour to complete. Next update once data merge has been completed and all webmail/IMAP data has been restored.

UPDATE Thursday 1am
Our new file server is now live with a full copy of all mails, old and new. At no point has any mail been lost or rejected.

There is a small risk that the mail indexes will have changed for IMAP users resulting in a mis match between the From/Subject line presented by your mail client and the mail downloaded. The fix is to right click the folder concerned and select 'rebuild index'

Thursday Technical Support

2009-09-09T21:49:00.002Z

Our key staff will be attending a conference from 7am to 1pm today (Thursday)

For the time we are away, we will be operating a reduced support service and no telephone support.

Emergencies will be handled by on call staff.
General support will resume early afternoon.

Network Stability 12:40pm

2009-08-26T11:44:00.002Z

A transit customers server has been subject to a DDOS causing in excess of 100Mbit of small packets to traverse our network. As a result there has been a disruption to connectivity and some packet loss.

Steps to mitigate the problem have been taken as as of 12.44pm stability has returned to the network with no packet loss apparent.

Cluster 150 - 5.15pm

2009-07-28T16:22:00.003Z

It would appear that a customer wordpress site has caused the web cluster to overload. We're currently waiting on the load to fall on the web servers to confirm the cause and to restore service.

We expect the servers in cluster 150 to have recovered within the next 5 to 10 minutes.

An update will follow shortly.

UPDATE - 5.25pm

Services have been restored and will be monitored over the course of the evening. The service restart prevented us confirming the exact process details that had caused the server load to climb.

"delayed emails" / duplicates.

2009-07-20T08:41:00.001Z

Friday saw a scheduled file system check of a mail cluster member. It would appear that a complication with files not being deleted properly after being cleared from the spool queue had arisen, and as a result on recovering these files to a state in which they could be deleted - the mail server diligently sent them on their way. The resulting phanton resending of emails - in some cases up to a month old.

As these mails had been marked as deleted, these mails are DUPLICATES, and these and can safely be disregarded.

We apologise for the confusion this may have caused.

Database restart

2009-06-08T15:56:00.003Z

The server hosting our main database has crashed and is in the process of being rebooted.

Logins for Mail/POP/Imap will be offline for a short time while the reboot completes.

UPDATE

The server has restarted without issues. All services are working normally.

Mysql1 Server Restart

2009-05-20T11:05:00.002Z

Mysql1 crashed a few minutes ago. It has been restarted and appears to be working normally.

Connectivity Issue

2009-05-17T14:33:00.006Z

We are aware that customers have been having issues connecting to sites on our network since this morning.

The issue has been traced to a hacked customer server which was pushing out denial of service attack. We will monitor the network over the next 24 hours to ensure that we can maintain a degree of stability.

UPDATE 19:00

It has become apparent this afternoon that a second denial of service attack was hitting our DNS servers ns1 and ns2. Customers will be aware of the importance of the name servers. Not only do they allow remote visitors to find the IP address of the server, they're also used by the web servers and Sql servers for performing reverse lookups.

The result of the attack has been to slow down all services including DSL/transit due to the high volume of UDP traffic hitting our routers.

We believe that at this stage we have been able to take action to reduce the effects of the Dos attack. We will be monitoring the network this evening and taking appropriate action as needed.

UPDATE 19:50
Network monitoring is showing that services have returned to normal. We will continue to monitor the network and services over the next 24 hours.

UPDATE 21:00
One issue with some static transit routes has been resolved. There are now no outstanding issues.

ADSL port change

2009-05-16T13:21:00.003Z

Our ADSL traffic has now been migrated from it's temporary port in our router back to it's permanent port location. Our engineering investigations on the link are not yet completed but we now have sufficient information to be able to identify the cause of the issues with the circuit provider.

Traffic is already flowing over the link so we are satisfied that the migration has been completed smoothly. The loss of IP connectivity lasted for 90 seconds during the cable transfer.

DNS server restart

2009-05-16T10:31:00.001Z

Both of our dns servers required a restart this morning causing some inconsistency with the resolution of domain names.

Maintenance Window

2009-05-13T20:51:00.005Z

We will be bringing our second router back online between 11am and 1pm on Thursday 14th May.

ADSL customers may experience a brief (60 second) outage while cables are re patched.

No other issues are expected as a result of the router replacement.

UPDATE 14:00

The work with the router replacement has taken longer than expected due to a faulty APC distribution switch causing our main load balancer to reboot. The result being that sites on clusters 150, 156 and 212 have suffered 3 outages in the last hour lasting up to 10 minutes.

Work is nearly completed to complete the router installation. Further updates will follow.

UPDATE 18:00

The work to replace router 2 was completed earlier this afternoon but is not yet in use. We will shedule out of hours work to enable the upstream connections on this router to reduce disruption during the settlment of our routing tables.

We have identified two issues during the work today.

An APC power switch which supports the load balancer for 150, 156 and 212 appears to have a loose connection which causes the load balancer, servers 29, 37 and 64 to reboot if the cables are moved. We will shedule another maintenance window to transfer the power cables to a new APC unit. As yet a date has not been set.

The second issue relates to a fault with a link to IFL1 which carries our ADSL customer traffic. The work on the new router today has highlighted a potential fault which we have raised with the link provider today. When the issue has been investigated and resolved we will shedule an out of hours maintenance window to transfer the connection from router 1 to router 2.

Mail, repeated downloads.

2009-05-13T14:09:00.005Z

We are noticing that a number of Outlook Express clients are experiencing issues. Downloads appear to be looping.

To rule out any corruptions to the upstream cache at the mail server, the local index has been removed. While this restructures there will be some delay.

If your mail client is repeatedly downloading email please take it off line. Then remove or reset any local cache and use the webmail facility in the interim as repeated downloads are currently forming a denial of service.

UPDATE 21:45

The load on the mail servers and file system is now down to within normal levels.
There should be no delays on mail delivery or slow mail downloads.

We would like to remind customers that they must set their mail clients to delete mail after a set number of days to prevent their mail clients from downloading mail repeatedly.

ADSL outage not directly related to router issue

2009-05-11T12:34:00.000Z

We are currently very aware of the ADSL issue remaining.

We have had confirmation that the system delivery is sound, and our infrastructure is ready to delivery the routing and transit, however we are no receiving the inbound traffic.

This issue is now clarified to be unrelated to the router outage of later Saturday night.

We are pressing / waiting on switch/circuit owners between our IFL1 and IFL2 installations in Manchester to pick up the ball.

Router Reboot

2009-05-09T14:31:00.002Z

Router 2 was rebooted a short time ago causing a brief interuption to network stability.

We are waiting on a new backup transit provider to deliver connection details to improve the reliability of the network and hope to have a new connection for router 1 completed by Monday.

UPDATE

The router has crashed twice and been rebooted twice. We're unable at this moment to identify the casuse of the crash but suspect environmental issues at the colo facility to be the cause of the issue.

We are working to migrate some of our backup transit routes to the remaining working routers to restore full routing to our network.

Further updates will follow.

Network Issue

2009-05-03T17:30:00.004Z

We are currently investigating what appears to be a network issue. Further information will be made available as we have it.

UPDATE

It would appear that we have a rouge AS advertisement being made by an ISP not providing service to us. we are attempting to work through the various channels to have their network management centre correct the issue.

UPDATE

A ticket has been raised with the noc of the ISP concerned we are awaiting the results of their investigations.

UPDATE

The issue has been identified relating to a single upstream provider so engineering are now working to ensure that the fixes are put in place with our supplier network to restore connectivity via that route.

Our network stats are showing working routes via two out of three upstream providers and all of our Manap peering. The faulty connection accounts for approximately 35% of our total capacity, which because of the nature of the fault, is not being re routed to the working connections.

A final update should be available within the next 15 minutes once full service has been restored.

Master Mysql Server

2009-02-27T12:41:00.004Z

We are investigating an issue causing the main mysql server to reject new connections from the web servers. Further updates will follow once we have identified the cause.

UPDATE

The service has been restarted a few times now but with no apparent resolution to the issue. We are continuing to look for the cause of the issue.

UPDATE

The issue with the mysql server has been tracked back to a faulty connection with one of the 4 dns servers. All services have been restored and are running normally.

LVS load balancer rteboot

2009-02-09T19:15:00.003Z

The LVS load balancer has been rebooted after a crash, the downtime was 14 minutes.

All services appear to be running normally.

Mail server issues

2009-02-03T11:24:00.002Z

We are currently investigating two issues with the mail servers.

1) Possible authentication issue for sending mail

2) Failure of the file server holding pop/imap and webmail.

A reboot is being undertaken of the file server and we are tracing the cause of the authentication issue for sending email.

Further updates to follow.

UPDATE

No issues with mail authentication have been found, inbound and outbound mail are working normally.

The pop/imap/webmail file server is having a disk check performed before we bring it live again. That should be completed shortly.

UPDATE

Mail storage services are all restored and working again. We have identified a single domain that has been mailbombed so are taking steps to reduce the impact of the attack on the mail servers.

All customer mail services should be working normally now.

DNS: Purge with flame

2009-01-20T12:36:00.000Z

We have had some issue over the past hours with DNS reliability.

We have taken the harsh step of sanity filtering the DNS zone records, removing invalid entries that contain chars outside the scope of [a-z|A-B|0-9|.] within the primary tuples, and with the added inclusions of things like = and : in text records (under 128 chars). The majority of these where in the form of trailing spaces and people putting things like http:// in as a cname.

Clearing these has resulted in a far better DNS platform in terms of throughput and reliability.

Please take this opportunity to check over any domains that are causing you issue today before contacting us with support queries.

Webmail / POP / IMAP

2009-01-06T15:33:00.004Z

A recurrence of the earlier issue has re emerged, we are currently working on a longer term fix. This will be effecting a small section of the users. For those it does effect the inbound mail servers are not affected in any way, however the inboxes are currently offline - so your connections will either time out or fail. We are aware of the situation and will update this entry once as soon as the issue is resolved. This will not be effecting the majority of users.

[update 1620] The back end for those effected, while struggling initially with the load of retries has settled down and is now servicing requests without issue.

Webmail / POP / IMAP

2009-01-06T01:46:00.003Z

The file server which supports the webmail / POP and Imap servers is currently suffering from a crash. IFL support staff will be rebooting it shortly at which point we hope to restore services.

The next update will be 9am.

UPDATE
The server has been rebooted and checked. All appears to be well so webmail/POP/Imap services are now working.

Any mail delivered in the last 5 hours will be stored on our inbound mail servers. It should be delivered to the mail accounts over the next few hours.

IMAP issue this morning.

2008-12-15T10:30:00.003Z

An IMAP server this morning has decided that being busy and slow is not its thing, and it has had enough. Having checked out we are running some tests before bringing it back on line. If your IP address is being diverted to this machine, you will be seeing connection errors. Expect a request for U&P when it comes back up and all will be well. We envisage about 10 mins tops of downtime. Many thanks for your patience.

Inbound email

2008-11-20T09:50:00.004Z

When it rains it pours.

We have located another issue with one of the primary MX cluster members, and the way another member uses DNS - these occured over night (19th-20th November).

These have been addressed, and we are seeing mail processing in at an expected rate given the time of day.

Please do keep in mind the way the mail RFC's work, and that unless you have asked the admin at the other end to force a flush, mails retry with a logarithmic delay, so be patient, and new mail may arrive before ones sent less recently. This is the way it is meant to work to assist with resilience and spread the load.

We are sorry for the delays, and are happy to say that there are now no issues.

SMTP

2008-11-20T08:58:00.004Z

Outbound email is currently being delayed.

Client abuse has resulted in a huge spam list that we have successfully cleared from the queue and contacted / blacklisted those involved.

Trying to clear the backlog appears to be being hampered as - quite logically users are sending themselves test messages, or mails that have not arrived have been resent. While this may make sense on the scale of one person, upscale this a couple of hundred times, and there is a fair amount of mail happening that doesnt need to.

We are aware of the issue, and we are processing the queue as well as we are able - once the queue has returned below 8K mails, we are confident it will start to recover of its own accord. Bear with us, and we will update you once we have the green light.

[update 0955] Blaming the tail end of an issue yesterday for this mornings issues quickly became a non starter. We have located the issues and made good. I have posted a new article to cover what happened.

Load balancer Issue

2008-08-02T11:11:00.001Z

The primary load balancer failed this morning.

Service has been reinstated and tested.

We are sorry for any inconvenience caused to the effected users.

New & recently changed email accounts.

2008-07-24T10:21:00.003Z

We are aware of an issue with slave replication that we hope to address today. This is effecting some email accounts that have been changed or created recently. As this will require replacing the out of step database, and then bringing it back up to speed, it will be a requirement to reduce service before restoring it to full strength. Every effort will be made to minimise further complication.
[update 1225] - this has been addressed, and is now up to speed - thank you for your patience.

Issue with cluster 1

2008-07-02T15:19:00.002Z

We are currently experiencing an issue with web hosting cluster 1. Cluster 2 and 3 are operating within tolerances if a little slow.

We are currently looking into storage issues, as yesterday an air conditioning failure in the data centre IFL2 resulted in a sizable heat spike for a prolonged period.

UPDATE 17:00

The cause of the issue has been traced to the file server and the underlying cause identified as an overheating CPU. We believe this is associated with failure of some air conditioning units in IFL2 yesterday afternoon.

We will be working on the racks over the next week to improve air flow through the servers as well as increasing our range of monitoring sensors to include more temperature sensors on servers.

Webclusters 150 and 156/212

2008-06-10T17:49:00.005Z

It has become apparent that the work on the database server today has not resolved all of the issues on the above clusters. While the service is working better than it was earlier, the process list on the web servers is continuing to climb to the point that new connections are not permitted.

At this point we have eliminated the possibility that the fault is within the LVS load balancer (that was replaced this morning) and the SQL server (replaced 1pm with a new server). We have also eliminated the network as the possible source of the issue as virtual servers and the mail service are working without issue.

The only other element of the service which is now in question is the nfs file server. While there are no obvious errors being produced we feel that it is the only possible cause of the issues left. A new file server is in the rack and we have just begun the process of transferring of data from the old server to the new. We expect that to be substantially completed within the next 4 hours.

UPDATE 8PM

The transfer of files to a new file server is underway and proceeding without issues. The web servers have been pointed to the new file server. Files are being restored from a to z so sites starting a and b have already been migrated. Judging by the first hour of transfer, we expect the process to complete in the early hours of the morning.

We would like to thank customers for their patience during this time.

UPDATE 1AM

We are approaching half way through the transfer of sites from the old file server to the new. We expect the remainder of the process to be completed by 6 to 8am.

Webmail services have been restored and are working without issue.

FTP access to the new file server will be suspended until mid morning Wednesday.

UPDATE 7AM

The file transfer is still running with about 75% of sites completed allbeit very slowly. Clients with sites still not available can email our support email address with any sites not showing so that we can push them by hand. Priority will be given to business sites.

UPDATE 11 AM

All of the remaining sites should have been resored within the next 2 hours. Once that is complete, ftp access will be made available to the new file server. Customers will not need to change any settings in their ftp clients.

UPDATE 2PM

All transfers are completed and there appears to be stability at last. A few bugs with sites have cropped up during the process but they have been ironed out. If you are aware of any site which is not working correctly please raise the issue with our support mail address and we will investigate it.

In summary,
Mysql has moved to a new server, no changes required from customers.
File server is on new hardware, no customer changes needed.
FTP service is up and working, no change to FTP settings needed.
Webmail up and working.

We would again like to thank our customers for their patience during this issue.

Web Clusters 150, 156, 212

2008-06-10T07:32:00.004Z

We are currently dealing with an issue on all web clusters whereby requests are not being serviced correctly resulting in slow page load times.

Our investigations over the last 12+ hours have drawn a blank with the result that there is no apparent reason for the poor performance of the web servers.

We are in the process of bringing in outside support this morning to double check our own investigations and are in the process of installing a new file server.

At this point, we have tried to eliminate all of the common points of failure in the web cluster service. The file server appears to be operating correctly but as it is such a major component in the delivery of the web pages, and given that we have no other potential sources for the problem we have decided to replace the file server as a precaution. We expect that work to be completed by lunchtime today.

UPDATE 9.40am

Further testing has identified that the bottleneck appears to be the SQL server not the File server as previously thought. Work to replace the file server has been suspended for the time being. We have a standby version 5 SQL server and are currently investigating the issues involved in making an upgrade from the current mysql4 service to mysql5 on the new server. If the issues appear to carry too high a risk, then we will prepare a Mysql 4 server. This work is expected to last for the next 3 hours.

UPDATE 1pm

The database tables and indexes on the existing sql server have all been repaired but this has not given us any significant performance increase. A new Mysql 4 server is being prepared to take the databases. Next update 2pm.

UPDATE 2PM

The new database server should be running within the next 10 to 15 minutes.

UPDATE 3PM

The main database has now migrated to a new multi core host server. Initial tests are showing that the loads on the web servers is now more stable with pages being served within normal times.

We will continue to monitor the service for the next 24 hours to ensure stability has been returned.

We would like to thank our customers for their patience during this outage and offer our sincere apologies for the intermittent service over the last 24 hours.

VPS/ vserver clients.

2008-06-09T15:42:00.001Z

VPS/vserver users please be aware that unless you specifically have a managed account with us then it is your responsibility to keep the server as secure as possible.

We have had two incidences already this week where clients have failed to keep on top of patches and updates that has resulted in machines being taken off line after they have been used in attacks on other networks.

If you have any questions regarding this or could do with some advice, then drop us a mail to support and we will assist as much as we can - we would prefer you to be on top of the situation as opposed to it coming as unwelcome news.

Network congestion

2008-06-09T15:39:00.001Z

We are currently experiencing an inbound UDP flood.

Once we have isolated the source we will be able to deal with it and return services to you. Technically the services are unaffected, however while the switches are so busy service will be slow or time out.

Phone Support

2008-06-02T11:11:00.001Z

We are currently moving offices.

While we are back online and answering support we are afraid we are yet to have our phones re routed.

In the meantime please continue to use email support (support@nsnoc.com), it is our preferred method of contact - we will post here again to update you on the phones status as and when we are able.

Mail collection for some users.

2008-05-19T07:58:00.003Z

We are currently working around a poorly timed drive expiry - some users may be experiencing an issue with mail collection. We are aware of this. No inbound mail is being lost, however you will currently be unable to get to it until we have repaired the drive; this includes the webmail facility. For the users this involves the data has been set read only to prevent damage to the inboxes. We will update you as soon as we are able.
[Update 0934]
We have recovered the data, and are brining live mail collection within the next five minutes. Inbound mail will be live hopefully within the next 20 minutes. Differences between restores and live will hopefully be bought up to speed within the hour.
[update 1030]
The inboxes will commence updates from inbound mail now. You may still find historical mail arriving in an odd order for the time being as retries are serviced.
[update 1147]
Users who had PROCMAIL configurations will now find that these are functioning again.

POP mail collection

2008-04-29T09:55:00.001Z

We are experiencing an issue with the popping service. This is not effecting those who have paid for an IMAP service.

Earlier sending issues where down to a single user the relay service as a mass mail host. Their IP has been blocked, and further action is being taken currently.

We are currently unable to give an ETA on the POP issue, however we are both aware of it and working on it.

[update 1116]
The issue is fixed, however the server is playing catch up. Please ensure you are not checking your mail every minute; step back to every ten to allow the status quo to return. Thank you.

Critical incident.

2008-02-18T14:35:00.008Z

Yesterdays critical incident has been narrowed down to the activities of one server, and one VPS machine.

An apparent distributed denial of service attack took place, possibly with what also looks like a MAC flood against a connected switch device.

Other devices then shut off connectivity to the flooding switch presenting as a loss of device. Replacement of device did not resolve the issue. Realisation of the intended target and steps taken to minimise the impact reduced the overhead, however removal of the target machine resulted in an immediate return of service.

Fall back service provision will be removed during the course of the morning to return to full primary service. We do not envisage more than a momentary loss of service during this fail back most likely presenting as an error to email clients.

Thank you for resisting the temptation to call in during this period

POP connections

2008-01-10T10:07:00.000Z

A faulty authentication database has been repaired this morning. A subset of POP mail users will have had issues collecting mail for the duration - the issue having surfaced at around 2am this morning - and become more obvious as loads built up. The issue is now rectified.

Mail delivery

2008-01-09T09:52:00.000Z

We are currently experiencing issues with the scanning mail servers mxs1 and 2. We are working on a remedy to this currently, and will have it back to normal as quickly as we are able.
However in the meantime please resist the temptation to phone and request information on this issue, or worse still, mail in causing more of a backlog to clear. Many thanks
This in no way effects your ability to collect email.

Failure to collect

2008-01-08T16:57:00.000Z

Authentication is currently failing for mail collection. We are aware of this, and are trying to remedy the issue - please do not call in so we are able to progress.
What this means is that you are unable to collect email - however processing of inbound email is unaffected.

POP Logins

2007-12-24T14:16:00.000Z

One of our 3 pop servers suffered an issue with its firewall which resulted in it blocking traffic from the LVS load balancer.

The issue has been resolved and should not cause any further error messages when customers try to download mail.

Christmas Support Hours

2007-12-21T10:25:00.000Z

From Saturday 22nd December through to Tuesday 1st January we will be handling support via email only. Emergency support via email will be available on 25th and 26th December. Our offices reopen on Wednesday 2nd January 2008.

As normal on call engineers will be available at all times for service related issues.

I would like to take this opportunity to wish all of our customers a happy Christmas and a prosperous new year.

Seasons Greetings from the Netserve team.

2007-12-14T15:48:00.000Z

Speculative Registrars advisory

2007-12-05T15:46:00.000Z

Keep your wits about you

We are seeing an increase in Spec Reg (speculative registrars) either offering to sell you a domain similar to yours, or even more amusing, reminding you of an expiry and offering to renew it. If its not in an email from us, then its not the real deal. People do fall for these otherwise they would not bother with them. As for people trying to sell you a domain - this is an insight into what happens when domains drop back into the public domain, and they will, from our experience try to sell you the most ridiculously cumbersome ones, so if you have a nice short name that draws traffic - be warned.

Here we have the DRoA (Domain Registry of America), they like to send letters, but here is an email as an example - they are nothing to do with us.

From: Transfer and Renewal Support [mailto:transfers@droa.com]
Sent: 27 November 2007
To: [deleted]
Subject: [deleted].com; Order # 1234567

Hello [deleted],

The transfer and renewal of your domain name, [deleted].com, is not yet
complete because your domain name is currently in a "REGISTRAR-LOCK" status
with your current registrar TOTAL WEB SOLUTIONS LIMITED TRADING AS
TOTALREGISTRATIONS

In order to complete the transfer and renewal, the "REGISTRAR-LOCK" status
needs to be removed and an EPP Key/Authorization code needs to be obtained.
Please see below for instructions on achieving this.

Solution:
- Log into your account with your current registrar, and change the status
of your domain, [deleted].com, from "locked" to "active".
- Then look for the EPP Key or Authorization Code.

- Alternatively you may call your current registrar, TOTAL WEB SOLUTIONS
LIMITED TRADING AS TOTALREGISTRATIONS (see phone number below) and ask them
to remove the lock status of your domain name, give you your EPP Key and
allow the transfer to Domain Registry of America.

- Once done please notify us that you have done so, by calling our toll-free
number below. We will then re-attempt the transfer and renewal of your
domain name.

As a convenience, we have supplied your current registrars phone number
below.

Domain: [deleted].com
Current Registrar: TOTAL WEB SOLUTIONS LIMITED TRADING AS TOTALREGISTRATIONS
Registrar Phone Number: Please visit their site to contact them

Transfer Department
Domain Registry of America
Toll free 1-866-434-0212 or for International Callers, dial +1(905)479-2533

.... and here we have a choice, yet more amusing example of why you should keep ontop of domains you like.

Recently, [deleted].com expired and we acquired it
in a domain name auction. Since you own the .net or .org
version of this domain name we wanted to provide you
with the opportunity to own the preferred .com version.

Our company specializes in recovering preferred expiring
domains and either selling them to individuals such as
yourself or building out our own web presence on those
valuable domains.

[deleted].com is a pretty darn good domain name and,
the truth is, the .COM is a far stronger version of the
name than then .NET is.

* .Com is the strongest brand on the internet.
When people think of a website, they intuitively
think ".com". Odds are people trying to get to
your website are inadvertently going to [deleted].com
because they assume that's where they can find you.

* .Com conveys Professionalism that .net & .org
cannot match.

If you'd like to own [deleted].com, you can buy it
now by covering our acquisition costs and a modest profit.

If you have any interest I encourage you to act quickly
because this domain name will only be offered for sale
for a limited time.

Click Here For Pricing On [deleted].com

https://secure.claimthisdomain.com/buy.php?[deleted].com

Please advise,

If you do wish to transfer to these people, or wish buy back a domain or whatever, just make sure you understand what their terms and conditions are, transfer limitations, costs and the like.

If you want to transfer away, give us a call, or drop us a mail and ask for our impartial advice; its always good to err on the side of caution.

Relay server

2007-12-04T09:33:00.000Z

We are dealing with an mail relay issue. We will have the service up and running as soon as we are able. We have had an influx from a spamming attempt, and are taking steps to remove these from the live queue and locate the source and take steps to prevent this flow before restarting the service.

[Update 1003]
The subsequent work has revealed an underlying file structure issue with a host machine serving data for a number of back office services. We have taken these offline temporarily to fix the issues before bringing those back online. We are sorry for the inconvenience - we should be back on within a few minutes.

[update 1030]
We have pulled the drive in question, and restored services from backup to other hosts. Primary services have been restored, and will be moving to tidy up any lose ends now until we can get new hardware instated.

Forwarding to BT

2007-11-30T11:59:00.000Z

We have had a few cases now where the nice people at BTinternet/Yahoo are delaying mail that is being forwarded to accounts with them.

Needless to say there is little we can do about this. If you are set up with a forwarder from your domain to a BT account, and mail is not getting through or is delayed, please contact them and ask for support on the matter.

In the meanwhile the sender may be receiving mails from our mail servers telling you that they have been unable to deliver so far; this is not our issue, they are just not accepting the mail.

Load Balancer reboot

2007-11-17T00:47:00.000Z

Our LVS load balancer required a reboot this evening. Sadly our monitoring and sms alerting provided by Tmobile failed to notify the oncall engineer of the issue for over an hour. Once the alert had been issued we were able to restore service within 10 minutes.

Services with issues incude, hosting on service 150 and 212 as well as pop collection from pop1.nsnoc.com

Temporary outage remedied.

2007-11-10T11:10:00.000Z

A short outage was alerted and remedied this morning. This involved a load balancer, and for those users that were involved, connections to the cluster one and three and mail collection will have been noticed. This has been remedied for the effected inbound users traffic.

During this period inbound email was unaffected for all users.

Rack failure

2007-11-01T19:48:00.000Z

We have experienced a complete power failure in one of our racks at the IFL data centre. This occured at the PDU breaker as opposed to in rack, and we are in the process of assessing damage and bringing up services as it is safe to do so.

This has taken out a number of primary services, however we are aware of the situation there is no need to mail in.

We will post progress as it occurs, and if you are experiencing issues above and beyond what is reported as fixed, then please do mail into support@support.nsnoc.com . Thank you.

[2209 Update] We have returned at this point domain name resolution and the primary of the three clusters. We are continuing to work on the situation both on site and remotely and will post more as we can. SSL server services have been returned.

[2240 Update] We have returned mail collection services and mail relay services. Inbound mail from external sources has not been effected. Clusters two and three are back in service delivering web traffic.

[2330 Update]
All core services have been returned to normal. We currently have 5 servers offline hosting about 30 customers which we have tracked back to a failed APC power distribution unit. It is unclear whether the unit failed causing the trip switch to cut power to the rack or whether the unit failed as a result of a power distribution fault. We are making arrangements for the unit to be tested and replaced if necessary as soon as possible.

Relay/Sending SMTP server failure

2007-10-23T18:38:00.000Z

One of the core SMTP relay servers relay.nsnoc.com is currently out of action and waiting on an egineer assisted reboot.

Further updates will be provided shortly.

UPDATE 19:54

Services have been restarted and will be monitored for the next 24 hours. The server had failed to reboot due to a some file system corruption.

Congestion

2007-10-05T14:49:00.000Z

We are aware of climbing loads on the mail cluster. This will not cause any issues with transit at all, however you may be experiencing slower responses than usual for those using the pop1.nsnoc pool this afternoon. This should continue to die off as the afternoon continues.

mySQL update

2007-09-26T14:06:00.000Z

We are planning to move mysql1.nsnoc.com to from 4 to mySQL 5 on Wednesday 10th October (two weeks time).

We are currently testing the platform we are moving to, and will be releasing details on how you can test your dev / staging code against the installation before it goes live shortly.

Any questions on the matter please contact support.

Load balancer event

2007-09-26T09:39:00.000Z

We have experienced a fault with one of our load balancer. Taking it out of service users will have experienced sporadic connectivity to the mail server for a five minute period during the diagnosis and change over. Some web sites will also have been affected for this period.

We failed over to the stand in load balancer as soon as we were able, and will be looking to switch back to the main box as soon as we are confident the issue has been dealt with.

Plus.Net and Forwarding

2007-09-21T12:43:00.000Z

As an advisory to Plus.Net users - it would appear that for some reason they are classing our primary secondary and mail scanning mail clusters as "Dynamic Addresses". While this is not the case, it is not preventing them from currently rejecting mail forwarded to accounts on their servers from our clusters.

Feedback has been along the lines of:

[snip]I'm sorry to hear you have been adversely affected by changed we have made to our email delivery system. The changes were designed to reduce the amount of spam our network receives. This is a generally accepted practice in the industry.

No hosting service should be using dynamic IPs to forward mail, you might want to discuss this with them. I have however, passed on the info you have provided, to see if we can make allowances in this instance.[snip]

We are all behind the less spam option and use many restrictions on our own scanning and non scanning servers - however this misapprehension that we are on a dynamic range will cause users with this ISP who usually use our forwarding service issue until they get their block lists corrected.

We will update this with more information as we get it.

[Update 1605]
The following URL was supplied to us by a client for the PlusNet forum. This surely cannot be the case since this is not entirely practical especially with load balanced clustered systems. We await some more confirmation.

[Update Monday 1052]
Thank you to Bob Pullen from Plus.Net for getting in contact with us to remedy this issue so swiftly. Apparently their list was showing the block in question being largely dynamic, and this has now been corrected.

Office Holiday Closure - Wednesday 29th August

2007-08-29T11:20:00.000Z

Our offices are closed today from 12 noon due to the marriage of our senior technician, Anthony. We will reopen for business 10am on Thursday 30th August.

In the mean time, limited email support will be provided for emergency cases only. On call engineers will still be available 24/7 for service related issues.

On behalf of all of the staff of Netserve and Designweb we would like to wish Anthony and Cally all the best wishes for their future together.

Webmail down

2007-08-24T12:07:00.001Z

Webmail has been taken down for maintainence, we plan on returning it as soon we as we able.

This in no way effects the ability to collect mail from desktop clients, remote webmail clients, or the transit of mail in and out of the service.

Some delays

2007-08-22T16:46:00.001Z

We are experiencing a delay on some mail that has been passing over one of our inbound email servers. We are aware of the issue and working on it - due to the clustered nature of our setup this should effect a small percentage of mail only. Thank you for your patience.

Sending mail / SSL

2007-08-22T08:56:00.000Z

We are currently having issue with a number of core systems, what appears to be a denial of service is effecting relay.nsnoc.com and one of the SSL servers. We are aware of the situation and working on it, please resist the temptation to phone in - we will return service as soon as we isolate the cause and make good.

[Update 1029] An automatic defensive shutdown occurred to a server that was being attacked from a massive influx of traffic. The issue was identified, and rectified. Service has resumed.

Manchester to London Transit

2007-08-17T09:55:00.000Z

Customers have made us aware this morning of connectivity issues from selected ISP's within the UK, which appears to have started at 8am this morning.

Investigation into the matter has proved difficult but has now been traced to one of our upstream providers advertising network routes to our network but then failing to deliver the traffic.

Under normal circumstances where a router connection fails, the advertisement to that route is removed automatically. The new routing table should then be updated to other routers that peer with it thereby preventing traffic being passed off to routers which cannot handle it.

We have closed down the peer with the London provider concerned and have raised the issue with them. When we have an explanation for the failed routes we will post further information here.

We would like to apologise to customers affected by this partial routing outage and the length of time it has taken to identify the source of the issue.

*** UPDATE ***

Further investigation suggests that the issue was restricted to a single /25 subnet used for routing transit to three customers and may have started some time after 11pm last night.

At this stage it appears that a reset filter on a single interconnect may have been the cause of the issue.

Switch of balancers

2007-08-16T07:58:00.000Z

we have encountered a partial failure of our backup load balancer this morning. So while services have been running without interruption, there has been no means to reach them.

The switch over has taken place back to the primary from the backup, and the opportunity has been taken to make some adjustments to the configuration. We are sorry for the inconvenience for those of you in the office too early this morning. This was rectified prior to 9am this morning.

Advisory: Reminder of renewal emails

2007-08-07T14:19:00.000Z

In addition to the DRoA - we have a new kid on the block of thoughtfully reminding you that your domain is about to expire.

While you are free to move your domain over to whoever you wish, you are best reading all the print on these kinds of emails, and looking closely at their terms and conditions. We have recently had a number of clients who have accidentally moved away and lost control of their domain.

Please be sure that the mail you are dealing with is in plain text from either DomainCity, or DesignWeb - or that you have compared the pricings T&C and what you are agreeing to by clicking the link.

Here is an example of one such mail.

The Domain Registry of America and the Domain Renewal Online are in no way related to us.

Load Balancer Issue

2007-08-06T11:51:00.001Z

We have a current issue with our primary load balancer, and are in the process of bringing our secondary service on line. This will be effecting both mail and web will be effected for some clients, others will see no impact what so ever. We will update this as we can.

[update 1255] - this has been resolved. The primary balancer has suffered a hardware failure, the secondary is now online and operating correctly.

[update 1307] Our previous month independent statistics are showing as 99.94% uptime, and so far this month 99.88% for the six days of August; we are quite pleased with these figures, and thought we would share them with you : )

AOL Customers

2007-08-03T14:40:00.000Z

Despite our requests to our AOL customers not to use the spam report features of their AOL mail accounts when dealing with mail sent through our network we have yet again had our AOL gateway server's IP address blocked. This is now the 5th block in as many months.

AOL have refused to whitelist the IP addresses of our mail servers.

AOL customers now have two options to ensure that they are able to have email services on their domain names.

1) Migrate your ADSL service away from AOL to ourselves. Our prices start at £11.75 per month so we're considerably cheaper than AOL.

2) Have your mail delivered to a POP box on our servers rather than forwarded to AOL.
Prices for mail accounts are available from our web site.

We will configure a new mail gateway today to handle mail bound for AOL customers and will contact all customers using AOL forwarders that the service will not continue to operate from the time AOL next blocks the new IP address.

If you are unhappy with the service provided by AOL please ask them to provide you with a MAC code which you can then forward to ourselves to migrate your DSL service away.

August Transit special offer

2007-08-01T11:51:00.000Z

Customers in IFL2 Manchester can now get 1Mbit of free PA address space with 8 IP's for 12 months. Simply pay the setup fee of £150 and arrange for IFL to install cable from your rack to our Meet-me-room switches to take advantage of this very special offer.

This special offer runs for August only and is limited to 10 new customers.

Additional transit is available at £30 per mbit. The contract can be upgraded to higher a CDR at any time which can reduce the rate to as low as £15 per Mbit.

Full details of the Manchester transit offer are available from our web site.

Protx Upgrade

2007-08-01T11:48:00.000Z

We have been advised by Protx that our payment gateway is currently down due to essential database maintenance and ugprades taking longer than expected.

While the service is unavailable we will be upgrading our own payment pages to support the new 3D-Secure service from Mastercard. We anticipate that the service will be unavailable for most of today.

UPDATE 02/08/07 : 15.00

Card processing services have now been restored. 3D Secure service will be available on Maestro cards within the hour.

July Transit

2007-07-27T10:58:00.000Z

Welcome on board to seven new transit clients this month, along with new branches to our internal infrastructure.

Supplying redundant and robust connectivity for transit in Reynolds House, Manchester Technopark, IFL2, for co-location, primary and secondary routes.

See Netserve website for more on our transit offerings.

Peering expansion

2007-07-27T10:52:00.000Z

In July we have added to our peering routes with NTL / Telewest / BlueYonder / Virgin Media group (approximately 250 routes) at gigabit speeds.

In simple terms this means shorter / faster paths to and from their networks - less congestion - more robustness.

London circuit advisory.

2007-07-27T10:37:00.000Z

In the spirit of reporting not just bad things - an advisory for you.
We are currently on an advisory that one of our peering providers is diagnosing a circuit in London. For this period there will be a loss of protection. We do not envisage this impacting delivery of transit or services beyond possible path should this fail - which is nice to be able to report.

Eu-id Registry / Notice

2007-07-10T17:14:00.000Z

We have noticed a number of emails arriving today from euidregistration.eu claiming they have a "obligation to reduce domain ownership disputes and brand name loss" and offering to register your .eu domain name.

Can customers please note this is simply spam mail and the company concerned has no obligation or position of authority to be contacting you. The email is best put in the spam folder!

If you're interested in seeing what this spammer has to offer, I'm sorry but this is all you'll see....

"Warning: include(index__en.html) [function.include]: failed to open stream: No such file or directory in /var/www/localhost/htdocs/mall/index.php on line 25"

If you wish to secure your .eu domain with ourselves, they're available as always at £8.50 + vat via our support site.

Manchester to London transit failure (avoided)

2007-06-27T12:58:00.000Z

We have lost transit between Manchester and London over both of our secondary providers. After a fibre break on the primary path, the secondary path has suffered a flooded repeater module in the Shefield area.

We continue to offer service without interruption or degradation through our other peers and providers. However this may not be the case for your provider, or your clients servers - please keep this in mind.

For anyone interested in purchasing transit from Manchester through us please have a look at our website.

UPDATE

The break at Shefield as been repaired so we appear at this moment in time to have restored London to Manchester transit. We will be monitoring the link over the next 24 hours and will revert to our two backup providers if the fibre does not remain stable.

During the course of the day we have been able to deliver additional transit at very short notice to another Manchester based service provider who was left with no connectivity from their Manchester racks.

Our offer of 1Mbit free backup transit to other service providers in Manchester IFL2 with their own AS number is still in effect. Simply pay for the installation of the service at £150 for free transit!

Manchester London Transit - Loss of Protection

2007-06-26T11:20:00.000Z

Due to last nights flooding in Sheffield, we have been informed that power at a network repeater point will need to be shutdown today during the clean up operation. No loss of service is expected this is a loss of protection only.

Temperature

2007-06-12T18:02:00.000Z

We have been suffering with elevated temperatures within our hosting environments. This is apparently a product of higher external temperatures and air conditioning filters becoming clogged with pollen / Dandelion seeds.

We monitor internal case and hard drive temperatures along with services around the clock, and are able to see trends when we are experiencing issues with our remote servers.

We are working to ensure that server losses are worked around, and that any down times are reduced to an absolute minimum. We are also awaiting some assurances from our provider that this is an end to the ongoing temperature rise.

Problem sending email

2007-06-11T11:43:00.000Z

We are currently dealing with an issue with the primary mail relay server - reporting as relay.nsnoc.com .

Currently the relay service has been disabled while we apply the means to counter the abuse.

We will continue the service as soon as we are able to secure the current situation. We will update this entry to reflect this.

This will in NO WAY effect the receipt of email from outside sources, and queued mails will be sent once the service is returned.

[update 1310]
This issue has been remedied, and tested as working - normal service has been resumed.

Cache corruption fix

2007-06-01T10:20:00.000Z

We are aware of a cache corruption on one of our cluster of mail servers from which you collect your email. We are currently taking steps to replace the damaged cache. Rather than letting it repopulate itself we are intending to seed the cache to reduce the impact to you.

This will currently be showing to an unfortunate few as a replicated mail, or mail you cannot delete or read. Deleting the cache will cause larger loads on the backend and therefore slower mail for all - we are hoping to avoid this by seeding the cache with valid but stale data.

We envisage this change taking place between 1130 and midday today Friday 1st June.

210 update and upgrade

2007-05-17T15:42:00.000Z

If you have a site that was listed on 210 (as stated in the list domains table) - you will find that you have a new home. These sites have been upgraded and migrated to the Netserve primary cluster. Users would do well to check their site functionality although we expect no issues, please do ensure you note that your FTP details will have changed, please check these using the control panel.

Mail relay

2007-05-16T09:25:00.001Z

We have temporarily taken the relay server off line while we deal with a spam issue.

[Update Wed 16 May 10:44]

An attack was dealt with, and for the period of dealing with the huge influx service was degraded to the point of denial. Faced with this we took the service offline to deal with the issue.

The service is back and fully operational as of 1040 and the source has been dealt with.

Secure gateway

2007-05-15T13:54:00.000Z

After a series of updates, our ProSSL server is back up and running. This includes the the payment gateway.

Users who had previously encountered issues placing payments should find that the payment gateway now fully functional.

DNS issues - Saturday evening.

2007-05-12T18:21:00.000Z

We are currently experiencing multiple repeated failures of our DNS front end system. While we are trying to keep the service up as much as possible, we are aware there are going to be perceived outages.

Every effort is being made to maintain service, however we are increasingly aware that this maybe due to malicious intent, and are looking into this currently. If you have a support query regarding DNS or lack of resolution, hold fire, and we will update this as often as we are able.

In overview a lack of name resolution for a number of the 64,000 domains we host is currently occurring.

[Update 14th May 12:30]

The issue has been dealt with before the close of Saturday. The secondary name server has been shut down, and a new one built configured and deployed.

This event bought to our attention that an older DNS server that many users have used on their DNS records was no longer valid. This has also been dealt with.

If you are still experiencing issues now, drop us a support mail to support, mails regarding issues over this period will not be prioritised so we are better able to address current and outstanding issues.

Online payments

2007-05-09T15:52:00.000Z

We are currently experiencing some issues with out payment gateway people. We have contacted them, and are awaiting some feedback. If in doubt, these do appear to be outright failures, and are reporting to us as such, so no funds transfers are going on, please do try again later.

The enlightening failures go something along the lines of :-

[snip]
The system could not authenticate your payment due to the Payment Gateway being erroneous
The system has informed us of this problem which will be fixed shortly. Please check back and try again later. If If this problem remains for more than 24 hours, please contact customer services for manual payment. NOTE VISA DELTA CARDS - If you are using a Visa Delta Card, please select Visa from the card type menu
[snip]

ADSL authentication

2007-04-26T11:26:00.000Z

We are currently experiencing some issues with authenticating our ADSL users. The mechanism that proxies the RADIUS authentication off of the BT network is not functioning. While we are seeing the requests from authentication and granting them where correct, this data is not getting back to you.

This issue is currently seeing as being within the Thus/Demon/YourComms network and a solution from them is being persued. The issue was first discovered at 10am today. This should in theory only effect users that are either connecting afresh, or sessions are dropping and need refreshing - since routing and transit mechanisms are fine, it is initial authentication that is failing.

Greylisting - A new antispam feature

2007-04-19T16:10:00.000Z

GREYLISTING
Our most recent addition to our spam scanning servers is to add greylisting to our spam scanning servers.

Greylisting allows us to delay the acceptance of email by 5 minutes for hosts and domains which are not known to us. Once the 5 minute delay has expired email from the host/domain is accepted immediately and will continue through the usual scanning processes.

A normal mail server sending mail will have no problems in retrying the mail delivery after 5 minutes has expired. Where the inbound mail is being sent from a compromised server or PC, the spam software does not normally run a queue for delayed mail and so the mail is never resent. The statistics of the last 48 hours suggest as many as 50% of the servers attempting to deliver mail which pass the RBL tests are not genuine mail servers as they never attempt to deliver the mail a second time.

Customers should be aware that mail will only be delayed by 5 minutes where that sender has never sent mail to our network previously. Our servers have already whitelisted many popular servers and mail addresses including hotmail, msn, yahoo and ebay.

For customers still seeing occasional spam mails it is worth ensuring that you do not have a catchall on your mail address. Where a catchall is unavoidable, identifying specific aliases that are targeted by spammers is a useful option. These aliases can then be set via our support site with a target of :fail: (include the colons) which will cause all future mail to those aliases to be rejected to the sending server.

Finally it is worth noting that the vast majority of the spam that we see can be traced back to mailto: links on websites that have been harvested by spammers. Remember, if you need to include an email address on a web page, avoid using a mailto: hyperlink