Netserve Service Status

Our offices will be closed from 5pm on Monday 23rd December and will reopen on Monday 6th January.

During the holiday period technical support will be provided via the support site ticketing system as normal with reduced hours on Christmas Day, Boxing Day and New Years Day.

We would like to thank our customers for their business over the course of 2013 and wish everyone a very Merry Christmas and prosperous 2014.

During the next 4 weeks we will be carrying out some housekeeping on the hosting and email servers.

Tasks we'll be undertaking include :-

New hosting servers will be added to the platform with PHP5.5 installed as standard.

Migration of hosting packages from pre 2013 Servers to the PHP5.5 platform.

Upgrade of Email forwarding, POP and IMAP servers.

New Webmail platform to be launched.  The old webmail system will be maintained for 3 months, then all accounts will be forced to use the new software.

Purging of old email accounts. 

Purging of closed and unpaid hosting accounts. All hosting accounts which have not been renewed or paid for will be terminated.  A list will be sent to each customer of hosting accounts being closed.

Purging of backup data.  We'll be keeping backups of the purged email and hosting data for 12 weeks after which time the backups will be deleted.

Review of all customer data storage and transfer limits.  Until now we've been fairly flexible with the number of email accounts and volume of data storage customers have used however from 1st January the limits for the package you select will be strictly enforced with respect to the number of POP email accounts you have and the storage used on both POP and FTP accounts.

The should not be any downtime associated with any of the changes being implemented.

We've been seeing some packet loss beyond our edge routers since about 11am this morning. Currently it appears to be only one upstream transit route that's suffering. Primarily UK/Europe routes which equate to about 40% of our connectivity.

We believe it may be related to a router upgrade the affected network performed at 2am last night.

Currently monitoring the situation, updates will follow.

UPDATE  11.48 am

Not seen any packet loss on the upstream route for the last 5 minutes although we're still waiting on confirmation from the provider concerned that they've seen the issue and responded to it.

We'll continue to monitor for the next few hours.

Should the issue happen again, we'll drop the peering session with the provider until their network regains stability.

Core Mail file server appeared to develop a go slow issue later this afternoon. Restart cleared the issue, however server 66 took a nose dive after the file system went awol so that's been restarted this evening.

Great to see so many people enjoying the sunshine over the weekend at the Kaya Festival in Bangor.

Our public Wifi appeared to be a great success, with coverage from the events field through to the camping fields.

Looking forwards to more big public events over the next few months. Stay tuned to find out where we'll be next!

St Asaph POP UPS to be reset within the next hour. May cause a short 20 second outage if the core switch resets during the work.

UPDATE
UPS reset completed without issue however mains disconnection test caused switch reboot. Further investigation will be required during the next week to confirm correct functioning of UPS during mains transfer to generator.

We've been investigating two reports of end users not able to reach our network. In both cases the points of failure appear to be well outside of our network. As yet we have been unable to identify the network at the heart of the issue. If anything develops we'll advise here, but at this point in time we're not seeing any changes in network usage or border routes.

The power supply to rack A5 has tripped due to the power supply on server 64 failing.

Power has been restored, servers in A5 are all working as normal.

Services (POP/IMAP mail) from Server 64 have transferred to servers 65 and 66 via the cluster management system.

The file server responsible for storing mail data has crashed three times in the last 3 days.

Investigations have shown one of the drives to be generating IO errors.

At midnight the failed drive will be replaced and re silvered

UPDATE 9am Monday

The drive replacement / resilvering is 50% completed. The file server is operating in a degraded state so is running with less performance than normal.

We expect the process to be completed by 6pm.

Until that time the mail servers will run with a higher than normal load and may be less responsive that usual.

UPDATE 1pm Monday

Work to bring the file server up to speed with a full disk set should now be completed by 4pm.

We have identified during the work that one of the POP cluster servers is suffering from a faulty network connection. Further investigation will continue this afternoon.

UPDATE 3 pm Monday

The process to re silver the missing drive is due to complete at 4.30pm.

During the monitoring of the server loads it has become increasingly apparent that there are a number of customers with POP accounts where mail is being stored for considerable periods of time. POP is not best suited to storing mail for longer periods and it is apparent that the loads on the POP servers are being caused by inappropriate use of the POP accounts.

We will be performing a review of POP usage over the next week and advising customers where their usage needs to be modified.

UPDATE 8pm Monday

File system completed the rebuilt at 4.30pm. Server loads all returned to normal within a few minutes of the work being completed.

We will monitor the situation over the next 24 hours to ensure there are no further issues.

Server 67 our core file server has suffered a crash. It's not viable today to attempt a reboot so services are currently running from our backup file server.

We'll reboot 67 in the morning and run a file system check.

At the moment we've disabled the FTP upload service to avoid any possible conflicts between the live server and backup.

Bad weather in North Wales has caused some power outages on the St Asaph Business Park overnight.

Capital Court hasn't suffered any power issues but we have been indirectly affected by switches on our transit routes being power cycled between mains and generator backup. We installed a UPS at 7,30pm last night on a switch that had been suffering from power cycling issues.

This morning the transfer from generator to mains at a local POP appears to have caused one of our Capital Court switches to lock it's network facing uplink port. We've made changes to the switch so service to Row A racks 3,4 and 5 has been restored.

An incident in the main energy centre at IFL has resulted in services for area C1 being run on generator. Suite C1 houses our core switches and routers so our core network is currently at risk.

Further information will be provided as soon as we are made aware of the situation.

A network upgrade yesterday has caused a number of our cisco switches to lock up.
We've replaced 4 switches, but now need one more to complete the process and restore the ADSL services. The last switch should be with us this morning to install and configure.

The upgrade to the North Wales circuit appeared to have completed correctly however further investigation revealed that the two core Cisco switches in Manchester had blocked the connections for North Wales post upgrade.

Switch reboots failed to resolve the issue resulting in us having to transport two new switches to Manchester.

As yet we've not had time to pick over the bones of the issue. The Ethernet circuit upgrade should not have caused any issues but the timing of the two events would suggest that they are related.

We'll be meeting with the Fibre providers in a few hours to see if they can shed any light on the event from their side of things.

Friday 17th September
Switch upgrade OpticGlyndwr St Asaph.
Customers at the site may see a short (3 seconds) loss of Ethernet circuit while the cables are patched between switches.

Sunday 19th September - IFL2
The Cisco switch located at C1-S5/A4 in IFL2 which supports colo equipment in Suite C4 at IFL2 will be replaced.
Patching of cables should take no more than 3 seconds for the Ethernet circuits to establish themselves on the new switch.

Monday 4th October 8pm
St Asaph, Bangor, Deeside, St Davids Park, Abergele
Our backhaul circuit over the Fibrespeed network will be upgraded between 8pm and 9pm. A brief outage is possible while the work is performed.

IFL are testing one of our core network cables in IFL2 within the next few minutes to identify the cause of some packet loss hitting our St Asaph network.

It's likely that there will be a few seconds of instability on the network during the test.

UPDATE

IFL cable proved to be working fine.

UPDATE

Fault traced to local circuit in St Asaph. Media converters replaced at both the Fibrespeed pop and OpticTechnium appears to have rectified the issue.

Please be aware that we have been moving to new offices over the weekend.

In order to finalise the move we will be splicing new fibre cable on Monday morning. We expect to have the work between 10am and 11am.

Until that time we'll have no telephone support available and email support will be restricted to high priority tickets only.

Our new office details are...

Netserve Consultants Ltd
Unit 44B
Capital Court
St Asaph Business Park
Denbighshire
LL17 0JG

The storage system that supports POP/IMAP and webmail has been rebooted after locking up at approx 10:15.

All email appears to be working without issue now.

We have identified that a number of files on Host server 8 have been modified by a remote third party via a vulnerability within a client web site.

As a precaution we have disabled a number of additional PHP functions on the server and increased the settings on mod_security.

The modified files have been replaced with valid files from our backup.

Thursday 7th Jan 4am.

The sql server mysql1.nsnoc.com has crashed and requires a restart. It has failed to respond to two automated reboot attempts and will require manual intervention to restart it.

An engineer will be on site shortly to attempt to resolve the issue.

Next update due 7am.

UPDATE
The server has responded to an engineer being present at the reboot. All services are working normally.

It is not clear at this time why the server is not rebooting on the remote switch when required so we will take steps today to migrate the database to new hardware.

UPDATE 2PM
The cause of last nights issue has been traced to a faulty network card. We are in the process of replacing the card and will provide an update shortly.

UPDATE 2.45PM
The network card has been replaced and hte service appears to be working now without issue.

We will monitor the system over the next 24 hours to ensure stability.

The power supply on server 32 the logs.nsnoc.com server failed at 2pm this afternoon.

The failed supply caused the trip switch to the rack to trip to protect the equipment. The UPS maintained service to the remaining servers for 15 minutes but it took longer than anticipated to identify the issue and gain access to the building.

An engineer was on site at 3pm to identify the issue and start the process of finding the server that caused the initial trip.

Power was restored to most servers by 3.30pm. We worked through all of the services to ensure that everything was working and by 5pm concluded that all remaining bugs had been resolved.

During the process we identified that one of the core switches had been incorrectly powered from the rack B UPS so the failure impacted on some services from other racks. This issue will be resolved on 4th January when the office reopens.

At this moment in time, the logs.nsnoc.com server is offline until a new power supply can be installed on 4th January. We will process the logs from today onwards once the server is working again.

All other services have been tested and confirmed as working. Please notify support @ nsnoc.com if you have any other issues.

The Mysql1 server has crashed and failed to reboot this morning.

An engineer will be on site for 8.30am to investigate the issue further.

UPDATE

A second reboot of the server brought services up without errors. We will have standby hardware available for the server as a precaution over the Christmas holiday.

If customers are aware of any issues following the reboot please raise a support ticket in the usual way.

We have a number of network cables which need to be re-patched over the course of Friday.

Network customers may notice a few seconds instability while the patch leads are moved.

Host server 8 has now been upgraded to PHP5 and Mysql 5.

We have migrated all databases and sites to the server and the service appears stable.

There are however some points :-

- If your hosting service has expired prior to June 2009 then your site has not been upgraded and is not live. Please contact us to renew the service and re enable your site.

- A small number of sites have databases with some locked/crashed database tables. There is nothing to suggest these are active tables or that the issue has only recently occurred but if you do have a table that requires a repair please let us know.

- The ftp server on the new host is being very 'Monday' and not authenticating requests at the moment. Assuming nothing major crops up after the migration we'll work on the fault over the remainder of the afternoon.

UPDATE 15:30

FTP accounts on the server are working without issue.

Support will run to 11pm via email this evening to resolve any remaining issues with users.

We will be performing mantenance on the web cluster from 5.30am tomorrow morning.

The following services will be taken offline, Cluster 150, 156 and 212.

We are anticipating that work will be completed and full services restored by 8am with a possible overrun to 8.30am at the latest.

While the work is being completed sites will show an under maintenance page.

The work is necessary to allow us to migrate to a new storage platform and add support for php5 to the web clusters

UPDATE 7AM
IFL have failed to renew the access pass for the engineer carrying out this mornings scheduled work. The pass expired 2 days ago thereby preventing us from completing the migration according to our original timetable.

We have no alternative dates to complete the work so it will commence at 7.30 am this morning with a target completion of 9.30am.

UPDATE 11AM
We're seeing high traffic levels at the file server caused by a single site. If services do not settle to an acceptable level within the next 5 to 10 minutes we will disable the site and look for stability to return to the servers.

UPDATE 1.30PM
We have been able to confirm the issues seen after the file server was returned to service this morning we caused by a single site. The site in question has been disabled until we can migrate it to alternative hardware.

We will be monitoring the servers for any further issues for the rest of today.

We've seen a batch of fishing emails arrive with customers today asking them to update contact details as part of a security update. We remind customers to be wary about clicking links in emails even when they appear to come from known sources.

We've been experiencing some instability with the web cluster 150 servers over the last hour. The cause has now been traced to a single site causing a the server to run out of memory.

The site in question has been disabled pending further investigations.

INITIAL 9:45PM

We're currently repairing some file corruption on the main mail server file store.

The process should not take more than a few minutes. Until the repair is completed there is no access to POP, IMAP, or Webmail services.

UPDATE 10:15PM
The file system check and repair has completed however a server reboot is required to clear a number of Dead/Zombie processes on the server.

We're going to assess the issues and decide on a course of action shortly.


UPDATE 10:50PM
An engineer will be dispatched to IFL2 at 3am to recover the file server to our new racks. We hope to have the service restored shortly after 8am.


UPDATE 8.15AM
The overnight work to recover the file server to our new racks was completed on time and without incident.


UPDATE 11:45AM
Heavy storage loads have resulted in the need to pull the file system and force a tree rebuild - this is going to be a lengthy process, so we are working to restore a backup in the interim with an ETA of a fifth of the time before inboxes are present. AGAIN I would like to reiterate this is not preventing inbound mail from arriving, merely the ability to collect it, an important difference.


UPDATE 13:38
Working on a solution to deliver / present fresh email to the end users. While historical data will be missing, all new emails will be arriving and visible - allowing us to migrate these to the existing file system once rebuilt. Light at end of collective tunnels.


UPDATE 13:45
Temporary solution in place. POP users will now be seeing all their new email arriving. Those who are paying for the IMAP solution will notice that all their mail has gone. DO NOT adjust your set. Simply work with what you have for now, and your inbox will be reunited with old mail on completion of the rebuild over the next 12 to 24 hours.

This completes updates on this entry, work from this point on will continue in a new entry. Over the last 10 minutes, over 467Mbytes of inbound mail have been stored, and we can see you retrieving email.

NB, if you use webmail or Imap to collect mail, please avoid creating new directory structures until your old mail account has been merged with the new server.

UPDATE Wednesday 11.30am
Work is continuing on the old file server. We are currently running a snapshot of the data to backup before starting the merge with the new file server. Further updates will follow at 2.30pm today.

UPDATE Wednesday 2.30am
The snapshot of the data is now 1/3rd complete. It's likely to be early evening before we can start the process of merging the new mails with the old. Next update 5pm

UPDATE Wednesday 5pm
The snapshot of the data is now 3/4 complete. We're expecting to be able to start the data merge at 9pm which should take less than 1 hour to complete. Next update once data merge has been completed and all webmail/IMAP data has been restored.

UPDATE Thursday 1am
Our new file server is now live with a full copy of all mails, old and new. At no point has any mail been lost or rejected.

There is a small risk that the mail indexes will have changed for IMAP users resulting in a mis match between the From/Subject line presented by your mail client and the mail downloaded. The fix is to right click the folder concerned and select 'rebuild index'

Our key staff will be attending a conference from 7am to 1pm today (Thursday)

For the time we are away, we will be operating a reduced support service and no telephone support.

Emergencies will be handled by on call staff.
General support will resume early afternoon.

A transit customers server has been subject to a DDOS causing in excess of 100Mbit of small packets to traverse our network. As a result there has been a disruption to connectivity and some packet loss.

Steps to mitigate the problem have been taken as as of 12.44pm stability has returned to the network with no packet loss apparent.

It would appear that a customer wordpress site has caused the web cluster to overload. We're currently waiting on the load to fall on the web servers to confirm the cause and to restore service.

We expect the servers in cluster 150 to have recovered within the next 5 to 10 minutes.

An update will follow shortly.

UPDATE - 5.25pm

Services have been restored and will be monitored over the course of the evening. The service restart prevented us confirming the exact process details that had caused the server load to climb.

Friday saw a scheduled file system check of a mail cluster member. It would appear that a complication with files not being deleted properly after being cleared from the spool queue had arisen, and as a result on recovering these files to a state in which they could be deleted - the mail server diligently sent them on their way. The resulting phanton resending of emails - in some cases up to a month old.

As these mails had been marked as deleted, these mails are DUPLICATES, and these and can safely be disregarded.

We apologise for the confusion this may have caused.