Wednesday, October 18, 2006

 

Service restarts

A primary MX server stopped responding at 0403 - the service was
restored for 0419.

While the resulting mail delivery backlog is currently catching up we do
not envisage any issues for start of business. Other primaries and
backups are functioning without interruption.

The issue was caused indirectly by a log rotate, and HUP/reload that
followed.


Monday, October 16, 2006

 

PHP Fopen - Remote URLs


One of our cluster web servers has been hacked this evening. While no damage
has been caused to customer web sites and our monitoring system prevented
significant damage, the server in question has suffered some damage to the
point where a full reinstallation is advisable.

We have disabled the web sites of customers using products from
www.comdevweb.com as despite warnings on 27th September they have failed to
adequately secure their software resulting in two hacks within 4 days even
after security patches were applied.

As a temporary measure we have disabled the remote fopen function within php
in order to protect our servers from any further risk over the next 24 to 48
hours. While we appreciate that this is likely to cause problems for some
web sites we feel that this is an essential step given that in the last 4
weeks our servers have suffered from over 10 semi successful php hacks as a
result of sites using insecure code and running fopen functions.

We would ask customers using software which includes fopen calls to remote
sites to ensure that the sites are not vulnerable to remote code injection
techniques. We would also remind customers that they should subscribe to the
update mailing lists of any commercial or open source software which they
load onto our servers.


Wednesday, October 11, 2006

 

Service Restarts

The LVS director failed at about 8.10pm last night and was restarted at 8.20pm.

Pop server 21 stopped responding at 9.20am this morning and was restarted at 9.40am.

The large mail queue issues of the last few weeks appear to have been resolved now. We will continue to monitor the queue size during the working day to ensure all mail is delivered as quickly as possible.

We would ask customers wanting to silently delete or reject email to use :blackhole: in their forwarders or catchalls as setting fake forwarding addresses only results in our servers having to handle large queues of frozen messages.

 

Email Service Issues.

Recent issues with our mail queue have caused some mails to arrive several hours later than normal.  We have been actively monitoring the queue processes and are now in a position to report that there have been no mail backlogs for 48 hours.
 
We will continue to monitor the situation to ensure that mail delivery is completed as quickly as possible.
 
We are in the process of reviewing our email services as traditionally they have been provided free of charge or as a bolt on service to hosting or reseller accounts.  We are now seeing business users becoming more dependant on their email service than their web site and feel that we must adapt our service levels to reflect the needs of business users. This will almost certainly require changes to our pricing structure but it will allow us to invest further in our current 9 email servers.

 

New Service Status Pages

In order to provide better communications with our customers we are now using blogger to provide service status information as well as deliver information about new services and changes to existing services.

As always we welcome customer feedback and suggestions for improvements to our services.

This page is powered by Blogger. Isn't yours?