Thursday, November 30, 2006
Reseller Billing
It has come to our attention that some invoices for November reseller
accounts have not been generated automatically as they should have
been.
We have corrected the error and raised the invoices as needed.
Invoices for December accounts will be issued on 1st December rather
than 30th November.
Thursday, November 23, 2006
Load Balancer reboot
Our LVS load balancer has been rebooted in error causing a short 3 to
5 minute outage on most of our core services.
(Web Clusters 150, 156, 212 and Mail Clusters POP1 and Imap)
The instruction to intelligent hands of "Reboot the 2U grey box with
the label 9....." was carried out as "Reboot a black 1U box"
PHP Session Handler
PHP has a fairly well documented bug that causes the following error.
PHP Fatal error: session_start(): Failed to initialize storage
module: user (path: /tmp)
http://bugs.php.net/bug.php?id=25876
While the error does not appear on all pages, it can appear at regular
intervals.
Our tests have suggested that the following code at the start of your
session startup will reduce or eliminate the problem.
session_module_name("files");
PHP Mail Function
We have been suffering from some issues today where customers were
unable to send mail using the PHP mail() function from cluster 150.
The issues have been tracked albeit after a long day of debugging and
software reinstallation to a fault with the bash shell on the server.
The problem has taken quite some time to resolve today as at no point
in the mailing process were any errors produced to either the visitor
or to ourselves on the server logs.
The installation of 3 new web servers to support cluster 150 will
continue as planned later this week.
Monday, November 20, 2006
Server Cluster 150
We are experiencing a high load on the remaining two servers within
cluster 150 which have caused problems.
Services have been started on a new standby server 29 while the issues
with 9, 36 and 37 are resolved.
By the end of this week, server 9 will replaced with a new higher spec
server and server 29 will be brought into the cluster permanently.
Server Restart
Two servers required restarts this morning.
Server 48 was restarted after being overloaded by a denial of service
attack against ftp and ssh ports.
All vservers on that host are now running normally. We will continue
to monitor this server and have blocked the IP addresses responsible
for the attack.
Server 9 failed at approximately 8am but due to faults with the
T-mobile messaging system our engineers were not alerted until 9am.
The server has been removed from the web server cluster 150 and will
be repaired later today as it has failed to respond to a remote reboot
request. Server 9 has failed twice in the last month so new hardware
will be installed this week to replace this server.
Friday, November 17, 2006
AOL mail users will be suffering an inability to receive mail currently.
This is due to them rejecting mail from almost all of our relay servers.
Work is under way to bring up IP's specifically to solve this issue.
This is due to two main issues:
1) AOL clients with domains set to catch all or forward that are marking
junk mail they are then receiving;
2) Clients with insecure web forms that are being used to send bulk mail
are either not securing their forms, or simply are not aware.
This is a tricky area both legally and technically as the likes of
Spamhaus are more than aware. We we have already taken steps to resolve
these issues through these mediums.
In the meanwhile - AOL users, check you are not blacklisting us (thus
shooting yourselves in the foot) and those with web mail could do worse
than referring to Google and the likes of <a
href="http://www.phpbuilder.com/columns/ian_gilfillan20060412.php3?print_mode=1">this article</a>.
