Pretty much everything apart from one of our DNS servers is now back online (not a big problem as long as the other one stays up, no reason to think otherwise), however there may still be a few problems for some customers. We have had reports of the websites not showing up correctly (shows a DNS Updating page) – if this happens then try rebooting your computer, this should do the job.

One problem caused by the DNS still updating is that some Paypal notifications are not being posted to the billing system, so if you have made a payment in the last day then please check to see that the invoice is listed as ‘Paid’, if it shows ‘Unpaid’ then please open a ticket. This won’t fix itself so we need to know.

One other issue that seems to have only affected one customer so far is that the AutoDJ control panel is running slowly – like I say, only one report of this so not 100% confirmed but this should improve over the next day (probably also due to DNS updating).

Sorry, I just turned off all streams on uk3-vn by accident – this caused about 5 seconds of downtime.

The reason is that the ‘stop all’ link is right next to the ‘next page’ link in the admin panel! I’ll be more careful in future.

Apologies for the downtime that we have experienced and thank you for your patience. We host our Shoutcast panel, billing and support system on a VPS server provided by Vaserv, a large VPS hosting company. They were using HyperVM which was targetted by hackers, who went on to delete data from many customers. There is an article on a popular news site here

http://www.theregister.co.uk/2009/06/08/webhost_attack/

The reason that we use another company to host our billing system is that this way if a customer server is down, it means that the support area is hosted separately and won’t be down at the same time – this ensures that customers are able to contact us during network outages.

We were fortunate in that we didn’t lose any data, however you will be pleased to know that we run full backups every day and if data had been lost then it wouldn’t have been a problem as our latest backup was only 3 hours old at the time of the attack.

Everything should be back online and working now, so if you do have any problems then please do get in touch and I’ll look in to this.

Over the next week or so, we will be carrying out the following to ensure that any future issues don’t affect anyone in the same way:

1) A full dedicated server will be set up for the control panel and billing/support at a separate location to all customer servers (please note that we always use full dedi’s for customer stream servers and hosting, we only use a virtual server for the control panel and billing/support)

2) One of the main issues that customers had is that those using the playlist files hosted on mixstream.net will have experienced broken links (although the servers were all still online and streaming). All playlist files (that’s listen.pls, listen.m3u, listen.asx, etc) will be hosted on the same servers as the streams so that they never go down.

3) Although our backups are good, I’ll be reviewing the backup procedure to see if it can be improved further (You can’t be too careful when it comes to backups).

We are also advising customers to change their billing account passwords as there is a small chance of a security issue. We don’t keep any payment/credit card details on file or paypal account details, so there is no threat at all to paypal accounts etc.

Finally, any payments made in the last day may not have been registered by the billing system so I’ll be going through all payments and manually applying them – there is a chance that I might miss some so if you have any invoices still showing as unapaid that you have made payment for then please open a ticket.

We would like to thank all customers for their understanding and continued support during the last day.

Some customers have reported issues with Nochex when using Internet Explorer. This has now been resolved, it turns out that the mailing list subscription form was causing a problem (nested