Bell Online Ltd Status Site » USA1-VN

[update] USA1-VN is back up

Auto — Wed, 21 Jul 2010 08:59:08 +0000

Thank you for your patience, USA1-VN is now back up and running.

USA1-VN.MIXSTREAM.NET (66.90.103.76)
UP at Jul 21 2010 09:56 AM (EU/Western)
Was down for: 05min

[outage] USA1-VN has just gone down

Auto — Wed, 21 Jul 2010 08:56:07 +0000

This is an automatic notification.

Our monitoring system has detected that USA1-VN is currently down.

Staff have already been notified of the outage and will look in to the issue shortly.

USA1-VN.MIXSTREAM.NET (66.90.103.76)
DOWN at Jul 21 2010 09:51 AM (EU/Western)
Err: ping: request timeout

[packet loss]

Gavin — Mon, 12 Jul 2010 15:24:53 +0000

There is currently an issue affecting most USA value network servers. This is being worked on and will be back to normal soon.

[packet loss] USA Value network

Gavin — Wed, 10 Mar 2010 14:50:25 +0000

There seems to be packet loss to some locations on some of the USA Value network servers. This is being worked on.

network maintenance – Chicago – 13/02/2010 – 6AM-8AM GMT

Gavin — Fri, 12 Feb 2010 16:49:02 +0000

network maintenance – Chicago – 13/02/2010 – 6AM-8AM GMT

We will be upgrading some of our equipment to increase our network`s capacity. We expect no network downtime. Some of our upstream providers may go offline during the maintenance but the affected traffic will be automatically re-routed through our other uplinks

USA Value network packet loss

Gavin — Tue, 26 Jan 2010 17:32:13 +0000

There currently seems to be some packet loss on one if the USA value networks. This will affect some customers on usa1-5 vn.

It may be another DDOS attack. Updates to come.

Value network update

Gavin — Tue, 01 Dec 2009 17:07:13 +0000

I will be performing an update on the systems used for stopping/starting streams today. All shoutcast streams will need restarting, but this will only cause a few seconds of downtime and then the servers will come back up automatically.

This is intended to resolve any issues with servers randomly stopping as reported by a small number of customers.

The restarts will be performed over the next few hours.

UPDATE: This is now complete on all servers apart from USA3-VN which needs some more work before the change can be made. Updates will be posted here on any maintenance. This will be within the next week.

USA Value network issue

Gavin — Mon, 12 Oct 2009 12:15:17 +0000

It looks like usa(1-5)-vn has just gone down, this is being worked on.

UPDATE 13:26: Servers are back online after a few minutes of downtime, I can see a small amount of packet loss but this should go away once the network has caught up with the demand.

Usa1-VN down

Gavin — Mon, 14 Sep 2009 14:58:32 +0000

USA1-VN just went down, an engineer is on their way to take a look at the server. If it’s something simple then we expect the server to be back on in 10 mintues.

Most of USA value network issues

Gavin — Thu, 23 Jul 2009 22:24:48 +0000

We are unable to access most servers on the value network from the office. Customers may be experiencing issues streaming on the usa value network (usa1-usa5).

Due to the network being down, we are currently unable to open a ticket with the datacentre, however I can assure you that they are very aware of the problem and will be working hard to fix it.

Update:

All back up now, this lasted about 2 hours.

Message from datacentre:

Our Chicago facility has experienced network issues this evening due to core network hardware failure.

The issue has been resolved and the network should be back to normal now.

The root cause of the problem was a failed 4x10GE blade on one of our core routers. When the traffic failed over to the other core router it has crashed the router thus causing complete network outage.

We have attempted to recover the core setup under existing configuration but once it became apparent that the blade was defective we had to move some uplinks and take the blade offline.