Incident on Fastnet's VMware platform paralyzes email and DNS • The Register

Brighton-based ISP and hosting provider Fastnet has had a busy week battling with technical issues at VMware/Broadcom that have resulted in many of its customers' websites going down.

The outage began on August 13, at which point Fastnet's technical team said they were “in contact with the provider at the highest level” and working as quickly as possible to restore services.

Fastnet has turned its homepage into a running live blog with progress updates, posting a new one about every two hours on weekdays. However, the company did not provide many technical details about the outage, attributing the problems only to its VMware hosting platform.

“Fastnet is currently experiencing a large, widespread service outage on our VMware hosting platform, affecting users with services that include these products,” the company's website said last week. “This includes web/email/DNS hosting including cPanel services, virtual data center and some other services that rely on the infrastructure. Connectivity, DSL and Ethernet services are not affected.”

“Fastnet Broadband customers who currently have a working internet connection are advised not to reboot their routers to resolve issues with our other services as this may cause subsequent reconnections to fail and your connection to go offline.”

A source at a customer who wished to remain anonymous said El Reg They expressed disappointment with the company’s communication regarding the incident:

“They offered to keep us updated via email, but we probably wouldn't have received it. [given] the problems, so the only updates we have had are those posted on the Fastnet website.” The level of technical detail was described as “minimal.”

The client told us that Fastnet is the registrar and DNS provider for their domain and that the MX records were unavailable and the SPF records could not be verified, so outgoing email was “unreliable”. Employees could access their email as it was provided through Office 365, but most organizations were not accepting messages sent from their domain.

Fortunately, the customer's DNS zone was back online by 11:00 a.m. on Thursday and digital services were restored, but the company still experienced an unwelcome 36-hour outage.

“I think this shows how important DNS is to the foundation of the Internet, yet it is overlooked,” the source told us.

The outage affected many Fastnet customers, as well as professional bodies such as the FICM and the Faculty of Pain Medicine (part of the Royal College of Anaesthetists, which also experienced an outage).

Small businesses, from wine merchants to fish restaurants, were also affected, and even the Caravan and Motorhome Club could only rely on telephone communication, although many were operational again on Friday.

Fastnet announced on the evening of August 15 (UTC time zone) that the majority of its shared hosting environment was restored and fully operational, but was still working to restore other customer-facing services.

“We continue to work to bring additional customer-facing services back online,” it said on the morning of August 16. “We understand the importance of restoring services and are working with our providers to do so as quickly as possible. We will provide further updates as they become available.”

El Reg We contacted Fastnet and Broadcom to ask for more details about the incident and whether there was any suspicion of a crime, as our source had feared, but neither responded.

From the outset, Fastnet promised to publish a comprehensive explanation of the disruption once the incident had been resolved and repeatedly apologized to the affected customers.

The last update came on Saturday afternoon, but showed no further progress. Some shared hosting platforms still needed to be brought back online, but DNS, hosting and email services were all fully restored at this time. ®