Spaceship service outage update

On August 13, we experienced a major service outage that impacted a number of Spaceship services.

The cause was a cooling system failure at the RadiusDC: Phoenix data center, which hosts some essential Spaceship operations. This forced us to take services offline to protect customer infrastructure from overheating and possible damage.

Everything affected by the incident has now been restored, and our teams continue to monitor systems closely to ensure they remain stable.

We are deeply sorry for the impact on customers who depend on us for their websites, email, and other online services. The disruption this caused, and the circumstances behind it, are being taken extremely seriously.

What happened

The incident began when cooling systems failed at the Phoenix data center due to a major storm, causing temperatures around our infrastructure to reach critical levels.

Continuing to operate systems in such conditions risked equipment overheating and suffering longer-term and catastrophic damage. Taking services offline caused significant disruption, but it was necessary to protect our customers’ infrastructure.

Keeping equipment offline until the cooling systems were back online also gave temperatures time to return to safe levels before recovery began. This helped avoid creating further problems that could have further prolonged an already serious incident.

How customers were affected

The outage affected services including Shared, VPS, email forwarding, and Spacemail. Spaceship.com remained online throughout the incident.

Here is a breakdown of how the situation affected key products and services:

Hosting operations

Customer access to websites and hosting services was affected, with many customer websites unavailable, slow, or returning errors. Some billing operations were also unavailable after the system responsible for processing them was taken offline during the incident.

EasyWP

EasyWP’s Dashboard and customer websites were disrupted. During recovery, some customer websites remained unavailable after EasyWP.com and the Dashboard were restored, with some returning database connection errors.

Spacemail

Email services were also disrupted. Affected customers were unable to send email, and incoming email could not be delivered while the relevant servers were offline.

Importantly, this did not mean incoming messages were automatically lost. Mail providers normally make further delivery attempts when a receiving server is temporarily unavailable. So messages would arrive, but later than usual.

How we restored services

RadiusDC began restoring its cooling capacity as soon as possible, while temporary chillers were installed and directed into our area of the data center to help reduce temperatures.

Our teams worked continuously on site alongside RadiusDC to restore cooling. We only began bringing services back online once temperatures had returned to safe operating levels, and we were confident there was no risk of overheating or longer-term damage.

With safe operating conditions restored, we began a controlled recovery across our infrastructure. This needed to happen progressively, so customers saw different services return at different times.

Why services were restored in stages

The scale of the incident meant different parts of our platform were disrupted at the same time. While customers experience these as separate services, the data center incident affected the infrastructure involved in delivering a number of them.

Services like hosting and email rely on multiple layers of technology, including servers, networks, storage, and databases. Restoring them meant bringing these different elements back online in the right order, rather than simply switching everything back on at once.

What happens next

We take this disruption and its impact on our customers very seriously. 

Our teams are conducting a thorough review of the situation to strengthen our systems and to ensure that similar incidents do not happen again.

We will be completely transparent about what we learn and will share the results with you, along with the actions we are taking.

CORRECTION: Our data center operator is RadiusDC, not PhoenixNAP as referenced in several, early posts on X on 13 August. PhoenixNAP is not involved in this incident.