Web host 123-reg accidentally slaughters customers’ sites

Customers of the web host 123-reg are frantically trying to rebuild their online businesses after the host accidentally erased an unknown number of their virtual private servers (VPS) during a clean-up.

“This will wreck my business & plenty of others I am sure,” said one of the customers who’ve been flooding the company on social media with complaints about its failure to give much detail.

123-reg admitted in an email that the data was removed by a clean-up script.

The company told the BBC that it doesn’t have a backup copy of all customers’ data – unless they’d previously paid for it, that is.

A VPS is a virtual machine sold by hosting providers such as 123-reg. It’s a far more affordable alternative to a private physical server, and it allows companies to install any software that runs on a given operating system, while retaining superuser-level access and control.

In a statement sent to Ars Technica, 123-reg said that it always tells customers to back up their VPS to protect themselves from these kind of situations:

Our VPS product is an unmanaged service and we always recommend that customers implement backups to safeguard against unexpected issues. Customers who had purchased 123 Reg backups can be online now as can those who are using another solution for website backup.

If customers restore from their own backups, this will not overwrite our efforts. Additionally, customers who have restored from their own backups are now hosted on new servers. By using new servers, we will ensure that we do not overwrite the previous servers and impact the data recovery process.

According to its status page, 123-reg is using data recovery tools to recover customers’ lost packages “bit by bit” while simultaneously checking data integrity: a process that’s “quite time-consuming,” spanning multiple servers in several locations, it said.

It’s also working with Kroll, a data recovery company, 123-reg told Ars.

On Monday, the company told customers with local backups to rebuild their own servers using these instructions.

The web host has been trying to restore data since Saturday morning, but it’s already told some customers that its team has “exhausted all options” and that their data’s gone for good.

One business said that it had talked to 123-reg on Sunday. It was told that it would take a week, minimum, for email update and “more than likely all data lost.”

123-reg hosts 1.7 million sites for 800,000 customers in the UK. It hasn’t specified how many sites were deleted, but it did tell the BBC that it was a “small proportion.”

It’s said that the fault was limited to 67 servers out of 115,000 across Europe, with only some of the customers hosted on those servers being affected.

Unlike our recent story of the sysadmin who deleted all his customers’ websites, which turned out to be a cautionary but apocryphal tale, this one looks like it really did happen.

As we said then, as ransomware has brought into sharp relief, and as 123-reg says it tried to tell customers before this boondoggle, backup is the way to go to protect yourself.

Make sure to encrypt your backups, too, to stop a data breach if somebody adds insult (and more injury!) by stealing your offsite disks.

Image of backup courtesy of Shutterstock.com