At 8:43 PM EST on 22 April 2021, Floofy Hosting was notified of a hardware failure on the node FLOOFY-CANADA-01. This node hosts VPS. The server previously crashed that morning, as well as the morning prior due to unknown reasons, however, self-tests came back fine.
The SSDs, which were in RAID-1, seem to be failing a SMART test (self drive test to make sure everything is working ok). On top of that, the NIC seems to have failed too. Everything seems to be pointing to a power surge by our data centre which was unsuccessfully stopped.
As of 9:49 AM on 23 April 2021, it is looking like we will be able to get the data back, we were successfully able to access the data, however, the drives could stop while we download the data to our backup server in Germany (for internal use only). We have to backup the data carefully as to not stress the SSDs to much to the point that they fail.
As of 2:06 PM on 23 April 2021, we are downloading the data to our backup servers. No data loss is expected, however, this statement may change.
We were successfully able to download all 700GB of data. After downloading the data, we had to process all the images to convert it to the new format for the new panel (previously were using raw _img, our servers converted all the VPS to raw .qcow2, then qcow2 .qcow2, which gives a performance boost, and is required on the new panel). The conversion took around 12-13 hours. We then added a new SSD into the server and transferred all the data back. We are now in the process of transferring all the IPs back, and the general installation of the new panel. We aim to make sure that everyone keeps their old IP, and that there is no noticeable difference in the VPS since before the issue.
ISSUE IS IN PROGRESS, ARTICLE WILL BE UPDATING...
