Backup and Recovery After a Ransomware Scare

Executive summary

In 2021 ransomware hit PCs in one department. We found it in time, so it never reached the servers, but some data on a user’s PC could not be recovered. That showed me a file server problem could stop the whole operation. I set up a 3-2-1 backup routine for the ERP, Payroll and file servers and tested that the backups could be restored. In 2023 a fire at the Thanlyin factory burned the point where the network cables for every department met. I had a temporary network and the servers back in use within one full day.

Context

The factory runs on 4 servers. The ERP and Payroll databases are on SQL Server; the file server holds the departments’ shared files.

The trigger: ransomware in 2021

• Ransomware hit PCs in one department.
• It was found in time and did not reach the servers.
• Some data on a user’s PC could not be recovered.

The lesson was clear: if a server fails, for any reason, the operation stops. Backups had to be routine, not occasional.

What I put in place

• 3-2-1 rule: three copies of the data, on two kinds of storage, with one copy off site. Copies are kept on an external drive and in the cloud.
• ERP and Payroll: SQL databases (about 2 GB) exported and backed up every day.
• File server: about 2 TB, so a full backup runs every three months.
• Tool: Windows Server Backup.
• Restore tests: restored SQL databases and a whole Windows Server, to prove the backups work.

1. Back up
2. Copy to external drive and cloud
3. Test the restore

The fire in 2023

On one day early in 2023, a fire at the Thanlyin factory burned the place where the network cables for every department met.

1. 01
   Fire

   The cable meeting point for every department burned.

2. 02
   Same day: temporary network

   Rebuilt the network cabling as a temporary cover, working the whole day.

3. 03
   Servers back in use

   All servers usable again within one full day.

4. 04
   Full repair

   Permanent cabling finished in about two weeks.

Challenges & risks

• Large file server: at about 2 TB, the file server is backed up every three months. The ERP and Payroll databases are small (about 2 GB), so they are backed up every day.
• Pressure during the fire: the priority was a working temporary network first, and the permanent repair after.

Results

• ERP and Payroll backed up every day; the file server every three months.
• Copies kept on an external drive and in the cloud, following 3-2-1.
• Restores proven on SQL databases and on a whole Windows Server.
• After the fire, the servers were back in use within one full day.

Lessons learned

• A backup is only real once you have restored it. Testing restores is what makes the routine worth trusting.
• Be ready before something happens. The fire was a very worrying day. Since then I keep backups current at all times, so whatever happens, we can bring the servers back quickly.