10tb data recovered after Multiple Hard Drive Failure for Cayman Islands, but 4gb may be lost forever.

The Cayman Islands Government are still trying to recover critical data after a series of hard drive failures in the last 2 years. The lost data includes important police files.

The saga started in February 2012 when a controller card in the police server in George Town failed, this was followed by a server hard drive failure in August 2012, plus two more hard drive failures in April 2013.

Still reeling from these failures in October 2013, three more hard drives failed at the same time, causing significant data loss for the Royal Cayman Islands Police Service.

It is not clear if the three concurrent failures were from a RAID array, though this seems very likely. One of the primary purposes of RAID arrays is that a hard drive failure will not cause loss of data, in fact some RAID systems can have 2 simultaneous hard drive failures without file loss. So why 3 hard drive failures at the same time. The likely cause is that when a RAID drive fails data is automatically re-distributed across the remaining drives, but this makes each individual hard drive work excessively hard as lots of data needs to be read and written to the drive, thus this extra work causes another hard drive to fail.

According to Cayman Islands computer services officials they restored 10tb of data, mostly from tape backups. Unfortunately Cayman officials report that human error meant that not all data was restored. After reviewing the backup logs, it was determined that the RCIPS server was only manually updated with three of the four logical hard drives. So a portion of the server did not get backed up.” Mr. Liebaers (The Acting Information Commissioner) asked. “How is it that a backup is corrupted and nobody knows? … Aren’t backups tested occasionally to make sure they actually contain real, usable data? Apparently that was not the case.”

Mr. Whittaker said the Computer Services Department had found that a portion of the RCIPS server “did not get backed up.”

It is reported that much effort has been put in by data recovery specialists who have managed to restore all but 4gb of data – however it is thought that some data will be lost forever.

The Cayman Islands Computer Services Department has reported that they have put “more checks and balances” in place to ensure that data will not be lost in the future.

One commentator suggested “Somebody was not doing a very good job – on a hurricane prone island disaster planning should make this level of data loss impossible. It is likely that the vast majority of this data is not updated on a regular basis, thus most of it could be remotely backed up multiple times also this amount of data could be easily stored on magnetic tape and stored off-site with further copies sent completely off the islands for protection from Fire, Flood or Hurricane? So even if a full backup was only done every quarter, over 90% is still protected.

This story has a lesson for any organisation that holds data:

Have adequate in house backup software with whatever redundancy is necessary to allow for multiple device failures.

Do periodic test restores to ensure that backups are actually backing up all data.

Have adequate offsite backups, whether physical media removed to a safe location or via a cloud backup service.