Backing up your CMMS

Although I have mentioned in this column before how important it is to back up your CMMS database, recent events have brought me to discussing it again.

In the past six months, I have seen some disastrous failures of data backup systems. Companies that thought their CMMS data was secure faced some incredible setbacks when their database servers failed.

I am going to tell you four stories about some recent database losses. They will demonstrate why backups and technical support for your database are so important.

The first story comes from one of my company’s clients in North Carolina. This brand-new facility just started production in January 2013 and it has been populating its CMMS software with equipment, inventory and preventive maintenance tasks since construction started on the plant two years ago. Hours and hours of critical equipment information was gathered and entered into its CMMS program. Hundreds of work orders were created to record equipment, process failures and corrective actions during commissioning that could be used at new plants in the future.

On July 22, the CMMS server failed. The facility IT person was confident that the server was successfully backed up every evening to a tape drive. When the staff checked the tape drive for the backup to retrieve the CMMS database, they found that there was none. Upon investigation, it was discovered the server backup log showed an exception error for the CMMS database.

What they found was the database was continually active 24 hours a day, every day. The technology they use for backups would only back up databases that were inactive during the backup process; therefore no backups of the CMMS were being successfully initiated.

After checking with a number of their other plants, they discovered several of them had the same problem. As I write this article, four days have passed since this incident at the plant. One backup has been found that is over a year old. Efforts by the plant’s IT person and the CMMS support team have not been able to recover any more recent data from the server.

They now realize that much of their data is missing and a plan will be needed to re-enter the data. This will take a substantial amount of time and effort by the maintenance department, time they don’t have available. Steps have been taken to correct the backup issue with scripts that will back up the database to the server even when the CMMS is active, and the solution has been sent to other plants with the same issue.

Next up is another large corporation. It manufactures bread products in Quebec. A young engineering student, although warned not to, attempted to manipulate some of the database tables in the company’s CMMS software. In the process of doing so, he crashed its CMMS program. The database was corrupted and inoperable. The staff went to restore its database from a tape backup, only to discover that the backup tape system had not been operating for several months.

With mountains of equipment PMs, repairs and sanitation records, there was no option but to try to recover the data from the corrupted files. It took over three weeks for its CMMS tech support team to recover and reconstruct the database. During this time, the CMMS was out of service, which caused a great deal of hardship for the maintenance and purchasing departments. 

This incident prompted the company to regularly check its plant and other locations to make sure its backup systems were actually working.

Another case comes from a building materials manufacturer in New Brunswick. Late last fall, its CMMS suddenly stopped working. The main server that housed both the CMMS database and the daily backup had failed. This particular server used a multiple drive technology that split the database over several hard drives, which was supposed to be more reliable and safer for data storage.

When the server failed, the staff discovered there was no local support for a server of this type and that data recovery was not a simple process. The server had to be sent to a Toronto data recovery company that specialized in these types of servers. Although the turnaround period was three days after the server was received from New Brunswick, it carried an expensive price tag. All the data was recovered and steps were taken to run the backup on another system, separate from the database server.

Finally, a Nova Scotia organization learned that backing up its CMMS was critical to its operations. It has several facilities, with each site using a stand-alone computer to host its CMMS program and database. The sites are not connected to the organization’s network, nor do they have backup systems in place.

Because its software and databases are all located on one computer at each site, the chances of losing the complete database is very probable and in fact has happened at three of the organization’s facilities in the past six months.

You might say they haven’t learned very much from their recent losses; however two of the facilities wanted to start their CMMS over again because their first ones were never properly formatted with correct naming conventions and nomenclatures. As a result, the software was hard to navigate and rarely used. Reloading the CMMS program was not a big issue, however it did point out to the staff that they must back up their data.

The corporate IT at this point does not have a common network where the facility databases can be stored and backed up properly to a common server. For now, each facility has purchased an external hard drive and has scheduled daily backups of its database in case there are any future failures of the stand-alone workstations.

There are alternatives to maintaining your own database and backup systems. Most CMMS companies offer web-based applications where the CMMS program is accessed over the web. The hosting company maintains the data and backup systems at secure data storage sites. However, the cost of ownership is often much more than owning and maintaining your own software and hardware. Also, some organizations have security protocols that do not allow web-based software applications; due to security risks, they must maintain their own server systems.

So let’s see what we have learned from other people’s CMMS data failures.

1. Even if you are sure your CMMS is being backed up, ask your IT department to double check the backup files.

2. If you are running a stand-alone com-puter, find a way to back up your data. The external drive solution works well.

3. Maintain your annual tech support service from your CMMS provider. In two of the cases mentioned here, that
support helped to retrieve lost data. Without this help, the data would not have been recovered.

Building a CMMS database takes a lot of time. The information it holds is critical to your maintenance operations. Treat it with care, protect its integrity and back it up regularly.

Like most things in life, you don’t know how much you will miss it until it is gone. 

Peter Phillips of Trailwalk Holdings, a Nova Scotia-based CMMS consulting and training company, can be reached at 902-798-3601 or by sending an e-mail to peter@trailwalk.ca.