Dead in the Water!

Like a line from a bad novel, it was a dark and stormy ike a line from a bad novel, it was a dark and stormy night in Halifax back on Jan. 14, 2009. Then the power went out. Two backup generators kicked in at the year-old Halifax Wastewater Treatment Facility (WWTF), operated by Halifax Water, bringing the plant’s systems back on line. Then one of the backup generators overloaded. Wastewater pumps shut down and control power to the inlet gate for the 2.4-m (8-ft) dia. sewage inlet tunnel failed.

In about 20 minutes flat, raw sewage roaring past the partially open gate flooded the 10.7-m-wide by 26.5-m-long by 20.2-m-deep (35x87x66 ft) wet well under the building, including underground chambers full of control and process equipment, rising to the hydraulic grade level – about one metre below street level.

This unexpected chain of events, triggered by an imbalance in how the generators shared the demand on them for backup power, resulted in massive equipment damage and a total shutdown of the facility. It took 17 months to return it to full operation.

The facility’s six-person maintenance team is made up of process technicians, operators and labourers performing the regular maintenance required for day-to-day operations. “All the bigger work [is contracted] out to Black & MacDonald and other contractors,” says Susheell Arora, director of wastewater operations at the facility. “Siemens, for example, does the thermographic scans. We don’t have electricians or mechanical people who can handle a problem this large.”

At the bottom of the wet well, only the lower four metres of which is actually supposed to be wet, sit five 650-hp pumps that weigh five tonnes apiece and stand 4.3 m (14.1 ft) high. When the rising sewage got into the junction boxes in the wet well, it travelled through the electrical cables and into the pumps, ruining them.

Other equipment that lay submerged and ruined under the sewage included the boilers that heat the plant, 17 sludge pumps, 10 chemical feed pumps, the water recycling system and its three 30-hp motors, all the processing systems that run the plant, and the 4.6 m by 6.1 m (15.1 × 20 ft) motor control centre – packed with starters, variable-frequency drives, big electrical panels and control panels that run the systems and equipment.

Adding insult to injury, a two-day snowstorm hit the city starting the very next day, yet the plant doors had to be left open because the ventilation system was down.

The city hired Black & MacDonald, a multi-trade contractor offering maintenance and other services, as the prime contractor for the plant cleanup and rebuilding operation, but first, the flooding had to be brought under control and the facility systems safely secured. “Water was still coming in, so upstream of the plant we lowered a pump into a chamber and pumped away most of the flow before it could enter the plant,” Arora explains. “We drained down the wet well to 3.7 metres (12 ft) below ground with an outdoor pump. Then we brought pumps inside, lowered them through an access way into the wet well, and drained the well.”

Removing the five-tonne pumps illustrated the many difficult tasks that the contractors faced. Since one of the two cranes built into the wet well for removing the pumps had drowned, crews had to bring in manual hoists to lift the pumps one-by-one, about six metres, then transfer them laterally to a second crane and out of the building. “It was a big task,” Arora recalls.

From the beginning of the recovery, the facility’s maintenance and operational staff made sure that conditions were safe for the cleanup and repair crews, and lent their expertise to the operation. Their first task was to secure the facility and make sure all personnel were safe. Then they began planning the restoration effort, from pumping out the wet well to organizing the ordering of replacement equipment.

“Our people coordinated the contractors [and] gave the guidance – for example, explaining the consequences of touching electrical panels or turning on pumps. These are complex systems. Our people were busy inventorying equipment, planning and working around the clock. Our operators are knowledgeable about the plant and could tell the contractors about the consequences of any actions they would take,” Arora explains.

Replacing the damaged equipment was a massive task. Arora outlines the challenge. “The control panels were custom built. Hundreds of components go into the control panels and the manufacturers in the United States had to build new control panels from scratch.

“The pumps are not off-the-shelf. You have to replace the pumps with the same model and make, to fit the piping and electrical systems. The five big pumps came from Germany, with a six-month delivery time. Other equipment had an eight-month lead time … parts came from all over the place. There were hundreds of subcontractors and thousands of suppliers for the original equipment.”

It certainly helped that Black & MacDonald has a big presence in Halifax and could come over to the facility right after the flood. Halifax has several shipyards and also services offshore oil and gas production platforms, and the local suppliers that serve them were able to obtain a lot of the replacement equipment.

The post-flood investigation resulted in some changes designed to prevent such an event from happening again. For example, says Arora, “The junction boxes and control equipment are now upstairs where they cannot be flooded. We installed backup equipment that can close the gate and we made operational changes to handle the flows better in the plant. We balanced the generators better so they would not overload.”

Halifax Water brought the facility back on line in stages: It began handling dry weather flows (sewage, but not rain runoff) in the fall of 2009; it returned the Advanced-Primary Treatment system to operation in May 2010; and the next month operators brought the ultraviolet disinfection back on-line, and the facility resumed full operations.

Among other measures to decrease the likelihood of a repeat flood, Halifax Water adopted the practice of conducting hazard operability assessments – a way to probe for problems and risks in plant designs and operations. Arora believes this is a valuable practice for any plant.

“Conduct tabletop exercises. Run all your scenarios and ask yourself ‘what will happen if this fails’? We are running a hazard operability assessment on a new plant we are building in the Eastern Passage. We bring in fresh eyes – facilitators who know the industry. It is a very open process.

“We now provide a lot more training to our people. You want to be prepared for any emergency.”

Montreal-based Carroll McCormick is the senior contributing editor for Machinery & Equipment MRO.