MD Anderson Cancer Center – Houston, TX
With 61 standby generators, 200 automatic transfer switches, 300,000 gallons of diesel fuel in 18 tanks to support 15 million square feet in 40+ structures spread over 3 campuses, the University of Texas MD Anderson Cancer Center (MDACC) in Houston, Texas needed one (1) comprehensive protocol for compliance, control and 100% reliability of their emergency power supply system (EPSS). The objective: to ensure the 24×7 safety of over 100,000 patients per year and 20,000 staff members, not to mention the protection of millions of dollars in fixed equipment, cash flow and permanent structures.
In addition to hardening the EPSS protocols, hospital facility directors wanted to investigate the rather large sums of Demand Response dollars seemingly available for just the promise to run some/all their generators when the local grids were stressed. An obvious question was, “can we take advantage of these opportunities without compromising the main objective of our emergency power systems?”
It was decided to first tackle the compliance issue by reviewing each EPSS’s components (generators, ATSs, breakers, fuel tanks, etc.) to ensure each was code compliant and installed according to the latest NFPA and building standards, regardless of “grandfathering” allowances. It was thought that since the variety of makes, models and components were installed by several different contractors over the past 25 years, non-compliance in some areas was highly probable. One of the main concerns was whether some of the essential system loads were being powered by the correct life safety, critical or equipment branch ATSs, or even by a non-essential panel. Related to this issue was whether there had been proper breaker coordination to prevent the tripping of an upstream circuit breaker which was feeding multiple distribution panels when a smaller breaker in a single panel would have cleared a single fault.
Compliant fuel systems were also a concern because of the large amounts of fuel being stored for indefinite periods. This fact coupled with having to deliver uninterrupted fuel supplies to multiple engines during times of extended outages made fuel management problematic. Redundant valves, multistage filtration, bypass plumbing and a wide range of ambient temperatures were all issues to be considered. Laboratory testing of fuel according to enhanced ASTM standards was implemented.
Arc flash studies were begun in earnest to meet the strict requirements of NFPA 70E and 29 CFR, 1910, Subpart S. While personal protection equipment (PPE) had always been used during electrical maintenance and repair tasks, some emergency panels where contractors were connecting load banks and portable generators needed labeling to be updated for conformity.
Comprehensive testing and reporting procedures had to be in place for review by multiple surveyors which dictated redundant and easily accessible files through indexed archives. Every item mandated by NFPA 110,Emergency and Standby Power Systems had to be included as all authorities having jurisdiction (AHJs) referred to NFPA 110 for guidelines and interpretations. While the Joint Commission’s Environment of Care standards referred to NFPA 110 for testing and maintenance practices, these standards were found to be actually less demanding than those suggested in OEM manuals. A thorough review of all OEM manuals was made and protocols enhanced.
While NPA 110 was the “minimum” standard mandated by the AHJs, Karen Mooney, Executive Director, Administrative Facilities and Campus Operations, wanted to go a step further and research all of the best practices for maintenance and testing now being employed by the most discerning individuals in healthcare facility design and management. Marching orders were given.
After hearing of one incident at another facility where a generator was found running without anyone knowing it for almost 2 days, it was decided that having a “must respond” reporting/monitoring system that would save a lot of embarrassment. Depending on remote annunciators capable of monitoring only basic functions of EPSS components required by NFPA 110 was inadequate. Since MDACC was continually expanding their patient services, the control system had to be scalable as clinics or labs came online and improved software became available. Instantaneous 24×7 response to any failure or anomaly was to be considered only a basic feature of MDACC’s standards.
Redundant components (N+1) and multiple EPSS contractors were to be considered in establishing a minimum protocol. The first step was to eliminate all obvious, and not so obvious, single points of failure in all electrical, mechanical and plumbing systems. Hi-mortality spare parts were purchased and stocked for use in emergencies and when annual services were completed. Contractors were required to replace these parts with fresh inventory. Contractors and staff were instructed to conduct a full system test after any service or repair of any EPSS component in order to conform to NFPA 110 standards and to ensure the service or repair was successful. Contingency contracts for the delivery of fuel and parts suppliers were executed. Laboratory testing of diesel fuel, lubricating oil and coolant was implemented. Infra-red and megger testing were made standard practice. Loading all generator sets beyond NFPA 110 minimum standards became mandatory. Careful attention was given to ensure the loading of the generators not only met the NFPA 110 minimum standards but also provided an exercise similar to a manufacturer’s acceptance test.
After several weather related utility power outages Karen realized that MDACC’s emergency power system had to be more reliable than the utility – available immediately and without failure.
After diligent research is was decided that Demand Response (DR) was no silver bullet. The three main questions were: (1) Can we use EPSS assets for economic benefit without decreasing the reliability of those assets; (2) would any patient or staff services be jeopardized, and; (3) were there existing environmental regulations which would prevent participation?
The first two questions were put to rest when Karen decided to proceed with caution and only use those EPSSs which contained N+1 components and served only the least critical loads.
MD Anderson is located in Harris County, which the Environmental Protection Agency (EPA) Green Book classifies as a “Severe 15” 8-hour ozone nonattainment area. The information to be considered included: 1) how are the generators permitted currently, 2) do those permits allow for participating in demand response or will a new type of permit be required, and 3) would participation increase the number of hours operated such that the resulting emissions increase triggers additional permitting requirements? A thorough research of the Texas Administrative Code (TAC) and the Texas Council of Environmental Quality (TCEQ) was performed.
As the EPSS team at MD Anderson continues to refine their operating methods, the recent improvements were a great opportunity for building operators, technicians and customers to reevaluate the approach to meeting demands as well as looking for new opportunities to enhance the operation with always an eye to costs. The expansion of MDACC over the last 15 years came quickly and although the ‘pockets’ of growth were managed well, there was not attention to the more strategic management of EPSS. Not all of the efforts described were easy to implement and some came at a cost whether that was financial or changing of processes and accountability. However every effort demonstrated a return on the investment by poising the EPSS team to not only operate efficiently, but also aligning resources to respond to future demands.
By: Dan Chisholm, Sr., Emergency Power Consultant, MGI Consulting