New Front-End Crate Low Voltage Power Supplies for HAWC --------------------------------------------------------- Original Version 4-JAN-2011 Most Recent Version 16-MAY-2011 Because the Acopian power supplies that were used for the Front-End crates in the Milagro Experiment are now 20 years old and because they do not provide an easy path for remote control or remote monitoring the decision was made by the HAWC Experiment to replace these supplies with modern ones that include remote control and monitoring. Front-End Crate Low Voltage Power Supply Requirements The following is a list of requirements for the new low voltage power supplies for the Front-End crates: 1. There will be 4 crates of Front-End Boards. Because we want to monitor and control these crates separately we will need 4 sets of Front-End Board power supplies, i.e. one per crate. 2. The Front-End Boards require positive and negative 5.0 Volt supplies for the Analog cards and a negative 5.2 Volt supply for the Digital card. 3. Per Analog-Digital card pair the power requirements are: +5.0 V Analog at 3.65 Amps -5.0 V Analog at 4.25 Amps -5.2 V Digital at 4.40 Amps These current requirements come from the nicely written "Milagro Front-End Electronics Manual" and they were confirmed by measurements in the MSU test setup. Note that to deliver the nominal supply voltages to the power planes on the Analog and Digital cards that the power supplies themselves will need to provide about 100 mV of additional output to compensate for the voltage drop on: the power cables to the backplane, the backplane distribution, drop across the per card power filter chokes, and drop across the on card fuses. Measurement of the choke and fuse drops are included in the MSU test setup notes. 4. The Front-End Crate power supplies must be sized to operate either 900 or 1200 PMT channels which implies either a 60 Front-End Board pair setup with 15 per crate or else a 75 Front-End Board pair setup with 19 per crate. 5. Rounding this up to 20 card pairs per crate gives the following current requirements per crate: +5.0 V Analog at 73 Amps --> 365 Watts -5.0 V Analog at 85 Amps --> 425 Watts -5.2 V Digital at 88 Amps --> 458 Watts The sum is about 1250 Watts of DC power used by 20 cards in a crate. Because 73, 85, and 88 Amps are all about the same there is considerable advantage in using the same type of power supply module to power each of these loads. 6. Using a normal rule of thumb we should select supplies that are large enough so that during normal operation they are running at under 2/3 of their rated capacity. We may even want to consider a more conservative, i.e. higher capacity supply selection rule, because of the high altitude and remote unattended operation of the HAWC experiment. A 2/3 of rated capacity rule implies that we need supplies that are rated for at least a 132 Amp continuous operation. 7. To this 1250 Watt DC power requirement listed above we should add about: 2% for the power distribution voltage drops noted in point #3 and at least 15% for the efficiency of the power supplies. This implies about 1470 Watts of AC line power required by each crate's low voltage power supplies. To this 1470 Watts of AC line power per Front-End crate we need to add the power required by the air blower for each crate which depending on the type of blower that is used may be another couple of hundred Watts per crate. 8. It is my understanding that 208 VAC 3 phase line power will be available at the HAWC site. 9. We should select supplies that do not have a large startup current spike that could cause trouble for the AC power system or other equipment at HAWC. 10. Monitoring and Remote Control of the Front-End Crate power supplies is required in the HAWC experiment. It would be convenient if the monitoring and remote control features of the Front-End crate power supplies matched with other components that will be used in HAWC. I believe that ethernet is the preferred media connection for monitoring and remote control in HAWC. 11. The cooling air flow direction of the selected power supplies will need to match that of other components that are located in the racks with these supplies. If a supply is located in a rack adjacent with its Front-End crate then vertical air flow will be required. Extra vertical rack space will be required if it is necessary to change the air flow direction. A realistic and detailed rack layout plan should be made to specify this aspect of these power supplies. 12. It is normal to require safety sensors (e.g. air flow, temperature, and smoke detector) on a high power system like the Front-End crates. This is especially true on a system that is designed for remote unattended operation. How will this be handled at HAWC ? For example: - Will each rack in the HAWC trailer have a uniform set of safety sensors feeding that rack's "safety monitor box" and a fault signal from any sensor will cause all AC line power being feed to that rack to trip off ? - Or will each component in the HAWC need to have its own unique safety system an sensors ? Specifically do we need to specify Front-End crate power supplies that have trip off interlock inputs, or make specialized "trip-off-able" AC line power distribution boxes for the Front-End crate power supplies, or with this be a uniform feature of all HAWC trailer racks ? I strongly suggest a uniform safety system for all racks in HAWC that causes a trip-off of the AC power be installed. 13. We need to select Front-End crate power supplies that can be removed from the rack, without disturbing too much adjacent equipment, when one of them fails. Here again a rational detailed rack layout would help specify this aspect of the requirements for the Front-End crate power supplies. Things that may make removal of a supply difficult include: - The supply is heavy and you can not get your hands in to the rack to lift it. Shelf bracket supports for the Front-End crate supplies will clearly be necessary. - The high current terminals are likely to be 5/16" bolts or something like that and thus we need to have room to get wrenches or socket drivers into the rack to work on these connections. - The high current DC cables are likely to be stiff and need to be tied down and thus hard to bend out of the way to remove things. - There is likely to be a mass of PMT coax signal cables of Front-End crate output twist-flat cables in the vicinity of these supplies and we should design things so that one does not need to disturb these cables to remove a supply. Wiener Model PL506 Power Supplies The following is an evaluation to see how well the Wiener model PL506 low voltage supplies will meet the above requirements. The model PL506 power supply is a modular system that is assembled from about 6 different types of modules to meet the requirements of a given application. 1. Reading between the lines, the original bid that we had from Wiener for these supplies (offer No. 1009-367 from 17-Sept-10) indicates the intent to purchase 2 supplies with each supply consisting of: 1x PBN506-3U-RASO "power bin" that mounts in the rack, the heavy DC output power cables attach to this "bin" 1x PBX506-EX "power box" that holds the various modules, has a front panel, has the Ethernet monitoring/control, and pulls out of the power bin as a unit 1x PFC-?? module the power entry / power factor correction module that makes "bulk" DC power at about 380 VDC 6x MEH-02/07 modules each one of which makes a DC output between 2V and 7V DC at up to 115 Amps 2. I believe that the intent of this original bid was to run the 4 Front-End crates off of just 2 supply chassis with each supply chassis providing 6 outputs. I believe that this will not work because: - We would not have independent power control of each Front-End crate - rather we could only turn on and off pairs of crates. - A pair of crates with 20 Front-End boards in each crate requires about 2500 Watts of DC power (from point #5 in the requirements list above). But The Wiener PL506 supply can provide only 1000 Watts of DC output when operating from 120 VAC line power. When operating from >210 VAC <264 VAC the PL506 can provide a maximum of 3000 Watts of DC output. This de-rating of the DC output Watts when operating from the lower AC line voltage is a common feature of modern supplies that have "auto-ranging" AC line input with power factor correction. I can provide details about why this is so if you are interested. Even running this supply from the higher line voltage would require it to operate at 83% of it maximum capacity to power two fully loaded Front-End crates. To me that is too heavy of a load for a system that needs to be highly reliable in unattended operation at high altitude. 3. For the rest of this evaluation I will assume that each of the 4 Front-End crates has its own Wiener PL506 supply and that each of these supplies has just 3 of the MEH-02/07 output modules and that these supplies are operating from the higher line voltage. We have now received a new bid for this configuration, offer number 1103-127 from 21-March-2011. The cost is about $5,862 per power supply chassis in this configuration. 4. The Wiener PL506 supplies are nice well thought out high quality equipment. The front panel, ethernet interface, PFC-?? module and MEH-02/07 modules are basically the same parts that are used in the their 6000 series VME crates, i.e. the VME-64X crate that HAWC will use. I have complete schematic print sets and maintenance documentation for the modules in the supplies. This is basically limited circulation public information. That is this information is available so that these supplies could be maintained by qualified electronics technicians. HAWC will not get stuck with secret proprietary equipment that no one but the manufacturer can take care of. I have had about 7 Wiener supplies with this type of internal modules in continuous operation at Fermilab for about 5 years. We had one failure during the first couple of hours of operation and then one more failure when we turned the equipment back on after a power outage. We keep spare supplies on hand at the Fermi experiment and Wiener has been good about quickly repairing the problems. 5. The 115 Amp maximum output from these supplies does exceed the 88 Amp requirement of a fully loaded 20 card Front-End crate. 6. The 115 Amp maximum output from these supplies does not meet the 2/3 maximum continuous use load limit rule of thumb for the -5.0 Volt and -5.2 Volt supplies on a fully loaded 20 card Front-End crate. The 115 Amp maximum output from these supplies does meet the 2/3 maximum continuous use load limit rule of thumb with up to 17.4 cards in the crate, i.e. it does satisfy the 2/3 rule for a 900 channel system with 15 cards per crate. 7. The Wiener supplies do "soft-start" both when the AC input power comes on and when the DC outputs are turned on. The AC inrush current is limited to 12 Amps at AC turn on. At DC output turn on the outputs ramp up to their set points over a default 50 msec period. These soft start features should help protect the UPS from spikes that could cause it problems. 8. As far as I know, the remote control and monitoring interface to these supplies is exactly the same as that provided in the Wiener VME-64X crates that HAWC will use. 9. The air flow direction through these supplies is vertical going up. This matches the air flow direction that will most likely be used in the Front-End crates and the VME crates that are in the same racks as these supplies. I do suggest the use of external fan trays to help push the air through these supplies. 10. The Wiener PL506 supplies do have some provision for temperature sensor input that can trip off their DC outputs if a temperature limit is exceeded. Their is no provision for air flow sensor or smoke detector inputs. In any case a more rational rack safety system has a fault signal from any of these sensors trip off the whole AC line power feed to the rack (not just trip off the operation of one of the objects that is in the rack). We should think about a uniform rack safety system setup for HAWC. All of the modern HEP experiments include such equipment. 11. The Wiener PL506 supply unplugs and is removed from the rack as a complete unit. Just the "bin" part of the supply remains in the rack - the whole "box" part of the supply, with all its various modules, comes out as a unit. This makes the removal of a supply from the rack about as easy as possible. All of the heavy stiff high current DC power wiring remains in place. This heavy DC power wiring is attached to 8mm threaded bolt connectors on the "bin". To get the most benefit from this nice power supply system HAWC still needs careful rational rack and cable layout design so that for example a water fall of signal cables is not blocking the removal of a power supply from it "bin". General Comments: 1. Using any new supplies will require replacement of the heavy DC power cables from the supplies to the crates. Making these cables will require money for parts and technician time. These cables need to be carefully designed, routed and installed. At the 100 Amp level cabling should be taken seriously and designs reviewed carefully. For equipment of this type at Fermi you need a design review to get an operating permit for it. Over the years I have learned that this is a smart policy.