Instructions for Powering Up Racks M101, M122 M123, M124 the Trigger Framework, the SCL Hub-End, and the Routing Master ---------------------------------------------------------------- Original Rev. 17-SEP-1999 Current Rev. 7-SEP-2010 These are instructions for controlling the power to and for cold starting the following systems: Trigger Framework, SCL Hub-End, and Routing Master. If there are any questions please contact: Dan Edmunds at 517-884-5521 or Philippe Laurens at 517-884-5522. Please verify that you have the current version of these instructions. The current version of this document is in: http://www.pa.msu.edu/hep/d0/ftp/l1/framework/hardware/rack_crate/ framework_power_control_procedures.txt Turning Power OFF to the TFW, SCL, and RM ----------------------------------------- Emergency Shutdown Open the front door to rack M124 and remove one of the two keys that are located at the right hand side about 3 feet up from the bottom of the rack just below the air inlet grill. Removing either one of these keys will immediately drop all power in racks M101, M122, M123, and M124. Normal Shutdown In each of the racks: M101, M122, M123, and M124 do the following: Open the front door and on each power supply chassis in the rack, first turn OFF the small front panel toggle switch that is labeled "DC Power" and then turn OFF the large front panel 3 phase breaker which is labeled "AC Power". In each of these racks there are either 2 or 3 power supply chassis and they are located in the bottom half of the rack. In racks M122, M123, and M124 turn OFF the "Rack Power" toggle switch. This toggle switch is located on the cable pass through panel just below the air intake grill and just above the Power Supplies at about knee height. The air blower will stop running when you turn the Rack Power switch OFF. Trigger Control Computer Power If you are doing a complete shutdown of the system (because there is going to be a power outage at D-Zero or something like that) then you also need to turn OFF the Trigger Control Computer, d0tcc1. Except for D-Zero power outages the TTC is left running all of the time. To turn it OFF please exit the TRICS application and then run shutdown and then turn its power off via the labeled power strip that you can reach from the back of the air conditioner rack M125. Turning Power ON and Starting the TFW, SCL, and RM --------------------------------------------------- To help keep these instructions readable they have been made fairly compact. Following this list of steps you will find a section with additional background information. Note: If you are restoring power to the TFW after an RMI power trip please complete the part of the power down procedure listed above where you open each rack door and turn off both the AC and DC switches on each power supply chassis in the rack. Only after these power supply switches are all off should you start the steps to restore power and turn the TFW back ON. 1. Master Clock The Master Clock must be turned on, down loaded, and running before you turn on and attempt to Configure and use the Trigger Framework. Failure to get the Master Clock running first will just waste time and cause confusion when trying to configure the Trigger Framework. There are separate instructions for starting the Master Clock which can be found at: www.pa.msu.edu/hep/d0/ftp/l1/framework/hardware/rack_crate/ master_clock_instructions.txt 2. Turn Power ON In each of the four racks (M101, M122, M123, M124) first turn on the "Rack Power" switch then turn on the power supply "AC" switches then, after a wait of at least 5 seconds, turn on the power supply "DC" switches. Rack M101 does not have a "Rack Power" switch. Turn on the Rack Power by flipping up the Rack power toggle switch and holding in the push button next to it for about 5 seconds (i.e. until the air blowers have had time to come up to speed). These switches are located just below the air intake grill in the front of each rack. On each power supply first turn on its large 3 phase "AC Power" switch and then after a delay of at least 5 seconds turn on its "DC Power" switch. The power supplies are located in the lower half of each rack. There are 3 power supplies in racks M122 and M123. There are 2 power supplies in racks M101 and M124. If it has been turned off then you will also need to turn ON the Trigger Control Computer, d0tcc1.fnal.gov Power for this computer comes from an outlet strip that is located in rack M125 on top of the air conditioner. There is a label on this output strip indicating its function and you can reach the switch on it by standing in back of the air conditioner. When TCC1 starts up it should automatically start up its TRICS application program. 3. Configure the TFW FPGAs The first step is to "Configure" the FPGAs in the Trigger Framework. Configuring the FPGAs is the process that takes a blank FPGA (i.e. their state right after power up) and puts logic into the FPGA. To accomplish this just click the "Configure" button in the Master Command File Menu of TRICS. The Configure button is the top left hand button in the TRICS Master Command File menu. It will take about 5 minutes for the Configure command to run. As it is running the log window will show its progress as TCC configures each FPGA (or groups of FPGAs). The final log file entry from the Configure command should be something like, "FPGAs Configured 1702, Errors 0". This step configures all the FPGAs except for those in the Routing Master. 4. Configure the Routing Master FPGAs and Start the RM Running The Routing Master is currently located in the bottom crate in rack M101. Power must have been turned ON to this crate long enough for its SBC to have booted before you start the following steps. This SBC will normally boot in less than 1 minute. To run the scripts that will be referenced in the following steps the d0lxmast machine must be powered up and running. a. From an online machine run 'l3xdaq_stop RM' which will stop the relevant readout processes in the Routing Master's SBC (and thus give TCC exclusive use of the VME bus in this crate to Configure the FPGAs in the RM). b. Use the bottom section of TCC's "Configure FPGAs" menu to "Execute" the dcf file that configures the Routing Master FPGAs. The dcf file to use is: D0_Config\M101_Routing_Master_All.dcf The button to take you to the "Configure FPGAs" menu is at the upper right hand side of the TRICS main menu. When you "Execute" the file D0_Config\M101_Routing_Master_All.dcf you will likely see an error reported in the TRICS log file on the left hand side of the screen. That is caused by previous VME cycles interfering with TCC's attempt to Configure the Routing Master's FPGAs. Just execute this dcf file a second time and it should run error free. c. Finally, from an online machine run 'l3xdaq_reset RM' which will restart the RM software running in this SBC. 5. Initialize the Trigger Framework During the Initialization process TCC will load default values into thousands of control registers in the Trigger Framework hardware to bring the system to a known quiescent state. To accomplish the Initialization of the Trigger Framework just click the Initialize button in the Master Command file menu. The Initialize button is the top right hand item in the Master Command File menu. The Initialize command will take about 5 seconds to complete. At the completion of this step the Trigger Framework is ready for COOR to use. Background Information about the above Steps -------------------------------------------- 1. The Master Clock must be turned on, down loaded, and running before you turn on and attempt to configure and use the Trigger Framework. This is because the TFW uses the Master Clock for all of its timing signals. The TFW can not even complete a VME cycle until the Master Clock is running. Turning power ON to the TFW before the Master Clock is running will just cause things to hang. The instructions for starting the Master Clock are at: www.pa.msu.edu/hep/d0/ftp/l1/framework/hardware/rack_crate/ master_clock_instructions.txt 2. For power to flow to the Trigger Framework the TFW "Contactor Box", which is located up above M124, must have received its "Global Permit" signal. Without Global Permit this Contactor Box will not allow power to flow to any of the racks: M101, M122, M123, or M124. To receive Global Permit: both of the keys must be in the panel in the front of M124, the VESDA must not see any smoke problems, the RMI based Smoke Detectors and Drip Detectors for the TFW must not see any problems, and none of the temperature limit detectors may see a problem. This RMI is located in the bottom of Rack M124. It is connected to Smoke Detectors and Drip Detector Strips in the 4 racks listed above. If the power to D-Zero has been off then you may need to push the Reset button on the front panel of this RMI before you get the Global Permit signal. Standing next to the air conditioner you can look up at the Contactor Box above M124 and see the LED labeled, "Global Permit". This LED is illuminated when the Contactor Box has Global Permit and is thus allowed to send power to the racks. If this LED is off then you have to investigate why it is off before you can proceed to turn the TFW ON. The Trigger Framework gets its power from breaker boxes along the East wall of the first floor MCH. Which breaker supplies which TFW rack is described in the following: www.pa.msu.edu/hep/d0/ftp/l1/framework/hardware/rack_crate/ power_panel_and_contactor_boxes.txt When you turn on the upper of the 2 power supplies in rack M124, i.e. the power supply for the SCL Hub-End, you will notice that the 6U VME crate at the bottom of M124 also comes on. This 6U crate is called the "Communications Crate" and it holds the equipment that lets the Trigger Control Computer "talk" with the various crates in the TFW, SCL, and RM. You need to have the Level 1 Trigger Control Computer, d0tcc1, turned ON and running its TRICS application program in order to Configure the TFW FPGAs and then to control the TFW. This computer is a normal PC and it should automatically start its TRICS program when it is turned ON. Except for when there is a power outage at D-Zero this computer is normally never turned off. 3. Configuring the FPGAs is the TFW is the process that takes a blank FPGA (i.e. their state right after power up) and puts logic into the FPGA. The configuration files that are loaded into the FPGAs are stored on the Trigger Control Computer. All of the FPGAs in the TFW are normally configured in one step. Because this requires many millions of I/O operations this process takes about 5 minutes to configure the 1500 FPGAs in the TFW. If the TCC has any problems configuring FPGAs you will see messages about this in red in the right hand log window on TCC's screen. When TCC is finished Configuring FPGAs you will see a final message in the log window indicating how many FPGAs have been configured and whether or not there were any errors. The error number should be zero. 4. The FPGAs in the Routing Master are Configured as a separate step. It is done this way because we need to have the Routing Master SBC stay off the VME bus while the RM's FPGAs are being Configured by TCC. During the Configuration process the only VME cycles that are allowed are those involved with the Configuration process. Once TCC has Configured the RM's FPGAs then the RM's SBC is told that it may start running it program again. 5. Manually telling TCC to Initialize the TFW is a good way to verify that all of its FPGAs have been Configured and that it is running OK. The log window on the right hand side of TCC's screen will show you any error messages in red if there are problems during the Initialization process. Once COOR is running and has connected to TCC it will issue its own command telling the TFW to Initialize. It is by this method that COOR knows the initial state of the TFW and can then proceed to setup the various triggers that people ask for.