Master Clock Instructions --------------------------- Original Rev. 23-MAR-01 Current Rev. 4-APR-06 The Master Clock is located on the first floor of the Moving Counting House in rack M100. These instructions are divided into 4 sections. 1. Power Up and Power Down of the Master Clock Rack. 2. Downloading the Master Clock, i.e. loading Time Lines, into the Sequencer module and controlling the various modes of the Sequencer and PCC Modules. 3. What you should see on the front panel LED display of the PCC and Sequencer modules. 4. A description of the alarms from the Master Clock that are seen in the Significant Event System. The current version of these instructions can be found at: http://www.pa.msu.edu/hep/d0/ftp/l1/framework/hardware/rack_crate/ master_clock_instructions.txt Power Up of the Master Clock ---------------------------- The Master Clock power switches are located near the top of the rack M100. There are 5 labeled switches on the power distribution box which controls the Master Clock. To power up the Master Clock turn ON the switches in the following order: Instrument Power Transformer Input Rack Monitor Master Clock - Water & Fan Push the RMI reset button. RMI is near the bottom of this rack. You will need to hold the button in long enough to give the water flow time to get going. Once the RMI shows no Faults (i.e. all green status LED) then turn ON the following: Master Clock - +5 Volt & Power Strip Master Clock - +-12 Volt & -5.2 Volt There is a NIM crate above the Master Clock 9U VME Crate. Turn on the front panel power switch for this NIM crates. It will take about 2 or 3 minutes for the processor in the Master Clock Crate slot #1 to down load its program and start up its various processes. You will know that this processor is loaded and running when the yellow "Module Select" LED at the top of each module in the Master Clock crate starts to flash once every 5 or 10 seconds. The flashing of the Module Select LED's once every 5 or 10 seconds shows that the process is running that maintains the EPICS view of the data in the various control status registers in the Master Clock. There is no point in going on to the next step in the start up procedure (downloading a Time Lines into the Master Clock) until this process is running. Power Down the Master Clock --------------------------- Turn off the NIM crate that is above the Master Clock VME crate. At the top of the Master Clock rack turn off the 3 Master Clock power switched in any order. Downloading a Time Lines into the Master Clock Sequencer and Controlling the Modes of the PCC and Sequencer ------------------------------------------------------------ During normal running there is no need to download the Master Clock. Downloading the Master Clock should only be necessary if the AC power has been off or if someone has disturbed the master clock. If you do decide to download the Master Clock, please do all steps in this procedure, i.e. do not reload just some of the sections of the Master Clock. It is NOT necessary to cycle the AC power to the Master Clock before downloading it. Downloading the Sequencer and controlling operating modes of the PCC and Sequencer modules is done via a program called the ClockGui. You can run the ClockGui from the d0run account. The current version of the ClockGui takes about 140 MBytes of memory which means that it will not run on some machines in the Control Room. The "epics variables" that control the Master Clock are locked so that the ClockGui program must be run from the machine where the DAQ Expect sits, i.e. node d0ol07. The current method to start the ClockGui is shown below. It is easiest to download and control the Master Clock by running two instances of the ClockGui. Starting and using two instances of the ClockGui is what is described in the following paragraphs. setup d0online cd /online/config/trg/master_clock/ ./clock.startup & ./clock.startup & On your screen you can place one instance of the ClockGui on the left hand side and leave it showing the "Status" display. Place the other instance of the ClockGui on the right hand side of the screen and click on the "TRG CSEQ M001" button. It will take about 2 minutes for the "TRG CSEQ M001" display to come up and show you a view of the Sequencer #1 Time Lines. The steps to download a Time Lines into the Sequencer #1 are the following: First load the Next Address Buffer 1. Left ClockGui Sequencer #1 Control: Clock Off Push the Sequencer #1 Control button and click Clock Off. Wait for Sequencer #1 Control green "Clock On" indicator to go out. 2. Right ClockGui From the top bar use: File Open and select /online/config/trg/master_clock/clk_sequencer_1_ddmmyy.txt Where ddmmyy is the date of this version of the Time Line data file for Sequencer #1. There should be only one version in this directory, i.e. the current known good version. Open the file and Wait for the ClockGui to read the file and repaint the time line display (10 seconds) 3. Right ClockGui Select Next Address Buffer 4. Right ClockGui Click Download (lower left side of window) Wait for status line to say Download Successful which should take just a second or two. Now load the Static Time Lines. 1. Right ClockGui Select Static Buffer 2. Right ClockGui Click Download (lower left side of window) Wait for status line to say Download Successful which should take just a second or two. Now load the Dynamic Time Lines. 1. Right ClockGui Select Dynamic Buffer 2. Right ClockGui Download (lower left side of window) You will be asked to confirm that you want to download the Dynamic Time Lines. Wait for status line to say Download Successful It will say that it loaded either the "A" or "B" dynamic time line buffer. 3. Right ClockGui Download (lower left side of window) You will be asked to confirm that you want to download the Dynamic Time Lines. Wait for status line to say Download Successful It will now say that it loaded the other dynamic time line buffer. Yes, you need to downloaded both the "A" and "B" dynamic time line buffers. Start the Clock, Clear Errors, and check the Clock Status 1. Left ClockGui Use the Sequencer #1 Control menu to issue the: Clock ON and then the Clear Errors commands. Examine the Sequencer #1 Green Status Indicators You need to have the following Green Status Indicators ON: Clock ON Sync Run Group A Output Enabled Select A MCLK Disabled Use Sequencer #1 Control with such menu options such as: Run Mode Sync, Step Mode Off, MCLK Disable and Clear Errors to obtain the list of Green Status Indicators shown above. If the display shows that you are running from the "B" dynamic time line buffer then use the right hand ClockGui to download the dynamic time line buffer yet one more time. Doing this will switch you to running from the "A" dynamic time line buffer. Wait 5 seconds for Sequencer #1 Red Error Display to update. For now it is normal to see errors for: BC Phase (BC PH), BC Phase 371, 742, 1113 If you have Red Error Displays for: Sequencer Hold, or Sequencer Halted, or Sequencer Parity Errors then there is a significant problem and the Master Clock is not running correctly. Sometimes it requires a second Clear Errors command to get rid of the Sequencer Hold error. You may want to go into the MCH and look at the Master Clock in M100 to confirm what Red Error LED's are on. Notes: 1. PCC Modes From the Left hand ClockGui you can control the operating modes of the PCC module. The only two useful modes are: FreeRun and Normal. Normal means locked to the Tevatron synchronization signals. The PCC should always be in Normal Mode (whether there is beam in the Tevatron or not). The only time that you would use the PCC FreeRun mode is when there is no beam in the Tevatron and there is some problem with the Tevatron synchronization signals. Being locked to the Tevatron is meaningful only when there is beam in the Tevatron. If there is a problem with the Tevatron Synchronization Signals, and you are not running with beam in the machine, then just put the PCC into FreeRun mode and the Master Clock will function and deliver the proper Time Line signals. The Sequencer module is always in Sync Mode. For beam running the PCC must be in Normal mode. 2. You should not need to reload the clock unless the power has been off. 3. We now have only one Sequencer module in the Master Clock. During Run I and Run IIA we had two Sequencers in the Master Clock. Master Clock Error LED's --------------------------- Both the PCC module and the Sequencer Modules have an array of red LED's near the top of their front panels that display various errors. The PCC module is the triple wide module located in slots 4,5,6. The number 1 Sequencer is located in slot 7. The number 1 Sequencer supplies timing signals to: the Trigger Framework, the Serial Command Link hub-end, the Luminosity Level 0 system, and the CFT Pulser. All other systems in D-Zero obtain their timing signals over the Serial Command Link (the SCL). In Run IIB we have only one Sequencer module in the Master Clock. The red LED Error indicators on these modules are "latched". That is an instantaneous error condition will set the LED on and it will stay on until it is cleared. During some Tevatron operation we have seen brief interruptions or discontinuities in the synchronization signals that come from the Tevatron. These interruptions to the Tevatron synchronization signals sometimes happen between stores especially during "studies" or shutdown periods. The Tevatron synchronization signals should never be interrupted during a normal Physics running Store. Now (since 19-APR-01) that we have a rational PCC module these brief interruptions do not cause the Master Clock, or the Trigger FW or the SCL Hub-End any problem but they do set various red error LED's. When you inspect the Master Clock you may see the PCC module with red error indicators set for: Sync_Missing, Sync_Timing, or RF_Missing. If these error indicators go out when you press the PCC's Clear_Errors push button located just below the error LED's then there is no problem. When you inspect the Sequencer you may see red error indicator for Sequencer_Hold. If this red error indicator goes out when you press the Clear_Errors push button located just below the error LED's then there is no problem. Note that for now it is normal to see red error indicators on the Sequencer set for: BC_Phase and BC_PH_371, and/or 742, 1113. This is normal. For now these error indicators will not go out and stay out when you press the clear errors button. Pressing the Clear_Errors on the PCC and the Sequencers does not do anything to "reset" these modules. It just clears the error LED's if the problem has gone away. If the problem currently exists then it will not clear the error indicator. Notes: Please do NOT press the Reset or Abort buttons on the processor module in slot number 1. The Sequencer #1 Green "BC PH TRIG" Status Indicator LED must be ON. If it is out this is an indication that the Time Line information has been lost. To verify that the Time Lines to the CFT LED Pulser and to the Level 0 Luminosity system are running OK, i.e. to verify that the Dynamic Time Lines are OK, check the following. On the Selector Fanout card in slot 9 verify that the green LED just to the right of the lemo output for Time Line 17 is ON. This indicates that the CFT LED Pulser is being sent its timing signal from the Master Clock. On the Selector Fanout card in slot 10 verify that the green LED's just to the right of the lemo outputs for Time Lines 12, 13, and 14 are all ON. This indicates that the L0 Luminosity System is being sent its timing signals from the Master Clock. Description of Master Clock Errors and Alarms -------------------------------------------------- The Master Clock can detect a number of error conditions. When one of these errors is detected by the Master Clock hardware it will cause an Alarm in the Significant Event System. These Alarms include: TRG_CPCC_M000/STIM = Phase Coherent Clock TeV_Sync Timing TRG_CPCC_M000/SMIS = Phase Coherent Clock TeV_Sync Missing TRG_CPCC_M000/RFMIS = Phase Coherent Clock TeV_RF Missing TRG_CPCC_M000/RFLCK = Phase Coherent Clock RF Not Locked TRG_CPCC_M000/MOD:R = Phase Coherent Clock Wrong Mode * TRG_CSEQ_M001/SQHLT = Sequencer Sequencer is Halted TRG_CSEQ_M001/SHDER = Sequencer Sequencer Hold error TRG_CSEQ_M001/SYMS = Sequencer Sync is Missing TRG_CSEQ_M001/PCMS = Sequencer P Clock is Missing TRG_CSEQ_M001/SCPER = Sequencer Static Memory Parity error TRG_CSEQ_M001/DCPER = Sequencer Dynamic Memory Parity error TRG_CSEQ_M001/CLKON:R = Sequencer Clock is OFF * TRG_CSEQ_M001/OUPEN:R = Sequencer Output is Disabled error * TRG_CSEQ_M001/SMOD:R = Sequencer is in Step Mode * TRG_CSEQ_M001/MOD:R = Sequencer is in Freerun Mode * TRG_CSEQ_M001/GRA:R = Sequencer Group A is OFF * TRG_CSEQ_M001/GRB:R = Sequencer Group B is ON * TRG_CSEQ_M001/SAB:R = Sequencer Group B is Selected * TRG_CSEQ_M001/MCLKD:R = Sequencer M Clock is Enabled * * --> This alarm implies an error that is most likely a problem with the programming of the module. See the information at the end of this note. During Beam Physics Running any of these Alarms will pause the run. When the Master Clock hardware detects an error condition it will "latch" that information. Thus even if the actual error condition is just a transit, the Master Clock hardware having "latched" this information will cause the associated Alarm to persist in the Significant Event System. If you have an Alarm from the Master Clock the first thing to do is to try to "clear the error". What I mean by this is you push a button on the Master Clock module that tells the module to re-examine its operating conditions and to stop asserting its error flag if everything is now running OK. Two of the Master Clock modules have these "Clear Errors" buttons. Note that pushing this button does not "reset" the module or change its downloaded control data. All that pushing the "Clear Errors" button does is to tell the module to look again at its running condition and determine whether or not it should still be asserting any error flags. - If there is still a problem with the operation of the module it will continue to assert the error flag. - If everything is now OK then it will clear the error flags that it had been asserting and the associated Alarm will go away in the Significant Event System. The Master Clock is located in the 9U crate in the middle of rack M100. The two modules that have "Clear Errors" buttons are: Phase Coherent Clock module 3x wide in slots 4,5,6 Sequencer #1 module in slot 7 If you need to clear Master Clock errors don't worry about which module is which. One at a time, push the "Clear Errors" button on both of these modules starting with the button on the Phase Coherent Clock module. During Beam Physics running the Master Clock should never set an Alarm in the Significant Event System. If it does and if pushing the "Clear Errors" buttons does not clear the Alarm then immediately get assistance from a clock expert. Between Stores the Master Clock will occasionally set an Alarm. This can happen when the beams division people reset the system that generates the TeV RF and TeV Sync signals. The system that generates these signals is call the "Low Level RF". When they reset this system the Master Clock PCC module notices that the TeV Sync signal is briefly missing and thus it sets error flags. Even through the TeV Sync signal immediately returns, the error flags remain asserted until someone pushes the "Clear Errors" button on the PCC module. If the actual error condition, e.g. the missing TeV Sync signal, still exists when you push the "Clear Errors" button then the error flags from the module will remain asserted. If the error does not clear when you push the button or if it repeatedly comes back then get assistance from a clock expert. A number of the alarms listed above are marked with an *. The cause of these alarms is most likely a programming error, aka a download error of the Master Clock. If you have one of these errors you can try to clear it with the "Clear Errors" button on the module. Most likely this will not clear the alarm and you will need to use the Clock GUI to correctly set the operating mode of the module that is causing the alarm. You probably should contact a clock expert because these alarms should never happen. Note that as used in Run II the BC Phase "BC PH" errors are not meaningful and do not set Alarms in the Significant Event System.