D-Zero Hall Log Book for 2008 ------------------------------- The most recent entries are near the beginning of this file. This file begins in January 2008. This file contains both Trigger Framework and L1 Calorimeter Trigger entries. Earlier D-Zero Hall Log Books are on the web in one of the following directories: http://www.pa.msu.edu/hep/d0/ftp/l1/framework/logs/ http://www.pa.msu.edu/hep/d0/ftp/run1/l1/inventory_logs/ ------------------------------------------------------------------------------ DATE: 16,17-Dec-08 At: Fermi TOPICS: check equipment in MCH-1, Calop meeting, PAB work ------------------------------------------------------------------------------ DATE: 10:12-Dec-08 At: Fermi TOPICS: check equipment in MCH-1, Calop meeting, #1 Bit-3 from PAB, Scope Pictures, Luminosity Group Brief Monitoring Blocks In the Calop meeting the changes to Mikes control code for setting the control registers in the TAB/GAB was discussed. This is in reference to running the Cal Calibration trigger on And-Or Term 190 and 191 and running the Purple Haze Noise trigger on terms 192 and 193. Mike was not there. It is not clear if there is logic behind all 4 of these terms, or logic that allows you to control just the count threshold and not the Et threshold or what. No one appears to know what is in the firmware. Mike and Gabriel tried various experiments and determined that there was at least enough logic behind them to run Cal Calib on 190 and 191, and PHN on 192 and 193. The Luminosity Group likes to receive the "Brief Luminosity Block" once every 5 seconds when they are doing HV scans. We carefully went through how to change between 5 second and 15 second Brief Luminosity Monitoring Blocks on the TFW TCC and the etiquette for using that console. We needed scope pictures of a normal EM energy deposit from the Tevitron for the Purple Haze note. On Friday I recorded normal BLS signals from +4,21 EM and put them on the web for Marvin. Brought back the #1 Bit-3 VME card from T962 at PAB for testing with DMA. ------------------------------------------------------------------------------ DATE: 1:4-Dec-08 At: Fermi TOPICS: check equipment in MCH-1, Calop meeting, spare L1 Cal Trig Bit-3, SCL_Init if once per minute rate dip is detected, TFW TCC TRICS startup In the Calop meeting Marvin presented a nice summary of the Purple Haze noise. Mike is going to try and order parts for the filter box. I should pickup the MCH-1 scopes and cables from the PHN investigation next week. I brought the spare L1 Cal Trig Bit-3 card set to MSU for testing so that we will know whether or not it works and to see if it will run DMA transfers. The MSU equipment in MCH-1 looked closed up and OK. Note --> On Nov 19th at the request of the run coordinators Philippe enabled the TFW TCC code that watches for once per minute dips in the L1-L2 rate to issue an SCL_Init when it detects that the DAQ system is in this state. The TRICS startup files have been edited so that TRICS will wake up enabled to issue an SCL_Init when it detects that the DAQ system is in the state where there is a rate dip once per minute. Note --> The TRICS startup files have NOT yet been edited to set a 15 second interval for the Brief Luminosity Scaler Blocks. For now when restarting TRICS one should use the "System Control/Status" menu and change the "Send Lum Blck Brief Every" from 5000 ms to 15000 ms and clicked on the "Set" button. ------------------------------------------------------------------------------ DATE: 19-Nov-2008 At: MSU TOPICS: L1 TCC allowed to issue SCL_Init L1TCC will now issue an SCL_Init to correct the L2 Fifo problem where the L2 Global is not issuing all pending L2 Decisions after L1 TCC pauses the framework e.g. to increment the LBN. cf. logbook entry for 22-Oct-2008 ------------------------------------------------------------------------------ DATE: 14-Nov-2008 At: Fermi TOPICS: Collaboration meetings. check MCH-1, Luminosity Scaler Brief Block Interval Walk through MCH1. Things look OK and racks are closed up with all visible fans running. On Wednesday Nov 12th at about 16:00, at the request of the Luminosity group, we changed how often we send the brief version of the luminosity scaler information from once every 5 seconds to once every 15 seconds. That is, in the "System Control/Status" menu we change the "Send Lum Blck Brief Every" from 5000 ms to 15000 ms and clicked on the "Set" button. There is no message of command file keyword implemented in TRICS to change this variable in the Boot_Auxi command file. We will thus have to manually change this variable via the GUI after we restart Trics. ------------------------------------------------------------------------------ DATE: 6,7-Nov-2008 At: Fermi TOPICS: Investigation of the Cal Purple Haze noise problem, check cables on the TAB 7 Chip 1 parity error problem, run the spare TFW power pans, Power Outage to switch Sub-Stations Run the two spare TFW power pans and check their outputs with no load. SN #2 SN #4 +5.0V 5.070 5.067 +3.3V 3.332 3.335 -2.0V 2.128 2.130 -4.5V 4.633 4.635 TAB parity errors. For the past few weeks or month there have been parity errors on the ADF-2 to TAB 7 Chip 1 link. This is the ADF-2 card in ADF Crate "C" Slot 18 and it is the "A" cable from that slot that goes to TAB 7 Chip 1. TAB 7 Chip 1 is the left most TAB (from the back of the crate) and the Chip 1 input is the next to the bottom group of 3 cables that come into this card. It is the "A" cable in this block of 3 that come from the ADF-2 card in Crate "C" Slot 18. Selcuk and I wiggled these cables and pushed their connectors full on and straight. There was some movement of the connectors at both ends. We checked the monitoring after this and things looked OK with the data that the TAB were receiving. So now we will wait and see if there are any parity errors during the next Store. Purple Haze Noise. Wednesday during the long access (made available by the site wide power outage due to the static line on the 345 KV pi poles coming loose) Dean broke out C22 into 16 feeds, i.e. 8 feeds from each end of what had all been feed by the C22 pod. Thursday we had no PHN until about 8 or 9 PM when Cal preamps had to be turned off to get ready for the power outage to switch back to the main sub-station at 4:30 Friday morning. Friday morning while experimenting with the feed to TT phi 23 we seemed to fix it by grounding one end and powering the other end. You can ground either end and power the other in which case TT phi 23 draws about 16 uAmp at 2 KV. It would make a lot of noise when one end was powered and the other end was not powered but was either grounded or loaded by the 100 Meg Ohm resistor in the supply. Having a standing current seems to be what makes it make noise. Friday afternoon we tried using the same technique on the PHN from TT phi 22. You can see the change in sign in the very first initial part of the PHN waveform from HD+ +4,22. The following is what we saw watching +4,22 HD+: Both end going up + initial noise pulse Both end going down - initial noise pulse North is up and South is held low by its pod - initial noise pulse North is help low by its pod and South is up + initial noise pulse North is up and South is grounded - initial noise pulse North is grounded and South is up + initial noise pulse North is open and South ramps up + initial noise pulse North is open and South ramps down - initial noise pulse North ramps up and South is open - initial noise pulse The noise in -4,22 HD+ is much smaller, but appears to be "in phase" with the noise in +4,22 HD+ and we think that the sign of the initial noise pulse is the same in both +4,22 and -4,22 HD+. Flowing current into the short did not appear to fix TT phi 22. When TT phi 22 was at 2 KV at one end and grounded at the other it draws about 7 uAmp. Wednesday late afternoon Philippe took care of getting the TFW and things running again after the 14:30 Chicago time site wide power outage caused by the pi pole static line problem Power up after the outage Friday morning: TFW configured 1531, Routing Master configured 51. A smooth startup but as usual it took the SBC in the Routing Master a couple of tries (pushing its reset button) before it come up. It still looks like there is something that can block the RM's SBC from starting up or that there is some not understood (by us) order issue in which the L3 stuff must come up before the RM's SBC can get itself going. The procedure is to just hang in there and push its reset button once every 5 minutes and eventually it takes off and runs OK (which you can tell because it starts to access the TFW cards in the Routing Master). ------------------------------------------------------------------------------ DATE: 23,24-Oct-2008 At: Fermi TOPICS: Investigation of the Cal Purple Haze noise problem, location of the TrgMon archive on Clued0, Fonts for pld_dmgr, L1 Cal Trig "Noise Trigger" setup Study the Purple Haze noise with a scope looking at the TT signals. 1. The PH noise is generally stronger in South (positive eta) than in the North part of CC. The PH noise is only in CC. 2. The noise waveform at a given TT is quite constant from one event to the next. 3. During typical beam running, after everything has settled down, you typically have a noise event every minute or two or else a burst of a couple noise events every minute or two. 4. Between noise events things look nice a quiet. 5. On adjacent TTs, the oscillation part of the noise waveform looks phase locked. At TTs far apart you see big phase differences but it looks like the same frequency. 6. The biggest PH noise signals are at about +4,22. 7. Some signals have just the about 2 MHz oscillation - others, typically stronger ones, start as a big narrow positive spike, then hold below zero for a few usec, then start their oscillation. 8. All channels show their "first evidence" of the noise at the same time. 9. You can see the noise in the MG signals. The eta 20 MG "EM" signals have a stronger PH noise than the eta 20 MG "HD" signals. The frequency of the oscillatory part of the PH noise in the eta 20 "HD" signals seems to be twice that of the PH noise in other parts of CC. The oscillatory part of the PH noise in eta +20 MG "EM" is in phase with its eta -20 counter part. Thursday night - Watching noise events at the normal rate of one every minute or two. The store is lost with a clean abort. Noise events stop for the next 15 minutes or so. - Noise events finally come back and we have one every 2 minutes or so. - Ramp down all Cal HV. See bursts of noise events while the HV is going down. - After the HV has been down for 10 minutes or so all noise events stop. For the next 45 minutes or so we have no noise events. - Ramp up the 1st half of Cal HV. See some burst of noise events during the ramp but then things are quiet again for the next 15 minutes or so. - Ramp up the 3rd quarter of the Cal HV and see some burst of noise events but then things are quiet. - Ramp up the 4th quarter and see much larger bursts of noise events and then things get quiet again. - Start a store and for the first 4 or 5 hours it is quiet and then the normal PH noise events return. Inventory of Purple Haze Noise scope pictures All scope triggers are from the L1 Cal Trig Term of >= 20 EM+HD trigger towers all over 7 GeV Et. The L1 Cal Trig Term #193 is in scope channel #4 and is the scope trigger. Scope channels #1, #2, #3 are the HD+ side of the indicated Trigger Tower. Note that Scope Standard Time is about 1:15 behind Chicago time. Disk #1 Scope Standard File Ch #1 Ch #2 Ch #3 Time Number Yellow Blue Pink Notes -------- ------ -------- -------- -------- -------------------- 13:54 0 +2,3 +2,14 +2,27 14:03 1 +3,23 +4,23 +3,22 14:11 2 +3,23 +4,23 +3,22 14:19 3 +3,23 +4,23 +3,22 14:41 4 +3,23 +4,23 +3,22 14:47 5 +3,23 +4,23 +3,22 typical noise event 14:50 6 +3,23 +4,23 +3,22 typical noise event 14:54 7 +3,23 +4,23 +3,22 typical noise event 15:10 8 +3,23 +4,23 +3,22 typical noise event 15:33 9 +2,23 +3,23 +4,23 typical noise event 15:44 10 +4,1 +4,2 +4,3 \ 15:49 11 +4,4 +4,5 +4,6 | 16:03 12 +4,7 +4,8 +4,9 | Tour in 16:07 13 +4,10 +4,11 +4,12 | Phi 16:14 14 +4,13 +4,14 +4,15 | 16:19 15 +4,16 +4,17 +4,18 | Note the 16:33 16 +4,19 +4,20 +4,21 | vertical 16:37 17 +4,22 +4,23 +4,24 | scales of 16:39 18 +4,25 +4,26 +4,27 | the channels 16:42 19 +4,28 +4,29 +4,30 | 16:44 20 +4,31 +4,32 +4,1 / Disk #2 Scope Standard File Ch #1 Ch #2 Ch #3 Time Number Yellow Blue Pink Notes -------- ------ -------- -------- -------- -------------------- 16:56 0 +4,1 +4,2 +4,3 16:58 1 +4,4 +4,5 +4,6 16:59 2 +4,7 +4,8 +4,9 17:15 3 +4,10 +4,11 +4,12 17:22 4 +4,13 +4,14 +4,15 17:24 5 +4,16 +4,17 +4,18 17:27 6 +4,19 +4,20 +4,21 17:32 7 +4,22 +4,23 +4,24 18:32 8 +4,25 +4,26 +4,27 8:58 9 +4,24 +4,25 +4,26 9:00 10 +4,25 +4,26 +4,27 9:03 11 +4,28 +4,29 +4,30 10:09 12 +4,28 +4,29 +4,30 11:04 13 +4,31 +4,32 +4,1 Disk #3 Scope Standard File Ch #1 Ch #2 Ch #3 Time Number Yellow Blue Pink Notes -------- ------ -------- -------- -------- -------------------- 11:39 0 +1,22 +2,22 +3,22 11:50 1 +4,22 +5,22 +6,22 11:56 2 +4,22 -3,22 -4,22 11:59 3 +4,22 -4,22 -4,23 12:03 4 +4,22 +4,6 +4,7 12:23 5 +20,22em +20,22hd +20,23em 14:18 6 +20,22em +20,22hd -20,22em Friday Afternoon - 16:30 Ramp down just the 4th quarter of Cal HV. See the expected bursts of noise events. - After 5 minutes the rate of noise events is back down to one every minute or so and the stuffing is really gone from the wave form on the scope watching the normal suspects, i.e. +4,21 +4,22 +4,23 HD+ - Zero noise events for the past 42 minutes. - At 17:45 ramp back up every one but the most suspicious HV channel in the 4th quarter, i.e. everyone but "Central 24" . During the ramp for 15 or 30 seconds get lots of scope triggers with big full noise events. - Then it is quiet for 10 minutes. - At 17:58 ramp up "Central 24" and see absolutely nothing. Zero noise events. - Leave the scope tied up to: +4,21 +4,22 +4,23 HD+ an triggering from AO Term 193. - Still no noise triggers at about 20:00 when the next Store began. L1 Cal Noise Trigger The L1 Cal "Noise Trigger", i.e. >= 20 EM+HD towers all over 7 GeV Et appears to find a pure sample of Purple Haze noise. It does not trigger on SMT noise. The only other thing that it appears to trigger on is when they inject p or pbars into the Tevatron. I don't know if it triggers on all of the PH noise but it gives you a pure sample of PH noise. From Mike Mulhearn, to setup the L1 Cal part of this: On d0tcc3 as d0cal: cd ~d0cal/l1cal2b/tablib/run ./bin/setup_noise --help ./bin/setup_noise --enable --verbose 20 44 d0tcc3|l1cal2b/tablib/run>./bin/setup_noise --help INFO: parsing file: param/vme.txt usage: setup_noise [OPTIONS] setup the TAB/GAB modules for a noise trigger which requires NUM_THRESH towers above ADC_THRESH. Noise algorithm sums EM+HAD ET, so there are two pedestals of value 8. Roughly 4 counts per GeV. This means: > PED ADC_THRESH 8 GeV 48 4 GeV 32 3 GeV 28 2 GeV 24 Arguments are assumed to be decimal unless preceeded by 0x in which case they are taken as hex. On clued0 the archive of the once per minute TrgMon is at: /work/poidog-clued0/mverzocc/TM/ Work with Mentor pld_dmgr using d0sunmsu1 as the display. This still works just fine once all of the fonts are loaded including: Mentor_1999_Fonts which I assume the Xilinx stuff wants and Fonts_bst2005 which seem to be good enough to run the current version of Mentor bst. ------------------------------------------------------------------------------ DATE: 22-Oct-2008 At: MSU TOPICS: Start TRICS version V11.3.A Trics V11.3.A was started remotely from MSU on Wednesday morning 22-Oct-2008 at about 11:15 Chicago time. New functionality in Trics V11.3.A for detecting L2 Global L2 Hardware Framework synchronization problems. L1 TCC has the opportunity to notice that the L2Global is behind in sending L2 answers to the L2FW when the Luminosity Manager process tries to make a new LBN boundary. Incrementing the LBN happens at least once per minute, and also on special transitions (pause/resume, start/end run, etc). The Luminosity Manager process is an independent process, separate from the main process (tied to the GUI), the COOR message executer process, and from the monitoring server process. Incrementing the LBN starts with pausing the L1FW and waiting for up to 1 second for the L2FW to become quiescent, i.e. until the L2FW thinks it has received and handled the L2 answers for all outstanding events. The reason for waiting for the L2FW to be quiescent is to capture a self-consistent set of scaler values to be pushed over the network to the Luminosity Server, so that this L1 and L2 accounting correspond to the same set of beam crossings. The Luminosity Manager takes these steps (summarized a bit and omitting some details) 1) acquire the synchronization lock on the capture monitoring data: we will need to capture monitoring data, and we want to prevent the monitoring server from initiate new monitoring data capture until we have captured and read the set we need during the LBN increment sequence. 2) acquire the synchronization lock on the L1FW pause/resume state: the Luminosity manager needs to pause the framework if it was not already paused, and resume it if it had paused it itself. We thus cannot let COOR or anything else change the pause/resume state while the Luminosity Manager is taking charge, or we would risk restoring a wrong pause/resume state. 3) Pause the L1FW (if it wasn't already) 4) Poll the L2FW for up to 1 second or until it shows that it has reached its Quiescent state. 5) do the required Register IO to increment the LBN in the L1FW hardware registers that will be readout with each event and become the event LBN number. 6) Force the immediate capture of monitoring information 7) resume the L1FW (if it had to be paused) 8) release the pause/resume lock note: Assuming the L2FW is already quiescent, the minimum time that the Luminosity Manager has to pause the L1FW is for 4 or 5 IOs, or less than 20 us. This could take a few hundred microseconds if the L2 Global was working on a few events at the time we paused the L1FW. This can take up to 1 second if the L2 FW thinks it is still holding one (or more) L1 Accepts waiting for L2 Answers. 9) advertize the new LBN number to the screen 10) acquire the synchronization lock on the monitoring pool data 11) read the L1FW and L2FW hardware data and update the monitoring pool data. 12) release the capture monitoring data lock note: now the monitoring pool could arm the capture monit data circuitry, and thus update the monitoring registers in the hardware, but this still does not allow the software monitoring pool data to be changed. 13) send/push the block of luminosity info to the luminosity server. 14) release the monitoring data lock The plan was to add logic to detect and try to correct any persistent loss of fifo synchronization between the L2 Global and the L2 FW. The simple plan of adding all this to step 4 above is not optimal. The current cure (we may think of something else some day) is to send an SCL_init, and it would not be very clean (nor very wise) to do this early in the LBN increment procedure, especially since an SCL_init also implies another LBN increment. The next simple plan of using step 4 for just noticing there was a problem and completely deal with it in a new step 15 would also not be optimal. We said we would like to read some registers to investigate this problem, and step 4 is the best time to do that, while the L1 FW is paused. There are 3 more or less separate things we want to do when we detect a problem: a) collect information about it b) determine if some action should be taken c) take action when it is called for (a) should thus happen during step 4, (c) should be during a new step 15, and (b) could equally well happen during step 4 or step 15. What was implemented was to do just (a) in step 4 and (b&c) in step 15. updated step 4: - when a problem is detected with the L2 Fifo remember that we will need to act during step 15 - remember the L1 rate last measured by the monitoring server (since this can change before step 15) - collect information on the symptoms - we already have the value of the status word of the L2 Helper, and its IO state printed due to the error itself - print to the screen the depth of the fifos of the FPGA #1 of each of the 5 TRM cards in the L2FW. This is tricky since we only have the pseudo gray code value of the read and write pointer, but the monitoring server has already read and calculated those values. - execute a command file where we could read or do additional things without having to change the version of Trics. FixL2Fifo_Investigation.mcf currently does nothing. new step 15: - if no problem was detected in step 4 any previous problem is considered cleared, and all low pass filters are thus reset. - if a problem was detected in step 4 examine the L1 rate saved in step 4 - if lower than threshold, there is no problem, and any former problem is considered cleared, and all low pass filters are thus reset. - if higher than L1 rate threshold (default 10 Hz) then we are experiencing a problem - if this is a new problem, remember the starting time - examine the duration of the problem - if less than the minimum duration threshold, do nothing else this time - if more than minimum duration (currently 2.5 minutes, i.e. we typically act only after the 3rd consecutive LBN increment with an active problem) - examine the time since we last considered it was time to take action - if less than the minimum cool-down threshold, do nothing else this time - if more than the cool-down period (default 10 mn) then it is time to take action - check if we are allowed to take action - if not ... do nothing (which is the default) but for every other purpose, consider an action taken, and problem fixed so that all time low pass filters will start over - if we are allowed to take action - acquire pause/resume lock and pause the L1FW if needed so that if we need to read or write some register, we are looking at a paused system. - execute command file FixL2Fifo_Correction.mcf which currently only sends an SCL_init message, but could do something additional, or even something else instead. note: because we are paused, the COOR message handler will not execute that SCL_init until we are cleanly 100% done here, which is another motivation for pausing the system. - resume the L1FW if paused earlier, and release pause/resume lock - consider problem fixed so that all low pass filters are reset. Note: in step 4 the information gathering and the command file is executed every time a problem is sensed (e.g. even if the L1 rate is 0 Hz). To change any of the parameters use the "Send COOR Message" menu and manually send messages. All these message are NOT case sensitive. - to change the minimum L1 rate to a new value of N Hz (N has to be an integer, eg. "100" for 100.0 Hz): "TrgMgr_FixL2Fifo_MinRate_Hz N" - to change the minimum time to wait before intervening to be S seconds (S has to be an integer): "TrgMgr_FixL2Fifo_MinTime_s S" - to change the minimum time between successive interventions to be S seconds (S has to be an integer): TrgMgr_FixL2Fifo_CoolDown_s S - to enable or disable executing the FixL2Fifo_Correction.mcf command file when Trics has decided that an action should be taken: "TrgMgr_FixL2Fifo_Enabled" "TrgMgr_FixL2Fifo_WatchOnly" ------------------------------------------------------------------------------ DATE: 8:10-Oct-2008 At: Fermi TOPICS: Remove Logic Analyzer, Wiener Power Supply Setup, L1 Cal Trig Communication and Control Crate, Single TT Noise Runs, Cut BLS Resistors, TT BLS Signal Work, PAB phone number, End of the one week Shutdown - recover from power outage Remove the Tek logic analyzer and the power supply for the ECL-->TTL board from the floor by rack M124. Leave all the inputs to the "bridge ECL buffers" connected and all of the cabling to the ECL-->TTL board connected. I.E. everything is connected exccept for plugging the logic analyzer back into it. The ECL-->TTL board is labeled where the various logic analyzer pods were plugged into it. See the log book entry for 29,30-NOV-2007 for the setup. For this trip I will leave MSU's Tek logic analyzer at D-Zero. Wiener 6021 Power Supply Control -------------------------------- Recall how to control the Wiener 6021 power supply via fan tray control panel. The general instructions from Wiener are: - Switch the POWER and the MODE switch up simultaneous for 5 seconds. The display shows, "Config: Wait....', and then, "Config: Ready !". Then release both switches. - If a sub-menu exists, you may now select the sub-menu item (MODE switch up/down). If no sub-menu exists, you may change the parameter value (MODE switch up/down) - To change a parameter of a sub-menu, select it (POWER switch up). The selected parameter is flashing now. - You may alter the parameter now (MODE switch up/down) - After finishing the parameter programming, leave the submenu or configuration menu (POWER switch down). The big point that these instructions fail to mention is that you should use the Mode switch to first select the power supply that you want to control before pushing up both the Power & Mode switches for 5 seconds to move into the "change the eprom settings mode". So if you are going to adjust things in a crate for "N" of its power supply voltages then you you need to enter and exit the "change the eprom settings mode" "N" times. The setup of the 4 ADF-2 crates is given in the 1:3-FEB-2006 entry in the log book. This also includes an operational definition of each of the Wiener parameter names that you can adjust. I read back the setup of the Run IIB L1 Cal Trig Control Communications Crate to document what it is: Control- Communications Crate Setting +5.0 V +3.3V -------------- ------- ------ Ilim 30 A 50 A Uadj 0 % 0 % Unom 5.00 V 3.30 V OVP 6.25 V 4.50 V Imax 25 A 45 A Umin 4.50 V 3.00 V Umax 5.50 V 3.50 V Fans 3300 rpm CAN-Bus Address 6 The only thing that seems strange is the heavy Ilim and high OVP on the 3.3V supply. I'm not certain who set this up originally. The: Bit-3, Vertical Interconnects, and SCLD use only the +5V. I assume that the control card for the TAB/GAB uses both +5V and +3.3V. Running the Single Trigger Tower "Noise Trigger" ------------------------------------------------ Exclude all TTs except for the one that you want to trigger on: --------------------------------------------------------------- 1) On d0tcc3 you will find Test_Exclude_Most_Trigger_Towers.msg in /tcc/L1Cal_IIb_Work/Config/. 2) You can edit this file to your needs and aim at the tower(s) of your liking. 3) From the GUI main menu select "TCC Com File" and enter the name of the file in the text box: /tcc/L1Cal_IIb_Work/Config/Test_Exclude_Most_Trigger_Towers.msg or click your way to it using "Locate TCC ComFile" 4) click on "Self Msg ComFile" to tell the TCS engine to ingest and execute this file. 5) If you need to switch to a different tower(s), repeat steps 1-3 6) When you are all done, *do not forget* to re-initialize the L1Cal system Setup the TAB & GAB for "Noise Trigger": ---------------------------------------- 1. On d0tcc3 cd ~mulhearn/tablib/run ( ./standard directory contains the functions that are not in the GUI). 2. Run the program: ./standard/setup_noise_trigger.pl --help (This will list the usage of this program) 3. Run the program: ./standard/setup_noise_trigger.pl NUM_TOWER: Number of towers in this trigger ADC_COUNT: 16+4*Threshold (per twoer in GeV) (16 = 8+8 pedestals for EM+HAD and 4 counts/GeV) 4. The output of this trigger is on And-Or Term 192. 5. To go back to regular running, initialize the system with the L1Cal_IIb_GUI. See the Running_L1Cal_IIb_GUI procedure for how to run this program. TAKER Trigger Configuration File: --------------------------------- Use the Trigger Configuration File: /commissioning/l1cal/l1cal2b/trial/l1cal2b_noise_wcal_wsmt_1.0.xml Review of Cut Resistors in Run II --------------------------------- I believe that the cut BLS resistors for Run II are the following: -6,22 EM -6,28 EM +6,23 EM +8,28 EM +5,23 EM +1,24 EM -4,7 HD For information about cut BLS resistors see the log book for: 12:15-AUG-2003 and 28:31-JULY-2003. 4:8-MAR-2003 log book entry contain a history of old cut BLS resistors. Work on Trigger Towers this trip -------------------------------- The recent problems with noise from +5,23 EM is that one of the trigger summer hybirds in this BLS is supposed to have a cut resistor but the BLS card that is supposed to be in +5,23 had been carried off to +5,15. Dean put the proper labeled BLS card back in +5,23 so not we should not see any more problems for +5,23 EM. +6,23 EM is currently Excluded. +6,23 has two problems: It is supposed to have a trigger summer hybrid with a cut resistor but the BLS card with this special trigger summer hybird for +6,23 EM has been carried off to some other location. Dean looked to try to find where this BLS card has been carried off to but he could not find it. On Friday Dean cut the appropriate resistor on the BLS card that is now servicing +6,23. It looks OK on the scope. In addition the whole BLS crate that services: eta +1:+6 phi 25 and 26 as well as eta +3:+6 phi 27 has some line locked noise on it. Try making a number of "single tower noise runs". The new "exclude all TTs except for bla" control file would just fine. We had the normal problem that the noisy TTs were not making much noise today. We could get a couple of triggers every 5 minutes out of +7,9 HD. PAB (Proton Assembly Building) phone number: 630-840-3330 or x3328 ------------------------------------------------------------------------------ DATE: 07-Oct-2008 At: MSU TOPICS: forensic on L1 TCC restart after power outage. L1 TCC was powered down on 06-Oct-2008 06:21. TRICS had been running since 13-Dec-2007 09:54, when we had switched from V11.2.A to V11.2.C for automatic M122 crate resets. That's one week short of 10 months of continuous TRICS running. After TCC power up, Trics Auto-Start started at 06-Oct-2008 12:37:41 and was used to configure the L1FW FPGAs, and RM FPGAs, and a data was flowing by 06-Oct-2008 13:44:11 But then somebody later started VME Access at 06-Oct-2008 16:23:17 closed it 5 seconds later then started VME access a second time at 06-Oct-2008 16:23:35 and closed it 8 second later then started a 2nd copy of trics via the V11.2 shortcut at06-Oct-2008 16:26:42 and wrote the value 60 three times at Master#1/Slave#2/Slot#21/Chip#16/Reg# 41 That's the tick and turn scaler card and this sets up AOIT 252 (Tick Select #1) That's quite weird, because I expect that person would have had to move the windows from the first copy of trics to be able to even reach the TRICS shortcut... unless the DAQ expert had shrunk the first trics windows after successfully starting up the system. I unfortunately did not think of that at the time, and brought up the windows from instance number one by clicking on the taskbar instead of shrinking the windows from instance number two... so we will never know how this confusion started. I don't think there was any ill consequence: COOR had already connected to the first instance, L1FW and RM FPGAs had already been configured, and a ZB run was already going. This second copy of TRICS was never initialized, thought the L1FW was non-operational, and thus never started to do any IO on its own (e.g. updating the LBN) and was not used for any further operator request (after the above Tick Select Register IOs). It was complaining about not having a connection from the Lum Server, and was writing LBN blocks (but all zeroes) to a result file. Monitoring clients obviously connected to the first instance, since the rollplot looked normal. I kept a copy at MSU and deleted all these confusing logfiles from L1 TCC. ------------------------------------------------------------------------------ DATE: 30-Sept:2-Oct-2008 At: Fermi TOPICS: MCH-1 check, Power Outage Our equipment in MCH-1 appears to be operating normally. To prepare for next week's power outages I turned off the MSU logic analyzer this week. I has not triggered in months. I'm going to remove it during this power outage. I will send a note to the Run Coordinators about taking care of the TFW and such on Monday. I will verify that Selcuk will take care of the L1 Cal on Monday. I will take care of both of these on Friday next week. ------------------------------------------------------------------------------ DATE: 18,19-Sept-2008 At: Fermi TOPICS: Check on the equipment in MCH-1, check some TT's These are the 7 TTs that are currently Excluded, plus +5,23 EM which we are keeping an eye on. Checked Thursday evening during a Store. -3,25 HD good normal looking signal - no noise -4,8 HD good normal looking signal - no noise +5,23 EM good normal looking signal plus negative energy pulses on both EM+ and EM- about 50 to 70 mV base to peak about 14 usec period and about 2 usec width +6,23 EM good normal looking signal - no noise +6,25 EM Can see the Calorimeter energy signals plus 150 mVpp almost sine wave oscillation on both EM+ and EM- with a 4.4 usec period this oscillation runs continuously for some minutes and then it stopped for min ? hr ? days ? +7,9 HD good normal looking signal plus a high frequency buzz on both HD+ and HD- about 10 mVpp with about 110 nsec period - not 132 nsec -9,22 HD good normal looking signal - no noise -9,25 HD good normal looking signal plus a 132 nsec buzz about 10 to 15 mVpp on both HD+ and HD- These 8 TTs were checked with the scope again Friday evening again during a Store. / Both of these have a good normal looking signal but -3,25 HD \ / many (most) TTs in this area have a small burst of buzz |--| with an envelope about 250 nsec long and an amplitude -4,8 HD / \ of about 25 mVpp with the buzz period of about 25 nsec. \ Is this readout related ? +5,23 EM Good normal looking signal plus negative energy pulses on both EM+ and EM- about 50 to 70 mV base to peak about 14 usec period and about 2 usec width. +6,23 EM Good normal signal with no extra noise. +6,25 EM AC line locked noise bump of about 20 to 25 mV amplitude that lasts for about 1 msec. +7,9 HD Negative energy buzz on both HD+ and HD- with an envelope that lasts for about 2 usec. The buzz has an amplitude of about 100 to 150 mVpp and a period of about 103 nsec. Is this readout related ? -9,22 HD Negitive energy buzz on both HD+ and HD-. The envelope of the buzz lasts about 10 to 20 usec with an amplitude of about 25 mVpp and a frequency of about 96 nsec. Is this readout related ? -9,25 HD normal signal plus a 132 nsec buzz with an amplitude of about 10 or 15 mVpp. I need to check these TTs again between Stores when I can pause the ZB run to determine which noise patterns are caused by readout. The previous scope check of these TT was done on 4:6-Sept-2008. ------------------------------------------------------------------------------ DATE: 4:6-Sept-2008 At: Fermi TOPICS: Check equipment in MCH-1, Check the currently Excluded TT's, L1 and L2 rate dips, Work at PAB The equipment in MCH-1 looks closed up and the visible fans are running OK. The L1 L2 rate dips were discussed in the Friday afternoon operations meeting. The conclusion of this discussion was to ask Philippe to add code to the TFW TCC to monitor for this rate dip situation (i.e. that the L2 Hardware FW thinks that L2 Global is not caught up after a 1 second wait). When it finds that we are in this situation (i.e. we are in a global beam Physics run and we have had "N" consecutive one minute periods with a rate dip then it should take the following action: - make a log file entry to indicate that TCC has found this situation, - readout some additional register locations that may help to diagnose what is going on and write this information in the log file, - optionally (if enabled to) TCC should issue an SCL Initialize. After issuing an SCL Initialize then, if this rate dip problem persists, some mechanism is needed to block TCC from trying to issue additional SCL Initialize signals to fix this problem for the next "M" one minute periods. The other thing that we should do is find an example of where L2 is known to be behind (e.g. stuck for a Muon L2 preprocessor input or something) and see what the system looks like in that condition i.e. see what the registers in the L2 Hardware FW look like in that condition. One theory was that L2 HW FW was getting an extra spurious entry in its input FIFO or in its L1_Awaiting_L2 counter. But the problem may be that either L2 Global or L2 HW FW have not responded to a kick to get there processing started for some event (but everything in their FIFOs is just fine). In that case the TFW needs the N+1 L1_Accept before it can issue the Nth L2_Decision. That is what Dean sees once in a while in his runs. Check the currently excluded TT's and the currently suspicious TT with the scope. This is to get ready for the mini shutdown in October. This check was made during a Store to see real signals. -3,25 HD OK - good signals - no obvious noise -4,8 HD OK - good signals - no obvious noise +5,23 EM signal is OK but it has negative pulses, i.e. negative energy pulse on both EM+ and EM- signals, amplitude is about 50 mV, width about 2 usec, period about 15 usec. +6,23 EM OK - good signals - no obvious noise +6,25 EM OK - good signals - no obvious noise +7,9 HD OK signal but it has 25 mV of buzz on it 200 nsec period -9,22 HD OK - good signals - no obvious noise -9,25 HD OK signal but it has 25 mV of buzz on it 132 nsec period See the log book entry from 24,25-APR-2008 for the previous scope check of the Excluded Trigger Towers. Work at PAB. All of the T962 channels are now installed. Bring back the Bo DAQ computer for library work. ------------------------------------------------------------------------------ DATE: 19:21-Aug-2008 At: Fermi TOPICS: Check on the equipment in MCH-1, close TFW TCC log file, work at PAB Closed the TFW TCC log file TRICS_II_20080808_V11_2_C.LOG;1 and opened the log file TRICS_II_200080819_V11_2_C.LOG;1. I copied the closed file to Faraday and then to Moto. All of our equipment looks closed up and running OK in MCH-1. I left the scope plugged into +2,29 EM just in case there is another BLS crate problem. I went to, but it appears there was no, CalOp meeting this week so I have not heard any more about the BLS crate 8 problem (but it has not been in the log book) nor have I heard any more about L1Cal work to get ready for the October shutdown. Installed a new wire order configuration file at PAB. ------------------------------------------------------------------------------ DATE: 5:8-AUG-2008 At: Fermi TOPICS: Check MCH-1 equipment, setup scope to monitor a TT, Close log file, Work at PAB All of our MCH-1 equipment looked closed up and running OK. All visible fans were running. The logic analyzer had not triggered. Closed the TFW TCC log file TRICS_II_20080731_V11_2_C.LOG;1 and opened the log file TRICS_II_200080808_V11_2_C.LOG;1. I copied the closed file to Faraday and then to Moto. Cal precision readout has had some trouble with a BLS crate. If this problem comes back it might be useful to look at a TT from this BLS crate on the scope in MCH-1. So after talking with Dan and Dean about which TT to plug the scope into I set it up to look at +2,29 EM. At PAB both Bo and T962 are full and there are tracks on Bo. ------------------------------------------------------------------------------ DATE: 31-JULY,1-Aug-2008 At: Fermi TOPICS: Check on the equipment in MCH-1, close TFW TCC log file, 1553 Box change, work at PAB All MCH-1 equipment looked closed up and all visible fans were running. Closed the TFW TCC log file TRICS_II_20080718_V11_2_C.LOG;1 and opened the log file TRICS_II_200080731_V11_2_C.LOG;1. I copied the closed file to Faraday. Thursday night starting at about 1:24 there were a number of run pausing alarms from L1CAL_LV_M101AB/-2VV. It was over its LowLow limit. By about 3:15 they disabled this alarm. They paged the L1CAL on call person a number of times but received no answer. This Run II TFW type power supply chassis supplies power to the Top and Middle crates in rack M101. The Top crate in M101 is the VIPA readout crate from the L1CAL. The Middle crate in M101 is a Run II TFW type crate, which used to hold the Spark cards for the Run IIA L1 Cal Trig and now holds only a Tom and a Ray card. I measured the actual Vtt Voltages: at power supply chassis test points -2.111 Volts, on the middle backplane -2.102 Volts, at the terminal block that feeds monitor Voltages to the Shea 1553 Box -2.102 Volts. The monitor system was saying -2.35 Volts. -2.400 was the LowLow trip threshold. In-fact all of the voltages monitored by this 1553 Box looked 200 to 300 mV low. So between stores I replaced this 1553 Box and now the monitored Voltages look OK. Mike Matulik got me the new 1553 Box and took the old one. Prep now takes care of them. The other issue on this subject is that the Vtt and Vee supplies in this chassis are no longer actually being used. The Top crate, the VIPA L1Cal readout, only uses +5 and +3.3 Volts. So change the alarms for the -2V and the -4.5V to NOT run pausing (but still keep these alarms on the -2 and -4.5 so that we wake up a human to the fact that something is going wrong in this power supply chassis). I also edited (i.e. made) a guidance file for the L1CAL_LV_M101AB alarms. This file is: /mnt/online/config/ses/guidance/l1c/L1CAL_LV.txt On the Control Room machines you can use Ctrl Right_Click to get a menu to make the font big enough for a normal human to read. Vladimir Sirotenko was most helpful in changing the alarm level and showing me what file to edit. I picked up the new pager that Bill had requested. Worked at PAB on the coherent noise level. Tested new pages from MSU and it works. Still the same pager number 517-232-1037. ------------------------------------------------------------------------------ DATE: 17:18-JULY-2008 At: Fermi TOPICS: Check on the equipment in MCH-1, close TFW TCC log file, work at PAB Our equipment in MCH-1 appears to be running OK and the racks closed up. I check all fans including the one in the back of M124. The upper M124 temperature is running 91.1 to 91.3 deg F. I should bring a new battery for that thermometer. Close the TFW TCC log file: TRICS_II_20080710_V11_2_C.LOG;1 and copy it to MSU in ascii. This starts the log file: TRICS_II_20080718_V11_2_C.LOG;1 I wrote a file from the logic analyzer with the date 18-July-2008 but I have no idea when the logic analyzer triggered. In this event there is almost a continuous string of junk on pod D0 changing from hex: 30 to 2a or 50 or 82 or 1c or 18 or 13 or 11. In the vicinity of the logic analyzer trigger it did something like: $10 for 60 nsec then $18 for 20 nsec (that sequence a bunch of times) then $10 for 40 nsec, $11 for 20 nsec, $1c for 20 nsec, $11 for 60 nsec, $18 for 20 nsec, @11 for 60 nsec, $1c for 20nsec, $10 for 60 nsec Recall that: D0 0 Mismatch Flag signal from L2 Global D0 4 L2_Rej for G.S. 0x20 D0 5 L2_Acpt for G.S. 0x20 Clear the TFW can not legally send out this kind of junk - but the logic analyzer also shows the mismatch signal from L2 Global being asserted multiple times for just 20 nsec and then negated. I'm pretty certain that it can not do that. Thus I do not know what the logic analyzer was recording. Bring DAQ-480 to Fermi and load it. Move the roll off caps on PFC-16 SN# 11 to 18 pfd. ------------------------------------------------------------------------------ DATE: 8:10-JULY-2008 At: Fermi TOPICS: Check on the equipment in MCH-1, work at PAB Our equipment in MCH-1 appears to be closed up and running normally. Checked fans except for those only accessible from the back. Leave the logic analyzer connected to the L2_Answer signals. In the CalOp meeting Selcuk reported that there has been only one TAB parity error since the cable swap and that was from a different channel on TAB 7 than the channel that had been showing about one error a day. Work at PAB testing 962 channels and Bo safety review. ------------------------------------------------------------------------------ DATE: 19,20-JUNE-2008 At: Fermi TOPICS: Trigger rate dips, Check on the equipment in MCH-1, L2Cal HW Scalers Gray Between about 19:40 and about 20:40 on Friday June 20th Chicago time there was a once per minute dip in the L1 and L2 trigger rate. I was not in the Control Room until right after the problem was "fixed" but reasonable people like Dean saw this once per minute dip in trigger rates. Normally (i.e. for 11 of the 12 TrgMon sweeps in a minute) the L1 rate was 650 Hz. For one TrgMon sweep per minute the L1 rate was 500 Hz. This behavior persisted for about 1 hour. Basically everything else was running smoothly. This behavior finally ended when an SCL_Init was needed for another reason (Muon). Philippe is checking the TFW TCC log file to look for this once per minute dip in L1 rate and to verify that it was not associated with the once per minute incrementing of the Luminosity Block Number. What else happens once per minute ? MCH-1 racks look closed up and the fans are running OK. The MSU logic analyzer is still in the center aisle. I did check the fans in the back of M124. Since some problem with the L2Cal system Thursday night (which required people poking at it to fix it) the L2 Cal pie chart in the Control Room has been gray. Being gray means that the display program saw no increment in scaler counts from one sweep to the next. This was traced to the flat cable not being plugged in that brings the "state scaler increment enable signals" out of the front panel of the L2Cal CPU module. ------------------------------------------------------------------------------ DATE: 30-MAY-2008 At: Fermi TOPICS: Check on the equipment in MCH-1 MCH-1 racks look closed up and the fans are running OK. The MSU logic analyzer is still in the center aisle. ------------------------------------------------------------------------------ DATE: 13,16-MAY-2008 At: Fermi TOPICS: Deliver a computer, Signal for monitoring the Heart Beat trigger, Swap two L1 Cal Trig LVDS cables, check TFW fans and doors Deliver a computer to Jim Kraus. Checked the TFW fans and doors. All fans are running OK - but I did not move things to check the two fans in the back transition card area of the SCL Hub-End. I need to do that on the next trip. The rear door to M123 had been opened and its latch was only half way closed. Get an L1_Acpt signal up to MCH-3 for Marvin. This is cable: EFS241-TB-21 /cfeBOT-00-?? EFS241-SCTB Trigger Out -J21. Marvin is checking into the problem that sometimes the SMT heart beat trigger signal is not firing when it should. Thursday afternoon May 15th at about 13:00 we swapped two ADF-2 to TAB cables in the L1 Cal Trig to start investigating the issue of the number of link errors on TAB #7 Chip #7. The link in question starts at M106 i.e. Crate "B" Slot 14 eta +9:+12 phi 1:4 ADF-2 card species "C" SN# C6 top link output i.e. LVDS Cable "A" The cable in the "A" port of ADF Crate "B" Slot 14 is Cable Number 71 Blue. The cable in the "B" port of ADF Crate "B" Slot 14 is Cable Number 69 Blue. First pull cable 71 from the "A" port and see error: TAB 7 Chip 7 status 8120 Now also pull out cable 69 from the "B" port as see errors: TAB 7 Chip 7 status 8120, and TAB 0 Chip 7 status 8080 Now plug cable 71 into the "B" port and see only the error: TAB 0 Chip 7 status 8080 Now plug cable 69 into the "A" port and see no more TAB input errors on the L1 Cal Trig TCC console screen. Watch for 2 minutes and no errors. After 24 hours Selcuk said that the TCC logs still showed no TAB input errors. But on the Sunday May 18th Selcuk reports that the errors are back on the link to TAB 7 Chip 7. On Monday May 19th he reports that Trigger Towers -5,29 and -6,29 EM & HD are dead and have been dead since the first run after the LVDS cable swap. -5,29 and -6,29 are serviced by the ADF-2 card in slot #12 of ADF Crate "B". Selcuk pushed on the PFC cable connectors and these two Trigger Towers came back to life. ------------------------------------------------------------------------------ DATE: 09-May-2008 At: MSU TOPICS: non-fatal SCL status concentrator initialization errors During routine check of TCC's logfile, these errors were discovered: %%08-May-2008 02:07 I$Initializing SCL Hub End Crate I$ Initializing the Trigger Hub Controller I$ SCL Trigger Hub Controller Status is 0x0860 I$ Initializing All Trigger Status Concentrator Channels E$ Failure Setting Status Concentrator for Geo Sect #53 Chan Int Req to 0x0000 I$ SCL Trigger Status Concentrator for Geo Sect #53 Chan Int Req is 0x0002 E$ Failure Setting Status Concentrator for Geo Sect #56 Chan Int Req to 0x0000 I$ SCL Trigger Status Concentrator for Geo Sect #56 Chan Int Req is 0x0002 E$ Failure Setting Status Concentrator for Geo Sect #57 Chan Int Req to 0x0000 I$ SCL Trigger Status Concentrator for Geo Sect #57 Chan Int Req is 0x0002 E$ Failure Setting Status Concentrator for Geo Sect #58 Chan Int Req to 0x0000 I$ SCL Trigger Status Concentrator for Geo Sect #58 Chan Int Req is 0x0002 Trics is trying to say that it wanted to write 0x0000 but read back 0x0002 for just these channels (i.e. each second I$-line should also be considered part of the preceding E$-line error) This was not considered a fatal error, and Initialization was reported to COOR as successful. Evidence in the code remind me that, at one time, 3 of the TSC cards did not DTACK, and Trics just walked around them, i.e. totally ignored setting all registers for them. This must have been cured, and no TSC card is currently ignored. Trics actions try to totally disable all interrupts - disable interrupts at the board level - disable interrupts for each channel - clear any previous interrupt for each channel There are 8 GeoSect per TSC, so the above messages probably correspond to read/writes from 2 separate TSC cards. These were 4 separate registers, but I did not try to reconstruct the exact VME address of these registers at this time. ------------------------------------------------------------------------------ DATE: 24,25-APR-2008 At: Fermi TOPICS: Look at Trigger Towers With the scope look at all of the TT's which at some point during RunIIB have had to be excluded. I looked at all of these TT's at least 2 times and most of them 3 times. Current TT Status Scope view of the BLS Signal -------- -------- -------------------------------- +5,9 HD included no noise, good waveform -5,11 HD included no noise, good waveform +7,23 HD included lots of 132 nsec sync noise, about 50 mV per side +8,22 HD included some 132 nsec sync noise, waveform is OK -8,9 HD included very big 132 nsec sync noise, about 75 mV per side +9,31 HD included small 132 nsec sync noise, good waveform, looks OK -3,25 HD Excluded no noise, good waveform -3,31 HD included no noise, good waveform -9,22 HD Excluded looks OK +2,25 HD included no noise, good waveform +7,9 HD Excluded wrong sign pulses, and lots of noise +7,9 EM has some noise too +8,9 EM&HD have some noise too -4,8 HD Excluded no noise, good waveform +6,25 EM Excluded no noise, good waveform Notes: We have tried running again with -9,22 HD. It will run OK for a week or so and then start causing trouble again in bursts of 1 minute to 30 minutes. As of May 19th it is again Excluded. It looks like the only trouble with some of the TT's, that were excluded early in RunIIB, but have run OK since the end of the shutdown in Nov 2007, was that they had a lot of 132 nsec sync noise that must not be stable. I assume that we are running OK with them now because of Philippe's active pedestal control. While looking at the above TT's I also noticed that +5,23 EM is very strange. +5,23 EM is not excluded. It has negative energy pulses about 1 usec wide that happen about once every 12 usec. On Friday monitored +11,24 HD ------------------------------------------------------------------------------ DATE: 27,28-MAR-2008 At: Fermi TOPICS: Look at some Trigger Towers, Master Clock glitches Thursday morning at about 8:30 the Master Clock set a PCC "Sync Timing" error and the Sequencer set a "Sequencer Hold" error. These cleared as soon as I pushed the "Clear Errors" buttons on both modules. We were able to get running right away again but people were still finding things wrong for the next hour or so. Many messages with Run Coordinators and with Steve. Friday afternoon at about 15:45 the AD decided to reboot the Low Level RF. They got LLRF running again at about 16:00. As expected while LLRF was off the PCC showed Sync Timing and Sync Missing errors. When LLRF came back it set a Seq Hold error as expected. Send a note to the run coordinators about what the Master Clock LEDs will look like if it glitches again and what buttons to push to get going again. Dean and I looked some more at -9,22 HD. See also the entry from 14,15 FEB 2008. It still shows bursts into oscillation - a nice sin wave with typically about 500 nsec period (but it varies). It also has other unusual glitches in the 2 to 4 usec range, e.g. spike pairs, spike with exponential decay. I looked at the "synchronous noise" from Philippe's list. The fun way to see this is to sync the scope with a once per turn marker and then it is easy to see the synchronous noise component (and not get mixed up with SMT readout noise and other types of noise). The amplitude of synchronous noise on the scope matches Philippe's list: -9,25 HD (rms 6.8) -9,24 HD (6.2) -9,21 HD (4.5) and -9,22 HD (2.7). ------------------------------------------------------------------------------ DATE: 14:15-FEB-2008 At: Fermi TOPICS: Work on +8,6 HD, Study -9,22 HD and +6,25 EM, Spare L1 Cal Trig Bit-3 On arrival I again found our logic analyzer in the middle of the aisle instead of tucked in up against the air conditioner as much out of the way as possible. Trigger Tower +8,6 HD has had an open circuit between the Patch Panel and the 1 Meg Ohm resistor on the ADF-2 card for some time. See for example the log book entry for 18:20-OCT-2007. Between stores Thursday morning first verify that I can still see this open circuit, i.e. with the Ohm meter plugged into the BLS cable connector of the Patch Panel verify that I still can not see the 1 Meg Ohm resistor on the ADF-2 card for the +8,6 HD+ signal. Swap the which PFC cable is plugged into which connector on the ATC card and now I can see the 1 Meg Ohm resistor from the Patch Panel, so it is not a problem with the PFC cable. OK swap the ATC cards. +8,6 is in Carte "A", rack M104, Slot 19, i.e. the 3rd slot from the left when looking at the back of the rack. Pull out ATC Card SN# 039. While this ATC Card is out (and thus LVDS cables unplugged) I see the following TAB input errors: TAB 0 Chip 6 Status 8120 TAB 1 Chip 6 Status C090 TAB 2 Chip 6 Status 8048 Install ATC Card SN# "Spare #4". Plug everything back in. The TAB errors stop and now the +8,6 HD+ connection looks OK on the Ohm meter. On the bench look at ATC Card SN# 039. +8,6 HD should be relative eta 3, relative phi 1, i.e. ADF-2 Channel #13 HD, i.e. P2 backplane pins: A15 for HD+ and A16 for HD-. Check with the Ohm meter and all looks OK ! Apply a gentle bend and it is still OK. Put it in the car overnigth to get it cold and check it in the cold in the morning and it is still OK. I will bring ATC Card SN# 039 back to MSU and test it for continuity using a 100 ma or so source. -9,22 HD On the scope I can see the problem with it. It is breaking into oscillation. The period of the oscillation is between about 500 and 530 nsec. The oscillation makes a good differential signal. It is between 100 mV and 200 mV of amplitude. It starts and stops rather suddenly, e.g. within about 3 or 4 cycles. Dean and I could see it on a scope Thursday night. It was not on the scope Thursday morning or Friday morning. -9,22 HD has some history in our Run 1A records. +6,25 EM was Excluded Thursday evening at about 23:00. Friday morning when I looked at it on the scope it looked fine. Friday afternoon after the between stores it still looks OK. +6,25 is not mentioned in the Run 2A notes about problem towers. Selcuk gave me the spare Bit-3 that is for the L1 Cal Trig system. It with its optical cable are in the brown cabinet. ------------------------------------------------------------------------------ DATE: 13-FEB-2008 At: MSU (action at Fermi) TOPICS: Exclude -9,22 HD At the beginning of Store 5905 there were high L1 rates from the L1 Cal Trig. This was especially seen in the Missing Et trigger. The control room could not get anyone on the primary or secondary L1 Cal pagers. Dean identified the problem TT from its high RMS value in Darien's display. They called Philippe at MSU and he Excluded TT -9,22 HD. This was at about 17:15 CST. ------------------------------------------------------------------------------ DATE: 16:18-JAN-2008 At: Fermi TOPICS: Work on TT +11,31, Work on the Logic Analyzer setup for the L2 MisMatch study, Deliver two BVDC cards to Mitch On arrival I found: The back door of M123 had been opened, the logic analyzer was in the middle of the aisle (instead of tucked back out of the way as much as possible where I had left it), the adaptor cable to see TT signals on the scope was gone. Make a new adaptor cable to let you plug the scope into the TT Patch Panel test points. Work on TT +11,31 EM Dean has replaced the Driver Hybrid on the BLS card and now both sides of the +11,31 EM signal are there but it looks a lot more "active" than its neighbors. It looks kind of like its gain had been turned up by a factor of 2 or something. The waveform of the EM+ and EM- look OK. +11,31 HD looks like its neighbors. I sent a note to Dean and Selcuk reporting the current status. Between stores Dean ran the pulser and now we can see that it is only the EM+ signal of +11,31 that looks too big. It is 2x too big so we suspect that it is not being terminated by the ADF-2 card. To further diagnose the +11,31 EM problem do the following steps: - Verify that the +11,31 EM signals at the trigger end of the long Blue cable look good. Do this by unplugging this blue cable from the patch panel and looking at its signals using the adaptor cable that lets you properly terminate the signals and look at them on the scope. These signals look fine so we are getting good stuff from the platform. Plug everything back in for the normal setup. - Next check to see if it is a problem on the Patch Panel Card. Monitor the +11,31 EM signals on the scope from the normal Patch Panel test points. Unplug from the Patch Panel the Pleated Foil Cable that carries +11,31. When you unplug this PFC cable you can see that both +11,31 EM signals now look unterminated. Next plug the other PFC cable for this Patch Panel into the connector for the +11,31 PFC and we saw that both sides of the +11,31 EM signal now look terminated. Thus the problem is not on the Patch Panel. Plug everything back in for the normal setup. - Next check to see if it is a broken wire in the PFC that carries +11,31. To do this swap where the two PFCs plug into the ATC card that receives +11,31. As you are monitoring the +11,31 EM signals on the scope you can verify that you have unplugged the correct PFC from its ATC card because when it is unplugged both sides of the +11,31 EM signal will look unterminated. We swapped which connectors on the ATC card these two PFC cables where plugged into and then both sides of +11,31 EM looked fine on the scope. Thus it is not a problem of a broken wire in the PFC cable. Plug everything back in for the normal setup. - So it is either a problem on the ATC card or a problem on the ADF-2 card. There are a couple of ways to investigate this. You could compare the Trigger readout for +11,31 EM either with its neighbors or with the Calor Precision readout. If the trigger readout is only 1/2 of what it should be then it means that the +11,31 EM+ signal is not getting to the ADF-2 card. EM+ is not terminated because it never reaches the terminator on the ADF-2 card and the readout from the ADF-2 card for +11,31 is only 1/2 of the expected size because one side of the differential signal is missing. If the trigger readout is 1.5 times the expected size then it means that the terminator resistor on the ADF-2 card is missing. The ADF-2 is seeing both sides of the differential signal: one side at its normal amplitude and the unterminated side at twice its expected amplitude. You could use an Ohm meter to look at the ADF-2 card input. Plug the Ohm meter into the connector on the Patch Panel where the Blue cable normally plugs in. You can not see the 80 Ohm terminator on the ADF-2 card but you can see the 470 nFd capacitor in front of it charge up and you can also see the 1 Meg Ohm resistor on the ADF-2 card that holds the center conductor of the ribbon coax cable at long term DC ground. We made the Ohm meter check and could see neither the 1 Meg Ohm resistor or the 470 nFd capacitor charge up. Thus the problem with +11,31 EM+ was most likely an open circuit on either the ATC Card or the ADF-2 card. First try replacing the ATC card. From the labels on the front of the ADF-2 Crates you can see that +11,31 is Slot #21 in the ADF-2 Crate in Rack M106, i.e. ADF-2 Crate "B". This should be an easy ATC card to work on because it is an end slot. Carefully uncable the ATC card in this slot and pull it out. This is ATC Card Serial Number #20. When this card was out (and thus its LVDS cables disconnected) I could see on the L1 Cal Trig TCC the following errors with each 5 second sweep of monitoring information: TAB Chip Status --- ---- ------ 0 #7 8084 6 #7 8120 7 #7 8080 Install spare ATC card #7. The TCC monitoring errors quit when the LVDS cables were plugged back in. Now both +11,31 EM signals look OK on the scope. Look at all of the TT signals in this Patch Panel and the adjacent Patch Panel to make certain that everything is plugged back in before turning off the pulser. Check ATC Card SN #20 with the Ohm meter. Yes you can see that pin 21D in the backplane connector is not connected to its pin in the 3M PFC cable connector. Label ATC SN #20 and put it in the spares cabinet. Logic Analyzer for L2 Mismatch Study Work to connect the logic analyzer to the L2_Decision lines for Geographic Section 0x20 i.e. 32 decimal. Put connectors on another set of the L2 FW FOM output cables so that the floating buffer card can get at L2_Acpt and L2_Rej for GS 0x20. The connections to the logic analyzer now are: Logic Analyzer Input Drew DeMux I.E. L2_Answers for Pod Signals Card Output L1 Specific Triggers --- ------- ----------- -------------------- A3 0:7 24:31 24:31 A2 0:7 16:23 16:23 A1 0:7 8:15 8:15 A0 0:7 0:7 0:7 C3 0:7 72:79 72:79 C2 0:7 64:71 64:71 C Clock_3 L2_Answer_Strobe D0 0 Mismatch Flag signal from L2 Global D0 4 L2_Rej for G.S. 0x20 D0 5 L2_Acpt for G.S. 0x20 The logic analyzer trigger is setup so that it is only looking at bit #0 from pod D0. In state 1 it requires D0(0) to be LOW for it to move to the next state. In state 2 it requires D0(0) to be HIGH for it to trigger. The trigger point is set for 97% of the way through the memory. Jim Kraus used the L2Glb code that pulses its flag to the logic analyzer on every event and we could see it and trigger the logic analyzer just fine. I watched things running during ZB and it does look like the coding of the at the output of the L2 FW is the following: To send out a L2_Acpt to this G.S. both of these lines are asserted. To send out a L2_Rej to this G.S. only the L2_Rej line is asserted. To issue a L2_Decision that does not involve this GS both lines are low. So at this point in the system the 3 values that you see on these 2 lines (with L2_Rej on D0(4) and L2_Acpt on D0(5) are: 3 This is an L2_Acpt for this G.S. 1 This is an L2_Rej for this G.S. 0 This L2 Decision is not for this G.S. Recall that the rest of the decoding to give the conventional meaning to the L2_Acpt and L2_Rej signals on a given SCL link takes place in the SLF card in the SCL Hub-End. See the mail with Ted Zmuda from 10-Jan-2002. The 34 pin flat cable connects from Fermi stock are 3M # 3414-6634.