D-Zero Hall Log Book for 2004 ------------------------------- The most recent entries are near the beginning of this file. This file begins in January 2004. This file contains both Trigger Framework and L1 Calorimeter Trigger entries. Earlier D-Zero Hall Log Books are on the web in one of the following directories: http://www.pa.msu.edu/hep/d0/ftp/l1/framework/logs/ http://www.pa.msu.edu/hep/d0/ftp/run1/l1/inventory_logs/ ------------------------------------------------------------------------------ DATE: At: Fermi TOPICS: ------------------------------------------------------------------------------ DATE: At: Fermi TOPICS: ------------------------------------------------------------------------------ DATE: 7:10-DEC-2004 At: Fermi Topics: L1 Cal Trig Trips, TAB readout tests, Waveform generator from Stefano, Power Pan Repair There have been 3 safety system power trips of the L1 Cal Trig over the past 2 weeks or so. I think that there is a leak starting in the radiator pack between M111 and M112. This is currently at the drop or 2 per day level. For now there is nothing to do except to move the drip detector out of the direct path of these drips and wait for the leak to get big enough so that it is possible to find it. There is a similar situation between M110 and M111 for the past N months. Made readout to L3 tests from TAB. There is now a trigger file in the commissioning area called something like "test tab". I made a different .vio setup file to put the 0x10 VRB crate into the correct configuration for the TAB tests, i.e. only one channel pair on just one VRB enabled. This vio file is in Scratch and is called something like "setup_0x10_for_tab_test". I need to get more geographic sections into the COOR Resource file: 124 $7c Cal Trig sidewalk #1 125 $7d Cal Trig sidewalk #2 $4e Dean's 2nd link to Cal 5k test something for Ken's Cal Track match Received the TWG from Stefano. It came 100% + complete with all cables and books and how to use instructions written up by him. Test Tier 2-3 Power Pan MM-2. It consists of bricks: 5V 50A #91 from 15-SEPT-1993 Pioneer SN# 438479 2V 325A #48 from 15-MAR-1991 Pioneer 406189 with MSU 6 CAP 5V 600A #17 from 14-FEB1992 PowerTec SN# 1726 just back from repair This Power Pan is now back in the cabinet as a spare. Start work on Power Pan PDM-19 which had its -2V brick die intermientently on 22-AUG-2004 while it was running Tier 1 in rack M110. PDM-19 consists of: +5V 600A #23 7-FEB-1992 PowerTec SN# 1722 -4.5V 200A #38 15-MAR-1991 Pioneer SN# 398980 -5.2V 200A #56 15-MAR-1991 Pioneer SN# 398962 -2V 250A #55 23-JAN-1992 Pioneer SN# 416247 Pull the -2V brick and take it apart. Replace the 680 pFd mica feedforward capacitor C122 with a 500pFd 3 KV cap. Put everything back together and run it for 1/2 hour. All looks OK so PDM-19 is back in the cabinet as spare. ------------------------------------------------------------------------------ DATE: 30-NOV-3-DEC-2004 At: Fermi Topics: Get L1 Cal Trig ready for Beam Physics Running, Delayed Busy for SMT Arrival: Desk lamp has been taken, cable press has been taken (but not the dies), L1 Cal Trig front doors not fully closed, Box of 25 mil pitch cable is missing from 6th floor, the box of TFW paddle cards was open and some had been pulled out. The two high eta CTFE cards that have bad pedestals are: +17:+20 phi 23 and -20:-17 phi 9. In both cases they have a pedestal of 0 for all channels or at least almost all channels. I assume that this is a Term-Attn-Brd soldering problem (cf. below as Find_DAC also had issues with these cards) +17,23 CTFE SN# 285 replace the first TAC card in the serial string -17,9 CTFE SN# 295 replace the first TAC card in the serial string I asked Pete Simon and he serviced the blower on the L1 Cal Trig. Wednesday swap the Drew DeMux card for the new type that has readout back to the MBT. The old know good working DeMux is labeled and stored in the L1 Cal Trig VIPA Readout crate. Could not test the new DeMux card with the full suite of 128 triggers because some down stream logging process was having trouble. Thursday Check CTFE EM HD PROM's page 7 eta +- 1:16 phi 1:32 all OK CTFE Px Py PROM's page 2 eta +- 1:16 phi 1:32 all OK CHTCR PROM's Ref Sets 0:3 EM and Tot eta +- 1:16 phi 1:32 all OK Make Find DAC runs (cf. below). One sweep eta +- 1:16 keep 2 No histograms and monitoring in L1 Cal Trig left running. One sweep eta +- 1:16 keep 2 with histograms. One sweep eta +- 17:20 keep 2 with histograms. Philippe updated the load gains and peds msg file to load this new set of pedestals. He studied the histograms from these Find DAC runs and we should not stop Excluding -4,25 HD and +9,31 HD. Work on Power Pan MM-2 This Power Pan had it -4.5 Volt brick die on 19-JULY-2004 when it was operating in M110 Tier II. Pull from this pan 600 Amp brick MSU SN# 16 from 11-JUNE-1991 PowerTec number 1264. Install in this pan -4.5V 600A brick MSU SN# 17 from 14-FEB-1992 PowerTec number 1726 which recently came back from factory repair. This pan is now ready for testing. Work with Ted and others on the Delayed Busy for SMT. This will be done in the Status Concentrator for the first 8 Geo Sections. We will pickup 8 control registers to handle the delay values. This should all be TFW post init auxi vio stuff. Find_DAC: --------- Run Find_DAC on eta 1:16. This corresponds to Find_DAC_V10_5_E_20041202.tti;1 Run Find_DAC again on eta 1:16, after excluding L1CT to stop TCC from collecting monitoring invormation. This corresponds to Find_DAC_V10_5_E_20041202.tti;2 with Find_DAC_V10_5_E_20041202.hst;1 Run Find_DAC on eta 17:20. This corresponds to Find_DAC_V10_5_E_20041202.tti;3 with Find_DAC_V10_5_E_20041202.hst;2 Compare Find_DAC result files to previous pedestal files: ----------------------- The previous file for 1:16 was Find_DAC_V5_0_F_20040109_Edited.tti;1 which had 8 towers with a Standard Deviation more than 50% different from the local average. Today's first run of Find_DAC for 1:16 found 19 such towers, and the second found 34. In particular HD(-4,25) was witnessed on today's first run with a huge amount of noise. HD(+9,31) has more than twice as much noise as its neighbors. These two towers were currently in the list of exlcuded towers. Other suspicious towers are HD(-4,8) HD(+9,24) and HD(-15,24) with more than twice noise than the local average while these towers are not currently excluded. Other suspicious towers are the ones with less than half as much moise as their neighbors. EM_TT(+ 7, 8) EM_TT(- 5,25) HD_TT(-14,12) In the range 17:20, Find_DAC_V5_0_F_20031212_Edited.tti;2 used to have 34 towers with noise 50% away from local average while today's run had only 12. There has been an improvement. One tower sticks out: HD_TT(+20,23) was found with a StdDev of ~10. The file for 17:20 was edited to simply accept the values for the two towers that Find_DAC had flagged as troublesome. They show excessive noise, wich caused jumps in pedestal average. This was for EM_TT(-20, 6) HD_TT(+20,23) 16 towers have a DAC to ADC ratio considered low. All these channels show "Warning: DAC to ADC Ratio Problem" with ratios around 12 instead of 13. Find_DAC is complaining (with details in the .hst;2 file) for all EM and HD channels of the two high eta CTFE cards that had been having low pedestals at +17:+20 phi 23 and -20:-17 phi 9. Find_DAC also reported "Warning: Failed Forcing Zeresp to Mid Scale" on 4 of these 16 channels. There is probably something going bad with the DAC reference voltage on these CTFE cards. Switch to using the new pedestal settings: ----------------------------------------- Edit Load_L1Cal_GainsPeds.mcf to now use Find_DAC_V10_5_E_20041202.tti;2 for eta 1:16 Find_DAC_V10_5_E_20041202_Edited.tti;3 for eta 17:20 Run L1 Cal Trig System Test: --------------------------- Tell TCC to Ignore L1CT to run Cla Trig System Test Configure the Test to cover Trigger Towers (1:16) using Lookup Page #7, Ref set #0:3 EM, HD Veto, Tot Et Ref Set Thresholds and verifying the EM and HD energy trees. The test blows up after some 20k loops as there is still a problem associated with EM(-13,2) where the CTFE Tot Et Ref Set #1 output would say 0 when it should say 1. This error is found on the global count at the Tier#3 CAT2 output and on the CHTCR input. Scale back coverage to RS #2 & #3 and run 200k loops then again 200k loops with RS #0. No problem. Set the range to eta(1:20). The test initialization consistantly complains that TT(+20,20) Tot Et RefSet #3 is 1 instead of 0. Run some 110k loops on all RefSets with no error except for a few at EM(-13,2). It is not clear why the initialization problem with TT(+20,20) does not appear again during the test loops. Also run 100k loops on eta(1:16) using Page #2 while checking the Px and Py momentum trees. No problem there. ------------------------------------------------------------------------------ DATE: 31-AUG-3-SEPT-2004 At: Fermi Topics: Running without LLRF from the Tevatron, New Geographic Sections, Optic SCL to 3rd Floor FCH, Meetings about new Cal Trig installation and patch paddle cards, Cables and covers for Optic SCL, Check about assembly companies and 2mm installation, Heartbeat trigger, Power Pans Delivered 2 boxes from the MSU Machine Shop to Kurt Krempetz. When I arrived I found that none of the screw down doors on the L1 Cal Trig had been closed fully after the power supply replacement on 22-AUG-04. At the time, both the electronics person and the Captain told me on the phone that they were correctly closed. Typically the latches had only been screwed in one turn. When the PCC is in "Free Run" the VCXO in the PLL in the PCC goes to the top of its range - about 53.10590 MHz. Talked with Dan Johnson in AD about the schedule for return of LLRF. It came back Wednesday morning and PCC was put back into "Normal Mode" with no problem. Wednesday meeting with Johnny Green and John Foglesong about BLS Patch and Paddle cards. There is an issue about what side of the Patch card the connectors go on. Received drawings to study and sign off on. Wednesday meeting with Norm Buchanan and Jon Kotcher and George Ginther about L1 Cal Trig and BLS Trigger Pickoff signals. Talked with Terry Toole about running monitoring stuff from the online machines. It sounds straight forward - need to pass the information to Philippe. SMT asks about a heart beat trigger. Email's with Scott. Talk with SMT about this again on Friday. They will try to put together a description of what they need. Add new Geographic Sectors to support the new DFEA Crates for CTT: Geographic Section Detector Crate # Crate # Crate Crate System (decimal) (hex) Name Location Comment -------- --------- ------- ----- -------- ----------- 1 : L1 NR 26 1A L1CT1 PC03-2 New DFEA Crate 1 : L1 NR 27 1B L1CT2 PC03-3 New DFEA Crate 0 : NR 122 7A L1JTO Electronics Area DAB3 Jamieson T Olson 0 : NR 123 7B L1CTW PW02-0 New DFEA Test Geographic Section $1A is carried on copper SCL cable 01SCL077 to rack PC03-2. I'm not going to activate this cable this week. Geographic Section $1B is carried on copper SCL cable 01SCL079 to rack PC03-3. I'm not going to activate this cable this week. Geographic Section $7A is carried on Optic SCL Cable #3 to Jamieson Olson's lab area in the 3rd floor Electronics Area. This was activated this week. Geographic Section $7B is carried on copper SCL cable 01SCL073 to rack PW02-0. I'm will activate this cable this week. Wednesday ran 3 duplex fiber cables to the 3rd floor FCH Electronics Area. The intent is to replace all the copper cables. These 3 new duplex fibers are labled Optic_SCL_#3 : Optic_SCL_#5. They are run via the official cable try path and arrive at the 3rd floor by the Combined Test Stand. These are 60 meter cables and all slack is at the 3rd floor end. Picked up from Johnny Green 1 Optic SCL Receiver box with 2 Finisar modules and 5 boxes with a single Finisar module. These suffered the standard problem of exposed 120 VAC connections and full exposed switching power supply. I talked with Victor and making covers and sent him a note about this. Things are full setup for $7A and gave an Optical SCL Receiver Box to Jamieson and sent a note to Jamieson and Stefan. $7B is also setup and ready for operation on the Platform. Visit Ted Neal Stefano Bill and Jamieson talking about assembly companies that they have used and installation of 2mm hard metric connectors. BEST in Elk Grove Village for proto-types Circuit Service in Wheeling for production 847.215.7171 SMA Surface Mount Assembly 252-5867 ManuTec 543-3022 Creative High Tech 847-718-0655 Clear overall vote is for Circuit Service. Talk about 2mm hard metric connector insertion. Use a backing plate, rational press, and dies that fit. No one knows of a press in for the 5 row side wipes - so you are stuck with: SMD, through hole side wipe connectors, and press in 2mm metric. With additional digging Stefano got the required numbers to begin the purchasing for the test waveform generator. Thursday meeting with Nikos, Alan, John F, John A, Johnny G, and Mario about BLS cabling and Run IIB L1 Cal Trig installation. The big change is to disassemble the existing L1 Cal Trig racks in place. The intent is to protect the BLS Cables even at the cost of taking a little longer to swap. Need more cables to enable the additional required Optical SCL channels. Need 8 cables for each end. Tom Regan is not around to pass this job to (he was given to AD for the shutdown). I was told to take the job to BJ. I refuse to do that and pass it to JA with a full note. Supplies the following connectors: MXC connectors for LMR-100 cable, e.g. to plug in the front of an SCL FanOut card are DigiKey J533-ND Johnson 133-3403-001 Qty = 10 SMB connectors for LMR-100 cable, e.g. to plug into an SCL Receiver Digi-Key J465-ND Johnson 131-1403-101 Qty = 10 SMA connectors for LMR-100 cable, e.g. to plug into SCL patch panel Digi-Key J525-ND Johnson 142-0403-011 Qty = 25 Returned 600 Amp bricks #17 and #22 to Fermi. No time to work on either of the two L1 Cal Trig Power Pans that are currently broken. I need to get on that to send the exploded 600 Amp brick back for repair and fix the 250 Amp brick that probably just has a leaking/shorted feedforward capacitor. This must get done next trip. ------------------------------------------------------------------------------ DATE: 22-AUG-2004 At: MSU Topics: L1 Cal Trig Power Supply Failure Early AM and morning on 22 Aug there were 3 or 4 occurrences of M110B -2V causing an Alarm and pausing the run. Each time the run would start up just fine again with no delay. I was somewhat suspicious that we perhaps had an alarm message problem and that it was actually M110A -2V that was causing the trouble. M110A -2V has been noisier than I would like (but has not tripped any Alarms so far). The first 1/2 of the afternoon M110B -2V ran just fine so I thought that it could make it through the next 12 hours until the shutdown started. Soon after the new store went in at about 3:30 PM this supply died. After 1/2 hr it started to run again and ran OK for about an hour and then died again. Tom Regan came in to swap it. The swap went OK. During the swap they ran the "full" trigger list OK by: acknowledging the 100 alarms, excluding G.S. 0x10, and NOT trying to re-load the trigger list. With M108 Tier 3 turned off, the Diff ECL did the right thing and the TFW saw no asserted And-Or Terms from the L1 Cal Trig. After replacing the supply (about 2 hours), the restart was fast and smooth (because everything else had been kept running). L1 Cal Trig Examine plots look OK for the rest of the night. No information about this has been added to the Run IIA Power Supply log book yet. No repair work has been done on this supply yet. Possible cause of the failure is C122 the servo loop feed forward mica capacitor. Fall 2004 Shutdown starts 4AM 23-AUG-2004 i.e. 7 hours after this power supply replacement. ------------------------------------------------------------------------------ DATE: 25-JULY-2004 At: MSU Topics: Exclude -6,25 EM Captain calls. They want -6,25 EM excluded. For the past day or two it has been clear that either EM or HD at -6,25 was causing trouble. It is now clear that it is EM and it is constantly causing enough trouble that we need to get rid of it. ------------------------------------------------------------------------------ DATE: 19-JULY-2004 At: MSU Topics: L1 Cal Trig Power Supply Failure M110 Tier 2 Power Pan blows its -4.5 V 600 A brick during a store. This is M110A -4.5V. It went with a fight and tripped the wall breaker which also knocked off M112 Tier 2. M110A was replaced by Victor and Tom between about 8PM and 11PM. I have not tried to repair this supply yet. Considering that it tripped the wall breaker, I assume that it blew apart the input filter and rectifier assemble (which is becoming the "standard failure mode" for the 600 Amps supplies). ------------------------------------------------------------------------------ DATE: 30-JUNE-3-JULY-2004 At: Fermi Topics: Replace the FPD NIM-ECL Converter that is in the Master Clock Crate, V13 L2 Fraction issue, Access connector on Mixer, Toroid Cal Noise TT +4,11 Meeting about ICR ICD MCP Replaced the FPD NIM-ECL Converter that is in the Master Clock's NIM Crate in rack M100. Stoped the Triggers (paued TFW when I did this. Everything came back to life in ZB except for 0x13 which took about 10 minutes to get going again. In Trigger List V13 there are a couple of places where there are pairs of triggers that are defined exactly the same at L1 and at L2. Some of these run in beam with a prescale of 1. But people have seen different L2 Reject rate for the two triggers that make up one of these pairs. The see this in the Luminosity reports. Things to check: Look at TrgMon while this is running (or at control room DAQ Mon). This has not been done yet. Verify the L2 Global to TFW communications. Run the list that has 128 triggers at L1 each prescaled to fire at a couple of Hz and each of these has a random 50/50 accept/reject at L2. Watch TrgMon. If L2 Global every receives an SCL L2_Decision that it does not expect then it will throw and error. This test has been done and looks clean. We watched TrgMon during these tests and things look OK. Thursday afternoon at 14:00 the Calorimeter became noisy in two regions in the South: a stip at eta +1 i.e. +1,17 through +1,24 and a strip in 2 phi's: +7,23 +7,24 through +12:23 +12,24. I could very clearly see the noise on the scope that was plugged into +11,2? HD. It was bursts of noise that came about once per second. Each burst lasted for some usec or 10's of usec and contained oscillations with period of a couple hundred nsec. The noise was symmetric on both sides on the diff signal and was 100 mV to a few 100's of mV in amplitude. At about 17:15 the toroid magnet was ramped down and about 1 minute after we thing the magnet reached zero current the noise bursts just stopped. Pierre and I were watching the scope when the noise stopped. We ran for about 1 hour at zero current and the Examines all looked clean and the scope remained clean. We rampped back up and after some minutes the noise returned and the scope showed noise bursts just as before. But after about 3 minutes this session of noise stopped. We did not touch the toroid magnet - the noise just stopped. Visit with Stefano and Neal. The connector that they used for the Logic Analyzer connection on the Mixer card is AMP "MICTOR 38" AMP 2-767006-2 which works with Agilent high density adaptors E5346A for the 16550 analyzer. They use bi-color LED's and the button protocol is a debounced push of less than 2 seconds steps the display - a push of more than 2 seconds causes the hardware reset. HP signal generator can be 488 or 232 they would like the relay box to be 232. Need to verify this with Philippe and Jason. Friday evening I Excluded TT HD +4,11 at the resuest of the CalMuon shifter and the Captain. There had been rate fluctuations at various times for the past day or two but the noise never lasted long enough to find the problem tower in the Examine plots. Finaly in the evening it stayed noisy for long enought and HD +4,11 was clearly sticking up by a factor of 5 or 10. I Excluded it at about 22:20. I put it on the Excluded list. HD +4,11 also was showing up in the precision readout as a problem. ------------------------------------------------------------------------------ DATE: 1,2,3-JUNE-2004 At: Fermi Topics: 12 Hours of Lost Beam This is a note about the Tick and Turn Number error problems that were seen in the CFT system. I first heard about this in a detailed note from George. Because all 4 CFT crates would have a problem at the same time it was very suspicious that they were receiving faulty commands over the SCL. The SCL Hub-End (in the top of rack M124) receives two clock signals from the Master Clock. These signals are a 53 MHz clock and a 7.59 MHz clock. For the SCL Hub-End to operate correctly the relative phase of these two clocks must be aligned to within about +- 1 nsec. The symptom of the 7.59 MHz clock being slightly late wrt the 53 MHz clock is that some of the SCL Fanout cards will occasionally send out incorrect SCL data. As the 7.59 MHz clock becomes an additional fraction of a nsec late wrt the 53 MHz clock then all of the SCL Fanout cards begin operating incorrectly. - The two slots in the SCL Hub-End crate that are most sensitive to a late 7.59 MHz clock are the slots that hold the SCL Fanout cards for Geographic Sections 0x50 : 0x57 and 0x78 : 0x7f. As the 7.59 MHz clock moves even later, by a fraction of a nsec, then all the SCL Fanout cards have trouble. - When the 7.59 MHz clock is too late wrt the 53 MHz clock there is a Red LED on the SCL Fanout cards that will flash. - When the 7.59 MHz clock is too early wrt the 53 MHz clock there is no indication except that the SCL sends out faulty data. - I last had to adjust the relative phase of the two clock signals to the SCL Hub-End in August 2002. This adjustment is made by inserting "delay cables" in series with the clock lines that run from the Master Clock (middle of rack M100) over to the SCL Hub-End. - I have a set of delay cables in 0.5 nsec steps. The procedure for making this adjustment is to determine the range over which the SCL Hub-End operates correctly (by inserting different delay cables) and then leave the relative phase of these two clocks set in the middle of the range of correct operation. - Note that this adjustment is not changing the Master Clock wrt the Tevatron beam crossings. - This adjustment was last checked in January 2003. Details about all of this are in our TFW log book. - I have no idea why the required setting of the relative phase of these two clocks seems to have changed since the feeder fault power outage last week. I know that it has a temperature sensitivity, but Daniel verified that we had normal cooling water temperature Tuesday morning. The problem reported by George and visually checked Tuesday morning by Daniel exactly fit the symptoms of the 7.59 MHz clock being just slightly late wrt the 53 MHz clock. As this was interfering with the operation of the experiment the decision was taken at about noon Tuesday to adjust the relative phase of these two clocks by removing a 0.5 nsec delay cable from the 7.59 MHz clock line that runs from the Master Clock to the SCL Hub-End. This delay cable is located in the back of the Master Clock crate. The Master Clock signals come out of paddle cards that are in the back of the Master Clock crate - one paddle card is behind each Master Clock Fanout card. The 0.5 nsec delay cable in the SCL's 7.59 MHz clock signal was between the Master Clock paddle card for the SCL and the line that runs over to the SCL Hub-End crate. Removing this 0.5 nsec delay cable should have been a clean 2 minute job. Unfortunately when this delay cable was removed the paddle card for the adjacent Master Clock Fanout was pulled partly out of its socket. This disturbed paddle card sends Master Clock signals to the Routing Master crate. The Master Clock paddle card for the Routing Master was still fully plugged in at the top but pulled out at the bottom. Because the Master Clock paddle card for the Routing Master crate was partially pulled out, the Routing Master did not work when the experiment tried to start back up. The symptoms from the Routing Master were very confusing. It kind of worked (it would do VME cycles that were spaced out by some usec's), but it would just hang VME cycles that were initiated by the full speed program running on the Routing Master's SBC. Note that the hardware cards in the Routing Master crate can not operate correctly without receiving their clock signals from the Master Clock. They can not even do VME cycles without receiving correct clock signals. I will skip all the details but the failure symptoms from the Routing Master were very confusing and it was only by Doug's hard work that we were able to conclude that there must be a hardware problem in that crate. We have had no other problems with the Routing Master crate since it began operating 2 years ago (and this was not really a problem with the Routing Master crate). ------------------------------------------------------------------------------ DATE: 30-MAY-2004 At: MSU Topics: Exclude EM(-6,24) Captain calls. This is the first store after recovery from feeder 45 fault. L1 rates are consistently high, and CalMuon shifter (or expert?) identifies EM(-6,24) as a hot tower. The L1Cal rate is not out of control, but they are concerned that there may be other trigger issues too. TT_ADCmon does not show gross pedestal problem (3 sweeps read 4, 8, 10). But our diagnostics "List Hot TT(RS#0)" shows this tower with 146, 172, 174, entries out of 3 rounds of 1000 point histograms. Edit and run Excluded_Trigger_Towers.msg to exclude the EM side of this tower. Note: the HD side of this tower was already excluded. This is done "live" just before switching runs, and the rate falls from ~1200Hz to ~1100Hz. ------------------------------------------------------------------------------ DATE: 24:26-MAY-2004 At: Fermi Topics: TRICS 10.5.E, L1 Cal Trig Testing, Shifter TFW Talk, Meetings with people about Run IIB L1 Cal Trig Have a meeting with Alan Stone, John Anderson, and Marvin Johnson about the cables for the BLS analog input to the ADF-2. John and Marvin suggest using mass terminate cables like they have used for the SMT and CFT systems. These would be used with a paddle board in the back of the ADF Crate and make a second pcb to do the transition. Stopped TRICS 10.5.D and started TRICS 10.5.E The page file usage by TRICS still looked fine after this 2 week run. Initialize full +-20 eta coverage of L1 Cal Trig. This now looks fine - there is no problem with cards in Tier 2 for |eta| 17:20 that to not exist. I looked at the log file from this Initialize and I believe that everything that should be there is there. We still need a cold start to prove this. Edit Init_Post_Auxi_L1CT.cio to change $CAT3_Corr_Value from 1 to 2 so that we have the proper 2's complement correction for Missing Et coverage over |eta| 1:16. Init L1 Cal Trig to pick this up. Run L1 Cal Trig Exerciser over: |eta| 1:16 phi 1:32 lookup memory page 7 EM HD Ref Sets 0:3 EM Tot HD_Veto Check Global Counts. After 11k loops it blows up on -13,2 Tot Et Ref Set #1 (this is the "standard" error) It had just changed -13,2 Tot Et Ref Set Comp #0. Keep the eta +- 16 but restrict to Ref Sets 2,3 EM Tot HD_Veto. This runs 200k loops. Keep the eta +- 16 but restrict to Ref Set 0 EM Tot HD_Veto. This runs 200k loops. Setup for: |eta| 1:16 phi 1:32 lookup memory page 2 Px Py Check Global Px Py. It appears to run OK. Switch to lookup memory page 3 and it blows up. Good. Switch pack to lookup memory page 2 and run 200k loops with no errors. What am I forgetting ? Switch to Memory Tests over |eta| 1:16 phi 1:32 CHTCR has no errors. Page 7 EM HD - no errors. Page 2 Px Py - no errors. Initialize full eta +- 20. Re-Load Gains and Pedestals. Set eta coverage to +- 1:16 for Physics running and give L1 Cal Trig back. Meet with Sten about the PLL for the ADF-2, with Stefano and Neal about the signal generator, and with Ted about making additional receivers for the fiber optic SCL. Meet with John Foglesong and Johnny Green about the BLS to ADF cables and patch panel and paddle card. Send them data on the crate. ------------------------------------------------------------------------------ DATE: 24-MAY-2004 At: MSU Topics: Analysis of last L1 Cal Trig Tests Study the logfile covering May 13th L1 Cal Trig system tests including verification of Px/Py. Observation #1: the momentum is off by one probably simply because Init_post_auxi_L1CT.cio loads the wrong 2's complement correction. It loads 1 instead of 2 in the final CAT3 cards: >> ! Define Correction Value >> !----------------------------------------------------------- >> >> $CAT3_Corr_Value= 1 ! 3 for eta(1:20) >> ! 2 for eta(1:16) >> ! 1 for eta(1:8) Observation #2: most of the other issues have to do with the ninth bit of Px and Py, which is the most significant bit of the input to the FMLN lookup. Observation #3: A lot of the remaining errors are of this type E$ Tier #3 MissPt Thresh Comp FMLN Px Input is 143=0x008f not -368=0xfffffe90 but this is just the same off-by-one issue applied to a large negative number, and truncated to 9 bits at the input to the FMLN. The error reporting is a bit confusing, and we need to think about what we can do about that. Observation #4: There are still a few errors that can't be explained the same way. These few errors are very localized during the tests in the logfile, which may mean an initialization issue. We have one weird error E$ Tier #3 MissPt Thresh Comp FMLN Px Input is 66=0x0042 not 329=0x0149 E$ Tier #3 MissPt Thresh Comp FMLN Py Input is -5=0xfffffffb not 316=0x013c E$_Changed HD_TT(+ 3, 4) EM/HD= 8/198 PROM EM/HD/Px/Py= 8/198/ 81/ 68 where things look quite off. followed by 3 instances all similar to this E$ Tier #3 MissPt Thresh Comp FMLN Px Input is 18=0x0012 not 275=0x0113 %%13-May-2004 16:43:51.610 E$ Tier #3 MissPt Thresh Comp FMLN Py Input is 14=0x000e not 271=0x010f %%13-May-2004 16:43:51.610 E$_Changed EM_TT(+ 2, 4) EM/HD= 57/ 8 PROM EM/HD/Px/Py= 57/ 8/ 27/ 23 %%13-May-2004 16:43:51.620 where (after the off-by-one correction) we just have bit #8 missing for both Px and Py. Conclusion: Fixing the 2's complement correction value should greatly help, and we will hopefully understand what else may be sick, if there are any errors left. ------------------------------------------------------------------------------ DATE: 12:14-MAY-2004 At: Fermi Topics: Optical SCL, TFW Power Supply, new TRICS, L1 Cal Trig Test, Replace and ERPB Started using the Optical SCL to the L2 Test Stand. The links for Geographic Sections $0A and $0B have been moved to the Optical SCL. The links for Geographic Sections $0C and $0D have been disconnected from the patch panel at the SCL Hub-End. All 4 of the coaxial cables that run to the L2 Test Stand have been disconnected from the SCL Hub-End patch panel and cable tied out of the way where they can not touch ground. I need to call Ted and ask for 2 more Return the TFW Power Pan SN# 4 to Fermi. This is the TFW Power Pan from M123 Middle that failed on Sunday evening Feb 29th It new has a factory repaired brick installed and appears to be in good shape. We are now back to having 2 spare TFW Power Pans at D-Zero. They are SN# 2 and SN# 4. Moved from TRICS 10.5.C to TRICS 10.5.D This should fix the memory leak, do a fancier job of draining the L2 system with each TFW Pause, make a best try at initializing Tier 2-3 CAT cards, and provide more tests for the L1 Cal Trig including the Global Counts. When setting the L1 Cal Trig eta coverage to eta -20:+20 (e.g. for a cold start) there is a problem that TRICS tries to initialize Momentum CAT2 cards in the Tier 2 for |eta| 17,18,18,20. The Initialize in this conditions throws a pile of errors in 3 batches but does run to completion. Once L1 Cal Trig is set for |eta| 1:16 the Initialize is clean. The Memory leak appears to be 100% fixed. Worked with the new L1 Cal Trig Test. It apppears that the smallest quanta that you are use is 4 in eta by 8 in phi, i.e. the area covered by a Tier 1 CAT2 card. That makes sense and is fine. When working with the Global sums it look like you should Initialize the full eta,phi space first and then back off to the eta,phi coverage that you want to use. That makes sense and is fine. I saw the same problem with -13,2 Tot Et Ref Set #1 as seen before. Backing off to just eta -12:+12 things ran fine with all Ref Sets and the Global Counts being checked. Checking Global Px Py did not work as well. Typically it was low by 1 count (both Px and Py) and at times was way off but would them come back. There is log file stuff from late Thursday afternnon that is worth looking at when we have time to understand this. Replaced the 3rd down ERPB card in Rack M112. Pulled out ERPB SN# 30. Installed ERPB SN# 78 from the spares box. Taged and put SN# 30 in the spares box, i.e. did not bring it back to MSU. Before doing this I could see screwed up data (starting with the 4th ERPB down in rack M112) once every so often. We also saw some shifter plots look screwed up. With the new ERPB everything looks OK so far. See the log book form 11:15-Nov-2003 for more details. ------------------------------------------------------------------------------ DATE: 21:24-APR-2004 At: Fermi Topics: Optical SCL, L1 Cal Trig Power Supply, Master Clock Frequency, Clock distribution in other systems, Integration Test, Presentation at Operations Meeting, Optical SCL The SCL copper_to_optic box and the SCL optic_to_copper box are back from Tom. They did put heat shrink over some of the exposed AC but they did not do as I asked in writting, cover over the exposed open frame power supplies that have exposed line AC. They did not make a cover for the optic_to_copper box as I asked. I give up. I asked again and Victor made a plastic cover for the optic_to_copper box. I will just run the copper_to_optic box with exposed line voltage. And yes there we still exposed dangling wires on the DC sides. Pickup the fiber optic cables from Bruce Merkel (x3263) for the Optic SCL connection. There are: 2x 15.5 meter, 4x 31 meter, and 4x 60 meter cables. This is 62.5/125 cable with Duplex LC connectors. They are from Fiber Instruments Sales 161 Clear Rd Oriskany NY 800-500-0347 www.fiberinstrumentsales.com www.fisfiber.com Part No: D2LLM15.50FIS Part No: D2LLM31FIS Part No: D2LLM60FIS Wednesday afternoon install the SCL copper_to_optical box above M123 and run 5 MCX-SMA patch cords into M124 SCL Hub-End. These new patch coards are labeled "OPT #1" : "OPT #5". Thursday norning run the SCL optic_to_copper box in MCH-1 and feed its output into the SCL Test Receiver - Logic Analyzer. At first it was all junk data but then I figured out that the front panel labels were wrong on the optic_to_copper box. They mixed up TX+ and TX-. Straighten that out and the data looks fine. It runs for some hours in ZB and no loss of lock. Thursday afternoon install two duplex 31 meter fiber optic cables up to the L2 Test Stand. These are labeled "SCL OPTIC #1" and "SCL OPTIC #2". There is a plastic sack with the dust caps left up above M123 by the copper_to _optic box. I sent Miroslav a note saying that things are ready but I have not heard back yet. L1 Cal Trig Power Supply Thursday the DAC Shifter, Maren Vaupel, just after coming on shift, came and found me and told me that there were some L1 Cal Trig Major Alarms. M110 Lower Tier 1 was not running. Its 30 Amp AC breaker had tripped. I tried restarting it but the +5 Volt brick was a no go. From M110 Lower Tier 1 Pull PDM-12 PDM-12 consists of: +5.0 Volt MSU SN# 17 14-FEB-92 <--- no output -2.0 Volt MSU SN# 41 15-MAR-91 -4.5 Volt MSU SN# 42 15-MAR-91 -5.2 Volt MSU SN# 39 15-MAR-91 The failed brick, +5.0 Volt MSU SN# 17 14-FEB-92 is PowerTec SN# 1726 Install PDM-09 See Log Book for 24:26-MAR-2004 for the recent work on PDM-09 and its current setup. Turn things back on and Run 2 passes of CHTCR PROM Test, and 2 passes of EM page 7 PROM test, and one pass of Momentum page 2 PROM test. Pedestals are loaded, its is initialized out to 20 and now set to run out through 16. TT_ADC_MON looks OK, Cal Trig Test TT's over Ref Set looks OK, rates from the default set of Ref Set triggers look normal, readout data to L3 looks OK. Give 0x10 back to ZB and thank the DAQ Shifter for saving us from a big problem when the store is loaded. While turning things back on it is clear that M110 Upper Tier 1 -4.5 Volt is making noise, has AC ripple and needs work done on it before it causes trouble. Using the StripTool M110 Upper Tier 1 -4.5 Volt is clearly not as clean as the other similar supplies. When turning back on after the Power Pan replacement the Fan Belt made noise. It did not make noise 3 weeks ago when powered up after the mini-shutdown. I told Pete but I should also send him a note. At the beginning of the next store on Thursday evening, the shifter's were concerned because the rate of the low Et EM trigs dropped 10%. The actual problem was that it had been hot by a lot. Watching the Quad Terms since the beginning of the store one had been almost an order of magnitude out of balance (AOT's 0,1,2,3). AOT #2 had been hot. They had not been looking at L1 Cal Trig Examine. Once we looked at it, it was clear that -6,22 EM had been hot. Now it is OK. Then put PDM-12 back together with a new +5 Volt brick. PDM-12 NOW consists of: +5.0 Volt MSU SN# 19 17-FEB-92 <-- PowerTec SN# 1723 -2.0 Volt MSU SN# 41 15-MAR-91 -4.5 Volt MSU SN# 42 15-MAR-91 -5.2 Volt MSU SN# 39 15-MAR-91 Note that the new +5V brick, MSU SN# 19 17-FEB-92 PowerTec SN# 1723, is a brick that had been sent out for repair in January 2003. This is its first use since it came back from factory repair. Master Clock Frequency Just for reference, look and see what does the Master Clock Frequency Meter say under different conditions: No current on the bus: 53.103692 MHz 149 GeV with beam in the TeV: 53.103693 MHz 980 GeV during a store: 53.104701 MHz The 150 GeV number is exactly what I expect but the 980 GeV number is 1 ppm lower than what I expect. The rock in the counter must be pretty good. D-Zero Clock Distribution Sequencer and Sequencer Controller. CY7B9910 and CY7B9920 these are used on the controller the clock then goes onto the backplane where it is received by the sequencer goes through another CY7B99x0 part and is used as the reference for G-Link. Mixer. Clock is point to point accross the backplane and then very carefully distributed on each card. It uses Cypress parts: and analog switches to select which clock to use. Very careful on card distribution to a large number of devices on each card. All devices are within 1 nsec of each other. www-ese.fnal.gov/D0_CTT_MIXER ADF. This does 8/7 TeV RF. It uses Cypress RoboClock parts: CY&B992 and CY&B9920. It uses a 7 stage shift register with with one bit up to generate the divide by 7 for the feedback to the PLL. The neat trick advantage of that is that asserted time of the feedback signal tracks the clock and is glitch free without latching the decoded counter output. Cal. Received from SCL on controller cards and from there careful dendicated point to point fanout down into the platform. No PLLing. Integration Test Readout some events to the VRB where VMEAccess was used to capture the formatted and raw version of the data and write it to a file. Next step it to make a configuration file and do this for real. Operations Meeting Presented the status of Excluded L1 Cal Trig Channels. Test of this talk is linked to the agenda. ------------------------------------------------------------------------------ DATE: 11-APR-2004 At: MSU TOPICS: COOR times out requesting SCL_Init DAQMON_Scraper ITC problems Restarting Trics makes it all better The Control Room calls during beam. The symtpoms experienced were that COOR was timing out while sending SCL_init messages to the L1 Trigger Control Computer (that's COOR's l1dnl channel). Pilippe managed to connect to TCC from home over VNC and could see the last messages that COOR had sent to L1TCC, including SCL_init. But something else was obviously wrong because there were many ITC Connect/Disconnect attempts from "DAQMON_Scraper".DAQMON_Scraper is the application which collects monitoring information from various places (inlcuding L1TCC) and feeds DAQ_AI, and the L2 GUI display. It didn't seem that COOR was failing to send its messages, nor TCC failing to receive them, but it looked like the Scraper program (or something else) was disrupting TCC's normal operation. Trics had been running for a long time, and the memory leak had grown to the largest ever (because of the now larger page file). TCC was only using 80% of its page file. With the smaller page file, we used to have zero problem up to the end at 95-99%. It didn't seem that this was the cause of the problem. Maybe it was. My thought at the time was that either the Scraper was out of control, or that all these were only symptoms of a bigger network problem. Unfortunately the daqexp didn't know how what the daqmon_scraper process was and how to restart it, or how to find out. We were losing beam, so I proposed to restart the control application on L1 TCC. This application is called Trics. This only takes a minute, plus another minute for COOR to restart the run. I figured that this would either get things back in order, or make the real problem more evident. Trics was restarted, COOR reloaded the run, and data taking restarted almost right away. So I am still not sure of what was truly wrong. I am not sure if Trics was really the cause of the problems, or if restarting Trics helped something else get back in order. It looks like we lost about 30 mn of beam. They probably took 20 mn to call, and it took 7 mn to decide to restart Trics, plus 3 mn for coor to start the run. Examining Trics' logfile on Monday didn't shed more light. Only DAQMON_Scraper showed obvious symptoms of ITC problems. Everything seemed to run slow. Even the 5 second monitoring capture was stretched to >10s. Maybe DAQMON_Scraper has its ITC timeout requirements set shorter than other host processes (coor, etc) and is the first one to panic and try disconnect/reconnect. ------------------------------------------------------------------------------ DATE: 7-APR-2004 At: MSU TOPICS: Exclude +9,31 HD and -9,22 HD We received a call Wednesday afternoon 7-APR-2004 that the rates were high and unstable and that both TT's +9,31 HD and -9,22 HD looked noisy to the Shifters. Using the L1 Cal Trig Test tool to sweep the Ref Sets 1000 times these 2 TT's typically showed up in the Jet Ref Sets 3 or 4 hits per 1000 sweeps. On TT ADC Mon these 2 TT's looked OK so it was not just a pedestal drift problem. We also got a chance to make 1000 Ref Set sweeps when the DAQ was stopped (i.e. no SMT noise) and these 2 TT's still showed up. Exclude +9,31 HD on the fly and the L1 rate dropped 250 Hz and now was stable. Exclude -9,22 HD on the fly and there was no additional drop in the L1 rate. The decision Wednesday afternoon (talking with Dean on the phone) was to put +9,31 HD in the file of Excluded TT's and to put -9,22 HD back into operation. About 3:00 PM a new run was started with +9,31 HD Excluded and things looked fine. Call Wednesday night and -9,22 HD had now gotten bad enough that it was causing trouble. At the end of the store Dmitri wanteded it Excluded. At about 10:30 PM add -9,22 HD to the file of Excluded TT's. ------------------------------------------------------------------------------ DATE: 31-MAR-2004 At: MSU Topics: Exclude HD TT(-3,31) The CALMUO shifter reports two hot Tot towers at (-3,31) and (-3,32). The captain (G.Fisk) reports that the L1 Accept rate is unstable with spikes that come and go. This is store #3335. Terry Toole plot shows that this may have started around 11pm yesterday, 30-mar-04. By the time we were called, DZero was in a special muon run not using L1CAL. There was no Reference Set defined. We cannot use our Diagnostics tools. After verifying with Captain Fisk, load the diagnostics suite of Reference Sets in Check_L1CT_setup_version1.msg. Our L1CAL diagnostic tool to look for Hot Towers shows that Trigger Tower (-3,31) is often above threshold (3 times in 1,000 point histogram, 1/1,000, 7/10,000, 4/10,000). TT_ADCmon also shows activity in this tower with unstable HD average of 6, 8, 10, and 10 counts. The last average of 10 counts was the contribution of 7 crossings (4,14,14,6,4,8,16). This tower indeed seems noisy, and this is not just a pedestal that has drifted. Our diagnostics tool to look at pedestals finds that HD(-3,31) has a standard deviation of 4.7, i.e. >3 times bigger than the expected 1.3. Dmitri and Gene request that this tower be excluded now, as opposed to waiting for the return to standard physics run. The Excluded Tower list has been edited to remove HD(-3,31) and executed. Looking at the Trigmon on the Web (with 1 mn integration time), we can see the rates decrease, including in the Quadrant and Region terms. And-Or Rate (wihle L1 Accept ~70Hz and Lum ~12E30) Term Before After excluding (-3,31) Tot RS #0 Quad 1 8 1376.7 1374.1 Tot RS #0 Quad 2 9 1896.4 1872.1 Tot RS #0 Quad 3 10 1025.3 1066.1 Tot RS #0 Quad 4 11 4594.9 1041.3 * Tot RS #1 Quad 1 12 314.68 306.11 Tot RS #1 Quad 2 13 331.37 326.38 Tot RS #1 Quad 3 14 244.95 271.53 Tot RS #1 Quad 4 15 1287.5 257.11 * Central Tot RS #0 20 8429.5 4890.8 * Central Tot RS #1 21 2113.3 1096.1 * Central Tot RS #2 22 897.19 399.47 * Central Tot RS #3 23 408.96 194.29 * Tot RS #0 Comp #0 136 10679. 7142.9 * Tot RS #1 Comp #0 137 2915.6 1907.9 * Tot RS #2 Comp #0 138 1142.2 653.31 * Tot RS #3 Comp #0 139 519.03 312.00 * Tot RS #0 Comp #1 140 1641.9 1629.8 Tot RS #1 Comp #1 141 237.59 234.82 Tot RS #2 Comp #1 142 65.37 70.55 Tot RS #3 Comp #1 143 28.70 30.82 Tot RS #0 Comp #2 144 687.58 682.01 Tot RS #1 Comp #2 145 54.98 54.41 Tot RS #2 Comp #2 146 14.12 14.20 Tot RS #3 Comp #2 147 5.39 5.84 Tot RS #0 Comp #3 148 364.96 365.47 Tot RS #1 Comp #3 149 13.61 14.71 Tot RS #2 Comp #3 150 3.40 3.89 Tot RS #3 Comp #3 151 1.51 1.61 The rate plot no longer shows the spikes for physics running, around noon CST. http://www-d0online.fnal.gov/www/groups/tm/daqrates/33XX/store_3335.png ------------------------------------------------------------------------------ DATE: 30-MAR-2004 At: MSU Topics: Exclude HD TT(-4,25) Dmitri called, concerned about high L1 rates during a special run. The L1Cal diagnostics tool that builds 1000 point histograms shows that TT(-4,25) is above its Tot Et Ref Set #0 Threshold 70-90 times. Even TT_ADCmon can see it averaged at 15 on some sweep. This is reported to Dmitri & Captain, but no action taken yet. This is the beginning of store #3335. 30 minute later Dmitri calls as problem seems to get worse as L1 and L2 busy percentages keep increasing. Exclude HD side only of TT(-4,25), on-the-fly, during non-recorder run, and while the accelerator is doing additional scraping. The L1 rate instantly drops 750->550Hz. Excluded_Trigger_Tower.msg has been updated and executed in verification. This was around 1:30pm CST ------------------------------------------------------------------------------ DATE: 24:26-MAR-2004 At: Fermi Topics: BLS Cable repair, COMICS CMC down load, ADF integration test, Work on isolation of SCL Link to L2 Test Stand, Check the Quadrant-Region Terms, Start up Cal Trig after mini-shutdown, PDM-09, Cal Trig Timing Windows OS MS04-007 patches, d0tcc1 pagefile, Power Supplies can now stop run, Start Trics V10.5.A,B,C Cal Trig Random Test. The official Trics version is now V10.5.C (cf. below for more details) Work on integration test: Tried with a different SCL Receiver and looked with a scope. The Geographic Sections for the Run IIB Cal Trig sidewalk test setup are decimal 122 and 123 $7a and $7b. By using auto disable and doing a SCL_Init right before moving one event we could see data from the TAB. Jovan said that the header looked OK but the rest of the data was zero's. I did not see the VRB saying anything bad about loosing G-Link frame lock - that looked OK. When trying to move a second event (after SCL_Init) it did not look OK. BLS Cables have not been repaired at the MCH end. The one open cable on the platform was repaired. The work requested 3 times in writting was: -19,30 HD There is a short in the HD- side of the cable at the MCH-1 end of the cable. +13,23 EM There is a EM + cable short at MCH-1 end. +14,11 EM There is a EM + cable short at MCH-1 end. +16,8 EM The EM- coax has an open center conductor. I have checked this with a capacitance meter and I believe that the open is right at the Platform end connector. The isolator boxes for the SCL link to the L2 Test Stand were passed to the Bob Jones group. The cable work will be done here by Victor. I have passed the written request to Tom. John passed the draft instructions to John. Master Clock download from COMICS Fritz has practiced this on the Sequencer #2 and that looks OK. Cal Trig is running on that. Then Friday morning test the COMICS download on both Seq #1 and Seq #2. As we leave the Master Clock is running from a COMICS download. Startup L1 Cal Trig after the one week mini-shutdown Start running TRICS 10.5.A The Upper Tier 1 Power Pan in rack M110 is making a lot of noise. Scan all CTFE and CHTCR PROM's. See the "normal" problem at Px +13,27 See the log book for: 15:16-JAN-2004 and 11:15-Nov-2003 and 25:27-JUNE-2003 and 9:13-JUNE-2003 and 8:11-JULY-2003 for the details of this CAT2 readback problem. (finally) Take decision to replace this CAT2 card. Replace Px CAT2 for eta +14:+16 phi 25:27 Pull CAT2 SN# 168. Install CAT2 SN# 227. Tag CAT2 SN# 168 as having a readback problem of value 16 at some Register Addresses. Run the PROM test again for all CTFE and CHTCR PROM's. Now all OK 2 passes. Verify that there are no TT's over Ref Set Threshold even once. OK TT +9,31 Friday afternoon Total Et Ref Set #0 was showing high rate (about 100 kHz) with the normal set of test check out triggers loaded during a ZB run. The other Total Et Ref Sets were also most like running Hot. Pulling the +9,31 BLS cable dropped the rates to normal. For a while, mostly because I was mixed up, there was concern that the PAL for +9,31 was dying. This was checked by pulsing +9,31 and watching on a scope the Hint outputs for EM RS0 and Tot RS0. Everything looked OK and looked the same as pulsing a different TT. Later in the day, with the standard set of test triggers loaded and normal ZB running underway all rates looked just fine. During the time that the rates looked hot there were some Calorimeter pullesers running. Cal Trig Timing Checks: As things stand the GAP signals wrt the Strobe has 2 RF Buckets of Hold time and 5 RF Buckets of setup time. The two copies of CMC Sequencer #2 Strobe and Gap (one from CMC Static Buffer and one from CMC Dynamic Buffer) are the same. Check the And-Or Term cable for AOT 128:143 from the L1 Cal Trig: AOT 128 wrt its Strobe Setup is 100 to 101 nsec Hold is 31 to 32 nsec AOT 136 looks the same. The Strobe on AOT cable 0:15 is about 4 nsec later than the Strobe on AOT cable 128:143. AOT 0 wrt its Strobe Setup is 98 nsec Hold is 34 nsec AOT 0 and AOT 128 are asserted for the same set of 3 Strobes. Conclusion is that Quadrant Region Term timing looks OK to use. Check Quadrant Region Terms Set 10.0 Gev Ref Set (EM&Tot) (other RS 100 GeV) Pulse TT And-Or Terms with Rate ---------------- ---------- -------------------------------------- Quad 0 -1,8 EM 128 136 0 8 16 20 #1 0 -1,8 HD 136 8 20 1 -1,8 EM 129 137 4 12 17 21 1 -1,8 HD 137 12 21 2 -1,8 EM 130 138 18 22 24 2 -1,8 HD 138 22 3 -1,8 EM 131 139 19 23 3 -1,8 HD 139 23 Quad 0 -8,12 EM 128 136 1 9 16 20 #2 0 -8,12 HD 136 9 20 1 -8,12 EM 129 137 5 13 17 21 1 -8,12 HD 137 13 21 2 -8,12 EM 130 138 18 22 25 2 -8,12 HD 138 22 3 -8,12 EM 131 139 19 23 3 -8,12 HD 139 23 Quad 0 +2,17 EM 128 136 2 10 16 20 #3 0 +2,17 HD 136 10 20 1 +2,17 EM 129 137 6 14 17 21 1 +2,17 HD 137 14 21 2 +2,17 EM 130 138 18 22 26 2 +2,17 HD 138 22 3 +2,17 EM 131 139 19 23 3 +2,17 HD 139 23 Quad 0 +7,32 EM 128 136 3 11 16 20 #4 0 +7,32 HD 136 11 20 1 +7,32 EM 129 137 7 15 17 21 1 +7,32 HD 137 15 21 2 +7,32 EM 130 138 18 22 27 2 +7,32 HD 138 22 3 +7,32 EM 131 139 19 23 3 +7,32 HD 139 23 Work on L1 Cal Trig Power Pan PDM-09 PDM-09 is the power pan from M104 lower Tier 1 that had its +5 Volt brick fail after a power glitch on Monday morning 8-MAR-04. From PDM-09 pull the +5 V brick which was MSU SN# 22 from 7-FEB-92 (PowerTec #1727) and install MSU SN# 25 from 7-FEB-92 (PowerTec #1725). Now PDM-09 consists of: PDM-09 +5.0 Volt MSU SN# 25 7-FEB-92 -2.0 Volt MSU SN# 25 15-MAR-91 -4.5 Volt MSU SN# 27 15-MAR-91 -5.2 Volt MSU SN# 59 JAN-92 Operate PDM-09 without load and all looks OK. Put PDM-09 in the D0 Hall spares cabinet. Bring the +5 V PowerTec brick MSU SN# 22 back to MSU. L1 and L2 TCC Operating System Patches Installed OS patch for MS04-007 (KB828028) on d0tcc1 and d0tcc2 This is to remove a vulnerability in the authentification modules. d0tcc1 (NT4) was probalby not vulnerable because it never had the MS03-041 patch applied which is what introduced the problem on NT4, as explained in the MS04-007 announcement. Install the patch anyway to avoid arguments. Note sent to Stu. L1 TCC Operating System Page File Add another pagefile to d0tcc1 on disk partition E: This is a 500 MB page file on top of the existing 128 MB pagefile on C: Diskeeper lite was used first to try and defragment the disk. The page file is still unfortunately in 888 fragments. The page file creation (at least on NT4) does not try to grab contiguous space. It simply allocates all the space available from the start of the disk. One can still do a better job, provided we have more down time. Method #1: temporarily copy all files from E: onto another disk, delete page file, reboot, recreate page file. If that still doesn't give a nearly contiguous file, one can reformat the disk. Method #2: delete page file, reboot, copy enough temporary large files to use up all the fragmented space (as seen with diskeeper lite), then create the page file. Trics does not use the page file very actively, just to store abandonned chunks of dynamically allocated memory that was not reclaimed properly. TCC probably mostly writes to the page file and reads little back. This page file fragmentation should not be an issue. Rack Power Supply Monitoring Update the Epics records for L1 Cal and L1 Framework power supplies. We had two goals. Renaming all L1FW devices into L1TFW devices to match the de-facto standard, and "enpower" all our power supply devices to cause a pause run if they get out of tolerance: 0) backup the file we are going to modify cd /online/ioc/node/d0olctl16/ cp d0olctl16.db d0olctl16.db-03-25-04 cp d0olctl16.dbg d0olctl16.dbg-03-25-04 1) Edit /online/ioc/node/d0olctl16/d0olctl16.db This generates the section of the Epics database that is downloaded into the d0olctl16 node which manages the 1553 controllers that serve MCH 1, 2, 3. 1.1) Rename all L1FW_LV_* to L1TFW_LV_* There are 4 devices appearing twice. 1.1.1) First in a section starting with file /online/ioc/templates/rm16.dbt within the lists of the full device names handled by our 2 RMI boxes. 1.1.2) Second in 3 separate sections starting with file /online/ioc/templates/lvl1*.dbt where the .dbt file is the template that actually defines the details of attributes and characteristics of each device. There is one template file for each different set of voltages in a power pan. We use a total of 4 separate lvl1*.dbt files for the L1 Cal and Fw (lvl1.dbt and lvl1c.dbt for L1CAL plus lfl1fw.dbt and lvl1fwc.dbt for L1TFW). 1.2) Change the alarm priority of all L1CAL_LV_** and L1TFW_LV_*** devices from 0 to 150 These appear in all 4 sections starting with file /online/ioc/templates/lvl1*.dbt This is the "expr" argument (standing for EXperiment PRiority!) that is passed to the .dbt files Change all occurences of "expr=0" into "expr=150" 2) Recompile the database: $ setup d0online $ setup onl_ioc $ cd /online/ioc/node/d0olctl16/ $ gmake d0olctl16.db 3) Edit the startup file for this epics node 4) reboot the epics node telnet to its terminal server port: telnet t-d0-mch3 2041 type : type : version and verify that the "boot line" shows this is indeed d0olctl16 type : The VxWorks reboots and starts its wait while it prints a lot of stuff and verify it has no problem loading the database which happens towards the end. type : in order to exit telnet 5) Change our display program to look for L1TFW_** and verify that the new record names are found. [in fact, missed one place in 1.1.1 and had to repeat.] Work area for this power supply display is /home/laurens/rackmon on d0online cluster. $ cd /home/laurens/rackmon $ setup d0online $ l1tfw.runme & $ l1cal.runme & 6) Note sent to Geoff. User "laurens" had been made part of group "onl_ctl" to be allowed to modify these files. Also cf. entry from 25:26-Nov-2003 Trics V10.5 (switched from 10.4.K) cf. 000_trics_ii_revision_notes.txt in http://www.pa.msu.edu/hep/d0/ftp/tcc/trics_ii/ Start Trics V10.5 Rev A. This version initializes the CTOM card to form the quadrant and region terms. It is also the first implementation phase of the L1 Cal Trig random test. The test appears to work correctly on Tier#1 energy inputs, but has a problem with CHTCR inputs. Run 100k loops separately on EM&HD (using lookup page 7) and PX&Py (page 2) to verify CTFE local sums at the input of Tier#1 CAT2 cards. Checking reference sets clearly has a problem as it complains about refset #1 when it is told to check #0, and #2 when it is told to check on #1. Checking all refsets #0:3 causes an access violation. Coding error found. We get no errors on checking energies, but we verify that the test indeed knows how to find errors when needed, by using VME_Access to overwrite the mux control mask (FA=81) of a CTFE and watch TTC quickly find an error for that card. This initial version of the cal trig tester does not yet check Tier#3 final sums nor counts. Start Trics V10.5 Rev B. This version correctly sets up the timing signal control on the CTOM board support function FPGA. It also fixes the above bug in L1Cal system testing of Ref Sets. The test still has problems with reference sets, but it now appears it expecting the wrong comparator output, while the hardware gives the right answer. Coding error found. Start Trics V10.5 Rev C. This version fixes the l1 cal system test expectation value for reference sets. A localized problem appears for TT(-13,2) Tot Et Ref Set #1. No errors with 100k loops on TT_eta(1:16) TT_phi(3:32) and all RefSets. No error on full TT_eta(1:16) TT_phi(1:32) coverage limited to RefSet #2:3. The problem with TT(-13,2) Tot Et RefSet#1 comparator is that it sometimes reads 0 (on the CTFE) when it should read 1. Many errors were captured in the logfile. PROM Output Comparator EM HD Threshold 175 178 163 after ~ 50k loops 116 244 163 after ~ 50k loops 154 102 118 after 17k loops 221 169 134 after 72k loops now reducing coverage to that tower only 187 163 162 187 85 101 8 149 49 8 151 82 175 151 103 Noticed that 10.5.C still has a cal trig test coding problem in its expectation value for reference sets because it uses the simulated raw EM and HD energies instead of the output of the PROMs to compare to the refset thresholds. That will cause a discrepancy only when the energies and thresholds are right around 8 counts. ------------------------------------------------------------------------------ DATE: 19-MAR-2004 At: MSU Topics: Update Rack Power Supply Display Following an example emailed by Geoff Savage, upgrade our Rack Power Supply Display Programs to use the new DevFrame objects of Fritz' onl_apps. This is a necessary upgrade to avoid the anoying warning pop-up box at program startup, and to keep up with the online environment. What we gain from this exercize is the ability to display all L1CAL devices in from one program and in one window even though some or our "epics devices" have different "epics attributes" (i.e. voltages in our case). The same thing is true for L1 Framework which can be displayed in one window. The development area is /home/laurens/rackmon $ cd /home/laurens/rackmon $ setup d0online $ l1fw.runme & $ l1cal.runme & The official area is /home/d0l1/rackmon $ cd /home/d0l1/rackmon $ setup d0online $ l1fw.runme & $ l1cal.runme & ------------------------------------------------------------------------------ DATE: 17:19-MAR-2004 At: Fermi Topics: BLS Cable repair, COMICS CMC down load, ADF integration test, Work on isolation of SCL Link to L2 Test Stand, Talked with Shoua and he will get started on repair of 4 BLS cables: MCH-1 end shorts -19,30 HD +13,23 EM +14,11 EM platform open +16,8 EM. L1 Cal Trig has been turned off and locked out. Shutdown the Master Clock Sequencer #2 so that Fritz could practice COMICS download on it. I need to verify that the L1 Cal Trig readout in M101 gets its timing from Sequencer #1. Yes, its all OK. Put the L1 Cal Trig Cold Start procedures on the web in the same directory as the power control procedure for the L1 Cal Trig. Swapped the L2 De-Mux card with its spare after Terry loaded the current 10-DEC-2002 firmware onto it. It is now screwed into the crate. Both cards have tags labeling them, their dates of service, and their firmware date. ZB is running fine with this De-Mux card and Terry is going to do the 128 trigger test. Visited with Ted. Picked up the SCL optical link isolation boxes. John is going to add covers over the exposed 120 V AC before we put them into service. Sent note to John,Tom, and Lin about this and the patch cables. Talked with Ted about managing incremental FPGA changes. Run IIB integration test. Fiber optic output from the Columbia card does not match the VTM input. No test attempted. Send note about the VRB VTM and Finisar documents and URL's. 48 V power supply 4A provided by John & Tom. ------------------------------------------------------------------------------ DATE: 8-MAR-2004 At: MSU action at Fermi Topics: L1 Cal Trig Power Pan Replaced. Early Monday AM March 8th a site wide power glitch tripped off much of D-Zero. Sections that lost power included the MCH. When power was restored and the L1 Cal Trig restarted, the Lower M104 Tier 1 power pan's +5 Volt brick did not start up. There was not a panic to replace this power pan because we were not in store nor was there likely to be one soon. Victor Martinez and John Foglesong took care of the power pan replacement. PDM-09 was pulled and PDM-08 was installed. When cold starting the L1 Cal Trig I forgot to initialize the eta coverage that is not used for generating L1 Cal Trig's. That caused trouble for the first couple of hours of the next store. ------------------------------------------------------------------------------ DATE: 3:6-MAR-2004 At: Fermi Topics: Toroid Magnet Noise, TFW Power Supply Most work during this trip was on the Toroid Magnet noise problem. Scope pictures from this are in the Cal Trig Pictures web page. We had the M123 middle power supply fail this past Sunday evening Feb 29th at about 9 PM. The brick with the +5, -2, -4.5 failed. The failed supply is Chassis SN #4. I will bring this Run II TFW power supply chassis back to MSU for repair. I got our other TFW power supply chassis back from the L2 people up on the 2nd floor. It is now down here in the normal storage place (behind the chair by the SUN tube) ready to use if needed. This is TFW power supply chassis SN #2. I test ran it without load and it is happy. SN #2 is now the spare at Fermi that is ready to go in if needed. ------------------------------------------------------------------------------ DATE: 25:27-FEB-2004 At: Fermi Topics: Run IIB All work during this trip was on Run IIB. On Friday morning during ZB running the VRBC in the TFW readout failed to kick the SCL Receiver and that readout crate hung without timing/trigger information. I have not seen this happen before in that crate. ------------------------------------------------------------------------------ DATE: 20-FEB-2004 At: MSU TOPICS: Restart Trics Restart Trics, Trigmon, and TTADCmon which brings pagefile usage from ~90% down to 15%. 1) Scan the logfile TRICS_II_20040205_V10_4_K.LOG;1 which covers 15 days of running, while COOR sent us some 31,300 messages and the increased pagefile usage was 90-15= 75%. 2) Scan the previous logfile TRICS_II_20040127_V10_4_K.LOG;1 which covers 9 days of running, while COOR sent us ~21,000 messages and the pagefile usage was 60-16 = 44% (cf. entry for 5-FEB-2004). 3) Scan the previous logfile TRICS_II_20031123_V10_4_J.LOG;1 which covers 10 days of running, while COOR sent us ~28,500 messages and the pagefile usage was 77-15 = 62% (cf. entry for 2-Dec-2003). 4) Scan the previous logfile TRICS_II_20031114_V10_4_J.LOG;1 which covers 9 days of running, while COOR sent us ~39,700 messages and the pagefile usage was 75-20 = 55% (cf. entry for 25-Nov-2003). 5) Scan the previous logfile TRICS_II_20030814_V10_4_H.LOG;1 which covers 14 days of running, while COOR sent us ~28,000 messages and the pagefile usage was 100-15 = 85% (cf. entry for 30-Aug-2003). The current understanding is that the memory leak is solely related to parsing COOR messages. There is no direct proportionality with time, or with number of messages, between these 5 measured instances. The pagefile usage is not an acurate direct measurement of the memory leak, as the memory usage remains "hidden" for a while behind the process' working set. This means we should expect an offset. The first (and most recent) 3 points indeed line up. They form a line with a x-intercept around 7,000 messages, and a slope of about 460 messages per percent of pagefile. The 4th and 5th points are way off and on opposite sides of the line. The memory leak is believed to come from the step that turns command lines into tokens, and this effect is thus linked to how complex the messages are. It is believable that the "average message" is quite different and more complex during physics running than during commisisoning. Data points #1,2,3 (which form a line) were during physics running. Data point #4 is below the line (lower leak rate) and was a period of commissioning. Data point #5 is above the line (higher leak rate) and was a period of physics running, but separated by 6 months from points 1,2,3. ------------------------------------------------------------------------------ DATE: 10-FEB-2004 At: MSU TOPICS: De-Exclude only EM of TT (+8,7) Trics trick to De-Exclude a TT We have returned the EM half of Trigger Tower (+8,7) to its normal L1CAL triggering operation, while the HD half of (+8,7) is still excluded from participating in any L1CAL trigger decision. We also looked at the L1CAL pedestal widths for the 32 Trigger Towers at TT_Eta=+8 and we do not see any noticeable oddity at TT_Phi=7, neither for EM nor HD. Additionally, we do not see any significant change while comparing to the width of these pedestals as measured when Dan ran our analysis tool 5 days ago. In this particular instance, the noise at TT(+8,7) does not seem to affect the pedestal seen by L1CAL, which is consistant with bursts of noisy activity during physics running. We De-excluded this trigget tower on the fly, and without re-initializing the L1Cal Trig: INCORRECT shortcut to "DE-exclude" a TT without re-initializing L1CAL: You cannot simply overwrite the Mux-Latch Clock Control Register of a CTFE at FA=81 with the default value 0xff and re-run the Exclude TT file. Trics will complain with a red error message about the register content having changed, and, more importantly, ultimately restore the old value. PROPER shortcut to "DE-exclude" a TT on the fly (e.g. during a ZB run): 1) run Find_DAC on the trigger tower(s) that need de-excluding 2) re-load the ped DACs (e.g. execute MCF Load_L1Cal_GainsPeds.mcf) 3) re-exclude the trigger towers (execute MCF Excluded_Trigger_Towers.msg) ------------------------------------------------------------------------------ DATE: 9-FEB-2004 At: MSU TOPICS: Exclude EM & HD for noisy TT (+8,7) The captain's 8am logbook entry says: "...there seems to be a hot tower developping. The L1 rate jumped by 150-200 Hz. The first time the problem fixed itself, but now it remains high." Run the Tool "List Hot TT (RS#0)" and histograms of 1000 entries find HD(+8,7) 80 times above threshold while a handful of the rest of the towers appear with only one entry. Looking in the Examine plots that the CalMuon Shifter put in the log book this morning at 8:22, TT +8,7 looked "hot" At the request of the CalMuon Shifter and with the agreement of the Shift Captain we Excluded Trigger Tower +8,7 from participation in the L1 Cal Trig. The file Excluded_Trigger_Towers.msg now excludes both EM and HD for (+8,7) P.Petroff email: " In precision readout :the noisy tower is 7/1/5/3 (ieta=16 iphi=13) and more precisely layer 11 (FH1). For this layer: in 0 bias mode: mean E = 2.5 Gev and occ = 22%. It looks that the hadronic TT should be affected only ?" ------------------------------------------------------------------------------ DATE: 5:7-FEB-2004 At: Fermi Topics: Cal Trig work Thursday after the BLS was repaired I made a run of Find_DAC over EM, HD, +-, 1:16, 1:32, 2, 8, check, check: This resulted in the file: D0_Log\Find_DAC_V5_0_F_20040205.tti;1 I looked at the deviations in the TT area where the bad BLS card/crate had been and things look mostly OK. The Deviation of +5,22 EM was bigger than its neighbors, but its HD looks OK, and the other 15 TT's look OK, so for now I hope it is OK. TT -4,25 HD had an Error Tag caused by a Deviation of 3.14 which is about 3x its neighbors. I edited the tti file to comment out the error tag and to crank down the pedestal on -4,25 HD a notch or two. This made the file: D0_Log\Find_DAC_V5_0_F_20040205_Edited.tti;1 With this file loaded, TT_ADC_Mon looks OK so I will try the next store with it. Edit \D0_Config\Load_L1Cal_GainsPeds.mcf to call D0_Log\Find_DAC_V5_0_F_20040205_Edited.tti;1 and verify that it loads OK. Etiquette for using TRICS Window location etiquette: On the right is the TRICS log file. In the center is the Trigger Monitor (TrgMon) with the mouse focus. On the left is the TRICS Main Menu. The VME IO Command File that should be selected is: D:\TRICS\D0_Config\Init_Post_Auxi_L1CT.vio In the Configure FPGA's menue: The "ECL Output Enabled" box must be checked. Selected the file: D:\TRICS\D0_Config\M101_Routing_Master_All.dcf In the Control Status Menu: check the "tti file loads DAC's" box. Watching a store: Luminosity of 56 to 46 in one hour. Luminosity of 45 to 37 in 2 hours 45 min. Cal Muon log book Saturday morning. There is the know sparky cell at 12/32/17 (CH3). It was an occupancy of 49%, a mean energy of 2.1 GeV and an RMS of 7.7 GeV. ------------------------------------------------------------------------------ DATE: 5-FEB-2004 At: MSU TOPICS: De-exclude BLS card input Robert Z. calls MSU around 11am, after the access to replace the BLS card. We can remove the 16 trigger towers from the Excluded_Trigger_Towers.msg file. At this time, we just comment them out, and will remove the lines from the file once t he fix is proven to be successful. Philippe takes advantage of this oportunity to restart Trics, as the DAQ expert needs to re-initialize the L1FW+L1CT. Restarting Trics brings the pagefile usage from ~60% down to 16%. We copy and archive the January logfiles. Right before restarting Trics, Philippe enables the "Log Monitoring Requests" option in the control/status menu. This is a fairly recent feature which had never been used at DZero. This generates the following messages in the logfile: at 05-Feb-2004 11:04:40.831 I$ @ requested Block_Type_Fw_Info_Brief I$ @ requested Block_Type_L2HardScalers I believe the "Scraper" is the alternate Dzero monitoring process which feeds Gordon's DAQ AI and Reinhard's L2 monitoring display. DAQMON_Scraper requests two blocks of data: the general summary of Framework information and the L2 Hardware Scalers counts. at 05-Feb-2004 11:04:42.424 I$ @ requested Block_Type_Fw_Info_Brief I$ @ requested Block_Type_L2HardScalers I believe this is Taka's official "DAQ Monitor" server. DM_Server also requests the same two blocks of data. at 05-Feb-2004 11:04:42.774 I$ sending Block_Lum_Brief to @ This is TCC sending a summary of per bunch scaler information to Michael's Luminosity server. Note that TCC __pushes__ this information every ~5 second (as well as a Block_Lum_Full every minute, and at begin/end run, etc). All other monitoring services are on a __pull__ basis where the client has to explicitely request the information. at 05-Feb-2004 11:04:44.867 I$ @ requested Block_Type_Fw_Info_Brief I$ @ requested Block_Type_L2HardScalers It looks like the DAQMON_Scraper asks for new data every 4 sec. at 05-Feb-2004 11:04:52.418 I$ @ requested Block_Type_Fw_Info_Brief I$ @ requested Block_Type_L2HardScalers It looks like DM_Server asks for new data every 5 sec (10 sec here, because we missed one inbetween). at 05-Feb-2004 11:04:52.879 I$ @ requested Block_Type_Fw_Info_Brief This is the process on MSUL1A collecting the data for Trigmon on the Web. This was a very short test, time limited by the need to restart Trics and return the system to the DAQ expert. This still verified that this new feature had no catastrophic effect on Trics, and already gives an idea of how the monitoring system is used and by which online processes. There was no surprise here, but we should collect this data over a longer period of time (e.g. 10 mn) to get a more complete picture. ------------------------------------------------------------------------------ DATE: 4-FEB-2004 At: MSU TOPICS: exclude whole BLS card input Dan reads in the log book this morning about the need to remove a BLS board from the Cal Precision Readout. L1 and L2 rates are still too high for the current prescale set and luminosity level. Using the "List TT above RSs" diagnostics tool, Dan sees a hot Trigger Tower at TT eta,phi +5,22. During the switch from one Physics Run to the next We exclude both the EM and HD parts of TT +5,22 from taking part in the L1 Cal Trig decision. The effect is to lower the L1 trigger rates to a more typical level. TT +5,22 is indeed on the BLS card that had to be removed from the Cal Precision Readout this morning. However the problem with this BLS card appears to be jerking the power supply voltage which in turn induces an apparent energy in the trigger pick-off signal from the other TT serviced by this card. We eventually Exclude both the EM and HD sections of the following 16 TT's corresponding to this BLS crate: +5,22 +6,22 +1,21 +2,21 +3,21 +4,21 +5,21 +6,21 +1,20 +2,20 +3,20 +4,20 +5,20 +6,20 +1,19 +2,19 These TT's were Excluded by editing the file Excluded_Trigger_Towers.msg that controls this function of the L1 Cal Trig. Thus these TT's will continue to be Excluded even if the L1 Cal Trig is Initialized. As soon as the access is made and the BLS card is replaced, we will be contacted to remove the Exclusion of these TT's from the control file. ------------------------------------------------------------------------------ DATE: 28-JAN-2004 At: MSU Topics: L1 Cal Trig Readout VRBC Hangs Wednesday Jan 21st at about 5:45 PM and then again on Thursday Jan 22nd at about 4:30 PM we had hangs of the VRBC in the L1 Cal Trig crate 0x10. We were not there to look at it but we assume that these were the "standard" SCLR sync loss and VRBC not telling it to re-sync problem. In both cases people were working at the L2 Test Stand at the time. It is assumed that this is an example of what Miroslav has observed, i.e. certain plugging un-plugging turning things on/off work at the L2 Test Stand can cause other SCL links to loose sync. It is assumed that this is a problem because these links cross the ground systems from MCH-Platform to Building. The written procedure that the DAQ Shifter uses to fix this problem is a full Initialize of TFW and L1 Cal Trig. It has recently be verified that this procedure is in their documentation. Because L2 needed to do some Test Stand work during beam we passed to Miroslav the following "fast" procedure for kicking the VRBC. I will include that procedure here because Miroslav is now familiar with it. Philippe points out that a cleaner way to do this may be to use the HSRO Test menu. I need to test that method on the real hardware and then write a procedure for it. The phone number of the L2 Test Stand Desk is x8645. Procedure to Reset the L1 Cal Trig VRBC 1. Tell the DAQ Shifter to "Pause" the run. There is NO need to "stop" the run or to "release the triggers". 2. Go into the MCH-1 to the console for TCC1. There will be 3 open windows on the screen. On the right is the TRICS log file. In the center is the Trigger Monitor (TrgMon) with the mouse focus. On the left is the TRICS Main Menu. 3. Move the mouse focus to the TRICS Main Menu by clicking on some "non active" part of that menu. 4. Now in the TRICS Main Menu - click on the "VME IO" button which is on the left hand side of this menu near the top. 5. Now in the VME IO Menu you will see that the section at the bottom is a called "VME IO Command File. The VME IO Command File that will have already been selected is: D:\TRICS\D0_Config\Init_Post_Auxi_L1CT.vio 6. In the VME IO Command File section of this menu click the "Execute" button. 7. It will take about 2 seconds for this command file to execute. I would then wait about 5 seconds and click the Execute button again. That is, reset the VRBC twice, to make certain that it gets reset. 8. Now at the very bottom of this menu click the "Done" button. Doing that will take you back to the TRICS Main Menu. 9. Return the mouse focus to the TrgMon window, i.e. the center window that is "safe" because it can not do anything to change the running triggers. 10. Tell the DAQ Shifter to "Resume" the run and verify that L3 monitoring is now happy and getting data from crate 0x10. ------------------------------------------------------------------------------ DATE: 15:16-JAN-2004 At: Fermi Topics: Cal Trig work on -10,11 CTFE PAL's CHTCR pull parts, Adjust some Pedestals by hand There has been no change to the Cold Start instructions but I put a copy of them at the end of this log book entry so that we have a copy in the 2004 log book. For a review of the Term-Attn-Brd gains, before the fall shutdown, since the fall shutdown, and theoretical see the log book for 15-SEPT-2003. Thursday during shot setup start looking at what is going on in the CHTCR for eta -9:-12 phi 9:16. Loaded the standard test setup of Ref Set's and Count Thresholds. Watch the rate of the Total Et Ref Set 0 (5.0 Gev) 1 TT over threshold And-Or Term with -10,11 BLS Cable plugged in and pulled out. This is during constant normal ZB running with SMT noise. Plugged In Pulled Out ---------- ---------- 42.1 kHz 19.2 kHz 42.8 kHz 19.2 kHz 40.6 kHz 19.2 kHz 48.2 kHz 19.3 kHz 42.9 kHz 19.7 kHz 43.3 kHz 19.2 kHz I did this test a couple of times separated by 5 or 10 minutes. This is about what was seen last Saturday but then I only knew that the rate was coming from somewhere in eta -9:-12 phi 9:16. With -10,11 cable pulled out you can see the CHTCR Tot_Et Ref_Set_0 Hint LED make just a very dim constant glow. With it plugged in you see the dim glow but also short periods of bright flash. ?? check other CHTCR for glow ?? OK the -10,11 BLS signal must be screwed up and pick up an extra portion of SMT noise. Does not look like it. Measure pp SMT noise. -10,9 EM 40 mVpp HD 190 mVpp no and-or rate change when cable pulled -10,11 EM 40 mVpp HD 125 mVpp big snd-or rate change when cable pulled -10,16 EM 28 mVpp HD 145 mVpp no and-or rate change when cable pulled Friday 16th Restart TRICS With everything back tied up as normal and the standard set of test trig's loaded into the L1 Cal Trig I see no excess rate from 1 TT over EM Ref Set #3. I do see the "normal" large fraction of SMT noise Tot Et Ref Set #0 Trigs coming from TT -10,11. Super check this BLS signal and it looks OK. The lemo output on the -10,11 CTFE looks OK. At worst there may be a very very low rate HD noise pulse. SMT noise pickup is just "normal". Run the CHTCR PROM test on eta -9:-12 phi 9:16. EM Ref Set #0 and #1 pass OK but all 6 other Ref Sets fail the CHTCR PROM test. With help from Philippe the test is indicating trouble with the CHTCR input signals from TT -10,11. I pulled the CHTCR for eta -9:-12 phi 9:16 to make certain that it looked OK. This CHTCR from M108 is SN# 36. Verified that the socketed latches all looked OK, and the 29525's were OK, and the PROM's were OK. I did what I want to start doing anytime a CHTCR is pulled out for some reason (which is not very often) - I want to pull off from it the socket parts that are just used for the Table Builder TBus stuff. We do not use that logic at all. Plug the CHTCR back in and it still fails the CHTCR PROM test in exactly the same way. Pull the CTFE for eta -9:-12 phi 11. It is SN# 343. At some point in the past it has been wet but it looks OK today. The PAL's on it are type. AMD PALCE-22V10H-25. All the spare parts are MMI AMD PALC-22V10. The only type that this programmer knows about are the AMD parts. The CTFE_2 type PAL's (the one from this card and a spare one) do not check against each other in the programmer - but they do verify OK agains other CTFE_2 type PAL's of the same manufacturer. Give up and replace the PAL and now the CHTCR PROM test runs OK. Run CHTCR PROM Test on eta -9:-12 phi 9:16 many times and it is now OK. Run full eta 1:16 CHTCR PROM Test and it is OK. Run full eta 1:16 EM and HD CTFE PROM Test on page 7 and it is OK Run full eta 1:16 Px and Py CTFE PROM Test and it is OK except for the known Px blow up at +13,27 +14,27 +15,27 +16,27 (assumed CAT2 readback problem - but I need to fix this). See 11:15-Nov-2003 and 25:27-JUNE-2003 and 9:13-JUNE-2003 and 8:11-JULY-2003. Run CHTCR PROM Test on eta -9:-12 phi 9:16 many times and it is now OK. Check the failed AMD PALCE-22V10H-25 on the Data I/O model 2700. When setting up the Data I/O for this part type it said to read application notes 983-0358 and 983-0372 for information about vector testing these parts (which is not what I'm doing today but may explain a mestery from 10 years ago). The check sum of a spare MMI part is $d30a. Checking agains another spare MMI part it does verify. Checking against the part pulled from the CTFE card for eta -9:-12 phi 11 and you get, "Device Problem 6027, Device Verify Error". This pulled AMD part has a check sum of 5832. But this "failed" pulled AMD part does check agains a spare AMD part. I'm very confused. But even if the readback check was OK it does not mean that this PAL does the logic correctly. The other times these PAL have died I do not think that I could "fix" them by reprogramming them. Should check logs. Poking through some AMD parts of uncertain quality I get for check sums: CTFE_1 PAL 58aa, CTFE_2 PAL 5832, CTFE_3 PAL 57ea, CTFE_4 PAL 57f6 During ZB running just before the shot, the and-or rates did look OK with the normal set of test triggers loaded. The rate for Total_Et Ref_Set 0 was at 19 or 20 kHz, i.e. it looks like what it did yesterday with -10,11 BLS pulled out. L1 Cal Trig Examine still looks a little un symmetric for HD i.e. the neg eta looks funny when compared to the pos eta. It's only slowly sinking in on my that these folks are running a 3.0 GeV Tot Et Ref Set. Good Luck. From the projection plots, the worst places looks like: eta's -6, -8, -10, and -11. It is especially bad for eta's -6 and -11. phi's 7, 9, 12, 24 especially bad for phi 12. I noticed earlier today that there are some HD pedestals a little off in this area, i.e. up at 10 or 11 or 12 counts. I'm going to edit the peds file and crank them down. Specifically: -6,1 is up at 10 counts, crack its DAC up by 30, 3598 --> 3628 -6,7 is up at 11 counts, crack its DAC up by 45, 3698 --> 3743 -7,6 is up at 10 counts, crack its DAC up by 30, 3659 --> 3689 -7,26 is up at 10 counts, crack its DAC up by 30, 3543 --> 3573 -7,27 is up at 10 counts, crack its DAC up by 30, 3653 --> 3683 -8,24 is up at 10 counts, crack its DAC up by 30, 3722 --> 3752 -9,21 is up at 10 counts, crack its DAC up by 30, 3599 --> 3629 -10,12 is up at 10 counts, crack its DAC up by 30, 3701 --> 3731 -10,24 is up at 10 counts, crack its DAC up by 30, 3722 --> 3752 -10,25 is up at 10 counts, crack its DAC up by 30, 3777 --> 3807 -11,11 is up at 10 counts, crack its DAC up by 30, 3668 --> 3698 -11,12 is up at 12 counts, crack its DAC up by 60, 3718 --> 3778 -12,9 is up at 10 counts, crank its DAC up by 30, 3677 --> 3707 Recall it takes an increment of 13.109 counts of the Pedestal DAC to move the Flash ADC down by 1 count. In theory with all the analog parts just perfect it should require 3614 in the Ped DAC to make $08 come out of the Flash ADC. Call the new eta 1:16 Pedestal File: \D0_Log\Find_DAC_V_5_0_F_20040109_Edited.tti;1 Edit \D0_Config\Load_L1Cal_GainsPeds.mcf to use this new eta 1:16 ped file. These new Peds were loaded in Friday night between Physics runs at about 22:00. Yes, this makes a difference in Bob's L1 Cal Trig Examine plots. OK, I need to start watching the Pedestals more closely. Socket Parts pulled from the CHTCR Card These are the socketed parts that are pulled from the CHTCR card. All of this stuff was for the Table Builder TBus stuff. This is all the socketed parts along the edge of the card that has the Table Builder TBus stuff. Pulling this stuff does not effect the main section of the card that we are using. By pulling this stuff off we cut the load on the +5 Volt supply by about 20%. "No Issues" means that I have checked the schematics and verified that the gates in this package are not split between the CHTCR's main function and Table Builder stuff (i.e. all gates in this package are part of Table Builder stuff) and that pulling this package does not leave some other inputs floating that will cause trouble. G1 Crystal Oscillator 10 MHz U119:U122 29525's - no issues U123 74AS151 - no issues U124 74AS869 - U125 a 74AS138 will now have 3 open inputs U126, U127 PROM's never had been installed U128 74AS869 - no issues U129 74AS08 - no issues U130 74AS32 - no issues U131:U134 29525's - some 10H124 translators will now have open inputs. U135 74AS74 - no issues U136 74AS32 - no issues U137 74S85 - no issues U153 74ALS541 - no issues - good it removes load on the CBus Data Bus U154 74AS74 - no issues Store that starts Thursday afternoon is store number 3172. The first Physics run is run number 188258. The last Physics run of store 3172 was 188263. This store had -9:-12 9:16 out of EM RS3 New store Friday afternoon, now with all eta,phi running in all Ref Sets. It is store number 3175 first Physics run is 188292 Cold start instructions Rev. 31-DEC-2003 We are running: TRICS V10.4.K See the full instructions at: www.pa.msu.edu/hep/d0/ftp/l1/framework/hardware/rack_crate/ master_clock_instructions.txt www.pa.msu.edu/hep/d0/ftp/l1/framework/hardware/rack_crate/ framework_power_control_procedures.txt www.pa.msu.edu/hep/d0/ftp/l1/cal_trig/hardware/rack_crate/ cal_trig_power_control_procedures.txt Overview: Only after Master Clock is running and loaded, then turn on TFW and then the L1 Cal Trig (one power supply at a time) then: Configure FPGA's. Use the normal TRICS "Master Command File" menu for this. It configures all FPGA's except for the M101 Routing Master. Get the Routing Master FPGA's Configured and Running 1) After the Routing Master is powered up, wait for its SBC to boot (less than 1 minute) 2) Login to the SBC with 'ssh d0runsu@d0sbc001b' The password is same as the online d0run account. 3) On the SBC run 'reset_all.sh stop' which will stop the relevant readout processes. 4) Use TCC's Configure FPGA's menue to execute the dcf that configures the Routing Master FPGA's. This file is D0_Config\M101_Routing_Master_All.dcf Verify that the "ECL Output Enabled" box is checked before executing this dcf file. If you see an error on the first attemp to configure these FPGA's try it again. It must say zero errors. 5) On the SBC run 'reset_all.sh start' Use the TRICS "System Control Status" menu to: Set the L1 Cal Trig Eta Coverage to 1:20 Check the "TTI File Loads DAC's" box Do a Full Initialize of TFW and L1 Cal Trig Use the TRICS "Master Command File" menu "Run a Master Command File" option to select and then execute the file \D0_Config\Load_L1Cal_GainsPeds.mcf This loads the current Gains and Pedestal files into the L1 Cal Trig. Use the TRICS "System Control Status" menu to: Set the L1 Cal Trig Eta Coverage to 1:16 Do a Full Initialize of the TFW and L1 Cal Trig verify that it is a clean init of all parts. For reference only, the current (16-JAN-2004) Gains Ped's files are: D0_Config\Gains_17_20_1_32_rev_a.tti # TT eta's 17:20 D0_Config\Gains_1_16_1_32_rev_5.tti # TT eta's 1:16 D0_Log\Find_DAC_V5_0_F_20031212_Edited.tti;2 # TT eta 17:20 D0_Log\Find_DAC_V5_0_F_20040109_Edited.tti;1 # TT eta 1:16 ------------------------------------------------------------------------------ DATE: 12-JAN-2004 At: MSU Topics: Background information from log files about eta -9:-12 phi 9:16 and a look at how COOR loads Global_CMT-12.32 Philippe dug the following background information about eta -9:-12 phi 9:16 out of various log files. From Thursday evening 08-Jan-2004 at around 19:18 Using the tool button "List TTs above RSs", all TT in the triggering coverage were scanned 6 times, all 2x4 Reference Sets were checked (as this tool is aimed at finding hardware problems) and this test tool found (-10,11) 3 times over threshold for Tot Et Ref Set #1. I$ Scanning All TTs in L1CT Coverage -> List TTs above Current RefSet Threshold I$_Over current L1CT Coverage eta(-16:+16),phi(1:32) E$ TT Comparator found Asserted at TT(-10,11) Tot Et RefSet #1 From Thursday evening 08-Jan-2004 at about 19:06: The Reference sets were programmed this way (not monotonic in energy order) The histograms from the last Find_Dac Run of 12-Dec-2003 show nothing weird for EM and HD (-10,11). Collecting some histograms during ZB run may or may not be useful, but it would still be interesting to know what they look like. The tool button "List Hot TT (RS#0)" allows to set the size of the data sample. This tool only checks one EM and one Tot Et RefSet (as this tool is aimed at finding input signal problems). If you decrease the Cal Trig covereage in Control/Status, you can crank up the sample size, and list the frequency of towers above RS#0. You need to set the RefSet #0 Threshold manually before running the test. This should do a qualitative and geographic evlauation of the SMT noise problem. On 20-Dec-2003 at about 9:17 the Find Hot Tower tool made 3 sweeps. -10,11 was not the only contender, but it did show up along with some neighbours. I$ Scanning All TTs in L1CT Coverage -> Look for Hot TTs above Ref Set #0 I$_Looking for TTs above EM Et Ref Set #0 I$_Looking for TTs above Tot Et Ref Set #0 I$_Over current L1CT Coverage eta(-16:+16),phi(1:32) I$_Taking 1000 samples from each CHTCR input I$_Estimated Diagnostic Run Time = 0.62 seconds I$ Monit Data -> L1 Accept= 0x00d93f83 L1 Rate=913.7 Hz Buf Dpth= 0 E$TT(- 3,10) was 1 time(s) above EM Et RefSet #0 E$TT(- 3,12) was 1 time(s) above EM Et RefSet #0 E$TT(+10,12) was 3 time(s) above Tot Et RefSet #0 E$TT(- 1,10) was 1 time(s) above Tot Et RefSet #0 E$TT(- 5,28) was 1 time(s) above Tot Et RefSet #0 E$TT(- 5,30) was 1 time(s) above Tot Et RefSet #0 E$TT(- 6, 7) was 1 time(s) above Tot Et RefSet #0 E$TT(- 7, 7) was 2 time(s) above Tot Et RefSet #0 E$TT(- 8,15) was 1 time(s) above Tot Et RefSet #0 E$TT(- 9,12) was 1 time(s) above Tot Et RefSet #0 E$TT(-10,11) was 4 time(s) above Tot Et RefSet #0 <---- I$ Monit Data -> L1 Accept= 0x00d9c2ce L1 Rate= 20.5 Hz Buf Dpth= 0 E$TT(+ 8,17) was 1 time(s) above EM Et RefSet #0 E$TT(+ 8,18) was 1 time(s) above EM Et RefSet #0 E$TT(+ 8,21) was 1 time(s) above EM Et RefSet #0 E$TT(+ 8,22) was 1 time(s) above EM Et RefSet #0 E$TT(- 4,17) was 11 time(s) above EM Et RefSet #0 E$TT(- 4,18) was 11 time(s) above EM Et RefSet #0 E$TT(- 4,19) was 10 time(s) above EM Et RefSet #0 E$TT(- 4,20) was 12 time(s) above EM Et RefSet #0 E$TT(- 4,21) was 11 time(s) above EM Et RefSet #0 E$TT(- 4,22) was 4 time(s) above EM Et RefSet #0 E$TT(- 4,23) was 12 time(s) above EM Et RefSet #0 E$TT(- 4,24) was 6 time(s) above EM Et RefSet #0 E$TT(-10, 1) was 11 time(s) above EM Et RefSet #0 E$TT(-10, 2) was 7 time(s) above EM Et RefSet #0 E$TT(-10, 3) was 6 time(s) above EM Et RefSet #0 E$TT(-10, 5) was 10 time(s) above EM Et RefSet #0 E$TT(-10, 6) was 6 time(s) above EM Et RefSet #0 E$TT(-10, 7) was 6 time(s) above EM Et RefSet #0 E$TT(-11,10) was 1 time(s) above Tot Et RefSet #0 E$TT(-11,11) was 1 time(s) above Tot Et RefSet #0 E$TT(-11,14) was 1 time(s) above Tot Et RefSet #0 E$TT(-11,15) was 1 time(s) above Tot Et RefSet #0 I$ Monit Data -> L1 Accept= 0x00d9c7dd L1 Rate=190.0 Hz Buf Dpth= 0 E$TT(-11,17) was 1 time(s) above EM Et RefSet #0 E$TT(-11,20) was 1 time(s) above EM Et RefSet #0 E$TT(-11,21) was 1 time(s) above EM Et RefSet #0 E$TT(-11,24) was 1 time(s) above EM Et RefSet #0 E$TT(+ 1, 4) was 1 time(s) above Tot Et RefSet #0 E$TT(+ 1, 6) was 1 time(s) above Tot Et RefSet #0 E$TT(+ 6,22) was 1 time(s) above Tot Et RefSet #0 E$TT(- 9, 9) was 1 time(s) above Tot Et RefSet #0 E$TT(- 9,10) was 1 time(s) above Tot Et RefSet #0 E$TT(-10,12) was 2 time(s) above Tot Et RefSet #0 E$TT(-11,12) was 1 time(s) above Tot Et RefSet #0 Philippe also captured a typical load of Global_CMT-12.32 into the L1 Cal Trig. From TRICS_II_20031219_V10_4_K.LOG;1 %%08-Jan-2004 19:06:04.035 L1CT_Ref_Set EM_Et_Ref_Set 0 Value 3.0 L1CT_Ref_Set EM_Et_Ref_Set 1 Value 6.0 L1CT_Ref_Set EM_Et_Ref_Set 2 Value 11.0 L1CT_Ref_Set EM_Et_Ref_Set 3 Value 9.0 L1CT_Ref_Set TOT_Et_Ref_Set 0 Value 5.0 L1CT_Ref_Set TOT_Et_Ref_Set 1 Value 3.0 L1CT_Ref_Set TOT_Et_Ref_Set 2 Value 7.0 L1CT_Count_Threshold EM_Et_Towers Ref_Set 0 Comparator 0 Value 1 L1CT_Count_Threshold EM_Et_Towers Ref_Set 1 Comparator 0 Value 1 L1CT_Count_Threshold EM_Et_Towers Ref_Set 2 Comparator 0 Value 1 L1CT_Count_Threshold EM_Et_Towers Ref_Set 3 Comparator 0 Value 1 L1CT_Count_Threshold EM_Et_Towers Ref_Set 0 Comparator 1 Value 2 L1CT_Count_Threshold EM_Et_Towers Ref_Set 1 Comparator 1 Value 2 L1CT_Count_Threshold Tot_Et_Towers Ref_Set 0 Comparator 0 Value 1 L1CT_Count_Threshold Tot_Et_Towers Ref_Set 1 Comparator 0 Value 2 L1CT_Count_Threshold Tot_Et_Towers Ref_Set 2 Comparator 0 Value 4 L1CT_Count_Threshold Tot_Et_Towers Ref_Set 0 Comparator 1 Value 2 L1CT_Count_Threshold Tot_Et_Towers Ref_Set 1 Comparator 1 Value 1 L1CT_Count_Threshold Tot_Et_Towers Ref_Set 2 Comparator 1 Value 1 L1CT_Count_Threshold Tot_Et_Towers Ref_Set 0 Comparator 2 Value 3 L1CT_Count_Threshold Tot_Et_Towers Ref_Set 0 Comparator 3 Value 4 ------------------------------------------------------------------------------ DATE: 8:10-JAN-2004 At: Fermi Topics: Cal Trig: List of Needed Cable Repairs, Check some TT's, Pull eta -9:-12 phi 9:16 from EM Ref Set #3. Brought a spare Distributor Cap here. I think it has all the fall 2003 modifications but it has not been tested run in the system or visually checked agains another Distributor Cap. It does have all the new/current PROM's and PAL's installed. It is DC SN#5. We now have 8x ERPB's and 1x DC at Fermi. Spent another couple or hours with Victor and John looking at / talking about L1 Cal Trig power supply replacement. The new plan is that if we need to replace a pan at 4AM they will both come in. Thursday I sent a note to Shoua and Tom Regan about repair work on 4 BLS cables. MCH-1 Shorts on -19,30 HD +13,23 EM +14,11 EM and a Plotform Open on +16,8 EM. Thursday night, -10,11 blows a gasket. It has added 200 Hz to the low threshold triggers. It looks like it is the EM section of -10,11 that is the problem. Exclude it and the rates return to normal and are stable. Thursday night +3,27 HD looks hot in the L1 Cal Trig Examine but it is not effecting the rates. Look at TT's on Friday Morning: Look at -3,21 -4,21 -5,21 because one of these (most likely -4,21) just started showing up as bad in Robert's plots. The plot shows the -4,21 HD at only about 0.5 in the Trig/Prec plot. Look at these BLS signals and they look OK. -4,21 HD may have had a little less noise than the others. I can see the line locked noise on -3,21 EM and -4,21 EM - did not look for it on -5,21. Hand Pulser test these 6 CTFE channels and they look OK. -4,21 has no history of a problem. -- Another note from Robert says that -4,21 is back looking happy again. -10,11 The EM and HD of -10,11 look OK. I'm NOT going to Exclude it from the next store. It looks OK on the scope. It was -10,11 EM that blew up last night. +3,27 HD has looked noisy in the L1 Cal Trig Examine plots. It looks noisy on the scope. Every few seconds it has a pulse of a hundred mV or more. The pulse looks more like signal than spark but it does not look exactly like signal. +3,27 does not have a history for problems. In the Friday morning run of Find_DAC the ped width of all these TT's looked OK. Friday morning run of Find_DAC EM HD + - eta 1:16 phi 1:32 keep 2 target 8 make tti and hst files This runs and makes: \D0_Log\Find_DAC_V_5_0_F_20040109.tti;1 scan this file for "error tag" and scan the hst file for "fail" and "warn". It all looks clean. Edit \D0_Config\Load_L1Cal_GainsPeds.mcf to use this new eta 1:16 ped file Saturday Morning During ZB running load the normal setup for testing L1 Cal Trig and the And-Or rates for EM Ref Set #3 Count Comparator #0 Count Threshold =1 look wrong. On some monitoring sweeps they are high by a few upto 100 Hz. Specifically that are > the rate for EM Ref Set #2. And-Or rates for EM Ref Set #3 Count Comparator #1 Count Threshold =2 looked OK. Use the Cal Trig Testing Tool for TT's over Ref Set but I can see nothing. It is only a few or 10's of Hz of junk so there is no chance to see anything. Pull output cables from Tier 2 Counter Tree CAT2's. The problem is clearly in the |eta| 9:12 section. Pull CHTCR EM output cables. It is clear that the problem is in the lower half of the upper Tier 1 crate in M108 i.e. eta -9,-12. That is the problem is in eta -9:-12 phi 9:16. Instead of plugging the EM output cable directly back into this CHTCR card I used a junper so that the counts for EM Ref Sets # 0,1,2 are passed through OK but EM Ref Set #3 is blocked at this point. All the And-Or rates now look OK. Notes: eta -9:-12 phi 9:16 includes -10,11 i.e. the TT that appread to blow up Thursday night during beam running. We never did see -10,11 look funny in the Examine plot or on the scope. So it could be that this intermittent problem just happened to go away at the same time as I Excluded -10,11 EM. The only reason Thursday night that we thought that -10,11 was the cause of the rate problem was that when using the TRICS Cal Trig Test Tools I saw its was over some Ref Set a couple of times. While working on this Saturday morning I noticed that during ZB running it looked like a lot of the SMT caused noise rate was coming from this same section eta -9:-12 phi 9:16. I could see the Tot Et 0 Hint LED flash and pulling the Total Et cable from this CHTCR cut the And-Or rate a lot for 1 TT over Tot Et Ref Set #0 term. I don't think that this is a Cal Trig problem but rather that this is a HD TT that picks up a lot of SMT noise. A good time to be careful - there may be a couple of problem things going on here.