D-Zero Hall L1 Framework and L1 Calorimeter Trigger Logbook --------------------------------------------------------------- Log book for 1994 is in D0_HALL_LOGBOOK.LBK_1994 Date: At: Topics: .............................................................................. .............................................................................. Date: 13-FEB-1998 At: Fermi Topics: ECB Meeting, Morning meeting with Walter knopf and company about Hub End of SCL. .............................................................................. Date: 30-JAN-1998 At: Fermi ECB Meeting, Bring back to MSU the reaction timer that Dan Owen was using, Walter Knopf come to ECB to show options for receiver. .............................................................................. Date: 13,14,15-JAN-1998 At: Fermi Lehman Review, SCL meeting The Lehman people at the breakout were: Stan Durkin (Ohio State) Roger Rusack (Minnesota) and the new person Tony Parker. Meeting with Walter Knopf, the engineers are John Anderson and Jameson .............................................................................. Date: 19-DEC-1997 At: Fermi L2 Mini Workshop, give L2 MBT talk .............................................................................. Date: 12-DEC-1997 At: Fermi L2 Meetings, Deliver 6 Digit Time to Dan Owen, Harry turns into a rock. .............................................................................. Date: ~1-DEC-1997 At: MSU but the action is at Fermi. Yet another water leak in the 6 pack in rack M100. Radiator was replaced to fix the leak. .............................................................................. DATE: 14-NOV-1997 At: Fermi TOPICS: Meetings, Trigger, Walter Knopf .............................................................................. DATE: 17,18-OCT-1997 At: Fermi-NIU TOPICS: Upgrade Trigger 2 day Meeting .............................................................................. DATE: 10-OCT-1997 At: Fermi TOPICS: ECB Meeting, UIC Meeting, Water update Bring a new micron pc to Fermi for Kate Frame. We had another water leak about two weeks ago in one of the lower radiators in the 6 pack in M100. The Multi Channel Leak Detector now has a connection to Firus via a DIP read relay that run right off of its output. Marvin was not at the ECB meeting so no discussion about SCL, SLIM, or L2 power. Meeting at UIC with their electronics people about MBT. .............................................................................. DATE: 28,29-AUG-1997 At: Fermi TOPICS: Trigger Meetings, L2 Meetings Water Leak There is a water leak between M102 and M103. Lots of stuff in M103 looks wet. Not too much in M102 looks wet. The leak is in the top radiator in M102. The new multi channel drip detector did turn things off. Most of the water was sprayed into the L1.5 FW. Some L1 Cal Trig cards in M103 are wet. Some L1 Cal Trig backplanes got wet, mostly where the power foils connect to the power bars. The power supplies in M103 may have gotten wet. I do not have any idea how long the system has been leaking; all of August ? .............................................................................. DATE: 20-JUNE-1997 At: Fermi TOPICS: Trigger Meeting, L2 Meeting .............................................................................. DATE: 9-MAY-1997 At: Fermi TOPICS: ECB meeting .............................................................................. DATE: 24,25-APR-1997 At: Fermi TOPICS: Online meeting, Run II Trigger meeting, Level 2 Trigger meeting, ECB meeting, Upgrade meeting, meeting with Mark Baert, New SCL_Description and And-Or Term Receiver Description on Web, Check Master Clock Rack Layout .............................................................................. DATE: 12-MAR-1997 At: D0 Hall TOPICS: Lehman Review, Final Names for SCL Talk with Mark Baert The command link to the Port Card will carry a string something like: Acquire, Acquire, ..., Acquire, Trigger (1 or 2 of these), --> Digitize (some number of these), Readout (Until the VRB all report "end of event"), Idel (8 or 16 of these, pipeline refill ?) Acquire, Acquire, .............................................................................. DATE: 1-MAR-1997 At: D0 Hall TOPICS: ECB Meeting, Talks about Running without gaps .............................................................................. DATE: 14-FEB-1997 At: D0 Hall TOPICS: ECB Meeting, Talks about the Level 3 Transfer Number .............................................................................. DATE: 16,17-JAN-1997 At: D0 Hall TOPICS: ECB Meeting, Talk with Marvin about VRB, Talk with Mark Baret about SCL SLIM, Talk with Jeff Bantly about Run II Scalers, Change the Mail_Server .com file. VRB Schedule talk with Marvin Johnson Right now the Ed Barsody VRB is partially working. It has 11 wires so far. D0 will have 2 on March 1st for the Test Beam. There will be a first production round of 16 VRB's. The PO for these will be issued mid April and it will take about 6 weeks to get them made. Thus there will be 16 of them in early to mid June. Marvin will try to push for a 3rd one by 1st to mid March. Getting one also means getting the transition module. Also problem of VME64 crate. But there are some of these around and there is going to be a big order of them. Perhaps can use a CDF controller for test or else they are going to make a simple test controller. Mark Baret is making the real D0 controller. This is due April 8th. We may be able to use Jay Wightman Test Beam VXWorks code to look at data. Mail_Server.com In TrigUser account I commented out the MAIL command in the Mail_Server.com file that TCC TRICS uses to send messages to us. There were 4412 messages in the TrgMgr account. I don't think that we need mail again until RUN II starts. I had trouble making this new version of the Mail_Server.com because there was not Quota for TrgUser. It was about 300 blocks over drawn on a quota of 20000. So in [TrgUser.TrgMon] I got rid of Test_Trgmon_main.Exe_Old from 2-MAR-1995. This ended up not being necessary because the Main_Server.com files are owned by TrgMgr anyway. TrgMgr has about 3k blocks left on a quota of 200k. .............................................................................. DATE: 20,21,22-NOV-1996 At: D0 Hall TOPICS: Collaboration meeting, Trigger meeting, ECB meeting, L2 look like L1 or L3 meeting, TCC First Collaboration Meeting that Harry is in charge of. Philippe goes to some OnLine and Simulation meetings on the 20th and 21st. ECB meetings now shifted to DEC 6th and 20th, Jan 3rd and 17th. FW was stuck with the zrl VME interface saying "4" on its LED display. Booting TCC via trigger did not help. Power Cycle and boot got things going again. .............................................................................. DATE: 8-NOV-1996 At: D0 Hall TOPICS: Level 3 Review, Bring back to MSU a Short MVME-214 MSU car is dead so drive Carol's car to Fermi. Level 3 Review lead by John Womersley. Bring back to MSU Short MVME-214 SN#? and convert it to a "Normal" MVME-214. .............................................................................. DATE: 1-NOV-1996 At: D0 Hall TOPICS: ECB Meeting, Meeting with Jerry Blazey about Schedules, Try to cook a 17256L for the Run II Xilinx 4013's. ECB Meeting about Muon L2 data transport and processing. Present the L1 FW, L1 Cal Trig, and L2 schedules to Jerry Blazey. He is getting trigger schedule stuff ready for the Lehman review next week. Try to cook a 17256L using the UniSite model 48 with its August 1996 software, but it can not cook the L part. .............................................................................. DATE: 17,18-OCT-1996 At: D0 Hall TOPICS: ECB Meeting, Trigger Meeting. ECB Meeting about Muon L2 data transport and processing. In the Trigger Meeting present the current "MSU Model" of Level 2 (e.g. 128 L1 Spec Trig's, L2 Global makes the decisions). In the Trigger Meeting talk with Jeff Bantley and others about Run II scalers. .............................................................................. DATE: 4-OCT-1996 At: D0 Hall TOPICS: Level 2 Muon Meeting, New MSU inventory tags, Return Hydra_II to Fermi, bring a Vertical Interconnect pair to MSU. I removed the Vertical Interconnect Slave and Master from the L15CT crate (the pair that communicates withIN the L15CT crate) and brought them back to MSU along with their short cable. This gives us one Master and two Slaves at MSU for Run II development work. HYDRA_II VME DSP Cards vs Fermi Inventory Tags ------------------------------------------------------- We (D-Zero Fermi) own a total of 5 Hydra_II Digital Signal Processor VME modules. These were all purchased in the summer/fall of 1993. They were purchased on a number of different PO's. Order of magnitude, there were probably 3 PO's: one for the running system, one for the Fermi "test stand", and one for the MSU "test stand". I can find copies of the Requisitions if this becomes necessary. During the normal Run 1B and 1C collider operation, three of these modules were in the running system, one module was at Fermi in the "Level 1 Spares rack/cabinet", and one was at MSU in the test stand. Which actual module was in what location varied from time to time as we at least twice send modules back for repair. Currently (4-OCT-1996) all 5 modules are at Fermi in the D-Zero assembly building. There are Fermi Property tags on 4 of the 5 modules. Location Property Tag Number ----------------------------- ---------------------------------- M124 Rack L15CT Processor "A" does NOT have a Fermi Property Tag M124 Rack L15CT Processor "B" Fermi Tag #84182 M124 Rack L15CT Processor "C" Fermi Tag #84183 L1 Spares Rack/Cabinet Fermi Tag #84184 L1 Spares Rack/Cabinet Fermi Tag #84338 If necessary I can find the Requisition Number and the Ariel Serial Number for each of the modules. I can also put together a complete history of each module, i.e. when we received it, any repairs, which locations it has be used in... New MSU Inventory Tags Applied ------------------------------------------------- Classic New Item PA Tag Description Number Number Additional Information Comments ---------------- ------- ------ --------------------------------------- Computer PDP11 PA12252 005390 This is in Rm 155 of the PA building. Digital "Its "PA" sticker is missing". It is old junk that should be taken off of the inventory. VAXSTATION 2000 PA12641 005393 This is in the D-Zero North Office Digital Building. This is old junk and should AB80100768 be taken off the inventory list. It is worth nothing and has not been used for at least 2 years. Scope Digital PA12928 005389 This is in Rm 156 of the PA building. Tektronix 2440 VAXStation 3100 PA13023 005400 This is a 3100 model 76. This is node Digital D0MSU2 in the D-Zero Main Assembly building. Computer PV416CH PA13080 005394 This is a 3100 model 76. It is in the Digital North D-Zero office building. It it AB12202VH8 node D0MSU6. Computer PV416CH PA13081 005399 This is a 3100 model 76. This is node Digital D0MSU4 in the D-Zero North office building. Computer PV416CH PA13084 005395 This is a 3100 model 76. It is in the Digital North D-Zero office building. It it AB11400VAR node D0MSU5. Computer 4000 PA13218 005392 This is a 4000 model 60. It is in the Model 60 North D-Zero office building on Dan Digital Owen's desk. It is node D0MSU7. Computer PV416CH PA13246 005396 This is a 3100 model 76. It is in the Digital M-West office building. It is used the disk drive listed next in this. Its serial number is something like AB12700AXZ but I could not read it clearly. Drive Hard PA13247 005397 This is a disk drive expansion cabinet SZ12C-CA used with the computer listed right Digital above in this table. It is in the M-West building. Its serial number is something like AB1260HPY but I could not read it clearly. VaxStation 4000 PA13310 005398 This is a VaxStation 4000 model 60. Model 60 It is located in CDF trailer offices Digital in room 138 E. Computer Q-700 PA13443 005391 This is an Apple "Quadra" computer. Apple It is located at Dr. Huston's house. F1239B4J954 I gave him a new sticker and asked him to put it on this machine. .............................................................................. DATE: 27-SEPT-1996 At: MSU TOPICS: Visit to ADCO Make a site visit to ADCO. Talk with Jack Zinke, Dean Lehocky, and the person who manages their rework operation. All looks OK. .............................................................................. Date: 11:13-SEPT-1996 At: D0-Hall Topics: Collaboration Meeting, Install new Drip Detector, ReStart the Framework, Cooling water temperature change, Pull cards from M114. Installed the new 16 Channel Drip Detector. It is located in the top of rack M110. A Green LED illuminated means that that channel is not sensing any drips. A Red LED illuminated means that that channel has been switched off so that it does not participate in the global decision (i.e. it can not cause a trip). The new 16 channel Drip Detector appears OK. The spare cards for it are in the spare card cabinet. The cables for drip strip M103-M104 and drip strip M104-M105 were (are) a bit flakey the way that they plug into the back of the Drip Detector box. One of them has to rotate CCW and the other has to rotate CW in order to make good contact. The output of the new 16 channel Drip Detector goes to the RPSS and then over to M114 where it feeds a Shea 1553 voltage monitoring channel to generate an alarm. Recall that the Drip Detector shows up in the RPSS as alarms for rack M113 (labeled Drip Detector). The problem of the drip alarm from 30-AUG-1996 was on the strip M102-M103. It looked like one drop of condensation and had left some typical white corrosion about the size of a nickle. The L1 and L1.5 FW and M114 are now running again. The blower is on slow and gives a differential air pressure of about 0.65 to 0.70 inches of water. The cooling water has been brought up to about 67 degrees. This helps a lot with humidity and condensation problems. Because of this warm cooling water I have pulled a bunch of cards out of M114. I pulled all of the Tier 1 and Tier 2 BBB cards. I pulled all of the Foreign Scalers. Before turning things on the thermometer in M114 was about 71.4 degrees (with just the VME running) and after running the FW for 1 day it is 75.7 degrees. The thermometers in the string of racks were about 69 degrees and now after 1 day of running they are about 78 degrees. .............................................................................. Date: 30-AUG-1996 At: D0-Hall Topics: NSF meeting about proposal for Run II L2 Trigger, Owen vs Truck Meetings with Patricia Rankin who is the new NSF program officer about a proposal to have NSF pur money into the Run II L2 Trigger. Our L1 stuff is not running. There is a Drip Detector Alarm. I checked for watter and found none. The water valves to the Cal Trig part are OFF as they should be. The pully, a belt, and a pully puller are all sitting by our stuff. I will put them in the cabinet. Bring back to MSU MVME-214 MSU SN#5, a "V" Type 214. Bring back to MSU an HP model 8011A Pulse Generator that has inventory lables PA-12598 X03631. I don't know when or by whom this generator was brought to Fermi. It was found on the second floor by Stu Fuess. Dan Owen and his bicycle tangled with a truck. He appears mostly OK. The smd hot air desoldering machine (which may be called a "Hart") is by Nu-Concept Systems Inc. 2725 Advance Ln. Colmar, PA 18915-9791 Phone: (215)822-8400 The solder paste dispenser is a Weller WD2000. The RF soldering iron is "Metcal" in Menlo Park, CA. 415-325-3291. .............................................................................. DATE: 19-JUN-1996 At: MSU TOPICS: kludge TCC ELN system to run without L1.5 CT and 36x36 backplane The system prepared 24-MAY-1996 was booted. There was still a problem with TCC trying to load the 36x36 SBSCs every 5 sec using the special scan/reset MTG in the 36x36 scaler backplane. This would generate 8 lines per 5 seconds in the logfile. A limitation/bug with the the MOD_HDB command prevented reaching this MTG to aim it somewhere else. Make and load a new system that does not try to load these SBSC cards: EWORK1:TRICS_V64.SYS_19JUN96 The old ("production") system is still in EWORK1:TRICS_V64.SYS_24OCT95 Edit the TRICS_INIT_AUXI.DAT file to comment out commands for the Tier#3 and the 36x36 scalers. Make backup copy of the new file in TRGCUR:TRICS_INIT_AUXI.DAT_BETWEEN_RUN Edit the TRICS_INIT_AUXI_L15CT.DAT file to remove all commands to L1.5 CT support MTG. Make backup copy of the new file in TRGCUR:TRICS_INIT_AUXI_L15CT.DAT_BETWEEN_RUN .............................................................................. DATE: 22,23-MAY-1996 At: D0 Hall TOPICS: ECB Meeting, Run 1 L2 Review Meeting, Learn about HSRO, Get ready for Boston Workshop, Start up the Framework. ECB Meeting. Officially decide not to approve the pipeline restart until more is under stood about it. There is not yet measurements to show that SVX-II can work with restart. These measurements will probably not be made until after the Boaton Workshop. The amount that restart gains you is not much. L2 Review had lots of information but it is hard to summarize. L2 was a big industry. The simulator was very connected to and supported by the L2 people. Mike Utis is now doing the HSRO Laser module. All continues the same except that it is not clear how the connector will go onto the laser card. The VRB stuff is on the WEB. Ed Barsotti's group is doing the input modules to the Vrb. Start up the Framework. The hardest part was finding the EtherNet connection to the MCH. It is located under the floor at the North end of the row of L2 racks nearest to the Host. The drip detector indicates that we have a leak but I could not find any water. We now have about 100 psi coming into the radiators. I put a small 5V power supply in M114 to jumper around the drip detector. We are still running with Data cable 0 looping through all of the muon trigger racks. Things ran OK and then the FW was turned off for the 3 day weekend. .............................................................................. DATE: 9,10-MAY-1996 At: D0 Hall TOPICS: TeV 33 Workshop .............................................................................. DATE: 25,26-APR-1996 At: D0 Hall TOPICS: ECB Meeting, VRB, Fan, The Fan in the 8700 is by Torin Engineered Blowers, a division of FASCO Industries, P/N A61795, Ref # 7134-0036 FASCO Industries is at phone number (704) 482-9582 Ruth Pordes is now in charge of the SAR aka VRB. The engineer is named Mark Bowden. Changes requested to the SCL document: 1. First RF bucket in a 132 nsec period having the particles is OK 2. Define the length of time that the signals are not defined. 3. State that there are not glitches, e.g. gap markers are stable. 4. Define that we will not advertize both L1 and L2 during the same 132 nsec period. 5. There is a minimum spacing between L1 Accpets and between L2 Decisions. 6. L1 Qualifiers and L3 Transport Number come from the same pins. 7. The beginning of Turn Marker will be at the 5th 132 nsec period before the end of the sync gap, i.e. ticks 0:5 are in the gap and tick #6 is the first tick out of the gap. Sync gap means the gap that has no possibility of generating triggers. .............................................................................. DATE: 11,12-APR-1996 At: D0 Hall TOPICS: Collaboration Meeting .............................................................................. DATE: 14,15-MAR-1996 At: D0 Hall TOPICS: Level 2 meeting, Atlas meeting, ECB meeting, Look into High Speed RO Level 2 meeting, Presented stuff about the interactions among Frameworks, L2 Pre-Proc, and L2 Global. Presented stuff about the hardware platform requirements for Cal Pre-Proc and Global. Presented current understanding of ALPHA trigger processor board. Atlas meeting at Argone. ECB meeting. Tuts want us to distribute the Serial Command Link stuff. Sten Hansen gave the 10kHz pitch again. No rotten fruit especially about his system for reading out Cal at 10kHz. In a talk with Boris; he is happy with the current SCL spec, he just want the responsibility of the Geo Sections emphasized. Marvin says to send him a note about our requirements from the Run II timing system. High Speed Readout Port Card End Low Speed command receiver: Uses HP HFBR-2416 receiver diode assembly which is 1 1/8" full length, (threads and barrel together are 5/8" long), 1/2" wide and 3/8" high. There are two of these; one for data and one for clock. These feed separate ML6622 "receiver" chips. The MicroLinear company manufactures the ML6622. The ML6622 makes ECL output signals that feed a 100314, which in turn feeds a 100325. The TTL output of the 100325 feeds a PAL which makes the 4 (I think 4 is right) command signals. The ML6622 is a 16 pin small outline i.e. about 3/8" x 1/8" not counting gull wings The ML6622 is called a, "Data Quantizer,High Speed". High speed uses 100S322 ?? to feed the Glink transmitter. This feeds the Finistar module. The Finistar is 2 3/4" total length 1/2" threads and barrel, 7/8" wide, 9/16" tall. Finistar is at 3515 Edison Way Menlo Park, CA 415 364-2722. The HP GLink is about 1"x1" or 1 1/8" x 1 1/8" counting leads. The Port cards will us a mezzanine with the HI Glink and the Finistar mounted on it. This will be a Fermi built exact copy of the Finistar prototype module. All of the Fermi prototype boards used 100k from Synergy. The SAR is going to make some assumptions about the format of the data that is send to it. I do not understand this at this time. I think that this mostly has to do with how the SAR determines the end of the data from a given event. As I currently understand it, the big point is that it does not use the DAV line from the receiver, i.e. we are not forces to send at 18.8 nsec. I think that there is some requirement that every other 16 bit word is an address and that the data following a certain address is the end of event marker. .............................................................................. DATE: 22,23-FEB-1996 At: D0 Hall TOPICS: Talk with Mike Matulik, Check how far the L1CT Pedestals move with BLS Clocking turned off, Run Random Cal Trig Cell Test, Start turning stuff off, Run L15CT_Prov, Talk with Christenson and Tuts about money and schedule. L1 Cal Trig Pedestals I wanted to see how big of a noise spike from the BLS we really have been living on top of. This is to give us some idea about how big of a switching noise spike we can live with in the Run II BLS. So we turned off BLS switching and then made two measurements. First we took a L1CT Find-DAC run. The results of this are in VWork1:DAC_BYTES_NO_BLS_CLOCKING.TXT Next we just used TrgMon (with the normal pedestals loaded) can captured a number of sweeps of the TrgMon ADC Counts display. The results from this are in VWork1: PEDESTALS_WITH_BLS_CLK_OFF.TXT As you would expect, the biggest change is at small eta. This is because the BLS switching noise is in Energy not in Et. Looking at just the eta +1 and -1 rings the average shift in Pedestal DAC is shown in the following table. This table is averaged over the 32 phi's in each of the eta rings. eta +1 EM 6.094 Pedestal DAC counts eta -1 EM 4.906 " eta +1 HD 4.750 " eta -1 HD 3.437 " With BLS clocking stopped the Find-DAC pedestal values went down by the amount shown in the table above. Thus during normal operation we have been digitizing during a dip in the BLS Trigger Pick-Off signals. Run Cal Trig random cell with everything enabled except locked on page 4 and not checking the And-Or Network view of the Cal Trig Terms. Ran 50,123 loops: S-HTT/PAR%rand% Loop 50000/50123, Error Count is 0 E-HRD/TST%rand% 50123 Loop Test Completed, No Error Detected Run L15CT_Prov on two pulser runs taken Monday afternoon this week i.e. 19-FEB-1996. These are runs 96956 and 96957 on Data3:[CAL]. All was OK. So as far as I know everything was working OK at the end of Run 1C. Turn Equipment OFF ------------------ Turned OFF all of L1 Cal Trig, i.e. all Power Pans in M103 through M112 except the Framework Expansion (L2 FW) Power Pan in the top of M104. Turned OFF water feed to: M103, M104, M105, M106, M107, M108, M109, M110, M111, M112, M113 M112 & M113 are the feed to the 8-Pack between M112 & M113. Edit TRICS_Boot_Auxi.dat to reduce the L1 Cal Trig coverage to 1 Trig Tower. So what is left running: M101, M102, Framework Expansion (L2 FW) in the top of M104, M114 green power pan and VME, M124 L15CT VME, and the two Power Pans on top of the air conditioner. Water is ON in M100 (i.e. the feed to the 8-Pack between M100 and M101) and in M101 and M102. I gave the current draft of the SCL specification to Mike Matulik and copies of the hand draw block diagrams of SCL Receiver Mezzanine and SCL SLIM module. Gave the multi point schedule to Christenson and Tuts. Tuts wanted to talk about money and now he finally understands all of the junk that has been given to him over the past year e.g. Jan 1995 estimate, Dec 22 Status report, Feb 10th '96 update. Maris gave Jim a copy of the 1996 MOU addendum. .............................................................................. DATE: 15,16-FEB-1996 At: D0 Hall TOPICS: Atlas L2 Meeting, Talk with Dan Owen about Luminosity Profile in Run II, Talk with Dean more about TT noise in Run II, picked up new safety stuff from Rick Hance, Talked with Jim Christenson about cost and schedule, Talked with Dean,Jan,Mike T about operation after end of Run I. Atlas L2 Demonstration Project Meeting, Abolins, Owen, Bob Blair, Schlarith, Dawson. Dawson will propose a demonstration project supervisor. Talked with Dan Owen about him talking to people in the Accelerator Division about the luminosity profile during Run II and the frequency of shots during run II. Dan full understood the reasons why we think that it is important to understand these aspects of run II operation now, and he is going to poke at accelerator people. Talked more with Dean about TT noise in Run II. I asked him about the channel to channel cross talked @ 400 and @ 132 nsec. i.e. Bo Pi's Thesis. He expects that things will be controlled at 400 nsec just because he will need them to be for the precision readout system to work. He remains very doubltful that we understand much about 132 nsec operation. In fact the TT signals at 132 could be so bad that L1 Cal Trig will not actually know which BX should have caused the L1 trigger. e.g. L1 threshold of 15 GeV to trigger, on BX N the TT has 17 GeV, we trigger, but that TT was really on its way up to 57 GeV so we picked the wrong BX to call the source of the action. Picked up "Electrical Design Guidelines for Electronics to be used in Experiment Apparatus at Fermilab" 22-DEC-1994 from Rick Hance. This is for us to follow in designing are Run II equipment. It says, "A fuse is required for each voltage that is capable of delivering enough power to damage the board". Jim Christenson wants schedule cost stuff that he can merge together. Most concern is about Fermi people load. Jan and Dean have the following idea about how we will run in the break between Run I and Run II. The DAQ system can be down but never for more than one month. The idea of this is to limit how far away from a usable condition the system is allowed to drift. .............................................................................. DATE: 8,9-FEB-1996 At: D0 Hall TOPICS: Collaboration Meeting, Talks with Dean about Run II TT noise. Collaboration Meeting lots of talk about triggers, Joan gave talk about Muon vs TT signals trigger, Terry Geld gave talk about more L2 Cal pre-processor timing stuff. Started talking with Dean about Run II noise in Trigger Towers. We need to think about this to get it into the L1 CT Simulator for Run II. More Johnson noise, crossing to crossing lack of time resolution noise, 400 nsec vs 132 nsec difference in noise. .............................................................................. DATE: 25,26-JAN-1996 At: D0 Hall TOPICS: Raja Special Run, Hand adjust -20,9 EM, Run Cal Trig Random Cell In making tests for the Raja Special Phi Symmetry Run we get some interesting numbers to check the early Monte Carlo Global Total Percent Et Threshold Exposure Luminosity And-Or Rate ------------ -------- ---------- ----------- 50 GeV 92% 5.31 E30 3,082 Hz 60 92% 5.31 E30 1,540 75 92% 5.31 E30 562 90 99% 5.20 E30 215 105 99% 5.20 E30 84 120 99% 5.20 E30 34 So now it is fun to compare this to the D0 Note 253 Monte Carlo. But one needs to take into consideration the multiple interaction stuff (which does make a big difference). There was one more trigger running; it had a threshold of 75 GeV and had the L0 single interaction flag required. It's And-Or rate was 97 Hz, i.e. down a factor of 5. -20,9 EM Hand adjust the DAC for -20,9 EM from 22 to 16. Its zero energy response had drifted up to 10. See also 13-14 DEC-1995 and 19-DEC-1995. Random Cell S-HTT/PAR%rand% Loop 100000/100123, Error Count is 0 E-HRD/TST%rand% 100123 Loop Test Completed, No Error Detected Everything enabled except locked on page 4 and no And-Or checking. .............................................................................. DATE: 19-JAN-1996 At: from MSU TOPICS: Pedestal values for +17,15 EM HD Joan sent a note saying that the zero energy response for Trigger Tower +17,15 EM and HD had slipped down to 7. By hand the DAC values for these channels were adjusted in the Init_DAC_Bytes.LSM files in TrgCur: and on D0HTCC::. +17,15 EM was moved from 35 to 37. +17,15 HD was moved from 31 to 33. .............................................................................. DATE: 9,10-JAN-1996 At: Fermi TOPICS: Run II L2 Meetings, L1 Cal Trig Exerciser run, Work on Huge Run the L1 Cal Trig exerciser program. It immediately finds a problem with Px being off by +-8 counts. This is the +Px coming from eta +5:+8 Phi 25:32. This appears to be exactly the same problem as described in detail in the log book entry from 27,28,29 June 1995 entry. We wiggle the paddle card for this Tier 2 CAT card (M105 3rd paddel down from the top) and things get better. Run 60k loops of Cal Trig exerciser with everything enabled except locked on page 4 and not checking the Framework Terms and all is OK. Huge Tool Work Greg Snow discovers that Huge Tool is not passing some Jet events that it should pass. There is a particular set of two jets phi combinations that should pass but Huge is not passing the events. This is all with the minimum phi distance parameter set at it "normal" value of 8. The problem is that in the Global DSP it really takes 31 compares and 30 rotates of the ORed local DSP Jet Phi Mask against the test pattern to guarantee that you will find all the back to back pairs with all possible values of the Minimum Phi Distance parameter. We were only making 16 compares and 15 rotates, which was appropriate only for real back to back, i.e. Min Phi Distance parameter value of 16. A new version of Global Huge Tool was installed in the late evening of 9-JAN-96 This was at the end of a day when the experiment had been stopped for accelerator access and before they started using the second version of the W->JJ Trigger List (drop double L15CT on Spec Trig #6, drop L15CT on Spec Trig #13, up the L15CT EM 1x2 from 7 to 8 GeV). Greg Snow's analysis now says that all is OK. Run II L2 Trigger Meetings on the 9th and 10th of Jan Getting rid of serialization is out, recyclying is out, model is pre-processor per data source feeding a Global L2 Stage, using the new L3 DCD is in, pre-processors are single server devices, Global stage can be a single server or farmlette, Cal precision readout will have to come up from its current transfer limit to L3 of somewhere 1 to 2 kHz, .............................................................................. DATE: 3-4-JAN-1996 At: Fermi TOPICS: Work on the installation of the HUGE L15CT Tool. To move from L15CT Tool #3 to Tool #4 the following things have to change: DSP executable files (12 BLX files on D0HTCC::DUA0:[L15CT$EXEC] ) L15CT Defaults file for TRICS ( L15CT_DEFAULT_CONFIG.DAT on D0HTCC::) 68k_Services abs file (L15CT_68k_Services.Abs in TrgCur:) and boot 68k_Ser The "Dan Owen Pulser Run" Configuration files The Cal_Trig_L15 Configuration file To automate the moving back and forth between L15CT Tool #3 and Tool #4 there are now two com files in VWork1: these are: Go_To_3.com and Go_To_4.Com These two com files do the appropriate Copy/OverLay commands on: the DSP BLX files, the L15CT_Default_Config.Dat file, and the 68k_Services Abs file to move back and forth between Tool #3 and Tool #4. The following is a review of the setup of the Dan Owen Pulser Run Configuration files: CALOR_PLS_TRIG_HIGH.CAL Calor_pls_trig_high.trig calor_pls_trig.lev1 L2_pass_fail.filt calelec_slow_detector.req trig_l15ct.req calor_mpls_high.req Calor_pls_trig_l15.trig calor_pls_trig_l15.lev1 cal_pls_trig.l15 <--- This sets up L15 Cal Trig calelec_slow_detector.req trig_l15ct.req CALOR_PLS_TRIG_LOW.CAL Calor_pls_trig_low.trig calor_pls_trig.lev1 L2_pass_fail.filt calelec_slow_detector.req trig_l15ct.req calor_mpls_low.req Calor_pls_trig_l15.trig calor_pls_trig_l15.lev1 cal_pls_trig.l15 <--- This sets up L15 Cal Trig calelec_slow_detector.req trig_l15ct.req The following is a review of the setup of the Cal_Trig_L15 Configuration files, i.e. the "Call" structure for the CAL_TRIG_L15 configuration: CFG:CAL_TRIG_L15.CAL @CFG_CAL:Cal_trig_l15.trig (defines one Spec Trig) @CFG_CAL:Caltest_l15.trig (defines another Spec Trig) CFG_CAL:Cal_trig_l15.trig @CFG_DAQ:crossings.lev1 (defines L1 requirements for this ST) @CFG_LV0_CRATE:trig_level1.req (readout Framework) @CFG_CAL_CRATE:trig_l15ct.req (readout L15CT) CFG_CAL:Caltest_l15.trig (ONLY USED BY CAL_TRIG_L15.CAL) @CFG:caltest_l15.lev1 (defines L1/L1.5 reqirements for this ST) ! @CFG:cal_detector.req (REMOVED 3-JAN-1996 ) @CFG_LV0_CRATE:trig_level1.req (ADDED 3-JAN-1996 ) @CFG_CAL_CRATE:trig_l15ct.req (ADDED 3-JAN-1996 ) CFG:caltest_l15.lev1 @cfg:caltest.l15 (defines L1.5 requirements for this ST) @CFG:no_main_ring.lev1 (defines L1 main ring veto for this ST) CFG:cal_detector.req @CFG_LV0_CRATE:trig_level1.req @cfg_cal_crate:trig_l15ct.req @CFG_CAL_CRATE:cetec_data.req @cfg_cal_crate:cal_laser_reset.req @CFG_CAL_CRATE:calor_inspill.req @CFG_CAL_CRATE:cal_data_ccn.req @CFG_CAL_CRATE:cal_data_ecnw.req @CFG_CAL_CRATE:cal_data_ecne.req @CFG_CAL_CRATE:cal_data_ccs.req @CFG_CAL_CRATE:cal_data_ecsw.req @CFG_CAL_CRATE:cal_data_ecse.req BLX Size problem The only problem encountered so far is that the BLX DSP executable files for some of the DSP's were so big that they overflowed the 8k longwords available in the Hydra II dual port memory. On the VAX the BLX files must be 192 blocks or less to fit in the Hydra II dual port memory. Steve changed the memory space reserved for the Ref Sets and the Object Lists from "initialized" to "non initialized" space to save enough space in the BLX file so that the whole thing would fit in the Hydra II dual port. Making this change to using "non initialized" memory brought the largest BLX file down from 199 blocks to 189 blocks. Note that the memory space for the Trigger Tower data was left as "Initialized" memory because this is how the zero energy response for eta's 21:24 is loaded in. First HUGE Tool Tests Initial timing tests, i.e. running at a couple of Hz on noise with low L1 thresholds and L15CT setup to give 50-50 confirm - reject using the EM-Fraction part of the HUGE Tool, the processing time per event is something like 94 usec. Problem going back to Tool #3 Now we find a problem when switching back to Tool #3 after running Tool #4. The problem is that the last Ref Set that was given to TCC is a Tot Et Ref Set. When we execute the Go_To_3.Com and then Initialize TCC all of the Local DSP's complain that they were expecting an EM type Ref Set and found a Tot Et Type Ref Set waiting for them. In the "panic" to get thing going (6 p bar bunches were now in) we booted TCC. This "fixed" the problem because it caused TCC to forget that it had been given a Tot Et Ref Set. TCC wakes up from the boot thinking EM Ref Set and the DSP's are all happy running Tool #3. The solution for this going back to Tool #3 problem is to add at the end of the Go_To_3.Com file the instructions to give TCC an EM Type Ref Set for L15CT. This is thought preferable to changing the L15CT_DEFAULT_CONFIG.DAT file for Tool #3 (where in principal one could have put the instructions to give TCC an EM Type Ref Set) just because this leaves us with EXACTLY the same setup as was used by Tool #3 during Run 1B and the first half of Run 1C. To help understand the details of how TCC handles the L15CT Term #0 Ref Set when it is defined multiple times or else not defined at all, Philippe looks at the details inside TCC. His report is the following: I investigated the behavior you discovered yesterday: 1) during load_code, the WHOLE parameter block data structure that is used to hold L1.5 CT programming is "erased" by painting FF over the whole space (plus the list of which Sptrg use which L1.5 CT term is zeroed) EXCEPT the list of reference set types that is stored in a separate variable, and was just "forgotten". This list is actually NEVER initialized to anything. But the VAXELN kernel zeroes the list when it is first created, which corresponds to EM_type. 2) during the load_param phase the reference set type of each term is looked at and the proper fields of the parameter block are set to 00 for EM or FF for TOT before the parameter block is copied to VME space. 3) when the programming for a term reference set is broken into several COOR commands, TRICS currently does NOT verify that the reference set types of the multiple commands all match -> the last refset type wins. I guess the cafeine level was low that day and the paranoia level with it. TCC (otherwise) tries to check the syntax and requests, and the DSP's check_params code performs the ultimate check. Interpretation: 1) It is clear to me that TRICS should have "initialized" the reference set type do a known state. The load_code step is the logical place for doing this. 2) It is not clear to me what it should be initialized to. EM type or none/illegal? 3) The early "grand scheme Level 2-like" system or the "single- term/single-tool" implementations did not need nor exercize this. The whole issue was overlooked. 4) COOR is certainly expected to defined all parameter of the terms (at least term 0?) before sending the load_params command. The use of the L15CT_DEFAULT_CONFIG and other presets are clearly and "undocumented" feature, or convenient side door that does not fully follow the "official" COOR guidelines. What should we do about it? a) fix TRICS to initialize to EM type (or illegal type?) b) Change nothing and keep use the L15CT_DEFAULT_CONFIG file (or the flip- flop tool # 3/4 command file) to pre-define the reference sets that matter (only term #0?) to EM type, with a high threshold, that will translate into 255 (which is already the current initialized state/default value). END OF NOTE FROM PHILIPPE We do not have a problem when going to Tool #4 because Tool #4 does not and can not check the Ref Set Type of the Term #0 Ref Set. It really is not illegal with Tool #4 to give Term #0 either an EM or Tot Et type Ref Set. It still is not clear to us what type of Ref Set TCC will give to the DSP's if TCC is first given a Tot Et Type and then an EM Type Ref Set for Term #0. But if we use the above procedure for switching from Tool #4 to Tool #3 then it appears that there is no problem. Procedure for switching between Tools #3 and #4 Initialize TCC ! to get the 68k_Service counters and flags into TCC's log file Execute Go_To_3 or Go_To_4 as appropriate (these are in VWork1:) Load the 68k_Service computer Initialize TCC ! to get the lots of good interesting stuff into TCC's log file. Expected swapping back and forth activity At the beginning of this section of Run 1C we expect to swap between Tools #3 and #4 only a few times. The push at this point is to get to stable running with Tool #4. There may be more swapping back and forth at the very end of Run 1C when people think of lots of special runs that they need to do for which they have old Trig Config files and thus would rather swap to Tool #3 than modify their Config files. But at this time the decision is not to give the DAQ expert the procedure for swapping. Having com files to do the work of swapping has been useful during our testing work and having them since the beginning of the testing work gives us a standard tested procedure for swapping which can be turned over to the DAQ Expt latter in this Run if necessary. Initial test configuration of L15CT for WJJ. This is NOT the final global trigger list but the L15CT part is likely to be close to correct. --> L15CT Ref Set: all 4 Ref Set definitions identical (except for Term Number), so only Term 0 saved in this log: L15CTERM REFSET CRATE(0) TERM(0) TYPE(TOT) SIGN_ETA(NEG) MAGN_ETA(17:20) PHI(1:32) THRESH(1000000) L15CTERM REFSET CRATE(0) TERM(0) TYPE(TOT) SIGN_ETA(NEG) MAGN_ETA(5:16) PHI(1:32) THRESH(4000) L15CTERM REFSET CRATE(0) TERM(0) TYPE(TOT) SIGN_ETA(POS) MAGN_ETA(1:4) PHI(1:32) THRESH(2000) SIGN_ETA(NEG) MAGN_ETA(1:4) PHI(1:32) THRESH(2000) L15CTERM REFSET CRATE(0) TERM(0) TYPE(TOT) SIGN_ETA(POS) MAGN_ETA(5:16) PHI(1:32) THRESH(4000) L15CTERM REFSET CRATE(0) TERM(0) TYPE(TOT) SIGN_ETA(POS) MAGN_ETA(17:20) PHI(1:32) THRESH(1000000) --> Term #0: Jet L15CTERM LOC_DSP CRATE(0) TERM(0) USE_TOOL( 4) WITH_PARAMS( 127.0000000 1.9000000 1.9000000 2 63.0000000 2.0000000 4 8.0000000) L15CTERM GLOB_DSP CRATE(0) TERM(0) USE_TOOL( 4) WITH_PARAMS( 8 2 8) L15CTERM FRAMECOD CRATE(0) TERM(0) PASS_ONE_OF( 10000) L15CTERM ST_VS_TM CRATE(0) TERM(0) SPTRG( 5 11 12 13 14) --> Term #1: EM L15CTERM LOC_DSP CRATE(0) TERM(1) USE_TOOL( 4) WITH_PARAMS( 7.0000000 0.0099999997765 0.8999000191689 1 5.0000000 63.0000000 16 127.0000000) L15CTERM GLOB_DSP CRATE(0) TERM(1) USE_TOOL( 4) WITH_PARAMS( 1 1 16) L15CTERM FRAMECOD CRATE(0) TERM(1) PASS_ONE_OF( 10000) L15CTERM ST_VS_TM CRATE(0) TERM(1) SPTRG( 6) --> Term #2: EM L15CTERM LOC_DSP CRATE(0) TERM(2) USE_TOOL( 4) WITH_PARAMS( 15.0000000 0.0099999997765 0.8500499725342 1 7.0000000 63.0000000 16 127.0000000) L15CTERM GLOB_DSP CRATE(0) TERM(2) USE_TOOL( 4) WITH_PARAMS( 1 1 16) L15CTERM FRAMECOD CRATE(0) TERM(2) PASS_ONE_OF( 10000) L15CTERM ST_VS_TM CRATE(0) TERM(2) SPTRG( 9) --> Term #3: Jet L15CTERM LOC_DSP CRATE(0) TERM(3) USE_TOOL( 4) WITH_PARAMS( 127.0000000 1.9000000 1.9000000 2 63.0000000 2.0000000 4 5.0000000) L15CTERM GLOB_DSP CRATE(0) TERM(3) USE_TOOL( 4) WITH_PARAMS( 8 2 8) L15CTERM FRAMECOD CRATE(0) TERM(3) PASS_ONE_OF( 10000) L15CTERM ST_VS_TM CRATE(0) TERM(3) SPTRG( 15) --> Start Crate L15CTSYS START CRATE(0) Timing measurement during this W->JJ Test Run During the above W->JJ Test Run, L15CT was the only L15 trigger running. Events were entering L15CT at about 74.4 Hz. About 79% of the L1 events received L15CT processing. The dead time measured between 1.0% and 1.1% on TrgMon. Averaging these numbers gives about 141 usec per event in L15CT. The dead time was also measured by pausing the run and hand reading the DBSC that tells the global number of events entering L15CT and hand reading the DBSC that tells the number of 3.5 usec periods spent in L15 FW Decision Cycle. Then resuming the run for one minute, pausing it again and hand reading these two DBSC again. This gave an average L15 FW Decision Cycle time of 139 usec. The two DBSC's used are CBus 2, MBA 106 CA 23 FA 1 for the number of L15 Cycles and CBus 2 MBA 57 CA 7 FA 1 for the number of 3.5 usec periods spend in L15 Decision Cycles. With the above configuration, about 10 events were captured using ZBD. These events are stored in VWORK1:L15CT_TOOL_4_SPECIAL_RUN_1.ZBD. Steve looked quickly at this file. All of the events in this file passed both L15CT Jet Terms. Some of the events also passed one of the L15CT EM Terms. The one event that was carefully examined was 100% consistent between Local Object Lists, Global Object List, and Terms Passed. The L1 Trigger Tower data was NOT examined. At the conclusion of this W->JJ Test we had a TCC Initialization at about 12:11 which will contain the counters from 68k_Ser for the duration of the this test. .............................................................................. Date: 20-DEC-1995 At: MSU Topics: "investigate" ZCD problem Jan G. calls Philippe to try and understand problems with ZCD running. Here are (what we understood of) the symptoms. The ZCD runs "work" fine (= they find the sinal in the data) on dedicated special runs, but does not work when they use sptrg #0 during global running. Some preparation tests were done for this sptrg #0 runs, with a file running on noise/no beam. The same configuration file modified for beam conditions is used during global run, and seems to take data, but the signal is not there. There are three differences in the L1 configuration between the beam/no_beam versions. The no_beam version only lists andor term 123 (required) which is the ZCD input signal while the beam version has additional requirements 28(veto)=CALNE_PLS 104(veto)=LV1_DBL_BUFFERED and 107(veto)=SKIP_ONE_BEAMX. We review what these terms mean. We try a test and define this sptrg #0 (leaving it disabled) during global running and try to remove these extra andor terms from the configuration one at a time, and the andor rate does NOT change. We also worry that andor term #123 might be sick on this sptrg (since their earlier no_beam tests that "worked" were using sptrg #1, and we recently had problems with a dead andor term on one andor card). So we run a test where we temporarily define sptrg #31 exactly like sptrg #0, and verified that the andor rates of the 2 sptrgs exactly match. We do a test with the sptrg enabled and try to remove the SKIP_ONE_BEAMX requirement, and the andor rate goes up by about 5 Hz out of 4,500 or about 1 part in 1,000 which is surprising as the global firing rate is currently about 100 Hz which means SKIP_ONE_BEAMX is blanking only 1 crossing out of 3,000. Other factors change besides the andor requirements between beam and no_beam, e.g. the crate download and pulser programming. No conclusion at this time. Philippe suggests they do another special run with both setups defined on 2 separate triggers. Jan says: "I've made a config file which will run two bits for tonight's run. The first bit will be the "old" one. The second will include the L1 dead time stuff. The shifters will set the prescales so that the first bit has about 2/3 of the events. So we'll see what happens. Mike Matulik looked at the trigger electronics and doesn't see any problems with the pulsers. We have no idea now what is causing the difference. Thanks for your help. I'll let you know if we learn anything." Jan later reports: "I also thought that you both would be interested in knowing that the ZCD run they took earlier this week looks good. I saw a plot today where they were seeing signals in each fiber. But I haven't seen a plot broken down by trigger bits yet. I hope that Norm will have that plot early in January. So far the ZCD bit-0 runs are still a mystery." .............................................................................. DATE: 19-DEC-1995 AT: MSU TOPICS: Hand set a of L1 pedestals. Joan called and the pedestal for -20,9 EM was now running at 6. By hand its pedestal DAC value was moved from 17 to 22. This channel appears to be moving around. Its capacitor could be in trouble. .............................................................................. DATE: 13,14-DEC-1995 AT: Fermi TOPICS: Have L15CT W->JJ meetings, TCB meetings, L15CT hang, Hand set a couple of L1 pedestals. They called the appartment at about 3:30 on the 14th because L15CT was 100% front end busy. Things had been running fine in a 630 GeV run when L15CT went 100% BEB. L15CT was not really being used, all events were innocent by stander events "i" or "I". They say that they initialized L1 and re down loaded and that did not help. All log file entries looked 100% OK. There was one lower case "e" on the screen. Triggers were enabled but we were hung 100% FEB. Aborting 68k_Ser, and reloading it did not help. It juet gave one more new lower case "e". The 68k_Ser was spinning on DSP "A", i.e. DSP "A"'s green light was ON indicating VME activity. The abort came out at $96AD4. $96AD4 means that 68k_Ser was in the "un-stick" DSP routine. It got there because Global DSP did not reach Step D3 with in the time limit permited. It sent an IIOF2 interrupt to all of the DSP's and then looked to verify that all DSP's were at Step D0. It spun at $96AD4 waiting for DSP A1 to reach Step D0. Why did 68k_Ser think that there had been a "That's_Me" event ?? Why did not the IIOF2 interrupt bring DSP A1 to Step D0 ?? I VME hardware reset the L15CT VME crate a number of times watching the display for all 3 DSP's and the 68k. Reloaded the 68k and they reloaded triggers and began a run OK. So, we could only get an "e" if 68k_Ser thought that it was a That's_Me event. But it clearly should have only been a "i" or "I" event. If this another case of IRONICS getting mixed up, i.e. some how 68k_Ser was reading That's_Me when none of the logic was thinking That's_Me. OK, being awake now finally I looked more at TRICS Log. It is clear that we were never initialized. Thus, I expect that all of this has been a joke. I expect that if we had been initialized and re down loaded at a little after 3AM that everything would have taken off just fine. It is very likely that DAQ experts do not know what we mean by "initialize L1". It's very likely that they think reloading triggers is "initializing L1". So I screwed up and don't know the answer; were the VME resets necessary or would an Initialization and down load to L1 gotten things working? L1 Pedestal File Joan noticed in the examine plots that a couple of pedestals had drifted in L1 Cal Trig. -20,9 EM had drifted up a count. Its pedestal file value had been 20; it was set by hand to 17. -7,9 EM had drifted down a count. Its pedestal was 25; and it was hand set to 27. .............................................................................. DATE: 8-DEC-1995 AT: Fermi TOPICS: Power Supply problem and work, Talk with people about W->JJ and L15CT. About 3AM on DEC 8th M105 Upper Tier 1 that was just installed died again. So PDM-19 was pulled out and PDM-10 installed. Both PDM-12 and PDM-19 were repaired and tested. Now they are in the spares cabinet at Fermi. .............................................................................. DATE: 7-DEC-1995 AT: MSU TOPICS: M105 Upper Tier 1 Power Pan failure, TCC problem: pQBA hangs. Plus 5V Power supply failure in upper tier 1 supply in rack M105. Dan Owen replaces the power pan with the spare at Fermi. Dan Owen and Mike Matulik pulled out PDM-12 and installed PDM-19. All L1CT power pans were turned off one at a time then turned back on one at a time after the work was done. The shifters took this oportunity to perform special runs reading the central detectors only. (probably) Right after the system was back on, all TRGMON sessions were dumped, and could not be restarted. COORs requests to TCC were not being serviced. Here is what could be reconstructed by looking at logfiles: 11:52 COOR Initialized TCC with Caltrig not turned on this must be to switch to the special run 11:54 Initialization completes with a bunch of errors since most of the hardware is turned off, but no problem here ... Everything is fine, COOR sends requests and TCC can perform IOs to the hardware all ok. 13:59 "SOMETHING" happens preventing all subsequent IOs from TCC to L1FW+CT 14:02 the Monitoring server, more exactly ITC, dumps all TRGMON connections and quits accepting new ones. The error in TCC's Mpool_server was ITC-E-NO_CHANNEL, Channel requested has not been activated COOR keeps sending requests to unwind from the special run. TCC is bogged down by slow and failed IOs as TCC it get control of the L1FW+CT bus (remember this bus is shared between TCC control and high speed event readout). Each individual IO has to time out (7s): this takes forever. 15 mn later TCC hadn't even got to the INITIALIZE request from COOR. Now for what the "something" was: we believe the ZRL pQBA interface in the BA23 QBUS enclosure tied to the microVAX 4000/60 TCC must have gotten corrupted. Flipping on power supplies in L1 CT might have been correlated to this. The pQBA stayed hosed until it could be reset. If the first message from COOR after the pQBA was hosed had been INITIALIZE we would have never been bothered, as step one is for TCC to reset the pQBA; unfortunately the first message was to deallocate the special run triggers. As to explain the ITC problems: I cannot know for sure, but it is not UN-believable that some timeout occured in ITC while TCC was dealing with the pQBA symptoms. The special thing that happens is that TCC asks for ownership of the L1 FW+CT bus and waits for a go signal. Since this bus is a highly time critical resource, TCC does NOT use an interrupt mechanism to find out when the bus is granted, but does register IOs on the status word at 100% duty cycle for up to 7 seconds. This is what could have bothered ITC. Whatever initial problem happened with ITC, it was then dead, but I was able to kill and restart the monitoring server with no problem. Conclusion: Turning L1CT power supplies back on one at a time was the gentlest thing that could be done, and it is not clear why it made the pQBA hang, but the symptoms exactly match this hypothesis. .............................................................................. DATE: 5-DEC-1995 AT: MSU TOPICS: Run Find DAC On 4-DEC-1995 run find DAC over all eta with no failures. The few changes are: DAC_BYTE increment -2 for HD,POS,E_2,P_7 40->38 DAC_BYTE increment -2 for HD,NEG,E_1,P_9 38->36 DAC_BYTE low 20 for EM,NEG,E_20,P_9 (was 22) DAC_BYTE increment -2 for EM,NEG,E_20,P_9 22->20 DAC_BYTE increment 2 for HD,POS,E_1,P_14 37->39 DAC_BYTE increment -2 for HD,POS,E_1,P_15 35->33 DAC_BYTE low 23 for EM,POS,E_16,P_15 (was 23) DAC_BYTE increment -2 for HD,POS,E_17,P_15 33->31 DAC_BYTE increment 2 for HD,POS,E_1,P_16 40->42 DAC_BYTE increment 2 for HD,POS,E_2,P_21 36->38 DAC_BYTE increment -2 for EM,NEG,E_16,P_22 37->35 DAC_BYTE increment -2 for HD,NEG,E_16,P_22 37->35 DAC_BYTE increment 3 for HD,POS,E_3,P_28 38->41 2560 towers have been examined 7 tower(s) incremented by -2 163 tower(s) incremented by -1 2284 tower(s) incremented by 0 102 tower(s) incremented by 1 3 tower(s) incremented by 2 1 tower(s) incremented by 3 On 5-DEC-1995 finally get the new Init_DAC_Bytes.LSM file onto TCC. .............................................................................. DATE: 28-NOV:1-DEC -1995 AT: FERMI TOPICS: Spec Trig #6 problems, L15CT ERPB MTG, M124-M125 Safety System, Trigger Tower -15,10 EM, Power Pan in Lower M108 Tier 1, Run II Level 2 talk, The problem with Spec Trig #6 was thought to be a bad signal to the 10H125 input receiver on the And-Or Card for Terms 128:255 of this Spec Trig. All of the solder connections and ElectroCircuit traces in this area of this card looked OK. So I pulled this card and replaced it with a spare, I pulled And-Or Card SN#7 and installed SN#70. This card is in the middle of the stack but it is lots easier to uncable the front CBus from the top down and not from the bottom up as this avoids pulling the connector out of the IML card which does not have ejection ears. Two smoke detectors were added to M124 and M125 and connected to RMI in the top of M124. This RMI is connected to FIRUS. This RMI output was connected to the Intergraph Power Controller-Distribution box from the MSU Test Rack. The RMI was set to enable it to use the smoke detector input: W W D A D A Sensor R S A T R S A T Local Input I M I E I M I E Fault Enable P K R R P K R R Enable N F F N N N N N As found "N" = ON N N F N N N N N Set to this "F" = OFF The Intergraph Power Controller-Distribution box is plugged into the Contractor Box above and for M102. So, M102's Contactor Box now supplies both the 208V 3 phase to the M125 36x36 Scaler Power Pan and also the 115V power to the L15CT VME crate in M124 and the L15CT Power Pan in M125. A separate 115V input to the Intergraph Controller Distribution Box was not allowed. The rule says that a box can have only one power cord. Both smoke detectors were tested and it was verified that they trip the RMI, the FIRUS, the D-Zero Alarm System, and the Intergraph Controller-Distribution Box. For these smoke detectors it takes a 3 or 4 second spray of smoke to get them to trip off (i.e. not the 3 nanosec spray for the VESDA). Replaced the L15CT ERPB MTG with one that has PROM Adrs-Clk line series terminating 47 Ohm resistors. I pulled out MTG SN#21 and installed MTG SN#20. Trigger Tower -15,10 EM is giving only 50% response. It was a bad Term-Attn Network. Replaced it. The lower Tier 1 Power Pan in M108 continues to make a Buzz sound. It looks like the -4.5V brick is dying. It has some AC and is reading only -4.533V So from M108 lower T1 pull PDM-10 and install PDM-25. PDM-25 was a spare stored at Fermi. Besides a new brick, PDM-10 needs the proper kind of terminal strips installed, and a proper pilot lamp. I expect that PDM-10 should return to MSU. Gave a talk in the Run II Level 2 Meeting about "Serialization" and Recycling". People have the idea of having L2 be only the Global part and each detector system would have its own private hardware box to make its list of objects to send to the L2 system. This is both to get around the serialization problem and the and "help hold cost down". .............................................................................. DATE: 26-NOV-1995 AT: MSU TOPICS: Trouble with Spec Trig #6 From: MSUPA::EDMUNDS "Daniel Edmunds" 26-NOV-1995 16:33:14.94 To: FNALV::DENISOVD,FNBIT::D0HSB::DAQEXP CC: EDMUNDS Subj: Current understanding about Spec Trig #6 this morning Hi, Although I do not understand the details my best estimate is the following: The "Level 1" scalers (e.g. L1 And-Or Count and L1 Fired Count) for Specific Trigger #6 have a problem with the configuration in the config file that you were running this morning. Neither of these scalers increments LIKE IT SHOULD with this configuration. Thus, in the "Global Level 1 Framework" display of TrgMon you see zero for Spec Trig #6 in the Firing Rate and the AndOr Rate columns. You see ****** in the Events Transferred column for this Spec Trig because the number displayed here is the L1 Fired Count MINUS the L1.5 Reject Count for this Spec Trig. Because the fired count is zero and the L1.5 Reject count is a slowly increasing positive number and because of 2's complement arithmetic; the value that TrgMon calculates is about minus 2 billion and getting slowly less negative. So it displays ******. I will now include 4 clips from TrgMon from this morning. The first is just the normal "Global Level 1 Framework" display showing the zeros and the ****** for Spec Trig #6. The next three clips are from the Level 1.5 Spec Trig display from TrgMon for Spec Trig #6. The quantities that I'm looking at are: Level 1.5 Confirm Count and L1.5 Reject Count. Over the 36 seconds of data that I have in these three clips; the values for these two quantities look rational. It shows that at Level 1 Spec Trig #6 was firing at about 2 Hz and that at L1.5 events from Spec Trig #6 were being confirmed and passed up to Level 2 at about 0.14 Hz. About 93% of the L1 Spec Trig #6 firings were being rejected by L1.5. This looks similar to Spec Trig #7 (and the Spec Trig #4 #5 pair). These clips show about 385 events from L1 Spec Trig #6 having been Confirmed by L1.5 (and thus transferred up to Level 2. So I do not understand what the "L1 Scalers" for Spec Trig #6 do not like about this configuration but I do believe that you were receiving events from this Spec Trig. I'm coming to Fermi on Monday and will continue the investigation then. Dan Four TrgMon clips follow: Global Monitoring of All Allocated Specific Triggers 26-NOV-95 11:43:37 ( for L1.5 Display) Integr Time DBSC/SBSC: 5.0/ 5.0 s Global Event Transfer Rate: 132.02 Hz Level 1: Running Information: Fresh Global Level 1 Trigger Rate: 452.69 Hz Fast L0: 20.8kHz SlowZ_L: 0.60 E30 Level 1.5 Input/Reject: 335.8 Hz/95.5 % Dead Beam X During Level 1.5: 0.5 % Time Since Last Initialize: 0 07:09:57 Events Transf Since: 2043832 |Tot|Tot |Total| Sp.|Firing| Andor|Prscl|L 1.5|Events|Globl|F-End|Level|And|Strt|Watch| Trg| Rate| Rate|Ratio|Rejct|Transf|Expos| Busy|2 Dis|Trm|Dgtz|Busy | ---|----Hz|----Hz|-----|----%|------|----%|----%|----%|---|----|-----|------- 1 |338.02|346.07| 1| 95.4| 18475| 94.8| 4.1| 0.5| 7| 31| 31|L1.5 2 |240.90|247.62| 1| 95.8| 12929| 94.8| 4.1| 0.5| 10| 31| 31|L1.5 3 |113.47|115.89| 1| 91.9| 10142| 94.8| 4.1| 0.5| 7| 31| 31|L1.5 4 | 12.16| 12.83| 1| 96.2| 2095| 94.8| 4.1| 0.5| 8| 31| 31|L1.5 5 | 3.99| 4.21| 1| 87.5| 843| 94.8| 4.1| 0.5| 8| 31| 31|L1.5 6 | 0.00| 0.00| 1| 0 |******| 94.8| 4.1| 0.5| 8| 31| 31|L1.5 7 | 1.20| 1.40| 1|100 | 312| 94.8| 4.1| 0.5| 8| 31| 31|L1.5 8 | 10.77|108.27| 10| 0 | 12047| 8.2| 4.1| 0.5| 7| 31| 31| 9 | 62.42| 63.96| 1| 0 | 65576| 94.8| 4.1| 0.5| 7| 31| 31| 10 | 4.59|961.61| 200| 0 | 5049| 0.4| 4.1| 0.5| 7| 31| 31| 13 | 0.00| 0.00| 1| 0 | 17| 94.8| 4.1| 0.5| 4| 31| 31| 16 | 1.60| 1.80| 1| 0 | 2725| 94.8| 4.1| 0.5| 9| 31| 31| 17 | 0.80| 0.80| 1| 0 | 1002| 94.8| 4.1| 0.5| 12| 31| 31| 18 | 2.99| 3.21| 1| 0 | 2505| 94.8| 4.1| 0.5| 8| 31| 31| 19 | 16.95| 16.84| 1| 0 | 20295| 94.8| 4.1| 0.5| 11| 31| 31| 20 | 34.30| 34.69| 1| 0 | 40243| 94.8| 4.1| 0.5| 7| 31| 31| 21 | 0.60| 74.19| 130| 0 | 599| 1.4| 4.1| 0.5| 6| 31| 31| 22 | 0.40| 64.16| 130| 0 | 510| 0.8| 4.1| 0.5| 6| 31| 31| 23 | 0.80| 0.80| 1| 0 | 732| 94.8| 4.1| 0.5| 9| 31| 31| 24 | 0.40| 0.40| 1| 0 | 260| 94.8| 4.1| 0.5| 11| 31| 31| 25 | 3.19| 31.88| 10| 0 | 3029| 7.0| 4.1| 0.5| 9| 31| 31| 26 | 2.79| 27.27| 10| 0 | 2585| 7.9| 4.1| 0.5| 11| 31| 31| 27 | 0.00|581.06| 100k| 0 | 7| 0 | 4.1| 0.5| 6| 31| 31| 28 | 1.20|276441| 250k| 0 | 1156| 0.0| 4.1| 0.5| 4| 31| 31| 29 | 1.20|276441| 250k| 0 | 1143| 0.0| 4.1| 0.5| 4| 31| 31| 30 |271484|276441| 1| 0 |******| 94.8| 4.1| 0.5| 4| 31| 31| 31 | 0.00|286275| 1| 0 | 2401| 0 | 3.3| 0.5| 0| 1| 1| Specific Trigger # 6, Level 1.5 Display 26-NOV-95 12:18:33 ( for L1.5 Display) Integr Time DBSC/SBSC: 5.0/ 5.0 s Global Event Transfer Rate: 124.37 Hz Level 1: Running Information: Fresh Spec Trig Fired Count: -1G/1073735036 Spec.Trig.Fired For This Event: No Specific Trig Level 1 Rate: 0.00 Hz This Event Used Level 1.5: No SpTrg L1.5 Input Rate: 0.0 Hz= 0 % L1.5 Skip Rate: 0.0 Hz= 0 % Sptrg L1.5 Confirm Rate: 0.2 Hz=***** % L1.5 Reject Rate: 1.8 Hz= 0 % Dead Beam X During Level 1.5: 0.0 % SpTrg L1.5 Exit by Timeout: 0 % Level 1 Fired Count: 0 SpTrg Expos. Cnt: 854269823 Level 1.5 Input Count: 0 L1.5 Skip Count: 0 Level 1.5 Confirm Count: 380 L1.5 Reject Count: 6788 Integrated Dead Time: 0 % SpTrg L1.5 Exit by Timeout: 0 L 1.5 Term #: 0.1.2.3.4.5.6.7.8.9.0.1.2.3.4.5.6.7.8.9.0.1.2.3.4.5.6.7.8.9.0.1 Term Required: T Term Don Stat: T T T T T F F F F F F F F F F F F F F F F F T F F F T F F F F T Term Ans Stat: F F F F F X X X X X X X X X X X X X X X X X T X X X T X X X X T Specific Trigger # 6, Level 1.5 Display 26-NOV-95 12:18:46 ( for L1.5 Display) Integr Time DBSC/SBSC: 5.0/ 5.0 s Global Event Transfer Rate: 129.51 Hz Level 1: Running Information: Fresh Spec Trig Fired Count: -1G/1073735012 Spec.Trig.Fired For This Event: No Specific Trig Level 1 Rate: 0.00 Hz This Event Used Level 1.5: No SpTrg L1.5 Input Rate: 0.0 Hz= 0 % L1.5 Skip Rate: 0.0 Hz= 0 % Sptrg L1.5 Confirm Rate: 0.0 Hz= 0 % L1.5 Reject Rate: 1.4 Hz= 0 % Dead Beam X During Level 1.5: 0.0 % SpTrg L1.5 Exit by Timeout: 0 % Level 1 Fired Count: 0 SpTrg Expos. Cnt: 858376308 Level 1.5 Input Count: 0 L1.5 Skip Count: 0 Level 1.5 Confirm Count: 382 L1.5 Reject Count: 6812 Integrated Dead Time: 0 % SpTrg L1.5 Exit by Timeout: 0 L 1.5 Term #: 0.1.2.3.4.5.6.7.8.9.0.1.2.3.4.5.6.7.8.9.0.1.2.3.4.5.6.7.8.9.0.1 Term Required: T Term Don Stat: T F F T F F F F F F F F F F F F F F F F F F T F F F T F F F F T Term Ans Stat: F X X F X X X X X X X X X X X X X X X X X X T X X X T X X X X T Specific Trigger # 6, Level 1.5 Display 26-NOV-95 12:19:09 ( for L1.5 Display) Integr Time DBSC/SBSC: 5.0/ 5.0 s Global Event Transfer Rate: 132.29 Hz Level 1: Running Information: Fresh Spec Trig Fired Count: -1G/1073734962 Spec.Trig.Fired For This Event: No Specific Trig Level 1 Rate: 0.00 Hz This Event Used Level 1.5: No SpTrg L1.5 Input Rate: 0.0 Hz= 0 % L1.5 Skip Rate: 0.0 Hz= 0 % Sptrg L1.5 Confirm Rate: 0.4 Hz=***** % L1.5 Reject Rate: 3.2 Hz= 0 % Dead Beam X During Level 1.5: 0.0 % SpTrg L1.5 Exit by Timeout: 0 % Level 1 Fired Count: 0 SpTrg Expos. Cnt: 863857592 Level 1.5 Input Count: 0 L1.5 Skip Count: 0 Level 1.5 Confirm Count: 385 L1.5 Reject Count: 6862 Integrated Dead Time: 0 % SpTrg L1.5 Exit by Timeout: 0 L 1.5 Term #: 0.1.2.3.4.5.6.7.8.9.0.1.2.3.4.5.6.7.8.9.0.1.2.3.4.5.6.7.8.9.0.1 Term Required: T Term Don Stat: T T T T T F F F F F F F F F F F F F F F F F T F F F T F F F F T Term Ans Stat: F F F F F X X X X X X X X X X X X X X X X X T X X X T X X X X T .............................................................................. DATE: 6:10-NOV-1995 AT: FERMI TOPICS: Repair M102 FW Power Pan, Spare 200 Amp brick to Fermi, Repair +15,28 EM pedestal drift, Work on MTG's, PAL PROM List from MTGs, Run Find DAC, Repair a CAT2 and a CTFE, work on -5,13 EM and -13,3 EM. They sent mail on Saturday 4-NOV-1995 to say the the M102 FW +5V had gone into alarm. I could open the rear door without causing a trip and I measured the voltage on the monolith terminals. The +5V supply was about 4.75 Volts. I pulled the Power Pan from M102 FW and it was the #1 +5V brick that was dead. It puts out about 4.2 Volts no load and falls with load. This brick is SN#296343 and it is a "old" type brick. The monolith in M102 is FW Monolith #1. The bricks in it are: Supply Brick SN# and type -------- ----------------------------- #1 +5 2500A-2 5D200-0-4-6 SN# 296343 No MSU ID Number or Date #1 -5.2 2500A-2 5D200-0-4-6 SN# 296344 No MSU ID Number or Date #2 +5 2500A-2 5D200-0-4-6 SN# 340239 MSU ID #4 20-JULY-1988 #2 -5.2 2500A-2 5D200-0-4-6 SN# 340256 MSU ID #1 20-JULY-1988 The #1 +5V brick was pulled and replaced by the spare type 2500A-2 brick that has been stored in the kitchen cabinet at Fermi for some years. This is 2500A-2 5D200-0-4-6 SN# 355005 and is labeled MSU SN#9 from 9-NOV-1988. The #1 supplies from this Power Pan connect to 000 cables that go to the top of the M102 power bars. The #2 supplies connect to the bottom of the power bars via 000 cables. The 000 cables to the top are 85" long and the cables to the bottom are 55" long. So how should you set the power supply voltages? Estimate that 000 cable has 0.000063 Ohm/ft. This comes from knowing that #6 wire is about 0.000403 Ohm/ft and that to go down one gauge number you multiple the resistance per foot by 0.793 So assuming 100 Amp loads the short bottom cables will drop about 29 mV and the long top cables will drop about 45 mV. For the +5 supplies lets add an extra 25 mV to guarantee that they get +5V. So the target voltages for the bricks under a 100 Amp test load will be: #1 Top +5 is 5.070 Volts -5.2 is 5.245 Volts #2 Bottom +5 is 5.054 Volts -5.2 is 5.229 Volts Now put this all back together, let it warm up and measure the voltages: M102 ----------------------------------------- #1 Top #2 Bottom ----------------- ----------------- +5 -5.2 +5 -5.2 ------- ------- ------- ------- Front Panel 5.071 5.248 5.057 5.234 Rear Lugs 5.058 5.239 5.041 5.220 Measure the voltages on the M102 power bars, supply bar WRT ground return bar: Top of Bars +5 is 5.019 -5.2 is 5.190 Middle of Bars +5 is 5.016 -5.2 is 5.193 Bottom of Bars +5 is 5.016 -5.2 is 5.193 Now we are at it, check the M101 voltages at the lugs at the rear of the M101 Power Pan: #1 +5 is 5.030 #1 -5.2 is 5.234 #2 +5 is 4.999 #2 -5.2 is 5.206 Brick SN# 87 The +5 200 Amp brick MSU SN# 87 was taken to Fermi and left in the kitchen cabinet as a spare. This brick was repaired and returned to us on 10-AUG-1995. Pedestal +15,28 EM Before the shutdown the channel +15,28 EM had suffered from slow pedestal drift. This is CTFE SN# 292. I replaced the normal 0.1 uF monocap. I could not see with the Ohm meter and soldering iron heat test that there was any trouble with this channel (but its drift had been very slow). MTG Work Pull the currently installed L1 Cal Trig MTG. This is MTG SN#20. The PROM's are: CalTrg1M, CalTrg2L, CalTrg3K, CalTrg4M, The PAL's are: Ch#1 : Ch#4 are MTGBit2 Ch#5 is a L15MTG06 Ch#6 : Ch#8 are MTGBit2 Ch#9 : Ch#14 are CTMapSel Ch#15 : Ch#28 are MTGBit2 Ch#29 is a L15MTG05 Ch#30 : Ch#32 are MTGBit2 The Terminator packs are installed at J4 all 4 of them and none at J5. There are no header jumper wires. Install these parts on MTG SN#22. The PROM's and the fancy PALs were actually moved from MTG SN#20 to MTG SN#22. Most of the Bit2 PALs were already on MTG SN#22 and are of the new fast BCN type. Pull the currently installed FW Main MTG which is SN#23. The PROM's are: FWMT1R, FWMT2L, FWMT3N, FWMT4L, The PAL's are: Ch#1 is a MTGBit2 Ch#2 is a L15MTG06 Ch#3 is a MTGBit7 Ch#4 is a L15MTG05 Ch#5 is a MTGBit9D Ch#6 is a MTGBit2 Ch#7 is a L15MTG06 Ch#8 is a L15MTG05 Ch#9 is a L15MTG06 Ch#10 is a MTGBit2 Ch#11 is a MTGBit2 Ch#12 is a MTGBit2 Ch#13 is a L15MTG05 Ch#14 is a FWMTCH14 Ch#15 is a BX2Skip Ch#16 is a MTGBit2 Ch#17 is a BX2Skip Ch#18 is a BX7Skip Ch#19 is a L15MTG05 Ch#20 is a L15MTG05 Ch#21 is a L15MTG05 Ch#22 is a MTGBit2 Ch#23 is a MTGBit2 Ch#24 is a MTGBit2 Ch#25 : Ch#32 are MTGBit2 The wire wrap wire on some of the special PAL's, e.g. the RA8's, is beginning to tarnish All of J4 has a fancy header and the 1st half of J5 has a fancy header. The second half of J5 has terminators. Because of the complexity of moving parts on this card with lots of hand wired PALs, this card will has the address line series terminators added directly to it. Pull the window 0:31 MTG which had been MTG SN#13 and replace it with MTG SN#24 which has the PROM ADRS line series terminators installed. Now add the series terminators to MTG SN#13 which was just pulled from service. MTG SN# 13 stays at Fermi in the spares cabinet. Spare MTG SN#15 from Fermi will come back to MSU to have resistors installed and to make sure that there is not a CBus clobber problem with this MTG. (see last weeks entry). Schedule as of 8-NOV-1995 Nov. 19 Collider Startup for 900 GeV Running. Dec. 22 (or slightly earlier) Start Holiday Shutdown. Jan. 3 Start 315 GeV Operation Jan. 14 Start 900 GeV Operation. Feb. 4 Continue 900 GeV/ Studies Before turning on the temperatures were: 60.4 59.4 59.0 60.5 60.2 59.8 and after running for a short time the water was 56.3 Now that it has run for a couple of hours the temperatures are: and the water temperature is 58.5 - 58.8 Power Supply Concerns: M107 Upper T1. This is the Power Pan that has trouble getting its +5 Volt brick running. Normally it only take one turn on then turn off for 5 seconds then turn back on to get the +5 V in this Pan running. At the second turn on the +5 V comes up right away followed 1 second later by the other 3 supplies. When the +5 V does not start, there is a -0.3 to -0.4 Volts across this supply. I assume that this supply looks out and sees this and wisely decides not to start. This time the normal two turn on process did not work. I ended up un bolting the +5 V cables, running the pan for 15 seconds (during which the +5 V brick ran fine) turning the pan off and reconnecting the +5 V cables and then using the two step process to get it started. I can not just replace a brick because as far as I can tell all 4 bricks in this pan are OK, they just come up in bad time order. M108 Lower T1 is making a buzzing sound. I think it is the -4.5 brick as this shows a little AC voltage at its output. Repair CAT2 SN#259 from HD Sum eta -9:-12 phi 25:32 This CAT2 card had bit of value 1 and bit of value 4 stuck together only on FA=23. The problem was in the resistor network R182 by U136 which terminates these lines. They had about 1.5 Ohms between them. The Ohm meter showed that the problem was in the resistor pack and not in traces or IC's. Heating pin #7 of this pack would make the problem go away until it cooled down. Pin #7 is bit of value 4 and bit of value 1 is on pin 5. Pin #8 is not used. So I did the very crude thing of cutting off this section of the pack. This must have mechanically shocked the pack enough that the short went away. Heating the pins of the pack did not make the problem come back or the resistance change. So all is OK now ? This resistor pack was a blue green Dale unit. I have turned out stuff off two time on this trip to Fermi. One time I used the Lower Limit Set Point on the PhotoHelix to trip the system and another time I used the drip detector from M101 or M102. So I have not tested the VESDA but I think that was done this summer. I have not tested the global upper limit trip but that was "tested" last spring. We have been running with 58.8 degree water. They found the cause of the hot cooling water. The actuator was not attached to the mixing valve. The computer readback of the actuator position was out of calibration. The control loop computer thought that the actuator was 100% cold when it realy was about 50% cold and 50% recirculated water. Make a Find DAC Run and find: Failure finding DAC byte NEG,E_13,P_7 EMETZ0 for ADC zeresp 8 within 19:45 HD looks like it may be up at 49 or so Failure finding DAC byte NEG,E_13,P_7 HDETZ0 for ADC zeresp 8 within 19:45 EM looks like it may be up at 49 or so Failure finding DAC byte NEG,E_14,P_7 EMETZ0 for ADC zeresp 8 within 19:45 HD looks like it may be up at 49 or so Failure finding DAC byte NEG,E_14,P_7 HDETZ0 for ADC zeresp 8 within 19:45 EM looks like it may be up at 57 or so The -1.000V supply on the CTFE card for eta -13:-16 Phi 7 was only -0.970 V. This was caused by a shorted monocap on the -1V rail of the -1V feed to the ADC for Ch#2 HD. This is CTFE SN# 307. Dan Owens pulser run analysis shows -5,13 EM at 3x and -13,3 EM at 50%. I can see nothing wrong at -5,13 EM using the hand test pulser and scope. A later pulser run is taken and -5,13 EM looks all OK. At -13,3 EM the L1 end of the cable has a short. I repaired this. The normal run 1 b version of VTC code was returned to the file RunMe68020.abs and loaded into the machine. The Trics_Boot_Auxi was returned to the full eta coverage. The 1 trigger tower version was named xxx.Dat_1_Trig_Tower and left on TCC's disk for use in Feb 1996. A second find DAC run was made with no failures. The compare between this second run a what had been in use shows: DAC_BYTE increment -5 for EM,NEG,E_14,P_7 34->29 <-- Note that the -1V DAC_BYTE increment -5 for HD,NEG,E_14,P_7 42->37 <-- Ref MonoCap started to die some time ago. DAC_BYTE increment 5 for EM,POS,E_17,P_15 31->36 DAC_BYTE increment 4 for HD,POS,E_17,P_15 29->33 DAC_BYTE increment 3 for HD,NEG,E_16,P_22 34->37 DAC_BYTE increment 6 for EM,POS,E_15,P_28 26->32 DAC_BYTE increment -4 for EM,NEG,E_20,P_9 26->22 DAC_BYTE low 22 for EM,NEG,E_20,P_9 (was 26) DAC_BYTE low 23 for EM,POS,E_16,P_15 (was 23) 2560 towers have been examined 2 tower(s) incremented by -5 1 tower(s) incremented by -4 9 tower(s) incremented by -2 137 tower(s) incremented by -1 2243 tower(s) incremented by 0 156 tower(s) incremented by 1 8 tower(s) incremented by 2 1 tower(s) incremented by 3 1 tower(s) incremented by 4 1 tower(s) incremented by 5 1 tower(s) incremented by 6 .............................................................................. DATE: 22,23,24,25,26-OCT-1995 AT: FERMI TOPICS: Install the 36x36 Bunch Scalers. Need a new spool of duct tape at Fermi. Jan ask us to turn L1 Cal Trig back on next week. Need to install the new drip detector. Installed the equipment in the new short rack M125. The previously existing L15CT Power Pan was installed on the bottom, next is the new M125 Power Pan, and at the top of the 36x36 Bunch Test Scalers. The M125 Power Pan gets its 3 phase power from the M101 Contactor Box. Edited the Trics_Init_Auxi file to include the setup of the 3 MTG's in the 36x36 Scaler backplane. Copied the new Trics_Init_Auxi file to TCC:: and to MSU::TrgCur:[DZero]. The somewhat flaky MTG that was running making the W0:W31 SBSC Gates did not look like it was doing a good job of this so it was replaced by a spare one from Fermi. During this it was discovered that a (in principal good) spare MTG at Fermi clobers the CBus if you try to use it. This is MTG SN# I will not bring it back to MSU yet as there is nothing much else here in the way of spares since I used the other spare in the W0:W31 location. In the TrgCur: directory I added the subdirectory {.Non_Critical_But_Save] From TrgCur: I moved into this new subdirectory all of the L15CT*Pallet* files, the Tree_offset_ets_16.dat file, and the Load_12_parameters.Abs file. Still some problem with the W32:W63 MTG. W32 which happens to stradal the PROM Counter carry from 511 to 512 appears to sometime glitch and have an extra edge in its gate signal. Is this a sick MTG or a characteristic of all MTG's ? This is the same as we saw in the original L15CT MTG. If it is the MTG then possible problems include: a carry problem in the PROM Address Counter, noisy PROM Address Lines, noisy Clk to the PROM's. I tried experimenting with noise on the PROM Clock line by putting in a series 27 Ohm resistor at the driving end and then a series 220 Ohme 56 pF cap at the far end. This made no diference at all. Right now both of the gate generator MTG's have the 27 Ohm plus 56 pF 220 Ohm treatment on their PROM Clock lines. This appears to make no difference. Note, if the problem is a 511 --> 512 problem then it is not a BX force counter preset problem, rather it is a counter carry problem. Is it the 10H104 in the carry circuit? Sometime during the past two weeks some brain dead idiot strung a bundle of red HV cables right through our And-Or Input Term patch panel on top of M101. Thats right, its strung and cable tied right through the extension on top of M101. Sources of the L0 signals for the 36x36 scalers NIM-ECL Pair on Module Lemo the 17 Pair on the M114 Connector Pair cable Signal to M124-M125 Cable ------------ ------------ ------------ ------------------ NT-NT-Top 15 P Halo 8 NT-NT-NT_Top 14 /P Halo 10 Bottom 2 L0 Fast Z 6 Pair on the M114 to M124-M125 Cable Function - Signal ------------------ ----------------------------- 1 N.C. 2 P6 Beam Crossing Marker 3 N.C. 4 Master Clock to the MTG's 5 N.C. 6 Luminosity to 36x36 Scalers 7 N.C. 8 P Halo to 36x36 Scalers 9 N.C. 10 /P Halo to 36x36 Scalers 11 N.C. 12 N.C. 17 N.C. See also the logbook for 22,23-MAR-1994 and 14-JULY-1994 for details of all of the connections from L0 into our scalers. We had to start the gates to the SBSC's almost 1 usec after BX to get the increment signals from L0 "timed in correctly" .............................................................................. DATE: 6-OCT-1995 AT: FERMI TOPICS: ECB Meeting, Dan and Steve go to the ECB meeting. Topics discussed include: changes to the Serial Command Link, Operation at 132 vs 396, triggering during gaps, Initialization, VBD count and amount of data from each Geographic Section. .............................................................................. DATE: 8-SEPT-1995 AT: FERMI TOPICS: ECB Meeting, Dan and Steve go to the ECB meeting. The requested changes to the Serial Command Link are presented and discussed. .............................................................................. DATE: 21,22-SEPT-1995 AT: FERMI TOPICS: Group meeting, Luminosity meeting, meeting with Jerry Blazey, deliver the C compiler Delivered the C compiler to Dan Owen. Specifically it is: 320 FLT-PT DSP OPTIMIZING 'C' COMPILER TMDS 3243855-02 PC Release 4.60 2617565-0741 REV C 25 AUG 95 It has 6 books: TMS320 Floating-Point DSP Code Generation Tools Getting Started Release 4.60 TMS320C3x Peripheral Control Library User's Guide Miroprocessor Development Systems Customer Support guide TMS320C4x Parallel Runtime Support Library User's Guide TMS320C4x Floating-Point DSP Optimizing C Compiler User's Guide TMS320C4x Floating-Point DSP Assembly Language Tools He is basically finished playing with the ATLAS stuff but he now will play with the Run II Level 2 C vs Assembler stuff. Presented the Processor Based Run II Level 2 Trigger stuff at the Group meeting and went over the paper with Jerry Blazey. It continues to look very much like we have the project. Jim Christenson wants a cost estimate by next week. Jerry has the following ideas for the note: Single system vs Multi system, drop the 30% argument and just put in what things go into deciding between the two. Find some way around the port limit. Need stuff about time out waiting for data and getting data from the right event. Fix the appendicies, conclusions, define terms Why the nodes need to wait for the Framework to advertize the decision. Also keep talking to Jerry about when we are ready for the next meeting and get the money estimate to him. We need to make 3 sets of 36 scalers to monitor 36x36 luminosity, p loss, and p bar loss. Dan Owen and Del will make 40" vertical inches of rack space available above the air conditioner. Need to install these scalers in about 2 weeks. Start Oct 2nd with one week of 6x0, then two weeks of 36x0. Oct 22 p bar startup and 36 x1. Nov 5 start 36x36 Nov 19 start 6x6 physics running one or two week shutdown at Christmas Jan 3 move to low energy for two weeks. Feb 19 stop everything. .............................................................................. DATE: 23,24-AUG-1995 AT: FERMI TOPICS: Run II Level 2 Trig meeting, meeting about luminosity monitoring at 36x36, meetings with muon people, Power Pan work, meeting with Ron Lipton and Bill Cooper about Si Vertex data. Blazey had a Run II Level 2 Trigger meeting. I presented the Jim Linnemann's queing stuff and the first draft of an impact trigger. They say that Bill Cooper is the person to talk to about F.O. cables from Si Vertex. Had a meeting about monitoring luminosity and looses at 36x36. Mike Tartaglia is going to dig at the AD people more to find out what is really necessary and what the bunch structure will look like exactly. Estimate at least one person month to get this working. Meeting with Boris Baldin. Trying to understand what Muon needs. The big discovery is that they can not readout unless they caused the L1 trigger. Finally repaired and tested PDM-19. Also brought a spare 600 Amp brick to Fermi (SN#23). Now all spare Power Pans are tested and ready. Need to bring Power Pan LED S6 bulbs and more big lug screws. More meetings with muon people: Boris Baldin, Sergi, Sten Hans. They do not want to have a level one trigger number but only want to use the BX number in their low level stuff. The L1 Trig Number will be on the data that comes out of their MAR cards. They want to "Request" L1 triggers vs just having their L1 Muon Trigger send its pattern of terms to the L1 FW each BX. They want to know the difference between someone else causing an L1 Trig and their own L1 Trig being accepted or rejected. Had a meeting with Ron Lipton and Bill Cooper about Si Vertex raw data. Yes, each Si barrel element (what they call detector) has its own flat cable coming out to the Port Card. 4 detector elements go to a Port Card. Each Port Card make one F.O. cable. 4 F.O. cables go to a SAR. There is almost complete freedom in the cabling of flat cables to Port Cards. There is complete freedom in cabling F.O. cables to SAR's. When you look at the end view its clear that you would rather have 6 detector elements go to a Port Card. This does not match the data to the F.O. bandwidth quite as well but Ron and Bill are thinking about it. .............................................................................. DATE: 9,10-AUG-1995 AT: FERMI TOPICS: Run II Level 2 Trig meeting, "trouble" with L15CT, Get the multi channel drip detector, reminder of Run 1b L15CT operation, Talk with Mike Matulik about the serial command link. Blazey and Fortner had a Run II Level 2 Trigger meeting. We presented the extra SAR buffer and Farmlet approach. L15CT was not running. Jan has been saying that Geo Section #5 was hung 100% FEBz since L1 Cal Trig was turned off. By rotating the RS232 switch and pushing the VME RESET it was clear that sometimes the DSP's would load in and run the "wakeup - sanity checker" and sometimes they would not. This was true for all three HYDRA-II cards. Turning off the power pan on top of the air conditioner curred the problem. So, with L1 Cal Trig off and the CRC cards running there is enough junk getting into the DSP Comm ports to cause trouble. I picked up the multi channel drip detector from the people who built it. John Fogelsong. As part of getting ready for the Run II L2 meeting Steve put together a reminder of how we used L15CT for the last 3 or 4 months of run 1B. With Trigger List V10.2 through V10.6 in Run Ib, the setup of the Specific Triggers which actually used the L1.5 CT to obtain rejection was as follows: Spec. Trig. 8: em_1_high L1 Requirements: em(1,10) jt(1,3) L1.5 Requirements: EM Ref Set: 7.0 GeV 1x2 EM Et: 15.0 GeV EM_Fraction: 0.85 Global Count: 1 L1.5 Rejection: ~65% (rejection factor ~3) Specific Trigger 9: em_2_med L1 Requirements: em(2,7) jt(1,3) L1.5 Requirements: EM Ref Set: 7.0 GeV 1x2 EM Et: 12.0 GeV EM_Fraction: 0.85 Global Count: 1 L1.5 Rejection: ~35% (rejection factor ~1.5) With the typical mix of events the overall rejection factor of the L1.5 CT for these Trigger Lists was approximately 2. I talked with Mike Matulik about the Serial Command Link from the Frameworks and the Master Clock to the Geographic Sections. His basic comment is that this is doable without aggressive use of serial link technology. He would like to come to the next ECB meeting. .............................................................................. DATE: 27-JULY-1995 AT: MSU TOPICS: Shutdown L1 Calorimeter Trigger To prepair to turn off the L1 Calorimeter Trigger I changed two files. In TrgCur: at D0:: only RunMe68020.Abs a file from 23-JULY-1994 was renamed to RunMe68020.Abs_Normal_1B_Running. Then RunMe68020.Abs_No_Cal_Trig_Error_Checking was copied to RunMe68020.Abs So next time they load the L1 68k using the standard filename and they will load the version that does not check L1 Cal Trig data. On TCC only, edit the file TRICS_Boot_Auxi.Dat so that L1 Cal Trig will use only one Trigger Tower. This was edited on TCC's disk only. Before I made the edit I verified that the version in TrgCur: at D0:: matched the copy on TCC's disk. The version on TCC's disk that we had been using for normal run 1B running was version 13 from 18-AUG-94. I did not purge the file that we have been using for normal running, so this fall all that we need to do is to delete the version on TCC's disk that I made today. Jan and Joan turned off the power to L1 Cal Trig and booted the L1 68k and TCC. I copied the last of TCC's log files from run 1B to MSU via COPY D0HTCC::[TRIGGER]TRICS_15JUL95.LOG MSU::MSUD01::DUA1:[BACKUP.TCC_LOG_IB] Before L1 Cal Trig was turned off all TrgMon displays for it looked OK. The only problem that I know of in the L1 Cal Trig that will probably need to be fixed when we turn back on in the fall is the +15,28 EM drift. .............................................................................. DATE: 20,21-JULY-1995 AT: FERMI TOPICS: Trouble with Spec Trig #6 and the Glbl_Low_Lum trigger configuration file, Install L15 stuff for E811, Spec Trig #6 New Glb_Low_Lum They started using a new Glb_low_lum trig configuration on the store that started early AM on the 20th. The And-Or rate and the Specific Trig Fired scalers look funny in TrgMon. Other trig configuration files run with no problem. In a quick test between stores it looks like the problem is that Spec Trig #6 does not properly see And-Or Term #212. During with shot setup, with the new Glbl_Low_Lum loaded I played makeing other Spec Trig's be setup the same and #6 and at changing what Terms #6 used. The problem appears to be limited to Spec Trig #6 view of Term 212. If I program Spec Trig's 4,5,6,7 all to look only at Term 212 then 4,5,7 all were running at the exact same rate of about 100Hz and Spec Trig #6 was running between 0 and 90 Hz varying in a random way. Mike Tartaglia says that there is no plan to run low luminosity again before the 24th so I will wait to pull this apart. Level 1.5 from E811 Andrew Brandt was interested in trying to setup the E811 to act as a L1.5 trigger to D-Zero. So I have provided connections to L15 Term #24. They access the Term #24 ANSWER and DONE lines through a the NIM-ECL converter module that they us. I installed an output cable on this module and ran it over our L15CT rack where it, via a couple of short jumper cables, plugs into the "Terms Returned" cable for the lower L15CT crate, i.e. L15 Framework Terms 24:31. So the complete setup for the E811 NIM-ECL module is: 4th from the top ANSWER to L15 Term #24 NIM is input 5th from the top DONE to L15 Term #24 NIM is input 6th from the top Specific Trigger #0 NIM is output 7th from the top Specific Trigger Fired Strobe NIM is output 8th from the top Global Total Et Comparator #3 NIM is output .............................................................................. Date: 11-JULY-1995 we were At: MSU all action AT: FERMI Topic: L15CT fails to evaluate some of its L15CT Terms because L15CT quits correctly seeing which L1 Spec Trigs Fired. They called from Fermi to say that all L2 nodes were crashing with a message something like, "MISMATCH OF TERMS BETWEEN ANDOR AND L15 BLOCKS". This message (which only the code's author could understand) comes from the L2_CONFIRM_L15 tool. What it means is that an impossible condition exists between the L1 List of Spec Trig's Fired and the "6th longword in the Framework section of the L15CT Data Block (i.e. Mask of Terms that were Evaluated, Mask of Terms that Passed Mask of Terms for which their Evaluation is Incomplete, Mask of Terms returned to the L1.5 Hardware Framework). For example L1 says that a given Spec Trig Fired but the L15CT Term associated with it was not Evaluated. Or possibly this could also mean that an impossible condition exists within this set of 4 masks. For example a Term was not Evaluated but its Evaluation is Incomplete. What we think happened is the following: At the very beginning of processing of each event, one of the "P2 Paddle Boards" in L15CT, the Term Select board, is responsible for mapping the L1 Specific Triggers Fired List into the list of L15CT Terms that need to be evaluated for this event. This P2 Paddle board is accessed by an Ironics VME digital I/O card. This Term Select Ironics digital I/O card allows the L15CT 68k processor to read both the List of L1 Specific Triggers Fired and the list of L15CT Terms that need to be evaluated for this event. The Ironics card reads this information over lines that are supposed to be "inputs" to the Ironics. The digital lines to the Ironics are programmable as either inputs or outputs. What we think happened is that the input vs output programming changed on a couple of these lines. We think that the lines that read L1 Specific Triggers Fired List for L1 Spec Trig's 8,9,10 all became outputs locked at the low voltage level. Because of this L15CT could no longer "see" that these L1 Specific Triggers ever fired. So, if for example you had an event where only L1 Spec Trig #8 fired, L15CT would not process it, but it would eventually be passed up to L2. It would be passed up to L2 because the L15 M103 Framework would time out waiting for an answer from L15CT and upon time out the L15 Framework passes the event to L2. Thus you end up with an event in L2 for which the L1 Specific Triggers Fired List in the L1 Data Block says that Spec Trig #8 fired, and L2 has been programmed to know that L1 Spec Trig #8 requires L15CT processing, but when the L15 confirm tool looks into the L15CT data block it does not see anything that makes sense to it so it flags an error. I think this is the same thing that happened on 12-JAN-1995 when L2 started dumping one half of the events because the "trigger number" coming from the L15CT crate was wrong one half of the time. This happened because the L15CT could not properly read the Trigger Acquisition Synchronization Number through an Ironics card. The Ironics card had one line set to output and stuck low. Assuming that our understanding is correct it would only be necessary to download the triggers to TCC again to fix this problem. When TCC loads triggers into the L15CT it resets all of the Ironics cards to their proper state. It should NOT be necessary to boot the L15CT 68k. The L2_CONFIRM_L15 tool needs to have a better message than "MISMATCH OF TERMS BETWEEN ANDOR AND L15 BLOCKS". If they had called at 4AM no one would have known what this ment. Luckily Sal was to MSU to help decode this message. The message should have said, "SPTRG #8 NEEDED L1.5 CAL TRIG TERM #0, BUT THE TERM WAS NOT EVALUATED". Another problem was that L2_CONFIRM_L15 did NOT check FIRST that the l1.5 crate data was valid or not. Flagging that the L1.5 caltrig did not cycle for a specific trigger that needed confirmation would have pointed more directly to the real problem. This test should be carried first, and before trying to interpret irrelevant data and generating complex error messages. We had the experiment running again in about 15 minutes. .............................................................................. Date: 6,7-JULY-1995 At: FERMI Topic: Deliver the Uranium to Fermi Receiving, Work on spare Power Pans here at D-Zero Hall, Drift in +15,28 EM, Check the Cal Trig. Uranium Delivered the Uranium to Receiving which is the big warehouse building at site 38 just North of the Stock Room building. All the the "travel documents" will be given to Harry when he gets back. All of the documents that were attached to the shipping container (i.e. the packet for Kathy Graden) were pulled off by people at Receiving. Power Pans Installed Varistors on the 4 spare Power Pans here at Fermi. These Power Pans are: PDM-19, PDM-25, MM-04, and MM-08. PDM-19 is the Tier 1 Pan that is still waiting for a 600 Amp brick to finish repairing it. Varistors were installed in two places on each Power Pan. On the Pan side of the power entry terminal strip, three varistors were installed. One side of each of these varistors connects to a phase lead and the other side of each varistor are all tied together and connected to the Power Pan chassis - Safety Gnd. On the switch side of the terminal strip that distributes power to the individual brick three more varistors were installed. Each of there varistors is connected between two phase leads. The varistor that has to stradle from one end of the terminal strip to the other has its leads supported on top of the nylon terminal strip cover as it crosses over the center two terminals. All varistors are type: V250LA40A The main problem installing these varistors was screws breaking off in the terminal strips as they were being re-tightened after the varistors were installed. This makes things a big mess to dig out a broken off terminal strip screw. Need to install new screws as the varistors are being installed ! These screws are: ??? So, three of the spare Power Pans down here have a terminal strip with the nylon cut up around one screws so that that contact could be repaired. +15,28 Drift Edit TCC ONLY The EM Trigger Tower at +15,28 had drifted its pedestal up to 9. Its DAC setting was 26. So in 8 or 9 days the pedestal drifted from 8 to 9. This drift looks slow enough that we can control it until July 24, so I will not turn off power and pull the card now. By hand I determine that a DAC value of 22 just gets this Trigger Tower down to 7. So I edited Init_DAC_Bytes.LSM ON TCC DISK ONLY to change +15,28 EM from 26 to 22. Cal Trig Exercise Ran 65k loops of Cal Trig Random Cell with everything enabled except for checking AND-Or terms and locked on lookup page #4. All was OK. .............................................................................. Date: 27,28,29-JUN-1995 At: FERMI Topic: Repair L1 Cal Trig Global quantity readout problem, Replace an ERPB card, Run Find-DAC, Deliver the OS2 IBM/PC to Fermi, Work on Px problem and run Cal Random Cell, work on a couple of Trigger Towers The problem reading out the Global quantities from the L1 Cal Trig was traced to the CTMBD card in the L1 Cal Trig final readout backplane section of the top card file in M103. Pin #16 of U47 was not soldered. This is a socketed 10101 that replaced a 10H101. It drives F7,F8,DIR,STRB. This is CTMBD SN#31. I checked the Timing Signal wire wrap wiring on this card. It is: 6-->A, 29-->B, 4-->C, 7-->S. So it is clear that this card is running from the L1 Cal Trig MTG. This CTMBD wiring fits what is in the [D0_TEXT.TIMING_AND_CONTROL] MBD_AND_CTMBD_TIMING_SIGNAL_WIRING.TXT file, with the 7-->S being added for some unknown reason. As described in this same file, the wiring of the CTMBD for the Level 1.5 Framework backplane clearly indicated that it is ment to run from the the L1 Framework Main Timing MTG. Thus it is clear that in the file TrgIntr: RACK_M103_CARD_ADRS it was showing the incorrect Timing and Sync Signal Cables going to Level 1.5 Framework and the L1 Cal Trig Final Readout. I have edited RACK_M103_CARD_ADRS to fix this. L15CT ERPB Card Replacement In a "Dan Owen Pulser Run" taken last Friday after we had recovered from the power outage, in the High Amplitude run there were two events that showed a problem. They are: Run 92438 Event 16: Data Mismatch Error Eta/Phi +1/ 4 LDSP B1 EM Et Rack 1 Expected $FF Found $12 Eta/Phi +1/ 4 LDSP B4 EM Et Rack 2 Expected $FF Found $12 Eta/Phi +2/ 4 LDSP B1 EM Et Rack 1 Expected $FF Found $12 Eta/Phi +2/ 4 LDSP B4 EM Et Rack 2 Expected $FF Found $12 Eta/Phi +1/ 4 LDSP B1 Total Et Rack 1 Expected $FF Found $23 Eta/Phi +1/ 4 LDSP B4 Total Et Rack 2 Expected $FF Found $23 Eta/Phi +2/ 4 LDSP B1 Total Et Rack 1 Expected $FF Found $23 Eta/Phi +2/ 4 LDSP B4 Total Et Rack 2 Expected $FF Found $23 Run 92438 Event 16: 8 Errors found Run 92438 Event 79: Data Mismatch Error Eta/Phi +1/ 4 LDSP B1 EM Et Rack 1 Expected $FF Found $13 Eta/Phi +1/ 4 LDSP B4 EM Et Rack 2 Expected $FF Found $13 Eta/Phi +2/ 4 LDSP B1 EM Et Rack 1 Expected $FF Found $13 Eta/Phi +2/ 4 LDSP B4 EM Et Rack 2 Expected $FF Found $13 Eta/Phi +1/ 4 LDSP B1 Total Et Rack 1 Expected $FF Found $1A Eta/Phi +1/ 4 LDSP B4 Total Et Rack 2 Expected $FF Found $1A Eta/Phi +2/ 4 LDSP B1 Total Et Rack 1 Expected $FF Found $1A Eta/Phi +2/ 4 LDSP B4 Total Et Rack 2 Expected $FF Found $1A Run 92438 Event 79: 8 Errors found Another high amplitude pulser run was taken on Monday and it showed no problems. So today I replaced the ERPB that services eta +1:+4 phi 1:4. I pulled ERPB SN#06 and installed SN#86. Both the Red and Green LED's were on on the ERPB that I pulled and on the new ERPB that I installed. I carefully labeled the "broken" ERPB and will return it to MSU. High amplitude pulser runs taken right after the new ERPB was installed show no problems. High and Low amplitude pulser runs taken on the 28th show no problem (runs 92527 and 92528). High and Low amplitude pulser runs taken in the morning of the 29th show all OK (runs 92548 Low and 92550 High). FIND_DAC Run a Find-DAC on all eta-phi to verify that things have not shifted around too much. The only channel that changed significantly was +15,28 EM. Guida's had spotted this channel in their CALIB as having a pedestal of 10 instead of 8. Find DAC showed: DAC_BYTE increment -6 for EM,POS,E_15,P_28 32->26 2560 towers have been examined 1 tower(s) incremented by -6 6 tower(s) incremented by -2 108 tower(s) incremented by -1 2307 tower(s) incremented by 0 135 tower(s) incremented by 1 3 tower(s) incremented by 2 The retired Init_DAC_Bytes.LSM file was put into [Trg_Current.Obsolete] and the new file copied to TCC:: and to MSU::[.DZero]. So we need to keep an eye on +15,28 EM just in case it has a dying capacitor on its pedestal DAC that is going to make it spiral out of control. CalTrig Random Cell Test This test was failing with problems of Px being off by a count of 8. The Px value after zero loops was low by a count of 8. Using Tree Browser I traced this to the section eta +5:+8 phi 25:32. After zero loops of Random Cell this section should have had a Px value of 60 but it was showing a count of 52. With this problem Cal Random Cell test would stop every 20 loops or so and need to get its Px value moved by 8 either up or down. The Tier 1 CAT2 Px adder for eta +5:+8 phi 25:32 was putting out the proper value. I ran a short cable out from this card to a 110 Ohm terminator and verified that it was driving OK. I then looked at the cable that ran to the Tier 2 +Px operand #4 input. I could see 100 mV across each input and I checked each side of each input input with respect to ground and saw the "right wrong" voltages. They were: Non-Inv Input WRT Gnd Inv Input WRT Gnd Bit 1 (value 1) -0.619 Volts -0.512 Volts Bit 2 (value 2) -0.615 Volts -0.507 Volts Bit 3 (value 4) -0.618 Volts -0.511 Volts Bit 4 (value 8) -0.618 Volts -0.511 Volts Bit 5 (value 16) -0.617 Volts -0.510 Volts Recall we are looking for a problem with Bit 4 value 8 but all looks OK. So the problem must be in the input section of the Tier 2 CAT2 (i.e. the value being sent to this card was correct, the cable looks OK, and the read back from this Tier 2 CAT2 input operand #4 is wrong so it must be a problem in the input section of this card). Pull CAT2 SN#099 and install SN#270 at the +Px adder in Tier 2 at the top of M105 (third card from the top). And the problem remains exactly the same. Double check that the Tier 1 is properly sending the right stuff. Double check the cable running up to Tier 2. At the Tier 1 end of this cable put a 110 Ohm terminator. This is just to cause some current to flow in the wires of this cable. In the voltage measurement above no current actually moved in the cable (i.e. the cable could have been 1000 ohm resistors and you would get the same number as above. Now with the 110 Ohm load at the L1 end of the cable and looking from the L1 end I see: Non-Inv Input WRT Gnd Inv Input WRT Gnd Bit 1 (value 1) -0.596 Volts -0.541 Volts Bit 2 (value 2) -0.592 Volts -0.538 Volts Bit 3 (value 4) -0.595 Volts -0.541 Volts Bit 4 (value 8) -0.615 Volts -0.609 Volts Bit 5 (value 16) -0.594 Volts -0.541 Volts OK so the Inv cable for Bit 4 clearly has a problem. I verified that it was not at the Tier 1 connector end by probing the conductors in the flat cable themselves (where they are exposed that the end of the cable). So I wiggle and push on the paddle card. Now the voltages at the Tier 1 end look OK. Run 60k loops all OK (all Cal Random Cell options enabled except locked on page #4 and no check of AND-OR terms). Then next day run 75k loops all OK. S-HTT/PAR%rand% Loop 75000/75123, Error Count is 0 E-HRD/TST%rand% 75123 Loop Test Completed, No Error Detected Note that I could not look into this cable from the Tier 1 end with an Ohm meter because with the Tier 2 running all the Ohm meter sees is the 100 mV across the resistor at the Tier 2 end, and with the power pan off the mill volts that the power supply caps still put out still screws up the Ohm meter measurement. Note what "right wrong" voltages to you expect to see across the 110 Ohm terminator and WRT Gnd. 5.6k 110 Ohm 3.9k +5 V o----------WWWW-----+----WWWW----+-----WWWW----------o -4.5 V | | | | o 109 mV o -0.536 V -0.643 V WRT Gnd WRT Gnd What I actuall see is everything shifted positive by 30 mV or so, i.e. the -4.5V is not quite a full -4.5V. This whole thing should be centered at -1.2V or something not at -0.6 V. Work on Trigger Tower +18,4 EM In recent pulser runs this Trigger Tower looks at 50% of what it should be. Dan Owen has checked the BLS and it is OK. I checked by looking into the L1 end of the cable with an Ohm meter and the scope box and scope and all looks OK including mild bending of the cable right next to the L1 connector. I used the CTFE pulser to check the CTFE response and all looks OK. So this channel will be left alone and checked on the next pulser run (which will be later today). This channel also looked bad earlier and then got better by itself. In the pulser runs on Wednesday afternoon (after I had played with this channel) all looked OK. This channel continues to look OK in the Thursday morning the 29th pulser runs. Work on Trigger Tower +5,20 EM Dan Owen says that for a long time this channel has been OK at low amplitude but about 20% low at high amplitude. He has changed the big BLS hybrids and this did not help. I checked this with the CTFE pulser and it looks OK, i.e. EM+ = EM- and pulsing just either EM+ or EM- gives 400 mV at the monitor Lemo and pulsing both EM+ and EM- gives 800 Mv but this is only 40% of full scale. So we need a bigger pulser. Work on Trigger Tower +16,8 EM This channel is just a bit low (perhaps 10%) but it has been dropping. I checked with CTFE pulser an all looks OK. Work on Trigger Tower -20,22 HD This channel is at 50%. There is little or no signal on the HD- coax. There appears to be an open at the BLS end on the EM+ coax so we rebuild the BLS end. Saw nothing wrong with the HD coaxes in the BLS end connector. The L1 end was rebuilt on 6JAN94. After this BLS end work we now see a signal on HD- at the L1 end but we need another pulser run to verify that all is OK. Final pulser run on Thursday says that -20,22 HD is OK. IBM/PC OS2 Delivered the 486 OS2 machine to Dan Owen. .............................................................................. Date: 22-JUN-1995 At: MSU Topic: Problem with L1 Cal Trig, Site wide power loss, They called at about 10AM. The story was something like: an Examine plot looked funny, and TrgMon Global Cal Trig display looked funny, so they power cycled Tier 3. Now of course things are really screwed up. All of this is with good beam luminosity of 10e30. Step one, encourage people to call often call early. They initialize L1, down load triggers, and resume the run. The only data that looks in trouble is data that comes from the M103 readout crate, i.e. Global Total Et, Counts of TT's over Ref Sets,... All of the data that comes out of the Tier 1, Tier 2, Tier 3 look OK. Trigger rates look OK. The examine plot that looked bad is the plot that uses Px,Py from L1 to make a Phi distribution of the missing Et. The Px,Py come from the M103 readout crate. So this plot should be screwed up. I verified with Jim Linnemann that L2 does not use any of the information from this readout crate. So, could we have a water leak in M103 ? or M107? Is it the MBD for final readout in M103? Now the power trips site wide. An "insulator blew" in West Chicago. By early evening all of the MSU equipment was powered up again. The TrgMon Global Cal Trig display still looks the same. .............................................................................. Date 20-JUN-1995 At: D0 Hall Topics: Water Chiller Problem causes L1 to be push button turned off. JUN 20 00:10 Chiller trips off (same as happened last night). We cannot run with magnets on in these conditions. JUN 20 00:37 Chiller seems to be working (thanks to the superior technical expertise available!). Toroid polarity reversed. JUN 20 06:11 end run 92291, 3334 events. this was because the york chillers tripped off again and the toroids had to be turned off. JUN 20 06:56 we also had to power down the trigger framework as the water temp had risen about 60 deg f. we are in the process of turning things back on now. .............................................................................. Date 19-JUN-1995 At: D0 Hall Topics: Power Pan repair, Trigger Tower repair, more setup for E811, change power up power down procedure. Tried for a second time to repair Power Pan PDM-19. I replaced its bad +5V brick SN#17 from 14-FEB-92 with a repaired brick SN#30 from 7-FEB-92. This new brick ran for about 5 minutes under a 200 amp load and then died. Repaired Trigger Towers -5,31 EM and -15,29 EM. Both were reading high in the Dan Owen check of the Dan Owen pulser run. Both had Terminator-Attenuators with cold solder joints. Both have been this was since about May 29th. Andrew Brandt needs the setup for E811 changed. The idea is to drift things in the direction of being able to run with a L15 trigger mode. So for now get some signals from a Spec Trig's Fired cable routed over to his ECL to NIM converter module in rack M122. Send him Spec Trig #0 on the 6th module channel and Spec Trig Fired Strobe on the modules 7th channel. Recall that on this modules 8th channel is the output from Global Total Et Comparator #3. Change all of the written Power Up and Power Down procedures to say that the L1 Cal Trig Power Pans in racks M103:M112 should be turned off by hand before turning RPSS off via the yellow push button and to turn the RPSS on first and then the L1 Cal Trig Power Pans in M103:M112. That's right I've decided to trade the real problem of blown power supplies from full power turn off's for the real potential danger of people playing in the rack and leaving doors open. .............................................................................. Date 18,19-JUN-1995 At: D0 Hall Topics: Water Chiller Problem causes L1 to be controlled method turned off. JUN 18 23:52 begin run 99264 V105_06E30 lum=6.22 E30 A chiller has tripped off (compressor number 4. Max is trying to restart it) JUN 19 00:28 Temperature dropping with magnets and cal trigger turned off. .............................................................................. Date 18-JUN-1995 At: From MSU work at D0 Hall Topics: Power Pan in M103 PDM-25 rack M103 Sunday morning at about 4AM there was a Drip Alarm that tripped all of the L1 equipment and the Lower Tier 1 Power Pan in rack M103 did not work when power was restored to the L1 system. The power pan that died was PDM-25. Its +5V brick died. It died in a way that pops the 30Amp breaker if you try to turn it on. PDN-25 was replaced in lower M103 Tier 1 by PDM-??. It still is not 100% clear what happened. I believe that there are two possibilities. 1). We really had a Drip Alarm and tripping off all of the L1 system killed this +5V brick. 2). The +5V brick died such a violent death that it caused current transits that caused the Drip Alarm. Alarm log entries follow: So on 19-June-95 the bad +5V brick SN#23 from 7-FEB-92 was removed and replaced by SN#15 from 11-JUNE-1991. This Power Pan now tested all OK so it was put in the storage cabinet on the 3rd floor as a spare. .............................................................................. Date 9-JUN-1995 At: MSU Topics: reboot TCC TCC was rebooted during a down time between stores, because it had quit writing to its TRICS logfile 2 days earlier. It had been up for almost 10 days, and leaking memory anyway. All other logfiles seemed fine and were still active. Remote file access was happy too. We can imagine that one IO to the logfile failed, and TRICS switched over to its protective mode of no longer writing to the logfile. There is no reason to believe this is related to the ongoing disk driver problem. As far as we know, this is the first time this happened. .............................................................................. Date 5,6,7,8-JUN-1995 At: D0 Hall Topics: Power Pan in M109 Lower Tier 1 Replace Power Pan in M114, Meeting with Jerry Blazey, PDM-19 rack M109 Replace the Lower Tier 1 Power Pan in Rack M109. This Pan's -4.5V supply went out of tolerance a couple of times Sunday night and caused a Fatal Alarm. For running on Monday the tolerance was increased 0.2V and people watched the Trigger Examine carefully. Nothing bad was seen in the Examine and there were no additional alarms. Before pulling the Pan I checked its -4.5V supply with the Fluke meter. It was clearly wandering around. Fluke read about 0.2 Volts on the AC scale. Pull PDM-19. Install PDM-21. When this Pan was out of rack M109 some water was noticed on the floor of this rack. It was a very slow drip from the front plug on the manifold on the side closest to M110. In the floor of the rack there was a puddle about 3" or 4" in diameter. No water was seen under the floor but it is hard to see anything except cables. The standard big hose clamp treatment was put on the manifold over the plug. No more water was seen to drip but more time would have had to have been spent waiting for a drip to "prove" that the dripping was stopped. The broken Power Pan from Rack M109, PDM-19, had its bad -4.5V brick, SN#57 15-MAR-91, removed and a repaired brick, SN#38 15-MAR-91, installed. PDM-19 was then tested using the toaster. All 4 supplies look OK under load except that the green "Power OK" LED on the +5V brick goes out as soon as the load is above 20 Amps. So PDM-19 is not considered ready for service. I have left it on the bench in the kitchen. I will take a replacement +5V brick to Fermi on the next trip to finish the repair of PDM-19. M114 Power Pan The four brick on the "new" Power Pan for rack M114 are: Rear SN#293943 no MSU# +5V supply SN#398974 MSU#47 +5V supply SN#416270 MSU#98 -5.2V supply Front SN#416268 MSU#97 -5.2V supply After final adjustment the voltages are: REAR +5V 5.088 Volts at the Test Points forward +5V 5.090 Volts at the Test Points rearward -5.2V 5.295 Volts at the Test Points FRONT -5.2V 5.295 Volts at the Test Points Typically the voltage at the brick studs reads 1 mV higher than at the Test Points. After final adjustment the voltages on the vertical power bars are (measured at the center of the power bars): +5V Power Bar WRT GND Power Bar 5.016 Volts -5.2V Power Bar WRT GND Power Bar -5.200 Volts The load currents measured with the Fluke clip on DC current meter are: +5V --> 105 Amps -5.2V --> 146 Amps Before the new power pan was installed in M114 we had turned off the old M114 power pan for about 15 minutes to make measurments. When M114 was turned back on it had the same funny problem of the Assistant COMINT thinking that it always needed to be building Data Blocks. Last week when this happened this was "fixed" by replacing the Pilot COMINT PROM's. This week this problem was fixed by cycling the M114 power and turning up the HP helper power supply to 85 Amps. Thus the PROM's that were pulled last week from the Pilot COMINT are most likely all OK. Disassemble the bad Power Pan that was removed from rack M114. The bad -5.2 Volt brick is SN#87 from 7-JAN-92. The +5V brick which is SN#50 from 15-MAR-91 was tested. It also is bad. So both bricks from the old M114 Power Pan need to be repaired. Run II Triggers Maris and I talked to Jerry Blazey about L1 Cal Trig during Run II. He is not counting on more than 4 EM and 4 Jet Reference Set and he does not appear to consider this limited number of Ref Sets a problem. He thinks that 4 and 4 is the best mix. He questions the usefulness of the Large Tile Ref Sets and he like more the idea of using just the Jet Ref Sets. The inefficiencies in Jet Ref Set triggering are better understood than the Large Tile based triggering. The one possible new idea is to get signals out of the EM Ref Sets for the central region that split the phi coverage into 4 sections. This is for use with the CFT. .............................................................................. Date 1,2,3-JUNE-1995 At: D0 Hall Topics: Meetings, E811 Setup and change to our Global Total Et Comparators, L1 M114 Problems, Replace MTGBIT15 PAL in L15CT, Reading L15CT counters into Trics Log. On Line Meeting Go ahead with the NT Alpha Messaging System for Run II needs to be talked about at this summers workshop. Run Meeting Nothing is to be done between now and 24th July (and nothing is planned to be done for fall winter running) to put the luminosity in the L1 Data Block or to have TCC/TrgMon take over the job of the Dan Owen Program. Want to keep all Run I data the same. All of this needs to be better organized for Run II. People like the idea of only ONE program calculating the luminosity numbers and serving these values to all of the consumers. E811 Setup E811 hardware setup and msg files. We have only one CAT3 card as the 1st Lookup Global Total Et adder comparator. Thus, there are only 4 Global Total Et comparators 0:3. But we at most use only one of these in global physics running. I removed the #3 comparator from feeding the And-Or network and instead have its output going via 34 conductor twist flat to the ECL-Nim converter in the top NIM Bin in rack M120. Andrew Brandt and Bob Hiroski use this signal in making up the signals that they send to Experiment E811. I was going to remove this And-Or Term from the Resource List but I did not because there are lots of other L1 Cal Trig terms defined in the Resource File that do not exist in hardware (but could if they are ever needed). I will just start picturing these terms as place holders to keep out the barbarians. L1 Problems in the vicinity of rack M114 Starting sometime about mid to late May there begins some indication of problem reading the And-Or Terms. This is seen by looking for "L0 vs L1 Bunch Number missmatch Error Messages" from the L2 system. One can quickly review an entire run by looking at the printed Sort Alarms listing for the run. This L0 L1 Bunch Number Mismatch started at about 1 per run and by May 31st had grown to about 8 per run. By June 1st the L0 L1 Bunch Number Mismatch was once every couple of minutes (so we could now see it on the VTC console by looking watching for 2-5 minutes) and we started having a couple of other strange things happen. Twice in the afternoon of the 1st (15:35 and 15:40) and then 24 time during the night of the 1st-2nd we had the L1 data block builder hang up. It would hang front end busy and after about 1 minute TCC would tire of not seeing any Data Blocks for TrgMon and TCC would reset the COMINT cards. When the COMINT hung it was the Pilot COMINT that was in trouble and it showed Req A Granted, Req B Waiting, Memory Available, Front-End Busy, DBB NOT Busy. The other strange problem that we saw in sort alarms was the Cal Readout ADC Controlers showing an error that is normally never seen but that could be caused by a timing problem in the signals that it receives from L1 FW. As a first step I slowed down the Data Block Builder by installing in the Main FW MTG the #3N PROM instead of the normal full speed COMINT #3M PROM. This did not help the And-Or read out. There is concern about the buzzing noise from the power supplies in M114. Using a nice HP supply I tried a 50 or 60 Amp boost to the +5V supply. This did not change the buzzing or stop the problems. I then tried the 50 Amp boost to the _5.2V supply and the problems went away. We ran an 18 hour store with zero L0 vs L1 Bunch Number Mismatches and zero COMINT hangs or TRICS resets of COIMINT. There are two side stories. At one point while working on things the Assistant COMINT kept always thinking that it needed to build data blocks. I changed the PROMs on the Pilot COMINT thinking that this was caused by a Pilot COMINT Start Jet List problem. Changing the PROMs made the problem go away but I expect that something else was realy going on, I expect that the original set of PROMs is OK. After the booster supply was added to the -5.2V bus we did have one 5 minute spell of COMINT hangs TCC COMINT Resets. This was before the 18 hour store began. But this spell of problems was during a time that the Master Clock was not working. It was sometimes sending out signals and sometimes not. Also remember that we get two feeds from the master clock: one goes to M101-M102 area and the other to M114-M124 area. After the Master Clock was running and the 18 hour store started there were no problems. I'm leaving the system running on the booster supply and will next week bring down a power pan, bricks, cable, lugs, crimper, checked fluke and DC current meter. M114 grew past what was ever planned for it and the power supply setup in here needs to be checked. What are the loads ? Do we need parallel supplies? L15CT To repair the problem of the LDSP's not getting L1 Data about once every 40 minutes in global physics running I installed a new MTGBIT15 PAL in the L15CT MTG Ch#7. In 18 hours of beam running I now see ZERO "un-sticks" of the DSPs. A spare of the new version MTGBIT15 is in the PAL/PROM box at D0Hall. I still need to move the "PDS" and "JDC" files for the new MTGBIT15 into the official PAL directories. See also 26-MAY-1995 entry. Because I can not quickly remember how to get the L15CT counters into the TRICS Log file I made a com file to do this. It is in VWORK1: and is called GET_L15CT_COUNTER_INTO_LOGFILE.COM It does it very "gently", i.e. there is a 10 second pause between each read. The concern being the system may be in trouble right at the time you want to read the counters so lets do nothing to disturb it. .............................................................................. Date: 29-MAY-1995 AT: MSU Topic: TCC "Hang" They called at about 23:45 Michigan time to say that TCC was hung in the normal way. Jan called. They had called here first. She said that trying to DIR on TCC's disk did not work. We had received two main messages from TCC. One at 23:17 and one at 23:18 From: FNBIT::D0::TRGUSER 29-MAY-1995 23:18:52.40 To: FNBIT::MSUHEP::EDMUNDS,FNBIT::MSUHEP::GROSS,FNBIT::MSUHEP::LAURENS CC: Subj: TRICS V6.4/11-MAY-1995/ Problem during Initialize, BAD sent to COOR Main Ring is in access so there was time for Philippe to look at TCC. Philippe looked and discovered that it was not the memory pool resource that had run out (although the PKCDriver was now consumming this resource) but rather PKCDriver had bumped into some other quota. TCC was running again by about 22:05 Chicago time. Philippe's understanding of the problem is that the SCSI port driver PKCDRIVER starts forgetting (after running fine for several days) to delete messages that pile up and consume "Pool Blocks" (one per message lost), and memory. On Monday night the 29th, PKCDRIVER had not run out of pool blocks, nor exhausted all the memory. After a 30 sec look, I still don't know for sure which quota it bumped against. .............................................................................. Date: 26-MAY-1995 At: MSU Topics: L15CT problem in 68k_Ser Error Recovery and MTGBIT15 PAL First store for 11 days. First beam running of the "new L15CT" system, i.e. the system that was installed on the 17-MAY-1995. Ir runs OK for the first 1/2 hour of beam global physics running and then hangs. The post mordem is that 68k_Ser ran its error-recovery routine and that there was a problem in its error-recovery routine. The problem is that the error-recovery routine did not "re-enable" the L15CT MTG to send out Transmit-Triggers. Thus, once the error-recovery routine ran (for what ever reason) the L15CT system was then doumbed to hang on every event that it tried to process after that. 68k_Ser error-recovery was repaired and installed. We now see that error-recovery executes about once every 40 minutes of beam running This problem is most likely in the MTGBIT15 PAL in Channle #7 of the L15CT MTG. If the skews in this part are bad enough and if the re-enable of ExtEnb happens right at the very end of the PROM clock pulse then it is possible for this part to quit remembering that it needs to send out a Transmit-Trigger and NOT send out a Transmit-Trigger all on the same beam crossing. So a new MTGBIT15 PAL was made that uses the SecHld signal to break the hold on the TMLtch signal. .............................................................................. Date: 22,23-MAY-1995 At: MSU Topics: Pulser runs, Pedestal run. On the 22nd Philippe made a Find-DAC run. It found good values for all channels. 4 Channels moved by -2, 1 moved by +2 and 1 moved by +3. The new Init_DAC_Bytes.LSM file was installed on the 23rd and copied to MSU. On the 22nd shifters made pulser runs 91584 (LOW) and 91585 (HI). Both runs looked good and L15CT_PROV saw no problems. .............................................................................. Date: 17,18-MAY-1995 At: Fermi Topic: Power Trips from the last two days, Water Flow Sensor is disconnected, Install the latest version of L15CT hardware and software, Run Meeting, Power Trips - Water Flow Sensor Bruce Gibbard installes a new COOR. ------------------------------- Power to all of the MSU equipment was turned off on 15-MAY-95 at 19:05 because the cooling water temperature was out of control. Power was not turned back on until 13:50 on the 16th. L1 was not needed until then because of other host work that was going on during the shutdown to repair the accelerator. Power to all MSU equipment tripped off at 23:15 on 16-MAY-95 when the coil in the water flow sensor for rack M103 opened. This water flow sensor was not replaced but just jumpered around and L1 turned back on at 15:32 on the 17-May. When time permits I need to work on this water flow sensor. L15CT Work ---------- Begin work on L15CT. First change the L1.5 CT Reference Set for the Pulser run Config (both HIGH and LOW) to 5 GeV (from 1000 GeV i.e. no contribution). The ETA +/- 20 rings were left at 1000 GeV, this matches global running and also how this Reference Set was originally defined last summer. This ref set is defined in the file CFG_ROOT:[CAL]CAL_PLS.RS, which is called (through a long series of indirection) from both the high (CALOR_PLS_TRIG_HIGH) and low (CALOR_PLS_TRIG_LOW) configuration files. Verify that the pulser runs still work with the current L15CT. Note that there are 2 Specific Triggers defined for this run, but only one moves events to the host. The "pure" L1 fires at a higher rate than the one which uses L15, but every time the L15 Specific Trigger fires, the pure L1 is also guaranteed to fire. Note also that the L15 Specific Trigger had a high L2 Disable rate (about 56%), but that was thought to not be too unusual for this run. Also change the L15CT "cosmic" configuration. This is defined in CFG_ROOT:[CAL]CAL_TRIG_L15.CAL. Many files hang off from this configuration, the only one which needed to be changed was CFG_ROOT:[CAL]CALTEST.L15. This file was appropriate only for the old "Isolation" Tool, not the new "EM_Fraction (+ Isolation)" Tool. Verify that this configuration works. L1 fires at about 2 Hz, and the L1.5 Reject rate varies from 0 to 100%. 68K error counters were checked after both tests and nothing funny was spotted. Install the new PAL which allows Transmit_Trigger to be "gated" by the 68K. This is an MTGBIT15 PAL in Channel #7 of the ERPB MTG. The ExtEnb signal that "gates" the Transmit_Trigger is controlled by 68k_Services. The electrical signal that is connected to ExtEnb of L15CT MTG Ch #7 is made by the LSBit of Port #1 of the Path Select Ironics card and is converted to diff ECL on the Path Select Paddle board. The Path Select P2 Paddle card was modified so that it has a differential ECL output that is controlled by the state of the LSBit in Port #1 of the Path Select Ironics card. The LSBit of this port arrives on this paddle card via VME P2 pin A2. This connects to pin #7 of U13 (the 10H124 TTL --> ECL). The diff ECL signal comes out of U13 on pins #1 (non-inv) and #3 (inv). Both of these signal are pulled to -5.2V by 510 Ohm resistors. The non-inv signal leaves the paddle card on pin #15 of the lower 34 pin rear connector. The inv signal leaves the paddle card on pin #16 of this connector. Verify that this works with the "old" 68K/DSP software, changing only TRICS_INIT_AUXI_L15CT.DAT. The only change that was made to the TRICS_Init_ _Auxi_L15CT.Dat file involved the setup of Ch #7 of the L15CT MTG. This had been a simple load of value 10 at location 3,89,35,6 to setup the old MTGBIT8 PAL. For the new MTGBIT15 PAL first we load a 0 at this location and then we load a 4. The reason for first loading the value 0 is to be certain that the MTGBIT15 is not "remembering" that it needs to send out a Transmit Trigger next time that it is ExtEnb enabled. Install new version of 68K_Services. This version controls the gate to the Transmit_Trigger, and also increases the "UnStick" timeout to about 10 mill sec. The idea is that we should not try to "UnStick" the DSP's unless they really are hung for some reason. Recall that when the LDSP's did not abort the scan loop after they filled their Local Object Lists, we had rare occurences of 68K_Services "UnSticking" the L15CT when it was actually close to finishing its processing. This was addressed by adding the "abort scan loop" feature to the Local DSP's. Now that the "abort scan loop" operation will disappear, we choose to increase the 68K "UnStick" timeout. Also note the new version/revision numbers: Data Block Version: 2 Hardware Revision: 2 Local DSP Code Revision: 3 Global DSP Code Revision: 2 68K_Services Code Revision: 2 This new version of 68k_Services was tested with the "old" DSP code in both the "cosmic" and "pulser" configuration files and worked OK. Load new version of DSP code. Test with "cosmic" configuration. Take a quick look at Data Blocks via ZBD, no problems seen. Switch to Pulser configuration and run. No "UnSticks" or any DAQ problems seen. Take a quick look at Data Blocks (now MFP) via ZBD, still no problems seen. Run the simplest rate test possible. Can't do this with Pulser config because of bottleneck to L2 (spending lots of time waiting for VBD). Use "cosmic" config. Crank L1-only trigger to low rate. Switch "L1.5" trigger to be purely prescale at L1. Crank up rate, and look at L1.5 Dead Crossings. Can't look at Geo Sect 5 FE Bz, does not increment during L1.5 Cycles. Recall that we get a better readout of L1.5 Dead X from TRGMON's L1.5 FW screen than from the L1 FW screen. With evenly-spaced events, see the following: L1 Rate L1.5 % Dead Crossings L1.5 Processing Time ------- --------------------- -------------------- 57 Hz 0.48% 84 us 113 Hz 0.97% 86 us So to first order, our 80 us estimate was close. We made a recorded run, Run number 91474. It is still on LBUF disk. It is perhaps 100 events. The idea was to have all types of events: overflow, non-overflow, Normal, MFP. We used the "L15CT cosmic" configuration file, MFP ration of 5, with the pulsers also turned ON for one section of CC. Cosmic noise triggers came in at about 1Hz and the pulser was firing at about 1Hz. We then ran for one hour using the "L15CT Cosmic" config file but with L1 Ref Set lowered to 1 GeV and using L15CT_CMD.Com and L15CT_Low_Ref_Set_CMD.Com files to lower the L15CT ref set and 1x2 threshold to 1.25 GeV. The pulser was also running. This gave us about 30 Hz of L15CT processing with 1 in 5 MFP ratio. There were no errors or 68k_Ser unstick DSP's after more than one hour of running. The idea was to exercise the system in a way that required the control of the Transmit_Trigger to be working properly or else there would be an error. There is a new version of COOR loaded which was supposed to fix the "acknowledge mismatch" problem for L1 Initializes by increasing the timeout to 100 seconds. This does not appear to have worked and in fact we think that the new COOR will generate a lot more of these messages. An 11 second ACKNOWledge delay appears guaranteed to cause a mismatch, but the timeout may even be shorter. How To Backup ------------- How to back up to the old version of the L15CT processing if necessary. People want the L2 executable and the L15CT processing to change at the same time. So if L2 is not ready then it will be necessary to backup the L15CT processing to the version that was used before 17MAY95. To do this: 1. Do NOT change the hardware or the Trics_Init_Auxi_L15CT.Dat back to the old versions. Keep the new hardware and the new auxi_l15CT.dat file. 2. Have people at Fermi load the L15CT 68K processor with the file L15CT_68k_Services.Abs_Removed_17MAY95 This file is in the TrgCur: directory. Instructions for loading the L15CT 68K are under the keyboard for its console on top of the air conditioner. 3. Make the pre 17MAY95 version of DSP code available for loading by: ReName D0HTCC::DUA0:[L15CT$EXEC]*.BLX .BLX_NEW_DONT_LOAD (or whatever) ReName D0HTCC::DUA0:[L15CT$EXEC]*.BLX_OLD_DONT_LOAD .BLX Run Meeting ----------- Most of the run meeting was taken up with trying to understand all of the sources of luminosity measurement and consumers of luminosity information. D0LUMS and D0LUMSD are the best measurements that D0 will have. They are also nice because they are not disturbed if a bunch is missing. I believe that things may workout so that only Dan Owens program makes the D0LUMS and D0LUMSD calculations and then he gives it to AccNet. Then consumers will get this information from Accnet via the "get_Accnet" routine. There is still much to be worked out. .............................................................................. Date: 11,12-MAY-1995 At: Fermi Topic: Work on the Per Bunch Scalers, Run Meeting, another spare brick. The Per Bunch Scalers that count L0 Slow Z Center have been clocked by the Live Crossing signal (from Spec Trig #30). Thus these per bunch scalers were counting Live-Crossing * Slow-Z-Center. For making a Slow Z Luminosity measurement they want them to count only Slow-Z-Center i.e. they need to be clocked every beam crossing period. It was too difficult to pull off the per bunch scaler wiring harness and I did not want to break anything else (i.e. it ends up that all of these per bunch scalers are important to Physcisc). Thus the Slow-Z-Center clock was changed by cutting up the DBSC cards themselves. It is the Clock signal to DBSC channel #4 on these 6 cards that was changed. The normal circuit is: 2 R44 +-WW- | 15 |\ J2-25 >-----+--x--+------------------| \ | | \ R41 Z | >---- Clock | | / Ch #4 J2-26 >-----+--x--+-----------------O| / U71 | 14 |/ +-WW- 2 R45 Note that the spreader resistor packs R44 and R45 are not normally installed. The trace was cut between: pin #2 of R44 and the the R41 J2-25 and between pin #2 of R45 and the the R41 J2-26. This left the R41 J2 pins 25,26 setup intact for looping the "normal" Live Crossing clock through the per bunch scaler cards and it isolated the input to the Clock for Ch #4 at the pin #2 of R44, R45. A white wire was run from R44 pin #2 to J2 pin #83 i.e. TSS signal "E" Non-Inv. The is the Non-Inv input to the Ch #4 Clock. A black wire was run from R45 pin #2 to J2 pin #84 i.e. TSS signal "E" Inv. This is the Inv input to the Ch #4 Clock. Thus Ch #4 of these 6 per bunch DBSC's is now clocked from TSS signal "E" which is routed on the Specific Baclplane CBus in the Lower M114 backplane. TSS signal "E" was not currently in use (neither are TSS "D" or "F". On the CTMBD that drives the lower M114 backplane TSS signal "E" was connected (with inversion) to L1 FW Main Timing MTG Ch #12 i.e. the Start Digitize TLM Clock. This signal always runs and it gives these per Bunch DBSC's a positive edge at just about the same time that they would have received a positive edge from the Live Crossing Clock. I haved edited the master wiring file for the MBD-CTMBD's to indicate this new connection. I had M114 turned off for sometime while working on this. It continues to be clear that the L1 Cal Trig gets hot if M114 is turned off. I should have turned M114 off, pulled cards, and turned it back on. I did turn off Tier 3 while M114 was off. I was surprised how warm the DBSC cards were when I pulled them out. There is lots of air flow over these cards but they were plenty warm. The power bricks in M114 continue to buzz quite a bit. Slot Bunch CA DBSC SN# ---- ----- -- -------- 18 P6 53 32 17 P5 50 34 16 P4 47 38 These are the SN#'s of the 15 P3 44 35 DBSC's that were cut up. 14 P2 41 40 13 P1 38 29 Philippe made a TEST version of TrgMon that let us look at the Slow Z Center scalers and the luminosity calculated from them. In the run meeting I learned that the accelerator will be doing studies much of next week. The stack will drop Monday morning, the Store will be killed Monday afternoon - evening. Stacking will resume on Thursday Morning and a first shot will be made Thursday afternoon - evening. At the run meeting Dan Owen presented information about what fraction of the integrated luminosity is accumulated above a given luminosity. This is based on the accelerator divisions store models. Even if they continue to shot 30's between now and 24-July then between now and 24-July we only get 1% of the integrated luminosity above 25 and only 2% above 20. He pushed a little on the idea that: we are OK below 20 and all of this work cutting up the system (e.g. hit finding in L2, L15CT filtering of TOP) just to run "efficiently" above 20 will never pay off. Lumisosity falls so fast above 20 and especially above 25 that we should just run with 25 prescale set, suffer a little Front-End Busy and soon the problem goes away. We don't do a bunch of stupid work and waist good beam getting new L2 and Trig Configs to work and we retain continuity of our data and its processing. Needless to say, he was boo'd down. I brought another spare brick to Fermi. This is +5 Volt 200 Amp brick SN# 38. It just came back from repair. In fact it has been repaired twice. I ran it at MSU for 30 min before bringing it here and it looked OK. I put it in our spares cabinet up in the high bay. Recall that the power pans in M101, M102, M114 and the L15CT power pan are only supported at the brick level. .............................................................................. Date: 10-MAY-1995 At: MSU Topic: Power OFF They York water chiller had a problem so they turned off the magnets and the MSU equipment. We were turned off a 7:00 AM by the Red button method. The Captain's Logbook says that we were turned back on at 7:46. It says that the York chiller tripped at 6:52. Magnet = 400 KW, Our stuff = 100 KW, CD = 100 KW, rest of the world = 100 KW Perhaps when 2 of the 3 chillers are still running then they really do not have to turn us off. It did start up OK including M106 Upper Tier 1 +5 Volt. .............................................................................. Date: 3,4,5-MAY-1995 At: Fermi Topic: Work on the Scaler, who is clocking me and why do you stop problem. DBSC's in M101 "Meena Discovers Problem with L15CT" -------------- In M101 it is clear that the clock input to the: BX scaler, Gated BX scaler, and the 6 VTC scalers is coming from the Frame Work Main Timing MTG Channel #13 "DBSC Increment Clock to the FSTD Cells" signal. This signal arrives at the DBSC's by a MTG Ch#13 to Specific Backplane CBus TSS signal E connection on the MDB that drives the top backplane in M101. This signal is picked off of the TSS signal E location on the MBD's backplane connector by a single twisted pair and then loops through the 3 DBSC cards that provide these 6 scalers. OK, to fix this; on the MBD wire FW MTG Ch#12 to Specific Backplane CBus TSS signal F. On the backplane move the cable from TSS signal E to TSS signal F. Other points noticed in working on M101 scalers: The gate signal to the "Gated BX" scalers is labeled "Geo Sect #1 COMINT Front-End Busy to Gate A DBSC Gated BX". I think that this comes from the Busy TLM Latch. The wiring for the bottom CTMBD in M101 must include the connection "MTG Ch #14 "Increment the Start Digitize Number DBSC" signal to Specific Backplane CBus TSS signal H. This connection is NOT in the official wiring list for this CTMBD. On the backplane there is a cable plugged into TSS signal H that is labeled "Increment the Start Digitize Number DBSC" and supplies the clock to the L1 Trigger per Bunch DBSC's in M101. I did NOT pull out the CTMBD from the bottom of M101 to verify this connection. But I DID add this connection to the official CTMBD-MBD wiring list file. ----> This connection needs to be verified. <----- I believe that the "Event Start Digitize Number" DBSC and the "Event Transfer Number" DBSC get their clocks from the CTMBD in the top of M102 (i.e. the Start Digitize backplane). In the text file that describes the wiring of MBD's and CTMBD's it shows the appropreate connections on the Top M102 CTMBD to get the clock signals for these two scalers but it labels them "only for monitoring". I know that the clock lines from these two scalers run over to M102. In the 30 page "Full System Timing Diagram" Framework MTG Ch #20 is labeled "Increment TAS Number". It should be labeled "Increment Transfer Number". TAS Number is made from the Event Transfer Number and the Event Start Digitize Number. M114 Foregin DBSC's ------------------- It is clear (and specific in the paper log book) that Foreign Scalers 1:4, 29:35 use the DBSC Increment Clock Signal WF MTG Signal #13. And I verified that this is how things are connected. This signal arrives at these DSP's via a FW MTG Ch #13 to Specific Backplane TSS signal J connection on the bottom CTMBD in M114. From the read CTMBD TSS signal J connection this clock is looped through these scalers on a signal twisted pair cable. To correct this situation pull the bottom M114 CTMBD and replace the MTG Ch #13 to J connection with a MTG Ch #12 to J connection. While doing this I notice that on this CTMBD there is a MTG Ch #11 to TSS G connection that is not specified in the official wiring list for this CTMBD. I can not image that this connection is needed, but I leave it in place, and add it to the offical wiring list for this CTMBD. Edit the official CTMBD-MBD wiring list file to make all of the changes noted above. Write the file [D0_Text.Scalers] Scaler_Work_May_1995.txt which indicates what scalers received the wrong timing signal. This file groups the scalers into their functions. Level 1 Cal Trig On the afternoon of the 5th the Cal Trig blows a gasket. With nothing going on the Total Count of Trig Towers over Total Et Ref Set #0:#3 started reading 1. Tree Browser pointed to Trig Tower +9,12. This is CTFE SN #157. Everything pointed to the PAL for its Ch#1. The strange thing is that Ch#1 PAL had a hand written label on it. PAL's for channels #2:#4 had LA50 printed labels. Replaced the Ch#1 PAL and things look OK. L1 Cal Trig was happy in the morning. Nothing was found in the electronic or paper log books about SN#157. +9,12 is in the upper M107 Tier 1 Backplane. When I turned this back on I had the normal problem with its +5V power supply. To make it work, turn it on until the -2, -4.5, and -5.2 supplies come up, then turn it off for 5 to 10 seconds, then turn it back on. The second time that you turn it on the +5V will actually come up first. While listening to other Power Pans (when turning the power off to work on the scalers) I noticed that the lower Tier 1 Power Pans in M108 and M109 are making noise, and one of the bricks in M114 is making noise. I checked M114 bricks with the Fluke on AC Volts and they looked OK. Dan Owen says that water is leaking some where at 2 cups a day. We have been asked (by someone) to prepair a sketch for triggering with 19nsec BX's. .............................................................................. Date: 2-MAY-1995 At: MSU Topic: Maris calls, trouble with TrgMon rates. Maris called to say that he did not believe the new TrgMon luminosity display. We start to dig into things from MSU and come to the conclusion that the Beam Crossing Scaler is running from FW Main Timing MTG Channel #13 "DBSC Increment Clock to the FSTD Cells" and that this clock stops during Level 1.5 Framework decision cycles. Thus the delta time looks low and this causes TrgMon to show higher rates than are correct. .............................................................................. Date: 28-APR-1995 At: MSU Topics: D0 Cntrl Room trouble with run summary Call from the control room (Jan): The run summary reports incorrect numbers, namely too low of a microb_lanking fraction and too high of a run efficiency. Extracted from the captain's electronic logbook: APR 28 22:28 Problem noted by previous shift: ublank term and eff_max_live on run summary sheet are abnormally high: 0.988, 0.996 compared with normal 0.93, 0.98. Attempted to contact Scott Snyder and Norm Amos with no success. Philippe remembered a similar episode from 12-DEC-1994, where the same thing happened to J.Butler for one run and didn't reoccur for the next run. The same thing happened today. For the December occurance, Philippe had checked in the begin and end run files and re-computed some quantities and produced different numbers that were in the "normal" range (e.g. average livetime about 91.3% where the run summary said ~98%). This time, Jan said she had already done that before she called, and that the numbers in TCC's files were ok. This time Jan pressed on Norm to find out what is going on. Mike Tartaglia and Dean were also informed. Norm answers that he checked the math and it was ok, BUT: "The Luminosity Server, when it writes the number which go into the RUN_SUMMARY does not insist that the scalers came from TCC files. Since RUN_SUMMARY information was always intended to be "on the fly", it just reads its current values of scalers from its internal memory. Some of the scalers come from TCC, some from global shared common and some come from both." Philippe points out that "the problem might be worse than is being reported. If the luminosity server was indeed looking at a WHOLE event from the pool, all the scalers would be there and CONTEMPORARY. I believe the readings would thus still be consistent and close to the final value; just a good but old reading." .............................................................................. Date: 27,28-APR-1995 At: Fermi Topic: Ladder by TCC, Talk, Test, Meetings Luminosity from TCC. Talked with Mike Matulik about the Slim Module Talked with Mike Tuts about Electronics Board and finishing the definition of the Run II equipment. Dean and Mike Fortner will NOT be at the summer workshop. Talked with Rich Partridge and Jeff Bantly about Luminosity measurement for Run II. Need to talk to Norm Amos. Ran some loops of Random Cell test; With every thing enabled except: locked on page 4 and not checking Framework And-Or Terms it looks OK S-HTT/PAR%rand% Loop 300000/300001, Error Count is 0 Now enable everything except checking Framework And-Or Terms (i.e. it is using all lookup pages) and standard problems appear: OK for first 5368 loops and then: Loop 5369, Global Py Momentum Sum is -852 instead of 3244, T1 trunc = -4096 ReSync then: Loop 5727, Global Py Momentum Sum is -5973 instead of -1877, T1 trunc = 0 ReSync then: Loop 5883, Global Px Momentum Sum is 13249 instead of 9153, T1 trunc = -16384 Global Py Momentum Sum is -1571 instead of -5667, T1 trunc = 0 ReSync then: Loop 6003, Global Px Momentum Sum is 12647 instead of 16743, T1 trunc = -20480 Global Py Momentum Sum is 6781 instead of 10877, T1 trunc = -12288 When repeating any of these cycles the same error reapears. All of this is exactly what has been seen before. Try running Cal random cell again and it makes it through 3240 loops and then Py off by 4096. Try running Cal random cell again and it makes it through 4519 loops and then Px off by 4096. Thursday Run Meeting: 1. It still is very unclear what the fall schedule will be. July 24th is still the end date for this segment of Run 1B (because of civil construction contracts). These contracts call for 2 shifts of 10 hours per day to get the civil construction done with in 3 months. 2. People clearly want (could use) the new feature of TrgMon showing L0 luminosity. There are problems in the area and having a digital based number will help them. Mont asked for it. I found a step ladder leaning against the back of rack M114. I expect that it was left over from when they power cycle booted the TCC to clear up the ELN Name Server problems that they ran into when trying to use a Model 90 as the L2 Supervisor. They do an ordered startup of all ELN nodes (including the Gateways and TCC) and I expect that they use the power switch to control the order of startup. But this does indicate that power cycle booting will be what some people (or primates) think that you need to do. .............................................................................. Date: 17-APR-1995 At: MSU Topic: "Hung" TCC They called about 3AM saying that something was wrong with L15CT because it was 100% FE-Busy. The Geo Sect #5 is 100% Front End Busy. L15CT L15CT 68k terminal says "Problem at LOAD ISR time Not all DSP's at Step D0" They had tried rebooting the 68k_Ser machine. When I tried to do a ReadLog from TCC it just hung with the DAP=01F77C54 message. Dir on TCC disk would not work and gave the same DAP code. TrgMon was running OK. Trics could read a register for me. TCC was NCP booted and all returned to OK. The principal point of this is that the call can be very strange when it comes. .............................................................................. Date: 13,14-APR-1995 At: Fermi Topic: Meeting with Jim Christenson about construction money for FY95. Need to be sure to spend the "maintenance" MOU before Oct 95. MSU must invoice Fermi for this before Oct 1st or we loose the money. Discussion about L1 overlap data. They want it preserved. Pickup from Dan Owen's office the MSU NIM modules (rate meter, spectrometer amp from Spence X-Ray) and return them to MSU. .............................................................................. Date: 6-APR-1995 At: MSU Topic: All L15CT MFP Events into the Monitor Stream, Change the L15CT 68k_Services code again. Not all L15CT MFP events are making it into the Monitor Stream. This is because the code in L2 that "selects" the MFP events checks to see that both the L1 Trigger that required processing by a L15CT Term fired AND that that L15CT Term is set in the MFP Term Mask in the 7th longword of the L15CT Crate Header. Note that the MFP Term Mask only indicates which Term caused the L15CT decision cycle to be run in the MFP "mode". This mask does not indicate "which Terms received MFP processing". Thus the L2 select tool is actually the cause of the current trouble. But we are wasting cycles running MFP events so we should at least get them into the Monitor Stream. After much discussion and mail messages the decision is taken to change the 68k_Ser code so that for MFP events it will show that all four of the L15CT Terms caused the event to be processed in MFP "mode". New 68k_Ser code was booted into the system on evening of 6-APR-95 just before Global Physics run number 90271. .............................................................................. Date: 5-APR-1995 At: MSU Topic: Water Chillers, Power Cycle L1 Cal Trig. Two water chillers have tripped off at about 14:10. The water temp started rising above 61 degrees on 1st floor MCH so Jan & Joan turned off the L1 Cal Trig but left the Frameworks running. The toroid magnets were also turned off. On the order of one hour later, when a sufficient number of chillers were running again, the L1 Cal Trig was turned back on, .............................................................................. Date: 4-APR-1995 At: MSU Topic: Begin using the offline Verify tool. Some running (production running) of the offline monitor stream L15CT Verification tool has begun today. Greg Griffin is working on this. As of today it is only checking that Trigger Tower data from the L1 Data Block matches the Trigger Tower data in the L15CT Data Block. They are working on the other aspects of the L15CT Verification tool. .............................................................................. Date: 31-MAR-1995 At: MSU Topic: New L15CT 68k_Services code, Pulser Runs Joan booted L15CT 68k to install a new version of 68k_Ser code. The only change is that now during MFP events the 68k_Ser properly loads the Terms that the Global DSP reported as Passed into the proper byte of the 6th longword of the Frame Code Section of the L15CT Data Block. Before this code change during MFP events only, the 68k was loading the Status Value that the GDSP returned into the Mask of Terms Passed byte of this 6th longword. Joan made two pulser runs, run numbers 90046 and 90049. Steve checked these with the L15CT_PROV tool to check that all was OK with the hardware and then he check some events in these runs with a raw data viewer tool to verify that the Mask of Terms Passed was properly being set. .............................................................................. Date: 22,23,24-MAR-1995 At: FNAL Topics: Swap Global DSP Tool Code, Collaboration meeting. On 23-Mar-1995 we migrated to Revision 2 of the Global DSP Tool 3 code. This is the version which, during MFP events, does not allow Objects which failed the Local Tool (but are still included in the Object List because this is an MFP event) to contribute to passing their corresponding Term (as reflected in the Terms Passed mask returned to the 68K, and also included in the Data Block. Note that, during MFP events, the Service 68K does NOT set the Terms Passed to the L15 Framework based on the Terms Passed by the DSP's, but instead sets Crate 0 Terms 0-3 (L15FW Terms 16-19) HIGH (=Passed) no matter what the DSP's say. This involved the following steps: 1. Move the new source and executable for the Global DSP ONLY to D0HSB (note: also take this opportunity to clean up the EXECUTABLES directory on D0HSB, which still contained the Tool #2 code as well as the Tool #3 code). 2. Move the new executable for the Global DSP ONLY to D0HTCC::DUA0:[L15CT$EXEC]. Also take this opportunity to clean up this directory which also contained old Tool #2 code. 3. Load the new 68K_Services code into the L15CT 68K. The only change in this code is changing the L15CT Global DSP Software Revision Number from 1 to 2. L15CT-68k_Services code comes from the [TRG_C40.SOURCE_3WORK] directory. Also verify that the new code will be picked up on subsequent reboots. After making the above changes, we made high and low amplitude pulser runs to verify that all was OK. These are run numbers 89831 and 89832. Their data is in Data3:[Cal]. The first global physics run with this new L15CT software is run number 89842. When this run started up I thought there was a problem. Spec Trig #8 was firing at 2kHz and we were moving events up to L2 at 400Hz. Some calorimeter element in Trigger Tower +5,26 EM had become noisy. It only remained noisy for about 5 minutes at which time it when back to normal. People did not have a chance to exclude it. Joan and Jan say the +5,26 EM has done this a couple of times before for a couple of minutes each time. .............................................................................. Date: 21-MAR-1995 Topic: Over Temperature Power Trip of Trigger Racks When he got to work Steve noticed, via TrgMon, that things were not well at D-Zero and he alerted the rest of us. Captains log confirmed that they had (through a screw up) lost two water chillers and the cooling water temperature was now out of control. Our Trigger Racks tripped off (assumed from a global over temperature sensor) at 7:35 just as Dan Owen was getting ready to do a controlled shutdown of the racks. Jan ended up turning off many other parts of the DAQ system because the temperature of the cooling water was out of control. The Trigger Racks were turned back on at about 8:20 and the Initialize at about 8:41. It is not clear if the L1 VME was ever turned off or if L15CT was ever turned off. I believe that these two VME crates were not turned off. TCC was not turned off. After things were turned back on and a 0.57 Hz test trigger was running things looked more or less normal except that the EM and HD pedestals in the range eta -1:-6 all phi's were up at about 9 and 10 instead of 8. All positive eta looked OK and eta -7:-20 looked OK. This caused the Global EM Et to typically be in the range of 32 GeV or greater. By the time they started a QCD no Tracks run all the pedestals looked OK. Still not clear what caused this. .............................................................................. Date: 13..17-MAR-1995 At: FNAL Topics: Repair L15 CT data transfer from L1 problem in M106, notes about Trigger EXAMINEs, power glitch At about 14:30 Chicago time on 14 Mar, Dan Owen called Dan Edmunds to report a L1.5 CT problem. The symptom reported was a failure of L1.5 CT to find an object located at -8, 23 (note: this was found with MONITOR stream compare). Dan E suggests that if something is wrong with L15CT, it is probably data transport from L1 to L15. Dan E and Philippe request "L15CT PROV" special runs which confirmed that a data mismatch between L1 and L15 was occuring for both EM and Total Et in the range eta -8..-5, phi 21..32 (for both "Racks"). This is the data from the lowest 3 ERPB's in Rack M106. Dan E and Philippe suspect that the daisy chain data transfer is broken in the 3rd ERPB from the bottom in M106. Dan E and Philippe have Dan O look in the back of M106 to verify that there is no smoke or spraying water and then decide to wait for Steve to arrive at Fermilab. Meanwhile the accelerator breaks so it looks like Steve may have a good shot at fixing the problem tonight without costing beam. Steve arrived at 17:00 and was brought up to speed via mail messages and talking to Dan E and the Guidas. He thought that it was significant that most/all of the bits of both EM and Total were being affected and after looking at the ERPB schematic noticed that if the REG_CLK signal to U32 and U33 were stuck, similar behavior would result. The REG_CLK signal is just a buffered version of XMIT_CLK which comes from the DC via the DC-to-ERPB Parallel Timing cable. Steve replaced this cable (which was easier to do than replacing an ERPB, and also did not require any L1 Cal Trig power pans to be shut off). That did NOT solve the problem but Steve left the new cable installed. The old one has been placed in the "L15CT spare parts box" (i.e. the HP cardboard box) and is thought to be a 100% good replacement. The new cable is installed reasonably nicely but the next time we have a L1CT shutdown we should look at it carefully. The low limit set point of the Photohelic is set back to normal. That knob has a lot of stiction, I hope people don't overcompensate and overshoot when turning the low limit back up. Steve then shut off the M106 lower power pan and replaced the suspect ERPB. The removed ERPB was S/N 25 and has been tagged as "bad" and will be returned to MSU. ERPB S/N 87 was installed in its place. According to L15CT_PROV this has solved the problem. Boy am I glad that Dan E pushed for this tool. Steve looked at Trigger Examine plots with Jan and Joan. Page 13 of the Trigger Examine plots shows L15CT Examines (the code for which was written by Greg Snow). It is difficult (impossible) to tell WHAT is being plotted, but it certainly looks different starting at about 5AM on 10 Mar 1995 (Run 89372). Jan and Joan will talk to Greg about these plots (to find out what is being plotted, the Y axis is not labelled on ANY of these plots and the title is typically something like "PHI"), and also remind shifters why they cut down trees to make these plots. Note that this evidence is only slightly inconsistent with Dan O's statement that everything was fine at 11:17AM on 10 Mar 1995 (Run 89385) but broken by the morning of the 13th (Run 89498). The problem was fixed between Runs 89540 and 89541 (which were both L15CT_PROV special runs). Both Dan E and Steve thought that the L15CT "rejection rate" as shown by TRGMON changed after this repair was made. Spec Trig 8 went back to its canonical 65% rejection (down from 70%) and Spec Trig 9 went back to 35% (down from 40%). Steve's monitoring of temperatures after the replacement of this card was complicated by the rise of the temperature of the MCH chilled water to 59.5 degF, and the corresponding 5 degree rise in the Cal Trig temperature monitors. This situation only lasted about 10 minutes while Pete Simon was working on the chillers. We should watch the water temperature closely though, because the word "chiller" has appeared in Captain's log a lot lately. On 15 March Steve ran 1 million loops of CalTrig Random, over all eta/phi, locked onto page 4, but without checking L1 Framework AndOr Terms. He also ran another L15CT_PROV special run (89569) afterwards. No errors were found. L1 68K console continues to show TF and T6 errors, about one per minute (total) equally distributed between TF and T6. On 17 March there was a power glitch at about 7:15 AM Chicago time. The stack and store were lost, and also platform power. 1st floor MCH power remained on. Steve checked power pan LED's, TCC console, and 68K consoles. L1 68K console was complaining about "No Slave Ready seen within max wait" every 2 minutes which is typically an indicator of COOR "sticking" the Framework. TCC was NOT trying to reset the COMINT once per minute. Steve tried TRGMON and DIR D0HTCC::DUA0:[TRIGGER] to verify that TCC was alive. Steve did NOT try TRICS_ACCESS. While Steve was gone to Mentor class, control room people noticed that COOR could not talk to TCC. TCC console showed a connection on Channel 1. "Restarting" COOR did not help, so they triggered D0HTCC without calling anyone. Steve thinks that Jan and Joan were both around when this happened, but doesn't know who did what. Also during this time both the L1 and L15 68K's were reloaded (again without a phone call). Note that the "old" (i.e. "no ANDOR term check") code was loaded into the L1 68K. .............................................................................. Date: 9,10-MAR-1995 At: Fermi Topics: Connect more And-Or Terms for new L0 signals, Bring the NWA Framework back to MSU, Watch VTC error codes, Discuss modification to L15CT EM-Fraction Global Tool. Connect two more of the signals from the NIM-ECL converter in rack M122 to And-Or Input Terms. This is for Norm Amos and Bob Hirosky. They need new special stuff from Level Zero. The new current setup is: Lemo Connector And-Or Term Pair in And-Or Term Name on NIM-ECL Module Number the Cable ------------------ ------------------- ------------- --------- CAL_RECOVERY 1st i.e. top #120 17 MRBS_AND_MICROBL 2nd from top #121 16 ASTUB_1 3rd from top #122 15 ASTUB_2 4th from top #123 14 MRBS_LOSS 5th from top #124 13 MICRO_BLANK 6th from top #125 12 MIN_BIAS 7th from top #126 11 LV0_HALOP 8th from top #127 10 MR_PERMIT 9th from top #116 9 MU_HV_RECOVERY 10th from top #117 8 LV0_HALOPB 11th from top #118 7 MR_VETO_LOW 12th from top #119 6 MR_VETO_HIGH 13th from top #115 5 FZN 14th from top #114 4 FZS 15th from top #113 3 MR_SPARE_16 16th from top (#112 not connected) 2 Pulled the NWA Framework apart to get it ready to load into the van and bring back to MSU along with its Power Pan. Returned the repaired Tier 1 Power Pan PDM-21 to D0 Hall. Talked with Dan Owen about the operation of the L15CT Global Tool during MFP events. With Steve write a note to Dan and Jim Linnemann about the modification of the EM-Fraction Electron Tool. I found that L1 68k had been booted sometime since I was here last week. They booted in the old standard program. I read the DAQ Expt log book and could find nothing about booting this machine or anything about problems with L1. I have no idea when this happened or who did it. I have reloaded the ABS_TEST version of VTC code. Finally get around to editing the TrigMisc: How_to_Replace_a_Power_Pan document so that it says to cover the ERPB, to pull of just the narrow piece of G10 cover, and it describes how to handle a Tier 2 Tier 3 Power Pan. Write a file in the TrgHard: [RPSS] directory that describes the new 16 Channel Drip Detector and the cables that connect it to the Drip Strips. Let's mount it in M110 to make the cabling easier. Send the cable information to John Fogelsong and schedule to Dan Owen. Watch the "error" codes coming from VTC. About once per minute you see a TF code (missmatch And-Or Terms 112:119) and at a much smaller rate you see a T6 code (missmatch And-Or Terms 40:47). Based on the source of these And-Or input both of them could be some what random. If a purely random signal came into the And-Or then what would you expect for the rate is mismatch between the two halves? There is about 10 nsec of cable between the two halves. It is a 3500 nsec cycle. So completely random data changing at 300 kHz would mismatch perhaps once every 350 reads due just to the cable skew between the two halves. One mismatch per minute at 100 Hz is about one in 5,000. At this one per minute rate I'm going to leave the ABS_TEST running in VTC. .............................................................................. Date: 28-FEB & 1,2,3-MAR-1995 At: Fermi Topics: Get things running again after the Feb '95 Shutdown Date: 28-FEB-1995 All of the following cards had their 10H101's removed from their Timing&Sync signal section during the February 1995 shutdown. Tier 2-3 CTMBD's RACK CTMBD SN# Mother Board Address and Switch Setting -------- ----------- ----------------------------------------- M105 T2 28 MBA = 177 1011 0001 M107 T3 02 MBA = 153 1001 1001 M111 T2 01 MBA = 249 1111 1001 Tier 1 CTMBD's RACK CTMBD SN# Mother Board Address and Switch Setting -------- ----------- ----------------------------------------- M103 U 07 MBA = 169 1010 1001 M103 L 04 M104 U 05 MBA = 172 1010 1100 M104 L 11 M105 U 18 MBA = 170 1010 1010 M105 L 29 M106 U 30 MBA = 175 1010 1111 M106 L 32 M107 U 16 MBA = 201 1100 1001 M107 L 23 M108 U 25 MBA = 204 1100 1100 M108 L 33 M109 U 06 MBA = 202 1100 1010 M109 L 21 M110 U 26 MBA = 207 1100 1111 M110 L 15 M111 U 12 MBA = 225 1110 0001 M111 L 27 M112 U 24 MBA = 228 1110 0100 M112 L 22 Check out the noise from the Upper Tier 1 Power Pan in rack M108. With the hose test it still appears to be the -4.5 Volt brick. Flukey says that there is between 10mV and 50 mV of AC coming from the -4.5 Volt brick, so I decide to pull it. Because the L1 racks are now even more crowded with the L15CT equipment we should make a couple of changes to the official procedure for replacing power pans: 1. Do not completely remove the Shockless System G10, rather just remove the two brass screws that hold on the small rear piece of G10, don't remove the 1/4-10 bolt. Removeing this small vertical piece of G10 from the shockless system cover gives enough access to remove the screws that hold the horizontal piece of G10 in place. 2. Using a paper towel or Chem-whipe, cover the L15CT ERPB card that is immediately under that area where you will be unbolting the big DC power cables from the Power Pan that you are removing. This is to prevent damaging the ERPB by dropping bolts, nuts, or washers on it and to prevent small pieces of metal that get scraped off of the DC power lugs from landing on the ERPB. Power Pan PDM-21 was pulled from M108 upper T1 and is being returned to MSU. It was replaced by PDM-14. Date: 1-MAR-1995 Remove the 10H101's from the Timing&Sync section of the Level 1.5 Framework CTMBD. This is CTMBD SN#34. The Level 1.5 Framework CTMBD has MBA=57. Remove the 10H101's from the Timing&Sync section of the Level 1 Cal Trigger Final Readout CTMBD in the Framework Expansion backplane. This is CTMBD SN#31. The Level 1 Cal Trig Final Readout CTMBD has MBA=154. In the Framework Main Timing MTG remove the PROM #3N and replace it with #3M. This returns the COMINT Clock back to full speed operation. The PROM #3N had been "temporarily installed last summer will investigating the problem reading out the two copies of the And-Or Term states. Remove the protective G10 covers from L15CT. Power up L15CT. Date: 2-MAR-1995 Remove the 10H101's from the Timing&Sync section of the CTMBD that is at the very bottom of rack M101. This is the CTMBD that is only used to drive timing signals into the And-Or Network and and scalers. This CTMBD does not have a MBA because it does not transfer anything on the data section of the CBus. This is CTMBD SN#03. Remove the 10H101's from the Timing&Sync section of the CTMBD that is at the bottom of rack M114. This is MBA = 106. This is CTMBD SN#20. Power up everything. The first problem is that we can not talk to the CAT2 cards in the bottom Tier 1 backplane in rack M112. This is CBus=1 MBA=228. We can talk to the CTFE cards OK. Perhaps something like the FA=32 line is stuck high, or the MBA switch does not work at 228 but did work at the test address at MSU. So Pull CTMBD SN# 22 out and install CTMBD SN#10 in lower Tier 1 M112. CTMBD SN#22 goes back to MSU to discover its problem. Delete the temporary (for shutdown only) version of TRICS_BOOT_AUXI.Dat and verify that D0::TrgCur: and D0HTCC:: both have the good proper version. Remove the funny extension from TRICS_Init_Auxi_L15CT.dat on D0HTCC:: Re-initializing was not sufficient (symptoms: about -2000 GeV of EM and HD 1st lookup, and EM 2nd lookup and 0 GeV of HD 2nd lokup). TCC had been rebooted while the trgtwr coverage was reduced, and had thus failed to read the normal INIT_DAC_BYTES.LSM. All TrgTwr DACs were thus loaded with the default value of 10 and gave a low ADC output of ~ 0 (instead of 8). Thus TCC needed to be rebooted again to read an use the correct ADC values. Load new system EWORK1:TRICS_V64.SYS_2MAR95 - old one is in ETRICS:TRICS_V63.SYS_19OCT94 - New Monit Pool Server for TRGMON with longer integration time. (only one mpool_server now) - Fix message length for mail message "BAD returned to COOR". (this problem was crashing the mail server) - Add new message to change the threshold for the error message filter. This would be useful to see ALL error messages during initialize. (boot default value is 50) $ @EENV:COMMANDS $ PHAT ERR_FILT nnn with 1 <= nnn <= 9,999,999 Update TRGUSERROOT:[TRGMON] release of TRGMON V6.1 with option for setting longer integration time - old files are renamed from xx.yy to xx.yy_OLD and will be saved for awhile - "improve" the menus for setting integration time and polling/refresh time (these are now 2 separate comands) - Add options during Print/Dump Screen to close the file or change its name - add between screen dumps. - sense and use full screen length during program startup. - Only one version of TRGMON left (no more TEST version) Made LOW and HIGH Amplitude pulser runs. They are run number 89102 and 89107. The data is in the normal place on Data3:[Cal]. Using L15CT_Prov these runs look OK. Ran 150k loops of CALTRG_RAND with all options enabled except locked on lookup page 4. S-HTT/PAR%rand% Loop 150000/150000, Error Count is 0 E-HRD/TST%rand% 150000 Loop Test Completed, No Error Detected Top is officially announced at the D0-CDF Fermi News Conference. Date: 3-MAR-1995 Edit the Power-Up instructions for the L1 racks to include the procedure for running with the Photo-Helic differential pressure gauge. In TrgCur: there are now 4 versions of VTC code: RunMe68020.ABS ! Standard version since July 1994. RunMe68020.ABS_NO_CAL_TRIG_ERROR_CHECK ! For running with L1 CT turned off. RunMe68020.ABS_ETA_16 ! Zeros the eta 17:20 data. RunMe68020.ABS_TEST The test version compares the two reads of all of the AND-Or terms (except for muon terms 32:39). I have left this version running. If they need to reload they will pickup the "safe" standard since July 1994 version. Because COMINT is back up to full speed I wanted to check that all was OK with the two And-Or reads. The decision was taken to cut the resistor on the BLS card from the noisy Calorimeter element that drives Trigger Tower +6,23 EM. We watched on a scope plugged into the CTFE monitor LEMO as Joan cut the appropriate resistor and the noisy when away. Jan has removed TT +6,23 EM from the Excluded TT List. See the 30-Nov-1994 log book entry for more details about the noisy Calorimeter element that is in the +6,23 EM TT. See the 12-MAR-1993 log book entry for details about the two other TT's that have a resistor cut: -4,21 HD +1,24 EM. Now with a flat 3 GeV EM Ref Set and requiring one TT the rate is about 1 Hz and with a flat 3 GeV Total Et Ref Set and requiring one TT the rate is about 3 Hz. Dan Owen looks at yesterday's Pulser Run and finds that TT +10,2 HD is at only half of what it should be. I look at things a find a bad Term-Attn Resistor Network at +10,2 HD. Replaced it and fixed the broken (cold solder) one. OK we try a Global Physics Download and have the following problem: REFSET TOTET SIGN_ETA(POS) MAGN_ETA(1:20) PHI(1:32) 0 (3000) SIGN_ETA(NEG) MAGN_ETA(1:20) PHI(1:32) 0 (3000) Failure Writing 18 @ cbus 0 mba 204 ca 45 fa 40 read 2 (cont) Data = Out 00010010 In 00000010 Mask= W 11111111 R 11111111 Failure Programming Trig Tower NEG,E_10,P_7 TOTET_REF,REF_0 Execution Failure at command #12 ACKNOW 000028CD BAD FAILURE %% time: 03-MAR-1995 13:59:12.17 I try loading value 18 into this location by hand with no problem. We had earlier in the morning loaded a Tot Et Ref Set of 3 GeV with no problem. Later in the day we repeat the global down load and have no problems. So we need to watch the log for a while to see if all is OK or if this location is getting sick. Tried the Run Number 100k test and all went OK. .............................................................................. Date: 13 evening,14,15-FEB-1995 At: Fermi Topics: Begin Feb '95 Shutdown Turn off the power to all of our equipment. Verified that it is the upper T1 Power Pan in rack M108 that is making the funny noise. It may be the -4.5V brick fan making the noise but it is hard to tell with the general background air flow noise. Pull 23 CTMBD cards to remove the 10H101's in the Timing-Sync Signal M103 Upper T1 Lower T1 M104 Upper T1 Lower T1 M105 Upper T1 Lower T1 Tier 2 M106 Upper T1 Lower T1 M107 Upper T1 Lower T1 Tier 3 M108 Upper T1 Lower T1 M109 Upper T1 Lower T1 Tier 2 already was de-H-ed M110 Upper T1 Lower T1 M111 Upper T1 Lower T1 Tier 2 M112 Upper T1 Lower T1 CTMBD's that were NOT removed but that DO need to be de-H-ed. M101 Very Bottom CTMBD (fans out Timing Signals to And-Or Terms) M103 Level 1.5 Framework CTMBD M103 L1 Cal Trig Final Readout CTMBD M114 Bottom Backplane (our and foreign DBSC scalers) CTMBD CTMBD's that were NOT removed because they were already de-H-ed. M102 Top backplane Start Digitize backplane M102 FSTD Backplane M109 Tier 2 Backplane 4 CTMBD's in the Spare Card Storage Rack Check the voltages of the Level 1.5 Cal Trig Power Pan before turning it off. Check at the front panel tip jacks Supply Voltage +5.0 V 5.198 V -2.0 V 2.027 V -4.5 V 4.521 V -5.2 V 5.407 V These all match within a couple of milivolts of the target voltage for these power supplies that are specified in the Power_Pan_Inventory.LBK file. Work on the problem that we have had for the past week of TCC not being able to program L15 FW Term #15 for L1 Spec Trig #11, i.e. Failure Writing 240 @ cbus 2 mba 57 ca 34 fa 27 read 248 Data = Out 11110000 In 11111000 Mask= W 11111111 R 11111111 I pulled this DIGIMEM card (SN# DGM-021) and the problem was obvious. 6 pins on the 74ALS574 register for this data were never soldered (pins 16:20). This card has in its history received other work (i.e. there are two white wires to fix Electro Circuits etch problems) so people have just not looked closely at this card while working on it. While working on this DIGIMEM card I noticed other strange funny things about the DIGIMEM in general: The print set (at least the copy at Fermi) never shows the address decoding anywhere in its 21+ pages. The logic to bring the results of the various terms together into the 4 final results has an extra unnecessary step in the logic. The driving of the results off card has an extra step. First the signals go through 10H124 TTL_to_differential_ECL converters and then the resulting ECL signals go through 10H101's before going off card. Why? That's right I said that there are more 10H101's in a critical place right on all of the And-Or and DIGIMEM cards. Connected the PhotoHelic to the RPSS. This is tied into the RPSS as the Air Flow sensor in rack M111 (I believe I have the number correct, it is labeled on the front panel of the RPSS). I verified that the lamp current in the PhotoHelic is 65 ma. The low pressure set point of the PhotoHelic was set at about 0.4" water. Note that we would like to set this higher (e.g. 1.2" water) but it can only be set to the value that the PhotoHelic has reached by the time that the PRSS tries to turn on the rack power. Currently with a 5 second delay in the RPSS this is about 0.52" water. If after a couple of weeks all appears OK with the PhotoHelic then I may change this delay in the RPSS from 5 seconds to 10 seconds but this requires opening the RPSS and then "re-certifying" that all of its functions are OK. Replace the VT320 terminal that is TCC's console. This terminal quite working the morning of Feb 10th. I replaced it with the second (spare) VT320 from our desk-workbench in the kitchen. I brought the dead VT320 back to MSU (along with the still good keyboard from what was the spare terminal). The VT320 that died is SN# TA81694567 it was replaced by the spare terminal from our D-Zero "office" SN# HK01650143. At this time I do not plan to repair this dead VT320. Installed a fan in the back of the L15CT Power Pan. This fan is bolted to the lip on the rear of the top cover and is connected to 115 VAC power via a standard fan power connector. To remove the top cover of this Power Pan first unplug the power connector from the fan. This fan does have a finger guard over it. VESDA test. On the 15th, with the Fermi Fire Techs we did a canned smoke test of the VESDA. Power to the racks tripped off and the FIRUS panel received the correct alarms. We are running with L1 Cal Trig turned OFF and L15CT turned OFF. To make TCC's job easier the following files were temporarily changed (just for the duration of the Feb Shutdown. TRICS_BOOT_AUXI.Dat This file had the RANGE of L1 Cal Trig coverage set to only 1 Trigger Tower. This file has only been changed on TrgCur: at D0: and on TCC's disk. The pre shutdown version was not purged. TRICS_Init_Auxi_L15CT.dat On TCC's disk only this file was renamed to .dat_not_during_shutdown These two things are the only things that need to be undone to get the system running again. Setup this way TCC can do an Init in about 50 seconds. During the RMI tests in the 1st floor MCH people got all cranked up because M124 (the L15CT rack) did not trip off when its RMI tripped. This RMI was installed in early summer 1993 as part of getting this rack ready for the installation of the L15CT. This RMI appears to have only drip and water flow sensors. It has no smoke sensor and they say that they have no more smoke sensors. The Power Distribution Box in M124 is not controllable via an external signal so it is impossible to setup anything to trip off when there is an Alram. This RMI is not connected to its Shea 1553 box so there is no way for it to send an Alarm to the Alarm System. The controllable Power Distribution Boxes require a 3 phase receptacle in the overhead wiremold and there is not a receptacle available for rack M124. So far the conclusion is that the RMI will be connected to its Shea 1553 and drip and waterflow alarms will be sent to the Alarm System. More serious work can be done in the summer 1995 shutdown. I have left the L15CT crate covered with G10 (front,top,back) and it is powered off. .............................................................................. Date 10-FEB-1995 At: MSU-Fermi Topics: VT320 TCC console dies and eventually stops TCC. My guess as to what happened is the following: The VT300 terminal that is D0HTCC's console died, and it died in such a way as it was sending out random characters (or in fact any character or just the start bit). This will eventually kill TCC and if the stream of junk characters keeps up it will prevent TCC from Booting and doing its job. Or the died VT320 may have sent a "control S" when it died, which would have eventually stopped TCC. The fact that the screen on this VT300 is always blank (even when typing on its Keyboard) and that it smells burned is a good indication that something in it died. The current situation is that the Level 1 68k console VT320 terminal has been swapped with the D0HTCC VT320 console and that the not working burned smelling VT300 is turned off. The Level 1 68k should have no trouble running without a terminal until the shutdown begins at which time I will replace it. .............................................................................. 7-FEB-1995 At: Fermi Topics: Repair of the water leak, Need more hose at Fermi, Questionable Power Pans, Start Up Procedure after L1 FW Power Outage, New Drip Detector, Trouble in L1 FW, Solder radiator repair kit stored at Fermi The accelerator was doing 12 hours of studies starting at noon on the 7th so when the store was dumped I shut off all of the L1 racks to repair the leaking radiator. I turned off the L1 Cal Trig Power Pans one at a time to try to learn which one in the vicinity of M109 is making the screaming arcing noise. When I turned off the upper T1 in M108 the noise when away. I do not know that anything is wrong in this Power Pan but it does make this strange noise. I do not know which brick in the pan makes the noise... There was a problem in removing the leaking radiator from the 8-Pack in M112-M113. I thought that the 8-Pack was a frame into which the radiators fit. This is wrong. Aluminum plates were welded to the sides of the radiators and these plates are bolted together to make up the 8-Pack. To avoid the long job of removing the whole 8-Pack, I cut the welds on the four corners of the leaking radiator. The new radiator is held in by the front screw in bars and I also RTVed the 4 corners of the new radiator to the plates left in the 8-pack from the old radiator. All of this went OK and looks strong enough. Mostly it is just stupid how the 8-pack is built anyway. The work that I did not like is putting the new water hoses onto the new radiator. I did not use 90 degree elbows like I should have, and the hose I had at Fermi was kind of stiff. It took a lot of force to push the hose onto the barbs, much more force than I like to use on these week radiators. The routing of the new hoses is not to my liking. The old leaking radiator was tagged, and left for Del Miller. Dan Owen was going to pass it to him. When turning the power back on to the L1 system, I had to cycle the Upper T1 M107 Power Pan 3 time to get the +5 Volt brick to start running. I left it turned on for about 30 seconds (with the +5 Volt brick not running) and then turned it off for about 5 to 10 seconds and then turned it back on to get it running. Note this is the same brick that Stu Fuess had trouble getting to run on Saturday night after the power trip. Note also that Upper T1 M108 makes noise and Upper T1 M107 is hard to turn back on; is there a book keeping error here or something else funny going on. Trouble in L1 FW When L1 was turned back on after the radiator repair, it started having an error come from a DIGIMEM card in the L15 FW. TCC would try to write 240 to an address and it would read back 248, i.e. it appears that the bit of value 8 is stuck on. I hope that this will not interfere with anything. This bit controls the "sense" of L15 Term #15 as it is used by L1 Spec Trig #11. Examples of the error are: Failure Writing 240 @ cbus 2 mba 57 ca 34 fa 27 read 248 Data = Out 11110000 In 11111000 Mask= W 11111111 R 11111111 Failure Programming Spec Trigger #11 Requiring Level 1.5 Term #15 Execution Failure at command #2 For now we are running with this (it should not cause any trouble). Jan has been sent mail about it. We need some new good soft 3/8" hose to keep at Fermi to make future repairs. I used up all of the hose that I had there. I left the radiator solder repair kit at Fermi in one of the desk drawers under the work bench next to the spare card storage rack. The kit is: the big nice soldering iron, the big diameter solder, steel brush, sand paper, stay clean flux. Procedure to Restart D-Zero after L1 FW Power has been off ---------------------------------------------------------- We tested to see what it takes to get everything running again after a L1 FW power outage. The official procedure is: Reset all CD crates (i.e. push the Reset buttons on their Vertical Interconnects) redo the Detector Down load (this appears to get Muon going again). That's all there is to it. New Drip Detector Dan Owen is cranked up about the idea of making a new fancy drip detector for the L1 racks that would have a separate LED indicator for each rack and a single global output that says that there is leaking water somewhere in the L1. He is going to talk to people about getting Fermi to make a 16 channel object with each channel like the insides of the RMI. Fermi would build this, I would install the separate run cables to each sensor. .............................................................................. Date: 5,6-FEB-1995 At: MSU-Fermi Topics: Water Leak in L1 Racks 5-FEB-1995 Water leak drip detector power trip of L1 System At 2:06 the Drip Detector for the Level 1 system turned off all of the racks. The racks were searched and no leaking water was found. This included removing the back doors from the racks where the cable bridge cables come into the 1st floor MCH. Because no leak could be found it was decided that it was OK to try to run again. To accomplish this the DIP switches in the top of the RMI at the top of rack M114 were set to ignore the DRIP detector. All L1 doors were closed and checked. The "green button" on the L1 Rack-Power_Safety_System was pushed and AC power restored to the Level 1 system. The +5.0 Volt power supply in the M107 Upper Tier 1 Power Pan did not start up when AC power was restored to the racks. Cycling the breaker on this Power Pan got all 4 supplies in it running. At 3:20 the L1 system was initialized and ready to go. Then there were problems with: Most of the muon crates being 100% Front-End Busy L2 begin run (and other ?) problems A vertex crate that needed to have its Vertical Interconnect reset Physics events started flowing again at 4:50 Current status: We are running the L1 system WITHOUT water leak protection. New paper towel drip detectors are being installed in the back of the L1 racks. These will be checked at a regular interval until the RMI drip detector in once again enabled to trip the power when it sees trouble. Dan Owen is currently in North Carolina. I'm not certain when he is getting back to Fermi. I will go to Fermi sometime on Sunday and plan to check again for water leaks between stores. I will also check to make certain that the RMI is working OK. I will take the radiator repair kit. The current (this weeks) version of L2 Supervisor does not let Spec Trig 31 run when other triggers are enabled (or something like that). Thus we spent time with STALE TrgMon data when it would have been nice to be able to have seen what was going on. Stopping Spec Trig #31 is the "standard" thing that Brown does every 6 months or so to "fix all the problems". 5,6-FEB-1995 Status of the water leak in the L1 system, Noise in Power Pan, How to trick the RMI and not turn off L1 racks The water leak is in the "8-Pack" of radiators between M112 and M113. It is in the next to the bottom radiator facing M113. I believe that the leak is in the joint between the short copper header pipe and the actual copper tubing of the radiator. The area around this connection has a lot of green corrosion on it. It is difficult to see this area on this radiator. The leak is about 5 drips per minute. It appears to be just drips and not the normal fine spray. I believe that this radiator can be removed without too much trouble. I have put a piece of plastic tubing around the end of this header pipe to direct the leaking water to the bottom of the rack. This water had been dripping on the M113 drip detector strip and the top of the RPSS. I do not believe that there was any damage to the RPSS. I can not see any water on the electronics cards in rack M112. If this drip does not get any worse, and if the plastic tubing works to direct the leaking water to the bottom of the rack; then I propose waiting until the February Shutdown to repair this leak. The RMI Drip Detector has been re-enabled to "look" for leaks in all of the racks except for M113. It has NOT yet been re-enable to trip the L1 power off if it sees a leak. If during the next store the RMI Drip Detector does not see any additional leaks in racks M100-M112 then at the next shot setup I will re-enable it to trip the L1 system power. Score Board. So far the RMI Drip Detector in the L1 system has found two real leaks and caused ZERO false alarms. Other notes: While searching for the water leak I noticed that one of the power supplies in rack M109 is making a lot of noise. It is kind of a square wave arcing type of noise. The fundamental is above 60Hz but it can't be too high because I can hear it. Using a 5 foot length of 3/4" hose I tried to locate the brick that is making the noise but I could not even tell which Power Pan it was coming from. Or even if it was coming from M109 and not a neighboring rack. Next time we turn off, lets turn the pans off in this area one at a time and listen after each pan is turned off. Trick to fool the RPSS into thinking that the RMI says all is OK. I wanted to remove the RMI to test it before looking for a leak. The drip sensor input to the RMI had already been disabled on the 5th (via the DIP switches on the top of the RMI box, so all the RMI was doing was acting as a +5V 10 ma power supply that was telling the RPSS and the Shea rack voltage monitor that all was OK. The BNC output connector of the RMI goes to a "T" connector and then to the RMI-to-RPSS box in the bottom of M113 and to the Shea rack voltage monitor (which can generate an Alarm System message about water leaksin the L1 racks). Between stores I pulled off the BNC cable that goes to the Shea rack voltage monitor (this temporarily generated a fatal alarm). I installed a second BNC "T" connector at the point where the cable to the Shea box had been connected. I then reconnected the cable to the Shea rack voltage monitor to one of the ports on this new "T" connector. On to the unused port of this new "T" I installed a new BNC cable that ran over to a small +5 volt power supply. I then could pull the original BNC "T" connector off of the RMI and never interrupt the "everything is OK" signal that was going over to the RMI to RPSS box in M113. Thus the RPSS never tripped the power off to the L1 racks and the RMI was now free to work on. This procedure can be reversed to reinstall the RMI. 6-FEB-1995 Status of the L1 system water leak. 22:00 Enable the RMI to Look and Trip Power, Check on L15 Cal Trig DSP UnStick. The water leak from the M112-M113 "8-Pack" of heat exchanges does not appear to have gotten any worse during the last store. The plastic tube continues to carry the leaking water to the bottom of the M113 rack. If this leak does not get any worst then the plan is to wait until the shutdown next week to repair it. The RMI drip detector for the L1 system is now enabled to look for leaks in racks M100 - M112 and to trip the L1 power if it sees a leak. L15CT look at how it is running during a store I have not checked the L15CT in the last couple of weeks to see if 68K services needed to "Un-Stick the DSP's". In fact I have not looked at this since Tool #3 was installed. So I looked at it for the last store. The triggers were loaded at 1:18 on 6-Feb and not disturbed until 18:04 (except to change prescales). There were 0 Un-Sticks required. Other statistics: "That's Me" With Transfer Count = 1919275 "That's Me" NO Transfer Count = 1729507 "Bystander" With Transfer Count = 4030187 "Bystander" NO Transfer Count = 21616954 "Mark&Pass" With Transfer Count = 364 .............................................................................. Date: 3-FEB-1995 At: MSU Topic: Translation of raw TT Energy to L1CT corrected Et and L1.5 CT Input Energy This is a summary of what we know, and have to re-invent every time we look at an event. - L1 CT Raw Et This is the data found in the L1 Crate of the TRGR bank. One can get this information using one of the following routines L1UTIL_ADC_COUNT_UNPACK.FOR from LEVEL1 L1UTIL_TRGR_ADC_UNPACK.FOR from LEVEL1 L1EXTRACT_TRGTWR_ADC_ENERGY.FOR from ZEBRA_UTIL Or look in array FADC_BYTE of L1C_EVENT.INC after L1SIM - L1 CT Corrected Et: This is the data after PROM Lookup. To get this data, one needs to let L1SIM unpack the TRGR bank, and use the lookup system manager, then find the data in the array Z_CORRECTED_ET of D0$INC:L1C_Z_CORRECTED_ET.INC Since no z-correction is performed, one can simply derive this information from the raw data (at least for EM and HD, but not PX and Py) Both EM Et and HD Et have a low energy cut. EM Et only requires a minimum count of 2 from the zero energy response HD Et requires a minimum count of 2 for TT_eta > 4 HD Et requires a minimum count of 3 for TT_eta <= 4 A raw EM or HD ADC count of: 5 6 7 8 9 10 11 12 Gives an energy lookup (Gev): -.75 -.50 .00 .00 .00 .50 .75 1.0 Except for HD (TT_Eta <=4) : 5 6 7 8 9 10 11 12 Gives -.75 .00 .00 .00 .00 .00 .75 1.0 - L1 CT Large Tiles: This is the same as the raw TT ADC, correctly summed. One can to let L1SIM unpack the TRGR bank, and find the data in the array LT_ENERGY of L1C_INTERMEDIATE_ENERGY.INC or The difficulty is in correctly adding the raw EM and raw HD for each tower and dropping the least significant bit before summing it into a large tile Tot Et. - L1.5 CT Energy: EM Et is the same as the L1 CT raw Et. Tot Et is the same as the L1 CT Corrected Et. .............................................................................. Date: 30,31-JAN-1995 At: Fermi-MSU Topics: Pulser Run On Monday Jan 30th Jan and Joan make a Pulser Run. On Tuesday from MSU look at this data and all appears OK. Send notes to Guida's and to Dan Owen. Ask that this data (runs 88340 and 88341) be kept on disk in case it is the last Pulser run before the shutdown and we need to use this data to decide about what (if and) BLS --> L1 work to do during the February Shutdown. .............................................................................. Date: 25,26,27-JAN-1995 At: Fermi Topics: More spare equipment to Fermi DRV11-J and pVBA, February shutdown schedule and activities, meeting with Jim Christenson, Run II Online system vs TCC. Bring two DRV-11J 's to D-Zero Hall as spares. These are in boxes hand labeled "DRV11-J Sept 93 D0" and "DRV11-J D0". Also bring the spare pVBA to Fermi. It is in its "mini VME 4 slot crate" with its special cables. The DRV11-J's and the pVBA are stored in the bottom of the Spare Cards Cabinet along with the spare VME cards and the spare Hydra II. See the Log Book entry from 9,10,11-NOV-1994 for an inventory of these cards. From the Run Meeting: Schedule for the shutdown: Last store will go in on Feb 12th, No beam No access on the 13th, D-Zero access starts on the 14th, Accelerator startup begins on March 6th, the schedule calls for one week of startup before HEP begins, but it is likely to be longer and/or start at low luminosity. During the February Shutdown they are installing RF cavities to make better coallessing. This should improve Luminosity by a factor of 1.3 i.e. we may see 20E30. Delivered the MOU to Jim Christenson and a copy of the Jan '95 Rev of our schedule for upgrade work. This included a cost per year out through '98. He wants to know if we could spend money this year. I said that I would get back to him within a week. Talked with a number of people about the Run II online host system. There have been some meetings about this and we need to start going to these meetings. Stu Fuess sets up the following basic question-path: either its unix (or OSF1) on some type of normal unix hardware or else the other choice is NT type of OS on PC type of hardware. In either case its thought that the hardware will have a PCI bus and we will use a memory mapped PCI to VME adapter (either ZRL or some other company). There is a white wheeled card transporter box at Fermi that can be used to bring the CTMBD's to MSU in February. For the February shutdown: Turn OFF, Tag OFF Tier 1, Tier 2, Tier 3 and Pull their CTMBD's. Load the 68k VTC program version that does not verify L1 Cal Trig data. Switch the toggle switches on the L15 CT Term Select card to the manual over ride position that forces Innocent-By-Stander. (Or we will get a phone call). Other things to do during the February Shutdown: Speed the COMINT back up to full speed. Start running as standard the 68k VTC program that tests all IMLRO data except for the muon stuff. Install the Photo-Helix differential air pressure gauge into the RPSS. Install a new System load file for TCC. Test the VESDA smoke alarm Add a second fan to the L15CT Power Pan Before turning things off for the Feb Shutdown: Run some random cell exerciser diagnostics. Have pulser runs to verify BLS --> L1 Cal Trig and L1 Cal Trig --> L15CT. .............................................................................. Date: 16-JAN-1995 At: MSU Topics: L15CT Pulser Run Jan and Joan make another pulser run. This time all works OK, i.e. the pulser is incrementing patterns properly. The L1 to L15CT data connection analysis looks all Ok. The new files are in Data3:[Cal]. The run numbers are: 87947 is the High amplitude 87951 is the Low amplitude. I have deleted the L15CT_Prov log files from last weeks runs # 87875 & 87877 where the pulser did not increment patterns. .............................................................................. Date: 12,13-JAN-1995 At: D-Zero Topics: Possible L15CT TAS Number Problem, Collect L1 Trig Overlap data for Trig List 10.2, Try to make a L15CT Pulser Run, February Shutdown Schedule. At about noon the Global Monitor noticed that the L2 nodes were only filtering events at about 50Hz. This woke up the DAQ Expt who looked at SupStat and noticed a huge number of events being dumped as they entered the L2 Farm because they had an "Event Number" missmatch i.e. a TAS Number problem. Recall that only the 4 lower bits of the TAS Number need to be OK for the L2 system to be happy while it is transporting the events. Then once in an L2 node Jan Hufton checks all 16 bits for a match. Jan Guida was called. She was first quickly able to see that it was a problem with a crate on the TRGR cable. Then she determined that it was the L15CT crate. Then she started ZBDump and for a bad event looked at the headers from all the crates on the TRGR cable. In the 16 bit TAS Number coming from the L15CT crate the following bit was always low: 2nd word in the Crate Header. 16 bit TAS Number ------------------------------------- 12 bit of L1 Trig # Sync# --------------------------- ------- MSBit x x x x x x x x x x x x x x x x FFFF LSBit ^ | The run was ended. The L15CT crate VME Bus RESET button was pushed and the L15CT 68k reloaded. The triggers were down loaded and things started up OK. Possible Causes: TAS Cable Connector Driver Receiver IC?? Possible Can the L15CT program (which has not been reloaded since 1-NOV-1995 have become corrupted in a very strange way so that it would stick this bit low? I doubt it. Could a IRONICS Module have something random written to one of its ports that read the TAS Number so that it sticks a bit low? Possible If this happens again this can be tested by doing something like: ABORT the L15CT 68k. Look at the Byte wide Short IO addresses used by the IRONICs that reads the TAS Number. The TAS Number is read by the Readout Control IRONICs Ports #5 and #5. Port #5 All bits are Inputs. They read the TAS Number bits 8:15 Port #6 All bits are Inputs. They read the TAS Number bits 0:7 Readout_Ctrl_Port_5 EQU $FFFFF049 Readout_Ctrl_Port_6 EQU $FFFFF04B Using the Bug135 one could read these locations and then write $FF into them. That's right, writting $FF allows them to be inputs. For example MD FFFFF049;B or MM FFFFF049;B Then Continue the L15CT 68k running and see if L2 is happy with the events. February Shutdown The current schedule for the February Shutdown is for it to begin on February 12th and to last for three weeks. At the monent we have the OK to pull apart the L1 Cal Trig. The idea is to do it early and then get it running again before the end of the shutdown. Level 1 Trigger overlap data was collected during three runs during Store 5333. This overlap data is in three separate files, one for each of the three first runs during this store. Run 87854 between about 1:00 and 4:03 on 12-JAN-1995 prescales V102-11E30 Luminosity at the start of this run 9.5 E30 Overlap data is in the file: SPTRG_FIRED_RUN_87854.TXT Run 87855 between about 4:14 and 8:41 on 12-JAN-1995 prescales V102-08E30 Luminosity at the start of this run 7.7 E30 Overlap data is in the file: SPTRG_FIRED_RUN_87855.TXT Run 87856 between about 8:45 and 12:05 on 12-JAN-1995 prescales V102-06E30 Luminosity at the start of this run 5.7 E30 Overlap data is in the file: SPTRG_FIRED_RUN_87856.TXT Level 1 Trigger overlap data was collected during the first run of Store 5334. Run 87880 between about 20:40 and 22:44 on 12-JAN-1995 prescales V102-15E30 Luminosity at the start of this run 12.4 E30 Overlap data is in the file: SPTRG_FIRED_RUN_87880.TXT All of these files are in User1:[TrgMgr.TRG_680X0.SOURCE_1WORK] Pulser Runs Modified the Config files for the "Dan Owen" Pulser Runs so that they would work with the L15CT now using Tool #3. Then we made two runs: 87875 is a high pulser run and 87877 is a low pulser run. Looked at both with L15CT_Prov. Everything looks OK i.e. there were no mismatches but we are back to the problem of the pulser pattern not incrementing. .............................................................................. Date: 11-JAN-1995 At: D-Zero Topics: ATLAS Meeting, Run L1 Cal Trig Tests Atlas Trigger meeting at Argon National Lab. Test of L1 Calorimeter Trigger Run 100k loops of CalTrg_Random_Cell test. This is over all eta-phi. Enable all options except: lock on lookup page 4 and do not check And_OR Terms. This gives: E-HRD/TST%rand% 100000 Loop Test Completed, No Error Detected Now try a run over all eta-phi and enable all options except, do not check And-Or Terms. After about 4000 events this gives E-HRD/TST%rand% Global Py Momentum Sum is -863 instead of 3233, T1 trunc = -4096 E-HRD/TST%rand% Pick was232,HD,NEG,E_11,P_13,LUP_1-4-7-7,TOTET_REF,REF_2,168,CMP_3 This does not go away when the cycle is repeated. This is the same kind of stuff that I saw during the tests on 23-DEC-94. Go back to running all eta phi and locked on lookup page #4 and all is OK for 10k loops. They want the system back for next Shot. .............................................................................. Date: 9-JAN-1995 At: MSU Topics: Migrate L1.5 CT to Tool 3 (from Tool 2) for Trigger List Version 10.2 Migrated the L1.5 Cal Trig to Tool 3 (from Tool 2) for Trigger List V10.2. This was done by Steve at MSU in consultation with Brajesh (TriggerMeister) who was in D0 Control Room. The chronology of events was (all times are Chicago time as recorded on TCC): 20:00 Brajesh calls to say he is 45 minutes away from performing a test download of V10.2 (we are between stores). Steve agrees to switch L1.5 Cal Trig code. Brajesh agrees to call Steve before starting download. Steve and Brajesh talk about what would be required to revert to Trigger List V10.1 in the event of problems. Steve emphasizes why he is interested in carefully monitoring the test download. 20:04 Steve starts moving the files required for L1.5 Cal Trig Tool 3. This consists of the 12 executable files (D0HTCC::DUA0:[L15CT$EXEC]*.BLX) and the "default configuration file (D0HTCC::DUA0:[TRIGGER]L15CT_DEFAULT_CONFIG.DAT) 20:05 COOR INITIAL Level 1. This of course fails. 20:08 Steve finishes moving the files required for L1.5 Cal Trig Tool 3. 20:08 Steve calls Brajesh to ask why Level 1 was initialized. Brajesh does not know. Brajesh asks DAQ expert (R Hall according to D0NEWS), who also does not know. Steve reminds Brajesh that the L1.5 Cal Trig is not usable right now and that Level 1.5 must be downloaded before it can be used. Steve and Brajesh agree that it is sufficient to wait for the test download of V10.2. 20:52 Brajesh calls, he will begin download now. 21:03 Brajesh calls, there was a problem with download which was unrelated to Level 1/1.5. No messages were sent to TCC. He is trying to fix problem. 21:15 Brajesh calls, download will begin soon 21:20 Download to Level 1 begins. It looks 100% fine EXCEPT that there was no INITIAL message. Steve calls Brajesh to say that there were no problems with the download that would cause L1.5 to be unusable. Steve tells Brajesh and Dan Owen (captain) about the lack of an INITIAL message. After checking with DAQ expert, Dan O. claims to understand why this occured and thinks it is not a problem. Steve, Brajesh, and Dan O. decide to force Level 0 and flow some events through the system. 21:43 Begin about 15 minutes of triggering on noise. Three of the four Specific Triggers requiring L1.5 Cal Trig confirmation are seen receive this confirmation at least once. Specific Trigger 9 is never seen to receive L1.5 Cal Trig confirmation, but it also was not seen to fire at Level 1. This Specific Trigger requires 2 Level 1 "electrons" so is unlikely to fire with just noise. 22:03 Level 1 is initialized, quelling Steve's fears about being left in some screwed up state. .............................................................................. Date: 8-JAN-1995 At: MSU Topics: Cal Trigger rates go up for Tot Et Jets Mike Fortner & John Butler called about 1 PM Sunday afternoon. Some of the Spec Trig's that depend on Jet Towers have been having their rates jump up for a period of a couple minutes at a time. Spec Trig's that depend on EM Towers or Large Tiles did NOT appear effected. Muon Spec Trig's are not effected. During the period of high rates, TrgMon showed -9,23 in the Tot Et Jet List very frequently. The overall L1 rate would about double during these periods. Mike says that Spec Trig's #13, #16, and #20 are good ones to watch. Calorimeter Examine and rates out of L2 did not change (i.e. L2 dumped all of the junk events that we passed to it). After doing this for a couple of periods of a couple of minutes each the problem "went away". Can this just be "sparking noise" in the Calorimeter? Is this an ECL differential input oscillating because the bias spreader network is not centered at Vbb ?? For a report of a similar problem see the log book entry 31-MAR-1994 near the end of the entry. .............................................................................. Log book for 1994 is in D0_HALL_LOGBOOK.LBK_1994 ..............................................................................