D-Zero Hall Log Book for 2010 ------------------------------- The most recent entries are near the beginning of this file. This file begins in January 2010. This file contains both Trigger Framework and L1 Calorimeter Trigger entries. Earlier D-Zero Hall Log Books are on the web in one of the following directories: http://www.pa.msu.edu/hep/d0/ftp/l1/framework/logs/ http://www.pa.msu.edu/hep/d0/ftp/run1/l1/inventory_logs/ ------------------------------------------------------------------------------ DATE: -10 At: Fermi TOPICS: ------------------------------------------------------------------------------ DATE: 15:17-DEC-2010 At: Fermi TOPICS: ADF-2 to TAB Link Errors, SCL Hub-End Power Supply Replacement, VME I/F Firmware Install, TFW Power Supply Work, Luminosity Scaler Function Map ADF-2 to TAB Link Errors ATC Card Swap Starting at about noon on Wednesday we started to see a few ADF to TAB link errors. These appeared as SES Alarms to the CalMuon Shifter, "L1CAL_TABGAB_M107/EVT_FAILED_GAB_STATUS". The link with the error was Tab 3 Chip 8 status 0x8100 this link comes from the "A" connector of slot #14 in ADF crate "C". The rate of these errors was about 1 per minute and it went down to about 1 per 20 minutes as the store went on. The decision with the RCs was to complete the store before working on this link. At the end of the store, early Thursday about 3 AM, we did the following: - Power cycle the TAB/GAB crate and power cycle the ADF crate "C". While ADF crate "C" was off I swapped the ADF-2 cards in slots 14 and 15 (both D type cards). Slot 14 was ADF-2 SN# D5. slot 15 was ADF-2 SN# D6. Now SNs D5 and D6 have swapped slots. The result of this was - no change. We still had the same Tab 3 Chip 8 error. So the problem was not the ADF-2 card. - Next we swapped the "A" and "B" cables on the ATC card in slot 14. The problem moved to a different Tab Chip - meaning that the problem stayed associated with the "A" output of the ATC card in the back of slot 14. Time was up as a new store came in. Everything was back to normal except the ADF-2 card swap between slots 14 and 15 was left in place. Thursday at about 10 AM the store was gone so I could try replacing the ATC card to see if that would fix the link error problem. Did the following: - Before doing anything, check errors and see TAB 3 Chip 8 0x8100 errors at a rate of about one per minute. - Pull out the cable #79 from the top "A" connector and now see continuous TAB 3 Chip 8 0x8120 errors - Pull out the cable #88 from the middle "B" connector and now in addition to the above see continuous TAB 4 Chip 8 0x8080 errors. - Pull out the cable #75 from the bottom "C" connector and now in addition to the above see continuous Tab 5 Chip 8 0x8049 errors. - Pull ATC SN# 053 out from the back of slot #14 and install ATC SN# "Spare 3". Plug back in the cables in the order shown above. - I did not see any more link errors so after about 30 minutes I re-enabled the SES Alarm "L1CAL_TABGAB_M107/EVT_FAILED_GAB_STATUS". - We need to watch for both link errors and to make certain that the BLS Trigger Tower signals in the eta,phi region around slot 14 look OK, i.e. that all the BLS cables are plugged back into the ATC cards correctly. - We have one spare ATC card left in the brown spares cabinet. - We clearly showed that the problem with ATC SN3 053 that was pulled out of slot 14 has to do with its top "A" connector and it may have to do with the LVDS pair that carries the BX Number and Parity. I checked this card with an Ohm meter and it has an open trace. - We have one spare ATC card left in the cabinet. SCL Hub-End Power Supply Swap There was trouble restarting the SCL Hub-End power supply after the cooling water problem 2 weeks ago. See the log book from 2 weeks ago. There was down time today so I was allowed to work on it. First I just tried turning it off both its DC enable switch and its main AC input switch. When I turned the AC back on the +3.3V bricks power entry module just flashed On and Off red. It's clearly in very bad shape so I swapped it. The only completely good TFW supply that I had here was the "proto-type" TFW supply that I just brought down to Fermi from MSU this trip. I pulled TFW Supply SN# 12 out of service supplying the SCL Hub-End and I installed the TFW Supply SN# 1 i.e. the proto-type in its place. The pulled out TFW Supply SN# 12 consists of: +3.3V brick SN# 99420446 and the +5V, -2V, -4.5V brick SN# 99420439. It is the power entry module on the +3.3V brick that is bad. The big fan on the +3.3V brick looks completely cooked and is not turning at all. One of the AC External Fans is making lots of noise and is about ready to go. After the newly installed SCL Hub-End supply had been running for a couple of hours I checked its output Voltage. Checking at the logs on the cables on the power supply terminals I see: +5.058V +3.332V -2.042V -5.212V Probing the studs on the SCL Hub-End crate I see: +5.017V - -2.014V - At the crate its not easy to probe the +3V and the -5V so I did not move things around to get access to these studs. Compare these numbers to the log book entry from 2 weeks ago and they are very close to being the same. Per Bunch Scaler VME I/F Firmware There was time before the store went in this afternoon so that I was allowed to swap in the ReBuilt VME I/F firmware for the Foreign Per Bunch Scalers. I put the new VME firmware into just the Slot 21 card in. This card was picked because it is easy to get it in and out and because it is the Slot 21 card whose VME LED flashed when Slot 18 is being configured. Thus putting the new firmware into Slot 21 gave me an easy thing to look for to see if things looked "better". See the log book for 22:24 Sept 2010 for the LED flashes during configuration details. With the new VME I/F firmware in Slot 21 I did not see the Slot 21 VME LED flash except when Slot 21 was being configured. There was trouble configuring the M122 Top crate after installing the new firmware in just Slot 21. I had trouble getting Slot 15 to Configure. It went OK on either the 2nd or 3rd try. I have no idea whether or not this trouble had to do with new VME I/F firmware in slot 21 (but it is hard to see how it could). I got from Yannis the current mapping of Foreign PBS to what quantity they actually count: 0: 'FastZ_NIM_raw' 1: 'PbarHalo_NIM_raw' 2: 'ProtonHalo_NIM_raw' 3: 'FastZ_PAIR_raw' 4: 'DeadTime_NIM' 5: 'FastZ_OR_raw' 6: 'FastZ_VME_raw' 7: 'DeadTime_VME' 8: 'ProtonHalo_VME_raw' 9: 'PbarHalo_VME_raw' Philippe is watching carefully and in the first 2 hours of running with the new VME I/F firmware there have been no VME Bus Hangs. Yannis has looked at the data and things look OK. The P_Bar Halo rates look normal. TFW Power Supply Work: TFW Power Supply SN# 12 until today this had been running the SCL Hub-End It consisted of +3.3V brick SN# 99420446 and the +5V, -2V, -4.5V brick SN# 99420439. Problems: +3.3V brick SN# 99420446 just flashes On/Off red LED and has a frozen fan +5V, -2V, -4.5V brick SN# 99420439 is OK but clearly needs new fans one or both of the External AC fans is making lots of noise and needs to be replaced Pull this supply apart and take brick +3.3V brick SN# 99420446 back to MSU to be sent out for repair. Leave the +5V, -2V, -4.5V brick SN# 99420439 here to get new fans on my next trip. Actually first experiment with the +3.3V brick SN# 99420446. Run it by it self on the bench and it pulses red On and Off as expected. Its big fan is completely frozen. Pull this supply apart and disconnect its big fan. Reassembly and try running. It starts up OK. So the power entry module that feeds the fans can detect the over current due to the frozen or dragging slow starting fans and that's what makes the power entry module pulse red on and off. So this module probably just needs a fan replacement. TFW Power Supply SN# 11 One of the spares at Fermi. This consists of +3.3V brick SN# 99420448 and +5V, -2V, -4.5V brick SN# 99420440. This needs to be a spare but it has 2 problems: one of the small fans in its +3.3V brick SN# 99420448 does not turn at all its +5V, -2V, -4.5V brick SN# 99420440 pulses red on/off 3 or 4 times at startup. For now TFW Supply SN# 11 must remain our #1 spare at D-Zero. Note that +3.3V brick SN# 99420448 has basically zero hours on it since it came back from repair at ASTEC. see the 3:4-Dec-2009 log book entry. OK the problem with the +3.3V brick SN# 99420448 small fan not turning was the length of the mounting screw under it. It is a short screw but it is too long. So for now I just have 3 screws holds this brick into the chassis. Now this fan runs just fine. Also note that +5V, -2V, -4.5V brick SN# 99420440 has all new fans in it. See the 12:14-AUG-2009 log book entry. So in fact TFW Supply SN# 11 should be in good shape. Setup TFW Power Supply Chassis SN# 11 to the following targets: +5.050V +3.350V -2.110V -4.610V TFW Power Supply SN# 4 One of the spares at Fermi. This consists of +3.3V brick SN# 96390262 and +5V, -2V, -4.5V brick without a visible SN# but it says "M123M" on it. All fans appear to run OK. The serious problem is that the +5V module's LED is a combinations of Red & Green when the supply is running. Put just a small load on the +5V module and now it is happy. I used a 1 Ohm i.e. 5 Amp load. So I think this is running fine in all our TFW applications. Setup TFW Power Supply Chassis SN# 4 to the following targets: +5.050V +3.350V -2.110V -4.610V with about a 5 Amp load on each supply as it is set and always a 5 Amp load on the +5V supply. Need to bring a log 4-40 flat head screw for the front panel test point connection to the 3.3V supply. The visible fans are turning. M124 back top temperature moves around in the range 95.0 to 95.8 They are now running on just pond water heat exchanger cooling. I see it in the range 55.8 to 56.8 Both the "Crash Kit" tool box and the main tool box were open when I got here. ------------------------------------------------------------------------------ DATE: 1:3-DEC-2010 At: Fermi TOPICS: Cooling Loss --> TFW Trip, MVME stuff back from Petr, L1 Cal Trig recovery after the Cooling Loss. Petr returned the MVME133-A20 and the set of books. For now he will push on the OmniByte cards. He showed me his work on them. It's not at all clear what the problem is. Thursday early AM there was a failure of the cooling water. There was circulation but no cooling and its temperature went above 80 deg F. The problem was ice plugging the pond input to the sump (i.e. wind in the wrong direction). Cal supplies tripped off on temperature and finally the TFW tripped off on temperature. They turned off the magnets and most of the other electronics. At TFW restart they had trouble because the SCL Hub-End crate pulsed On/OFF. They were very rational and turned the DC switch on that supply back off and let things sit. Geoff came to the high rise to get me before 8. With the AC still running I checked the status LEDs on the SCL Hub-End supply - they all looked OK - so I put the DC switch ON and it started up just fine. The TFW Configured and Initialized fine. Things have been OK for the past 34 hours. The plan is to leave a labeled supply sitting out on the bench in case Mike needs to swap it in. I checked the terminals right on the running SCL Hub-End supply. It is running: +5.056V +3.347V -2.044V -5.220V See 26:28-AUG-2009 for a recent check of this crates voltages. I have setup TFW Power Supply SN# 11 to match this and will leave this supply on the bench. Note that the power entry module on this supply flashes a couple of times at AC on time. See 2:4-AUG-201 for details of recent work on TFW Power Supply SN# 11. The other spare supply that is here, TFW Power Supply SN# 4, appears to now also not be 100% happy. The red/green LED on its +5V module does not turn 100% green. It's clearly oscillating between red and green. It does appear to be running OK though. I have no load here to test it. Both TFW Power Supply SN# 11 and SN# 4 were tested this past summer. I need to get the supply from the MSU test stand back here now that the Scaler Card VME IF FPGA work is done (I hope) at MSU. I need to make and get here at 1 Ohm / 1/2 Ohm small test load. Note that the TFW Power Supply in the MSU test rack was also doing some funny "pulsing" stuff at its turn on when I first started working on the scaler VME FPGA stuff. The problem with it ended up just being a dirty corroded (from no use) DC enable switch. Only a subset of its outputs were pulsing and I think that they were doing it at both turn ON and at turn OFF. After the cooling water was restored Thursday morning it was clear that L1 Cal Trig was in trouble. I pushed the button at the top of the SCLD card to get the Yellow LED on its SCL Receiver to come back ON. Then I did an Initialize and that made things look OK for the link status in the once per 5 second monitor window. The L1 Cal Trig and all the TCCs were left running during the cooling water problem. Note that when I first tried to use the L1 Cal Trig TCC that one of the windows was "frozen". I forget which one - but it was not the gui command window. The "frozen" window turned into just a frame. I "ionized" all the windows and then brought them back and then things appeared to run normally. The visible fans are turning. M124 back top temperature moves around in the range 94.9 to 95.3 They are now running on just pond water heat exchanger cooling. "Crash Kit" tool box was open when I got here. ------------------------------------------------------------------------------ DATE: 11,12-Nov-2010 At: Fermi TOPICS: MVME Board and Books to Fermi, PAB Meeting, Collaboration Meeting I brought a MVME133-A20 and a complete set of its books down to Fermi and gave them to Petr. He is still working on the Omnibyte card problem. I talked with Stu and he is passing the information to Mike Matulik. Petr kept all the HW manuals but I'm taking the Bug book back to MSU. PAB Meeting - the new goal is to have all the hardware ready by 1-Jan-2011. Collaboration meeting - the next big indication of the run extension will be if the 35M/yr is in the president's budget in February. If so then the lab says that they will commit to it. Talked with George and Stefan about VME Interface firmware and being ready to test. They asked not to let this slip for too long. The visible fans are turning. M124 back top temperature moves around in the range 91.7 to 92.6 Tool Box was open and the clip board has been taken from the desk. ------------------------------------------------------------------------------ DATE: 28,29-SEPT-2010 At: Fermi TOPICS: Ops meeting, High rate tests for VRBC buffer loss, VME IF ReBuild work, Spare blower mounting bolts Between stores tried running both the main DAQ system and triggers to the SMT test stand at a high rate to look for a VRBC "buffer loss" problem. With the STT crates and some Muon crates removed from the run, things were pretty stable at a L1 rate of 3.1 or 3.2 kHz and a rate into L3 of about 1/2 of that 1.6 kHz. The VRBC buffer loss problem is kind of like it never processed a L2_Decision message. If the VRBC dropped a L2_Reject then nothing would happen in the rest of the DAQ system - rather that VRBC would just have one less buffer to use. If it drops an L2_Acpt then a broken event will go to L3 - and L3 does have some back ground rate of broken events. ---------------------------------------------- DZero Level 1 Trigger Trigmon Display Snapshot ---------------------------------------------- Level 1 Trigger Framework Specific Triggers 28-Oct-2010 15:39:34 Integrat Period L1/L2FW = 61.5 s / 61.5 s L1 Accept = 3106.68 Hz/ 29745839 Operational:Yes Current:Yes Triggered:Yes L1 FW Paused= 0.0 % /NowRunning L2 Bypassed:No Outstanding L1 Accept: 0 L2 Accept = 1566.71 Hz/ 18469238 Last FPGA Configure = 14-Oct-2010 11:06 L2 Accept/Reject= 50.4 % / 49.6 % Last FW Initialize = 28-Oct-2010 05:23 L2 FW Stalled by L2 Busy = 0.3 % Last SCL Initialize = 14 mn 16 s Luminosity Block Num = 0x 0075 3e33 Last LBN Increment = 15 s Tick / Turn = 72/ 1777878765 Allocated SpTrg: 8 ExpGrp: 2 GeoSect: 71 Spec| L1 | L2 Accept |And-Or|Prescl|Total|ExpGp|ExpGp| L3 | COOR|Ex Trig|Accept| | Fired| Ratio|Expos| Live|L1 Bz|Disab|Disab|Gp ---#|----Hz|----Hz|----%|-------|----Hz|------|----%|----%|----%|----%|----%|-# 0| 8.44| 8.43|100 | 167934|1.702M|191000| 0.0| 0 | 9.0| 0 | 0 | 0 1|779.10| 0.00| 0 | 67565|1.702M| 2000| 0.0| 0 | 9.0| 0 | 0 | 0 2|785.43|785.74|100 |7079182|1.702M| 2000| 0.0| 0 | 9.0| 0 | 0 | 0 3| 0.05| 0.05|100 | 1335|1.702M|17.20M| 0.0| 0 | 9.0| 0 | 0 | 0 4| 8.73| 8.71|100 | 53971|1.702M|191000| 0.0| 0 | 10.9| 0 | 0 | 1 5|762.90| 0.00| 0 | 0|1.702M| 2000| 0.0| 0 | 10.9| 0 | 0 | 1 6|764.40|764.66|100 | 765097|1.702M| 2000| 0.0| 0 | 10.9| 0 | 0 | 1 7| 0.05| 0.05|100 | 126|1.702M|17.20M| 0.0| 0 | 10.9| 0 | 0 | 1 ------------------------------------------------------------------------------- Level 1 Trigger Framework Geographic Sections 28-Oct-2010 15:39:34 Integrat Period L1/L2FW = 61.5 s / 61.5 s L1 Accept = 3106.68 Hz/ 29745839 Operational:Yes Current:Yes Triggered:Yes L1 FW Paused= 0.0 % /NowRunning L2 Bypassed:No Outstanding L1 Accept: 0 L2 Accept = 1566.71 Hz/ 18469238 Last FPGA Configure = 14-Oct-2010 11:06 L2 Accept/Reject= 50.4 % / 49.6 % Last FW Initialize = 28-Oct-2010 05:23 L2 FW Stalled by L2 Busy = 0.3 % Last SCL Initialize = 14 mn 16 s Luminosity Block Num = 0x 0075 3e33 Last LBN Increment = 15 s Tick / Turn = 72/ 1777878765 Allocated SpTrg: 8 ExpGrp: 2 GeoSect: 71 Geogr| L1 | L2 Busy | L1 GS Accept | L2 GS Accept |L1|L2|SCL Sect| Busy| Raw |Delay|Cycl| |L1Acc| |L2Acc|Er|Er|Sta --#|0x|----%|----%|----%|---%|----Hz|------|----%|----Hz|------|----%|--|--|--- 0|00| 3.3| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.32|16.26M| 50.7| | | Ok 1|01| 3.0| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.32|16.26M| 50.7| | | Ok 2|02| 2.8| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.32|16.26M| 50.7| | | Ok 3|03| 0 | 0 | 0 |0.00|3106.6|27.75M|100 |794.32|16.66M| 50.7| | | Ok 4|04| 3.2| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.32|16.26M| 50.7| | | Ok 5|05| 0 | 0 | 0 |0.00|3106.6|27.75M|100 |794.32|16.66M| 50.7| | | Ok 6|06| 3.1| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.32|16.26M| 50.7| | | Ok 7|07| 3.2| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.32|16.26M| 50.7| | | Ok 10|0a| 0 | 0 | 0 |0.00|3106.6|29.67M|100 |1567.3|18.41M|100.0| | | Ok 11|0b| 0 | 0 | 0 |0.00|3106.6|29.67M|100 |1567.3|18.41M|100.0| | | Ok 16|10| 2.0| 0 | 0 |0.00|1572.7|28.13M| 50.6|794.34|17.63M| 50.7| | | Ok 17|11| 0 | 0 | 0 |0.00|1572.7|28.13M| 50.6|794.34|17.63M| 50.7| | | Ok 19|13| 4.0| 0 | 0 |0.00|3106.6|27.74M|100 |794.34|16.66M| 50.7| | | Ok 20|14| 0 | 0.0| 0.0|0.00|3106.6|29.66M|100 |794.34|17.63M| 50.7| | | Ok 26|1a| 0 | 0 | 0 |0.00|3106.6|29.66M|100 |794.34|17.63M| 50.7| | |Bad 27|1b| 0 | 0 | 0 |0.00|3106.6|29.66M|100 |794.34|17.63M| 50.7| | |Bad 31|1f| 4.0| 0.2| 0.2|0.00|3106.6|29.74M|100 |1567.3|18.46M|100.0| | | Ok 32|20| 0 | 0 | 0 |0.00|3106.6|29.74M|100 |1567.3|18.46M|100.0| | | Ok 33|21| 0.6| 0 | 0 |0.00|1572.7|28.13M| 50.6|794.34|17.63M| 50.7| | | Ok 34|22| 0.2| 0 | 0 |0.00|1572.7|28.13M| 50.6|794.34|17.63M| 50.7| | | Ok 35|23| 0 | 0 | 0 |0.00|1572.7|28.13M| 50.6|794.34|17.63M| 50.7| | | Ok 36|24| 0 | 0 | 0 |0.00|3106.6|29.66M|100 |794.34|17.63M| 50.7| | | Ok 37|25| 0 | 0 | 0 |0.00|3106.6|29.66M|100 |794.35|17.63M| 50.7| | | Ok 48|30| 0 | 0 | 0 |0.00|1572.7|28.13M| 50.6|794.35|17.63M| 50.7| | | Ok 49|31| 0 | 0 | 0 |0.00|1572.7|28.13M| 50.6|794.35|17.63M| 50.7| | | Ok 50|32| 0 | 0 | 0 |0.00|1572.7|28.13M| 50.6|794.35|17.63M| 50.7| | | Ok 51|33| 0 | 0 | 0 |0.00|1572.7|28.13M| 50.6|794.35|17.63M| 50.7| | |Bad 52|34| 0 | 0 | 0 |0.00|1572.7|28.13M| 50.6|794.35|17.63M| 50.7| | | Ok 53|35| 0 | 0.0| 0.0|0.00|1572.7|28.13M| 50.6|794.34|17.63M| 50.7| | |Bad 54|36| 0 | 0 | 0 |0.00|1572.7|28.13M| 50.6|794.34|17.63M| 50.7| | | Ok 55|37| 0 | 0 | 0 |0.00|1572.7|28.13M| 50.6|794.34|17.63M| 50.7| | | Ok 56|38| 0 | 0 | 0 |0.00|1572.7|28.13M| 50.6|794.34|17.63M| 50.7| | | Ok 57|39| 0 | 0.0| 0.0|0.00|1572.7|28.13M| 50.6|794.34|17.63M| 50.7| | | Ok 58|3a| 0 | 0 | 0 |0.00|1572.7|28.13M| 50.6|794.35|17.63M| 50.7| | | Ok 59|3b| 0 | 0 | 0 |0.00|1572.7|28.13M| 50.6|794.35|17.63M| 50.7| | | Ok 64|40| 0.1| 0.1| 0.1|0.00|1572.7|25.29M| 50.6|794.35|16.22M| 50.7| | | Ok 65|41| 0.1| 0.1| 0.1|0.00|1572.7|25.29M| 50.6|794.35|16.22M| 50.7| | | Ok 66|42| 0.1| 0.1| 0.1|0.00|1572.7|25.29M| 50.6|794.35|16.22M| 50.7| | | Ok 67|43| 0.1| 0.1| 0.1|0.00|1572.7|25.29M| 50.6|794.35|16.22M| 50.7| | | Ok 68|44| 0.1| 0.1| 0.1|0.00|1572.7|25.29M| 50.6|794.35|16.22M| 50.7| | | Ok 69|45| 0.1| 0.1| 0.1|0.00|1572.7|25.29M| 50.6|794.35|16.22M| 50.7| | | Ok 70|46| 0.1| 0.1| 0.1|0.00|1572.7|25.29M| 50.6|794.35|16.22M| 50.7| | | Ok 71|47| 0.1| 0.1| 0.1|0.00|1572.7|25.29M| 50.6|794.35|16.22M| 50.7| | | Ok 72|48| 0.1| 0.1| 0.1|0.00|1572.7|25.29M| 50.6|794.35|16.22M| 50.7| | | Ok 73|49| 0.1| 0.1| 0.1|0.00|1572.7|25.29M| 50.6|794.35|16.22M| 50.7| | | Ok 74|4a| 0.1| 0.1| 0.1|0.00|1572.7|25.29M| 50.6|794.35|16.22M| 50.7| | | Ok 75|4b| 0.1| 0.1| 0.1|0.00|1572.7|25.29M| 50.6|794.35|16.22M| 50.7| | | Ok 76|4c| 0.1| 0.2| 0.1|0.00|1572.7|25.29M| 50.6|794.35|16.22M| 50.7| | | Ok 80|50| 5.6| 0 | 0 |0.00|3106.6|27.75M|100 |794.35|16.66M| 50.7| | | Ok 81|51| 5.6| 0.0| 0.0|0.00|3106.6|27.75M|100 |794.35|16.66M| 50.7| | | Ok 82|52| 5.6| 0 | 0 |0.00|3106.6|27.75M|100 |794.35|16.66M| 50.7| | | Ok 83|53| 5.5| 0 | 0 |0.00|3106.6|27.75M|100 |794.35|16.66M| 50.7| | | Ok 96|60| 3.6| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.34|16.26M| 50.7| | | Ok 97|61| 3.8| 0.0| 0.0|0.00|1572.7|25.45M| 50.6|794.34|16.26M| 50.7| | | Ok 98|62| 3.5| 0.0| 0.0|0.00|1572.7|25.45M| 50.6|794.34|16.26M| 50.7| | | Ok 99|63| 4.0| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.34|16.26M| 50.7| | | Ok 100|64| 3.8| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.34|16.26M| 50.7| | | Ok 101|65| 3.9| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.34|16.26M| 50.7| | | Ok 102|66| 4.0| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.34|16.26M| 50.7| | | Ok 103|67| 4.0| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.34|16.26M| 50.7| | | Ok 104|68| 3.6| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.34|16.26M| 50.7| | | Ok 105|69| 3.6| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.34|16.26M| 50.7| | | Ok 106|6a| 3.5| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.32|16.26M| 50.7| | | Ok 107|6b| 3.8| 0 | 0 |0.00|1572.7|25.45M| 50.6|794.34|16.26M| 50.7| | | Ok 111|6f| 0 | 0 | 0 |0.00|1572.7|25.45M| 50.6|794.35|16.26M| 50.7| | | Ok 118|76| 0 | 0 | 0 |0.00|3106.6|29.67M|100 |1567.3|18.41M|100.0| | |Bad 120|78| 0 | 0 | 0 |0.00|1535.6|1.555M| 49.4|773.29|790463| 49.3| | | Ok 121|79| 7.7| 0 | 0 |0.00|1535.6|1.555M| 49.4|773.29|790463| 49.3| | | Ok 124|7c| 0 | 0 | 0 |0.00|1572.7|28.13M| 50.6|794.39|17.63M| 50.7| | | Ok 125|7d| 0 | 0 | 0 |0.00|1572.7|28.13M| 50.6|794.39|17.63M| 50.7| | | Ok 127|7f| 0 | 0 | 0 |0.00|3106.6|29.74M|100 |1567.3|18.46M|100.0| | | Ok ------------------------------------------------------------------------------- Test online of the VME Interface ReBuild is on hold while the Luminosity folks figure out which of their quantities is measured by which set of PBS Scaler cards 0:9. Continued with design work to implement stopping the JTAG clock after the card is up and running. I installed the mounting nuts in one of the spare blowers so that it is ready to install now if it is needed. The visible fans are turning. M124 back top temperature moves around in the range 94.5 to 95.2 the cooling water is moving around in the range 55.6 to 56.5 There is another round of compressor work starting next week. ------------------------------------------------------------------------------ DATE: 14-Oct-2010 At: MSU TOPICS:VESDA trips TFW A VESDA alarm (triggered by the Fire Tech) tripped the TFW Configure FPGAs TFW : 1531 downloaded, 0 error Read M122 Top Slot#21 Reg#2 0x006c VME Error bit NOT set (and MA FPGAs output not enabled) Configure Routing Master 51 downloaded, 1 error Configure Routing Master 51 downloaded, 0 error Intialize TFW Read M122 Top Slot#21 Reg#2 0x0027c VME Error bit is set ------------------------------------------------------------------------------ DATE: 7,8-OCT-2010 At: Fermi TOPICS: Check VME Reg #2 contents, Ops meeting Work on VME Interface ReBuild, TRICS Console with 2 TRICSs Between stores on Thursday I checked the contents of the #2 register in the VME Interface chip on a number of cards in a number of crates. Do reads on: Rack Crate Slot Chip Reg Contents ---- ----- ---- ---- --- -------- m123 Bot 21 0 2 $007C m123 Bot 9 0 2 $007C m123 Mid 5 0 2 $007C m123 Top 2 0 2 $007C m122 Top 5 0 2 $007C m122 Top 18 0 2 $007C m122 Top 20 0 2 $007C m122 Top 21 0 2 $027C <-- VME Error Bit These values match what was expected and specifically that we keep: Bit #5 "JTAG Scan Path Controller Enable" set to 1 ---> JTAG Controller outputs enabled Bit #6 "JTAG Scan Path Controller Test Logic Activate" set to 1 ---> JTAG Scan Path active i.e. the bullets are in and the safety if off (but it works). Wednesday night for a special run during a store the Lumi folks wanted to change from 15 sec to 5 sec monitoring data. My best guess is that the following happened: while trying to put the mouse focus on the TRICS window they clicked on the master command file button, they did not see the TRICS window with the master command file (because it is smaller ?), they thought that they had minimized the TRICS window, so they double clicked on the TRICS icon on the desk top (not in the task bar), thus they started a second instance of TRICS. They were very nice and rational and tried to page me to let me know that there had been a problem - but I was on I-94. Thursday morning Philippe killed the second instance of TRICS and did what ever clean up was necessary. The rest of the store during which there were 2 instances of TRICS had no useful Physics because the Calorimeter system was dead. Thursday I got to visit with the Lumi folks. Sent a note to remind people about TRICS console etiquette and to remind the RCs that if this happens during a good store that then need to get ahold of us. M124 top back of SCL Hub-End backplane is 91.8 95.1 deg F. I know that various chillers are still being turned on and off. All externally visible fans are running. ------------------------------------------------------------------------------ DATE: 22:24-SEPT-2010 At: Fermi TOPICS: M122 VME Errors, TFW RMI Drip Alarm, Tick Selector #3, MCH Check Ever since the Master Clock power supply failure on Sept 13th we have had a significantly increased number of monitor data read VME hangs in both middle and top crates in M122, the Per Bunch Scalers for the L1 Triggers and for the Luminosity System. We have not seen any hangs in M122 Mid for about 1 year and M122 Top has been at the rate of about 1 or 2 a month. Since the Master Clock problem we are seeing 1 and sometimes 2 hangs a day. As a first step to investigate this we cold started M122 Mid and Top early Wednesday afternoon. DC power to the crates was off for > 1 minute to make certain that the VME Interface FPGAs would all re-Configure from their PROMs. Then TCC configure M122 Top. This was 340 configured with 0 errors. While the card in slot 18 (SM-7) was being configured I also saw the VME LED on the card in slot 21 (SM-5) flash a few times. The configured M122 Mid which gave 272 configured with 0 errors. The system looked fine at Initialize and was given back to the DAQ Shifter by about 14:00 Fermi time. I also checked the cables and paddle cards in the back of the Master Clock rack to see if anything had been bumped during the work in it. Everything looks fine. Friday at noon Philippe pulled the logs. The problem still exists. We still have 1 or 2 VME read errors per day between M122 Mid and Top. Fritz is working on the SES Alarm for the TFW drip detector. It ends up that for the type of RMI (which he calls "cook" and I would call "Hance") that the cables need to be: RMI Status goes to RM P7 which is an RM input thus switch up RMI Control comes from RM P8 which is an RM output thus switch down Fritz currently has this as a CTL (control system alarm). I need to verify the the Guidance text got in OK. Request from Andreas Jung to move the default value for Tick Selector #3 so that it will issue L1_Acpts on tick #78. Thursday evening after the store I edited ...\D0_Config\Init_Post_Auxi_L1FW.rio to make this change. I got this done before the Initialize for the next Store. M124 top back of SCL Hub-End backplane is 91.8 93.6 deg F. I know that various chillers are still being turned on and off. All externally visible fans are running. M123 back door was open. Back door on M101 was found half open. 1075 1993 ------------------------------------------------------------------------------ DATE: 13-SEPT-2010 At: MSU all action at Fermi TOPICS: Master Clock Power Supply Failure Monday morning we noticed that the web TrgMon display suddenly started showing blank fields. An initial concern was that the TFW had a power trip. We called the control room and the problem was the Master Clock. It's -5.2 Volt power supply was dead. Steve Chappa got the supply replaced and Bill down loaded the clock. We then basically cold started the TFW. We saw the expected VME I/O errors at places that the TCC had tried to talk to when there was no 53 MHz clock. At one point we configured just M122. Doing just M122 shows 846 configured 0 errors. It was lucky that all of this happened during a machine studies period and that the DAQ system was running again by the next store. ------------------------------------------------------------------------------ DATE: 8:10-SEPT-2010 At: Fermi TOPICS: Operations Meeting, Assumed False Drip Trips of the TFW, Parts Delivered, Forward Muon Trigger, MCH-1 Check Between stores Wednesday evening I replaced the RMI in rack M124 that has provided both Drip and Smoke protection for the TFW. On the RMI that I installed I enabled only its Smoke sensor input. Thursday I plugged the TFW Drip sensor cable into another RMI that I on top of M122. This "new" RMI has only its DRIP sensor input enabled. This new RMI is connected to the RM with terminal address 26. The Status from this new RMI is connected to P8 on the RM and the Control to the RMI is from P7 of the RM. The switches on the front of the RM are set so that P7 is an output (switch down) and P8 is an input (switch up). The 8 bit "Geographical Address" switches on the RMI are currently set to 00 (but could easily be changed to something more unique to help confirm that status from the correct RMI is being looked at. Fritz is working to setup an SES Alarm from this new RMI (there currently are no TFW RMI Tripped SES Alarms). If this "new" RMI sees a Drip then we will cause an SES Alarm but it will not actually trip off power to the TFW. I have written a guidance text file for this new alarm and sent it to Fritz. The plan is to figure out what has caused the two recent apparently false Drip trips of the TFW, without causing any additional trips, and will still providing protection. Once everything is working correctly then we will go back to using the normal M124 RMI to provide both Drip and Smoke protection for the TFW and connect the M124 RMI so that it can cause the TFW RMI Tripped SES Alarm. After the work Wednesday evening I opened up the RMI that I pulled out of M124. It has a burned up transformer but I think that this may be the smoke detector power supply transformer. At the last couple of power ups (after scheduled power outages) I have not had to reset the RMI to clean a smoke trip. It *could* be that when the smoke detector 20V power supply dies that it quits providing protection and does not fail safe trip the system. I do not know and could not tell from the schematics. When I turned on the newly installed M124 RMI Wednesday not I did have to reset (as normal in the past) for a smoke trip. So far Fritz has not come back down to test the TFW SES Trip Alarm. Dean gave me an output listing from the thing that watches for calorimeter system monitor channels that appear to be stuck. Using the Channel Archiver and my poor instructions from the 23,25-JUNE-2010 log book entry I looked at the L1 Cal Trigger (at least at the 4 ADF-2 crates). U0 M104 moving around +- 30 mV from 5.00 Digital +5 M106 sits at 5.00 M109 sits at 5.01 M111 sits at 5.00 U1 M104 sits at 5.00 Analog 5 M106 moving around 4.98 and 4.99 M109 sits at 5.01 M111 moving around 4.98 and 4.99 U3 M104 moving around 3.29 and 3.30 Digital 3.3 M106 sits at 3.29 M109 sits at 3.27 M111 sits at 3.30 U3 M104 sits at 4.99 Analog 5 M106 sits at 4.99 M109 sits at 5.00 M111 sits at 5.02 The interesting question is is the +- 30 mV channel sick or is are the channels that do not move even +- 10 mV the sick ones ? I tried to make links to make it easy to see the one day channel archiver output. To help muon folks dig into a forward muon phi distribution problem I looked at some trigger configuration files and such. I record the path names here just to remind myself where stuff is. /online/data/coor/configurations/official/special/muon_L1L2L3-5.00.xml /online/data/coor/resources/coor_resources.xml Confirmed that there is only one L1 Muon Trigger Term for the L1 Trigger that has a funny forward muon phi distribution. Dean S on site night number x5635 We still have not seen the expected oscillatory odd behavior that we have seen before during Cal Phi Calibration Triggering. Dean reminds me that the period was about 25 to 30 minutes and it was mostly in the lowest energy calibration trigger. I can not imagine that this is the first time that we have done a phi calibration since the L1 Cal Trig Active Pedestal Control has been turned on. Delivered the new GE motor run capacitors for the blowers to Fermi and the quality 1000 uFd 25V and 50V capacitors for replacing parts in the safety control system. I changed out the run cap on one of the spare blowers. I need the 10-24 hardware for 2 spare blowers down here on the next trip. M124 top back of SCL Hub-End backplane is 92.2 93.3 deg F. There has been another round of chiller problems and I know that things are not stable at full cooling yet. All externally visible fans are running. 1791 2227 3082 3257 ------------------------------------------------------------------------------ DATE: 26,27-AUG-2010 At: Fermi TOPICS: SCL FanOut, Operations Meeting, Blowers from IMS, MCH-1 Check When I got here the gray cabinet next to the desk was open and our tool box was open. Visited Ted over in Feynman. The bulk of Feynman 3rd floor is being turned into more computer farm space, in this case with all glass walls around it so that people can look in and be impressed. The lab area where the VRB and SCL test stands were is all gone. Ted has the SCL test stand running (which requires a W95 machine to run) and gave me a tested known good spare SCL FanOut for the one from out D-Zero cabinet that has a yellow dot sticker on it. Our new spare SCL FanOut is SN# 007 B. I got the 3 blowers back from IMS with new bearings installed. These are the 3 that were replaced during the shutdown. The tag is ripped off one of them (it was sucked into it during testing) and the motor run capacitor is smashed in on it (but it still works). I need to replace this capacitor. It is: GE 7.5 uFd 370 VAC +- 6% 50/60 Hz "Protected" P921 97F9001 A10000AFC 6243GA28 0452-06 598 Operations meeting, the Cal phi calibration triggers will be added starting now to the luminosity < 150 end of each store for the next N weeks. This means that we will start to see something periodic and funny in some rate or busy or something in the monitoring but I forget what. M124 top back of SCL Hub-End backplane is 91.3 92.6 deg F. All externally visible fans are running. 1830 ------------------------------------------------------------------------------ DATE: 11:12-AUG-2010 At: Fermi TOPICS: Shutdown work, 0x51 SCL Hub-End Problems, Master Clock Frequency, L2 Computers Since restarting after the power outage last Saturday Aug 7th we have had trouble with crate 0x51. From the 0x51 SBC's log file Gordon figured out that bit #13 was stuck low in the L3 Transfer Number. Jadzia said there there were also Tick/Turn mismatch errors in the data from the crate. Only about 1/2 half of the events readout OK - which should have been a big hint. All that I did to "fix" the 0x 51 problem was replace the small serial transmitter mezzanine card for that channel on the SCL Fanout card. We are still running the same set of 16 SCL Fanout cards. I do not know for certain that the serial transmitter card was the problem - it could be that wiggling something while replacing the transmitter mezzanine "fixed" the problem. I have been in contact with Ted Zmuda, Neal Wilcer, and Thinh Pham over at 3rd floor Feynman about this problem. As usual they are very cooperative and ready to help. They are restarting the SCL system test stand in their lab. Specifically they will verify that the one spare SCL Fanout card that we have at D-Zero is working OK. Our spare has a Yellow dot sticker on its front panel and it should in fact have a Green dot sticker like the 16 SCL Fanout cards in the running system. Red = does not work in the test stand, yellow = rare failure in the test stand, green = always OK in the test stand. They are also digging through their records to see if they know what was wrong with the SCL Fanout card that failed in the same way (a bit stuck low in all fields of the SCL frame) back in November 2000. That was a different SCL Fanout card, a different channel on the card, and a different bit stuck low - but otherwise a similar failure. See our log book for: 29,30-NOV-00 and 7-Nov-00. We switched the Master Clock to "Normal" mode yesterday so it is following the Tevatron RF system. Currently the Tev RF frequency is up at the value for 980 GeV. I do not know whether they will "ramp down" or just glitch us (boot Low Level RF or something) when they get ready to run. A couple of months ago Mike from L2 brought down some 11 year old computers for me to pull out Bit-3 stuff from them and then he was going to recycle/junk them. I pulled the Bit-3 stuff, see the log book for 29,30-APR-2010, but he never took them away. Today I talked with Pete Simon and he will take care of them. This is 4 computers, 1 monitor, and 1 keyboard. I moved the stuff to a pallet in the high bay. Some numbers from the 4 computers are: node name: d0ntmsu7 Micron SN: 1307147-0001 Fermi Property Sticker Number: S54382 node name: d0ntmsu8 Dell Dimension XPS-T600 Number: 0338D-02952-99G-3080 node name: d0ntmsu1 Micron SN: 797561-0003 node name: ? Digilink SN: 21351 M124 top back of SCL Hub-End backplane is 91.7 92.7 deg F. Work on the chillers is still going on. All externally visible fans are running. ------------------------------------------------------------------------------ DATE: 2:4-AUG-2010 At: Fermi TOPICS: Shutdown work: Edit L1TCC Config File, Replace Power Chassis Fans, Check Drip Strips, Replace Shea Box at Address 26, 5V Supplies in the Contactor Box and Vesda Permit Box, Print Sets to Fermi, Test Spare TFW Power Supplies, No Scaler Module Swap I removed the upper M101 TFW power supply chassis, the L1 Cal Trig Readout Crate supply, and replaced the 6 internal fans in its two ASTEC VS3 bricks. These Astec bricks are: SN# 99420447 (+3.3V) and SN# 99420438 (+5V, -2V, -4.5V). I labeled the backs of these bricks saying fans replaced in AUG-10. I used standard AMP contacts and housings for the fan connections but I had to solder them - our crimp tool is about worn out. This work goes OK but it is dirty and takes a long time. One could replace fans in about two TFW power supply chassis per day if they had been pulled out and were ready to work on. The very stupid thing that I did was to not replace the "external" 120V fans. For the work involved in pulling these things apart we clearly should replace all the fans - not just the "internal" DC fans. Things to remember when working on the Astec VS3 bricks: the power factor correction module comes off sidewise, on the lower current 60 Amp modules (where you can get at it) it is worth while tightening the two sets of screws on the filter choke connections (they all seem to be a little loose). The internal fans that were installed are: Description: FAN 119mm x 38mm 24VDC EBM-Papst Part Number: 4214H Digi-Key Part Number: 381-1085-ND Price: $46.30 each Description: FAN 60mm x 25mm 24VDC 23.5 CFM EBM-Papst Part Number: 624N Digi-Key Part Number: 381-2272-ND Price: $29.61 each Because of the un-explained water drip detector power trip of the TFW on Friday July 23rd at about 11:15 AM I check the drip detector strips in the TFW and M101. All the TFW strips except for the East strip in M124 are covered with paper and look OK with cables fully plugged in. The East one in M124 was not wrapped so I wrapped it. The ones in M101 are bolted to the sides of the rack, not wrapped, and very hard to clean because you can not get at them. I did what I could and things look in OK condition. While working on drip strips I also cleaned the ones in ADF-2 Racks "C" and "D" M109 and M111. One of these has had a couple of false trips at one time. There are strips: at the radiator water connections, on each side of the top cable shelf, and on the floor at the back of the rack. I especially think that this back one can get junk on it and trip. None of these are wrapped. Because of the 4 failures of our home made low current supplies in the past year at MSU, I checked the Contactor Box and Vesda Permit Box power supplies. The Contactor Box has one of our supplies but it is setup as an un-regulated 12V supply with a big Sprague capacitor (not the kind that have been failing at MSU) so I did not change it. The Vesda permit box has a commercial 12V low current regulated supply with about 19V going into the regulator. I did not have an optimum cap with me for replacing that one, and that one appears to have good hold up time so I left it alone. The big think I learned is that the Vesda permit box (necessary for the old L1 Cal Trig safety system could be dumped and just send the Vesda contact closure to the Contactor Box control circuit. That would make a better system and should be done (as well as setting up M101 rationally). For sometime we have had the problem of the M122 -2V monitoring readout being right at or even at times under the -2.2V "Low" Alarm Threshold. Looking at this monitor point voltage with the Fluke and it looks just fine and is noise free whereas the RMI monitor readout jumps around in a range of about a 100 mV window and is offset negative by about 100 mV. This is RM Shea Box terminal address 26 analog input channel 18. Mike Matulik provided a replacement box to swap in back in February. Because this was not an emergency, making the swap has been delayed until there was a safe time to work on top of the rack. The replacement box was swapped in on Tuesday and the new one fixed about 1/2 of the problem. I gave up and put a 1.1 k Ohm resistor resistor from this monitor point to ground to pull this monitor readback into the correct middle of its range. I still do not know why these readbacks are so noisy - the Fluke shows them being nice and clean. Note that for a while we ran with "special" alarm thresholds form M122 -2V. See 12-FEB-2010 log book. Test run the spare TFW power chassis stored at D-Zero: SN # 4: +5.068 V +3.335 V -2.130 V -4.636 V SN # 11: +5.047 V +3.351 V -2.112 V -4.610 V SN # 1: +5.066 V +3.333 V -2.131 V -4.634 V SN #11 is the supply that died a year ago during shutdown while running the Routing Master. It has its original Astec VS3 bricks back in it: +3.3V brick rebuilt by Astec, +5V -2V -4.5V brick with MSU installed new internal fans. SN #1 is the some what proto-type from the MSU Test Stand. The potential correlation between how long the red LED in the PFC module flashes at power up and whether or not the PFC is on it last legs remains a mystery. I will leave the LED on a stick power supply fan inspection light and its supply here at Fermi. I have not swapped Scaler Modules in the M122 Top crate as was planned for the shutdown. This still needs to be strongly considered. The Scaler Modules stored in the spares cabinet are: GS-48 (tagged as having run in M122 Middle Slot 13), GS-50, GS-54, GS-55, SM-1. On the L1TCC that we have now been running for the past 2 weeks I edited \D0_Config\Configure_FPGAs.dcf to comment out the call to M101_L1CT_All.dcf I do not know, and need to check, if this is the way that Philippe turned off the call to configure the old L1 Cal Trig M101 readout system in the other "new" L1TCC. This worked fine during the restart after the power outage Tuesday morning, I need to bring here copies of the print sets for all the power and "special" monitor stuff, e.g. rack power control, contactor box, M101 fan monitor, Vesda permit box, If we continue to run then I need a plan to clean up the Vesda permit and to "normalize" the power control of M101 The front door of M101 was completely open when I arrived. M124 top back of SCL Hub-End backplane is 91.7 93.5 deg F. All externally visible fans are all running. 1123 1213 1993 2187 3065 3082 ------------------------------------------------------------------------------ DATE: 28-JULY-2010 At: MSU action at Fermi TOPICS: Un-Scheduled Power Outage Wednesday July 28 there was an unexpected power outage (apparently related to a feeder fault in the proton area). The power was off from 2:45 to 4:15 Fermi time. George and Bill got power, computers, and Clock on and paged me at about 8:45 Eastern time. The TFW Configuration and Initialization were smooth. - There was still the message on L1TCC, "Add New Hardware You do not have sufficient security priveledge to install devices on this computer. .... I just clicked "OK" and as usual TRICS had no problem using the Bit-3 VME stuff. - The SBC in the Routing Master was not listening to any commands, e.g. STOP. I *think* this is because L3 was not turned ON yet, specifically L3 Supervisor was not ON. George and I just did the: RM DC off, wait 10 sec, RM DC on, Configure RM (before the RM's SBC boots and gets on the bus). Later once L3 was ON then the RM SBC was following commands. - Marvin had trouble with the SCL Receiver in the SMT Heartbeat crate getting it to lock onto the SCL data stream. ------------------------------------------------------------------------------ DATE: 23,24-JULY-2010 At: MSU action @ Fermi TOPICS: Un-Scheduled Power Trips Friday July 23rd at about 11:15 AM Eastern time the TFW racks tripped off. The trip was caused by the RMI in rack M124 detecting a water leak drip. All 4 racks were examined looking for water. Everything looked dry. Dean was there and he reset the RMI and turned on the power. Starting up the TFW went without any problems. The only issue with our equipment was getting the L1 Cal Trig running correctly. We ended up powering it down and then cleanly starting it back up. The assumed problem was getting the serial links all running again correctly after the SCL clock went away. The startup all went ahead without any problems except that the L1 Cal Trig TCC was power cut crashed during the power down process. Saturday July 24 at about 8 AM Eastern time site wide power was lost at Fermi. This may have been caused by a very heavy rain storm in Chicago at the time. Power was on and the computers running and the master clock running by about 1 or 2 PM. George worked at the Fermi end to get our stuff running. TFW startup went OK expect, it took two tries to get a couple of the TFW crates Configured and in the post init auxi vio the TFW TCC could not talk to the SCL Hub-End crate basically at all. The issue ended up being that the outputs were not enabled on the master clock. So inability to do vio to the SCL Hub-End crate is a good clue that the master clock is not running correctly. ------------------------------------------------------------------------------ DATE: 19:21-JULY-2010 At: Fermi TOPICS: Shutdown work: TFW Blower Replacement, Test Spare L1 TCC, Check and Replace Power Supplies, L1 TCC Log Files, MCH-1 Liebert Monday evening at about 18:50 I closed/started a new log file on TFW TCC. The original log file was 20100127, the just now closed file was 20100709, the just now started file is 20100719. The just now closed file was about 41 meg bytes long (in 10 days ?). I copied this file to a temporary location on Force. Note that this 20100709 log file includes all of the beam running before the shutdown started. There has been no beam since early Sunday morning. I verified that I can see this log file OK on Force. Tuesday morning I start a new log file and move the log file 20100719 with just last night's running to MSU. The current spare L1TCC is Del Box #4. I got it out of the cabinet and fired it up. It appears to be running OK but the clock was quite a way off so I set it. I also set the Luminosity Block Number file to $ 0071 Fb00. Power Outage then: Replaced the M122 front and rear blowers and replaced the M123 front blower. All blowers have now been replaced except: the M101 Front blower which we normally do not use and the M124 rear blower which is still an original. M101 rear was replaced about 1 1/2 years ago and M124 front was replaced a long time ago (check this date). Made a survey of all 60 internal power supply fans. I found one dead one. It is in the M101 top power supply on its right hand brick when viewed from the front, it is the upper small fan in that supply. It does not turn at all. As far as I can tell all the other internal fans are running. I have no idea whether or not they are running at the normal speed or if they are making a lot of noise. I will try to obtain replacement fans and repair this supply rather than replace it during the next power outage. Looked at the water hoses and radiators. Hoses look OK. Radiators look a bad as usual. If we have more than one more year then they need to be replaced. Checked the TFW Comm Crate fans and they are OK but getting quite dirty. Checked the 5 fans in the back of M124. They appear to be running OK but I have no idea whether or not they are making noise. Started the TFW up. It configured with 1531/0 and the Routing Master configured with 51/0. The Initialize looked clean. Then I swapped the running L1 TCC with its spare. Specifically: Dell Box #3 has been the Online Running L1TCC since 5-Nov-09. Today I pulled Del Box #3 out of service with the intent of leaving it in the cabinet as the tested known good Spare L1 TCC. Dell Box #4 has been in the cabinet since 18-Nov-09 as the un-tested spare L1 TCC. Today I put Dell Box #4 into service as the Online Running L1 TCC. The intent is to leave Del Box #4 as the Online Running L1 TCC. When Del Box #4 booted up and auto started up TRICS in the trigger account it showed the expected message about not having enough privilege to use the Bit-3 hardware. I stopped TRICS, logged out of trigger and into administrator, and did not see the message. I did nothing in the administrator account. Log out of administrator and back into trigger account and start up TRICS. Things look OK, i.e. I could do a clean Initialize of the TFW - so TRICS from the trigger account could use the Bit-3 OK. I put notes in the log books for Del Boxes #3 and #4. Next I tried a Configure TFW from the Dell Box #4. It run through all of the real hardware OK and then blew up, as expected, trying to configure the no longer existent M101 Mid Crate Cal Trig Spark and Bougie cards. This configure showed 1702/21. I started up TrgMon the the Luminosity Block Number looks fine. With Dell Box #4 the Lum Blk Num started up at about $ 0072 2200 I gave the system to the control room at about 17:45. The first event moved at 22:20. Swap the spare L1 Cal Trig ADF-2 Crate power supply into service. Recall the original setup of what power supply was in what ADF-2 crate. From the 1:3-FEB-2006 log book, this is how it ran for 3 or 4 years: ADF-2 Crate Power Supply MSU Property Crate Serial Number Serial Number Tag Number ----- ------------- ------------- ------------ A 5196039 5196008 017734 B 5196041 5196009 017721 C 5196038 5196007 017760 D 5196040 5196010 017747 Then we did the "Diode Upgrade" which put the power supply that had been for 3 or 4 years the spare, i.e. 5196006 into service in ADF Crate "A" and moved everyone else down a crate. From the 26:29-JULY-2009 log book entry you can see what the setup has been since the summer 2009 shutdown: ADF Crate Wiener Supply SN --------- ---------------- A 5196006 B 5196008 C 5196009 D 5196007 With 5196010 being the spare for the past year. Now, Tuesday night, I pull 5196007 out of ADF Crate "D" and it becomes the spare. I put 5196010 into crate "D". I verified that the supply that I put into crate "D" had the correct CAN-Bus address. After the power outage the Liebert in MCH-1 had lost its set points again and woke up heating the room. Pete Simon reprogrammed its set points. He keeps a list of them inside the front panel. He told me that if power is ON in the building and you want the unit off then there is a way to do that that keeps power to the controller and thus the non-volital memory is not lost. I put the back on the TAB/GAB rack. This week I saw the M124 top back of SCL Hub-End backplane temperature low and hi of 60.1 and 96.7 deg F. Since things have been stabe today it is in the range of 92.4 to 92.5 which is up a little from normal. ------------------------------------------------------------------------------ DATE: 7,9-JULY-2010 At: Fermi TOPICS: CalOp, Operations meetings, Walk-through, Remove preamp box at Lab F, Last Blower for Shutdown and the Power Supply Inspection Light are Ready Removed all the the preamp box down to the signal flange but left the flange bolted up on the feedthrough card for now. Preamps and the External Readout Cables are coming back to MSU. Cable mapping is written on the inside of the top of the preamp transport box. To reinstall: back of inner box, then bottom of outer box, the bottom of inner box, then left side of inner box, then card file, and from there on it is obvious. Installed the mounting bolt nuts in the last of the replacement blowers. This is the NuMi blower and it has had new bearings installed at IMS. The new bearings feel fine but the blower and motor were still full of a lot of dirt and two of the motor through bolts were very loose. --> Need to check the through bolts on all the replacement blowers. The correct wrenches are now down here for the shutdown work. The power supply inspection light is now down here and has been tested on a spare supply (and modified so that it will actually work). When I got here I found the back door is completely off rack M107 the TAB/GAB rack. M124 top back of SCL Hub-End backplane is 91.2 91.3 deg F. M123 front blower bearing noise is about the same. Externally visible fans are all running. ------------------------------------------------------------------------------ DATE: 23,25-JUNE-2010 At: Fermi TOPICS: CalOp, Operatins, and Bo DAQ meetings, Walk-through, Use of Channel Archive, View of L1 Cal Trig power supplies & Wiener Info. You can view the Channel Archive data on the web at: http://www-d0online.fnal.gov/chan_archiver/index_cal_py.htm Click the "START" box. Enter the Pattern "L1CAL" (case sensitive) and clock on "LIST". Now typically what you want to do is select a subset of the displayed list of channels by highlighting sections of the displayed list by dragging the left mouse key over it and the while black background highlighted hit the space-bar or carriage return and these selected channels will be removed from the list. Now typically hit the "Plot" radio button then then click "GET" and it will generate and display a plot of the selected data. Sometimes I had to hit "GET" two times. You can control the time span of the plotted data with the obvious fields in this web page. A useful want to use these plots is to plot the same quantity from each of the 4 ADF-2 crates and see if anything looks funny. Recall that in the ADF-2 crates the U0 supply is the heavy digital +5V, the U1 and U5 supplies are the light analog +-5V, and the U3 supply is the +3.3V for the ADCs. The LSBit on the Wiener Voltage Monitoring appears to be 50 mV and the LSBit on the Wiener Current Monitoring appears to be 50 mA on the low current supplies and about 150 mA on the high current supplies. So far I believe that everything looks rational except that the M104 +5V digital current seems to vary between about 16 and 19 Amps. The other 3 crates show this supply at about 17.5 Amps. The Voltage of the M104 +5V digital appears to flop around by 50 mV. The setup of the L1 Cal Trig Communications Crate is in: 8:9-JULY-2009 The general setup procedure for the Wiener power supplies: 8:10-Oct-2008 Some of the Wiener Crate inventory information: 9-FEB-2007 L1 Cal Trig Wiener power supply setup and crate inventory: 1:3-FEB-2006 L1 Cal Trig Wiener power supply expect current draws: 29-NOV,2-DEC-2005 Wiener crate MSU Inventory Numbers: 13:15-JULY-2005 M124 top back of SCL Hub-End backplane is 91.1 91.4 deg F. M123 front blower bearing noise is about the same. Doors are closed up and the externally visible fans are all running. 1203 1268 1760 1993 ------------------------------------------------------------------------------ DATE: 8,9-JUNE-2010 At: Fermi TOPICS: CalOp meeting, Walk-through, Mini- PDT Teststand Readout Problem Worked with Al on the mini pdt test stand readout problem. Verified that the TFW and SCL stuff was all OK. Most likely the problem is with the replacement readout SBC that was recently installed not being setup correctly (or maybe at all). He will check with L3 folks. Checked with Marvin and George at the Cal Ops meeting. The MCH-1 racks look closed up OK. All externally visible fans are running OK. The M123 front blower noise is about the same as last trip. M124 top back of SCL Hub-End backplane is 91.3 91.5 deg F. 1075 1268 1987 train 3161 3432 truck ------------------------------------------------------------------------------ DATE: 26:28-MAY-2010 At: Fermi TOPICS: Becane_too disk backup, Spare L1 Cal TCC, Online nodes to use, TFW SES Alarm Guidance files, L1 Cal Trig Crate EPICS Record Name/Attributes, Shutdown TFW Blower Schedule, M101C -4.5 Volt Routing Master, TFW Power Supply Fan Inspection Light When I arrived I found the back of M123 completely unlatched - open. Made a backup of the Becane_too disk. Slice 5, i.e. /home was made with dd so all the licenses should be OK. Made a test boot with just this fresh backup disk and it looks OK. This backup will go to MSU. Talked with Bill and he said that the spare L1 Cal Trig TCC is node d0ol148. There is a script on it that will move it to the correct hostname and internet address so that it will appear as the L1 Cal Trig TCC. Bill says that for general work we should use online nodes d0ol140 : d0ol146. Work on the SES Alarm Guidance files for the TFW and the L1 Cal Trig Readout Crate: You can see the existing SES Guidance files at: http://www-d0online.fnal.gov/groups/ses/guidance/ The SES Alarm Guidance files are actually on the online system at: /mnt/online/config/ses/guidance/ The mapping from SES Alarm to what Guidance file is shown is dynamic. It is controlled by the Name of the Alarm and the FileName of the Guidance .txt files. See Geoff's SES Tutorial talk on the web and his presentation to the Operations meeting on 19-MAR-2010 on the web from the agenda server. Alarms follow the official D-Zero naming convention: __/ The Trigger Framework is the detector "TFW" We have one guidance file for the: L1TFW_LV_M122ABC/, L1TFW_LV_M123ABC/, L1TFW_LV_M124ABC/, and L1TFW_LV_M101C/ alarms (power supplies), and a separate guidance file for the L1CAL_LV_M101AB alarms (power supplies), and a separate guidance file for the L1TFW_LV_AIR-TEMP alarm. Yes, it is L1TFW_LV_M124ABC/ in the alarm names and not L1TFW_LV_M124AB/. And yes, the L1 Cal Trig Wiener Crate supplies have their own guidance file. Symbolic links are used to point the 4 main TFW racks all to the same guidance file. All guidance files are (and should be) owned by d0run. I now have a k5 to d0run. Something that I did not do (but Geoff showed me that it can be done) is to put indirect expressions into the guidance file that are translated at guidance file display time. E.G. In the Guidance .txt File Shown in the Display of the Guidance ------------------------- ------------------------------------ Show the attribute $(attr). Show the attribute $(attr) $(det) l1tfw $(devtype) lv $(loc) air-temp $(devname) l1tfw_lv_air-temp You can test what guidance file(s) will be shown for a given alarm by: > setup d0online > ses_guidance (and then type in an Alarm Name) Note that this does not test whether or not an Alarm of the Name actually exists. What it does show is what guidance file(s) will be displayed if an alarm of the name that you type in happens to occur. If you look in: /mnt/online/config/ses/guidance/l1t you will now find: L1TFW_LV_AIR-TEMP.txt guidance file L1TFW_LV_M122ABC.txt symbolic link L1TFW_LV_M124ABC.txt symbolic link L1TFW_LV_M101C.txt symbolic link L1TFW_LV_M123ABC.txt symbolic link l1tfw_power_supply_guidance.txt guidance file The L1 Cal Trig guidance files are in: /mnt/online/config/ses/guidance/l1c including the one for: L1CAL_LV_M101AB. To see the EPICS Record Name/Attribute information about the items in the L1 Cal Trig crates that are being monitored look in: /mnt/online/ioc/ppc/mv2300/d0olctl50/d0olctl50.db d0olctl50.db is really just a .txt type file. To find the EPICS Record Name/Attribute information you need to grep on things like M108 and the look for things like: L1CAL_LVPS_M108/U0 L1CAL_LVPS_M108/I3 L1CAL_LVPS_M108/TEMP L1CAL_LVPS_M108/FAN-1 I dug out the 64 EPICS Record Name/Attribute items from the 6 Wiener Crates in the L1 Cal Trig and mailed them off so that they will now be included in the special Vladimir achieving and trend monitoring of CAL systems. The scheduled date to replace the TFW M122 and M123 blowers is Tuesday July 20. Need to make a power supply fan inspection light and test it next trip. For the past 3 or 4 days we have had no more L1TFW_LV_M101C/-45VV alarms and the scope monitoring this supply has not triggered, i.e. no information yet on whether this is a power supply problem of a monitoring problem. All of our Shea Box analog monitoring channels have more noise than they should. It could be normal mode of common mode noise gets into it. During shutdown I should try installing some 1 uFd Tantalum caps on the Shea Box side of the cabling. M124 top back of SCL Hub-End backplane is 91.1 91.3 deg F. 1203 1977 1993 2348 2586 3257 ------------------------------------------------------------------------------ DATE: 18-MAY-2010 At: MSU action at Fermi TOPICS: L1TFW_LV_M101C/-45VV Alarm We had a couple of bouts of L1TFW_LV_M101C/-45VV coming into alarm both minor and major. Mike Matulik checked the front panel test points and they look OK: On his check of this crate today he measured: 5.03V 3.31V -2.09V -4.61V On 28-AUG-09 (about one month after this supply was installed during the summer 2009 shutdown) I measured: 5.034 3.313 -2.095 -4.611 There are 3 issues: - Is this a power supply problem or a monitoring problem ? Mike is going to monitor the -4.5V test points with a scope. - Why was not the M101C -4.5V major alarm setup to be run pausing ? Wait until the power supply or monitoring is fixed to change this alarms severity. - Why where not the proper guidance file displayed ? Fix the guidance files. Associated and confusing factors: See 14:16 & 26:28-AUG-2009 for the setup of the air-temperature and air-flow alarm in M101 L1TFW_LV_M101/AIR-TEMP See 31-JULY,1-Aug-2008 for the special setup of the L1CAL_LV_M101AB alarms. This file is: /mnt/online/config/ses/guidance/l1c/L1CAL_LV.txt See 12-FEB-2010 for the special thresholds of the TFW_M122/ABC_-2VV alarm because of the about 100 mV offset in its Shea Box. ------------------------------------------------------------------------------ DATE: 13,14-MAY-2010 At: Fermi TOPICS: Collaboration and CalOp meetings, hardware for blowers, Desmo_too disk backup, shutdown power out dates Shutdown schedule power outage dates and switch overs: Tuesday 20 July switch over Monday 2 August switch over Saturday 7 August outage day for DAB distribution maintenance blower hardware and wrench - I must get a special wrench to allow blower replacement - 1/8" allen that fits in a 1 U frame. Give up on getting a disk backup made while I'm here - bring Desmo_too back to MSU to work on it there. The racks looked closed up and the externally visible fans are OK. The M122 front blower motor noise is about the same as last trip. M124 top back of SCL Hub-End backplane is 91.4 91.8 deg F. 1203 1963 1966 1977 3257 ------------------------------------------------------------------------------ DATE: 29,30-APR-2010 At: Fermi TOPICS: Look at the Front-End Busy strip chart features, Monitor jack for the L1_Acpt, Reference for L2 not sending all the L2 Answers, Walk Through, Trigger List Download Example, Disk Backup of Becane_Too, Old L2 MSU Machines, L1 Busy Strip Chart meeting, 3rd blower back from IMS. Philippe put a recent full Physics Trigger List download is on the web at: www.pa.msu.edu/hep/d0/ftp/tcc/coor/coor_to_tfwtcc_15-apr-2010.txt Looking just at andor terms that are used in the Exposure Group definitions: L1FW_Expo_Group 0 And_Or_List 243 255 -224 -247 L1FW_Expo_Group 1 And_Or_List 224 243 255 -247 L1FW_Expo_Group 2 And_Or_List 243 255 -224 -247 Note that Exposure Groups 0 and 2 are programmed identically, and that the only difference for Exposure Group #1 is that it requires Andor Term #224 while the other two veto it. Reminder: these andor terms are 224 AOT's controlled by Dean's Calorimeter Timing and Control card. This involves synchronization with the Calorimeter precision pulser. Cable Label "Calo TC" 243 Live Accelerator BX Marker SCL Helper MSA_Out_49 247 Skip Next N #0 FOM++ MSA_Out_40 255 Permanently Tied HI Andor Term #247 is initialized by TRICS // Program the Skip Next N Crossing Comparator #0 // for 2.6 us, on the FOM++ i.e. ~20 Beam X, or 18 counts The first 91 L1 Triggers (0:90) use Exposure Group 0. L1 Trigger 91 uses Exposure Group 1. The final 19 L1 triggers (92:110) use Exposure Group 2. When reading the trigger list recall the basic layout of the L1 And-Or Terms: 0:31 L1 Cal-Track 32:95 CTT 96:127 L1 Muon 128:143 not used (this had been forward proton) 144:207 L1 Cal Trig 208:223 Level Zero 224:239 Dean's Cal Timing and Control Card 240:255 Generated by the TFW After the Operations meeting there was an almost 2 hour meeting about the features in the L1 Busy strip charts. The two basic features are spikes (that look quantized in amplitude and that are I believe one sample wide, and they always spike up) and stepwise shifts in amount of L1 Busy from some crates. Both effects are order of 1%. Note that a 1% step up in L1 Busy means that something was stopped for about 50 msec. Many ideas developing for how to dig into this. We are not certain what is being plotted as "Global L1 Busy" but it is probably something like Exposure Group #0 L1 Busy. Recall that although the per L1 Specific Trigger Correlated Disable has some some And-Or Terms in it, the Exposure Group L1 Busy does not (it is just the OR of the L1 Busies from the crates in the Exposure Group. Recall also that there is a very complicated correlation among the about 100 G.S. L1 Busies in the Exposure Group. - Do we see these effects during ZB running ? - Do the spikes or shifts in leading L1 Busy match the rate of broken events seen in the L3 Farm ? - Turn off L2 Collect Status - is this state about 50 msec ? - Is there a natural order of which crate shows the most L1 Busy at a given event rate and given event size (luminosity). If so, when the order changes does an SCL Init always restore the same natural order. - Scope of monitor the time structure of L1 Busy Global L1 Busy The information about watching on the L2 monitor the "rate dips" aka "L2 not sending all the L2 Answers" is in the 27:29-MAY-2009 log book entry. Mike Mulhearn brought down from the L2 Test Stand area 3 old no longer ever used computers. They are labeled with paper tags: d0ntmsu1, d0ntmsu7, and d0ntmsu8 131.225.231.64 131.225.231.114 131.225.231.115. Before these are thrown in the junk I pulled out 2 Bit-3 cards. There is a card to go from a computer's pci bus to a Bit-3 pci expansion box. This card says: 200910 85224036 Rev A. There is a card to go from the computer's pci bus via copper to a Bit-3 VME card. This card says: Model 617-201 85221510 Rev U 170031. I put both of these cards in static sacks and put them in the "spare cards cabinet". Note that these may be useful this summer in the tests of the "new spare L1 TCC" and the "new spare L2 TCC". On the side of M101 I installed a bracket and a BNC connector to see the 132 nsec wide NIM pulses with each L1 Accept. This comes from an ECL to NIM converter in the M100 NIM bin right hand side. This monitor jack is labeled. Mike Matulik has the 3rd blower back from IMS. He has made a test run on it. There are now 3 blowers ready to install. I need to bring down to Fermi the mounting hardware for another blower and a cut off Allen wrench. Our racks in MCH-1 look closed up and in good order. The external fans are running OK. - The M123 bearing noise is a little louder and it sounds very dry and like things are flopping around. - It was very clear that someone had been in the gray cabinet next to our desk here. And they had dug around in the red tool box and even left some of the tools on the floor in front of the gray cabinet. M124 top back of SCL Hub-End backplane is 91.7 91.9 deg F. 1866 1203 1230 1268 1294 ------------------------------------------------------------------------------ DATE: 15,16-APR-2010 At: Fermi TOPICS: Shutdown Work, Walk Through Sent note to the run coordinators about the TFW and L1 Cal Trig work during the summer 2010 shutdown. Our racks in MCH-1 look closed up and in good order. The external fans are running OK. M124 top back of SCL Hub-End backplane is 91.5 91.7 deg F. 1075 1993 ------------------------------------------------------------------------------ DATE: 31-MAR:2-APR-2010 At: Fermi TOPICS: 3rd TFW Blower, Shutdown Schedule, Walk Through I brought the blower assembly from Lab F to D-Zero and removed it from the rest of the rack support chassis. The blower itself is tagged and ready to go to IMS for new bearings. I sent a note to Mike and George. This will give us 3 blowers for this summer's replacement work. Someone has walked off with the scope monitor cables for the BLS signal patch panel. I made another set and sent a note to Selcuk, Gabriel, and Joe Haley. On walk through it looks like all visible fans are running OK and the racks are closed up. There is a little more high pitch noise than normal by the TFW. With the hose and ear covers this appears to the the east bearing on the front blower in M123 - which is an original blower. The summer 2010 shutdown schedule is scheduled to start 19 July and last for 4 weeks. I need to bring the stuff to Fermi to make a 2nd backup disk of Becane_Too. It has the licensed and usable Xilinx 3.1 on it which we need to keep. I got a sample of the 3M 25 mil polyolefin cable from Walter. Splitting this cable does not appear to work. M124 top back of SCL Hub-End backplane is 91.5 91.9 deg F. 1001 1120 1203 1268 1830 3082 ------------------------------------------------------------------------------ DATE: 17:19-MAR-2010 At: Fermi TOPICS: From Lab F get the ADF-2 cards from the NuMi T962. Pull from: Top Crate: A21, A17, A18, ,B19, B20, B21, D33, D34 (A21 is Maestro) BOT CRATE: C3, B17, B18, ,A19, ,D35, D36, A20 ( C3 is Maestro) Stored at D-Zero was: C5, D37 (D37 is Maestro) Now stored at D-Zero will be: A20, A21(M), B20, B21, C3(M), C5, D36, D37(M) B21 has been but is not now a Maestro. I will make it again Maestro. To MSU will go: A17, A18, A19, B17, B18, B19, D33, D34, D35 The blower is still over at Lab F. CalOp, Operations, and PAB meetings. Bo PreAmp Green Box pulled apart. Return from PAB to MSU: Bo white preamp box, 5x Bo preamp cards, Bo LV fuse panel, Bo bias Voltage final filters. MCH-1 walk through: racks are closed up and visible fans are running. M124 top back of SCL Hub-End backplane is 91.5 91.8 deg F. CR657 1993 1203 1824 1052 1075 (D6/H2) (H5/A3) (A4/H1/D3) ------------------------------------------------------------------------------ DATE: 22:24-FEB-2010 At: Fermi TOPICS: TCCs to and fro, ReBuilt Blower Returned to D-Zero the Dell box #2 "new spare" TCC for the Level 2 system. This machine had been at MSU for Philippe to install the new version of the Bit-3 drivers. Current state of TCCs at D-Zero: In the brown cabinet: Dell box #2 is a spare L2 TCC without a Bit-3 pci card installed. The intent is to make this the online L2 TCC during this summer's shutdown. Dell box #4 is a spare L1 TCC with a Bit-3 pci card installed. This is The Spare L1 TCC that should be put online if there is a problem. The intent is to make this the online L1 TCC during this summer's shutdown. Currently this box will try to configure the M101 Run IIA Cal Trig readout Bougie Spark cards but that is not a problem. The original 12 year old L1 TCC without a Bit-3 pci card installed. This is an emergency backup spare L1 TCC and will remain at Fermi. In the rack M125 above the air conditioner: Dell Box #1 is the online L2 TCC. Dell Box #3 is the online L1 TCC. The original 10 year old L2 TCC with a Bit-3 pci card installed. This is The Spare L2 TCC that should be put online if there is a problem. All 6 of these boxes have labels. I'm taking the original HP "dual personality" spare TCC back to MSU. It is no longer useful here and leaving it sitting here with the current space machines may just confuse people at 4AM. On the blower that was returned from IMS last trip I have now glued in the mounting hardware and made a final short test run. This blower is now ready to install, labeled, and stored in the brown cabinet. Check MCH-1 all visible fans are running and the doors are closed. M124 at the top back of the SCL Hub-End is 91.7 or 91.8 Deg F. NuMi T962 beam run ends and is taken apart after a 10 TByte run. ------------------------------------------------------------------------------ DATE: 11,12-FEB-2010 At: Fermi TOPICS: TCCs to and fro, ReBuilt Blower, M122 -2V Minor Alarm, Trigger Tower Noise I brought back down to D-Zero the original 12 year old L1 TCC. It will stay here as a "back up" spare L1 TCC. I'm taking back to MSU the "new" L2 TCC that can not be used because it has the old Bit-3 software that does not appear to work correctly on these "new" L2 TCC boxes. Philippe will put newer Bit-3 software on it which does appear to work correctly. I will then bring it back to D-Zero and this summer during the shutdown we will swap it in. We are currently running on "new" boxes for both L1 TCC and L2 TCC. I have labeled both of these boxes as the "running online TCCs". The original 10 year old L2 TCC box (with its new disk and which had been back running online until about 27 January 2010) is still in the short rack on top of the air conditioner. I have labeled it as the "to be used if there is a problem spare L2 TCC". We also have down here in the brown cabinet: - The second "new" spare L1 TCC. It has the Bit-3 pci card in it but has not yet been tested. As of right now it will try to Configure the old M101 Cal Trig Readout cards but aside form that it should work just fine. The intent is to swap it in during the summer 2010 shutdown. - The "original" HP dual personality spare TCC. As far as I know this box will soon no longer needed/useful as a spare TCC and should come back to MSU. I have gotten back from Mike Matulik the second TFW air blower that IMS has rebuilt. I need to bring to Fermi a set of bolts and nuts to be "glued" in place so that it is ready to install. This blower is in the brown cabinet. Friday morning Geoff Savage told me that we have had had a non run pausing alarm, a "Low" alarm, for a set of TFW power supplies for sometime. I'm not certain how long this alarm has been set. The name of this alarm is approximately TFW_M122/ABC_-2VV. The -2 Volt supplies in rack M122 are just fine. I checked them again this morning and in August 2009 I did a complete check of all TFW supplies. The problem is that the Rack Monitor channel that is watching the M122 -2V supplies does not read correctly. This Rack Monitor channel typically reads just short of its "Low" alarm threshold. Over the past few years there have been other times when this Rack Monitor channel has drifted a little more extra negative than "normal" and gone below its "Low" alarm threshold. - This morning Geoff decreased the "Low" alarm threshold for this channel by 100 mV (i.e. increased its absolute value by 100 mV). It's fine to run this way for now. - At an appropriate opportunity it would be nice to replace this Rack Monitor. - Once the Rack Monitor is replace we will need to check all the readback values and then return the TFW_M122/ABC_-2VV "Low" alarm threshold to its normal value. - I do not know if we can watch the SES Alarm Display from MSU. - This is the rack monitor with Address 26 on top of M122. It is the upper one of the 2 rack monitors on top of M122. Looking at the terminal block at the top of the rack in the voltage monitoring cable runs I see: +5.080 Red +3.331 Blue -2.133 White -4.638 Orange this is wrt the Black wires at the terminal block. The TOM card test points for the -2 Volt supplies read: Top -2.003 Mid -2.023 Bot -2.016 Late Friday afternoon Mike Matulik brought me a spare Rack Monitor to swap in. I have labeled this spare Rack Monitor and left it on our 1st floor bench. Selcuk was on CalMuon shift Friday during the day. Before noon there were some L1 Cal rate fluctuations. Trigger Towers -4,25 HD and -8,28 HD looked noise in the TT Monitor Gui Display. I looked with a scope. -8,28 HD has 250 mVpp of reasonable good looking sin wave oscillation. it is almost stable it has about a 1.1 usec period none of its neighbors that I looked at had this 1.1 usec period sin wave - but many of them have 132 nsec sin wave content. --> -7,28 HD has a very nice clean 50 mVpp 132 nsec sin wave. I should verify that this TT has stable 132 nsec sin wave over time and then use it to watch the power down this summer. Later in the afternoon when I looked -8,28 HD looked pretty clean - no more 1.1 usec period junk -4,25 HD I did not have time to look at this TT until late Friday afternoon. At that time it looked OK. Note that -4,25 HD has had trouble before - see our Run iia TT BLS log file. The current summer 2010 shutdown schedule is for a 4 week shutdown starting July 17th. All visible fans are running and door were closed. M124 top of SCL Hub-End backplane 91.8 - 92.2 Bill said that the two diodes fixed his circuit board. NuMi T962 check looks OK. ------------------------------------------------------------------------------ DATE: 21,22-JAN-2010 At: Fermi TOPICS: Walk Through Inspection and Temperature Check, Blower Rebuild The racks look closed up and all visible fans are running. The M124 back temperature is 91.3 to 91.7. Thursday late afternoon the belts in the MCH-1 air conditioner were changed. While the unit was off it lost its set points or something. When it was turned back on it thought it should do 100% heating. This was not spotted for 10 minutes or so. M124 back went up about 11 degrees F. The Ops shifter reloaded the set points. This is a known problem with some of the air conditioners. Mike has a price of $201 to rebuild the blower that was pulled out from M123 rear 2 weeks ago. George gave Mike a "Task Code" for this (aka an account number). I sent George a note about replacing the rest of these blowers during the summer 2010 shutdown and checking the Internal PS fans. NuMi T962 check looks OK. ------------------------------------------------------------------------------ DATE: 6:8-JAN-2010 At: Fermi TOPICS: Replace M123 Rear Fan, Walk Through Inspection, Bring original TFW TCC back to MSU, L1_Actp NIM signal to MCH-3 for Mike & Marvin We were called at MSU late Tuesday afternoon January 5th because a rear blower in M123 had started to make a lot of noise. From what the folks a Fermi could tell the blower was still running at full speed but was just making a lot of noise. The decision was made to try and continue running through Tuesday night with the bad blower in operation. I checked with the shift captain a couple of time Tuesday evening and things continued to run OK. Arriving at D-Zero Wednesday morning I repeatedly offered the RCs the option of just turning off the bad blower and continuing to run normally until there was a good opportunity to replace the bad blower. Their decision was that they wanted it replaced between Stores Wednesday afternoon. Using ear muffs and a 2 foot long hose it was verified that it was the M123 rear blower that was making noise. It was replaced with the blower that was rebuilt by IMS in August 2009 (i.e. the old M101 rear blower that failed in Feb 2009). Time stamps of the various major way points during the replacement: 12:44 Turn off and start work. 13:05 Front cabling is off and all crate M123B cards are out. 13:53 The old fan is out. 14:33 New fan is in and is OK in a 30 second test run. 15:20 All cards are in, the cables are installed but not dressed, TFW power is back on. 15:35 TFW FPGAs have been Configured, TFW is Initialized, give the TFW back to the control room. 16:22 New Store is in. 19:35 The first Physics run of the new Store is started. In the later part of the afternoon: dress cables and Philippe checks the start up log files. - My initial attempt to Configure just M123B, without power to the TFW Communications Crate clearly could not have worked. - Philippe is checking into the attempt to Configure the old Run IIA Bougie cards in M121. - M123B Slot 18 Gated Scaler SM-57 the current theory is that this card is not currently used by TRICS although it may have been at one time during 1999 or 2000 like that. TRICS does Configure it as a Gated Scaler TRICS does not Initialize it TRICS does not turn on its BSF LED TRICS does not read Monitor Data from this slot Drawings showing the function of M123B Slot 18: /l1/framework/drawings/the_layout_racks_m123_crates.gif shows M123B slot 18 as the Sequential L1 Acpt Num /l1/framework/hardware/rack_crate/run_ii_m123_card_addresses.txt shows M123B slot 18 as "open" Both of these files are dated 2-Jan-2002 Log Book clips about M123B Slot 18: DATE: 5:7-DEC-01 At: Fermi M123 Bot Slot 18 has a Scaler Module installed but nothing is connected to either its P2 or P3. DATE: 7-NOV-2000 At: Fermi (Information about spare cards stored at Fermi and then) Note that slot #18 of M123B also has a Gated Scaler Module (Blue front panel) in it that is currently not being used by TRICS. It is Scaler Module SN# 57. DATE: 9-FEB-2000 At: Fermi L1 CNX appears to be working again. Philippe modified Trics to include the environment variable %CNX% which points to D0_CNX here and MSU_CNX at MSU. We're now using 7.2 Rev 3. Additionally, the old cnx files had to be modified because 1) bla 2) M123 bottom has scalers in 18, 19, and 21 only (internal connectivity only) 3) bla DATE: 26,27,28-JAN-2000 At: Fermi Install SM Gated for Sequential L1 Accept Num in slot 18 of M123 Bottom and Install the FM-Latch in slot 20 M123 Bottom. We now have all the cards installed for the currently understood running configuration. Kirsten updated the what card is where in the FW inventory file and started the Down Load Command file loading up all the cards. I installed allen cap screws and mounted the self-locking nuts with hot melt on the one remaining spare blower at D_Zero. I then ran it for a few minutes. It was built in February 1996. I need a plain 5/32" allen key that has been cut off to keep at Fermi for use during blower installation. The Tick Select that is regularly used for the CFT scans is programmed via: Mst 1 Slv 2 Slt 21 Chip 16 Reg 41 This must be Tick Select #1 which is AOIT 252. If there was a problem with this one I would have heard about it Philippe would like the old original TFW TCC back at MSU to double check some files. I will take it back with me. M124 Back Top temperatures: 91.6 91.8 Visible fans are running NuMi T962 check looks OK. ------------------------------------------------------------------------------