D-Zero Hall Log Book for 2005 ------------------------------- The most recent entries are near the beginning of this file. This file begins in January 2005. This file contains both Trigger Framework and L1 Calorimeter Trigger entries. Earlier D-Zero Hall Log Books are on the web in one of the following directories: http://www.pa.msu.edu/hep/d0/ftp/l1/framework/logs/ http://www.pa.msu.edu/hep/d0/ftp/run1/l1/inventory_logs/ ------------------------------------------------------------------------------ DATE: At: Fermi TOPICS: ------------------------------------------------------------------------------ DATE: At: Fermi TOPICS: ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ DATE: 14:16-DEC-2005 At: Fermi TOPICS: Work on the Sidewalk ADF system, work on the Purple Haze Viewer, Edit the Init_Post_Auxi_L1CT file Sidewalk ADF Work Tested the 10 spare LVDS cables from Nevis and then pulled apart the test system. Its pc and bit-3 and signals generator and mux box and channel link tester are all going back to MSU. Swapped the two ADF Crates that are in racks by them selves at the South end of the row of racks so that the labels on these crates match the drawing that was made after the labels were put on the crates. Take apart the test setup for: ATC cards, PP cards, PFC cables, and LVDS cables. All of this equipment is going back to MSU. Before pulling it apart I played around with the issue of VME sometimes not working with some applications in this non Vertical Interconnect system. I did the following: - Power cycling the VME crate made no difference this afternoon. - The order of starting up first the GUI or the Engine made no difference. - To begin a test I always killed both the GUI and the Engine and then restarted both of them in one order or the other. - If the first thing that I tried was Configure it always failed. After configure failed I always could read from 0x2c0000. After reading OK from 0x2c0000 the configure would still fail. I made many tests of this. 0x2c0000 is just some register address at slot #11 i.e. the Miestro. - If the first think that I tried was Reading from 0x2c0000 then everything always worked OK. I then could configure just fine. I made many tests of this. - When the first thing that I try to do is Read 2c000 there is a noticable pause between the "port connection" message in the engine window and the logging of the read operation. - When the first think that I try to do is Configure then I don't think that I see this pause between the port connection message and the logging of the I/O activity associated with trying to start the Configuration process. All 80 ADF-2 cards are installed and look OK so far. We need to look at the Find DAC Log file on Monday. Jorge has all 80 tested ATC cards installed. SCLD connections look OK. The Comm Crate looks OK - all cards installed and all of our cables are labeled and tied down. Purple Haze Scope setup to record Purple Haze noise from the Calorimeter. - As I understand it this noise only happens at the highest luminosity. To have a chance of seeing this noise, the scope setup must be left in place because there would not be enough time to quickly put everything together at the beginning of a good Store - This scope setup is using an internal signal from the L1 Cal Trig to trigger the scope. The signal that was stolen from L1 Cal to do this is the Count Comparator #3 for Total_Et Ref Set #3. This resource is not currently being used by the Global Physics Trigger List. This was checked by downloading 14.82 last night and was verified during the trigger download for the Store this afternoon. If a new Global Physics Trigger List does need this resource I can explain how to restore it over the phone. The And-Or Term from the L1 Cal Trigger that was disconnected from the TFW is And-Or Term #151. - The scope is setup to monitor the BLS signal waveform from the Hadronic section of 3 Trigger Towers in the eta,phi region that is most effected by the Purple Haze noise. These analog signals are obtained from monitor points in the L1 Cal Trig. The current scope connection to these monitor points will not effect the operation of the L1 Cal Trig. - Right now the trigger for the scope is 20 or more Hadronic TTs over 7.0 GeV in the TT_Eta range -9:9. I have never seen this trigger fire (but we could prove that it will fire with the Calorimeter Pulser). If I set it to 10 or more Had TTs over 7.0 Gev then SMT readout noise fires it about once a minute. Please let me know if you would like the Count Threshold changed. -----> The Init_Post_Auxi_L1CT.mcf file was edited to by default set -----> the Tot_Et Ref Set #3 Count Comparator #3 Count Threshold to 20. - The Tot_Et Ref Set #3 Count Comparator #3 output is plugged into Channel #4 (Green) of the scope. This signal from the L1 Cal Trig is once side of a Differential ECL signal with the other side of this twisted pair cable at ground. The trigger level knob on the scope is set correctly. This cable has the correct termination at the end. Do not turn on Channel #4 internal scope terminator. - The 3 BLS signals that are connected to the scope are: Channel #1 Yellow Had TT signal +2,15 Channel #2 Blue Had TT signal +4,15 Channel #3 Violet Had TT signal +2,18 If you need to verify where these TTs are, the map of TT eta,phi is in the MCH right above the TCC console. - These 3 scope channels are set to 500 mV per division. The quiescent voltage level is about -1.00 Volts which indicates 0 GeV in the Calorimeter. These TTs are at about 90 degrees so +1.00 Volts is about 60 GeV. The dynamic range of this signal is -1.0 Volt to +1.0 Volt. These signals are back terminated. Please do not turn on the internal scope terminators or you will effect the operation of the L1 Cal Trig. - I have been using single sweep to capture scope pictures in this setup. Be careful - the "Auto Setup" button on this scope is right next to the "Single Sweep" button. Once I accidently hit the Auto Setup button and that screwed up everything - I watched during the beginning of the Store Friday afternoon and saw nothing that looked like Purple Haze noise. Look at a download of Global Physics Trigger List 14.82 to see what Ref Sets are set to what thresholds. Ref Set EM Total_Et ------- ---------- -------------------- 0 6.0 GeV 5.0 GeV 1 12.0 3.0 2 9.0 4.0 TT_Eta -13:13 3 3.0 7.0 TT_Eta -9:9 ------------------------------------------------------------------------------ DATE: 29-NOV,2-DEC-2005 At: Fermi TOPICS: M109-M110 Water Leak, Sidewalk BLS signal delay, ADF Crate and Comm Crate Currents, Tuesday gave the DAQ Shifter talk on TFW... Run 2B L1 Cal Trig meeting - specifically asked if it was OK to take apart the test setup for the ppc, atc, pfc, and lvds cable so that that crate could become part of the full 4 crate system. That question was not answered by the managers. M109-M110 Water Leak This was in the next to the bottom radiator. It was at the joint between the header and one of the two pipes that runs through the radiator itself. I used the now standard technique: wire brush and scotch brite the area, turn off and de-pressurize that pack of radiators, apply the epoxy, allow two hours to cure, start up the water flow again. For practice I did two additional patches (over the ends of headers) on that radiator. It basically works OK but its clear that you want to "apply and peel off" to clean the surface and then work in a well as possible for about a minute. The pot life is just fine. For another test I put a glob on an uncleaned section of the center aisle manifold. For that test I just did the apply to clean, peel off, mix it a final time, and apply. Checking that "uncleaned" patch a day later and I was able to knock it off with a screwdriver. This stuff is not magic in its ability to bond. Wednesday after the cooling water interruptions the new M109-M110 patch still looked OK. Need to get more of the same type of cable ties that were used for the original installation of the mud-flaps. Having the same kind saves pulling them the whole way out. The original ones are Panduit PLT2I-M Newark 78F588. Things still looked dry on Friday. I have not yet put back the drip detector strip between M109-M110. Wednesday morning, all turned off in MCH and on the Platform for the HVAC UPS work. As part of this re-start TRICS on TCC1. Turn power back on and everything was normal except that the RM's SBC did not really start up right. In order to let the DAQ shifter stop it, it I first had to push its RESET button so that it would boot a 2nd time and then it was rational and the DAQ Shifter could control it. Loading the Master Clock went with zero problems and was done from the Global Monitor set of tubes, i.e. the Master Clock GUI is no longer locked to the DAQ Shifters tube. After this HVAC UPS work, start up the L1 Cal Trig. The Upper Tier 1 supply in M105 is making a lot of noise. Run Diagnostic-Exerciser on L1 Cal Trig: 2x CHTCR, 2x Eng, 2x Mnt, 2x 1 meg loops all looked OK. Give it to ZB. Sidewalk BLS Signal Delay Needed to double check the delay of the BLS signals on the Sidewalk wrt their timing as they come out of the long Blue BLS cables in the MCH. That is, what is the delay going through the Saclay Splitter and the added ribbon cable to reach the Sidewalk. Measure this with a scope and the Hand Pulser. The delay in the splitter is about 3 nsec. The delay on the cable run from the splitter to the Sidewalk is about 82 nsec. So the total extra delay in the Sidewalk BLS signals is 85 nsec. I check the currents in a fully populated ADF Crate while it is running T7 Data Path firmware under SCLD control. I check the currents under the following conditions: Wiener Current drawn from that supply in Amps Front Panel under the following conditions Power Supply ----------------------------------------------------- ------------ Turn-On Pre FPGA Pre Ctrl Reg Normal Name Volts Spike Configuration Initialization Running ---- ----- ------- ------------- -------------- ------- +5V0 5.0 6 12 15 17 U1 5.0 10.8 10.8 10.8 10.8 +3V3 3.3 44 10 6 34 U5 5.0 9.5 9.5 9.5 9.5 The "Turn-On Spike" is the highest reading that I saw on the front panel display as the crate was powering up, i.e. during the Wiener soft-start process. The "Pre FPGA Configuration" reading indicates the stable current draw after the crate was fully powered up but before any of the firmware had been configured into the Xilinx FPGAs. The "Pre Control Register Initialization" reading indicates the stable current draw after all the Xilinx FPGAs had been configured but before any of the Control Registers had their default values written into them by the TCC "Initialization" process. The "Normal Running" current draw reading was the stable value after everything was setup and the full ADF Crate was performing its normal function. Note this is with Data Path T7 version firmware and the crate was under SCLD control. Then check the current drawn by the Communication/Control Crate Wiener Current drawn from that supply in Amps Front Panel under the following conditions Power Supply -------------------------------------- ------------ Turn-On Normal Name Volts Spike Running ---- ----- ------- ------- +5V0 5.0 17 16 +3V3 3.3 1 1 To startup the TCC for the Run 2B L1 Cal Trig: log into TCC3 with the d0l1 account > startx (return) from the "tool bar" at the bottom start the following: "TCC" for the engine part of the new system "gui" for the human interface part of the new system ------------------------------------------------------------------------------ DATE: 9:14-NOV-2005 At: Fermi Run 2B L1 Cal Trig work, Chan Link Test problem, Routing Master, M109-M110 water, Power Supply for CTT, Master Clock, LArTPC meetings, TFW Power Supply Fan, Master Clock error and Guidance The work on the Routing Master was completed with no problems. The Routing Master's new fast VME CPU card is now cable tied to the 6U/9U adaptor card that lets this VME card plug into the "TFW style" crate. Aran got the Fermi tag number that he needed. There were no problems restarting the Routing Master using the standard instructions. The fan in the M122 Middle TFW power supply was replaced. There are 10 TFW power supplies chassis running in MCH-1, each with two of these fans. This was the first failure. They have been running since February 1999. We had the "normal" set of problems getting everything running again after TFW had been off. The new M109-M110 Cal Trig water leak, that started last week, is in the next to the bottom heat exchanger. The previous M109-M110 leak was in the bottom heat exchanger and was fixed on August 31st with the special epoxy. Good, this is a new leak and is not a problem with the epoxy falling off after 2 months. I have not yet actually seen the leak itself but I assume that it will be in one of the standard locations in the copper pipe. As before it will take about 4 hours to take things apart and patch this leak with the special epoxy. There was not enough time to do that work Thursday evening between Stores. The current leak rate is order of a drop every 5 seconds. Bill was around and got to see the drip from the bottom of the mud flap. After looking for the water leak I restarted the L1 Cal Trig and ran the tester/exerciser: PROMs: 2x CHTCR, 2x Energy, 1x Momentum, and then 1k loops followed by 1 Meg loops of exerciser. Thur night Fri early morning at about 1AM the Master Clock threw an error and the resulting Major Alarm in the Significant Event System paused the Beam Physics run. It looked exactly like the BD reset the LLRF - but how could they have done that ? Spent 3 hours and wrote the Guidance files for the Master Clock and added that to the Master Clock instructions. Since Thursday afternoon there was a problem with the Channel Link Test not running OK. The Chan Link Test Receiver just stayed in the "Waiting" state. I absolutely verified that the ADF end was OK, i.e. SCLD signals looked fine on the ADF Crate backplane and tried a different ADF-2 module. The receiver looked OK, i.e. I checked with a scope the data lines taking the decoded Chan Link data down to the FPGA and it all looked believable. The solution was to replace the ATC card and then all worked OK. Step #7 Channel Link Test started up just fine. I tried that 6 or 8 times. I then let it run a long time, > 6 hours, and Step #8 Check Again was looking just fine. Power Supply 3.3V for Stefan Grunendahl and John Anderson There are 2 spare TFW Power Pans at D-Zero. They are TFW Power Pans SN# 2 and SN# 4. See the TFW inventory pages on the web. I pulled apart SN# 2 (which had previously been lent to the Level 2 people) and let Stefan and John use the 3.3V brick out of it. First I made certain that both of these spare TFW Power Pans are working OK. This will leave us only one spare TFW Power Pan at Fermi until I can get the one down here from MSU that is in the test crate there. I send John/Stefan a note about using this supply. I thought that it was just going to be for a quick test to determine if power supply was the problem with CTT or not. Instead it looks like the technicians are making a fancy setup to install this supply. They made a plastic cover over the lug end of this supply that mostly blocks all the air flow. I put the user's guide for the ASTEC Spectrum VS3 supplies on the web in the TFW rack/crate directory. ------------------------------------------------------------------------------ DATE: 26:28-OCT-2005 At: Fermi Run 2B L1 Cal Trig work, CJT(2,3) noise, Routing Master, Thursday at about noon, during a Store, the L1 rate went up 400 Hz. This showed up mostly in L1 Spec Trig #108, the CJT(2,3) plud no hits in Luminosity. This L1 Trig went from its normal 2 Hz up to about 400 Hz. This sounds exactly like what happened at the end of May - fist day of June 2005. The new L1 Examine Display did not show any hot towers. The old L1 Cal Trig Examine showed +9,10 and +10,10 HD hot by a factor of 5 compared with neighbors. Checked these with a scope and the noise only appears when the DAQ system is taking events. The noise stoped when the DAQ system was paused. On the scope, at the normal BLS signal viewing horizontal sweep rate, the noise looks like the base line jumping around. If you slow down the scope what you see is a negative going bump followed 290 usec later by a positive going bump. Both bumps are about 50 usec wide. We Excluded +9,10 and +10,10 EM and HD and resumed running with normal rates. On Friday between Stores, we verified that we could still see the noise on the scope with ZB running. Cycling the BLS power supply OFF and back ON did not change the noise. Then Norm swapped BLS Crate Controllers between PS00 Mid (where +9,10 and +10,01 are) and PS02 Mid. PS02 Mid covers eta +7:+14 phi 14 and eta +11:+20 phi 15. After the swap we checked +9,10 and +10,10 and eta +9:+12 phi 14 and eta +9:+12 phi 15 and all was quiet. Cycling the BLS crate power and all was still quiet. Thus we "un-Excluded" +9,10 and +10,10 which were Excluded during the Store on Thursday. Friday morning the L3 people wanted to put a faster processor into the Routing Master. This SBC sits in one of our TFW 6U-2-9U TOM modules. This swap went OK and now the Routing Master can handle a higher L2 Accept rate and more L3 nodes, ... Details from Andy are, "... swap the old RM (VMIC 7750: Pentium 3, 933MHz) and put in the new VMIC 7850, which is a Pentium 4, 1.7GHz with 256MB RAM. The system booted up fine and we were able to do several tests in Zero Bias.All in all, extremely succesful morning! We have a brand new Routing Master! Thomas, Andy, Dan and Aran" This needs to be visited on the next trip and if all is still OK then tie this new processor card to the TOM module. The fear is it pulling part way out so that only a few of the +5V pins are running it. Installed the real Wiener Communications/Control crate in the Run 2B L1 Cal Trig and moved to the unix TCC. Ran the VI Links from the new Comm Crate to the two newly installed Wiener ADF Crates. Big problem getting VME cycles to work in the upper of the two newly installed Wiener ADF Crates. The problem was that the lower lefthand edge (when viewed from the front) of the backplane was not screwed into the crate mechanics. Did Wiener miss it or did our friends at Fermi screw it up ? Ask Linda and John if they knew which Wiener ADF crate had been repair. Neither of them had kept track. As an aside, John wanted to pull apart the Wiener power supplies and add a 15A AC line fuse inside the supply. He has already done this to the Comm Crate supply. I said "please no" wrt the ADF crates. If a fuse is needed then clearly it needs to be at the source end of things. The 4 Wiener ADF crates are: Rack S105 JB Test Stand Wiener SN# 5196041 VI Link #1 Rack S106 Orig Sidewalk ADF Wiener SN# 5196039 VI Link #0 S109/S111 new this week top Wiener SN# 5196038 SI Link #2 bad bkpln S109/S111 new this week bot Wiener SN# 5196040 SI Link #3 Does Alan care which crate in S109/S111 is which ? Need to find out before putting the label on. Where are the remaining 2 labels and their flat flat-head screws ? The first two crate slot labels were put on in early June. We also ran a VI Link into the Readout Crate. That appears to be working OK. A practice address to use is A24 $090070 or A32 $14090070. This is a register in the slot 9 VRB. I put into a box 4x spare VI Mst and 4x spare VI Slv. They are in static sacks. The box is well labeled and in one of Linda's cabinets on the Sidewalk. Need to send a note to Linda. Moved the SCLD to the T5 firmware. After this change, we now have SCLD SN#1 installed on the Sidewalk and SCLD SN#2 is going back to MSU. I need to talk with Denis about SCLD SN#3. Made timing measurements and scope pictures of SCLD T5 in operation. It came out as expected. As part of moving the Sidewalk Run 2b L1 Cal Trig test stand to normal running it needed to be plugged into the TFW. Marco Verzocchi was consulted and determined that it should be plugged into TFW And-Or Term Inputs 176:191. SiFi Cable #6 had been plugged into this location. Still need to fix the fan in the front of the power supply for M122 Middle. Must schedule this for the next trip to Fermi. The d0tcc3 node was brought to the sidewalk. The copper Model 617 PCI module that had been used as a temporary target for the bit3 software was removed and returned to the rack of spare cards, as it is the spare for L1 TCC. The Model 618 which was just bought was installed and connected using a 10 meter optical cable. This node is on building power (not on detector ground) and part of the online cluster. The VME module of this Model 618 interface was plugged into the new Communication Crate in Slot #1 Crate Arbitrer. The existing 618 VME module was jumpered to no longer be the crate arbitrer and installed in slot #2. This second interface is available to the Columbia team as an independent VME interface to be used from their Windows PC or the tbilisi Clued0 node. Basic communication to the ADF cards worked. Configuring ADF FPGAs worked with zero problem. Using the "Configure Crate" button of the "Configure FPGA" GUI dialog caused some kind of lockup where no further VME IO could be performed via the linux PC or via the Windows PC. Power cycling the ADF Crate or pushing on the Reset button of the Vertical Slave would clear the problem. This must mean that a VME cycle to a non-existent address was left hanging in the vertical slave module. This is not reproducible by simply reading and writing to a few registers. It must be the more intense configuration phase that is locking the bus. These are memory mapped IOs, but the control code explicitely asks the bit3 driver if it had encountered a problem and even requests and check the current status word to examine the appropriate error bits. If we swap the optical cable with the older 618's VME module, there is still no bus error, but it reads back 0xffff instead of the value it tried to write. The windows driver using the newer 618's VME module detects bus timeouts as usual. The "Configure Crate" button was trying to detect which slots have an ADF card by reading BCPAL Register #7 and TCC is supposed to detect a VME Bus Timeout for non existent slots. The windows bit3 driver does, but the linux driver does not. It does not detect any error, and even seems to read back the value just tried to be written. This means that the GUI thinks the slot exists and tries to send a configuration file. The python code was probing slots 0:21 for simplicity, and was thus trying to configure non-existent slot 0 and 1. Changing the range to 2:21 solved the problem, and the slot detection was also made smarter by trying to write 0x00 and 0x01 to Register #0 and doing a separate read to verify that the VME write was successful. This needs to be investigated further. One idea is to swap with the model 618 from the MSUL1B test stand PC that Jorge uses to test cables. This would further determine if this is just a driver issue, or an interface issue. Philippe needs to review what user softwar esettable options exist for the linux driver interface. Write a Configure_FPGAs.mcf which calls Configure_FPGAs.dcf which calls crate0_all.dcf to load T7 firmware in all cards of the crate (note: we can/should skip the intermediate dcf file and call the per crate dcf file(s) directly from the mcf file). Sending a "Configure_FPGAs" message causes TCC to execute this file. Successfully. The TCC software configuration files are currently in /tcc/L1Cal_IIb_Work/Config/ and the Fpga configuration files are in /tcc/L1Cal_IIb_Work/exo/ The first populated ADF crate includes coverage for eta(-4:4)phi(1:32). TCC can be told to restrict the L1Cal coverage to this eta(-4:4) range by sending the following COOR-like message "TrgMgr_L1Cal_Coverage TT_Eta(1:4)". In fact we put this command in the Boot_Auxi.mcf file to be executed automatically at startup. If we send an "Init" message, TCC initializes the first 16 cards of the ADF crate. successfully. Mike Mulhearn had new TabGab code to test. We go over the differences again between Configuration, Initialization, and Run Programming. We remove the Configuration code from the Initialization routines Mike had prepared, make some adjustments, and compile the new code into the windows development version on Philippe's laptop. Then into the linux work area on d0tcc3. This new code is now able to initialize the TAB cards. We already had code that could configure the TAB and GAB, while this had never been fully tested on the windows version. The coverage needs to be further restricted to eta(-4:4) and phi(1:4) to limit the intialization to the first TAB card only. Send an "init" message. Successful. We tested TAB configuration from d0tcc3. This was successful as well. We tested the Setup_Readout_Crate.cmd GUI python command after adjusting it to control VRB slots 9 and 10. This seems to be successful and a run was started and data was flowing through the SBC. Tested reading back and dumping an event from the SBC, after pausing the run, and this seems to be working as well, as Mike recognizes his payload. The initialization of the ADF FPGAs was changed to match the new SCLD timing and load an adjustement delay of 0 counts (instead of 8 counts previously). Made a pair of icons in the bottom panel of the gnome windows manager to start the TCC and GUI applications. We are currently using the d0l1 account to log onto the d0tcc3 console, and we have to manually invoke "startx". A dedicated account will be created for d0tcc3 so that it will be independent of d0l1 which is used for other things, and we can put it into the same "d0_trig" user group as the account "laurens". The screen resolution is stuck on 800 horizontal pixels and it is not clear where the limitation is, as both the video card and the screen can handle much higher resolutions. This also needs to be investigated further. The arangement of directories is not optimal, as the compilation area, the GUI python code, the logfile area, the GUI, and the firmware are all at the same level. ------------------------------------------------------------------------------ DATE: 24&28-OCT-2005 At: MSU Topics: Errors in TCC logfiles This was found in Trics' logfile. %%24-Oct-2005 06:55 E$Pre-Write Check Mismatch: Read = 0x0001 / Last Written = 0x0000 E$_Read Mask = 0x0001 / Write Mask = 0x0001 E$_@Master#0/Slave#1/Slot#20/Chip#17/Reg# 1 This is in rack M122, middle backplane 20. SM Per Bunch Scaler for Exposure Group #0 for Ticks 159:81 This Chip is Board Support Function FPGA and this is the register for 1 R/W Chip Control Status Register (MSW) This register only has one allocated bit (0x0001) and it is used to turn the computer controlled LED on/off. TCC was thus trying to display a scaler count when this happened. This means that Trics found a different value (0x0001) in this register than it had written the last time (0x0000). %%28-Oct-2005 20:07 M$Receive Msg #26298 : 000000000000048493 L1CT_Ref_Set TOT_Et_Ref_Set 0 Value 5.0 I$Process Msg Command : E$ Post-Write Check Mismatch: Read = 0x01 / Last Written = 0x11 E$_Read Mask = 0xff / Write Mask = 0xff E$_@MBA#175/CA#45/FA# 40 I$ Recover Attempt: Try Writing Register Again I$ Second Register Write Successful This is in rack M106, upper half of the upper CTFE backplane backplane This is CTFE for Phi 7 Eta -5 through -8 This register is 40 Reference for Total Et Threshold Comparator #1 Channel #2 This means this was for eta (-6) Trics was trying to write 0x11 but found 0x01 when verifying. Trics tried writing 0x11 a second time, and this second attempt was successful. There was no harm done to data quality. The power supplies for this backplane do not show excessive noise. ------------------------------------------------------------------------------ DATE: 12:15-OCT-2005 At: Fermi Topics: M107-M108 Water Leak, Run 2B L1 Cal Trig work, Power Trip of TFW The water leak in the M107-M108 pack of 4 radiators was in the next to the top radiator. It was in the side of the header pipe on that radiator that is next to the yellow support bar. It was a pin hole right in the side of the junction between the header and the thinner pipe that runs through the radiator. I.E. it was not in the cap of the header. Wire brush and Scotch-Bright this section, remove water pressure, dry, and then Mike put on his special epoxy. After 1 1/2 hours put back on the water pressure. Things still looked dry 1/2 hour later so put back on the G10 mud flap and shockless system shields. All work was done Thursday between Stores and then while BD changed a power supply before the next Store. At the next major shutdown we should change all radiators in the TFW. M122 has two radiators with clamps. Run 2B L1 Cal Trig. Tentative plan is to move racks around and get things in the Sidewalk setup for running during Stores during the week of 24-Oct. There is a tentative request/proposal to move the ADF Crate timing to it 1458 nsec location. Saturday morning at about 6:25 AM about 20 minutes into a new Store the TFW tripped off. Its RMI reported a smoke alarm. I was told that the RMI reset without any problem. The DAQ shifter was making progress getting the TFW running again by the time that I got there. They were confused by the mixed together format of TFW and L1 Cal Trig that is presented by TCC. I was told that some other systems also tripped off at the same time: power supplies on the Platform and Brendan had to replace some fuses in the Luminosty system. Store was then lost at 8:16. After things were running again, spent sometime looking at the power supply displays. On the L1 Cal Trig display (L1 Cal Trig did not trip off) the bricks with the most problem are: M111B -2V reads -1.86V M106A -4.5V reads -4.38V M111B -4.5V reads all over the map as low as -4.37V M110A -2V is noisy. Look at TFW which did trip. M122 -2V is reading high. It reads -2.2V. I checked with a Fluke meter and the supplies look just fine. This appears to be just a high readback. I checked all the fans in the TFW and M101 to make certain that they all started to run OK. The fan on the right hand side of the middle power supply in the bottom of M122 is not running. I need to find and bring a replacement fan for it to D-Zero and ask for some downtime. ------------------------------------------------------------------------------ DATE: 06-Oct-2005 At: MSU action at Fermi Topics: L1Cal Power trip Bill Lee calls. L1Cal has shut itself off. The drip detector display in M110 shows a problem at M107-M108, but they cannot see any water on the detector strip. The display in the bottom of M114 shows a global Air Flow fault and a water pressure fault. Pushing the fault reset button does not clear the error, and the rpss does not allow powering up. Philippe is confused by what he (wrongly) understands is a *global* water *pressure* error and can't imagine where there would be a sensor for this. While they continue looking at Fermilab, Philippe finds the rpss documentation, and the photohelic sensor description that describes jumpering out the global water pressure input to the RPSS. But this is not related. After a while, Bill Lee finds Dan's note taped in M110 about drip faults causing this exact symptom. Philippe also google-found the following entry in our logbook. From Logbook entry 18-FEB-1994 To do this we installed the RMI to RPSS Box. This box is located in the bottom of rack M113 behind the RPSS. It connects to the RMI in the top of M114 via a long green RG58 BNC cable. This box is plugged into the M114 slot of the RPSS. If the RMI detects a water leak the RPSS will trip showing "Air Flow" and "Water Pressure" faults in M114. A matching note is now added to cal_trig_power_control_procedures.txt in http://www.pa.msu.edu/hep/d0/ftp/l1/cal_trig/hardware/rack_crate/ Philippe suggested to wipe and dry the detector, and they are also ready to install a spare if needed. Bill calls back later to say that the detector was indeed moist, and that it is now standing on its side. Bill confirms that they carefully closed all the doors. Norm restarted L1Cal (instead of taking up the offer to let Ph. do it), and then calls to be reminded of the source of the red error messages during eta(-20:+20) initialization. Watching the power supply display while they were turning the power supplies back on, the -2V in M111B started out yellow, then flickered yellow/green before stabilizing green after 30-60 seconds. The average voltage of this power supply remains lower (in absolute value) than the others, at around -1.85V and barely on the plot_m11b.stp chart. Also noticed that the -2V of M110A has about 200mV of noise. The -2V of M111C, M112B and M112C are also high (in abslolute value) at about -2.15 V. These are also off the stripTool chart. Beam time lost around 1.5 hours at about 35E30 ------------------------------------------------------------------------------ DATE: 28:30-SEPT-2005 At: Fermi Topics: Collaboration Meeting, Splitter Meeting, Work on L1 Cal Trig, TFW Tick Selects TFW Tick Selects There was a requested and approved change in the Tick Selects to make one available for the Layer 0 Readout Testing. The old setup for the past "N" years was: Tick Selector #0 issues L1 Acpts for tick 24 used by Muon Tick Selector #1 issues L1 Acpts for tick 37 used by CFT Tick Selector #2 issues L1 Acpts for tick 10 various users Tick Selector #3 issues L1 Acpts for tick 28 used by CFT The new setup is: Tick Selector #0 issues L1 Acpts for tick 24 used by Muon Tick Selector #1 issues L1 Acpts for tick 37 used by CFT Tick Selector #2 issues L1 Acpts for tick 10 various users Tick Selector #3 issues L1 Acpts for tick 119 used by Layer 0 Friday morning I edited the \D0_Config\Init_Post_Auxi_l1FW.rio to implement this change. L1 Cal Trig Run 2B Splitter Meeting Finally a decision - No new splitters. Work on the L1 Cal Trig Shut down to grease the blower bearings and tighten the belt. Turn the system back on and it was a smooth startup. No massive noise from the blower. I would have asked Jorge to practice this startup but it was at 5AM. Then run the L1 Cal Trig Exerciser: Two scans of the CHTCR PROMs +-1:16 1:32 0:3 EM & Tot Two scans of the CTFE Energy PROMs +-1:16 1:32 EM & HD Pg 7 One scan of the CTFE Px Py PROMs +-1:16 1:32 Px & Py Pg 2 Exercise +-1:16 1:32 Pg 7 EM&HD 0:3 EM Tot Veto Check Global Sum Ran 1k 1Meg 1 Meg with no errors. Review / document the list of ideas for verifying the L1 Cal Trig Readout: What could Cause a Problem -------------------------- 1. Loss of G-Link Sync 2. Something falling apart in the VRB - VRBC - SBC equipment 3. Something falling apart in the SPARK cards 4. Something falling apart in the Cal Trig Readout Helper 5. Something falling apart in the ERPB - Distributor_Cap 6. Something falling apart in the Timing Signals to CTFE or ERPB/Dist_Cap How to Search for a Problem --------------------------- 1. "Exclude" all the TT Channels and paint in an eta,phi ID pallet. 2. Exclude all TT Channels. Set all to 8 except for the last channel in each ERPB chain which you set to something easy to see. Load this up during ZB running. Look at 100 events by hand with the TRICS formatted dump routine. 3. Exclude all TT Channels, put in a full normal pallet pattern, during ZB capture 100 formatted dumps in the log file, pull each dump out into a fill and use a trivial script file to diff all of them. 4. Send one more word after the last ERPB card, just from a header plugged into the input of the last ERPB card. Then: - Have the Examine look for this word - Have the Spark look for this work and cut it off before the data goes to the VRB - Run this extra word only between Stores with special Spark firmware to look for it Collaboration Meeting Very interesting talks Friday morning: Physics and hardware and software talks. There is a ton of stuff going on. ------------------------------------------------------------------------------ DATE: 7-SEPT-2005 At: MSU action at Fermi TOPICS: Hot -4,31 HD Run coordinator and Dean called. L1 rate had gone up from about 700 to about 1400. During this Store, earlier there had been a couple of spikes but now the L1 rate had been sitting high for 30 minutes and was causing trouble. It was the Jet triggers that were running hot. Nothing had shown up so far in the L1 Cal Trig Examine and the assumption was that L3 was throwing everything away. They did not try stopping all the OK triggers and running with just the effected Jet triggers at a reasonable rate to try to find the problem tower with the Examine. Looked with Philippes tool that shows TT > threshold even once. In 7 sweeps that showed anything, -4,31 showed up 3 times in the Tot Et thresholds. Excluded both -4,31 EM and -4,31 HD. All OK. Put -4,31 EM back in and all was still OK. Put -4,31 HD back in and immediately back to 1400 Hz. Excluded -4,31 HD and added -4,31 HD to the Excluded file at Initialize time. ------------------------------------------------------------------------------ DATE: 31-AUG, 1-SEPT-2005 At: Fermi TOPICS: M109-M110 Water Leak, 2 Amp Fuses for ADF-2, +- eta verification During a 4 hour access period between Stores turned off the L1 Cal Trig and removed the "shockless system" and "mud flap" shields on the bottom radiator between M109 and M110. The water pressure to this 4 pack of radiators had been turned off. The area was cleaned with a SS wire brush and Scotch Brite pad. Then Mike put his patch cap on the leaking header pipe on the bottom radiator using the special copper loaded epoxy from McMaster. The epoxy was given one and 1/2 hours to cure and then the water pressure was turned back on. No visible leaks. Let the water run for about another hour and it still looked dry and felt dry. Then installed the shields and turned the system back on. Ran two cycles of CHTCR memory test and two cycles of Energy lookup memory test and one cycle of PxPy lookup memory test. Ran 2x 1 meg loops of the exerciser over the full in us eta coverage checking final values. All tests were error free. Gave the system back to ZB. Arranged to have 2 Amp fuses sent directly from Newark to Linda and thus to Jorge for the analog supplies on the ADF-2 cards. Jorge will replace the fuses in the 21 running cards. Verified with Mike that he understood that TTs at +eta and -eta are plugged into the sampe relative place on the ADF cards and thus will come out of ADF cards in the same relative places. He does and did understand this. ------------------------------------------------------------------------------ DATE: 24:27-AUG-2005 At: Fermi Topics: L1cal Technical Readiness Review ------------------------------------------------------------------------------ DATE: 18:20-AUG-2005 At: Fermi Topics: MSU L2 Inventory Tags, Install Full Crate of ADFs, L1 Cal Trig exercise, CMD files, Log file MSU Inventory Tags MSU Inventory Tag Number 017917 from the envelope labeled "Muon Detector" was put onto the Level 2 Trigger Forward Muon crate. This is the lower VIPA crate in rack M324 which is Geographic Section 0x22 name L2MUF. MSU Inventory Tag Number 018092 from the envelope labeled "Rack Calorimeter" was put onto the Level 2 Trigger Calorimeter crate. This is the lower VIPA crate in rack M121 which is Geographic Section 0x23 name L2CAL. ADF Crate For testing the ADF to TAB connection we now need a number of ADF cards in a crate. The only thing that makes sense is to install a full crate's worth. I do this knowning that the mechanical work in that rack is not finished because the people who are required to do that work have been distracted by the 100% worthless task of setting up the GlenAir cable tester to test the BLS Cable to ADF Tansition Components. The crate that was loaded with ADF-2 cards is in rack S106 i.e. the rack next to the TAB/GAB rack. This ADF Crate covers eta +1:+4 phi 1:4 through eta +5:+8 phi 13:16. All of the card that officially belong in this crate were installed except for slot 11. Slot 11 is the slot that receives actual BLS signals and is typically connected to a TAB input that can readout. Because the BLS signals that are connected to this ADF are from TT eta,phi -9,1 -10,1 -11,1 -12,1 and thus reguire a "C" Species ADF-2 Card I have left in slot 11 the ADF-2 card SN# C3 i.e. a "C" Spicies card. I'm doing this because Todd and Sabine are starting to try to think about calibration. The ADF card that should be in this crate's Slot 11 is ADF-2 SN# A10. I have but ADF-2 card SN# A10 into an SBS VME box in the "spares rack" and labeled that box. The 20 card transport box for this set of cards is going back to MSU. Note both ADF-2 cards A10 and C3 are Maestro ADF cards. Run the L1 Cal Trig Test: CHTCR PROM 2 passes, CTFE EM/HD Pg 7 2 passes, CTFE PxPy Pg 2 2 passes, Exerciser +-1:16 1:32 EM/HD Pg 7 EM/HD R.S 0:3 EM Tot HDV 1k pass 1M pass 1M pass. CMD files Made new CMD files for the sidewalk. They are: Sidewalk_pallet_ on_ten.cmd and Sidewalk_track_match_on_ten.cmd They use the Raw ADC Simulation Data Memory Block to put out all zeros 0x00 for all ticks execpt for tick 10 i.e. the 2nd Live BX in which case "track_match" puts out 0xff just for 0,0 EM/HD and "pallet" puts out the normal eta,phi pallet except for 0,0 EM/HD which is 0xcafe. Started a new TRICS log file. ------------------------------------------------------------------------------ DATE: 15-AUG-2005 At: MSU action at Fermi Topics: TrgMon to Web Socket Error Monday morning August 15th at about 9:53 there appears to have started some kind of network error problem with TrgMon V3.1.B Direct running on MSUL1A. At high rate there was a message saying, "Socket Error ... Returning 0 or 32", scrolling by in the TrgMon window on MSUL1A. I did not spot this problem until about 11:15 Monday morning. MSUL1A did not send me an email message saying that it was having trouble. For this hour and 15 minutes the "Rolling L1/L2/L3 Rate Plots" on the web look like a flat line. Looking at TrgMon on the Web you just saw the output from 9:53 AM Monday Aug 15th. I stopped and restarted the TrgMon V3.1.B Direct job on MSUL1A and things looked OK again. I did not tough the job that spins the TrgMon output to the web server. ------------------------------------------------------------------------------ DATE: 9-AUG-2005 At: MSU action at Fermi Topics: No Luminosity Monitoring Data I was paged this morning, Tuesday Aug 9th at about 6:15 Chicago time, because the Luminosity Monitor system could "not connect" to TCC. In the office noticed that the "Rolling L1/L2/L3 Rate Plots" on the web were all flat (no data). I forget what TrgMon on the web looked like. VNC to TCC. Mouse focus was on the TRICS Log File. I know for certain that I did not leave it that way last week. TrgMon on TCC showed nothing. Every time you gave TrgMon another carriage return it would: paint nothing onto the screen, wait 5 seconds, and then display its normal prompt line, "s for spec trig ...". The TRICS Log File screen looked OK and it was showing the correct L1 rate information and number of buffers in use information. I looked at the operating system display and every thing was fine. There were not page file problems or anything like that. I decided to kill and restart TrgMon. After killing it, I noticed that there were too many boxes saying "TRICS" in the panel that comes up on the right hand of the screen. One of these boxes was a release notes screen. I "de-iconized" that box and it was a locked display of the TRICS release notes. I killed this just via the "X" in the top right hand corner. As soon as I killed this a ton of messages posted to the TRICS Log File screen showing things like Luminosity Block updating. I restarted TrgMon and everything looked normal. I strongly doubt that I needed to stop and restart TrgMon. I wish that I had noticed the locked release notes screen first and left TrgMon alone. Michael dug into his logs and thinks that the problem started just after the end of the Store last night at about 19:26 last evening (Monday 8-AUG). At the end of this Store I will close the current TRICS Log File and start a new one to make it easier to recover the monitor data. This TRICS Log File has been running since last week. The next day, Wednesday Aug 10th, I learned that people who were trying to use the Register I/O memu to adjust the Tick Select #1 thought that TRICS was "hung" Monday night a little after 7PM. They found Norm and to Norm TRICS looked "hung". So Monday night a little after 7 PM, Norm killed TRICS by "X" in the corrner of its command menu window and restarted just TRICS. He did not restart any of the other jobs, e.g. TrgMon or ADCMon. ------------------------------------------------------------------------------ DATE: 8-AUG-2005 At: MSU Topics: Documentation and Understanding of the Saclay BLS Signal Splitter. To make certain that we do not loose the documentation for the BLS Signal Splitter card from Denis/Saclay I started a directory for this stuff on our web site. It is the "splitter" directory under: www.pa.msu.edu/ hep/d0/ftp/run2b/l1cal/hardware/. So far it has the schematics from 2003 and a report about the testing of the Splitter card. From studying the schematic of the Splitter card it is now clear that the cables that carry the Trigger Tower signals from the Splitter out to the Side Walk Test Area need to swap the signals end for end in order to have the signals appear in the correct order out on the sidewalk. That is the Splitter to Sidewalk Cables need to look like: +---+ 4 Coax +---+ |O |--------------------------| | | | | | | | | | | | | | | |--------------------------| O| +---+ +---+ Last week Thursday evening I modified and then tested 3 of the 4 TT cables that are currently running from the Splitter to the Sidewalk so that they looked like this. I had previously modified the first cable. Now all 4 TT signals cables are setup correctly. I left all 4 of them plugged into the Splitter in the MCH and into the Patch Panel out on the Sidewalk. ------------------------------------------------------------------------------ DATE: 2:4-AUG-2005 At: Fermi Topics: Work on the M109-M110 water leak, work on CHTCR PROM Test error, HP - ADF - TAB Readout When the store ended at 7 PM Tuesday evening I turned off the L1 Cal Trig and investigated the water leak between M109 and M110. The leak is in the bottom heat exchanger of the 4 heat exchangers between these two racks. As normal, the leak is right in one of the copper pipes. It is in the short header pipe that connects the external supply line to the two parallel pipes that run through the heat exchanger. It is right at the edge of the lower end cap on this short header pipe. I installed a small funnel type thing made out of various size hoses to catch the water from this leak and route it to a container under the floor. Currently the leak rate is about 4 drops per minute. I hope to connect a pump to this and get this leak water to a drain. After this work, I re-installed all the G10 shields over the heat exchanger plumbing. Tentative plan - The goal is to run until October 31 with no lost beam time. - Run this way and see if things get worse. Typically the pin hole leaks in these heat exchangers get worse rather slowly. I have never seen one of these pin hole leaks get catastrophically worse in a short period of time. I.E. we should have plenty of warning before the water loss gets out of control. - Ask Russ to carefully watch the water level in the cooling system and keep us informed if the system starts to loose water more rapidly. Fully turn over to Russ the job of watching this. - If the leak gets out of control and we need to patch it then I think that we have a chance of doing so. - The leak is were the lower end cap joins the vertical cylindrical part of the header pipe. Thus if we disconnect the water lines and blow out all the water and dry this area we could pour into this area (with a hypodermic needle or something) a few cc's of epoxy or whatever. The needle would have to be the right length to reach in through the barb and input fitting and end right over the end cap of the header pipe. The intent is to fill up the end cap with epoxy (or whatever) thus covering the pin hole leak but not blocking the normal water flow path through the heat exchanger. We could pressurize the heat exchanger with air before the epoxy cures to try to force it into the pin hole. - This work could not be done between Stores. It would take a full day to do this work plus time for the epoxy (or whatever) to cure before water was put back into this heat exchanger. There is very limited access to this area of the rack - that's why this work would take so long. - There is some risk, i.e. lots of pushing and pulling is needed to get the 14 year old hoses off and back onto the barbs. All of this force is put directly onto the weak header pipes one of which could break and thus make the heat exchanger unusable. - I prefer not trying to run for a long period of time (weeks) with this heat exchanger disconnected. My concern is that something will run hot and fail and thus result in lost beam time. - In an emergency, e.g. catastrophic leak during a Store, we always have the back up plan of just pinching off the hoses with small "C" clamps and continuing the run. - Another idea we to pull a vacumm on the heat exchanger and suck the glue stuff into it. CHTCR PROM Test Tuesday evening right before the new Store can in I ran the L1 Cal Trig diagnostics and exerciser programs. 100k loops with zero errors but CHCTR PROM shows something funny (that it did not show last week). CHCTR shows something in eta -13:-16 phi 9:16 for EM Ref Set #1. The problem is in the First Pass PROMs. The typical error is that, "CHTCR output reads $4xy instead of $0xy", e.g. $404 instead of $004. The detailed log messages are: I$ Checking EM RefSet #0 Proms on CHTCR for Eta(-13:-16),Phi(9:16) I$ Using Tier #2 EM Et Ref Set #0 CAT2 for Eta(13:16) Op#3 @MBA#154/CA#12/FA# 6 I$ Scanning First Pass Prom # 0 I$ Scanning First Pass Prom # 1 I$ Scanning First Pass Prom # 2 I$ Scanning Second Pass Prom I$ Chan #233 of256 - Failed = 0 ... I$ Checking EM RefSet #1 Proms on CHTCR for Eta(-13:-16),Phi(9:16) I$ Using Tier #2 EM Et Ref Set #1 CAT2 for Eta(13:16) Op#3 @MBA#154/CA#14/FA# 6 I$ Scanning First Pass Prom # 0 E$ CHTCR Output=0x404 inst of 0x004 after rel Eta,Phi=0,5 set to 1 for EM_RS#1 E$_1stPass#0 @0x00001111000 Input FA_4=0x0f FA_5=0x00 FA_6=0x00 FA_7=0x00 E$ CHTCR Output=0x405 inst of 0x005 after rel Eta,Phi=0,3 set to 1 for EM_RS#1 E$_1stPass#0 @0x00001111001 Input FA_4=0x1f FA_5=0x00 FA_6=0x00 FA_7=0x00 E$ CHTCR Output=0x405 inst of 0x005 after rel Eta,Phi=0,3 set to 0 for EM_RS#1 E$_1stPass#0 @0x00011110100 Input FA_4=0x47 FA_5=0x01 FA_6=0x00 FA_7=0x00 E$ CHTCR Output=0x403 inst of 0x003 after rel Eta,Phi=0,3 set to 0 for EM_RS#1 E$_1stPass#0 @0x00101000010 Input FA_4=0x21 FA_5=0x02 FA_6=0x00 FA_7=0x00 E$ CHTCR Output=0x403 inst of 0x003 after rel Eta,Phi=0,3 set to 0 for EM_RS#1 E$_1stPass#0 @0x00100001100 Input FA_4=0x48 FA_5=0x02 FA_6=0x00 FA_7=0x00 E$ CHTCR Output=0x404 inst of 0x004 after rel Eta,Phi=0,2 set to 0 for EM_RS#1 E$_1stPass#0 @0x01010100001 Input FA_4=0x12 FA_5=0x05 FA_6=0x00 FA_7=0x00 E$ CHTCR Output=0x406 inst of 0x006 after rel Eta,Phi=0,2 set to 1 for EM_RS#1 E$_1stPass#0 @0x01011010011 Input FA_4=0x35 FA_5=0x05 FA_6=0x00 FA_7=0x00 E$ CHTCR Output=0x404 inst of 0x004 after rel Eta,Phi=0,4 set to 0 for EM_RS#1 E$_1stPass#0 @0x01000110100 Input FA_4=0x46 FA_5=0x04 FA_6=0x00 FA_7=0x00 E$ CHTCR Output=0x403 inst of 0x003 after rel Eta,Phi=0,3 set to 0 for EM_RS#1 E$_1stPass#0 @0x01000011000 Input FA_4=0x0c FA_5=0x04 FA_6=0x00 FA_7=0x00 E$ CHTCR Output=0x405 inst of 0x005 after rel Eta,Phi=0,3 set to 0 for EM_RS#1 E$_1stPass#0 @0x11000101010 Input FA_4=0xaa FA_5=0x04 FA_6=0x00 FA_7=0x00 E$ Test Aborted for First Pass Prom # 0 E$ First Pass Prom # 0 Failed the test I$ Scanning First Pass Prom # 1 B$CL1ct_TrgTwr::f_okRefCmp: Illegal Parameter E$ CHTCR FA_5=0x00 but bit#7 should be 1 as rel Eta,Phi=1,0 EM_RS#1 Thr=0.000 E$ CHTCR Output=0x000 inst of 0x001 after rel Eta,Phi=1,0 set to 1 for EM_RS#1 E$_1stPass#1 @0x00000000001 Input FA_4=0x00 FA_5=0x00 FA_6=0x00 FA_7=0x00 B$CL1ct_TrgTwr::f_okRefCmp: Illegal Parameter E$ CHTCR FA_6=0x00 but bit#0 should be 1 as rel Eta,Phi=2,7 EM_RS#1 Thr=0.000 E$ CHTCR Output=0x000 inst of 0x002 after rel Eta,Phi=2,7 set to 1 for EM_RS#1 E$_1stPass#1 @0x00000000011 Input FA_4=0x00 FA_5=0x00 FA_6=0x00 FA_7=0x00 B$CL1ct_TrgTwr::f_okRefCmp: Illegal Parameter E$ CHTCR Output=0x000 inst of 0x001 after rel Eta,Phi=1,0 set to 0 for EM_RS#1 E$_1stPass#1 @0x00000000010 Input FA_4=0x00 FA_5=0x00 FA_6=0x00 FA_7=0x00 B$CL1ct_TrgTwr::f_okRefCmp: Illegal Parameter E$ CHTCR FA_6=0x00 but bit#1 should be 1 as rel Eta,Phi=2,6 EM_RS#1 Thr=0.000 E$ CHTCR Output=0x000 inst of 0x002 after rel Eta,Phi=2,6 set to 1 for EM_RS#1 E$_1stPass#1 @0x00000000110 Input FA_4=0x00 FA_5=0x00 FA_6=0x00 FA_7=0x00 B$CL1ct_TrgTwr::f_okRefCmp: Illegal Parameter E$ CHTCR FA_5=0x00 but bit#7 should be 1 as rel Eta,Phi=1,0 EM_RS#1 Thr=0.000 E$ CHTCR Output=0x000 inst of 0x003 after rel Eta,Phi=1,0 set to 1 for EM_RS#1 E$_1stPass#1 @0x00000000111 Input FA_4=0x00 FA_5=0x00 FA_6=0x00 FA_7=0x00 B$CL1ct_TrgTwr::f_okRefCmp: Illegal Parameter E$ CHTCR Output=0x000 inst of 0x002 after rel Eta,Phi=2,7 set to 0 for EM_RS#1 E$_1stPass#1 @0x00000000101 Input FA_4=0x00 FA_5=0x00 FA_6=0x00 FA_7=0x00 E$ Test Aborted for First Pass Prom # 1 E$ First Pass Prom # 1 Failed the test I$ Scanning First Pass Prom # 2 B$CL1ct_TrgTwr::f_okRefCmp: Illegal Parameter E$ CHTCR FA_7=0x00 but bit#2 should be 1 as rel Eta,Phi=3,5 EM_RS#1 Thr=0.000 E$ CHTCR Output=0x000 inst of 0x001 after rel Eta,Phi=3,5 set to 1 for EM_RS#1 E$_1stPass#2 @0x00000000001 Input FA_4=0x00 FA_5=0x00 FA_6=0x00 FA_7=0x00 B$CL1ct_TrgTwr::f_okRefCmp: Illegal Parameter E$ CHTCR FA_7=0x00 but bit#3 should be 1 as rel Eta,Phi=3,4 EM_RS#1 Thr=0.000 E$ CHTCR Output=0x000 inst of 0x002 after rel Eta,Phi=3,4 set to 1 for EM_RS#1 E$_1stPass#2 @0x00000000011 Input FA_4=0x00 FA_5=0x00 FA_6=0x00 FA_7=0x00 B$CL1ct_TrgTwr::f_okRefCmp: Illegal Parameter E$ CHTCR Output=0x000 inst of 0x001 after rel Eta,Phi=3,5 set to 0 for EM_RS#1 E$_1stPass#2 @0x00000000010 Input FA_4=0x00 FA_5=0x00 FA_6=0x00 FA_7=0x00 B$CL1ct_TrgTwr::f_okRefCmp: Illegal Parameter E$ CHTCR FA_7=0x00 but bit#4 should be 1 as rel Eta,Phi=3,3 EM_RS#1 Thr=0.000 E$ CHTCR Output=0x000 inst of 0x002 after rel Eta,Phi=3,3 set to 1 for EM_RS#1 E$_1stPass#2 @0x00000000110 Input FA_4=0x00 FA_5=0x00 FA_6=0x00 FA_7=0x00 B$CL1ct_TrgTwr::f_okRefCmp: Illegal Parameter E$ CHTCR FA_7=0x00 but bit#2 should be 1 as rel Eta,Phi=3,5 EM_RS#1 Thr=0.000 E$ CHTCR Output=0x000 inst of 0x003 after rel Eta,Phi=3,5 set to 1 for EM_RS#1 E$_1stPass#2 @0x00000000111 Input FA_4=0x00 FA_5=0x00 FA_6=0x00 FA_7=0x00 B$CL1ct_TrgTwr::f_okRefCmp: Illegal Parameter E$ CHTCR Output=0x000 inst of 0x002 after rel Eta,Phi=3,4 set to 0 for EM_RS#1 E$_1stPass#2 @0x00000000101 Input FA_4=0x00 FA_5=0x00 FA_6=0x00 FA_7=0x00 E$ Test Aborted for First Pass Prom # 2 E$ First Pass Prom # 2 Failed the test E$ Test Failed for EM RefSet #1 Proms on CHTCR for Eta(-13:-16),Phi(9:16) I$ Chan #234 of256 - Failed = 1 ... I$ Checking EM RefSet #2 Proms on CHTCR for Eta(-13:-16),Phi(9:16) I$ Using Tier #2 EM Et Ref Set #2 CAT2 for Eta(13:16) Op#3 @MBA#154/CA#16/FA# 6 I$ Scanning First Pass Prom # 0 I$ Scanning First Pass Prom # 1 I$ Scanning First Pass Prom # 2 I$ Scanning Second Pass Prom I$ Chan #235 of256 - Failed = 1 ... The details of the error messages would change from run to run, e.g which rel eta,phi was changed just before the read with error, but all bad reads were of the form, $404 instead of $004. I think that these are reads of the CHTCR output as read by the Tier 2 CAT2 input operand readback. Try moving the backplane paddle card. Try changing the Tier 2 CAT2. This did not fix the problem. Try changing the CHTCR, pull CHTCR SN #28 from eta -13:-16 phi 9:16 and install CHTCR SN #??, a spare from the cabinet with its excess parts removed except for 4 of the 29525s that I forgot to pull (see log book 15:16-JAN-2004 ). This did not fix the problem. Tried changing the CTFE PAL for: rel eta,phi 0,3 i.e. phi 12 Ch 1 rel eta,phi 0,2 i.e. phi 11 Ch 1 rel eta,phi 2,7 i.e. phi 16 Ch 3 This did not help fix the problem - but it did have the doors open and the power supply turned On/Off many times. Wait, OK an input operand value read back from the CAT2 when used with a CHTCR can not be $4xy. The CHTCR only sends 6 bit data to the T2 CAT2. The upper 6 bits of each oprand are just held a zero by the bias networks. This must be just two flakey CAT2's in a row. The 12 bit CAT2 operands are held in 2 socketed 6 bit latch chips. So pull out the high order latch chip and let the 50k Ohm pull downs hold the inputs of the down stream chips at Vee for the high order operand bits. The CHTCR PROM Test calls this CAT2 operand #3 but I assume it is counting from 0 and the CAT2 documentation counts its operand from 1. So pull out U37 and U39. This is from the CAT2 card that was originally in this Tier 2 locations which I now put back into the running system. OK now the CHTCR PROM Test runs OK. Now run lots of tests: CHTCR PROM Test x x x x CTFE Eng PROM Test x x CTFE Pt PROM Test x 1 Meg Loops of Exerciser x x x x ADF TAB Readout Test Thursday morning we did a readout test with the Hand Pulser injecting a signal in front of the Splitter so that both the ADF - TAB and the current L1 Cal Trig could see it. We moved it from one TT to the next turning on EM and HD separately. We had to move the MCH to SW cable while doing this because only one of the 4 cable sections was working correctly. The signals in the other 3 sections were running backwards. I can fix this by flipping the connector at one end of the cable (which I had done to one of the 4 sections and have now done to all 4 sections). I had though that this was caused by the cables being made backwards. The other thing that could cause it is if Denis ran the signals backwards to the SW connection on the Splitter. For now all 4 sections of the cable are setup so that the signals look OK. NOTES: In the TRICS log window, Buffer Depth normally says "0" and sometimes "1" or more during normal Global Physics Beam running. Holding the Log Window with the vertical scroll position control so that you can read something in the log file that has gone off the top of the screen clearly appears to lock up something and block other activities from going forward. Started a new TRICS log file. ------------------------------------------------------------------------------ DATE: 1-AUG-2005 At: MSU action at Fermi TOPICS: L1 Cal Trig Testing Between stores check that the L1 Cal Trig is happy and go through the full cold start procedure. Run the exerciser on the Run IIA L1 Cal Trig: run 1 passes of the CHTCR PROM checker It does not like EM Ref Set #1 at eta -13:-16 phi 9:16 This may be the standard EM Ref Set #1 problem, see for example 29-MAR-2005. Need to check in detail on Tuesday when I get there. run 1 passes of the CTFE EM HD PROM checker run the exerciser on 1:16 +- 1:32 EM/HD Ref Set 0:3 EM HD HDVeto check global counts runs: 2x 100k The exerciser see no problems. ------------------------------------------------------------------------------ DATE: 31-JULY-2005 At: Wayne action at Fermi TOPICS: Water Leak Was called at about 2:15 in the afternoon because the L1 Cal Trig had a water leak between M109 and M110. Norm and George came in to take care of getting things going again. The leak is a couple of drops a minute and it appears to be captured by the mud flaps. They moved the drip detector strip and cranked things up again. This was all during a Store. Beam lost about 2 hours. ------------------------------------------------------------------------------ DATE: 27:29-JULY-2005 At: Fermi Topics: Check L1 Cal, Test GAB AOTs. Remember that the d0mino machines that one can use now are: d0mino01 d0mino02 d0mino03 Run the exerciser on the Run IIA L1 Cal Trig: run 2 passes of the CHTCR PROM checker run 1 pass of the CTFE Px Py PROM checker run 2 passes of the CTFE EM HD PROM checker run the exerciser on 1:16 +- 1:32 EM/HD Ref Set 0:3 EM HD HDVeto check global counts run: 1k, 1M, 1M, 1M All looked OK. Check the And-Or Term cable from the GAB by plugging it into the TFW. It looked OK on the scope, i.e. Strobe is OK, Gap is OK, and they are generating an And-Or Term on D15 of the cable which is asserted for Tick #7 and looks OK. The TFW could not lock onto the GAB signals. Look more and their GAP signal is only 16 Ticks long instead of the 17 Ticks long Sync Gap signals sent on the SCL. I do not know if the first or last ticks worth of the GAB Gap signal is missing. Mike worked on the GAB firmware and then its Sync Gap signal looked OK. Tested this by having GAB generate a L1 Term signal that was asserted for one tick on each turn. By hand setup TFW to look at this term from GAB and AND it with a Tick Select Term. Saw the correct operation when they were both asserted on the same tick or offset by +-1. Added the web director for the SCLD stuff. So far this is just the SCLD information from Denis and Saclay. It is in: www.pa.msu.edu/hep/d0/ftp/run2b/l1cal/hardware/scld/ Started a new TRICS log file. ------------------------------------------------------------------------------ DATE: 13:15-JULY-2005 At: Fermi Topics: Meeting about monitoring, Vertical Interconnect to Readout crate d0tcc3 + borrowed spare L1FW bit3 card Integrating TAB software into L1Cal_II software Specification for COOR, MSU inventory stickers Meeting with Darien, Suharyo Sumowidagdo, Adam Roe, Geoff Savage. Adam is a 6-month "summer student" working with Darien and is being asked to work on a monitoring client for L1Cal data. We had a discussion about serving monitoring data, about pedestals, about monitoring events (from L1 accepts and forced) vs examine events (passing L3). They were shown the current TT_ADC_mon and the difference between single sample and 7-fold average. Geoff encourages everybody to think of anything else we need from the online group. In particular, we can report error conditions to the SES for the purpose of notification, stopping run, archiving occurences, etc. I don't know of anything we can cleanly detect and report, but we should keep this in mind. Haryo will help with the database aspect of adding the L1Cal power supplies to the Epics database. Geoff is currently working on an interface between CANbus and 1553. We can then use the current python code displaying L1Cal power supplies to display the new items by just writing a new simple configuration file. Adam was assigned to do the display as well, but he will have little to do, and we may just prefer to keep control of this, as there will be nothing new to do. Vertical Interconnect to Readout crate The goal is to let the windows PC setup the Readout crate via easy to understand command files rather than Excel. Removed the front panel of the vertical interconnect slave (the one which was sitting unplugged in the communication crate). Not sure of an apropriate temporary/permanent home for this front panel and left it on top of the sun. From reading our setup guide for the VI slave, I think it is only needed to remove jumper J15 to stop the module from being the VME arbitrer. I was never successful in doing VME IOs in the readout crate. The A32 VME address in the communication crate to do remote IOs through Master#1/Slave#0 is 0x 1400 0000 and one can see the VI slave LED flash, so this must be right. Tried running the Setup_Readout_Crate.cmd that Philippe prepared for this, and also tried manual read/writes. The VME cycles would often time out, but sometimes not, but the automatic reading back of the VME addressses would read mostly 0xcccc. This was first tried with the Interconnect Slave in an upper slot (~#20), past all other modules, then again in slot #2 as controller, then in slot #1, then even in slot #1 with the SBC and VME/SCL pulled out and only the VRB, VRBC and FIC card. Note: I am not sure how or if the SBC knows not to be the crate controller/arbitrer. Slot #1 is the 618 (with 9U extension) and must be set to be the crate arbitrer. Tried also pulling out the wirewrap modification to pull down BG1. This vertical slave was tested (again) in an upper slot of an ADF backplane, and was able to set and read one of the BC registers. Double-checked also that these SBC (short IO space), VRB, and VRBC VME registers can indeed be read back in the IIa readout crate. The Vertical Interconnect was left in the crate after the last attempt, I believe this is in slot #6. d0tcc3 + borrowed spare L1FW bit3 card Work with Stu on installing bit3 software V2.3.0 onto d0tcc3. There was no problem compiling the driver on this newer release of linux. We installed the PCI module of the spare Model 617 interface into d0tcc3. This is the L1FW spare located in the yellow-blue rack accross from Dan's "office space". We need to use an optical Model 618 on the sidewalk for ground isolation, but the Model 617 copper interface is identical from a user point of view. It was not clear how to make linux automatically start the driver at boot time. After some unsuccessful attempts at adding information about the driver into the standard /etc/modules.conf file for module dependencies, we just added a line into the local startup file /etc/rc.d/rc.local: insmod /lib/modules/btp.o The other issue in the way was some aparent incompatibility with ssl that caused the initialization of python's XML parsing software to fail. This no longer occurs, and I am not sure why. There are no more known issue in the way for using d0tcc3 on the sidewalk. The L1Cal_IIb software now compiles and runs on d0tcc3, and just complains about "remote crate off, or cable disconnected" which is the expected response, and proves that the user application does indeed communicate with the driver. Stu says that leaving the box where it sits now (in the 2nd floor rack) for another few weeks is not a problem and is not in the way. When we move it, we need to plug it directly into the wall plug #42 located on the north wall just accross from the sidewalk racks. This is the online network, and we patched this ethernet on the second floor to make it live and ready to go. None of the other hubs on the sidewalk benches are connected to the online network. We should plug into the building ground for AC power. We need to use an optical bit3 to the isolated sidewalk equipment. Integrating TAB software into the overall L1Cal_IIb control software Philippe has a set of source code files that Mike uses to write his dedicated test programs. These files are devided into several categories: 1) test programs that setup vme access and call the subroutines 2) somewhat card level routines to manipulate card resources 3) mid level IO routines to handle the VME/SCL serial IO 4) low level VME access routines to handle the bit3 interface L1Cal_IIb_TCC can get a) examples/inspiration from category 1 b) include and use categories 2 and 3 at the source code level c) cleanly replace category 4 with its own vme access code which is advantageous because: - the nevis bit3 code was derived from examples - we benefit from the MemoryMap/API/Fake/NoIO switches to be able to run development code without hardware. For a first attempt, two tests were chosen. A simple test is to set the VME/SCL Online/Offline and watch the result by reading the VME/SCL status. A more complex test is to configure the 10 Altera Stratix FPGAs on a TAB card. From Windows: There were no real issue in compiling the tab/gab code on windows. In order to not (or least) modify the source code, an emtpy unistd.h file was added. This was tested on the sidewalk by swapping the bit3 optical cables and connecting the windows PC to the readout crate which currently hosts the VME/SCL card. There was first a problem with the bit3 memory mapping, and a bug was found in the Nevis software which generates 32bit VME addresses (with upper byte set to 0xff) while accessing A24 addresses. I am not sure why this doesn't affect the Nevis software (or is it the linux version of bit3 software that ignores it?) Configuring the stratix FPGA was not a clear success, but Mike had to go to a meeting. It all seemed to proceed and complete normally, but I don't think there is a clear report of configuration success/failure. We were getting a single error message while configuring all 10 FPGAs. something like "stritix #0 pll error", and another time it was complaining about stratix #1. Mike says this is a sign of failure. The procedure they use is to power down and power up the stratix chips before they get configured. This is done over VME and has been included to happen automatically in later versions of L1Cal_IIb software. I noticed that the tab software was using a 'usleep' function to wait 100 microseconds at times, and that function does not seem to exist in the standard windows run time library. I had made a replacement using the thread_util::delay function, but I don't know how it behaves on intervals smaller than the scheduler interval of 10 ms. Maybe that was the source of the problems. I made it wait a minimum of *****. This hasn't been tried. From Linux: I think we should be able to run this same test of control software on the "tbilisi-clued0" node they use. I don't think we need to build the software on tbilisi directly. We can build on d0tcc3 using a d0 release common between the online and clued0 clusters (here p17.05.00) and copy the executable and shared objects (while xerces needs to have been built on d0rel3 if needed at run time). I was not able to converge on this until late afternoon Friday, and the Columbia people were gone by then. We can try this remotely. Specification for COOR Had several conversations with Mike towards defining control messages for COOR, and towards defining an interface between overall control code (i.e. MSU/Philippe) and tab/gab specific control code (i.e. Nevis/Mike). There are some documents from which one can imply the tower level TAB resources, but we found very little to describe the GAB level final cuts and And/Or Term definitions. Mike is somewhat familiar at that level with the TAB resources, and not yet acquainted with the goals for the GAB. This is going to take some effort before we can converge. Philippe will start this process and will start written discussion with Mike and Hal. MSU inventory stickers The MSU property stickers were affixed to the 4x ADF-2 Wiener crates. There are presently 2x crates on the sidewalk, 1x crate in a rack in the technician room directly east of the machine shop in the high bay area, and 1x crate pulled out of its rack and sitting near the middle of FCH3 on the north side (I think Geoff is practicing CAN-bus with it). MSU # Wiener S/N 017721 5196041 017734 5196039 017747 5196040 017760 5196038 Software on Sidewalk PC The TCC engine is now V3.0.B The previous version was V2.3.B and the executable is still there. The GUI client is now V4.0.B The L1Cal software was also updated to use the officially assigned IP port numbers. The V4 GUI is compatible with V3 TCC Engine, but not with V2.x TCC engine. If it becomes necessary to back up to the V2.3.B TCC executable, one needs to switch two files in C:\L1Cal_IIb\L1Cal_IIb_Gui. First rename the two Itc_***_Client.py files to Itc_***_Client.py_new_ports, then rename the two files Itc_***_Client.py_old_ports to Itc_***_Client.py, and restart the GUI. The only difference is that the V4 GUI uses IP ports #52345 (and 52346) for the Coor (and Gui) access to the TCC executable while earlier versions were using #52160 (and 52164). ------------------------------------------------------------------------------ DATE: 7-JULY-2005 At: MSU Topics: L1Cal VRBC SCL lost synch Control Room calls (Miroslav captain): "L1Cal missing Events in L3" Shifters had tried restarting the SBC in L1Cal crate Intuition: VRBC has lost SCL signal lock. Bill Lee confirms missing amber LED. Manually running Init_Post_Auxi_L1CT.vio fixed it. This had not happened in a long, long time. Note that there was Run IIb testing activity on the sidewalk at the time. ------------------------------------------------------------------------------ DATE: 28-JUNE,1-JULY-2005 At: Fermi Topics: Gave TFW shifter talk, Check hot Towers again, Worked with Mike on Checking the And-Or Terms from the GAB, Worked with Jorge on ATTM-TWG at D-Zero. Between Stores I used the "check_l1ct_setup_8_single_tt.msg and .cio files in /Scratch files to look at the rate from the currently Excluded TT's. Currently 10 channels now actually excluded. Nothing looked bad. All rates between stores at a 3 GeV Tot Et threshold looked OK. The current Excluded List is: +6,23 EM -4,25 HD +9,31 HD +4,11 HD +4,25 EM -6,25 EM -10,25 HD +10,12 HD +2,2 EM. +4,25 HD This week Dean has replaced the BLS summer for -6,25 EM with one that has a cut resistor for the Calorimeter eliment that they know is noisy. They see noise on this channel in the precision readout. Not known if this is in the Cal or a preamp problem. It is also not clear how many of these Excluded channels still need to be Excluded. Norm will be gone for a week and then after everyone is back we may stop excludeding any TT's and see how it goes. In any case we should be able to stop excluding -6,25 EM. Working with Mike we looked at the And-Or Terms from the GAB. All that we see is the Strobe signal on their cable. We checked a number of the And-Or Term outputs from the GAB and all the same - just the Strobe clock signal. I ran a And-Or Term 40 conductor cable from the sidewalk to the MCH-1 TFW. Jorge brought the ATTM-TWG to D-Zero. Philippe installed the necessary software to run it. The intent is to be able to test the components in the BLS Cable Transition system. We now have the card file mechanical components installed in the back of the backplane in the 4 ADF crates. In general this looks OK but it also looks like there is about a 1 mm gap in the connectors, i.e. that the ADF Transition Cards should slide in an extra 1 mm (or be 1 mm longer or that their card file mechanics should be 1 mm closer to the back plane). Both ADF crates that are out on the sidewalk now are running with ATD cards and their ADF card is in slot #11. The SCLD signals are going through the ATC and it looks OK. Jason confirms this with tests at MSU. So we have checked the BLS signals, the Channel Link signals, and the SCLD signals through the ATC. I sent out sign off on the ATC to Linda and company. ------------------------------------------------------------------------------ DATE: 25-JUNE-2005 From: MSU Action At: Fermi Topics: Exclude +4,25 They paged Saturday morning and wanted +4,25 both EM and HD excluded. They had been living with high rates for about 2 hours and it did not look like it was going to "fix itself". Order of magnitude 1/3 of the L1 rate was from +4,25. They asked for both EM and HD to be excluded. I did not see plots to support that. ------------------------------------------------------------------------------ DATE: 15:17-JUNE-2005 At: Fermi Topics: L1 Cal Trig tests, Noisy TT tests, Patch Panel and ATC tests, Run the various tests on the L1 Cal Trig. 2 passes of CHTCR prom check, 2 passes of EM-HD prom check on page 7, 2 passes of Px-Py prom check on page 2. All prom checks are over +- 1:16 1:32. Then run the system exerciser over +- 1:16 1:32 1k, 10k, 1 meg, 1 meg. All looks OK. There currently are 8 channels Excluded. They are: +6,23 EM -4,25 HD +9,31 HD +4,11 HD -6,25 EM -10,25 HD +10,12 HD +2,2 EM. Between stores look at these with single TT Ref Sets. The one that is clearly in trouble is -6,25 EM. With a threshold of 5 GeV it looks OK but with a threshold of 3 GeV it is running at 10 kHz. It runs about 1/2 of the time and is quiet about 1/2 of the time. The other 7 of these Excluded channels actually looked OK today. In the scratch directory there are setup eight_single_tt .msg and .cio files for setting up the L1 Cal Trig to look at and record noise events from these Excluded channels. The .cio file "un-Excludes" them. We have a final version Patch Panel Card and a final version ATC cards. In the sidewalk test stand Alan's student Cosmin Dragoiu tests all 32 channels and it looks OK. I'm bringing these cards and two final version Pleated Foil Cables back to MSU for Channel Link testing and Stefano TWG JJB testing. ------------------------------------------------------------------------------ DATE: 1:4-JUNE-2005 At: Fermi Topics: CJT(2,3) problem, Optical SCL to mini_PDT, VI's for ADF and Readout Crates, BLS Signal Splitter, Things brought to Fermi D-Zero this trip: L1Cal TCC, ADF-2 cards for rack M106 SN# B5:B16 and SN# C6:C13 and the ADF-2 cards for rack M111 SN# D13:D32. Return to Fermi D-Zero the full ADF Crate that was used at MSU as the system test crate, the spare SDF Crate Power Supply, and the spare ADF Crate Fan Tray. All of the ADF-2 cards and the spare power supply and spare fan tray are locked in Lin Bagby's cabinet on the sidewalk. Work with Dean on what is called the, "CJT(2,3) problem". Based on what had been worked on right before this problem started, Dean guessed that the noise was coming from the middle BLS crate in either rack PS00 or PS02. In terms of TT's these map to: PS00 Middle BLS Crate is eta +7:+14 @ phi 10 and eta +11:+20 at phi 11 PS02 Middle BLS Crate is eta +7:+14 @ phi 14 and eta +11:+20 at phi 15 Looking on the scope at the CTFE Lemo Monitor connector you can see the noise. It is by far the strongest in BLS signals from PS00 and you can see it in PS02 at about 1/4 of the amplitude that it is in PS00. In other BLS crates you can just barely barely see a hint of switching noise at the same points in time. A typical BLS signal from PS00 has a negative bump of about 40 mV that is about 50 usec wide followed about 290 usec later by a positive bump of 40 mV also of about 50 usec width. This happens during the L2_Acpt and Digitization sequence on the BLS card. This probem was taken care of by replacing the BLS power supply. Then a big problem Friday afternoon of a blown fuse in the Cal pre-amps that required opening the detector to fix. Install the equipment to run Optic SCL to the mini_PDT test stand in DAB-1 clean room East. This is Geographic Section $3F which is non-readout. This has been fully installed (both Optical SCL to mini_PDT and optically isolated SCL Status from mini_PDT) but it has not been tested. They need to setup their test stand for normal running and not standalone running. Get 2 VI slaves and 1 VI master. They all appear to be working OK. Setup all 3 slaves as per the ADF-2 crate notes document. Setup the dip switch on the masters are follows: Master SW1 Key Number --------------------- 1 2 3 4 5 6 - - - - - - 0 0 0 1 0 0 VI Master for the ADF-2 Crates 0 0 0 1 0 1 VI Master for the Readout Crate Both VI masters are plugged into the communications crate. I tested all 3 VI slaves. All of these VI cards are working. I started running the 2nd ADF Crate out on the sidewlk. It is slave #1 on master #0. VME access to it is working fine. It has one ADF-2 card in it. The paper slot labels are on both of the ADF Crates on the platform. I ran RRT that included registers in both crates. Repaired the problem with the BLS signal splitter. The problem was the power supply - not the splitter card. I now have it running on an un-cased "ugly" type open frame supply that can make 12 Amps. The splitter cards appear to take about 2 Amps +. While working on it I went ahead and installed on of Denis' now final design splitter cards. This is in TT's -9,0 -10,0 -11,0 -12,0. ------------------------------------------------------------------------------ DATE: 18:20-MAY-2005 At: Fermi Topics: Deliver ADF-2 cards to Fermi. Start running T7_Phy at Fermi, SCLD_T3 at Fermi, BLS Splitters, Wiener orders. Brought to Fermi: SCLD SN #2, ADF cards for crate M104 SN A1:A16 and B1:B4, ADF cards for crate M109 SN C14:C21 and D1:D12. Return to MSU: SCLD SN #1. Pull SCLD SN #1 out of the Communications Crate. Pull the normal SCL Receiver from SCLD SN #1 and put it on SCLD SN #2. Put SCLD SN #2 into the Communications Crate. SCLD SN #2 is running firmware SCLD T3. The SCL Receiver is SN #2?? and is labeled 24-MAY-2002 5980A9. It is not a 10 nsec delay SCL Receiver. Now measure the time betwee the peak of a TT eta -9 EM BLS signal and the beginning of the Live_BX signal in the ADF-2 Crate Backplane and find it to be about 360 nsec. By Thursday the BLS signal Splitter was not working correctly and was removed. It had been in TT Eta -9:-12 TT Phi 1. It's power LED's were flashing On and Off at a couple of Hz. Cooked DC/DC converter ? Put the Stand Alone SCL Receiver onto SCLD SN #1 and return it to MSU. On this trip 40 ADF-2 cards were delivered to Fermi. They are: ADF cards for crate M104 SN A1:A16 and B1:B4, ADF cards for crate M109 SN C14:C21 and D1:D12 Neither shipping container has been opened. Both shipping containers are now locked in a cabinet under Lin Bagby's control. I have installed the ECO #1 capacitor C1354 on the two ADF-2 cards that have been here at Fermi for sometime. They are cards SN #C3 and SN #C5. Start running T7_Phy as the standard version of Data Path firmware at D-Zero. Sidewalk TRICS is updated to support T7_Phy and more ADF to TAB test patterns are made available in the command files. Meeting to finalize the order for the Wiener Communications Crate and the TAB GAB Wiener power supplies. ------------------------------------------------------------------------------ DATE: 3:6-MAY-2005 At: Fermi Topics: Review Documents for Run2B Cal Trig, BX_X8_Clock during Tev Ramp, Opto-Coupler of Dean 5k test stand SCL Status, SCL signal to HeartBeat Trigger card on MCH-3, Startup of after Power Outage, Work on L1 Cal Trig -13,2 Check TFW -2V, BLS Splitter, BX to L1_Acpt Spread timing change, safety document reference, BLS signal splitter is back in. Reviewed 6 sheets of documentation about the ADF Transition card and send a written comments to John. Reviewed BLS cable layout plan and confirmed there there were errors in the current plan. Measured two BX_X8_Clocks on infinite persistence during the RAM down and the Ram up. Full width of jitter remains 1.0 nsec. Can not tell the triggered trace from the just observed trace. Installed the cables and opto coupler stuff for the Cal 5k Test Stand SCL Status signals, i.e. G.S. $4E and $4F. Looked at the SCL Coax cable to the Heartbeat Trigger card up on MCH-3. The cable looks OK. Dig around for 2 hours trying different SCL Hub-End outputs and different patch cords and checking with the 112 Ohm target. Give up and carry the test receiver up to MCH-3. It works fine. So swap SCL Receivers on the HeartBeat trigger card and then it works. So I expect that the SCL Receiver that they had been using either has a bad connector (so that it works on some cables and not others) or that it has a zapped input so that it now has a low sensitivity. It appears to be working fine with its new SCLD Receiver. Now I have only a single (non delayed) SCL Receiver in the spares box. Checked the signals on the SCL Status cable from the HeartBeat Trigger card and the Status signals look fine. The HeartBeat Trigger card is G.S. $6F and runs on Spare #2 cables to MCH-3. We had a scheduled power outage Thursday Morning at 5:30 AM to change some parts in a D-Zero safety system. I did the shutdown without a problem. At startup it took about 1 hour to get the Master Clock to load up and start running. Both sequencers kept saying "Halted". It's very hard to tell what is going on. One can not directly see or control any registers, it is all just click and pray gui. After a while it worked just fine. The counts during FPGA Configuration (and the steps) are: Master Command Configure All: 1702 with 0 errors Configure Command M101 Routing Master: 51 with 0 errors Configure Command M101 Quad Terms: 17 with 0 errors After power outage before turning on the L1 Cal Trig I looked at -13,2 which has shown Tot Et Ref Set #1 problems in the Cal Trig exerciser. Pulled -13,2. It looks fine. Can not spot any problems with the Ohm meter in the vicinity of -13,2 Tot Et Comps. As a random guess I replaced the CTFE #1 PAL. It is in the path of the -13,2 Tot Et Comp output signals. -13,2 now appears to be OK. Test CHTCR PROMs 1:16 1:32 RS 0:3 EM & HD 0 errors EM and HD on page 7 0 errors Px and Py on page 2 0 errors L1 Cal Trig System Excerizer 1:16 1:32 page 7 EM & HD Trees RS 0:3 EM Tot HD_Veto Check Glb Counts 100k loops zero errors then 1 mega loops zero errors The TFW Power Supply Voltage Monitoring has shown that M122 ABC is a little bit high. It reads about 2.190 to 2.200 Volts and is in Hi alarm (but not Hi Hi Alarm). I checked the actual supplies and the actual voltage on the backplane bus in M122. M122 Top power supply 2.140 V on the crate bus 2.005 V M122 Mid power supply 2.123 V on the crate bus 2.025 V M122 Bot power supply 2.110 V on the crate bus 2.025 V So this looks like a Shea Box RM problem and not an actual power supply problem. Put the BLS Signal Splitter back into TT eta -9:-12 TT phi 1. Look at the signals out on the Sidewalk. The safety review requirements document is at: www-ppd.fnal.gov/EEDOffice-w/Infrastructure_group/Reviews/ Electrical_Safety_Review.htm Talking with Darien and Al I learned that the BX to L1_Acpt timing is supposed to change for Run 2B. This is to allow time for the Track Cal match. In theory this was approved by everyone 2 years ago or something like that. I did not remember this at all. So the firmware will need to be made to move the TFW Spread. I don't know is this is going to be tested before October 3rd, or if we will come out of the shutdown the way we are now and then move. The current L1 Cal Trig should not need to move. ------------------------------------------------------------------------------ DATE: 20,22-APR-2005 At: Fermi Topics: L1 Cal and TFW Power Supply Monitoring, SMT Heartbeat, +2,2 EM On this trip bring to D-Zero: serial number labels for ADF-2 SN# C3, 10 fuses for the contactor boxes 20 Amp, 10 of the SMB 90 degree connectors for LMR-200. We thought that we had a problem with L1 Cal Trig power supply L1CAL_LV_M110B/45VV. It went out of tollerance for a couple of seconds Monday morning and again at about 12:20 on Wednesday. The Physics run pausing alarms on Monday and today Wednesday afternoon from M110B -4.5V really apprear to be a monitoring problem. All 56 supplies monitored by the Shea Box remote terminal 26 went to about zero Volts for just one second and then all returned to being OK. The Significant Event Server logs are at: /online/log/ses/logs/ They are listed by their start date (I think). This is in their filename. The time stamps are in seconds since 1970 (linux standard). There is a routine to convert this number. The assumption is that this is a RM (Shea Box) problem and John will take care of this. Remember, for now to spin use the following: spin -rn www.pa.msu.edu -rd temp filename Thurday afternoon I dug around under the floor on MCH-3 and found the spare SCL cables. I have routed the SCL cables with the label MCH-3 Spare #2 over to rack M300. Both the SCL serial data coax cable and the SCL Status cable have been connected at the SCL Hub-End on the first floor. I have sent a note to Scott Snyder asking that Geographic Section 0x6f be added to the COOR Resource file. Geographic Section 0x6F Crate Name smt_hb Location M300-3 Type Non-Readout I have added Geographic Section 0x6F to the D-Zero Crate List file at. Friday early AM when the Store went in the rates from L1 and L2 Cal Trig were wrong and very high. The TRICS L1 Cal tools List TT's Above Ref Sets and List Hot TT's showed the problem to be +2,2 EM. The Shift Captain asked that +2,2 EM be Excluded. The Excluded TT's File was edited to make +2,2 EM Exclusion persistent. ------------------------------------------------------------------------------ DATE: 29-MAR-2005 At: Fermi Topics: Repair of two Tier 1 Power Pans, 1-APR-2005 Optical SCL to Cal 5K Test Stand, L1 Cal PROM and System Tests, Install SMT "Delayed" L1_Buzy Status Concentrator, FPD AOTs, Peds in two CTFEs, Contactor Box plugs and fuses, PowerTec supply repair, Rev 2 BLS only Paddle Brd Spare Bricks at D-Zero before any of Power Pan repair work was started this week: Powertec 5V 600A MSU SN #16 11-JUNE-1991 # just back from repair in early March 2005. Pioneer 5V 200A MSU SN #97 7-JAN-1992 # Factory Refurbished Pioneer 5V 200A Factory SN #416254 ?-?-1992 # Factory Refurbished Pioneer 5V 200A MSU SN #67 7-JAN-1992 # MSU 6 Cap ReBuild Pioneer 5V 200A MSU SN #72 7-JAN-1992 # MSU 6 Cap ReBuild Pioneer 5V 200A MSU SN #95 7-JAN-1992 # MSU 6 Cap ReBuild Pioneer 5V 200A MSU SN #98 7-JAN-1992 # MSU 6 Cap ReBuild Pioneer 2V 325A MSU SN #79 25-ARR-1991 # MSU 6 Cap ReBuild Repair of PDM-13 Tier 1 Power Pan Sunday evening Power Pan PDM-13 was pulled from M109 Lower Tier 1 because there was an incident the day before of its -4.5 Volt brick sagging below the lower major alarm limit for a few minutes. All the bricks in this Power Pan, including the -4.5 Volt brick, then ran fine for about another day until this Power Pan was pulled out. Diagnosis and repair of this -4.5 Volt brick will have to be very cafeful. Power Pan PDM-13 had been running in M109 Lower Tier 1 only since 24-JAN-2005. At the time it was pulled, Tier 1 PDM-13 consisted of: +5.0 Volt 600A MSU SN# 6 -2.0 Volt 250A MSU SN# 70 with Capacitor Band-Aid -4.5 Volt 200A MSU SN# 47 -5.2 Volt 200A MSU SN# 26 There is one interesting point about this failure: - The brick that dipped low Saturday afternoon, the -4.5V brick, is a fancy MSU rebuilt 6 capacitor brick. So far these MSU rebuilt bricks have given no trouble. Repair PDM-13 by installing MSU SN #72 as its new -4.5 Volt brick. Test PDM-13 +5.0 Volt Load = 210A Output = 5.055V 0 mV AC -2.0 Volt Load = 105A Output = 2.106V 0 mV AC -4.5 Volt Load = 107A Output = 4.611V 0 mV AC -5.2 Volt Load = 101A Output = 5.229V 0 mV AC Power Pan PDM-13 now consists of the following: +5.0 Volt 600A MSU SN# 6 -2.0 Volt 250A MSU SN# 70 with Capacitor Band-Aid -4.5 Volt 200A MSU SN# 72 -5.2 Volt 200A MSU SN# 26 It is a spare in the power supply cabinet. Now look at Power Pan PDM-10 PDM-10 was pulled from M105 Upper Tier 1 last Sunday night because first its -5.2V brick had no output and then after a couple of power cycles none of its outputs were running. It is thought that perhaps there was an AC power input problem to this Power Pan because of possibly a bad socket in its concator box. At the time it was pulled, Tier 1 PDM-13 consisted of: +5.0 Volt 600A MSU SN# 13 11-JUNE-1991 -2.0 Volt 250A MSU SN# 29 15-MAR-1991 with Capacitor Band-Aid -4.5 Volt 200A MSU SN# 50 15-MAR-1991 with Capacitor Band-Aid -5.2 Volt 200A MSU SN# 30 15-MAR-1991 with Capacitor Band-Aid Test PDM-10 under the toaster load box: +5.0 Volt Load = 209A Output = 5.056V 0 mV AC -2.0 Volt Load = 108A Output = 2.096V 1 mV AC -4.5 Volt Load = 110A Output = 4.609V 0 mV AC -5.2 Volt - no output - Repair PDM-10 by removing brick MSU SN# 30 and installing MSU SN# 95 for its -5.2 Volt supply. Now retest PDM-10. +5.0 Volt Load = 213A Output = 5.056V 0mV AC -2.0 Volt Load = 108A Output = 2.103V 0mV AC -4.5 Volt Load = 109A Output = 4.608V 0mV AC -5.2 Volt Load = 107A Output = 5.229V 0mV AC The repaired PDM-10 now consists of the following and is stored in the spare power pan cabinet. +5.0 Volt 600A MSU SN# 13 11-JUNE-1991 -2.0 Volt 250A MSU SN# 29 15-MAR-1991 with Capacitor Band-Aid Mallory -4.5 Volt 200A MSU SN# 50 15-MAR-1991 with Capacitor Band-Aid Cornel-D -5.2 Volt 200A MSU SN# 95 7-JAN-1992 MSU 6 Cap ReBuild Wednesday between Stores Take time to investigate what is going on in rack M105 that caused the trouble Sunday night. First notice some things about how the M105 Upper Tier 1 Power Pan was replaced: The Parallel cable on the right hand side of the next to bottom ERPB in the back of M105 had been pulled loose. This was the problem Sunday night. The power connector on the next to bottom ERPB in M105 was bent to the side and bent up. This could short circuit +5V and -5.2V power. The output cable from the top CAT2 card in the M105 Lower Tier 1 was trapped under the Power Pan that had been replaced. The cover panel at the top of the M105 power pan that had been replaced was put on backwards. I can see with a flash light that the AC feed cable to the M105 power pan that was replaced does not have all strands of it yellow cable into the terminal block. I hope that it has enough and is tight enough that it does not cause trouble. Sunday night there was also the problem of one of the sockets on the Contactor Box that feeds M105 Tier 1 having gone dead. Today I traced this to two blown fuses in that Contactor Box. To get the system running Sunday night John and Victor had moved some rack power cords. They had plugged one of the M105 Tier 1 cords into the next Contactor Box and to make room in that Contactor Box they had moved the Tier 3 power cord down to yet another Contactor Box. This was a good way to get running Sunday night but it has two problems: we now had a Contactor Box running 2x Tier 1's and Tier 3 which is so much load that it is close to blowing the breaker in the wall panel and it leave the labels wrong about what panel breaker feeds what rack (a safety issue). The M105 Contactor Box fuses have been replaced and all the power cords put back in their normal locations. Specifically: M105 Upper Tier 1 was moved from Contactor Box "F" back to box "E" Tier 3 was moved from Contactor Box "G" back to box "F" The wall breaker to Contactor Box layout and the Contactor Box to object serviced is shown in: www.pa.msu.edu/hep/d0/ftp/l1/framework/hardware/rack_crate/ power_panel_and_contactor_boxes.txt Need to purchase more Gould TRM20 fuses <----- Look at the M105 Upper Tier 1 Power Supply Test Points with the Fluke: (this is the Power Pan that was replaced) +5.0 Volt 5.054 V 0 mV ripple -2.0 Volt 2.103 V 0 mV ripple -4.5 Volt 4.604 V 5 mV ripple initial 40 mV ripple after 1 minute -5.2 Volt 5.226 V 0 mV ripple This is not supper good but should be OK for now. Look at the M105 Lower Tier 1 Power Supply Test Points with the Fluke: +5.0 Volt 5.065 V 0 mV ripple -2.0 Volt 2.106 V 0 mV ripple -4.5 Volt 4.604 V 2 mV ripple initial 19 mV ripple after 1 minute -5.2 Volt 5.230 V 0 mV ripple Look at the M109 Upper Tier 1 Power Supply Test Points with the Fluke: +5.0 Volt 5.059 V 2 mV ripple initial 2mV ripple after 1 minute -2.0 Volt 2.110 V 0 mV ripple -4.5 Volt 4.612 V 1 mV ripple initial 1 mV ripple after 1 minute -5.2 Volt 5.226 V 0 mV ripple Look at the M109 Lower Tier 1 Power Supply Test Points with the Fluke: (this is the Power Pan that was replaced) +5.0 Volt 5.050 V 0 mV ripple -2.0 Volt 2.094 V 1 mV ripple initial 5 mV ripple after 1 minute -4.5 Volt 4.611 V 0 mV ripple -5.2 Volt 5.229 V 0 mV ripple Then ran some L1 Cal Trig System Tests. CTFE EM HD PROM's page 7 full eta +-16 coverage pass twice CTFE Px Py PROM's page 2 full eta +-16 coverage pass twice CHTCR PROM Test on full eta +-16 coverage This failed at eta -1:-4 phi 1:8 Tot Et Ref Set #1 as it has for the past couple of tries. The error messages indicate that the bit of value 8 is not seen at the Tier 2 CAT card. I moved pushed in the CHTCR card, the Tier 2 CAT card and its Paddle Card and then it passed 3 runs of this test. Which was "loose" ? I bet the paddle card. Now try some Full System Tests. Setup eta +-16 phi 1:32 EM HD page 7 EM HD Ref Sets 0:3 EM Tot HD Veto Check Global Counts. This typically fails after between 1k and 50k loops with a problem at -13,2 Tot Et RS #1 output is 0 not 1. See the log book for 30-NOV-3-DEC-2004. Set the full system test to just eta +-12 and the rest of the setup is the same. Now it passes 1 Meg loop twice. FPD And-Or Terms Many notes on conversations with Duncan Brown over the past couple of weeks about problems they were having with AOT's from their NIM based FPD trigger. Between Stores I found that the problem was an incorrectly made connector on their AOT cable. This cable has been pulled off and on the AOT Patch Panel by Trigger Manager people who are trying to setup some other equipment for the FPD people. Wednesday after the 2 hour Store SCL Status Concentrator for G.S. 0:7 A thunderstorm ended the Store and there is no stack so Taka says swap the SCL Hub-End Status Concentrator for Geographic Sections 0:7. I pulled out the standard Status Concentrator card (which has the funny Firmware) and installed the hardware modified one with its special firmware for making the "Delayed" L1 Busy from the SMT Sequencer Controllers. I did not need to pull all the cards. It took about 1 hours to swap. The very stupid thing that I did was forget to install the JTAG pigtail cable. After things settled down and we had stable ZB we tried loading $3F into the "Delay L1 Bz" control registers. Kristian said that the changes in the amount of TrgMon displayed L1_Busy from the Sequencer crates was as expected, i.e. it works. I put the Post_Init_Auxi_L1CT.vio back to loading zeros into these 4 control registers and Kristian saw the percent of L1 Busy move back to its standard value. Thursday Optical SCL for Dean Cal 5k Test Stand Dean needs two SCL runs to the Cal 5k Test Stand to replace his current copper Geographic Section $4F run. At the SCL Hub-End set this up as: Geographic Section $4E cal5ktc no-readout patch cable Optic-4 Geographic Section $4F cal5k no-readout patch cable Optic-5 I send a note to Scott to have the $4E cal5ktc added to the resource file. I edited the official Crate List File on the web. PowerTec Supply PowerTec 5V 600A PowerTec SN# 728 MSU SN# 5 is back from factory repair. I have brought it back to Fermi. They clearly looked at it but did they fix anything ? Something still rattles inside. This is the brick that we think went low on 24-JAN-2005 and let the ERPB's in its backplane loose configuration. Open up MSU SN# 5 to see what part is rattling around. Find some issues in this supply. - There is a 6-32 screw fully loose floating around in the supply. I found it near the output fan end of the supply but it could have moved around anywhere. It does not look like a screw used for manufacturing this supplies - nor does it look like a screw from any of our stuff. It is round head, philipps, 6-32, 5/16 long. - Two screws that carry current from one card in the supply to heat sinks that hold switching transistors were loose. They were half unscrewed not holding anything or carrying any current. - There is a joint in the 600A internal bus that is held together with some #10 studs. This joint has gotten very hot. Some of the steel studs have turned blue. This was caused by a burr of metal in the joint that kept the joint from ever coming together. There was no surface to surface contact. The copper bus bar in this area had cooked and its surface turned dark. Clean it all up and counter sink the holes so that the surface joint will make up and then bolt it back together. I assume that this is what cooked and caused the supply to drop low on 24-JAN-2005. It has been hot for a long time. - There is a ferite choke over a bus bar that the repair people must have thought was the rattle problem. They had put a second blob of RTV on it. - There is a stud on one of the AC input wires that is no longer tight in its circuit bus. I soldered around that connection. Test this supply. It does not work. At 10 Amp load it is OK. At 235 Amp load it is OK most of the time and then about once per minute or so it jumps up to 100 mV or 200 mV of moise on the Fluke. After some seconds its settles back down. Its output Voltage wandrs around by 10 mV or so. Send it back again. Return PowerTec MSU SN# 5 Pioneer MSU SN# 47 Pioneer MSU SN# 30 to MSU. CTFE Cards High Eta with zero pedestal For sometime there have been two full CTFE cards that have had low or zero pedestal values. This is up in "high eta" so it has been on the back burner. The two CTFE cards with this feature are: eta +17:+20 phi 23 and eta -17:-20 phi 9. In both cases this problem is with the full CTFE so I do not suspect the Term-Attn-Brd. Look at eta +17:+20 phi 23. The reference supplies read -1.360V +1.00V and -0.806V. OK the -1V reference is being pulled down - expect a bad bypass cap. Pull the card, it is CTFE SN# ??. The -1V Ref bypass cap on Ch 4 HD is shorted so replace it. Look at eta -17:-20 phi 9. The reference supplies read -1.360V +1.00V and -0.720V. OK the -1V reference is being pulled down - expect a bad bypass cap. Pull the card, it is CTFE SN# 295. The -1V Ref bypass cap on Ch 2 EM is shorted so replace it. Measure Ohms of the -1V Ref Bus, the ADC center, and the +1V Rev bus wrt GND. They are about: 54 Ohm, 84 Ohm 54 Ohm. Put both cards back in, run them for a short while, and set their Ref Supplies. Load Pedestals and things are way off. On one of the cards most of the Peds are around 36 and on the other card most of the Peds are about 24. Make two single CTFE card Find_DAC runs. These are: eta -17:-20 phi 9 EM and HD keep = 1 write both files Find_DAC_V10_5_E_20050331.tti;1 eta +17:+20 phi 23 EM and HD keep = 1 write both files Find_DAC_V10_5_E_20050331.tti;2 The tti file that the Load_L1Cal_GainsPeds.mcf file currently points to for the |eta| 17:20 pedestal values is: Find_DAC_V10_5_E_20041202_Edited.tti;3 Make a copy of this called: Find_DAC_V10_5_E_20041202_Edited.tti;3_pre_30mar Then into Find_DAC_V10_5_E_20041202_Edited.tti;3 place the 16 new Ped DAC values from the two single CTFE cards runs of Find_DAC that were just made. If I knew how to do it I would then have differenced this tti file and its "_pre_30mar" version to verify that I did not screw up while typing. Load Peds and look with ADC_Mon and things look happy. Verify data in HSRO and return the system to DAC Shifter as they are loading protons. BLS Only ADF Crate Paddle Card Rev 2. Run through all channels using the Rev 2 BLS only Paddle Card and the Patch Panel card. Two assembly problems were spotted in the Patch Panel card. The HD- 2,3 circuit has no connection to the PFC connector. The problem is a non soldered series resistor. The EM+ 1,0 circuit has no connection of the front panel monitoring connector. The signal on HD 3,3 also looks lower than all the rest. I think this must be on the ADF card. Both sides of the differential signal are there but the captured ADC output is only about 2/3 of normal value for the other channels. Looking at the Patch Panel with the door folded down, i.e. sitting in front of the rack and fold down the Patch Panel panel and look down on it. +--------------------------------------------------------------------+ | | | 0,0 2,0 0,1 2,1 0,2 2,2 0,3 2,3 | | | | 1,0 3,0 1,1 3,1 1,2 3,2 1,3 3,3 | | | | | | PFC Conn to Upper PFC Conn to Lower | | Paddle Board Conn Paddle Board Conn | | | +--------------------------------------------------------------------+ Pin #1 of the BLS Cable Connectors in this diagram is in the Lower Lefthand corner of each of the 16 connectors. Now fold up the Patch Panel panel door, stand in front of the rack, and look at the Monitoring connectors. They are arranged as shown below. Note that the Ground row is the Top row of these connectors. ETA 0 0 1 1 2 2 3 3 0 0 1 1 2 2 3 3 0 0 1 1 2 2 3 3 0 0 1 1 2 2 3 3 PHI 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 EM E H E H E H E H E H E H E H E H E H E H E H E H E H E H E H E H or M D M D M D M D M D M D M D M D M D M D M D M D M D M D M D M D HD ------------------------------------------------------------------------------ DATE: 26,27-MAR-2005 From: MSU action at D-Zero Topics: L1 Cal Trig Power Over the weekend of 26,27 March 2005 we had a failure in the L1 Cal Trig Power Supplies. - Right at the beginning of the Store Saturday afternoon the -4.5 Volt brick in supply M109-C (Lower Tier 1 M109) dipped below the Major Alarm level and stopped the run. It may have dipped low a couple of times. It then recovered and ran the rest of that Store (Saturday afternoon through Sunday afternoon) without further incident. - At the conclusion of the Store Sunday afternoon John Anderson and Victor Martinez were ready to change the supply M109-C (M109 Lower Tier 1). They pulled out Power Pan PDM-13 and installed PDM-21 PDM-13 had been running in M109 Lower Tier 1 only since 24-JAN-2005. This swap went OK and doing the normal test run for 5 minutes with just running the new PDM-21 installed in M109-C (Lower Tier 1 M109) everything looked OK. - But when the full system was restarted there was now a new problem with the -5.2 Volt supply in M105-B (M105 Upper Tier 1). The M105-B Power Pan was cycled a couple of times and then none of its bricks were making any output. The tentative conclusion Sunday evening was that there were two problems affecting M105-B. - Its AC power feed from the contactor box that feeds M105 was bad. The theory at the time was that there is a bad socket on this contactor box. - All 4 supplies in this power pan had died. - Two actions were taken to fix the situation in M105-B - The M105-B Upper Tier 1 M105 Power Pan was replaced. Power Pan PDM-10 was pulled from M105 Upper Tier 1 and Power Pan PDM-19 was installed. This used both of the spare Tier 1 Power Pans at Fermi. - The AC power feed for M105-B was shifted down to the next Contactor Box. To do that a couple of more plugs were moved. - At this point the L1 Cal Trig was turned back on and appeared to be running OK. ADC Mon, And-Or Terms, and HSRO were all checked and then a new Store was put in. - At about 4AM, a couple of hours into this new Store, the Control Room called because there was a hole in the L1 Cal Trig Examine at TT eta +5:+8 TT phi 25:28. Philippe could see this in the HSRO data (I had not looked carefully enough earlier). I asked the CALMUO shifter to open the back door of M105 and look at the next to bottom ERPB card. Barry Spurlock did this. He reported that the back door to M105 was already open when he got there ! His log book entry then says, "The connector to the signal line on the right side of the board was no longer locked in place." This is from Monday, March 28, 2005 4:33:38 AM CST. ------------------------------------------------------------------------------ DATE: 22:25-MAR-2005 At: Fermi Topics: Brought ADF-2 SN #5 back to D-Zero. ADF-2 SN #5 is safe in a box in the spares cabinet and SN #3 is running on the sidewalk. I needt to get more 90 deg LMR-200 SMBs Pasternack PE4522. Pull the Wiener crate from rack S111 and bring it back to MSU. We now have at MSU from Fermi: the 1 spare power supply, the 1 spare fan tray, and the complete crate from S111. I have not yet seen the 4th Wiener crate at Fermi. I guess that it is here somewhere. Install the SCL Cable for G.S. $2D. This is the Test L1 Cal Trig sidewalk Readout Crate. I tested it and it is fine. Ran TRICS on the sidewalk and everything ran fine: SCLD locks on, configure ADF-2 DP FPGAs OK, run multiple 10**7 passes of RRT. ------------------------------------------------------------------------------ DATE: 8:11-MAR-2005 At: Fermi Topics: Bring the first ADF components to D0 Test of L1 Busy stretch ("delay") ADF control software on Nevis' PC Test Patterns to TAB ADF Capture Pulser, Noise, Tev deposit L1 Cal Trig trip off durning a Store Things brought to D-Zero. ADF-2 cards SN #3 and SN #5. Powertec power supply SN #1264 MSU #16 11-JUNE-1991 (just back from repair). MSU optical Bit-3 PCI to VME. Run 2B L1 Cal Trig Communications Crate (i.e. Run I L1.5 Cal Trig Crate). Installed SCLD with a normal SCL Receiver in the Communications Crate. It is running SCLD_T2 firmware. It locks onto the SCL stream just fine and it stays locked. Front panel LED's show the status. There was an issue getting the LVDS cable plugs in correctly. There appears to be a difference between the AMP cables and the ERNI cables. The cable end at the ADF-2 crate has its white plastic index fingers removed and has been flipped over. Used scope to check the timing signals on the ADF Crate backplane to make certain that everything from the SCLD was making it. I see the following: pin Z13 Begin Turn marker once every 21 usec 132 nsec wide Z15 Live BX asserted for 132 nsec Z17 always LOW - this is Save_Monitor_Data Z19 always HI - this is SCL Initialize Z21 BX_Clock direct Z23 BX_Clock complement Z25 Spare - HI with random 20 nsec spikes to Gnd Connect SCL Links for the Run 2B L1 Cal Trig Sidewalk test setup. I labeled the required additional SCL Hub-End Patch Panel Cables with Optic SCL names because that is most likely what they will eventually be used for. We are using 3 consecutive Geographic Section Numbers $2D, $2E, $2F because they are on an SCL Hub End FanOut card that currently had no connections. I put these sidewalk Cal Trig Test at the end of this Fanout card to give more space of its real owner (L2) to expand. SCL Hub-End Patch Panel Function in the Optic SCL Hub-End Patch Panel Cal Trig Sidewalk Test G.S. # Cable Num. to Sidewalk Cable ---------------------- ------- ---------- ------------------- VRBC in readout crate $2D 6 Cal Trig Test #3 SCLD in Comm. Crate $2E 7 Cal Trig Test #1 VME/SCL in Comm. Crate $2F 8 Cal Trig Test #2 Add Geographic Section Number labels at the sidewalk end of the cables. Other people working in MCH-1 tripped of the L1 Cal Trig during a Store. Back to MSU: ADF-2 SN #5, SCL Receiver with "stand alone firmware", Label Printer, * 9-Mar-05 Test of the new feature to stretch L1 Busy from SMT Sequencers. Ted Zmuda loaded new firmware in the SCL Hub-End Status Concentrator which handles Geographic Sections 0..7 Updated Init_Post_Auxi_L1FW.vio to setup the L1 Busy Stretch. cf. status_concentrator_control_of_l1_busy_delay.txt in http://www.pa.msu.edu/hep/d0/ftp/scl/ The desired stretch officially requested by Kristian Harder is 4.85 us. We can thus set this delay to 4900ns which corresponds to programming the value 0x31 into the Concentrator. VME_Address: 0x18800416 Write_Value: 0x3131 VME_Address: 0x1880041e Write_Value: 0x3131 VME_Address: 0x18800436 Write_Value: 0x3131 VME_Address: 0x1880043e Write_Value: 0x3131 There was no apparent increase in the L1 Busy percentage for these geographic sections. We tried the maximum stretch of 0x3f which gives 6.3 us. At the ZB rate of 900 Hz this should correspond to 0.5% additional dead time. We see nothing. We tried the delay to GS #0 only, to GS #0 & 1, to GS #0..7. No difference observed. These 4 new VME addresses were expected to be only availble on the new firmware, but doing direct VME IO to the address corresponding to the next concentrator shows that these addresses already were holding values on the old firmware. This was not understood. The new Init_Post_Auxi_L1FW file is left in but it programs the values that correspond to no additional stretch, i.e. 0x0000 Ted later realized that the current hardware directly drives the L1 Busy through the concentrator, while the FPGA only gets a copy of the signal but cannot change what is sent to the L1 FW. * ADF control software for sidewalk tests The linux build of the L1Cal_IIb software for the dzero environment was not ready for this week, so we used the windows version, squatting on the Nevis PC on the sidewalk with address l1cal2bpc.dhcp.fnal.gov. The old faithful V2.4 of the bit3 software drivers was installed. This version is not "Windows XP aware" and thus does not register itself with the operating system. This means that after every boot the OS notices there is a "new" piece of hardware installed and will propose to go hunt for a driver. Just click "cancel" or "no" through the 1-2 windows after each boot. The account was created as "laurens" but later renamed as "L1Cal_IIb" with a password that may be shared with our colleagues. One directory c:\L1Cal_IIb was created to hold all of the MSU software. - C:\L1Cal_IIb\L1Cal_IIb_TCC has the C++ executable and currently also receives the logfile and all result files. (In the future we will switch to have a separate directory for logfiles). - C:\L1Cal_IIb\L1Cal_IIb_Gui has the python gui code - C:\L1Cal_IIb\ComFiles has the python gui code - C:\L1Cal_IIb\AdfWaveGen stub of the python code to handle the waveform generator (because our standard command files try to "park" the waveform generator out of the way). - C:\L1Cal_IIb\python has python V2.3 with TkInter (no installation needed) - C:\L1Cal_IIb\exo holds the data path fpga firmware - C:\L1Cal_IIb\misc to save miscellaneous files There are 2 icons on the bottom right of the screen for starting the two parts of the software: - the TCC application (C++) - the GUI application (python) * Test Patterns to TAB Three new command files were created specifically for the sidewalk tests to send specific patterns to the TAB(s). All 3 files start by configuring the adf Slot #8 with the T6 firmware and execute the play_wake_up command file to allow the channel link connection to lock. - Sidewalk_ChLink_Test_PseudoRandom.cmd Loads and starts the pseudo random generator with the same seeds used with Denis' Big Brother at MSU. - Sidewalk_ChLink_Test_FixedEm55HdAA.cmd Loads the seeds (but does not start the shifters) to send 0x55 on all EM channels and 0xAA on all HD channels - Sidewalk_ChLink_Test_Fixed_Pallet.cmd Loads the seeds (but does not start the shifters) to send a different recognizable value for each trigger tower but the same value for EM and HD. The value for relative trigger tower (eta,phi)=(e,p) is 0xep (e,p=0..3) e.g. for EM&HD(1,3) send 0x13 and for EM&HD(3,1) send 0x31 Also make a special version of the GUI's "Command File" dialog to - remove the buttons about serial IO to the waveform generator - and add 3 buttons to call these 3 new command files directly. This will allow the Nevis folks to setup the ADF card for Channel Link tests with minimum effort. * L1Cal Hand Pulser captured by the ADF Use the hand held L1Cal pulser to send signals through the transition sytem to the ADF card and record the waveforms. * Noise measurement without beam captured by the ADF Plug several Trigger Tower BLS signals into the splitter and measure the noise seen by the ADF card using Find_DAC. EM(-9,1) 1.02 1.06 1.08 HD(-9,1) 1.56 0.86* 1.64 * bad connection on ADF end of BLS extender to sidewalk EM(-9,2) 0.90 0.88 0.91 HD(-9,2) 2.73 2.72 2.72 EM(-10,1) 0.84 0.87 0.85 HD(-10,1) 2.68 2.46 2.51 EM(-10,2) 0.78 0.82 0.78 HD(-10,2) 3.66 3.64 3.73 For comparison, Find_DAC on TRICS (IIa) finds for these towers: EM(-9,1) 0.64 HD(-9,1) 1.06 EM(-9,2) 0.49 HD(-9,2) 1.72 EM(-10,1) 0.47 HD(-10,1) 1.47 EM(-10,2) 0.43 HD(-10,2) 1.82 * Energy Deposits with beam captured by the ADF Use the new "Try and Capture" ADC Analysis feature to hunt for real energy deposits during Tevatron running. We capture two EM and two HD deposits while listening to TT(-9,1). TCC is told to keep collecting 4k slices of ADC values until it finds a value above a threshold, set to 125 counts. We had previously run Find_DAC to set the zero energy response at 50 counts. Also capture some streams to file from EM&HD(-10,1) through the splitter with no beam in the machine. ------------------------------------------------------------------------------ DATE: 12:15-FEB-2005 At: Fermi Topics: L1 Cal Trig Water Leak, Repair PDM-21 The L1 Cal Trig water leak was in the bottom radiator between M110 and M111. The leak is in the side of the input header pipe right up against the vertial yellow support bar. It is impossible to clamp or solder over in place. Pulling all 4 radiators from this location would be complicated - impossible because of the T2 crate in M111. I think that all the ones that we have ever pulled out had T2 Power Pan in the higher number rack pair. I disconnected the water feed to this radiator, plugged the hoses, and re-installed the mud flap G10 pieces. To compensate for the loose of cooling (especially for the air coming into the T1 Power Pan in the very bottom of M111, I have pulled a number of cards from the Tier 1 Crates in M111 and M112. M111 and M112 are used for readout only and are not currently part of the eta coverage for generating L1 Cal Trigs. I pulled the Tier 1 CAT2 cards, and the CHTCR cards from these 4 Tier 1 crates. That is 4x CAT2s and 2x CHTCRs per crate. Note that these 4 Tier 1 crates only had T1 Momentum CAT2s installed. Repair of PDM-21 which was pulled from M109 Lower Tier 1 on 24-JAN-2005. PDM-21 consisted of: PDM-21 +5.0 Volt MSU SN# 5 -2.0 Volt MSU SN# 45 15-MAR-91 -4.5 Volt MSU SN# 21 7-JAN-92 MSU_6_CAP -5.2 Volt MSU SN# 46 15-MAR-91 Replace the +5.0 Volt 600 Amp brick MSU SN# 5 with MSU SN# 22 which is recently back from repair. Test without load for 20 minutes. PDM-21 now consists of: PDM-21 +5.0 Volt MSU SN# 22 7-FEB-92 -2.0 Volt MSU SN# 45 15-MAR-91 -4.5 Volt MSU SN# 21 7-JAN-92 MSU_6_CAP -5.2 Volt MSU SN# 46 15-MAR-91 Bring the bad +5.0 Volt 600 Amp brick MSU SN# 5 back to MSU. Brinning back to MSU: the wire stock and lugs and tools for making the DC distribution harnesses that are in the Run II TFW racks and in M101. For now bring back our good scope, along with its books and probes. ------------------------------------------------------------------------------ DATE: 24-JAN-2005 At: MSU Topics: L1 Cal Trig Power Pan Replacement A strip of the L1 Cal Trig display had started to look funny to the shifters. Norm called us. This strip was TT eta +13:+16 TT phi 17:32. Looking at a dump of the raw readout data it looked funny (all zeros or f's or something) A guess was that the ERPB's had lost their minds in this crate. ReConfiguring them made the raw readout data look OK. Looking at the power supply display, the +5 Volt for this Tier 1 crate looked noisier than any other Tier 1 +5 Volt. Theory is that a drop out on the +5 V caused the problem. Because there was no beam we took the decision with Norm to replace the Lower Tier 1 Power Pan in rack M109 which is eta +13:+16 phi 17:32. Norm worked with John Foglesong and Victor and did the swap. From M109 Lower Tier 1 they pulled out PDM-21. Into M109 Lower Tier 1 they installed PDM-13