Post Release Log Book for the Hub Module -------------------------------------------- The most recent entries appear first in this log book file. This section of the Hub Module Log Book starts immediately after the design was released for manufacturing. For information about the design stage of the Hub Module see the Log Book "hub_log_book_pre_release.txt". ------------------------------------------------------------------------ DATE: -Oct-2019 Topic(s): ------------------------------------------------------------------------ DATE: 4-Oct-2019 Topic(s): Receive Hubs SN-07 and SN-09 back from Yuri, Work and Status of the 4 HTM Cards that "did not work" HTM Cards SNs 02, 05, 07, 10 Hub modules SN-07 and SN-09 were shipped back to MSU by Yuri. They arrived late afternoon Oct-3. They were shipped to the Cyclotron on DHL Air Way Bill 6540110113. This shipment was the result of the "over temperature event" on Sunday 15-Sept-2019. This is all a bit non-intuitive because, as I understand it, the two cards in the STF during the over temperature event were SN-08 and SN-09, with SN-08 powered On and SN-09 with its ROD powered Off. Additionally at one time Wade's plan was to send back to MSU Hubs SN-07 and SN-08. Over the past 3 weeks or so I have tried to diagnose the 4 HTM Cards that "did not work". This diagnostic work was carried out by replacing the 4 HTMs that are normally in the 6SS with the 4 HTMs that had been reported as not working. The JTAG pod with ID ...E501 which is normally used in the bench test setup was moved to Wade's new lab an used in Slot #6 of the 6SS so that I had JTAG access to all 6 slots. The setup with the 4 "not working" HTM cards installed in the 6SS is: Slot #6 HTM SN-05 Slot #5 HTM SN-02 Slot #4 HTM SN-07 Slot #3 HTM SN-10 As part of this testing the FPGA Mezzanines were swapped between HTMs SN-05 and HTM-02 and between HTMs SN-07 and SN-10. Note that this FPGA Mezz swap was not un-done because there was no point in it. Note also that anytime it was necessary a so called "working" HTM Card was used in a given HTM slot in the 6SS to verify that the Hub's FEX Data Receivers and Combined Data Transmitters were working correctly. Tests of HTM Cards SN-05 and SN-02: HTM SN-05 originally had the problem of requiring multiple power ONs before its SiLabs chip made a 320 MHz output but once this clock was running then this card's MGT Links worked without error. FPGA Mezz SN-500138 was removed from this HTM and FPGA Mezz SN-500128 was installed on this HTM. Now this HTM has 320 MHz clock on its initial power ON and its MGT links work without error at the 3.2 E-16 BER level. HTM SN-02 originally had the problem that sometimes some of its MGT Links did not work (or vivado did not show them as working). FPGA Mezz SN-500128 was removed from this HTM and FPGA Mezz SN-500138 was installed on this HTM. Now this HTM requires multiple power ONs to get its 320 MHz clock running but once it is running then all of its MGT Links work OK at the 3.2 E-16 BER level. Conclusions: Both HTM SN-05 and SN-02 circuit boards are OK. FPGA Mezz SN-500138 requires multiple power ONs to get its 320 MHz clock running. FPGA Mess SN-500128 once showed some MGT Links as not working (or vivado showed them as not working) but now this FPGA Mezz seems fine. Tests of HTM Cards SN-07 and SN-10: HTM SN-07 originally showed no problems with any of its MGT Links. FPGA Mezz SN-500137 was removed from this HTM and FPGA Mezz SN-500135 was installed on this HTM. Now none of the MGT Links involved with Quad 110 are working (that is: CD from Hub #2, FEX L2 & L4 to Hub #1, FEX L0 & L1 to Hub #2 now do not work). HTM SN-10 originally had the problem that none of the MGT Links involving Quad 110 worked, i.e. CD from Hub #2, FEX L2 & L4 to Hub #1, and FEX L0 & L1 to Hub #2 did not work. FPGA Mezz SN-500135 was removed from this HTM and FPGA Mezz SN-500137 was installed on this HTM. Now all MGT Links on this HTM are working OK. Conclusions: Both HTM SN-07 and SN-10 circuit boards are OK. FPGA Mezz 500135 Quad 110 does not work. FPGA Mezz 500137 works fine. Notes: FPGA Mezz 500137 does not have a heat sink on its FPGA. HTM SN-02 originally did not have MiniPODs installed but as part of this diagnostic work I installed both Rec and Trans MiniPODs on to HTM SN-02. We have both straight through and cross over MPO to MPO 12 fiber ribbon cables so by using both cable types one can test 8 of the 12 MiniPOD Receivers on an HTM Card by using just the 4 MiniPOD Transmitters on that card. ------------------------------------------------------------------------ DATE: 13-Aug-2019 Topic(s): IPMC Software Downloads, 3 more Hubs brought to Final Form, 6SS Running in Wade's Lab, Move 14SS and all remaining equipment from old test room All of the IPMCs currently at MSU (except for IPMC SN-31) have now been programmed with Fatih's constant 25 deg C software. The IPMCs that I programmed are: 9 from the "Initial Batch", 10 from the "Second Batch", and SN-203 from Hub SN-05. In addition to these 20 IPMCs Wade had programmed the 4 IPMCs that are on the 4 Hub cards currently running in the 14SS and the 6SS with Fatih's 25C software. All 24 of these IPMCs are now at IP Adrs: 192.168.1.89. Philippe made a nice script to change the IP Adrs range of the 2nd Enet port on HubDev2 to make it possible to program these parts. This is script is called by: from /root ./switch_private_net.sh with and argument of 10, 134, or 192. So far we do not see any Enet activity from IPMC SN-31. I have log files from all programming of the IPMCs on the web at: .../hub/reference/ipmc/lapp/msu_ipmc_programming_logs/ Hubs SN-04, SN-05 (with the original MSU ROD), and SN-06 have all been brought up to Final Form including that their IPMC now controls there Power On-Off state. See their trailer sheets. Hubs SN-04 and SN-06 are currently in the 6SS. The two Hubs in the 14SS (SN-11 and SN-12) are also in Final Form except that they still have a "ByPass" on their power control. The 6SS with 2 Hubs under IPMC power control and 4 HTM cards and the HubDev2 computer have now all been running in the new lab for about 4 days. So far all looks OK. Work to move the remaining "old test rack room" equipment will begin tomorrow Aug 14th by pulling out the 16 or 18 small Enet Switch test computers from the Rack #3. The actual move of the 14SS is scheduled for Thursday the 15th Aug. The remaining equipment in the old test rack room includes: 14SS, HubDev computer, TTC VME crate, HubTTC V706 computer, HubPSMA computer, 16 or 18 little computers, the black video roll around table, my scope, and Wade's scope. ------------------------------------------------------------------------ DATE: 7-Aug-2019 Topic(s): IPMC Software Download, Move of the 6 Slot Shelf and HubDev2 It became clear a couple of weeks ago that the Hub power On-Off cycling is caused by the lack of a temperature monitor signal being sent from the IPMC to the Shelf Manager. We have lots of temperature sensors on the Hub but the default "MSU Hub" software from Fatih does not read out any of them and send the required temperature monitor information to the Shelf Manager. As a work around to get past this Fatih added a constant 25 deg C "sensor" to his MSU Hub IPMC software. We believe that this fixes the power On-Off cycling problem and this new IPMC software provides for much faster Ethernet download of new software to the IPMC. The IPMCs currently at MSU and not installed on a Hub are: The Initial IPMC that Wade brought to MSU is SN-031. Of the Initial Batch of IPMCs the ones that we currently have at MSU are Serial Numbers: 132, 134, 135, 137, 138, 139, 140, 141, 143. 9 total Of the Second Batch of IPMCs the ones that we currently have at MSU are Serial Numbers: 195, 196, 197, 198, 199, 200, 201, 202, 208, 209. 10 total Spencer's instructions for running his script to download Fatih's constant 25 deg C software are the following: The script can be found at /home/hubuser/Documents/ICARE/workarea-MSU/FEX_Hub/save/program_IPMC.sh. It is a simple bash script that issues the firmware upgrade commands one would normally run by hand. You specify the IOIF MCU firmware/binary with --IOIF_f and the IPMC MCU firmware/binary with --IPMC_f. For example, if I am in /home/hubuser/Documents/ICARE/workarea-MSU/FEX_Hub/save I would do from the terminal: ./program_IPMC.sh --IOIF_f bmc_IOIF_Fatih_July4.bin --IPMC_f bmc_IPMC_Fatih_July4.bin You can also specify the IP Address with --IP_Addr, but if you do not the script defaults to 192.168.1.89. Also note that this script should ONLY be used to download firmwares created from the FEX_Hub project in the directory /home/hubuser/Documents/ICARE/workarea-MSU/. This is because in the ICARE ethernet upgrade instructions we are told to source a script in the project area of the firmware we are uploading. I noticed that I cannot communicate with the IPMC immediately after issuing the IOIF upgrade (cannot send another upgrade command, cannot ping), so I put in a 5 minute wait time between issuing the IOIF upgrade and issuing the IPMC upgrade. The time until I can communicate again seems to be around 1-2 minutes, so I used a 5 minute wait to be on the safe side. With Spencer's script the 10 IPMCs from the Second Batch were programmed with Fatih's Constant 25C software. Currently I can not ping or download software to the IPMCs from the Initial Batch. That may be caused by these Initial Batch IPMCs having a different default IP Address. Looking into this now. Status of the Move to Wade's new lab: The power distribution system and its control loop and safety sensors are signed off on. The new cooling system was tested with a 6 kW load and signed off on. Rack 3 is fully installed with all of its equipment, Rack 2 is installed with only: the HubDev2 and the 6 Slot Shelf installed in it. All of the mechanical work in Rack 2 is finished so that it is ready for the rest of its equipment to be installed: HubDev1, TTC VME Crate, and in the bottom of Rack 2: the PerfRadar and the VC-709 TTC computers. The 6 slot shelf and the HubDev2 computer are both running in Rack 2. ------------------------------------------------------------------------ DATE: 17-July-2019 Topic(s): CMX move to new lab, IPMCs back from CERN All of the CMX test stand equipment has been moved to the new lab. This includes the CMX L1Calo special VME test crate, its power supply, and the CMX test stand computer "MSUL1C". The box of test program CF memory modules and the box with a TTC Receiver Mezzanine card have been stored in an open space in the CMX test VME crate. All of this equipement is in Rack #3 in the new lab. The only thing that I could not find when moving the CMX test stand to the new lab is the special cable for testing the CMX's front panel and backplane LVDS signals. The power supply connections between the CMX test VME crate and its power supply are (viewed from back of crate) : - At the lower back of the crate, the three power bars are: Red Bar next to the backplane Black Bar in the middle Blue Bar furthest from the backplane - DC Power Cables from the Power Supply: On the Right hand side: Red PS wires to Red Bar Black PS wires to Black Bar Brown Sense wire to Red Bar Blue Sense wire to Black Bar On the Left hand side: Yellow PS wires to Blue Bar Blue PS wires to Black Bar Brown Sense wire to Blue Bar Blue Sense wire to Black Bar As agreed in a Monday Meeting, Yuri has returned to MSU most of the IPMC cards that I sent to him a couple of months ago. These were received back at MSU on about 3-July-2019. From the "Initial" batch there are SN: 132, 134, 135, 137, 138, 139, 140, 141, 143 a total of 9 From the "Second" batch there are SN: 195, 196, 197, 198, 199, 200, 201, 202, 207, 208, 209 a total of 11. I do not fully understand the thick stack of printed material that came from Fatih but for each IPMC it includes: the names of the .bin files that it is programming into the devices, e.g. bmc_IPMC.bin and bmc_IOIF.bin (both to the same processor ?) and a MAC Adrs and IP Adrs for the IOIF. As far as I know we still do not understand if the Annecy IPMC does DHCP or if it has an IP Adrs built into its software. ------------------------------------------------------------------------ DATE: 17-June-2019 Topic(s): Receive a ROD from Ed Received ROD V3 No 3 from Ed and installed it on Hub SN-11. Philippe started up the MGT link tests. The hope/intent is to get test data from this ROD to Ed before the ROD's pre-production review which is on the 25th of June I think. ROD V3 #3 arrived without any of its MiniPODs and FO cables. This shipment from Ed also contained the "mechanical only" Hub card, aka Hub SN-00. This shipment from Ed was made through Yuri which may now be standard practice. ------------------------------------------------------------------------ DATE: 10-June-2019 Topic(s): Production Hubs from Debron, Hub Serial Numbers, FA and Test of Hub SN-11, Remove Enet Swch B to C link, Racks and Pulizzi power ready, Ship Hub SN-08 to Yuri Pickup the 20 Production Hub cards from Debron on April 30th. This was right at 12 weeks from release to collect finished cards. We are skipping Hub SN-10. Hub SN-00 is the Mechanical Only Hub. Hub SN-01 is the one without an FPGA on it. Hub SN-02 through SN-09 are the 8 Prototype Hubs that have FPGAs. Hub SN-10 is skipped. The 20 Production Hubs are SN-11 through SN-30. The first Production Hub SN-11 was through MSU Final Assembly and running on the bench by 9-May-2019. It is 100% final form except that SN-11 does not have a ROD and its IPMC power control is bypassed. After bench work it goes to MGT Link tests and Enet Switch tests which so far look basically OK. Monday meeting 13-May-2019 receive verbal instruction to split the Enet Switch between the B and C sections and to split by pulling the coupling capacitors off of the card. Ask for written confirmation but as usual do not get it. By about 31-May-2019 the rails and shelves and brackets are ready in the racks for Wade's new room. The 4x Pulizzi boxes with new power cords and the Control Loop for the Pulizzi boxes are all ready and tested. No word yet about wanting or not a drip detector and sprinkler stopper. Friday 7-June-2019 ship Hub SN-08 to Yuri on air bill 8112 7236 3029. SN-08 is full final form except that it does not have a ROD. SN-08 IPMC does have power control. ------------------------------------------------------------------------ DATE: 7-Mar-2019 Topic(s): Production Hub parts to Debron, Hub SN-09 + ROD SN-04 to Yuri On Wednesday March 6th I dropped off the remaining parts to build the 20 production Hub Modules at Debron. This delivery included the 20 FPGAs for the Production build. George, Josh, and I reviewed the contents of this delivery, verified the box with the FPGAs, and reviewed the status of the Kit for the Production Hub build. On Friday February 22nd I shipped Hub SN-09 with ROD SN-04 to Yuri on air waybill 8112 7236 3235. It was in Geneva Meyrin by Monday and Yuri reported that it arrived in his office on Wednesday February 27th. ------------------------------------------------------------------------ DATE: 5-Feb-2019 Topic(s): Release Build of 20 Production Hub Modules, Combined Data Signal Strength Study, Hub and HTM MGT Link Types with Equivalent Routing At the 10 AM Monday Meeting Wade approved releasing the build of the 20 Production Hub Modules. That afternoon I sent a note to Geogre asking Debron to start the work to build the production Hubs. Hub Module Output Combined Data Signal Strength: ------------------------------------------------- The most worrisome result from the study of the MGT links on the Hub Module is probably the apparent low level of the Combined Data link as it is received on the HTM Cards. As received on the HTM the Combined Data signal appears to be quite low in amplitude but OK in fidelity, i.e. it has an OK horizontal opening. We have used the following steps the "officially prove" that the cause of this problem is not on the Hub Module and thus it is OK to go forward with the build of the Production Hubs. - Besides receiving the Hub's Combined Data output on an HTM Card we can also receive it on the Other Hub. Files in the following directory show eye diagrams of the Combined Data as received on the Other Hub as a function of the Drive Level on the sending hub. https://web.pa.msu.edu/hep/atlas/l1calo/hub/testing/Combined_Data_Initial_Study/ files: scan_hub1_oth_hub_combdata_XYZmv_0_0_eq_off.png scan_hub2_oth_hub_combdata_XYZmv_0_0_eq_off.png Because the Combined Data links as received on a Hub Module do not have a Motorola Fanout chip, that provides a limiting amplifier function, we can see a monotonic increase in open eye area as the Drive Level is increased. Only the very lowest Drive Level results in an eye pattern that looks like the Combined Data eye pattern as received on an HTM Card. - The traces on the Hub Module that are involved in sending out all of its Combined Data links are exactly the same as the Hub Module traces involved in sending out its Readout Data. The Hub Readout Data, as received on the Other Hub, passes through a Motorola Fanout chip but otherwise has much the same routing as the reception of the Combined Data link. Thus comparing the reception of the Combined Data and the Readout Data on the Other Hub gives us the best view that we have of the isolated effect of the Motorola Fanout chip. Further, assuming that the output of the Motorola Fanout chip is a good full CML signal, we can compare the raw Combined Data signal to the Motorola Fanout amplified and limited Readout Data signal to learn if the Combined Data signal is of the appropriate amplitude. That is we assume that the Motorola Fanout provides a standard candle CML output signal. Eye diagrams of the Hub Readout Data as received on the Other Hub, as a function of sending end Drive Level, are available in the directory: https://web.pa.msu.edu/hep/atlas/l1calo/hub/testing/Hub_to_Hub_Readout_Initial_Study/ These eyes show that even at low sending end Drive Levels the open area of the received signal soon saturates. This gives us confidence that the CML output from the Fanout chip is at the full CML signal level and thus it provides a good full CML signal to compare the Combined Data signal with. What we see is that for Combined Data signal Drive Level of 640 mV or greater it has the same open area as the Readout Data signal that has been amplified and limited by the Fanout chip. Conclusions: ------------ - The Combined Data signal as sent out from a Hub Module has good signal strength. - There is no significant problem with the high-speed differential signals that are routed on the bottom side of the Hub pcb. - There is no significant problem with the high-speed signal characteristics of the pcb material that is used on the Hub circuit board. The good place to see all of this data in one place is in Philippe's files in the directory: https://web.pa.msu.edu/hep/atlas/l1calo/hub/testing/FanOut_Initial_Study/ Recall that the Fanout Equalizers only exist on the Readout Data links (not on the Combined Data links) and that they were only controllable on the Hub #2 receivers. The Fanout Equalizers were always ON on the Hub #1 Readout Data links. Possible Evidence for an Overall HTM Card MGT Link Receiver Strength Problem: ---------------------------------------- On the HTM Card 2 of the MGT Receivers are used for the Combined Data links, 12 MGT Receivers are used for the MiniPOD Receiver channels, and 2 MGT Receivers are not used. On the HTM Card the high-speed trace routing of the Combined Data input signals and the trace routing of the MiniPOD Receiver circuits are the same. Thus assuming that the electrical output signals from the MiniPOD Receivers are full strength CML signals we can use them to judge how well the HTM Card's MGT Receivers are working. The eye diagram plots of HTM Card MiniPOD Receivers the following directory show signals that are similar to the Combined Data signals: https://web.pa.msu.edu/hep/atlas/l1calo/hub/testing/Hub_MiniPOD_Initial_Study/ This study should be expanded to include, as a function of Drive Level, look at Hub MiniPOD transmitted signals as received on both Hub Module and on HTM Card. This could be done as part of a study to verify that the polarity of the Hub and HTM MiniPOD MGT links is correctly understood. Equivalent Routing Sections On Hub and HTM: -------------------------------------------- On the HTM card the following MGT link traces are routed in the exact same way - fully on the top surface of the pcb: - both Combined Data inputs to the HTM FPGA - all 12 MiniPOD Data inputs to the HTM FPGA (includes caps) - all 4 MiniPOD Data outputs from the HTM FPGA - 4 FEX Data outputs from the HTM FPGA (includes capacitors) Hub #2 Lane #3 Hub #2 Lane #4 Hub #2 Lane #5 Hub #2 Lane #6 The MiniPOD input and output traces are longer than both the Combined Data Input and FEX Output Data traces to the Zone 2 connector. On the Hub card the following MGT link traces are routed in the exact same way - fully on the Bottom surface of the pcb: - all 12 Combined Data outputs to the FEX cards - This Hub's Combined Data output to the Other Hub - This Hub's Combined Data output to its ROD - This Hub's Readout Lane 0 & 1 to its ROD All of these links include a capacitor. The trace runs to the ROD are slightly longer than the trace runs to the Zone 2 backplane connector. ------------------------------------------------------------------------ DATE: 20-Nov-2018 Topic(s): Engineering Tests of the Hub Besides the two current main engineering studies of the Hub: to verify the operation of the Hub Module's Ethernet Switch and to verify the reliable transfer of 6 lanes of IBERT FEX data from each of the 12 FEX cards to the Hub and ROD the following are additional studies that we could/should complete before releasing the Hub for its production build: - Test all of the Select I/O connections to the Hub FPGA, especially the Spare Hub <--> ROD Select I/O connections. - Test the Optical MGT - MiniPOD channels on the Hub Modules and HTM cards. - Test all of the "one of a kind" MGT Links between Hub-Hub and between Hub-ROD, e.g. Hub Readout to the Other Hub, Combined Data between Hubs and to the ROD, ROD Readout Control. - Understand the Vertical Scale of the Eye-Diagrams; is the vertical scale in Volts or is it just a relative scale with meaning only to a specific Eye-Diagram. Test by adjusting the Driver Output Voltage at the FEX end of the Copper links. - Make a study that moves around the various HTM cards to understand how much of the Slot-to-Slot differences that we see are due to the Backplane and how much of the current Slot-to-Slot differences are due to the particular HTM card that is driving FEX data into a given Slot. - DeBug the ROD SN-00 (or is it SN-01) problem with 2 lanes from FEX #6 - where the two problem lanes depend on whether this ROD is in Hub #1 or Hub #2 Slot. - Study different MGT Line Rates for the Backplane FEX data. Try a big change, e.g. to 10 Gbps just to see how much of the system works at 10 Gbps. Try a small, e.g. 15% change, where the actual attenuation is not changed by much, but the effect of trace routing problems (e.g. stubs) could change the signal fidelity a lot. - Study sending FEX MGT data in the 6 Slot Shelf. As much as possible keep everything the same, e.g. same cards used, same FEX to Hub distance, and see if the Eye is better or worse in the 6 Slot Shelf. - Get the IPBus based monitoring of the IBERT based test of the Combined Data channel from Hub to HTM (FEX) working so that we can finally study the reliability of the Combined Data that the Hub must provide to the FEXs. This is currently a big unknown. ------------------------------------------------------------------------ DATE: 16-Nov-2018 Topic(s): Slot to Slot Delay Test, HWmonit test with and without IPMC Using the HTM front panel Lemo clock monitor connectors we should be able to see the slot to slot delay of the 40.08 MHz reference clock in the 14 Slot Shelf. I'm using transformer coupled probe cables to make this timing survey. These are 1:1 ratio MCL T1-6 transformers with a 100 Ohm 100 nFd series network in the primary (to give the CMOS driver in the HTM something rational to work into) and direct connection to a 5 ft RG58 BNC cable on the secondary. This results in a clean scope signal with a risetime of about 1.5 nsec. The TDS3054 itself has a rise time of about 700 psec, a typical channel to channel differential delay of 100 psec, and at full bandwidth a channel to channel crosstalk of > 30:1. The ATCA slot to slot pitch is exactly 1.2 inches. Assuming 1/2 c for the speed of the differential backplane signal we should expect a slot to slot delay of 204 psec. c is about 85 psec per inch. The backplane reference clock is driven from Hub #1. On the left-hand side we should expect a delay of about 1020 psec between FEX Slot 3 to FEX Slot 13 (a total of 5 pitches). On the right-hand side the delay between FEX slots 4 and 14 should be the same but the whole pattern is delayed by an additional slot (crossing the Hub #2 location). There are additional uncertainties in the timing of the Lemo monitor signals caused by the ICs in the signal path. The signal path is: 65LVDS2 receiver, PLL, CDCLVD1204 fanout, and then a 65LVDS2 that drives the front panel Lemo. The expected delay in the 65LVDS2 is: 1.4 nsec min, 2.5 nsec typ, 3.6 nsec max. The expected delay of the CDCLVD1204 is: 1.5 nsec typ, 2.5 nsec max. At a given operating condition the chip to chip delay difference is 600 psec max. So the uncertainty in the signal path delay going through these 3 chips could be large compared to the expected 204 psec slot to slot delay. The scope pictures of the various delays are in the web directory: https://web.pa.msu.edu/hep/atlas/l1calo/hub/hardware/drawings/Slot_to_Slot_Timing_Scope/ For the left-hand side pictures scope channel #1 was always the signal from FEX slot #3 and channel #1 triggered the scope. Scope channel #2 moved through FEX slots: 5, 7, 9, 11, 13. For the right-hand side pictures scope channel #2 was always the signal from FEX slot #4. Scope channel #1 moved through FEX slots: 6, 8, 10, 12, 14. The scope triggered on Ch #1. For the most part the measured delays are monotonic and have the correct overall (3 to 13 or 4 to 14) delay. The main exception is FEX slot 9. HWmonit tests continue. So far it appears that on Hub cards without a ROD module one must not enable the I2C Buffer to the ROD. The kind of makes sense because the pull up for the bus on the ROD side of this buffer is on the ROD itself - so without a ROD the I2C Bus on the ROD side of this buffer just floats. I thought that the LTC4315 could take care of this but it makes sense that it can not if one of the ROD side signals actually floats back and forth between Hi and Low. It also appears that HWmonit shows lost arbitration only if the IPMC module is actually on the Hub. With the IPMC removed then there are no cases of lost arbitration. This was checked on Hub SN-04 in the 6SS by running it with and without an IPMC. This is being verified on Hub SN-08 in the 14SS but that test is still under way. Note that in Friday afternoons HWmonit test on Hub #1 in the 14SS that the IPMC does not seem to be running normally. It is continuously flashing its OK to extract LED - so it may not be trying Sensor I2C Bus cycles. ------------------------------------------------------------------------ DATE: 15-Nov-2018 Topic(s): Hub SN-07 shipment to Yuri, HWmonit running with IPMC, The 47 parts from Alpha arrived Hub SN-07 and the 24 IPMCs have made it to CERN - details are below. This shipment is air waybill: 8112 7236 3224. Note that FedEx now wants to see the destination address exactly as listed on this waybill, i.e. the city is Geneva 23 and the "zip code" is 1211. Gabriel sees a difference in how HWmonit runs with the IPMC on the Hub vs no IPMC on the card. This test is on Hub #2 in the 6SS. ------------------------------------------------------------------------ DATE: 12-Nov-2018 Topic(s): Ship Hub SN-07 and IPMCs to Yuri 47 part job still at Alpha Metal Shipped Hub SN-07 to Yuri on air waybill: 8112-7236-3224. Hub SN-07 does not have a ROD on it but it has spent a lot of time in the 14SS doing MGT Link and Enet tests. This Hub has an IPMC on it but the IPMC does not actually control the enable signal to the 12V Isolated supply. The 12V Isolated supply is controlled by the bottom handle switch (which also feeds the IPMC). Philippe has a lot of MGT Link test results for SN-07, it has always looked good, and it was given a final test of all 72 links right before shipping. In the 12-Nov-2018 shipment of Hub SN-07, return to Yuri the following 24 IPMC cards: 9 from the Initial batch of IPMC cards that we received SN: 132, 134, 135, 137, 138, 139, 140, 141, 143 15 from the Second Batch of IPMC cards that we received SN: 195, 196, 197, 198, 199, 200, 201, 202, 207, 208, 209, 210, 212, 213, 214 Keep 3 IPMCs from the 2nd batch at MSU SN: 215, 216, 217 In addition we have IPMCs on the 4 Hubs currently at MSU, Hub SNs 04, 05, 06, 08, and on Hub SN-07 which is now on its way to Yuri. Thus we are keeping a total of 8 IPMCs - all from the 2nd batch. The job of 47 FPGA and MiniPOD heat sinks is still at Alpha. They say that they thought I was coming to pick it up. The actual instructions for the job very clearly said to ship the parts back to me. This anodizing job started on 26-Sept-2018. ------------------------------------------------------------------------ DATE: 7-Nov-2018 Topic(s): Optical FTM, HW_monit, HWmonit, MDIO to Enet Switch chips, Start work prepairing Hub shipment Received 2 Optical FTM cards from RAL on 5-Nov-2018. The working theory is that the Hub's Sensor I2C Bus can get into trouble if the I2C Buffer to the ROD is Enabled when there is no ROD installed on the Hub. The pull-up resistors for the ROD section of the overall Sensor I2C Bus are on the ROD itself. Thus when there is no ROD mezzanine on the Hub and the I2C Buffer to the ROD is Enabled then the un-defined CMOS inputs on the ROD side of this I2C Buffer can cause problems. The intent is to test running the HWmonit many times on all of the Hub cards that are available while enforcing the rule that the ROD I2C Buffer is Disabled on all Hubs without a ROD. Yuri makes a new version of the IPBus FW from which he has removed the MDIO connection to the Enet Switch chips. I verified with the scope and Hub SN-08 that the MDIO lines now look OK with Yuri's new version of IPBus FW configured into the Hub FPGA. What I see on the scope is MDC Low and MDIO is undefined. Remove the modifications from the Hub cards that were cut up to debug the FEX 8, 14 Enet - MDIO running problem: pull the Switch Phys reset push-buttons off Hub SN-07 pull the Switch-FPGA flag wires off of Hub SN-08. The intent is to ship Hub SN-07 to Yuri and get back Hub SN-09 with ROD SN-04 on it. We will probably also ship to Yuri some/all of the IPMC cards from Annecy. Hub SN-09 with ROD SN-04 was shipped to CERN 21-Feb-2018. ------------------------------------------------------------------------ DATE: 5-Nov-2018 Topic(s): TTC VME Crate Finish about a week's work getting the TTC VME Crate running. The intent is to have the Optical Output from the TTC crate contain: - LHC Beam Crossing Clock 40.08 MHz - L1As randomly spaced at a controllable rate between 1 Hz and 100 kHz. The rate can be changed or the L1As can be stopped by a VME register write. - Bunch Counter Reset this is a Short Format, Synchronous, B Channel, Broadcast Command with the value 0x01 that is automatically sent out at a fixed position in each turn. - Event Counter Reset this is a Short Format, Non-Synchronous, B Channel, Broadcast Command with the value 0x02 that is only sent out when a human writes to a specified VME register. Our collection of TTC documents for Hub testing is at: https://web.pa.msu.edu/hep/atlas/l1calo/hub/reference/ttc_documents/ The CERN TTC web area starts at: http://ttc.web.cern.ch/TTC/intro.html We need only the TTCvi and the TTCvx modules to generate the required TTC output signals. Recall that we use/need an external 40.08 MHz clock signal to go into the TTCvx module. The remaining LEMO connections between vi and vx are: TTCvi TTCvx ------- ------- Clk In Clock Out BC ECL <------------< ECL Channel Out Channel In A ECL >------------> A ECL B ECL >------------> B ECL If you want to look on a scope at the raw A Ch or B Ch output from the TTCvi module you can do that but note that you will need to supply the ECL pull-down current. As is rational ECL - the TTCvi transmitter does not provide pull-down current rather the TTCvx receivers provide the pull-down. Scope pictures are on our Hub web site. The A24 base address of our TTCvi is $C0 0000. The TTCvx has no VME access. VME access to the TTC crate is provided by the low level VME functions in Philippe's TRICS II software. The full startup script for the TTC VME crate is still under development. ------------------------------------------------------------------------ DATE: 26-Oct-2018 Topic(s): First tour of Rm 1235 Mark gives us a first tour of Rm 1235. The main section of Rm 1235 is about 30 ft in the East-West direction from the counter on the West wall to the East wall. It is about 20 ft in the North-South direction. The door to the hallway is in the SW corner. There is a counter along the West wall with a sink in the NW corner. There is an over-head cable tray running East-West in about the middle of the room. The cable tray has power outlets. The door to the alcove room is in about the middle of the North wall. The alcove room is about 9 or 10 ft North-South and about 20 ft in the East-West direction. The single width door is in the South wall of the alcove room about 5 ft from the West end of the South wall. The alcove room is painted black. It does have in/out air vents. Number of AC circuits ? The alcove room door and the hallway door are both 7 ft tall. The noise test (a Dawn 9U VME fan tray running near alcove room door) result was so so. You could hear the fan tray in the main room. Pulling the door closed reduced the noise in the main room. The back ground building ventilation air flow noise in Rm 1235 was low (much lower than Rm 1200C). We can make a temperature rise test in the alcove room, e.g. about 3600 Watts, the VME fan tray, and a good thermometer. There is no building compressed air in Rm 1235 but there is compressed air in the teaching lab just East of Rm 1235. It is through that Rm 1200C (our current lab) is about 1200 sq ft and that Rm 1235 with its alcove is about 850 sq ft. If so then the new room + alcove is about 71% the size of Rm 1200C. If we want a 3 rack test setup then this will require about 80 sq ft minimum. This would be about 10 ft min width and about 8 ft min depth. Width is based on: 2 ft air plus 3x 2 ft rack width, plus 2 ft air = 10 ft. Depth is based on: 3 ft of air in front plus 3 ft rack depth plus 2 ft air in back = 8 ft. Rm 1235 is currently used for an interesting freshman lab in vacuum and optics. ------------------------------------------------------------------------ DATE: 25-Oct-2018 Topic(s): FEX 8 and 14 Ethernet Links FEX 8 & 14 Ethernet Link Testing on Hub #1 in the 14SS: Go from the state of all Enet links working at 1 Gbps to the state pf links to FEX 8 & 14 broken (running at 10 Mbps) by loading the IPBus FW. Now load the Safe Foundation FW. Check and FEX 8 & 14 are still at 10 Mbps. Now un-plug, leave un-plugged for about 30 seconds, and then plug back in the cables for FEX 8 & 14 (that run to the little computers). See that when plugged back in these two links come back up at 10 Mbps, i.e. they did not renegotiate. Thus whatever the IPBus FW does to these Enet Switch Ports the problem seems to persist, e.g. the duration of the problem does not end when some static signal is returned to the correct state by loading the Safe Foundation FW. Testing on the Bench with Hub SN #8: Check the 4 signals that run between the FPGA and each Switch chip - these are: MDIO, MDC, Enable Loop Detect, Loop Detected This was done by adding flag wires to Hub SN #8: R2020 MDIO, R2021 MDC, R2022 ATC_Loop_Detect, R2023 Loop_Detected. This is Switch Chip A. With the Safe-Foundation FW: - At power up both MDIO and MDC go to about 1.5 to 1.8 V, i.e. undefined 3V3 CMOS signals. When the Hub FPGA finishes configuring MDC goes Low and MDIO stays undefined. - At power up Enb_Loop_Det is undefined 1.5 to 1.8 V and Loop_Deted is Hi at 3V3. After the Switch comes out of its reset Loop_Deted goes Low and no change to Enb_Loop_Det. After FPGA Config finishes Enb_Loop_Det goes Low. With the current IPBus FW: - During FPGA Config: Enb_Loop_Det undefined 1.5 to 1.8 V. Loop_Deted is Hi. After FPGA Configuration both are Low. - During configuration: both MDIO and MDC are undefined at about 1.5 to 1.8 Volts. At the end of configuration: MDIO switches between valid Hi and Low logic levels and a valid 3V3 clock starts up on MDC at about 2 MHz. Well after configuration is over: MDIO is undefined at 1.5 to 1.8 V and the valid 2 MHz 3V3 continues to run on MDC. See the scope pictures in the following web directory to see the difference in the MDIO and MDC signals with Safe-Foundation and IPBus FW: https://web.pa.msu.edu/hep/atlas/l1calo/hub/hardware/drawings/FEX_8_14_Enet_Links/ enet_swch_a_mdio_ipbus_1st.png End of Configuration enet_swch_a_mdio_ipbus_2nd.png MDIO Data enet_swch_a_mdio_safe_1st.png End of Configuration Need to review the requested state of these 4 signals as specified in the web document: ..../hub/hardware/details/hub_0_ab_fpga_signal_types.txt ------------------------------------------------------------------------ DATE: 22-Oct-2018 Topic(s): FEX 8 and 14 Ethernet Links Investigate the problem of the Enet links to FEXs 8 and 14 dropping out or dropping down to 10 Mbps. This investigation is being done in the 14SS and it is secondary to the MGT Link Tests that are the top priority for the big shelf. This is a list of what was learned so far last week and today. - Note that the Enet link to FEX 8 is Port 0 on Switch Chip C and that the Enet link to FEX 14 is Port 0 on Switch Chip A. - Using the Hub without an FPGA the MacBook in 1200C ran without error for a full week using Port 0 of Switch Chip C on that Hub. - Once the drop out/down problem starts then the Enet links from the Hub Switch to the Little Computers are still running but they only run at 10 Mbps. - Once the drop out/down problem starts then the Enet links from the Hub Switch to the HTM FPGA Logic Phys Chip do not work at all. There is no link. - Once the drop out/down problem starts then the Enet links from the Hub Switch to the HTM FPGA ARM Processor Phys Chip do make a link if they are plugged in but the link is at 10 Mbps not at 1 Gbps. - When tested with other Enet Switches it appears that the HTM FPGA Logic Phys Chip can only make a link with at 1 Gbps port. The Phys Chip on the HTM FPGA ARM Processor can make a link at 10/100/1000. - Once the drop out/down problem starts then that port on the Hub Enet Switch seems to be stuck a 10 Mbps for ever. Un-plugging and plugging back in does not clear it. Plugging into a known good 1 Gbps device (e.g. the MacBook) does not clear the "stuck at 10 Mbps problem". The stuck ports on the Hub Switch just do not re-negotiate until the Switch Chips are Reset. - The only way that I have seen to clear the stuck ports on the Hub Enet Switch is to Reset the Switch and doing that appears to always clear the problem. - I've only ever seen Port 0 and Switch Chips A and C get stuck this way. Note that Port 0 on Switch Chip B is not used. - Once brought back from the broken state than FEX 8 & 14 Enet links can run for hours of constant pinging with no errors. - Plugging a 10 only or 100 only device into a Hub Switch port brings that port down to 10 or 100 but does not break it, it is not stuck at 10 or 100. Unplug the device and the port will re-negotiate for 10/100/1000 with the next device that is plugged into it. - Broken means stuck at 10 Mbps and will not re-negotiate. - From all FEX links working at 1 Gbps to FEX 8 & 14 broken what appears to cause these Switch ports to break is Configuring the Hub FPGA with certain pieces of FW: . Configuring either IPBus or IBERT + IPBus into Hub #1 disturbs the FEX 8 & 14 Enet connections. . The disruption happens right at the end of the Configuration process, right when the FPGA is starting to do its logic, right when the FPGA is turning On which ever of the 3 Hub FPGA Status LEDs that it is going to turn On. . The disruption consists of the 1 Gbps Enet links to FEXs 8 & 14 first completely dropping out and then a few seconds later coming back up as 10 Mbps links. . Holding the Hub's PHY Chips in Reset during and after Configuring the Hub FPGA with either IPBus or IBERT + IPBus FW does not prevent the disruption of the Enet links to FEXs 8 & 14. . Configuring in either Safe-Foundation or just plain IBERT FW causes no problem to any of the Hubs Enet links. . I ran all of these tests many times. ------------------------------------------------------------------------ DATE: 3-Oct-2018 Topic(s): Modifications to balance the Hub ROD Status LED Brightness This section covers the modifications in the area of the 8 LEDs that show ROD & Hub status, i.e. LE45 through LE52. - Blue series resistors R248 & R249 remove 330 Ohm and install 2k Ohm. - Yellow series resistor R251 remove 240 Ohm and install 1k Ohm. - Red series resistors R247 & R252 remove 240 Ohm and install 1k Ohm. - Remove the current Green LEDs: LE45, LE46, and LE50 and install the higher output version of these LEDs. Check the currents with the new value series resistors on Hub card SN-08. The 3V3 supply measured 3.306 Volts The 5V0 supply measured 4.993 Volts V_Res_1 is the voltage wrt ground at the high side V_Res_2 is the voltage wrt ground at the low side LED Color V_Res_1 V_Res_2 V_LED V_Res R_Ohm LED_mA --- ----- ------- ------- ----- ----- ----- ------ 47 Red 1.546 0.046 1.760 1.500 1k 1.50 48 Blue 2.233 0.011 2.760 2.222 2k 1.11 49 Blue 2.218 0.012 2.775 2.206 2k 1.10 51 Yellow 1.453 0.038 1.853 1.415 1k 1.42 52 Red Off during bench test On the 20 Production Hub Modules the Green LEDs in the array of 40 near the top of the front panel (LE5 through LE44) will be replaced with the new higher output version of these LEDs. ------------------------------------------------------------------------ DATE: 28-Sept-2018 Topic(s): Front Panel LED Brightness Check the LED currents for the 8 LEDs: 5 driven by ROD and 3 driven by Hub. In general: the Blue and Red are clearly too bright the Yellow may be slightly too bright the Green is too dim and looks washed out The 3V3 supply measured 3.313 Volts The 5V0 supply measured 5.045 Volts V_Res_1 is the voltage wrt ground at the high side V_Res_2 is the voltage wrt ground at the low side LED Color V_Res_1 V_Res_2 V_LED V_Res R_Ohm LED_mA --- ----- ------- ------- ----- ----- ----- ------ 45 Grn 1.328 0.139 1.985 1.189 240 4.95 46 Grn 1.324 0.142 1.989 1.182 240 4.93 47 Red 1.496 0.159 1.817 1.337 240 5.57 48 Blue 2.145 0.067 2.900 2.078 330 6.30 49 Blue 2.127 0.066 2.918 2.061 330 6.25 50 Grn 1.321 0.134 1.992 1.187 240 4.95 51 Yellow 1.413 0.143 1.900 1.270 240 5.29 52 Red 1.493 0.151 1.820 1.342 240 5.59 49 Blue 2.293 0.032 2.752 2.261 2k 1.13 51 Yellow 1.464 0.040 1.849 1.424 1k 1.42 52 Red 1.558 0.042 1.755 1.516 1k 1.52 Compare the LEDs that have been used to date with the 8x light pipes: LED_LGR971_GRN Osram Part No. LG R971-KN-1 0805 2.2 V 20 mA 570-572 nm 26 mcd Lens: White, Diffused, Rectangular, Flat Top, 1.30 x 1.25 mm, 160 deg 0.9 mm max height LED_LSR976_RED Osram Part No. LS R976-NR-1 0805 2.0 V 20 mA 633-645 nm 104 mcd Lens: White, Diffused, Rectangular, Flat Top, 1.30 x 1.25 mm, 160 deg 0.9 mm max height LED_LYR976_YEL Osram Part No. LY R976-PS-36 0805 2.0 V 20 mA 588-591 nm 163 mcd Lens: White, Diffused, Rectangular, Flat Top, 1.30 x 1.25 mm, 160 deg 0.9 mm max height LED_HSMR_C170_BLU Avago/Broadcom Part No. HSMR-C170 0805 3.4 V 20 mA 473-469 nm 55 mcd Lens: White, Diffused, Rectangular, Flat Top, 1.40 x 1.25 mm, 155 deg 0.8 mm max height Verified that R249 is 330 Ohm. Verified that R251 and R252 are 240 Ohm. Within that array of 8 ROD-Hub controlled LEDs the: current estimate is that 2k Ohm is good for the Blue LEDs. current estimate is that 1k Ohm is good for the Yellow LEDs. current estimate is that 1k Ohm is good for the Red LEDs. There are 2 basic types of Green LEDs: InGaN on Saphire bright 2.8V AlGaInP on GaAs less bright 2 V Because the Vcc supply for the Green LEDs is 3.3V we can not use the InGaN type Green LEDs - nor do I think that most of the drivers, e.g. the Switch Chips, could handle at higher Vcc. What is shown in the Digi-Key comparision tables and in the manufacture's data sheets does not match - nor do I understand the binning vs part number scheme used by most of the manufactures. KingBright part number: APT2012MGC 70 mcd Colorless Clear Rect Flat 1.30x1.25mm 120 deg 0.75 mm 2.1V 20 mA 20min-60typ mcd <---- data-sheet ----> 140 deg Broadcom (Avago) part number: HSME-C170 50 mcd White Diffuse Rect Flat 1.40x1.25mm 170 deg 0.90 mm 2.1V 20 mA 18min-50typ mcd <---- data-sheet ----> 170 deg Both: Wavelength 570-574 nm To Purchase: Manufacturer Kingbright Manufacturer Part Number APT2012MGC Description LED GREEN CLEAR CHIP SMD 25 0.46480 $11.62 Broadcom HSME-C170 Digi-Key Part Number 516-3542-1-ND Description LED GREEN DIFFUSED 0805 SMD 25 0.40760 $10.19 ------------------------------------------------------------------------ DATE: 26-Sept-2018 Topic(s): Ship 47 to Alpha Today I shipped 11 Hub FPGA Heatsinks and 36 MiniPOD Heatsinks to Jamie at Alpha Metal. This should be the last of the Black anodizing that we need to Hub & HTM. To go there are still the 28 or 29 new Hub Front Panels for green anodizing and how ever many spare MiniPOD assemblies we may want to make at this time. Once this shipment is back we will have anodized a total of 29 Hub FPGA Heatsinks and 60 MiniPOD Heatsinks. ------------------------------------------------------------------------ DATE: 19-Sept-2018 Topic(s): Work to get ready for the Production Build of 20 Hub Modules Today gave the M Shop the request for making: 20x Top Brackets, 20x Bottom Brackets, and 28x Hub Front Panels. I passed to the M Ship: drawings / instructions, stock material, and examples for all 3 parts. Other mechanical parts work steps that need to be taken: debur the 11 or so raw Hub FPGA Heat Sinks clean the ?? or so raw MiniPOD Heat Sinks send for black anodizing the FPGA and MiniPOD Heat Sinks Assemble the 23 or so Hub FPGA Heat Sink + Filler Blocks including fly-cutting the bottom of the Filler Blocks Assemble all available MiniPODs and their Heat Sinks with thermal epoxy Remove the tang from 40 or 41 extractor/inserter handles Eventually need to green anodize and silkscreen the 28 Hub Front Panels Replace the Front Panels on the 8 Prototype Hub Modules Work on the additional parts that are required to complete the kit to assemble the 20 Production Hub Modules. The following items are now available at MSU and need to be delivered to Debron to complete the kit to build the 20 Production Hub Modules. Count Count Ship to Ship to Count Debron Debron Item MSU per 23-Sept 8-Mar Ord Have Number Part Number Card 2016 2017 Flag Num Qty ------ -------------------- ----- ------- ------ ---- --- --- 25 Conn_Condo_RJ45 3 29 0 % 62 70 27 FCI_Conn_74221 2 20 0 % 62 50 32 IC_74AVCH2T45 2 40 0 % 62 50 44 IC_Linear_LT1764A 2 24 0 % 62 50 47 IC_NB7VQ14M 74 700 0 % 63 1600 54 IC_XCVU125 1 8 0 % 75 20 67 Molex_877823003 1 12 0 % 66 22 68 POL_MDT040A 1 10 0 % 66 25 69 POL_PDT012A 3 30 0 % 66 70 70 POL_UDT020A 3 30 0 % 67 70 71 Pulse_HX5201NL 10 99 0 % 68 210 95 SynQor_IQ65033 1 8 0 % 64 24 96 SynQor_PQ60120 1 8 0 % 64 25 97 TE_1766500-1 1 10 0 % 69 22 98 TE_2065657-1 5 50 0 % 70 110 This is 15 items packaged in 4 boxes plus the Ultra Scale FPGAs which are still upstairs. All 15 items are now fully labeled and ready for delivery to the assembly house. Items that still need to be ordered for the MSU Final Assembly: 4x Light Pipe, Mid-Board MPT Connector, Thermal Paste ------------------------------------------------------------------------ DATE: 11-Sept-2018 Topic(s): Edit 4 of the Hub's Design Drawings, "Finished" the MSU Final Assembly on the 5 Hub Modules currently at MSU. Hub designs drawings: 7, 14, and 30 were edited so that the Optical Coupler that enables the Iso_12V power supply now has the correct reference designator, i.e. OPT1. Hub design drawing 13 was edited to add the WTERMs and the wire colors for the connection to the ATCA Front Panel Handle Switch. Worked to finish the MSU Final Assembly of the 5 Hub Modules that are at MSU and to thus make them equivalent except as noted below. This work to finish their MSU Final Assembly included: - Remove all of the special and for test modifications, e.g. On/Off Switches, Temp Sensor, Separate Phy Chip Reset, I2C and I2C Buffer flag wires, Phy Chip RGMII flag wires, LED to monitor the IPMC power control pin. - Remove the temporary FPGA Heat Sinks and install bonded final design FPGA Heat Sinks. - Install the ATCA front panel Handle Switch. - Install the IPMC to test it for On/Off power control and so that it will provide pull-up current to the Sensor I2C bus. - Install Enet Switch PROMs on all cards so that the LEDs that are controlled by the Enet Switch Chips will show: Green for 1 Gbps, Yellow for a Link, Flash Yellow for an Active Link. - Move all 5 of these Hub Modules to 3 rear mid-board MPT connectors and 1 air block. Top MTP for the ROD, Bottom and next to Bottom MTP for the Hub's MiniPOD Transmitter and Receiver (which use 1 ribbon MPT for now at least). Next to Top is and air block. - Install MiniPODs with Heat Sinks on all 5 of these Hub Modules. This work was done on Hub Modules SN: 4, 5, 6, 7, amd 8. Currently only Hub SN-5 has a ROD attached and has Fiber Optic Ribbons on its MiniPODs. Hub SN-5 has been and is Hub #1 in the 14 Slot Shelf. Hub SN-7 has been and is Hub #2 in the 14 Slot Shelf. Hub SN-8 has been and is Hub #1 in the 6 Slot Shelf. Hub SN-4 has been and is Hub #2 in the 6 Slot Shelf. ------------------------------------------------------------------------ DATE: 15-May-2018 Topic(s): Production FPGAs for Hub Received the 20 FPGA for the production Hub Modules from Dean at Stony Brook. World Courier waybill 106073428 ------------------------------------------------------------------------ DATE: 6-Mar-2018 Topic(s): Clobber the Hub's Drill Files Today while working on the HTM Card I clobbered the two drill files in the Mentor /mfg/ area of the Hub design. I probably just re-generated them but I have not check the Hub Release Proceedure to see if there are any post generation scripts or hand edits that need to be run on the raw Hub Drill Files. The Hub Drill Files in the Manufacturing Release area on the web were not touched. The sizes of the new Hub Drill Files in the /mfg/ directory and their sizes in the Manufacturing Release area on the web match but I did not diff them. Note that this also generated a new Hub drill_table_37. The Hub drill table that had been used to generate the final manufacturing data a year ago was drill_table_36. Diffing shows that these two drill tables have the same internal content. More coffee needed. ------------------------------------------------------------------------ DATE: 4-Mar-2018 Topic(s): Text Files vs Drawing Number Hub "As Built" Text Files vs Hub Drawing Numbers ----------------------------------------------------- General Introduction Drawings: ------------------------------ Drawing Subject ------- ------------------ 1 TTC Distribution 2 Readout Data Distribution 3 Ethernet Switch Connections 4 All Ethernet Connections 48 Hub Overall Block Diagram Technical Description of Hub Module: ------------------------------------ hub_0_ab_FEX_MGT_fanout_design.txt Drawing 5, 52 hub_0_ab_FEX_MGT_fanout_map.txt hub_0_ab_clock_generation.txt Drawings: 39, 40A, 40B, 41 hub_0_ab_ethernet_line_circuits.txt Drawings: 43 hub_0_ab_fpga_mgt_transceiver_usage.txt Drawings: 22, 23, 58 hub_0_ab_fpga_select_io_usage.txt hub_0_ab_fpga_signal_types.txt hub_0_ab_ipmc_connections.txt Drawings: 9, 13, 14, 16, 37 hub_0_ab_jtag_string.txt Drawings: 33 hub_0_ab_jumpers.txt hub_0_ab_led_front_panel.txt Drawings: 44, 50, 53 hub_0_ab_minipods.txt Drawings: 21 hub_0_ab_non-MGT_ROD-Hub_connections.txt Drawings: 42, 57 hub_0_ab_phys_chips.txt Drawings: 34, 35 hub_0_ab_power_system.txt Drawings: 6, 7, 8, 24, 25, 26A, 26B, 26C, 27, 28. 29, 30_Blk, 30, 31_Blk, 31, 32 hub_0_ab_switch_ethernet_base_if.txt Drawings: 17, 18, 19. hub_0_ab_switch_ethernet_jumpers.txt hub_0_ab_virtex_fpga.txt Drawings: 36, 38 Hub Technical Files Not Included in the BIG Doc: -------------------------------------------------- fanout_component_selection.txt hub_0_ab_pcb_layer_strategy.txt hub_0_ab_power_distribution_strategy.txt hub_0_ab_trace_routing_strategy.txt hub_0_ab_trace_routing_details.txt hub_0_ab_switch_ethernet_prom.txt (& other Enet Swch PROM files) Hub Drawings Not Included in the BIG Doc: ------------------------------------------- Drawing Subject ------- --------------------- 10 Drawing Number 10 - Not Used 11 mechanical 4 column LEDs 12 mechanical 1 column LEDs 15 Hub to Hub Backplane Connections 20 Drawing Number 20 - Not Ued was SFP+ conn 45 Hub FPGA Bank 68 Breakout 46 Hub South FPGA MGT Breadout 47 Hub East FPGA MGT Breadout 51 Power System - drawing not used 54 TI 40400 Standard Power Control 55 Lineage Standard Power Control 56 Lineage External LC Power Control ------------------------------------------------------------------------ DATE: 23-Feb-2018 Topic(s): Checking Hub's IPMC Recall the setup for a terminal connection to the Shelf Manager in the 14 Slot Shelf. Use the RJ45 labeled "Serial #2" in the upper left-hand corner of the front of the 14 Slot Shelf. The cable is: Computer Shelf Terminal Serial #2 DB-9 RJ45 Function -------- --------- ------------ 2 3 Data to PC / Terminal 3 6 Data to Shelf 5 4-5 Ground Signal Shield Chassis Ground Shield On the "white" PC: Start --> Programs --> Accessories --> Communications --> HyperTerminal> --> atca_1.ht shmm700 Login: and its the "shelfman" programs that is providing the management function and you interect with it via its command line interface "clia". Using "clia boards" its clear the the Shelf Manager sees the IPMC on the Hub. You see the strings "FEX-Hub" and "MSU Shelf FRU". Working on the board addressing: Physical Slot, Logical Slot, I2C address of IPMB bus, FRU Number, Backplane Zone 1 Hardware Address. Currently with the Hub in Hub Slot #1 "boards" reports slot 7 and with the Hub in Hub Slot #2 "boards" reports slot 8. ------------------------------------------------------------------------ DATE: 22-Feb-2018 Topic(s): Check out of the Hub's IPMC I have worked to verify the connections to the IPMC socket and investigated the operation of the IPMC modules that we have with whatever software is in them. - The IPMC clearly communicates with the Shelf Manager over the IPMB bus because the IPMC behaves differently when the Hub card is operated on the bench and in the Shelf and the Shelf Manager can see the Hub's IPMC from clia commands. - The IPMC can control it 4 front panel LEDs. These LEDs are seen to flash at verious times in the operation of the IPMC. - The Ethernet connection to the IPMC was tested at the level that the Ethernet Phy Chip on the IPMC can make a link connection with one of the Hubs Ethernet Switch Ports. - The Pay Load Enable connection from the IPMC to the Hub's power supply system clearly works because the IPMC can tell the Hub to power up or power down. - The front panel Handle Switch connections are clearly OK because the IPMC sees when the Handle Switch is moved and changes the state of the LEDs that it controls. The correct wiring of the Handle Switch (with whatever software is currently running on the IPMC modules that we have) is between Hub terminals WTERM71 and WTERM72 with terminal WTERM73 left open. The Handle Switch contacts should Close when the handle is moved to extract the card. When the card is fully inserted then the contacts are Open (no connection). For use with the current IPBC hardware/software we need only SPST switch connections. - Hardware Address from the Backplane Zone 1 connector - Sensor I2C Bus - Management I2C Bus - IPMC to Hub FPGA 8 lines. - Alarm signal from ATCA Power Entry Module ------------------------------------------------------------------------ DATE: 21-Feb-2018 Topic(s): Shipment to CERN I shipped Hub SN-09 with ROD SN-04 to Yuri at CERN. Initially it will be used in Building 104 for tests with the FTM and the jFEX. FedEx waybill 8040 6091 5515 ------------------------------------------------------------------------ DATE: 17:21-Feb-2018 Official Test of Hub SN-09 + ROD SN-04 ------------------------------------------- Saturday 17-Feb-2018 13:30 Things ran OK overnight except for the problem that we saw Friday afternoon of the 3rd MiniPOD circuit showing No-Link. Recall the 4 MiniPOD circuits in the overall MGT Link Display: Fiber 8 X0Y0 TX X0Y0 RX OK Fiber 6 X0Y0 TX X0Y0 RX OK Fiber 4 X0Y1 TX X0Y1 RX No Link Fiber 2 X0Y2 TX X0Y2 RX OK Recall that this run has been going since Friday evening and that all links (expect for 3rd MiniPOD) are now showing zero errors and BER of 3 10**-15 at 6.4 Gb/s. Recall that FTM is in FEX Slot #3. Recall also that for the above run Hub SN-09 + ROD SN-04 were in the Hub #1 Slot and that Hub SN-05 with its ROD were in the Hub #2 Slot and were powered up but not used in the test. Pull Hub SN-05 with its ROD from the 14 Slot Shelf and put in the storage rack. We will run the rest of this series of tests with just Hub SN-09 + ROD SN-04 in the Hub #1 Slot. Pull Hub SN-09 and do the following to try to get its MiniPOD links running. Mill 40 mils off the top surface of the center bracket. Now I can clearly see a gap between the bottom of the MiniPOD Heat Sink and the top surface of the center bracket. Pull both MiniPOD dust caps and re-seat both Prism FO connectors. Work to resetart the test (now with just Hub #1 Slot occupied). - Step 3-ROD-Config dies with "Error common 17-55 set property expects at least one object" - Step 5-hub-rod-ibert dies with "Error common 17-161 Invalid option value specified for hw_sio_rx" Continue cycling through the steps. Hang in step 5 with "Refresh HW Device" Recall that Philippe has proven that there is a mix up between "This" Hub and the "Other" Hub in the Hub Readout Data as received by the ROD. I think that this is just a labeling mixup. I need to check the emails about the 4x Hub Readout lines in the official MegArray pinout list. This is not a problem of one of the two Hubs using both of the two "instrumented" Hub readout connections on the ROD. 14:55 Test running in FEX Slot #3. All links show OK. Scope shows FTM's 160 MHz is locked to Hub's 40 MHz. Recall that Friday evening we confirmed that the Combinded Data link from Hub #1 to FEX Slot #3 was working OK and that the Hub power supply currents were OK. 15:21 Conclude the test of FEX Slot #3 zero errors on all ----- 19 links that are received by Hub & ROD BER 6 x 10**-14 Start work to move to FEX Slot #13 Left most FEX Slot. 15:31 Combined data to FEX Slot #13 is OK but I had to by hand tell the FTM receiver to invert the RX data. When doing Step 5 for Slot 13 I get "Error 44-163 hw_sio_tx Digilent bla 57644 is already in a link. Remove the existing link before creating this link." I fight this error for 20 minutes including a complete cold start of all cards. Get this error every time I execute step 5 for slot 13. Give up on FEX Slot #13 and try FEX Slot #5 15:57 FTM is in FEX Slot #5. Combined Data is OK without needing to flip by hand. Hub PS currents are OK. FTM clock is locked to Hub clock. All links are runnning OK. 16:34 Conclude the test of FEX Slot #5 Zero errors on all ----- 19 links that are received by Hub & ROD BER 7 x 10**-14 Start work to move to FEX Slot #7. 16:40 Running all links in FEX Slot #7. Combined Data is OK as received by FTM without needing to hand flip. Clocks are locked. Power Supply currents are OK - 12.81 Amps is the maximum MGT supply. All 19 links received by Hub + ROD are running OK. 17:23 Conclude the test of FEX Slot #7 Zero errors on all ----- 19 links that are received by Hub + ROD BER 6 x 10**-14 Start work to move to FEX Slot #9. 17:32 It required two tries of Step #5 for FEX Slot #9 to get things running. First try of Step 5 hung at ReFresh HW Device. Now running all 19 links in FEX Slot #9. Combined Data is OK as received by FTM but I did need to hand flip the FTM Combined Data Receiver. Clocks are locked. Power Supply currents are OK - 12.81 Amps is the maximum MGT supply. All 19 links received by Hub + ROD are running OK. 20:51 Conclude the test of FEX Slot #9 Zero errors on all 19 ----- links that are received by Hub and ROD BER 1.3 x 10**-14 Start work to move to FEX Slot #11. 21:03 Now with FTM in Slot #11. Clocks are locked. Combined Data to the FTM does NOT WORK with the FTM receiver either straight through or with it flipped. For now give up on the Combined Data to Slot #11. All 19 links that are received by Hub and ROD are running OK. Power Supply MGT current is 12.81 Amps max. Note In General: All Hub reported links show a line rate in the range 6.409 to 6.417 Gb/s. All ROD reported links show a line rate of 6.400 Gb/s. 21:31 Conclude the test of FEX Slot #11 Zero errors on all 19 ----- links that are received by Hub and ROD BER 9.3 x 10**-14 Work to understand the problem of Combined Data to FEX Slot #11 NOT Working. Do a Step #1 to configure the Hub FPGA. Skip directly to Step #4 to configure the FTM FPGA and put up the display of the FTM receiving the Combined Data. Specifically skip an Step #5 where the unused Hub MGT Transmitters are turned OFF. Hand Flip the polarity of the FTM Combined Data Receiver and the FTM starts to receive good Combined Data. Conclusions: Step #5 is turning OFF the MGT Transmitter that sends the Combined Data to FEX Slot #11. FEX Slot #11 requires a polarity flip to work. While here at the debug point try Step #5 for FEX Slot #13 again and get the same Fatal Error 44-163 as before as shown at 15:31. While trying this the first time it hung during "ReFresh HW". Start work to move to FEX Slot #4. 21:47 Running all links in FEX Slot #4. Combined Data is OK as received by FTM without needing to hand flip. Clocks are locked. Power Supply currents are OK - 12.69 Amps is the maximum MGT supply. All 19 links received by Hub & ROD are running OK. Sunday 18-Feb-2018 14:25 Conclude the test of FEX Slot #4 Zero errors on all 19 ----- links that are received by Hub & ROD BER 2.6 x 10**-15 Start work to move to FEX Slot #6. 14:33 Now running in FEX Slot #6. Combined Data is OK NO hand flip is needed. Clocks are locked. 19x links are running OK. Power Supply currents are 12.69 Amps max. 15:18 Conclude the test of FEX Slot #6 Zero errors on all 19 ----- links that are received by Hub & ROD BER 5.7 x 10**-14 Start work to move to FEX Slot #8. 15:24 Now running in FEX Slot #8. Combined Data is OK without hand flip. Clocks are locked. 19x links are running OK. Power Supply currents are 12.69 Amps max. 15:59 Conclude the test of FEX Slot #8 Zero errors on all 19 ----- links that are received by Hub & ROD BER 7.3 x 10**-14 Start work to move to FEX Slot #10. 16:xy Now running in FEX Slot #10. Combined Data is OK but it does require a hand flip. Note that FTM reports a Combined Data line rate of 6.413 Gb/s. Clocks are locked. Power Supply currents are 12.69 Amps max. Try using v5p1 of Step #5 for Slot #10. It does not show and of the 3 links between Hub and ROD (i.e. Hub Readout, ROD Readout Control, or Combined Data to ROD). It hung at "ReFresh HW Device". I now understand via email from Philippe that it should not show these 3 direct Hub <--> ROD links but that it should fix the fatal problem with the Slot #13 Step #5 v5p0 script which he says probably also affects Slot #12, #13, and #14. Try using v5p0 of Step #5 for FEX Slot #10. This looks OK but only 18 of the 19 links are running. The first "Hub1 from FEX" link is not running. All 6 FEX links to the ROD look OK. I tried flipping the polarity of this not working link but that did Not help. This first "Hub 1 from FEX" link is listed as X0Y22. This first of the "Hub1 from FEX" links also did not work with the Step #5 v5p1 Slot #10 script. 16:50 Conclude the test of FEX Slot #10 Zero errors on 18 ----- of the 19 links that are received by Hub & ROD BER 6.4 x 10**-14. The first "Hub1 from FEX" link does NOT work. It shows No-Link. Polarity flip did not help. All 6 of the FEX links to the ROD are OK. Start work to move to FEX Slot #12. 17:07 Now working with the FTM in FEX Slot #12. I did need to flip the FTM Receiver polarity to get the Combined Data link to this FEX Slot to work. The FTM reports a Combined Data line rate of 6.413 Gb/s I tried v5p0 of the Step #5 script for Slot #12 and it dies with a fatal error as Philippe expected. Try the v5p1 version of Step #5 for Slot #12. As expected this has only 16 links - the 3 direct Hub <--> ROD links are not displayed. Only 15 of the 16 links are running. The last of the "Hub1 from FEX" links is NOT working. This is listed at X0Y19. Try polarity flip of this link and it does NOT help. All 6 FEX links to the ROD are OK. As an experiment try with Step #5 Slot #10 script (with the FTM still in slot #12) to see if its missing Hub1 from FEX link would show up. First try hang at ReFresh HW Device. The second try loads OK but none of the 12 links from FEX to either Hub or ROD show and data connection. Give up. Back to v5p1 of Step #5 for Slot #12 with FTM still in Slot #12. Only 15 of the 16 links are running. The last "Hub1 from FEX" link is not running X0Y19. Power supply MGT currents are OK - MGTs are 12.69 A max. 17:44 Conclude the test of FEX Slot #12 Zero errors on 15 ----- of the 16 links that are received by Hub & ROD BER 6.9 x 10**-14. The last "Hub1 from FEX" link does NOT work. It shows No-Link. Polarity flip did not help. This is listed as X0Y19. All 6 of the FEX links to the ROD are OK. Start work to move to FEX Slot #14. 17:52 Now working with the FTM in FEX Slot #14. The FTM did require its receiver to have a polarity flip in order to get the Combined Data to FEX link to run. The FTM reports Combined Data line rate of 6.413 Gb/s The clocks are locked. Use Step #5 v5p1 for Slot #14. The first try hung at ReFresh HW Device ROD. All 16 exected links as reported by Hub and ROD are now running OK. Highest MGT power supply current is 12.69 Amps. Monday 19-Feb-2018 9:27 Conclude the test of FEX Slot #14 On 16 of 16 links ----- there were Zero errors in the data received by Hub & ROD BER 2.8 x 10**-15 Start work to move back to FEX Slot #13. 9:40 Now working in FEX Slot #13 Clocks are locked. It does require a FTM receiver polarity flip to receive the Combined Data. Combined Data line rate is 6.413 Gb/s. Try Step 5 v5p1 Slot 13 and it dies with a fatal error 44-163 as it has before. Try again with v5p1 for Slot 13 and it has the same error but notice that it gets far enough so that some MGT links are being displayed. 17 MGT Links are being displayed and it looks like these 17 include the 6 FEX to Hub FPGA and the 6 FEX to ROD and that these links are all running OK. MGT power supply current is 12.56 Amps maximum. 13:41 Conclude the test of FEX Slot #13 On 16 of 16 links ----- there were Zero errors in the data received by Hub & ROD BER 1.07 x 10**-14 Note that this final check of Slot #13 was made using version v5p2 of the Step #5 script Start work to move back to FEX Slot #10 to look for the missling link. 14:xy Now working with the FTM in Slot #10. Using the normal set of scripts it still shows a missing "Hub1 from FEX" link. Using a special Step #5 script that Philippe just made, a script that shows all "Hub1 from FEX" links, it shows only 5 links. Just doing Step #1 to Configure the Hub FPGA and then running Hardware Manager and asking it to look at all links - we still see only 5 links from the FEX in Slot #10. Note that we also see the 4 MiniPOD links and the Readout Control from ROD link. In FEX Slot #10 we think it is Lane 4 that is missing. This should be channel 49 of the MGT Fanout, which runs to pins N1 / N2 on the FPGA, and uses capacitors C737 and C738. FEX Slot #10 uses MGT Fanout Ch 45 : 50. In FEX Slot #12 we think it is Lane 5 that is missing. This should be channel 62 of the MGT Fanout, which runs to pins T3 / T4 on the FPGA, and uses capacitors C830 and C831. FEX Slot #12 uses MGT Fanout Ch 57 : 62. 15:xy Now working with Hub SN-05 which had the 1st MSU ROD on it test and verify that both this Hub's FPGA and the ROD see 6 good lanes of FEX data with the FTM in Slot #10 and #12. 16:57 Now working with Hub SN-04 which is without a ROD and has an un-thermally bonded FPGA heat sink test and verify that this Hub's FPGA does see 6 good lanes of FEX data with the FTM in both Slot #10 and #12. This Hub's FPGA Si Temp was in the range 50 - 52 deg C after Step #5 had turned off the unused MGT transmitters. With the FTM in Slot #10 it saw good Combined Data. 17:35 Now working with Hub SN-07 which is without a ROD and has an un-thermally bonded FPGA heat sink test and verify that this Hub's FPGA does see 6 good lanes of FEX data with the FTM in both Slot #10 and #12. This Hub's FPGA Si Temp was in the range 52 - 53 deg C after Step #5 had turned off the unused MGT transmitters. With the FTM in Slot #12 it saw good Combined Data. 18:10 Now working with Hub SN-06 which is without a ROD and has an un-thermally bonded FPGA heat sink test and verify that this Hub's FPGA does see 6 good lanes of FEX data with the FTM in both Slot #10 and #12. This Hub's FPGA Si Temp was in the range 48 - 50 deg C after Step #5 had turned off the unused MGT transmitters. With the FTM in Slot #10 it saw good Combined Data. Tuesday 20-Feb-2018 9:30 Now working with Hub SN-08 which is without a ROD and has an un-thermally bonded FPGA heat sink and is the card that is normally in the 6 slot Enet test Shelf. I tested and verify that this Hub's FPGA does see 6 good lanes of FEX data with the FTM in both Slot #10 and #12. This Hub's FPGA Si Temp was in the range 52 deg C after Step #5 had turned off the unused MGT transmitters. The FTM saw good Combined Data when it was in both Slots #10 & #12. ---> Conclusion - it is only Hub SN-09 that has a problem receiving 6 lanes of good FEX data from slots #10 & #12. Morning I ran Hub SN-09 with ROD SN-04 in the vertical test bench. Hub FPGA auto-configured Safe-Foundation and I don't think that ROD FPGA Configured anything. Both FPGAs were cool. I checked the DC bias level of the Fanout Chip Outputs and of the MGT Receiver Inputs, i.e. check DC levels on both sides of the DC Blocking Caps. Fanout Chip output is about 1.8 V and looks the same on the not working Slot 10 & 12 channels as it does on the other channels. MGT Receiver Input is about 300 mV and again is the same on the working and not working Fanout channels. No sign of leaking DC Blocking Caps. With the Ohm meter I see no shorts to Gnd or other planes, can see the MGT input terminator resistors, and can see the Fanout chip output resistors. Everything looks good and the same for working and not working Fanout channels. The soldering looks perfect. Afternoon Install the ROD's 4x FO ribbon cable. Change the Hub FPGA heat sink to the one with the slot cut for the Hub MiniPOD Transmitter FO ribbon. Make Cu wire clamp for the Hub MiniPOD Receiver FO ribbon and RTV the clamp in place. Note that removing the original Hub FPGA heat sink with air pressure works well - it takes about 5 minutes. Milled the new heat sink to also have an air bar. Now after the work on Hub SN-09 with its SN-04 ROD I want to re-check them with FTM in FEX Slots: 9, 10, 11, 12, 13. 16:04 FTM in Slot #12, receives good Combined Data, Clk OK, 44 deg C ----- 12x Zero errors on FEX links to Hub and ROD 1.0 x E-13 i.e. it now works. Using v5p1 at Step #5 16:32 FTM in Slot #9 Clk OK, receives good Combined Data, 46 deg C ----- 12 x Zero errors in FEX links to Hub and ROD 1.15 E-13 Using v5p1 at Step #5 16:56 FTM in Slot #11 Clk OK, NOT good Combined Data, assume that ----- the Combined Data transmitter for Slot #11 was still being turned OFF by v5p1 for Slot #9 above, 46 deg C 12 x Zero errors in FEX links to Hub and ROD 1.45 E-13 Using v5p2 at Step #5 17:11 FTM in Slot #13 Clk OK, receives good Combined Data, 46-47 deg C ----- 12 x Zero errors in FEX links to Hub and ROD 2.6 E-13 Using v5p2 at Step #5 18:02 FTM in Slot #10 CLK OK, receives good Combined Data, 46-47 deg C ----- currently 12x Zero errors in FEX links to Hub and ROD 5.7 E-14 i.e. it now works Wednesday 21-Feb-2018 9:30 Ran over night in Slot #10. This morning see 12x Zero errors ----- on the FEX links to Hub and ROD BER 2.66 E-15 9:41 FTM in Slot #12 CLK OK, receives good Combined Data, 42-43 deg C ----- currently running with 12x Zero errors on FEX links to Hub and ROD 10:10 12x Zero errors on FEX links to Hub & ROD BER 8.0 E-14 Pull the Hub+ROD out of the Shelf for pictures and to check the fiber optic cable routing. Remove Hub+ROD from Shelf to check FO ribbon routing and for pictures. 11:13 Hub+ROD is back in the Shelf with the FTM still in Slot #12. Try to power things up from scratch. Clks are locked and FTM receives good Combined Data. Hub FPGA 43 deg C. But Lane #2 from FEX as received by Hub FPGA is NOT working. Screw around for a while with various test steps and Lane #2 is still NOT working. Give up and turn off Hub+ROD, leave FTM running, do steps 1,2,3,5 v5p1 for Slot 12 and now at 11:22 we have 12 good lanes from FTM to the Hub and ROD receivers. Run in Slot #12 until 11:51 and end with 12x Zero errors 8.8 E-14 Move to Slot #10. 12:49 Running in Slot #10, Clks good, Combined Data Good, 43 deg C use Slot #10 v5p1 12x Zero errors 4.9 E-14 Pull out for pictures. 13:50 Back in Slot #10 after taking the official pictures. Everything starts up without problems and runs 12x Zero errors. Move to final try in Slot #12 13:59 Running in Slot #12, Clk OK, Combined Data good, 43 deg C 12x good Hub + ROD FEX Data Zero Errors. Shutdown to ship. UnDo the Modifications to the FTM: - Remove the MSU JTAG cable from the FTM board surface JTAG connector (not its front panel JTAG connector which we do not use). - Remove the Ext/Int Clock Selector switch: Remove switch connections to Gnd LK31 and U87 LK43. Reconnect U87 Pin #1 to its Layout Pad. This is signal Clock_Master_N. See sch pages 2 and 27. - Remove the single ended Clk Monitor from SK27 to the BNC connector. This is the 40 MHz Clk Monitor. See sch page 27. - Remove the differential Clk Monitor that ran to the 2x BNCs. This was connected to U63 pins 30 & 31 aka R92 & R93 aka L011 Pads. BNC gnd was to Gnd LNK14. See sch page 28. The was the 160 MHz monitor. - Remove the JTAG "skip select jumpers " to skip past CPLD3. Remove jumper L027 Re-Install jumpers R69, R73, R325. This is all near U52 CPLD3. Recall that U62 is the "Control FPGA" that makes the FEX Output Data and that the Control FPGA's JTAG nets are named C_JTAG. - ReInstall the 10 MiniPODs with their heat sinks and FO Ribbons. - Replace the two broken backplane MPO mid board connectors. - Power up the card and run hardware manager to verify that JTAG is seeing all of the parts and is a complete string. - Recall tht U62 is the Control FPGA that makes the FEX output data and that it is shown on sch pages 7-11 and has C_JTAG nets. - Recall that the best way to run the FTM was to make certain that the Hub was powered up first and providing backplane reference clock and then power up the FTM. That way the FTM seemed to always lock up OK with the Hub. If the FTM was turned on first, when there was not backplane reference clock, then sometimes in that situation the FTM clock would never lock to the Hub. Package and ship on Air Bill 8040 6091 5515: - The FTM card and its optical RTM that MSU received on Wednesday 31-May-2017. - Hub SN-09 with ROD SN-04 attached to it. - A USB JTAG Cable for use with the Hub module. - A static control wrist strap. This shipment made it on the plane out of town on the night of 21-Feb-2018. Review of Problems and Observations Hub SN-09 with ROD SN-04: ------------------------------------------------------------- - The Hub's MGT Transmitter for the Combined Data link to FEX Slot #11 is being turned off. - Still fatal error at Step #5 for Slot #13 with v5p1. v5p2 is OK. - In FEX Slot #10 the Hub FPGA sometimes did not receive the first "Hub1 from FEX" link which is listed at X0Y22. This is Lane 4 (0:5) MGT Fanout Ch 49 45:50 It appeared that this not working link situation could always be fixed by starting the test over again from power up. - In FEX Slot #12 the Hub FPGA sometimes did not receive the last "Hub1 from FEX" link which is listed as x0y19. This is Lane 5 (0:5) MGT Fanout Ch 62 57:62 During another test it did not receive Lane 2. It appeared that this not working link situation could always be fixed by starting the test over again from power up. - Hubs SN 04, 05, 06, 07, 08 receive 6 good lanes of FEX data from Slot #10 & #12. - The following polarity flips of the FTM's Combined Data Receiver were required to allow the FTM to correctly receive Combined Data. FEX Slot Receiver Flip FEX Slot Receiver Flip -------- ------------- -------- ------------- 3 no 4 no 5 no 6 no 7 no 8 no 9 Inv 10 Inv 11 Inv 12 Inv 13 Inv 14 Inv This list of the experimentally determined FTM receiver polarity flips required to receive Combined Data does match the Hub's MGT link specification as given in drawings 22 and 23. - If vivado hardware manager is running (and the JTAG cable plugged in) when you power up the Hub or FTM then their FPGA does not auto-configure from the on-board memory. - If the Hub is not running and providing the LHC Reference Clock to the backplane when the FTM is first powered up the often the FTM will not lock to the Hub provided reference clock once it is turned ON. - When testing Hub+ROD with FTM about 5% to 10% of the time when vivado hardware manager starts up on Hub+ROD it hangs forever doing"Refresh HW Device" on either the Hub or on the RUD. The fastest way out of this is to "background" the Refresh HW Device process, kill the vivado hardware manager and start over with that step of the test procedure. - Sometimes vivado IBERT display says a MGT link from FEX to Hub is "No Link". The currentl solution of to power cycles the Hub+ROD and start the test over. It seems like you can leave the FTM running and that its clock will re-lock to the Hub once the Hub+ROD is powered up again and the Hub's FW is configured into its FPGA. ------------------------------------------------------------------------ DATE: 22-Jan-2018 Topic(s): Receive 2nd ROD card Received the 2nd ROD card at MSU. This card is to be attached to a Hub, tested, and then shipped to CERN. This ROD arrived with it FPGA heat sink not installed and its 4 MiniPOD not installed. 3 of the 4 MiniPODs were in un-opened packages. The MiniPOD heat sinks and 4x FO Ribbon cable were not installed. ------------------------------------------------------------------------ DATE: 28-Aug:1-Sept-2017 Topic(s): Phys Chip Enet work, Sole Source Orders, Switch Test, Monday Meeting Moved the scope to the Hub SN-04 in the 6 Slot Shelf and tied it up to the building Enet so that Yuri can see it. This more or less works OK but I expect that the scope up date from Europe is pretty slot. Yuri can change the relative Rx Clk Data timing and see it on the scope. I think that he now has the extreem settings (max Clk delay and min Data delay) and now the Data transitions are almost 2 nsec before the Clk edges. It would be nice to see a full 2 nsec or a little bit more of Setup time. Start the orders of all sole source items to support: assembly of 20 Hubs at Debron, long term support of all Hub cards, all sole source items for MSU Final Assembly, and additional quantities of 4 items (Zone 1 and 2 connectors, Enet magnetics, 48V to 12V isolated converters) to support Brian's 20 HTM cards. There are about 21 of these sole source items. Two will be purchased directly from the manufacturers (PLL and power entry and converter modules). Work with the FPGA less card SN-01 to measure the Rx Clk to Data timing. I've seen 2 cases: simultanious Clk Data transitions and Data transitions just before Clk edges about a half nsec before the Clk edge. Scope pictures are in: hub/hardware/drawings/Phys_Chip/ Using the FPGA less card SN-01 and the single Enet cable to plug into the Hub backplane connectors scan through all 12 FEX slots and the Enet link to the Other Hub. They all appear to work OK. During this process I bent the SN-01 Hub backplane connector pins to FEX Logical Slot 8. I kind of fixed these bent pins but Hub SN-01 should NEVER be plugged into a Shelf. Without a Switch Chip PROM the LEDs appear to be Green ON for 1000 Base-T and the Yellow LED is ON for 100 Base-T. All Switch Ports have now been operated except for the Port to the Phys Chip on This Hub and that has been tested a little bit with Yuri's loopback. Currently SN-01 has an added Enet cable off the back that was soldered on for FEX Logical Slot 8. With the current System-D cables we can operate Hub SN-01 on the bench with 7 of its Switch Ports connected to something. The FAN_1V8 supply on Hub SN-01 is turned Off and it appears to operate OK without any forced air fan. The Switch Chips get just slightly warm and nothing else appears warm at all. Monday Morning Meeting on 28-Aug with all 6 present. We did not have an official meeting last Monday. Reviewed the tests from the last 2 weeks, e.g. Hub only scan through all FEX Slots, operation at 40 Watts. Yuri is pushing on Phys Enet. The Phys chip will loopback from its twisted pair side and run that way OK. Loopback from the MAC side does not work. Yuri is digging into Receiver Clk & Data relative timing. Brian is working on padstacks and libraries and hopes to start placement this week. ------------------------------------------------------------------------ DATE: 24,25,26-Aug-2017 Topic(s): Work in 14 Slot Shelf Thursday the on going FTM in Logical Slot 11 run is at: 10 links running at 6.4 Gbps until shutdown for power work with zero errors on 4 MiniPOD & 6 FTM links BER 8E-16 status: SysMon 44 deg C 1.811 V 0.949 V Toughbook CORE 0.94 V AVCC 1.02 V AVTT 1.22 V Toughbook CORE 7.88 A AVCC 9.38 A AVTT 12.69 A On Thursday I got the Power Distribution Box for the rack that holds the 14 Slot Shelf wired up correctly with a proper 12 AWG power cord and plug. Thus I was finally able to remove the power extension cord that had been running across the floor to the wall on the other side of the room. The 1/4" quick connect to the main 20 Amp front panel breaker in this power box is 99% OK but should be replaced at some point. Friday install ROD SN-01 onto Hub SN-05. I do not know which version of the ROD card this is. It does have one wire on its upper side. I made two supports to put under the Hub's 400 pin MegArrays so that all force to push the ROD and Hub together went straight down into the bench with no flexing of the Hub. As Ed and I had discussed, I removed the heat sink from the ROD to do this installation so that all force was applied directly over the ROD's 400 pin MegArrays. This technique basically works OK and it is much much better that what Ed and I did in his lab. I carefully cleaned and air hosed the top of the ROD's FPGA and the bottom of its heat sink before reinstalling it. The ROD's TIM was not stuck to either its FPGA or its heatsink. I used the ?? mm M3 screws, with a metal flat washer under the screw head (against the Hub) and a metal flat washer under the hex nut (against the ROD). I used ?? mm dia 4 mm tall nylon spacers at all 3 locations. This set of hardware looks all OK and final now except of something to lock the threads. Power up Hub SN-05 with the FTM still in Logical Slot 11. Let the Hub auto-Config with the Safe / Foundation design and see status: SysMon 27 deg C 1.811 V 0.955 V Toughbook CORE 0.94 V AVCC 1.00 V AVTT 1.20 V 1.20 1.79 1.82 Toughbook CORE 1.19 A AVCC 0.19 A AVTT 0.00 A 2.00 0.62 12.00 With FTM still in Logical Slot 11 and the ROD still NOT turned ON start a run using ver2 All GTY + All GTH and use Pawels TCL scrips to: setup links, reset receivers, turn off transmitters. In a little while see: 10 links running at 6.4 Gbps with zero errors on 4 MiniPOD & 6 FTM links BER 1.7E-13 SysMon 45 deg C 1.811 V 0.949 V Toughbook CORE 0.94 V AVCC 1.03 V AVTT 1.22 V 1.19 1.79 1.82 Toughbook CORE 8.00 A AVCC 9.38 A AVTT 12.44 A 1.94 0.62 12.12 Now with VIO turn ON the ROD and it looks OK as far as I know. It does not appear to have auto Configured anything. Start a new vivado and use its Hardware Manager to Configure it with the bitstream that Ed emailed to me. I think/know that this is the same ROD bitstream that we used when working in his lab. Now back to the Hub's vivado HM and creat the 160 links, i.e. click 160 times. This picks up the Hub and the ROD MGT receivers. I have not powered off the Hub so its MGT receivers still know if that are to be input inverted and its transmitters still know to be turned OFF. Status: Links 0,2,5,8 are the 4 MiniPOD links. They are the correct Chip and Quads to be the MiniPOD Receivers. Link Chip Quad Link Chip Quad ---- ---- ---- ---- ---- ---- 54 550 214 74 125 230 58 550 214 77 125 230 61 550 215 80 125 230 63 550 215 83 125 230 66 550 215 86 125 231 70 550 215 89 125 231 After running well long enough for things to settle down: Hub SysMon: 45 deg C 1.811 0.949 ROD SysMon: 53.5 deg C This is 16 links (4 Hub MiniPOD, 6 Hub FTM, 6 ROD FTM) all running error free at 6.4 Gbps. Saturday at 15:51 this run from Friday with Hub and ROD is still going with: 16 links running at 6.4 Gbps with FTM in Logical Slot 11 with zero errors on 4 MiniPOD & 2x 6 FTM links BER 1.7E-15 Hub SysMon 44 deg C 1.811 V 0.949 V ROD SysMon 53 deg C Toughbook CORE 0.94 V AVCC 1.03 V AVTT 1.22 V 1.20 1.80 1.80 Toughbook CORE 8.06 A AVCC 9.50 A AVTT 12.44 A 1.94 0.62 12.12 This was an OK Hub+ROD run with the FTM in Logical Sot 11 so now move the FTM the whole way to the left-hand side to Logical Slot 13. Just power down the FTM, physically move it, and re-Configure it using its vivado with its HM. Do not touch or change anything with the Hub+ROD. This is the technique that worked almost all of the time last week with the Hub only slot scan. Play around for 30 minutes trying to get the correct set of Links to show that they are up and running. Play with Receiver Reset, BER Reset, resize window and everything else that I could think of. In the end see: Hub MiniPODs FTM FTM ---------------- ---------------- ---------------- Link Chip Quad Link Chip Quad Link Chip Quad ---- ---- ---- ---- ---- ---- ---- ---- ---- 0 125 124 14 125 225 24 550 212 2 125 224 17 125 225 27 550 212 5 125 224 18 550 211 30 550 212 8 125 224 20 125 225 33 550 212 21 550 211 62 125 229 23 125 225 65 125 229 The above 16 Links are up and running error free and are the correct Chips and Quads for Logical Slot 13. But - vivado also shows the following two links as up and running error free: Link Chip Quad ---- ---- ---- 11 125 224 111 550 219 I worked for 20 minutes trying to get these two "false links" to go away. All power draws and temperatures look the same as recorded earlier for today, i.e. look fine. ------------------------------------------------------------------------ DATE: 22-Aug-2017 Topic(s): TCL Files to run All GTY + All GTH Pawel worked on TCL files for the All GTY + All GTH configuration to: setup the links, setup the Receiver Input Polarity (inverted or right-side-up), turn off the unused MGT Transmitters to reduce the power required by the FPGA to a value similar to the real L1Calo application Currently to use these TCL files, access them in the directory: /home/hubuser/Desktop/Instruction and then: copy the contents of each file, paste into the TCL input field text box near the bottom of the vivado window and press Return. The contents of the TCL lines includes the JTAG pod so that can not be changed and it may include a "number" which may change when the Hardware manager is re-started. Configure the FPGA with Ver 2 of the All GTY + All GTH firmware. Then in the TCL input box near the bottom of the screen paste run the following files: 1. HUB_TCL_CREATE_LINK 2. HUB_TCL_RX_POLARITY 3. HUB_TCL_TX_POWER_DOWN_L1CALO This appear to work and it executes rather fast. After setting it up this ran over night until Wednesday morning with the FTM in Logical Slot 11: 10 links running at 6.4 Gbps until Wednesday noon with zero errors on 4 MiniPOD & 6 FTM links BER 1.9E-15 status: SysMon 44 deg C 1.808 V 0.949 V Toughbook CORE 0.94 V AVCC 1.02 V AVTT 1.22 V Toughbook CORE 8.00 A AVCC 9.38 A AVTT 12.69 A ------------------------------------------------------------------------ DATE: 18,19,20-Aug-2017 Topic(s): Work in the 14 Slot Shelf, Continue 6.4 Gbps FTM Slot Scan FTM in Logical Slot 13 ran over night with zero errors which got it to a BER of about 2 E-15. The status Friday morning: Supply Core MGT_AVCC MCT_AVTT ------- -------- -------- Voltage 0.93 V 1.02 V 1.23 V Current 8.06 A 9.50 A 17.06 A SysMon 46 deg C 1.808 V 0.949 V Note that this is about 38 Watts of power in the FPGA. Recall the layout of Physical Slots vs Logical Slots in the 14 slot Asis Shelf. Physical Slot: 1 2 3 4 5 6 9 10 11 12 13 14 Logical Slot: 13 11 9 7 5 3 4 6 8 10 12 14 The following FEX Logical Slots are GTY only: 3, 4, 5, 6, 7 The following FEX Logical Slots are GTH only: 10, 11, 12, 13, 14 The following 2 FEX Logical Slots are mixed GTY/GTH: 8, 9 Move the FTM to Logical Slot 9 10 links running at 11:26 end run at 12:14 with zero errors on 4 MiniPOD & 6 FTM links BER 5 E-14 status: 46 deg C CORE 8.00 A AVCC 9.62 A AVTT 17.06 A Move the FTM to Logical Slot 8 10 links running at 12:28 end run at 13:34 with zero errors on 4 MiniPOD & 6 FTM links BER 4 E-14 status: SysMon 46 deg C 1.811 V 0.946 V Toughbook CORE 8.00 A AVCC 9.50 A AVTT 17.06 A Move the FTM to Logical Slot 6 10 links running at 13:43 end run at 15:40 with zero errors on 4 MiniPOD & 6 FTM links status: SysMon 46 deg C 1.811 V 0.949 V Toughbook CORE 8.00 A AVCC 9.50 A AVTT 16.38 A AVTT is down a little because during this run Pawel was able to turn off the MGT Transmitters on links: 65, 66, 67, 68 I have not looked up what Quads and Channels these links are. Move the FTM to Logical Slot 10 10 links running at 17:44 end run Sat. at 13:00 with zero errors on 4 MiniPOD & 6 FTM links BER 2E-15 status: SysMon 45 deg C 1.808 V 0.946 V Toughbook CORE 0.94 V AVCC 1.03 V AVTT 1.23 V Toughbook CORE 8.00 A AVCC 9.50 A AVTT 16.38 A Move the FTM to Logical Slot 12 10 links running Sat. at 13:12 end run Sat. at 14:34 with zero errors on 4 MiniPOD & 6 FTM links BER 3E-14 status: SysMon 47 deg C 1.808 V 0.946 V Toughbook CORE 0.94 V AVCC 1.03 V AVTT 1.23 V Toughbook CORE 8.06 A AVCC 9.62 A AVTT 16.44 A Move the FTM to Logical Slot 14 10 links running Sat. at 14:42 end run Sat. at 16:10 with zero errors on 4 MiniPOD & 6 FTM links BER 3E-14 status: SysMon 45 deg C 1.808 V 0.946 V Toughbook CORE 0.94 V AVCC 1.03 V AVTT 1.23 V Toughbook CORE 8.00 A AVCC 9.62 A AVTT 16.38 A Move the FTM to Logical Slot 7 6 links running Sat. at 16:49 I could not get all 6 of the FTM links running no matter what I did. The thing that finally worked was re-Configuring the Hub FPGA. I did not want to configure it with All GTY + All GTH becsuase I do not know how Pawel turned off some of the unused MGT Transmitters and thus it would have run at full 17 Amp load on AVTT. So I configured it with All GTY and that got me all 6 FTM links from Logical Slot 7. Before the Hub re-Configure it was either Ch 0 or 3 in Quad 132 that would not come up. end run Sun. at 14:17 with zero errors on 6 FTM links BER 2E-15 status: SysMon 40 deg C 1.811 V 0.952 V Toughbook CORE 0.94 V AVCC 1.02 V AVTT 1.22 V Toughbook CORE 4.69 A AVCC 3.94 A AVTT 14.50 A After this long over night run with FTM still in Logical Slot 7, I tried a 5 minute run with All GTY + All GTH and all 10 links came up fine. Move the FTM to Logical Slot 11 9 links running Sun. at 14:44 using All GTH still running Sun. at 16:40 with zero errors on 3 MiniPOD & 6 FTM links BER 2E-14 status: SysMon 39 deg C 1.808 V 0.952 V Toughbook CORE 0.93 V AVCC 1.02 V AVTT 1.20 V Toughbook CORE 4.56 A AVCC 6.50 A AVTT 3.06 A During one FTM move, while resting my hand against the Shelf to help steady and align the card with the slot guides, I hit the Hot Swap button on the left most Fan Tray. The Shelf Manager brought the fans up to full speed as part of the its Hot Swap proceedure. The solution seems to be just to hit the Fan Tray Hot Swap button again and then after 30 sec or so the Shelf Manager started to step the fans down to Level 6. After all of this stabilized I tied up the terminal to the Shelf Manager to verify that all was OK and fanning at Level 6. Using 2 vivados, one for Hub and one for FTM seems to work well most of the time. The FTM vivado remembers the correct bitstream file to load into FTM. Once FTM is moved and configured and running again just use the Rest Receivers in the Hub's vivado and with luck 6 will come up. The 4 MiniPOD links continue to run when FTM is down. ------------------------------------------------------------------------ DATE: 17-Aug-2017 Topic(s): Work in the 14 Slot Shelf The All GTH FW with the FTM in Logical Logical Slot 13 ran over night with zero errors on the 3 MiniPOD and 6 FTM links. Pawel provided the All GTY + All GTH firmware. Try running this with the FTM still in Logical Slot 13. For the first time this should let us see all 4 MiniPOD links. This All GTY + All GTH configuration pulls a lot of MGT_AVTT power. Philippe's Toughbook power supply display is: Supply Core MGT_AVCC MCT_AVTT ------- -------- -------- Voltage 0.94 V 1.02 V 1.20 V Current 8.08 A 9.62 A 17.06 A The Si temp is 47 deg C and there is about 38 Watts of power being dumped in the Hub FPGA and the Fan Level is 6. Let this run over night. This is 10 MGT links running with the MGT_AVTT supply near its maximum load and thus probably making lots of noise. ------------------------------------------------------------------------ DATE: 16-Aug-2017 Topic(s): Work on the Shelves, All 40 GTH, and on the FTM module Both Shelf Managers are now setup and working OK. Currently if you power cycle a crate you should not need to do anything special to get the correct fan speed and such. The only two things that have been changed are the Time/Date and the MinFanLevel. Time/Date is set in the normal unix way: > date MMDDhhmmYYYY and then you need to tell the battery operated cmos clock to move to this new time setting by: > hwclock -w On the Comtel Shelf Manager I also had to connect the battery. The minimum fan level is set in the shelfman.conf file: On the 6 Slot Shelf this is: /etc/shelfman.conf.comtel6 On the 14 Slot Shelf this is: /etc/shelfman.conf.asis The original shelfman.conf.comtel6 was saved for reference. The original shelfman.conf.asis was lost. The easiest command to check and see what the fans are doing at the current time is: > clia fans At this time I have no plans to change anything else in the Shelf Manager - perhaps just adjust fan speed if necessary. The MiniPOD heat sinks on the FTM card are a problem because they make the card too tall to fit into the the slot immediately to the left of the Hub and the right most slot. I do not want to just remove the heat sinks. I pulled off as a unit all 8 of the MiniPODs, with their heat sinks, and with their fiber optic ribbon cables. I put dust caps on all fiber optic cables and I put pickup caps on all MegArray connectors both on the MiniPODs and on the FTM pcb. All of these MiniPODs, heatsinks and fiber optic cables were packaged in a labeled box and put in the NW blue cabinet. Today with Pawel we tried for the first time the All 40 GTH configuration. With all 40 GTHs running, in the ToughBook display I see (all running at 6.4 Gbps all Trans & all Rec): Supply Core MGT_AVCC MCT_AVTT ------- -------- -------- Voltage 0.94 V 1.02 V 1.20 V Current 4.69 A 6.62 A 3.19 A Estimated 3.06 A 6.75 A 2.66 A Amps from Pawel Vivado System Monitor showed 38 deg C Si temperature in the Hub's FPGA and 1V8 voltage of 1.811 and CORE voltage of 0.952. When checking the Si temp 5 minutes seems like fully enough time for things to settle to a new value after making a heat or cooling change. I moved the FTM to Logical Slot #13, i.e. furthest slot to the left when viewed from the front). Started FTM and Hub with the All 40 GTH configuration. See the 3 MiniPOD links and the 6 FTM links running with zero errors after an hour or so. Leave this way for an over night run. At the end of the date the currents, voltages and Si temp looked the same. ------------------------------------------------------------------------ DATE: 15-Aug-2017 Topic(s): Work with Andrew and Pawel on FTM to Hub and Hub Power Tests in 14 Slot The first test today was running a configuration from Pawel with all 40 of the GTY Transceivers turned ON. With all 40 GTYs running, in the ToughBook display I see (all running at 6.4 Gbps all Trans & all Rec): Supply Core MGT_AVCC MCT_AVTT ------- -------- -------- Voltage 0.94 V 1.02 V 1.23 V Current 4.50 A 3.81 A 14.44 A Estimated 3.27 A 3.26 A 15.00 A Amps from Pawel Vivado System Monitor showed 37 deg C Si temperature in the Hub's FPGA. With the all 40 GTY transceivers running Configuration the 6 IBERT links from the FTM in Logical Slot 4 did show up as running without error at 6.4 Gbps. Andrew did a lot of work to learn the test systm, practice running FTM to Hub tests and set up a directory structure to capture results from the tests. ------------------------------------------------------------------------ DATE: 14-Aug-2017 Topic(s): Monday Meeting, Work with 6 Slot and 14 Slot Shelves, Programming the Flash Configuration Memory Monday Meeting with: Yuri, Pawel, Brian, Wade, Dan. Yuri reports that he sees the right number of received packets come out of the Phys chip and that they have the correct number of bytes in them but that they are all marked as invalid. I think that I understand that correctly and that the data sent out by the Phys chip is correct down to the level of the actual 1s and 0s inside the packets, i.e the transmitted data is fully correct. Brian is working on pad stacks for the HTM - I think the pad stacks for some of the connectors on that card. I reported on the current setup and use of SN-04 and SN-05 hubs: 04 in 6 slot for Enet development for the past 12 days and 05 in the 14 Slot crate with FTM getting ready for the FEX Slot Scans. Talked a little about how is the Enet test going to work during MSU Production Testing of Hubs, i.e. on the bench throgh an octopus cable or in the 14 Slot Shelf through RJ-45s on the FTM cards. Finally got the Terminal connection to the Shelf Manager in the 6 Slot Shelf working. It is 115,200 baud, 1 stop bit, no parity, no flow control. The required cable pin out is: 6 Slot Shelf Manager RJ-45 RS-232 Connector PC 9 Pin RS-232 Pin Number and Color Conn Pin Number Signal ------------------------ ---------------- ---------- 4 Blue 5 Signal Gnd 5 Blue/Wht 3 Data to SM 6 Green 2 Data to PC Summary of the Shelf Managers and their Unix OS 14 Slot: PP Shelf Manager version 3.4.0 from 29-May-2014 Unix 2.6 from May 2014 PP model 700 ShMM ARM Processor 6 Slot: PP Shelf Manager version 2.8.2 from March-2012 Unix 2.4 from March 2012 PP model 500 ShMM MIPS Processor For now my intent is to only set the Date/Time in both Shelf Managers and to edit the correct startup configuration file to set the desired MinFanLevel to provide good cooling for the electronics but not make any more noise than necessary. I prefer to make no other changes to the Shelf Managers for now. From Pawel here are the instructions to program the Flash Configuration Memory on the Hub Modules: 1. Open Hardware Manager 2. Right click on the xcvu125 3. Add Configuration Memory device 4. choose: mt28gu01gaax1e-bpi-x16, click OK 5. Add the configuration file: mcs and prm 6. Click Apply and then OK. 7. Wait about 10 minutes. 8. Done. ------------------------------------------------------------------------ DATE: 10-Aug-2017 Topic(s): Work with the 14 Slot Shelf Started working on the 14 slot shelf on Tuesday. The rack that I know was in the HEP area in the sub-basement is gone. This was the nice rack on wheels that I setup for Roger with the Magic Bus crate. The two power supplies that I modified to run the Magic Bus crate were still in the rack but it is gone from the sub-basement and a lot of new stuff is in that "room" and the whole West side of that "room" is all changed. Pull out of Rm 1200C the tall rack on wheels that had the VME Communications crate, the VIPA crate for readout, and a Run II Trig Framework crate in it. Pulled all of that stuff out of the rack and cleaned the rack inside and out. I installed the 14 Slot Shelf in it using support"L" brackets. I have 3 more pairs of "L" brackets in this rack in case we want shelfs for the 17 little Enet Test computers. I installed the 14 Slot Shelf up rather high in the rack so that it is at a level that makes it easy to move cards around in this Shelf, e.g. to walk the FTM or to swap Hub cards. The Cable Management Bracket that you can install near the top front of this Shelf gets in the way of moving cards around so for now I have not installed it. The 4 power supply modules are not setup the way that I though they would be. All 4 power supply slots deliver power to a common bus. This common bus delievers the 48V power up to te backplane on both the right and left sides but this is a common bus. This common bus then drives both the "A" and "B" buses to the cards in this crate. There are not some power supply modules associated with the "A" bus and other power supply modules associated with the "B" bus. We have the 2500 Watt modules. For the initial tests I'm running just one power supply module and running it from 120 VAC. The unused Cable Management Bracket and the 3 currently unused power supply modules for this crate are under the CMX test stand L1Calo crate. We now have the user's manual for the 14 Slot Asis Shelf. The sales person tht Yuri worked with to get this crate pointed us to the corrct person in the company and they supplied us with the manual. We also now have two Pigeon Point Shelf Manager manuals - one about the Shelf Manager mezzanine card and one about the Shelf Manager software. On Thursday I was finally able to make a working terminal connection to the Shelf Manager. This runs RS-232 at 115200 baud (why). The terminal emulator on the "white" computer (CMX test, VME development, LArTPC development computer) will run at that baud rate. It is running with 115200 baud, 1 stop bit, no parity, and no flow control. Comm #1 9 Pin RS-232 on the PC Pin Color in RS-232 Num Function Cable Scrap --- ---------------- --------------- 1 Carrier Detector Black 2 Received Data Red 3 Transmit Data Brown 4 Data Term Ready Green 5 Ground Yellow 6 Data Set Ready Orange 7 Request to Send Violet 8 Clear to Send Blue 9 Ring Indicator Gray RJ-45 RS-232 on the 14 Slot Shelf Front Panel Pin Color in Enet Num Function Cable Scrap --- ---------------- ------------- 1 Request to Send Orange/Wht 2 Data Term Ready Orange 3 Transmit Data Green/Wht 4 Ground Blue 5 Ground Blue/Wht 6 Received Data Green 7 Data Set Ready Brown/Wht 8 Clear to Send Brown Cable Setup on 14 Slot Shelf Using No Hardware Flow Control Comm #1 RJ-45 RS-232 9 Pin PC Conn. Shelf Front Panel Signal -------------- ----------------- -------- 2 3 Data to PC 3 6 Data to Shelf 5 4 and 5 Ground Shield Chassis Chassis Gnd It's clear that a lot of setup of the Shelf Manager and its unix is needed before the Shelf Manager should ever go on the network. The main command to operate the Shelf Manager is "clia" command line interface agent. As delivered to us: the account to run it is wide open, all the network setup is wrong, un-secure programs are running, e.g. ftp and telnet. What we received is PP Shelf Manager version 3.4.0 from 29-May-2014. My guess is that the small 6 Slot Shelf is much the same. There is a lot to learn about this and its small ROM unix but I think that we can do a lot of what we actually need just from the terminal - as the web and snmp and all of that is just on top of the command line stuff. I think that the ShMM in the 14 slot shelf is a Pigeon Pt 700 and that the one in the 6 slot crate is a PP 500. ------------------------------------------------------------------------ DATE: 7-Aug-2017 Topic(s): Monday Meeting Wade, Pavel, Yuri, and Dan for the Monday meeting. Pavel has been at CERN doing CMX VME work that I do not know any details about. He will be headed from Poland to Sweeden at the end of the week. Discuss the 12 slot test and how many scans are needed and what FW is needed. No new FW is needed immediately to get the FTM - Hub - ROD setup running in the 14 slot crate. Expect to receive the FTM & ROD from Ed tomorrow. Discuss the problem of always getting vivado to show known working links as up and running OK. Pawel says that he will send the how to program the Flash Config Memory write up. Yuri reported on his work on the Hub's Enet and IPBus stuff. He is now seeing received packets from the Phys chip but they are all marked as bad data. May try a scan of the timing of the received data bus. ------------------------------------------------------------------------ DATE: 2-Aug-2017 Topic(s): Enet Hub Test Setup Hub SN-04 is running in the 6 slot shelf with its JTAG connection to the hubdev machine. I have moved the hubttc machine to the Rm 1200B lab so that it is also next to Hub SN-04. The hubtcc machine has the VC709 demo card with it. Yuri has used this demo card to check the Ethernet and IPBus operation to the FPGA. ------------------------------------------------------------------------ DATE: 31-July-2017 Topic(s): Monday Meeting Monday meeting with: Wade, Pawel, Brian, Dan. Discuss Wade's idea of taking Hub to Argon for development with Felix. An interesting point is that Argon wants to work on Hub's reception of the trigger information data stream and separation of the LHC Clock. Could also work on the data readout direction of information to Felix. For a test could go full slice: 12x HTM, Hub, ROD, Felix. Need to talk with and include Ed. Brian reports work on the pad-stacks for the geometries that HTM requires. I'm pushing on Final Assembly of 2 Hubs, one for Enet test setup, one for 12 slot test. Wade says that Andrew may be able to help with 12 slot shelf setup. ------------------------------------------------------------------------ DATE: 24-July-2017 Topic(s): Monday Meeting Monday meeting with: Wade, Yuri, Pawel, Brian, Philippe, Dan. Yuri will be away starting tomorrow through Aug 2nd. Philippe will be mostly away starting July 31 through about Aug 18. I will push on Final Assembly of 2 more Hubs so that we can run at MSU an Enet test setup for Yuri and the 12 slot FEX test. Brian is working on the Mentor database for the HTM as it is the database that will let him import geometries from the Hub and from the FPGA company's card design. He reports that it can not work with ascii geometries. Philippe says that for Yuri's Enet work on the Hub card that either the hubttc machine with its installed Enet tools, e.g. wireshark, can go to the lab Rm 1200B or that he can install these Enet tools on the hubdev machine that is in Rm 1200B. ------------------------------------------------------------------------ DATE: 19-July-2017 Topic(s): Work while at Ed's lab Tues. 11-July About 1 hour working with Ed - see the lab, unpack the box, discuss with Ed what we should try to accomplish on this trip. Wed. 12-July No current version of Vivado has been installed on Ed's lab machine - only a 2015 version of Vivado exists and it does not work with the current version of components in the Hub FPGA bit-streams. No success installing 2017 Vivado because of problems with the Xilinx distribution system. Nothing useful accomplished until Yuri arrives mid-afternoon with his laptop. Using Yuri's laptop we make the Hub SN-02 (without ROD) run the MiniPOD loop test and receive data from "our" FTM. Thur. 13-July Xilinx distribution system now working - the 2017-1 Vivado installs with little problems, run again the MP loop test and the FTM data test to Hub SN-02 - all looks OK, then install a ROD on the Hub SN-02 and make quick tests, then make a serious IBERT test from FTM in FEX slot #4 to the Hub and ROD, both Hub and ROD receive IBERT data on all 6 lanes without error with runs of about 20 minutes, collected labeled eye diagrams from both ROD and Hub and verified that errors injected at FTM are seen as expected on both receivers. What appears to work: +12V Power to ROD still running on 3 Amp Hub fuses, Power Control to/from the ROD, Automatic switching of the JTAG string to include the ROD once it is powered up, ROD runs from the Hub's Reference Clock ??? The ROD's Ethernet connection works OK using the Hub provided magnetics and front panel RJ-45 and works OK when running through the Hub's Ethernet Switch. Note that Ed's current ATCA crate is 5 slot, has small inadequate fans, and that its backplane is rated for 6.2 Gbps per lane. He has no blank front panels. I try to keep cardboard covers over the open back and open front slots but folks don't take cooling seriously. Hub Si temp is OK. FTM and ROD Si temps are ?? ??. Note that yesterday or today we did see the FTM get stuck at something like a 5.6 Gbps line rate when you check it in loopback, the solution seems to be to keep it powered off until the Hub is up and running, and only then power up the FTM. The problem is clearly the TI CD???? clock generator in the FTM running off to its high end and getting stuck there. Setting up our FW for the FTM to hit the reset on this clock generator is another way to take care of this problem. Fri. 14-July Much of the day spent trying to get the eFEX to Configure - the problem was the aggressiveness of the new Vavido Hardware Manager to take control of configuration - the simple solution: kill the Vivado Hardware Manager, power cycle the eFEX, wait for the eFEX to fully configure all of its FPGAs from its Flash (or whatever and its DONE LED comes on), then startup the Hardware Manager, and configure the eFEX output FPGA with the IBERT bitstream. Ed in the driver's seat gets the IBERT test from eFEX to ROD and Hub going OK with no errors in 20 minutes or so, test injected an error into one eFEX lane, also looked at eye diagrams for that lane. Verified that the eFEX is locked to the Hub's backplane Reference Clock (by showing that its MGT PLLs are not locked if the Hub is Off). Ed works on the ROD's Enet FW to fix some reset problem and then Enet works to the ROD from the Hub Front Panel and through the Hub's Switch. The transmit Enet from the Hub is working but not the Enet receiver. Working to understand if there is a Hub Phys problem or a MAC problem. Recall that all work to date at Ed's is with Hub SN-02. Sat. & Sun. Could not go to Ed's lab because of power outage and lack of safety training. Old Cavendish and the Eagle. Pawel starts back early Sun AM. Saw some cricket but still don't fully understand it. Mon. 17-July Verified from the Phys chip datasheet that its activity LED should flash with either Rec or Trans data by the Phys (does not require MAC). The Phys to This Hub's SWCH is U22 with LEDs LE41 Grn Link and LE42 Yel Activity. We see Grn On and Yel blink when Yuri sends data from his laptop to Hub. Yuri made code that removes the MAC and ties ChipScope to the 5/6 signals from the Phys Receiver: RX_D_3:0, RX_DV, RX_Clk. When sending data to the Hub ChipScope shows activity on these lines, i.e. 4x RX_Data, RX_DV, and RX_Clk. Between Enet packets the, i.e. when RX_DV is Low the state of the 4 RX data lines matches what the Phys Chip datasheet pg 27 for the states for these lines when nothing is coming in. Phys U22 goes to SWCH-B Port-4 which has LEDs LE31 and LE32 which look as expected for now. I now understand more about how the Enet UTP Auto- Negotiation works, i.e. you advertize the Best that you can do not the Range that you can do. Thus I will change Final Assembly to advertize 1000 Base-T Full Duplex by default, i.e. Mode_3:0 equal to 1100 that is: Mode_3 install R1907 (R1957) 10k Ohm Mode_2 install R1908 (R1958) 10k Ohm Mode_1 install R1910 (R1960) 1k Ohm remove R1909 (R1959) Mode_0 install R1912 (R1962) 1k Ohm remove R1911 (R1961) Note that this is how Hub SN-02 is already setup and it does not really make any difference anyway as the Switch port can connect to any of the standards and if necessary the Phys can make a link to any of the standards even if they fail at auto-negotiate (by just looking at the incoming data). Work with Ed on installing a ROD on Hub SN-03. Recall that the intent is to leave Ed with two working ROD+Hub setups. This time I want to include the bolting together hardware in the assembly. A concern is how close some 2V5 pads are to the screw hole in the ROD's NW corner. For now we will use a nylon spacer (instead of the brass ones) and a nylon washer on top of the ROD (instead of the normal steel ones) at just this one location. Recall that there is a 4mm spacer between the cards and washers on both the Hub and ROD outer surfaces. I'm using the 14mm long screws which look fine for length but I need to get them with locking compound. This time I pushed the MegArrays together. I do not like it at all. Too much force is required. It's best to press in the south edge first and then roll up to the north edge. We did this with the heat sink on the ROD (like last time) and that is a big mistake. We need to remove the ROD heat sink so that you can press directly over the two MegArrays. Pressing on the ROD heat sink clearly flexes the ROD pcb. I need to make a jig for this operation that supports under the Hub and lets one press down on the ROD without its heat sink. This was way scary. Ed works to bring up the eFEX, Hub, ROD system still using the Hub SN-02 before we move to the untested Hub SN-03 with its just installed ROD. As is almost usual it took two tries to get Vivado to see all of the running Links, 6 on ROD and 6 on Hub. SN-02 is fine move to SN-03 with its just installed ROD. After a small fight with Vivado, SN-03 with its just installed ROD is running all 6 lanes OK to both ROD and Hub FPGAs from the eFEX. Ed tries experiments using the Equalizer in the Hub's MGT Fanout and a test running 32 bit prd vs the 8 bit prd that we have been using for all Hub MGT tests to data. Everything is at 6.4 Gbps. Looking at a few unorganized eye diagrams things probably look better with the equalizer On. Over night leave the system running 32 bit prd from eFEX to Hub and ROD with the Hub's MGT Fanout Equalizer turned On. ROD Si temp is 68 deg C all MGT Quads are powered. Hub Si temp is 37 deg C only 3 MGT Quads are powered. Tues. 18-July Ran overnight from Ed's eFEX to Hub SN-03 and its ROD. This is at 6.4 Gbps in 32 bit prd with the Hub's MGT Fanout Equalizers turned On. This ran overnight with zero errors on the 6 Hub links and zero errors on the 6 ROD links. eFEX was in slot #4. This got us to a bit error rate of 3 x 10**-15. Working to setup the Two Source Test Shelf Slot Contents ---------- -------- 5 eFEX 4 3 FTM 2 1 Hub + ROD This lets us fit in the with the overly tall MiniPODs on both the FTM and Hub and should give us good air flow between cards. In the initial test of this setup ROD is seeing all 12 of the live "FEX" input signals and Hub is running its FEX slot #3 FW which lets us see 6 signals from the FTM and 2 signals from the eFEX. As an alternative we could run the Hub's FEX slot #5 FW which would let us see the 6 signals from the eFEX and 2 signals from the FTM. Ed looked into building a version of Pawels FEX MGT test FW that would let us see all of the eFEX and FTM channels in the above setup (i.e. to see all of Quads: 125, 126, 127, and 130) but he does not have license to build for UltraScale with his 2015 version Vivado or any licenses for 2017 Vavado. We ran this for a while at 7 bit prd and at 31 bit with the equalizer On and it looked OK. Then Yuri wanted to test with Hub card SN-02 so we pulled out SN-03 and put in Hub SN-02. Yuri worked for a while and then we setup for eFEX and FTM data to flow into Hub SN-02 for an over night run. This is with 31 bit prd at 6.4 Gbps with the Hub's MGT Fanout Equalizers all On. Wed. 19-July Only a half day at Ed's lab. The test setup: eFEX and FTM data flow to Hub SN-02 with 31 bit prd at 6.4 Gbps and Hub's MGT Fanout Equalizers all On ran overnight with zero errors on the 12 ROD lanes and 1 error on 1 lane and 2 errors on another lane of the 8 Hub lanes that are visible with the Hub's FEX slot #3 FW. We are not confident that these 3 errors were not already in the system at the conclusion of our playing around with things Tuesday night right before we went for a drink by the river - a typical example of why one wants computer control and logging of test setups. We will continue running this "Two IBERT Data Source" setup for a couple of days and use both the Hub's FEX slot #3 and slot #5 FW so that we can in combination look at all 12 data source lanes as received by the Hub. The ROD FW for this test always shows us all 12 lanes. Recall the eFEX + FTM + ROD/Hub card setup as shown in the Tuesday entry above. The mapping for data from FEX 3 and 5 on the Hub is: FEX Hub Quad Slot Lane Quad Channel ---- ---- ---- ------- 5 5 130 0 5 4 130 1 5 3 126 2 5 2 126 3 5 1 127 0 5 0 127 1 3 5 125 0 3 4 125 1 3 3 125 2 3 2 125 3 3 1 126 0 3 0 126 1 For this test Two Data Source test from FEX #3 & #5 all Hub Quads are GTY. Discuss various small points with Ed about how we want to handle the ROD+Hub cards. ------------------------------------------------------------------------ DATE: 5-July-2017 Topic(s): Sketch of the last three weeks: June 12 Monday meeting: status of Hub board, what next, focus of the tests while Yuri and Pawel are here. June 14 through June 28 Pawel and Yuri are here at MSU. June 16 over night Hub SN-02 runs with 3 MiniPOD links at 6.4 Gbps June 17 over night Hub SN-02 runs with 3 MiniPOD links at 9.6 Gbps June 18 over night Hub SN-02 runs with 3 MiniPOD links at 11.2 Gbps June 19 Work on and finally get the FTM to lock onto the backplane reference clock from the Hub June 20 Over night with FTM in slot 5 to Hub SN-02 at 6.4 Gbps 6 links June 21 Over night with FTM in slot 4 to Hub SN-02 at 6.4 Gbps 6 links June 22 Over night with FTM in slot 3 to Hub SN-02 at 6.4 Gbps 6 links June 23 Over night with FTM in slot 3 to Hub SN-02 at 10.24 Gbps 6 links June 24 Over night with a Pawel Aurora test on 3 MiniPOD links June 25 same as the 23rd June 26 SN-02 ran some test in the crate over night June 27 Over night with FTM in slot 5 to Hub SN-03 at 10.24 Gbps 6 links June 28 Over night with FTM in slot 3 to Hub SN-03 at 6.4 Gbps 6 links June 28 Over night with FTM in slot ? to Hub SN-03 at 6.4 Gbps 6 links June 30 First on SN-03: repeat the MiniPOD tests on it at 9.6 Gbps 3 links and repeat all 3 slots of FTM tests at 6.4 Gbps with all 6 links, then remove the MiniPODs, and pack up SN-03. Then on SN-02: repeat the MiniPOD tests on it at 9.6 Gbps 3 links and repeat all 3 slots of FTM tests at 6.4 Gbps with all 6 links, leave the 10 Gbps MSU MiniPODs on SN-02, and pack up SN-02. Then pack up the MSU FTM. Pack up the overall shipment to Cambridge. It includes" - Hubs SN-02 and SN-03 - the MSU FTM card - the bare ROD pcb and the bare Hub pcb - the ROD to Hub M3 mounting hardware - two static control wrist straps - an MPO fiber optic ribbon cable and MPO connectors - a Hub front panel J2 "access cable" JTAG, I2C, BNC - 3 Amp, 5 Amp, and 7 Amp fuses for the Hub The FedEx Air Waybill tracking number is: 8040 6091 5537 weight 37 lbs July 3 Monday meeting: Pawel is away, review items from the Yuri Pawel work at MSU that need a final clean up, review the initial tests at Cambridge of just the Hub (to verify that the Hub arrived and is working OK and that Vivado at Hub bitstreams at Cambridge are all OK) and then the tests with the ROD, review the 14 slot MSU tests with ROD and Hub that I need after the Cambridge trip, HTM report: schmatics almost in, need the database to import the components from the FPGA Brd and from the Hub design (the database has the translators), need Vivado to work with the FPGA boards. Receive a note from Richard Shaw at about noon that he has the box. It was a very fast trip and appeard not to have any customs delay (vs the typical 3 day delay getting into CERN). ------------------------------------------------------------------------ DATE: 9-June-2017 Topic(s): Work on Hub SN-02 Configuration The push today was to work with the Configuration Flash Menory on the Hub. Pawel loaded the "Safe" configuration into the Flash memory and from vivado he could tell the Hub FPGA to configure itself from the Flash and that worked. At first Configuring automatically at card power up did not appear to work. If the JTAG cable to the running vivado system is plugged into the Hub at the time that I turn ON the Hub, then the Hub does not automatically Configure from its Flash memory. If the JTAG cable is not plugged into the Hub when I turn it ON, then the Hub automatically Configures from its Flash. It only takes perhaps 7 or 8 seconds for it to Configure. So my guess is that the automatic power up Configuration function of the Hub card is more or less working OK. I assume that when the JTAG to vivado connection exists at the time of power up that vivado more or less takes control of things and must block the automatic Configuration from Flash. Need to read more to check the details on this. ------------------------------------------------------------------------ DATE: 8-June-2017 Topic(s): Work on Hub SN-02 Yesterday Hub SN-02 was far enough through Final Assembly that I tried to power it up. The 48V output from the ATCA Power Entry module would just pulse on once ever 2 seconds or so. The Iso_12V supply was jumpered to run all the time. The problem appears to be that the ATCA Power Entry will not enable its 48V output if it sees a load at the time that the ATCA Backplane Enable goes true. That is the Power Entry module is more or less enforcing that the IPMC will delay enabling the Iso_12V supply until it gets the OK from the Shelf Manager. I added an Enable Iso_12V switch to work around this problem. See the Final Assembly doc for details. This morning was more machining of the prototype Hub FPGA Heat Sink that I get back from the Machine Shop. The trench for the MGT_AVTT capacitor must move East and I need to mill the NW corner of the Fill Block to make space for the MGT_AVCC capacitors. See the Final Assembly doc for details. JTAG was not working on Hub SN-02. Traced this to the wrong value of the ROD Power Good signal "coming back from the ROD". This is probably caused by pull-up from the un-configured Select I/O pin on the Hub FPGA that receives the ROD Power Good signal. Changing resistor vaiues should fix this. See Final Assembly document for details. Pawel was able to Configure SN-02 and change its LEDs. Still need to finish and check the Clocks on SN-02 and figure out what to do to make a delayed Iso_12V enable. ------------------------------------------------------------------------ DATE: 5-June-2017 Topic(s): Monday meeting, Rm 1200 meeting, Shipping information Normal Monday morning Hub meeting with everyone present, two from Europe even though its a CERN holiday and four at MSU. Philippe is pushing to get the HubDev machine working and Pawel is making some tests on it. Yuri and Pawel will be here next week on Wednesday June 14 at 18:31 from Detroit, flight KL5438 and depart on Wednesday June 28 at 13:45 to Detroit, flight KL5793. I sent out a note requesting about 8 changes in the Safe Hub design and now including the spare oscillator input to the reference selector switch. Report that Hub power supplies look OK for now: sequence, noise, stability. Need to figure out Rm 1200 setup and FTM setup. Brian reports that the HTM schematics are going well. From MSU the Polycom video meeting ran through the Hub switch. Sent the custom broker shipping information for the ATCA Shelf to Yuri and CERN Logistics. Meeting with Wade, Philippe, Brian, and Dean about space in Rm 1200. The big unknown is where the setup for the small muon drift tubes will go. For now as a temporary setup the Hub testing will move into the North most Rm 1200 and I may use the old going to salvage Polycom table as a base to set the ATCA shelf and HubDev computer on. ------------------------------------------------------------------------ DATE: 1-June-2017 Topic(s): Final Assembly and Hub testing to date, IPMCs, FTM The shipment of 9 IPMCs from Yuri arrived on Tuesday June 30. The FTM module from Ian and company arrived on Wednesday June 31. All Hub final assembly and testing work to date has been on Hub SN 01, i.e. the Hub without an FPGA. - The steps in the final assembly document is not in perfect order but I think that all of the electrical steps are now written down. All of the parts are here except for the recently required (and ordered) mmbt3904. Some of the steps take quite a long time to complete. - The documentation of the mechanical final assembly steps is still just an outline. So far the Machine Shop still has the Brackets, and Heat Sinks. They have gotten busy with Cyclotron work and I've not pushed but the deadline for getting things anodized is almost here. So far I have not selected the best way to fit the board thickness into the front panel handles. Which ever of the two ways to do this is selected, I will practice on SN 01. - There is one pcb design layout problem found so far, i.e. Q2951 is layed out backwards. The current solution for this is just to pull the assembly house part off the card and replace it with one put on up-side-down. All that I have in the lab right now are mmbth10s so I will use them for now but I want to replace them with the specified mmbt3906 because I do not want to risk the high frequency RF mmbth10s getting into parasitic oscillation. - I have installed the "post L RC network" on SN 01 using: 1500 uFd & 25 mOhm on the AVCC and Fan_1V8 and two parallel 820 uFd & 25 mOhm on the AVTT rail. Electrically and mechanically it looks OK but it takes a long time to intall these parts. - Initial scope pictures of the Hub power supply startup are in the series: scope_hub_pwr_xyz_abjkl17.tif which will be moved to the web and will have a readme file. - Somewhat randomly tried the BI Switch on the Hub from its front panel RJ45 connections. So far this looks OK both for 100 Mb and 1 Gb connection and either just going through Switch B or going through A-B or B-C or A-B-C. The LEDs are screwed up as somewhat expected. Without the PROM the LEDs mean very approximately green on --> 1 Gb connection and yellow means active. - The I2C PMBus from the Hub Front Panel appears to be working as far as looking at the DCDC Converters. One can see the first 6 DCDC Converters with the DPI_GUI. You start this on the Tough Book by something like > cd C:\DPI Suite\DPI Suite\DPI_GUI > dpi_gui 6 This is how you get the 6 "POL" windows. Running the DPI GUI with 6 POL windows what you can monitor and control is: Not Seen Address: 40 41 42 43 44 45 46 0x28 0x29 0x2a 0x2b 0x2c 0x2d 0x2e DCDC: 1 2 3 5 6 7 8 Vset Res kOhm: 34 30 20 20 10 10 4.42 Power Bus: FPGA MGT MGT Swch Bulk Fan Bulk Core AVCC AVTT 1V2 1V8 1V8 3V3 Output Amps: 2.06 0.00 0.06 1.81 0.12 12.38 --- There was no load on the MGT_AVCC and MGT_AVTT rails. There was a 0.5 Ohm load on the FPGA_Core rail. ------------------------------------------------------------------------ DATE: 24-May-2017 Topic(s): Pickup prototype Hub cards Picked up the 9 prototype Hub Modules from Debron. Delivered to Debron the correct: 74AVCH8T245 TSSOP-24 74LVC07A TSSOP-14 ------------------------------------------------------------------------ DATE: 22-May-2017 Topic(s): Monday Meeting All present at Monday meeting. The schedule is to pick up the prototype Hub Modules this week Wednesday morning. Topics include: when/where to setup the Hub test setup in the north most Rm 12xy, computers and JTAG pod, office space for Yuri and Pawel, I still need to check Pawel's initial safe design. Exchange a series of email notes with Ed about the initial Hub + ROD get together trip, e.g. potential dates, to whom to ship, cables, what to ship. ------------------------------------------------------------------------ DATE: 15-May-2017 Topic(s): Monday Hub Meeting, Power Supply Tests, Felix Arrive We had the normal Monday Hub Meeting with everyone present with Wade from CERN. Reviewed topics getting ready for the initial tests of the Hub card at MSU before the first get together with ROD in Ed's lab. The minimum required pre ROD + Hub get together tests are: 1. the Hub's Power Supplies work OK 2. the Clock System and its fanout work OK 3. JTAG works OK 4. the Hub FPGA can be Configured from JTAG or its Flash Configuration memory 4.1 verify that front panel access to the I2C PMBus for monitoring and control of the DCDC Converters is working OK 5. that we have some way to control and monitor the Power Control lines between ROD and Hub 6. that IBERT "FEX data" makes it across the backplane, through the Hub's MGT Fanout, and can be received by the Hub's FPGA from at least a couple of FEX slots at some line speed. About item #6: - Do we know what card we will be using for a FEX IBERT data source ? - Do we have the firmware for it and know how to run this firmware ? - Have any of us ever seen or worked with this FEX data source card ? When will it arrive at MSU ? - Do we know how to tell this FEX data source card to lock its clock system to the Backplane Reference Clock (so that this sender card and the Hub receiver are trying to operate at the same speed) ? - Has this FEX data sender card's Backplane Reference Clock input function ever been tested and proven to work ? (If so this test was done without a Hub) - Do we know how to change the line rate of the sender card ? We may need/want to run at a slow speed to get things going. - Do we have/need the firmware source code for this FEX sender card ? - If we get into an MGT power supply current limitation problem on the Hub are we ready to back off to running a minimum set of Hub MGT Transceivers ? Our first Hub MGT test is going to be from Transmitter to Receiver MiniPODs where both of these MiniPODs are on the same Hub Card. If we then get into trouble making the link work from the FEX IBERT data source card, we may want to try a Hub to Hub link, perhaps first using a Hub to Hub MiniPOD optical connection, and then trying one of the few MGT backplane links that exist between Hub 1 and Hub 2. Do we need anything special to get ready for such Hub to Hub tests ? Are we ready to have Hub #2 get its Reference Clock from the backplane ? The Hub 20 Amp DCDC Converter with External LC Filter power supply tests are finished for now. Now I need to edit the power supply documents and drawings and the MSU Final Assembly document to make everything current. Received from Yuri the Felix Optical TCC card, i.e. a Xilinx VC 709 demo board and a TTC fx card. ------------------------------------------------------------------------ DATE: 12-May-2017 Topic(s): Polarity of Handle Switch for the IPMC, IPMC Software Via Yuri we received the reccommended polarity for the ATCA Handle Switch for the IPMC. The official story is "We recommend to implement a normally closed handle switch which means, "when the card is fully inserted in the crate and operational  the switch contacts will be open circuit"." I also learned that we are in the IPMC software business. "Only the implementation of ATCA specification (i.e. firmware in IPMC MCU)  and firmware update code (i.e. running on IOIF MCU) are developed and maintained by us. However, you (i.e. USER) must provide the FRU and SDR information which describe your board as well as the code to register and to read your sensors. Some sensors are supported by us and we provide libraries (e.g. see ICARE framework) to read/write these sensors. o AD7414 o LTC4151 o LTC2499 o IQ65033QMA10" I know that we needed to provide the FRU and SDR information to describe the Hub board but I had not understood that we needed to write any software to readout out the Hub+ROD sensors. ------------------------------------------------------------------------ DATE: 8-May-2017 Topic(s): Monday Hub Meeting, TSSOP parts coming via ground, Hub build without U551:U554 We had a normal Hub meeting with everyone present. The emergency order last Friday of the TSSOP 74AVCH8T245 and 74LVC07A is mistakenly being shipped to us by ups ground. I asked Brenda to place a 2nd order for these two parts this time by air. I'm scheduled to get this tomorrow but it ends up that is too late. Debron was ready to run Hub today. Debron will build the first 9 Hubs with U551:U554 not populated. I'm sending parts to them so that they can populate these locations in the production build. ------------------------------------------------------------------------ DATE: 5-May-2017 Topic(s): Parts kit problems at Debron, schedule and tests note to Ed Email notes from Josh about 2 Hub Parts Kit problems: - I supplied the 74AVCH8T245 in the TVSOP-24 package and it needs to be in the TSSOP-24 package. - I supplied the 74LVC07A in the SOIC-14 package and it needs to be in the TSSOP-14 package. These were also screwed up in the Components Description file. Brenda did an emergency order with shipping that should get these two parts here on Monday. Josh is going to let me know how soon he needs them. A corrected version of the Components Description file is now ready that fixes these two errors and 4 other changes that need to be made to the file. I will contact Josh about making this corrected version official. Today I send the schedule and tests note to Ed that I said I would do in the last Monday meeting. ------------------------------------------------------------------------ DATE: 4-May-2017 Topic(s): Hub Front Panel, Heat Sink Extrusions, Old pcb mount handle switch at Debron Josh emailed about the pcb mount snap switch in the Kit but not in the current BOM. I should have taken care of letting him know about this earlier. He is just going to hold onto them and return them with the assembled cards. Rob finished the front panel late yeaterday afternoon. All looks OK but: - I need to move the 4-40 screw holes for the center bracket to get better/enough clearance for the Receiver MiniPOD Heat Sink. Move these two screws to the "left" 4.9 mm so that they are now centered between the top surface of the pcb and the bottom surface of the Rec MiniPOD Heat Sink. This locations lets them be maximally long. Regrires a rework of drawings M2 and M5. - Decision made to put the Insulated LEMO connector by the J2 Access Connector and not in either of the other two possible locations. The next to J2 location give the best chance of ration labels on the Front Panel. Requires redo of drawing M2 and a new drawing M7. All of this was passed to Rob today along with 8 blank panels and the one panel that I had hogged out for an initial test. All drawings on web are current again. Order FPGA Heat Sink Extrusion: ATS-EXL1-254-R0 We only get 2 FPGA Heat Sinks per bar so for now order 5 bars. Order MiniPOD Heat Sink Extrusion: ATS-EXL71-300-R0 We can probably get 9 MiniPOD Heat Sinks per bar and we need 2 per Hub Module so for now order 3 bars. ------------------------------------------------------------------------ DATE: 3-May-2017 Topic(s): BOM note from Josh, Power Supply test with external LC Filter Email notes with Josh about Hub BOM and the Manufacturer's Part No. vs Hub Part No. All OK with the documentation that is on the web for now. The setup of the USB - I2C Pod cable to the power supplies appears to be the following: - Cable connector pinout appears to be standard rational with triangle marking pin #1 and then pin #2 directly across the short dimension of the connector from pin #1. - The Pod's I2C PMBus pinout is: pin #6 Ground pin #8 SMBAlert_B pin #9 I2C Clock pin #10 I2C Data - All other pins (1,2,3,4,5,7) should be open at the power supply end of the cable. - I need to check to learn if there are additional grounds in this cable, i.e. at the pod end of the cable. - There appear to be various versions of the PMBus GUI software available from GE / Lineage Power that differ in the maximum number of targets that they can control and in other features, e.g. logging of data, graphical display of data. All of this now apprears to be from www.geindustrial.com ------------------------------------------------------------------------ DATE: 1-May-2017 Topic(s): Monday Meeting, 20 Amp PS Work Monday morning meeting with everyone present including two May Day holiday day off from Europe. Discussions included: - Update on Hub schedule from Dan: now expect to get together with Ed after his June 26 : July 10 time away, Dan will contact Ed with new schedule, must agree on minimum set of Hub tests before the first get together and note that this is not the same a the real MSU Production Hub Testing, work at MSU with Pawel and Yuri is in the range June 15 - July 10. - Discussion of mini-pods: what do we have, what's needed and when are they needed: decide to decouple this immediate purchase of MiniPODs from what is really needed for production Hubs, maximum credit card is $2,499, today ordered 4 Trans and 5 Rec 14 Gbps for $2,063, will order all of the FO Ribbon cable and MPOs for 8/9 prototype. - Discussion of Brian's most recent HTM specification document: new version of document now with some drawings, concern that both the MGT reference clocks and the clock to the FPGA Fabric come from backplane LHC locked source, "Mentor project" started. - Status and schedule of firmware development: mostly everything, Yuri ready to get the software necessary for IPBus installed on the recently arrived MSU machine. - Issues related to the VC709/TTCfx PCIe mating: Philippe suggests having a separate machine for the VC709/TTCfx, i.e. not putting the VC709/TTCfx in the MSU Hub Test machine. - Mechanical: an example of everything is ready except for the front panel which is being done now, can check FP and 3x Brackets on the existing bare board, plane to OK machining of full set of FP and Brackets just as soon as the examples are OKed on the bare board. - IPMCs: Wade thought the 8.9 more had been ordered, Yuri will check, intent is to ship production Hubs will all parts installed, i.e. with IPMC installed. - Hub Switch Test: need backplane connector/cables, are we planning to be ready to program PROMs - 20 Amp LC PS testing continues OK - report later. - Wade away: 9-16 May, 18-28 June. Weekend 20 Amp LC power supply work includes review of all Tant caps on all Hub power supplies and what comps file they come from and verify that the Bulk_3V3 caps have an OK Voltage rating. All looks OK. Add this data to the Hub power supply description. Work on the test setup with an additional 3x 470 uFd at the ouput of the external filter. That looks better now. Work on next MSU Final Assembly parts order. ------------------------------------------------------------------------ DATE: 26-Apr-2017 Topic(s): MiniPODs, Components File Error, ATCA Bench Supply Test, MSU Final Assembly Note, 20 Amp LC Filter testing It's time to order the MiniPODs that will be used with the Hub Modules. There are many questions issues surrounding this: - Need to have a matching set of MiniPOD Heat Sinks manufactured but that can not start until I have proven that the design is OK and that requires having an assembled Hub Module. - Need to have more of the Wakefield 155 thermal epoxy and use it to assembly MiniPOD heat sink combinations while it is within its shelf life period and arrange to get rid of the unused part in a safe way. Need to get access to fume hood for assembly. - Need to have matching fiber optic ribbon cables: probably 500mm length from Molex I currently have 8 spare male ribbon cables recall the standard, modules are male, patch cables are female is this still the L1Calo standard ? Molex Part No: 106267-2011 i.e. 500 mm VersaBeam Prizm to male MTP/MPO is still available from DK for $77 each with 10 in stock. - We also need to decide how many of the backplane MPO/MPT connectors we should install. They are Molex 106088-0100 with 19 in stock at DK for about $25 each. - I currently have only 1 un-heatsinked 10 Gbs Receiver MiniPOD and 4 un-heatsinked 10 Gbs Transmitter MiniPODs. There are more Trans and Rec MiniPODs "in stock" here but they all have the CMX type heatsinks installed. - I assume that all Hub Modules will have a Receiver MiniPOD installed on them. - Will all Hub's also have a Transmitter MiniPOD installed ? - Will we use the 10, 12, or 14 Gbit/sec MiniPODs ? - I assume that we will not use the 10 Gbit/sec if we actually want to test at that line rate. - It appears that HP optoelectronics, became Avago, then purchased by Broadcom, then the MiniPODs sold to Foxconn Interconnect Technology (aka FIT) part of Foxconn. It appears that the only distributor for FIT is Avnet. - Avnet appears to have zero of the MiniPODs in stock. Where have other L1Calo folks purchased MiniPODs recently ? - Estimated Prices for MiniPODs (without heatsink): 10 Gb Trans $334 12 wk 10 Gb Rec $209 8 wk 12 Gb Trans $297 12 wk 12 Gb Rec $175 12 wk 14 Gb Trans $297 12 wk 14 Gb Rec $175 12 wk - If we want to purchase enough Transmitters and Receivers for all 8/9 prototype cards this will require a P.O. and the associated delay. - I put the 3 page datasheets for the 12 Gbs and 14 Gbs parts into the Hub web site components - optical - Because of the same price, could the 12 Gbs and 14 Gbs parts be exactly the same except for how they are tested and specified, e.g. 12 Gbs part tested/specified for 100 m link at 12 Gbs 14 Gbs part tested/specified for 50 m link at 14 Gbs In any case fully testing these parts can not be easy, fast, or inexpensive. - Philippe has "pdf diffed" the full 10/12/14 Gbs datasheets and is the keeper of that knowledge. There is a serious typo error in the Components file, dcdc_4_converter_non_rpcs_comps It calls out R1152 as Res_1070_Ohm_0603 when it should specify it as Res_464_Ohm_0603. This does not effect the bare pcb or the assembly house work (because the assembly house does not install this part) but the correct value of this resistor must be installed during MSU Final Assembly. This is the Voltage Set Resistor for the MGT_AVAUX supply. The ATCA bench supply is finished and has been tested. On the input side, the "front panel" and the case on this supply are both tied to the 3rd wire Safety Ground. On the output side the Shelf Ground, Logic Ground, and 48V_Return are all tied to the ESD work bench ground through appropriate resistors. Running with a 20 Ohm load connected to the supplies Zone 1 connector cable, it made about 47 Volts across the 20 Ohm load and the current monitor resistor read 23.4 mV so the current calibration looks plausible but still needs a carefull check. Working on the MSU Final Assembly instuctions. As with CMX I'm going to do some of the mechanical stuff first, e.g. mount the front panel to help protect the card, and then do the small electrical stuff, and then do the rest of the mechanical stuff, e.g. install FPGA heat sink and MiniPODs. The current list of in stock electrical RC parts for MSU Final Assembly is: 1% 0805 Resistors: 180, 220, 10k, 15.4k, 23.7k, 36.5k, 54.9k, 84.5k, 130k 0.1% 25 ppm 0805 Resistors: 4.42k, 10k, 20k, 30k, 34k NP0 50 Volt 0805 Capacitors: 10 nFd, 15 nFd I must sort out and order the other values that will be needed during MSU Final Assembly. Testing of the 20 Amp DCDC converters with the external LC filter continues. The noise level looks OK coming out of the external LC filter. The issue remains understanding the best setup for stability considering the limited set of the servo loop components that we have access to. ------------------------------------------------------------------------ DATE: 24-Apr-2017 Topic(s): Monday Meeting, Bare Board check Monday meeting with everyone present. The decision was made to purchase a 14 slot ATCA shelf from ASIS. Yuri will order it and then ship it to MSU. The current plan is that we do not need to have this 14 slot shelf before we ship Hubs to Ed. That is the current plan is that we do not need to check the FEX data path from all 12 slots to the Hub FPGA before we ship Hubs to Ed. Brian reported on HTM work. Philippe is working on getting accounts or whatever so that Pawel, Yuri, and Brian can get through the perimeter firewall. I finished the inspection of the Mechanical Sample bare Hub pcb. With proper light and magnification I could verify that the top and bottom Ground Planes were the correct ones. I also finished checking more but not all of the configuation signals. The press-in connectors went in OK and their hole diameters feel OK. I still need to figure out how to handle the board thickness in the front panel handle. For now on the Mechanical Sample I just ground aobut one layer off the back side of the card. Grinding through the back most ground plane seems about right. The card appears to plug into the 6 slot crate OK. I send a note to Josh at Debron saying that it is OK to build with these bare boards. ------------------------------------------------------------------------ DATE: 21-Apr-2017 Topic(s): Bare Board arrival and checks, 20 Amp power supply testing The Mechanical Sample Bare Board arrived on Thursday April 20th. In a general view it looks like a very nicely manufactured card. Everything, Cu, Silk, G10 looks nice and crisp. I made many Ohm meter checks looking for connectivity and for shorts. I think that I have checked all of the power supply buses and fills. I think that I have checked all of the various types of back-drills. I think that I have checked the JTAG. I'm working on checking all of the Bank-0 and Configuration signals to the FPGA but there are a lot of them. I want to confirm that the 10 ground planes are all in the pcb in the correct order. I will have to section the card in two or three places to verify this. Ground plane issues are: the oval relief on all 10 ground planes around differential via pairs, the ground plane cuts under the AC coupling capacitors, and the over vs circle ground plane relief on just some differential via pairs where both traces to the via are in the same half of the stackup i.e. either upper or lower. I must verify were to section the card to see these features. Working on the 20 Amp LC filtered supplies. Currently making a scan of 4 feedback RC values. The setup of the scope is: Web URL: http://scope-hep-1.pa.msu.edu Instrument Name: scope-hep-1 Instrument IP Adrs: 35.15.225.89 Domain: pa.msu.edu DNS Address: 35.8.2.41 Gateway Adrs: 35.8.2.3 Subnet: 255.248.0.0 Http Port: 80 ------------------------------------------------------------------------ DATE: 19-Apr-2017 Topic(s): Mechanical Sample to MSU The bare boards arrived at Debron on Tuesday April 18th. Josh tried to ship the MSU Mechanical Sample to me the same day but I missed an early Tuesday afternoon email from him (because I was downstairs all day) with a question about how to ship the card. So the mechanical card is being shipped via UPS leaving Debron on Wednesday the 19th. Once I get it we have agreed that I have a two day turn around on saying that it is OK or not. The two types of PLLs from the September parts kit delivery are coming back with the mechanical card. ------------------------------------------------------------------------ DATE: 17-Apr-2017 Topic(s): Monday Morning Meeting, bare boards About 9AM or so Josh sent an email confirming that the bare boards had shipped on Saturday as expected. Monday morning meeting with just MSU present as this is a holiday in Europe / CERN. We used the whole meeting to discuss the HTM. ------------------------------------------------------------------------ DATE: 14-Apr-2017 Topic(s): Bare board schedule Josh reports that the bare board house missed the expected Wednesday April 12th ship date but that they now expect to ship on Saturday April 15th. ------------------------------------------------------------------------ DATE: 10-Apr-2017 Topic(s): Monday Meeting, Mentor Software Monday morning meeting with Pawel, Yuri, Brian, Philippe, and Wade. Pawel has a final version of the definition of the "safe" firmware and I'm signed up to read through it to look for any mismatches with the hardware. Brian is going to start the Hub Test Module based on the German FPGA module. He reports that Board Station is no longer on their download screen. In the definition of the various testing scheems it would be good to define what tests the Hub must pass before getting together with the ROD for the first time. Yuri is looking at what 14 slot ATCA Shelf to purchase. I passed to Brian and Philippe the "secret" URL to get Mentor Graphics software and our site ID and the password for my account at Mentor. Looking over Brian's shoulder at our Mentor download site I see the expected packages like Expedition, and HyperLynx and Tau and that they still have Boardstation on the download site and that for some reason PADs was also offered on our download site. There were also a couple of wizard things that I have not idea what they are. ------------------------------------------------------------------------ DATE: 6-Apr-2017 Topic(s): Bare Board Schedule, Wiener, Hub Mechanical Parts Josh at Debron talked with the bare board house earlier this week. If there are no problems they expect to ship the bare boards on April 12th. Thus I should plan on verifying the MSU bare board during the week starting Monday April 17th. Yesterday I talked with Andreas about Wiener perhaps getting into the ATCA Shelf market. I've emailed him to try to verify what is going on. The Front Panel and the MiniPOD Heat Sink are into the M-Shop. The drawings for the FPGA Heat Sink are ready to check with them tomorrow. ------------------------------------------------------------------------ DATE: 4-Apr-2017 Topic(s): Hub MiniPOD Heat Sink & Bar Stock Drawings and instruction text for the MiniPOD Heat Sink passed to the Machine Shop and on the web. Recall that the bar stock for the current MiniPOD Heat Sink design is: Advance Thermo Part Num: ATS-EXL71-300-R0 Digi-Key Part Number: ATS2187-ND and that the bar stock for the tentative FPGA Heat Sink design is: Advance Thermo Part Num: ATS-EXL1-254-R0 Digi-Key Part Number: ATS2193-ND ------------------------------------------------------------------------ DATE: 3-Apr-2017 Topic(s): Hub Monday Meeting, Hub Front Panel Monday Hub meeting with the full complement: 4 from MSU and 2 from Europe. Mostly status reports on the normal topics: FW, Testing, HTM, Hub Hardware. Drawings and instruction text for the Front Panel passed to the Machine Shop and on the web. This Front Panel version is minus the insulated LEMO. ------------------------------------------------------------------------ DATE: 30-Mar-2017 Topic(s): Discrete Power Wiring on Hub Because of lack of pcb power plane layers, the Hub design includes a number of Discrete Power Wire connections in order to provide enough copper cross-section to support the current loads with reasonable voltage drop. List of the Discrete Power Wire connections and their Manhattan length: Hold-Up Pos WTERM41 to WTERM42 (91.0 8.3) (276.7 246.4) --> 424 mm Hold-Up Neg WTERM31 to WTERM32 (97.4 8.3) (266.3 246.4) --> 407 mm 12V_ISO WTERM21 to WTERM25 (16.0 8.3) ( 30.5 217.5) --> 224 mm 12V_ISO WTERM22 to WTERM26 (20.5 8.3) (112.0 213.0) --> 296 mm 12V_ISO WTERM23 to WTERM27 (25.0 8.3) (252.0 227.0) --> 446 mm IPMC_3V3 WTERM1 to WTERM2 (96.5 40.5) ( 30.0 271.4) --> 297 mm CNST_5V0 WTERM11 to WTERM12 (101.0 40.5) ( 30.5 306.0) --> 336 mm BULK_2V5 WTERM51 to WTERM52 (28.9 137.7) (174.5 86.0) --> 197 mm So there are 8 discrete power wires which have a total Manhattan length of about 2.6 meters about 8.5 ft. ------------------------------------------------------------------------ DATE: 29-Mar-2017 Topic(s): Work on Mechanical Drawings, No direct SysMon of MGT Buses Work has been on the mechanical drawings for the Front Panel and for the 3 types of Front Panel Mounting Brackets. - The Front Panel currently has: 60 holes of 4 different sizes 4 rectangular cutouts 2 sizes Labels for: 54 LEDs 6 RJ-45s 1 LEMO 1 Access Connector - The 3 types of Front Panel Brackets can all be made from 1" x 1/2" Delrin bar stock. Currently that looks to be the best choice as the top bracket which holds the LEMO must be insulating. Be careful of the front panel picture in the Elma data sheet - what looks like the bottom is actually the top of the front panel. Front Panel Issues: The length of the wires on the stock Elma front panel handle switch is too short. I'm attempting to purchase just the switch and then add to it our own wiring. The gap for the pcb is too narrow in the Elma front panel ergonomic handle. The gap can take 2.6 mm drop dead max. Hub is about 3.07 mm thick. There are a number of ways to make this work, e.g. cut off the not used plunger part of the handle and EDM its thickness. Move the drawing numbers of all of the Hub's Mechanical Drawings to an "M" series so that they are not mixed in with the electrical circuit drawings. Pawel reports that the SysMon block does not provide for direct monitoring of the MGT Buses. It would have been quite nice to have had that. ------------------------------------------------------------------------ DATE: 27-Mar-2017 Topic(s): Monday Meeting, Yuri, Pawel, Wade, Philippe, Dan at the Monday meeting. - HW: No news since the all OK on Wednesday the 22nd Working on Mech drawings and test of supplies with LC filter. - FW: Talked about the "initial safe FW", i.e. all Select I/O lines have VIO, ILA plus SysMon via JTAG of both Super Logic Regions and that we need this forever, i.e. it is not a temperary toy. Changing names from Safe_x, feature driven versions, where/how maintain. -Shelf: Will look at 2 versions of 14 Slot Shelf, Schroff and Isic (or something like that, i.e. the type that they have in the current test stand at CERN). ------------------------------------------------------------------------ DATE: 22-Mar-2017 Topic(s): 2nd Technical Query, Out of Office One Monday the 20th received the 2nd Technical Query from Via Systems with 3 points: stop backside milling, move via, approve plots: - Backside thickness milling will end 7.5 mm from the front edge of the card (not 20 mm). - They will move the R958 via up 30 mils. - approve 45 all gerber plots The plot file names seem to be: cs. & ss. "component" side and "solder" side copper cbt. & sbt. top & bottom Plugs (see note) cpt. & spt. top & bottom Paste Stencil csk. & ssk. top & bottom SilkScreen csm. & ssm. top & bottom Solder Mask l2 ... l21 Layer 2 through 21 Copper drl_bak 1-2, 1-6, 1-13, 1-17 back-drill from Top cutting through x-y sdrl_bak 22-21, 22-17, 22-11 back-drill from Bottom 22-10, 22-6 cutting through x-y drl. Drills Through Plated dbe. Drills Through Not Plated (see note) th. Tool Holes smlcomp. Mill from Top routcomp. Thickness Mill from Bottom outline. Outline Notes: The way that the top and bottom Plugs work is not clearly understood - but it is clear that they can provide pcb features to prevent wicking from both the top and bottom. It is clear that these two plug files divide things up in a rational way. It appears that much Plate vs Un-Plate adjustment is done including moving Un-Plate to Plate if you can end up with the required finished hole diameter that way. On Tuesday the 21st send "OK" to all items in this Technical Query. Note that this Technical Query was just a request for approvals and not a request for any changes. On Wednesday morning the 22nd verify with George that there are currently no known issues - then away Wed afternoon, Thur, Fri. ------------------------------------------------------------------------ DATE: 16,17-Mar-2017 Topic(s): Check delivery of files, work on drawings and PS, Wade meeting, tar Archives On Thursday verified with George that Debron and the bare board house had the new files, that all topics were now clear, and tht there was nothing else that I should be working on to help keep the build moving along. Working on drawings and tests of the 20 Amp LC filtered supply. Wade stops by and he, Philippe, and I talk about HTM. He has a test layout to prove that a Xilinx Demo Board would fit onto a carrier card in the ATCA format. He needs this to make a rational resentation about money. Note that since the design work this week to make new Solder Mask and Plugs plots there is a new Official Archive file. The full Official set of tar Archives is: 46373888 Feb 15 backup_hub_0_wed_1.tar_archive pre release build 123458048 Feb 15 backup_hub_0_wed_2.tar_archive post release build 123460096 Feb 16 backup_hub_0_thur_2.tar_archive with release text 124993536 Mar 15 backup_hub_0_wed_4.tar_archive new Mask and Plugs ------------------------------------------------------------------------ DATE: 15-Mar-2017 Topic(s): Bare PCB Manufacture Conference Call, New Plots Released Morning bare board conference call. Milling from the back will be stopped 20 mm short of the front edge (left-hand edge), R958 cut via is OK. Solder Mask Tents now all fixed. Debron would like all vias under all BGA components plugged. Note that this is really just an extension of what was thought about as a potential problem with the Enet Magnetics yesterday. Ask about the thickness of the 1/2 oz surface copper - yes good it is plated up to almost 1.5 oz during production. Request no changes in copper thickness - but note that so far I still have not received any Vdrop calculations for the Hub. Edit the following Geoms to add Top side Plugs to all of their Pin/Pad stacks that include a Via: MiniPOD Trans and Rec 8 stacks each Flash Config Prom 64 BGA 4 stacks MegArray S1 (used by S2) 16 stacks FPGA flvc_2104 16 stacks Ethernet Magnetics Dual 2 stacks 54 stacks total Generate new Plugs plots using the traces with full set of fills and such. I check them and Philippe checks them with his fancy viewer. I now finally understand what his fancy viewer can do, e.g. look from the back and prove that nothing was lost. Send note to George and all with summary of conference call and official release of new Solder Mask and Plugs plots. Note that all details of the pcb work since I received the Technical Query last Friday are on the web in: ..../components/hub_pcb/hub_pcb_review_march_2017/ and the "official" communications back to them are in: ..../manufacturing/Production/Bare_Boards/ ------------------------------------------------------------------------ DATE: 14-Mar-2017 Topic(s): More Plugs, Answers to TTM Technical Questions Start with concern about the soldered wire connection in the Enet Magnetics rigth above the via in the BGA foot print for these components. Could the Hub pcb vias wick out these soldered connections in the Enet magnetics ? Test plugging these vias. Full written replies to the TTM Technical Query sent out. George works on setting up connference call. ------------------------------------------------------------------------ DATE: 13-Mar-2017 Topic(s): Work on Solder Mask Tents, Monday meetings Over the weekend and today I would on the Solder Mask Tents problem. The design is correct, i.e. the correct Solder Mask Relief is in the design files when you look at them with Layout and such - but these Solder Mask Flashes are clearly missing from the Gerber plots. This must be a plotting problem but attempts on the Durand system to fix it by changing the Artwork Order file did not work. If I change the Pin/Pad Stacks in the required geometries then the Solder Mask Plots are OK. So start editing all of the Pin/Pad Stacks in the following geometries that include a Via: MiniPOD Trans and Rec 8 stacks each Flash Config Prom 64 BGA 4 stacks MegArray S1 (used by S2) 16 stacks FPGA flvc_2104 16 stacks Ethernet Magnetics Dual 2 stacks QFN_16_TI U 1 stack QFN_48 U504 1 stack QFN_32 U507 1 stack DCDC9 1 stack ESD_Strip_Front 1 stack 59 stacks total Make new SolderMask plots (using the full traces file with fills). These Solder Mask plots now look OK. But note that the problem with the original plots had been very obvious but no one picked up on it. A split Europe and US Monday meeting because of US time change and orther schedule issues. Quick review and then look at the HTM. Brian is looking at a Xilinx demo board and cables to a bridge board in the crate. Push to dump the cables and move to a carrier card, i.e. no external Enet and no wall wart PS, rather make these connections on the carrier card to the backplane. Nothing written to study yet. Need minimum requirements, and description of how this setup will meet the requirements, e.g. backplane locked clocks for Logic and MGT. Will continue pushing with demo board design. ------------------------------------------------------------------------ DATE: 10-Mar-2017 Topic(s): PCB Review from Bare Board House Received via George a review of the Hub pcb from the bare board house. There are holds on 8 items. 5 or so of the items are simple. 2 will require some work. George is scheduling a conference call. I will start by working on the Solder Mask problem - item #6. Currently I have no understanding of the real cause of this problem. It looks fine in the actual design. ------------------------------------------------------------------------ DATE: 9-Mar-2017 Topic(s): Hub Kit parts shipped to Debron The 26 pound box of Hub Kit Parts shipped today. I believe that it is on Stores Pickup No. 5423765217366. 42 different part types, 194,570 pieces total. Back to work on Drawings. ------------------------------------------------------------------------ DATE: 8-Mar-2017 Topic(s): Hub Kit parts and Inventory ready, Reply from Janice The Hub Kit parts (42 different part types) and the Kit Inventory are ready. There are 3 inner boxes of components. Put a copy of the new Kit Inventory on the web in the ..../ Production/Assembly/ directory as hub_kit_inventory_8mar17.txt Received a reply from Janice - she will make an amendment to the Assembly PO. ------------------------------------------------------------------------ DATE: 6-Mar-2017 Topic(s): Morning and Afternoon Meetings, Revised Quote for Hub Assembly Normal Monday morning meeting with all 6 Hub people. Afternoon meeting about HTM with Wade and Brian. I will put the HTM drawing and notes that I made for the Feb 20th meeting on the web (made a ..../hub/hardware/other/ directory) and Brian will look things over and talk with Wade on Wednesday. Follow up with email note with possible minimum functions list. Received the Revised Quote for Hub Assembly. Verified that the expected cost increase looked OK with Wade, wrote a cover/ explaination note and sent things off to Janice. She is away until Wednesday. ------------------------------------------------------------------------ DATE: 1:3-Mar-2017 Topic(s): Post Release Geometry Edit, Mechanical design work and orders, Friday meeting with Wade Edited the Hub's FPGA Heat Sink Geometry on March 2nd so that its outline matches what is currently being designed. Changed its outline from being on SilkScreen to being on the special layer Conformal_Mask_1. March 2nd got the last order to Brenda to complete the kit for the assembly house. Need to double check that all is now complete with the Kit. Friday meeting with Wade and Philippe: review of what has gone on over the past two weeks with the Hub project. Agree with Wade that the mechanical parts will be made for the full build of 32 or 34 cards. ------------------------------------------------------------------------ DATE: 27-Feb-2017 Topic(s): Monday Meeting, Parts Order vs Component Description file, Mechanical Designs We did not have a Monday Meeting. Continue working on parts orders for the Assembly House Kit. There are 3 issues with the component_description files so I started component_descriptions_corrected. These problems were picked up during the Kit parts ordering: - The component description file incorrectly or unnecessarily asks for 0.1% tolerance for the 180 Ohm 0805 resistors for the Kit. Even though the 0.1% tolerance is not needed for this part I would have just purchased it to keep all of the documentation straight. But the 180 Ohm 0805 with 0.1% telerance is not in stock in a convinient packaging so I've revert to want it actually should be, i.e. a 1% telerance part. No that the 180 Ohm is used only in the loop compensation in the power supplies - not as the voltage Rset. - The 150 Ohm 0402 was not in stock from the specified manufacture so switch the purchase to Viahay/Dale and changed the information in component_descriptions_corrected. - The wrong part number was given for the 2.49k 0805. The part number that was originally given was for the correct value but 0603 size. ------------------------------------------------------------------------ DATE: 22,23,24-Feb-2017 Topic(s): Mechanical Design Parts Order Issue one parts order per day. All orders so far are for items to complete the kit to the Assembly House. Work on the Mechanical Design of the FPGA Heat Sink and of the Front Panel. - The FPGA Heat Sink can be either high up and thus have fewer holes and cut outs in it for tall components on the Hub or can be lower down and have taller fins but require more holes or cut outs for tall components. - If a thicker base plate material is used then the FPGA Heat Sink could have fins under it to direct air to the DCDC Converter that is in the shadow of the FPGA. - For the Front Panel I need to design a standard bottom "L" bracket, a middle notch bracket to keep the pcb in the center, and a fancy top bracket to hold the LEMO. Have not started any work on the brackets for the MiniPOD Heat Sinks. The MiniPOD Heat Sink has the issues of: - Can it fit onto the card past the lip on the Front Panel after the Front Panel has been installed ? - Can one make a twin MiniPOD Heat Sink that goes on after both MiniPODs have been installed ? ------------------------------------------------------------------------ DATE: 21-Feb-2017 Topic(s): Kit parts ordering started again, Edit to ATCA_Comps file, Summary of changes to the Inventory list, edit ab_jumpers, Finally got the build kit parts ordering started again. I started with the capacitors. Made a "post release" edit to the ATCA_Comps comp file to change all three instances of Res_10Meg_Ohm_1206 to Res_1_Meg_Ohm_1206. I did not rebuild comps or pass it to Mentor. This change is to make the comps file fit the BOM and the Inventory and the intent of the design. The following is a summary of the changed to the "Inventory" to make the Inventory list match the Assembly House BOM for the design as it was released. This list ignores changes to just the Item Number. Add as item #2 CTS_742C163_4.7k_Ohm increase count of Cap_100_nFd_0402 from 107 to 181 increase count of Cap_100_nFd_0603 from 174 to 177 Add as item #8 Cap_10_nFd_0402 increase count of Cap_10_uFd_10_V_0805 from 141 to 152 increase count of Cap_1_uFd_0603 from 101 to 102 increase count of Cap_2.2_uFd_0603 from 47 to 51 increase count of Cap_220_nFd_0603 from 70 to 73 increase count of Cap_22_uFd_10_V_1206 from 34 to 37 increase count of Cap_330_uFd_Tant_V from 6 to 7 Remove item #19 Cap_4.7_nFd_0402 increase count of Cap_470_uFd_Tant_V from 26 to 27 increase count of Cap_47_nFd_0402 from 197 to 205 increase count of IC_74AVCH2T45 from 1 to 2 Add as item #39 IC_CDCLVD1204 increase count of IC_NC7SV08 from 3 to 4 Remove item #49 IC_PLL_320M6296 Remove item #50 IC_PLL_40M0787 increase count of Res_100_Ohm_0603 from 18 to 20 Change item #77 Res_10Meg_Ohm_1206 to Res_1_Meg_Ohm_1206 increase count of Res_1k_Ohm_0603 from 26 to 36 increase count of Res_2.7k_Ohm_0603 from 4 to 5 increase count of Res_22_Ohm_0603 from 31 to 34 decrease count of Res_4.99k_Ohm_0603 from 57 to 52 increase count of Res_470_Ohm_0603 from 17 to 18 increase count of Res_49R9_Ohm_0603 from 80 to 92 Remove item #92 Res_62_Ohm_0603 increase count of Res_Zero_Ohm_0603 from 10 to 13 Remove item #95 Switch_SDS002R decrease count of Wurth_742792116 from 12 to 11 Edit hub_0_ab_jumpers.txt to include a description of the 4 jumpers associated with U1541 the Sensor I2C Bus EEPROM that was added right before the design release. ------------------------------------------------------------------------ DATE: 20-Feb-2017 Topic(s): Monday meeting, MGT test source for Hub, Parts Inventory, Debron Normal Monday Meeting with 4 from MSU and Pawel and Yuri. Production of the FTM cards to act as a MGT data source for Hub testing may be hung up on Euro Practice. Over the weekend I made a sketch design of a HTM aka Hub Test Module the intent being to sketch a minimal MGT data source card for Hub testing. Worked on getting the Hub Parts Inventory up to date with the actual design. An issue is that some Inventory Item Numbers have changed as different parts have come in and out of the design. Another issue is that some parts that I had assumed had been purchased in full quantity for both prototype and production builds - we now do not have enough of, e.g. IC_74AVCH2T45. The intent had been that the kit was complete except for Rs and Cs but it is now also missing: IC_CDCLVD1204 as that is new to the design since parts/inventory was last worked on. Email with George. They have looked at the design. He just wanted to verify the PLLs that they are now not to assemble and the setup with missing chicklets on J20 and J24. ------------------------------------------------------------------------ DATE: 17-Feb-2017 Topic(s): Work on documentation cleanup, DRC Runs, Verify trace length match, Parts Purchase for Kit, update web mentor documents Bring some of the Hub documentation up to date to reflect the changes that were made in the last week or two, e.g. hub_0_ab_trace_routing_details.txt and diff_pair_geoms_description.txt Make a DRC run on the design with and without the Fills in place. Record the results in the hub_drc_run_notes.txt notes file. I verified that the trace length information and thus the differential pair length match information that I used for the most recent runs of Philippe's DiffLengthCompare exactly match the production traces file. Thus the most recent runs that I made of DiffLengthCompare match the pcb design that is out for build. Update the Mentor: Geom, Comps, Nets files that are stored on the Hub's web site so that everyone can see them. Work on the parts to purchase list for finishing the kit to the assembly house. The issue is that the actual set of parts being used on the Hub Module has changed since the original inventory files were setup for it, i.e. specifically the Item Numbers have changed. There is nothing to do but start over with a new (i.e. current) set of Items in the inventory files to match the Assembly House Bill of Materials. ------------------------------------------------------------------------ end - go to the older section of this log book