Hub Module Testing -------------------- Original Rev. 26-Oct-2017 Current Rev. 10-Nov-2017 In many cases, at this time, all that we can test is the electrical operation of the Hub because we do not have the full firmware and other components of the L1Calo that are required to test the actual operation of the Hub in the overall system. For example we can make a link by link IBERT tests of receiving data from the FTM but we can not tests 6 bound links from eFEX transporting Auora formatted data. We need 4 different lists of tests: - those for the Final Design Review - those that we ourselves want to make before releasing the Production Build of 20 Hub cards - tests that are done on only one or only a few Hub cards to characterize some aspect of the card's operation - tests that are done on all Hub cards as part of their MSU Final Assembly and official MSU Production Test procedures before a card is qualified for use in the real system or shipped to another lab for use there. As listed here I have tried to group the various tests by by "functional block" on the Hub Module. Category: MGT Link Tests ------------------------- 1. Verify that the 13 Equalization Group Enable Signals are all working OK and routed to the correct sets of MGT Fanout Chips. Status: This test has been started on one card but it has not yet been completed. 2. Verify that the 4 MiniPOD Receiver MGT channels and the 8 MiniPOD Transmitter MGT channels are all working OK. Status: So far the design has been checked for only 4 the MiniPOD Transmitter MGT channels and for the 4 MiniPOD Receiver MGT channels. 3. Verify that the Hub FPGA can receive 6 lanes of MGT data from all 12 of the FEX slots. Status: Most slots have now been tested on a couple of Hub cards. 4. Verify that the Hub can receiver 6 lanes of MGT data from all 12 of the FEX slots and correctly pass this data to the ROD. Status: So far this test has been made for only 3 or 4 FEX slots. 5. Verify that the Hub FPGA can send out Combined Data to the 12 FEX slots, to the ROD on This Hub, and to the Other Hub. Status: Most of these links have not yet been tested on even one Hub card. 6. Verify that This Hub can Receive Combined Data that was sent out by the Other Hub. Status: This test has not yet been tried. 7. Verify that This Hub can Receive the Readout Control data that was sent out by the ROD on This Hub. Status: This test has been made on one or two Hubs. 8. Verify that This Hub can send out two lanes of Readout data to the Other Hub. Status: This tests has not yet been tried. 9. Verify that This Hub can Receive two lanes of Readout data from the Other Hub Status: This tests has not been tried yet. 10. Verify that This Hub can Receive and correctly pass to its ROD one lane of readout data from the Other Hub. Status: This tests has not been tried yet. 11. Verify that the FPGA on This Hub can send one lane of its readout data to the ROD on This Hub. Status: This tests has not been tried yet. MSU Production Tests must include at least items 2:11 and probably need to run these link tests at a speed that must work (i.e. 6.4 Gbps) and at a speed that may fail just so long as it fails in a way that is consistent with the other Hub cards (e.g. 10 Gbps). Note that for now many of these test will require "walking" a FTM card through the 12 FEX slots and will not be testing with the full level of noise that the system will have when all links are running at once. Category: Power Supplies and Hub Component Cooling Tests --------------------------------------------------------- 1. During MSU Final Assembly and Bench Testing - Verify (measure) 11 power supply output voltages and 7 power supply output currents while the FPGA is configured with the Safe Foundation firmware. - The 7 DCDC Converters on each Hub are tested and setup for: Vin_On, Vin_Off, Vout_Margin_High, Vout_Margin_Low, and Vout_Scale_Loop. Status: These tests have been done on all 8 prototype Hub cards. 2. Verify that the I2C PMBus monitoring of the 7 DCDC Converter output voltages and currents is working OK by reading them from the Front Panel I2C Bus connector. The intent is to verify that the IPMC has access to this data. Status: These tests have been done on all 8 prototype Hub cards. 3. Verify that the FPGA's System Monitor reads out correct data for the Si Temperature, 1V8 Aux power, and 0V95 Core power. Status: These tests have been done on all 8 prototype Hub cards. 4. Verify the sequencing and ramp rate for at least a couple of the power supply voltages, e.g. DCDC-1 FPGA_Core, DCDC-5 SWCH_1V2, and DCDC-8 Bulk_3V3. 5. Verify that the board power enable circuit, the board start-up reset circuit and the two front panel board power status LEDs are all working correctly. Status: These tests have been done on all 8 prototype Hub cards. 6. Verify the expected Hub FPGA Si Temperature with the Hub running in the Shelf with standard air flow rate and the highest wattage FPGA firmware that we have available (currently about 32 Watts). Status: This test has only been run on a couple of the prototype Hub cards. The FPGA heat sink must be installed and thermally bonded before running this test. 7. Verify that the expected operating temperature is readout from the two MiniPODs via their control/monitoring serial bus connection to the Hub FPGA. Status: This test has not yet been tried. This test requires that the MiniPODs have their final heat sinks installed, are operated in a shelf with standard air flow rate, and Hub FPGA FW to access the MiniPOD control/monitoring serial bus. During MSU Final Assembly or MSU Production Testing all Hub cards must go through all of the Power Supply and Cooling tests. Category: IPMC and Slow Control & Monitoring Bus Tests ------------------------------------------------------- 1. Verify that the IPMC can negotiate via the IPMB Buses with the Shelf Manager and turn On or Off the Hub card appropriately. 2. Verify that the IPMC can detect the front panel handle switch position and correctly follow the protocol to to turn the Hob module power On and Off. 3. Verify that the IPMC can correctly read the Slot Address from the backplane pins for both Hub Slots. 4. Verify that the IPMC can correctly get the Crate Address from the Shelf Manager and make this address data available on the IPMC's general purpose I/O pins. 5. Verify that the IPMC can correctly gather the Hub's voltage, current, and temperature Monitoring Data by initiating the required cycles on the Sensor I2C Bus and that it can forward this Monitoring Data to the Shelf Manager where it is made available to the DCS system. 6. Verify that by one of its general purpose I/O pins the IPMC can be told to suspend its Reads of Monitoring Data over the Sensor I2C Bus and then later to resume its normal reads of the Hub Monitoring Data. 7. Verify that the IPMC can correctly read FRU and SDR data from the PROM on the Hub card via the Management I2C bus so that a generic Hub type L1Calo IPMC can be used on any Hub card. 8. Verify that the IPMC can correctly read the Monitoring Data from the Hub's ATCA Power Entry Module via the Management I2C Bus. 9. Verify that the three I2C buffers in the overall Sensor I2C Bus are working and be enabled or disabled via their control lines that run to the Hub's FPGA. 10. Verify that the Front Panel JTAG connector has access to the Hub's FPGA and that when the ROD is installed and powered up that the ROD's FPGA also appears in this JTAG string. 11. Verify that the two MiniPOD serial control & monitoring I/O buses are working and communicating with the Hub's FPGA. Status: Basically the first 8 tests and the last test in this category have not been tried yet. So far only tests 9 and 10 have been made on the prototype Hub cards. During MSU Final Assembly or MSU Production Testing all 11 of the tests listed above need to be made on all Hub cards that will be used in the L1Calo system or shipped to other labs. Category: Clock Tests ---------------------- 1. Verify that the 25.0 MHz and 40.0787 MHz oscillators are running at the correct frequency and are stable. 2. Verify the the 40.0787 and 320.6296 MHz PLLs have the correct LHC frequency tracking range. 3. Verify both sides of the First 40 MHz Fanout to the: Hub FPGA, ROD, 320 MHz PLL, and Second 40 MHz Fanout. 4. Verify both sides of the 13 outputs from the Second 40 MHz Fanout: 12 to the FEXs and 1 to Other Hub. 5. Verify both sides of the outputs from the 320 MHz Fanout: 8 connections to MGT Reference Inputs and 1 Logic Clock to the Hub FPGA. 6. Verify that when operated as the 2nd Hub, i.e. the Hub that does not directly receive an optical TTC signal from Felix, that this Hub card can correctly receive its LHC Reference Clock from the backplane. Status: All of the tests 1:5 have been done on the 8 prototype Hub cards. So far we have not tried clock test #6. During MSU Final Assembly all 6 of these tests must be made on all Hub cards. Category: Ethernet Connection and Ethernet Switch Tests ----------------------------------------------------------- 1. Verify that both Enet links to the Hub's FPGA are working correctly (link to the Enet Switch on This Hub and link via the backplane Base Interface to the Enet Switch on the Other Hub). Status: Most testing so far has used the link to the Enet Switch on This Hub. Transmit appears to work OK, Receive is not yet working. 2. Verify that all 22 Enet Switch Ports that are used on the Hub card carry Enet traffic OK. Status: The above test has been done for one card. The four Front Panel RJ-45 ports are bench tested during the MSU Final Assembly process. 3. Verify that the Hub's Front Panel Enet connection and Enet Magnetics for the ROD are all working OK. Status: This was tested on the two Hub's currently at Ed's lab. 4. Verify the the Hub's Front Panel Enet connection and Enet Magnetics for the IPMC are all working OK. Status: This test has not yet been made. 5. Verify that the current (16-Sept-2017) Enet Switch PROM content is OK and Final and start installing Enet Switch PROMs with this content on all Hub Modules. Status: Waiting for the OK of the PROM's contents. 6. Characterize the performance of the Enet Switch, i.e. test in isolation the data flow rate to each of the Switch's 18 external ports and test the switch when all 18 of its external ports are trying to flow data at the same time. Status: This test has not yet been started. MSU Production Tests must include at least items 1:5. Category: Miscellaneous Tests ------------------------------ 1. Verify that all 66 Front Panel LEDs are working and are of the correct color. 4 driven by the IPMC, 2 power supply status, 7 driven by the ROD, 3 driven by the Hub FPGA, 2 not driven by the IPMC, 4 driven by the Hub's Phys chips, 44 driven by Enet Switch. 2. Verify the Physical aspects of this Hub card: dimensions look correct (e.g. none of the brackets are in backwards), the board's corners are in good shape, the Insertion and Extraction forces feel correct and are smooth, the Front Panel Labels are clear and not scratched, all of the backplane connectors line up in a straight row and are fully inserted, nothing is loose or likely to vibrate in the air flow (e.g. the discrete power wires or fiber optic ribbons, or power supply LC filter comps), MiniPOD and FPGA heat sinks are correctly installed, the MiniPODs are fully seated and screwed down, the MiniPOD Fiber Optic Ribbon Cables are correctly routed and not blocking air flow. 3. Verify that all 9 ESD ground points are correctly connected so that the card is safe to handle. 4. Verify that the card auto-configures its FPGA from its Flash memory at power up 5. Verify that the ROD can control the open collector output signals from the Hub's Front Panel LEMO Connector. 6. Verify the correct operation of the 4 Power Control signals and the 8 Spare TBD signals between the Hub and ROD cards. 7. Verify the the correct fuses are installed. 8. Verify that both Front Panel FPGA output Access Signals are working OK. Status: To the extent that the prototype Hub cards are complete and in final form, all tests except for 5 and 6 have been made on the prototype Hubs. Some of these test are made very early during the MSU Final Assembly process, e.g. the check of the ESD Grounds. Others tests listed in this category require special firmware, e.g. to verify the operation of the 8 Spare TBD signals between ROD and Hub. Still other tests listed in this category are basically a final visual inspection. All Hub cards should receive all of these tests.