Hub Module FPGA Firmware Topics ----------------------------------- Initial Rev. 5-Mar-2015 Current Rev. 8-Jan-2016 The intent of this note is just to list the various firmware components that need to be developed for the Hub Module's Virtex FPGA. In no particular order this firmware includes at least the following sections: 1. Understand all modes in which people may want to run the Hub's GTH & GTY Transceivers, both now and in Phase 2. These transceivers are used to transport different types of data, e.g. FEX Readout, Combined TTC + ROD1 + ROD2 Information, and to receive the Optical Timing signal. These may be at different line rates. What externally generated reference clocks will be needed to support all of the anticipated GTH line rates (while running the internal transceiver clock systems in an optimal way) ? What different GTH signal types can share GTH Quads in the Hub's FPGA - both transmit and receive ? How many places in the 20 Quads do we need to distribute the reference clocks to, e.g. just one place on each side, to every 3rd Quad ? Understanding all of this feeds directly into the design of the Hub PCB. 2. Firmware to make the initial tests of the Hub Module at MSU when we first receive some prototype cards. This will include: - Test IPBus communications over the Base Interface with registers inside the Hub's FPGA. This is just like the initial work to make VME communications possible with the CMX card. We can not do any serious testing until we can reliably communicate with the Hub's FPGA over its normal IPBus data path. What desk top computer equipment and software programs do we need at MSU to make some IPBus register tests to a Hub card in the ATCA Shelf ? - Test the GTH connection over the Fabric Interface from one Hub Module to another. This is possible because there are a few transceiver links that run between the two Hub Modules in an ATCA crate. See drawing number 15 in the following directory: http://www.pa.msu.edu/hep/atlas/l1calo/hub/hardware/drawings/General/ This lets us make the initial Hub backplane transceiver tests using only Hub cards. This requires firmware for both the transmitting and receiving ends. This test can then be run (initially anyway) from Xilinx Chip-Scope via the JTAG connection to the Hub's FPGA. This is very similar to the MiniPOD tests on the CMX cards. That is, most of this firmware comes from Xilinx test Cores. This is something that one could practice doing with the Xilinx demo board. 3. Firmware to operate the Hub card as a Hub in L1Calo: - Help manage the Power Up of the ROD mezzanine on the Hub. A description of the power up process and of the signals involved in this process is in the available in document about all of the non-GTH ROD-Hub connections. - Decode the incoming GTH Optical Timing Signal into its Clock component and its TTC Information component. - Combine the TTC + ROD1 + ROD2 information and send this out over GTH Fabric Interface links to the FEX cards and to the Other Hub. As you know the details of all of this are still in flux. - Develop a rational IPBus Register Map for the Hub's FPGA to facilitate development of online software that must talk with and control/monitor the Hubs. - The 2 Hub's in a Shelf must have different behavior because they have somewhat different functions in the overall L1Calo system. In fact there are really 3 flavors of Hub behavior because the test racks operating with just a single hub module will be different from the full system operation with 2 Hubs. There are many ways to handle this, e.g. separate firmware designs with different "bit streams" and you configure in the correct bit stream for a given application, all possible modes are in one design and a hardware jumper on the Hub PCB tells the firmware which mode to run in, all possible modes are in one design and IPBus registers tell the firmware which mode to run in. Picking the correct approach typically depends on how many differences there are between the different modes of operation. - Does Wade want to define being able to Receive a block of FEX Readout Data and make it visible for inspection by reading some IPBus Registers to be part of the basic Hub FPGA functionality ? - Does Wade want to define Reading out any "event" data from the Hub to the Rod to be part of the basic Hub FPGA functionality ? - Just as with the CMX BSPT firmware, the Hub's firmware will need to allow monitor readout and control of the MiniPODs optical modules on the Hub. On the Hug the serial bus communications with the optical modules must connect to IPBus registers vs the CMX's VME registers. - The Hub's FPGA needs to support 2 types of I2C, SMBus, PMBus communications. The System Monitors in the Hub's FPGA have an I2C port that is connected to the IPMC's Sensor I2C bus. The Hub's FPGA also needs to provide a Master on this I2C bus so that we can control the Hub's DCDC Converters. Drawings of the IPMC Sensor Bus connections (including the run to the ROD) are available in the circuit diagrams directory on the web. - A Hub module knows its Geographic Address within a Shelf based on the Zone 1 Geo Adrs pins. These lines will be routed to the Hub's FPGA. The Hub's FPGA must then "Learn" which crate it is in. The Crate's Shelf Manager knows which crate it is in. This Crate Adrs must be communicated to the Hub's FPGA via some path, e.g. the IPMC learns the Crate Adrs from the Shelf Manager over the IPMB bus and then sets this Crate Adrs value on some of its General Purpose IO pins and these IPMC pins are tied to Hub FPGA HP IO pins or something like that. This IPMC function is being worked out by some one ??? Then the Hub's FPGA must send the full Geo Adrs within the overall L1Calo System to the ROD mezzanine over 8 HP IO signals that run directly from Hub FPGA to ROD FPGA. - Firmware to Generate IP / Ethernet Hardware Addresses: Based on its Geo Adrs in the overall L1Calo system the Hub FPGA Firmware must generate IP Addresses for both of its Base Interface Ethernet connections. I *assume* that some algorithm is going to be written to do this and that all of L1Calo (or all of ATLAS) will use this algorithm. I assume that David knows about this or is thinking about it. I *think* that this is how it will work, i.e. make an IP Address based on the card's overall Geo Adrs within L1Calo. Then I guess that via the ARP mechanism you would learn the card's Ethernet Hardware Address and the Hub's FPGA would then start using this Ethernet Hardware Address for its "non-broadcast" packets. I clearly do not know the details of this and I expect that people are still working on it. - Firmware to connect the Hub FPGA to Ethernet, ATCA Base Interface, IPBus: I *assume* that most of this comes as a "MAC Core" either from Xilinx or from L1Calo or from CERN/Atlas. This Core of logic in the FPGA talks to Ethernet by talking RGMII (Reduced Gigibit Media Independent Interface) through some FPGA HP IO pins to a Micrel KSZ9031RNX "Phys Chip". The Phys Chip makes the hardware connection between RGMII and the actual 10/100/1000 Base-T Ethernet medium. Drawings of the Phys Chip's connections with the Hub FPGA are included in the circuit diagrams web directory. On the other side of this Core of logic are the IPBus visible Registers that allow control and monitoring of the Hub via IPBus protocol over the Ethernet. Obviously one must have some kind of bus structure in the FPGA's logic to connect this Core to the various IPBus visible registers. Besides the RGMII connection, as part of the MAC core, there should be also be an MDC, MDIO connection between the MAC and the Phys chip. This allows the Hub FPGA to control and monitor the Phys chip. The details of this MDC, MDIO connection will need to be understood. Note that the only control and monitor connection that we have between the Hub FPGA and the 3 Switch chips is also via a MDC, MDIO path. Does that mean that we also want to instance MACs for the 3 Switch chips just to get this MDC, MDIO connection to them ?