Prototype Hub-Module Specification -------------------------------------- Rev. 15-Sept-2014 Functions of the Hub-Module: ---------------------------- 1. Support of the ROD Mezzanine Card The Hub-Module physically holds the ROD Mezzanine Card and provides electrical connections to it through two 400 pin Meg-Array connectors. 2. Readout Signal Distribution The Hub-Module fans out the Fabric Interface Readout data to the ROD and to the Hub's Virtex FPGA. This is 6 channels of Fabric Interface data from each of 12 ATCA "node" slots and 2 channels of data from the FPGA on the other Hub. The data rate per channel will be 10 Gbps or less. 3. TTC Clock and TTC "Data Stream" Distribution The Hub-Module uses a TTC-FMC mezzanine card to receive the TTC signal. One Fabric Interface channel is used to fanout the recovered TTC Clock signal to each of the 12 ATCA node slots and to the other Hub-Module. The 8th and final Fabric Interface channel is used to fanout the recovered TTC "Data Stream" to each of the 12 node slots, to the ROD Mezzanine Card, to the Hub's Virtex FPGA, and to the other Hub-Module. 4. Base Interface Switch The Hub-Module provides a non-managed 10/100/1000 Base-T switch that has the following 19 connections: 1 connection to the front panel i.e. the "up-link" 1 connection to the ROD (or IPMC) on this Hub 1 connection to the ROD (or IPMC) on the other Hub 1 connection to this Hub's Virtex FPGA 1 connection to the other Hub's Virtex FPGA 1 connection to the Shelf Manager 1 spare front panel connection 12 connections to the "Node" boards in this crate 5. Power Supplies The Hub-Module provide all of the normal ATCA redundant power input, power isolation, and power control from the Shelf Manager via a IPMC card. Bulk +12 Volt power is provided to the ROD Mezzanine Card. DC/DC converters are used to provide the power rails for the Hub-Module itself. The required voltages are supplied to the TTC-FMC card. Notes about the Hub-Module Functions: - The Hub-Module is not providing Fabric or Base Interface connections to the 2 slots that do not exist in 14 slot crates, i.e. crates with 2 Hub slots and only 12 Node slots. - As shown the Hub-Module's Base Interface switch provides a connection to only one Shelf Manager. Hardware Implementation of the Hub-Module: ------------------------------------------ 1. Physical Layout The Hub-Module is implemented as a standard size 6 HP ATCA card. The Hub-Module holds the ROD mezzanine card. The ROD is located near the top edge of the Hub and is expected to run from the Hub's front panel edge for 220 mm towards the Hub's backplane edge. In the direction along the front panel the ROD is expected to run for 100 mm. The Hub and ROD are electrically connected by two 400 pin Meg-Array connectors. A short 4mm stack height is used so that the Hub and ROD PCBs are quite close to each other. The component sides of the Hub and ROD both face in the same direction. The intent is to keep the path of the high speed differential signals from the Hub to the ROD as short as possible and to give the maximum available height for the MiniPODs and other components on the ROD. The Hub and ROD are mechanically connected to each other using standoffs. The Hub-Module holds the fiber optic pig-tail cables and connectors that run from Zone 3 on the Hub to the MiniPOD devices on the ROD. In about its middle near the front edge the Hub-Module holds a TTC-FMC card. As its name suggests the TTC-FMC is electrically connected to the Hub via a 400 pin FMC connector. Four standoffs are used to mechanically mount the TTC-FMC onto the Hub. The TTC-FMC has a high standoff and most of its components are between the Hub and TTC-FMC PCBs. The Hub-Module has penetrations through its front panel for the TTC-FMC's LEMO, optical, and LED devices. Other Hub-Module front panel penetrations include those for the ATCA required LEDs, for the four front panel Ethernet connections, and any that are required for the ROD Mezzanine Card. The Hub-Module includes heat sinks for its Virtex FPGA, for its Ethernet switch components, and for it MiniPOD. Along its backplane edge the Hub-Module uses a full complement of connectors J20 through J24 and P10. 2. Readout Signal Distribution The Hub-Module receives readout data on 6 channels of the Fabric Interface from each of the 12 node slots in the shelf. This is 72 channels of high speed readout data from the node slots. In addition the Virtex FPGA on the other Hub-Module provides 2 Fabric Interface channels of readout data. This makes a total of 74 channels of readout data from other slots in the shelf. The FPGA on the Hub-Module holding the ROD also provides 2 GTH channels of readout data. That a total of 76 GTH receivers on the ROD are required to field the readout data from all sources in the shelf. The readout data from other slots in the shelf is received by the Hub-Module with On-Semi 2 way fanout chips that have built in termination. The exact chip used will depend on the final decision about the data rate of these readout signals. One output from these fanout chips runs to 74 GTH transceivers inputs on the Hub-Module's Virtex-7 FPGA. The other output from these fanout chips is routed through the 2 Meg-Array connectors to the ROD mezzanine card. The pinout of the Meg-Array connectors to the ROD has been designed to provide optimum signal fidelity for these high speed differential signals. The intent is to provide a clean, uniform, and short route for the traces on the ROD that connect the Meg-Array pins to its GTH transceiver inputs. On the ROD the Meg-Array connectors are located adjacent to the edges of its Virtex-7 FPGA that hold the GTH transceivers. In the Hub-Module design we are not providing a predetermined mapping of backplane Fabric interface channels to Meg-Array differential pin pairs going to the ROD. Rather this mapping will be determined during Hub PCB layout. Whatever mapping provides the cleanest layout of these high speed differential traces on the Hub-Module will be used. The only (and presumably week) constraint that this mapping will follow is that all 6 Fabric Interface channels from a given node slot will be routed to only 2 GTH Quads on the ROD's Virtex-7 device and to only 2 GTH Quads on the Hub's Virtex-7 device. The intent of this constraint is to allow an effective power down of unused GTH Quads. Note that for this layout technique to work the Hub PCB design must be aware of the Meg-Array to GTH connections on the ROD. In addition the direct and complement sides of these high speed differential signals will not be conserved during the Hub PCB trace layout. Whatever arrangement of the direct and complement sides of a given differential signal provides the cleanest layout will be used. Differential traces from the backplane connectors to the fanout chips, and from the fanout chips to the Meg-Array connectors, and from the fanout chips to the Hub-Module's GTH transceiver inputs will all be length matched. After PCB routing a final overall document will be prepared that lists which Virtex GTH Quad and transceiver a given backplane Fabric Interface channel is actually connected to and whether or not the overall routing on the Hub and on the ROD has resulted in an inversion of the signal. 3. TTC Clock and TTC "Data Stream" Distribution The Hub-Module uses a TTC-FMC mezzanine card to receive the composite TTC signal. The TTC-FMC card extracts the LHC locked clock and the TTC "Data Stream" and passes them to the Hub-Module. The Hub-Module distributes the TTC Clock and the TCC Data Stream to 15 different objects that use these signals. The objects that use these TTC signals are: 12 ATCA Node Slots, the ROD mezzanine card on this Hub, this Hub's own Virtex FPGA, and finally distribution of these TTC signals to the other Hub-Module. Distribution of the TTC Clock by the Hub is purely by fanout. Note that the TTC-FMC can provide a clock signal even when it is not receiving a composite TTC input signal. Distribution of the TTC Data Stream by the Hub is more complicated. As shown in the TTC Distribution drawing the TTC Data Stream is mixed with the "back data" coming from both the ROD on Hub-1 and the ROD on Hub-2. A small part of the logic available in the Hub-1 Virtex FPGA is used to combine these 3 data streams. Fabric Interface Channels are used to carry the TTC Clock and the combined Data Stream from Hub-1 to the Node Slots and from Hub-1 to Hub-2. When the Hubs are used this way all Node slots receive both their TTC Clock and the combined Data Stream from the Fabric Interface channels to Hub-1. Note that the PCB traces on both Hubs are the same so that distribution of TTC Data combined with back data from the ROD on Hub-1 on one set of Fabric channels while separately distributing TTC Data combined with back data from the ROD on Hub-2 on another set of Fabric channels is possible. We assume that extraction of the information that a given object requires from the combined TTC plus ROD Data Stream will be performed by FPGA firmware in that object. Further we assume that all objects will receive the combined Data Stream using a Virtex GTH Transceiver. As noted the Hub-Module that holds the TTC-FMC will distribute the TTC Clock and combined Data Stream signals to the other Hub-Module. This connection is necessary to supply these signals to the ROD and Virtex FPGA on the "other" Hub-Module. The physical path to carry these signals from the Hub with the TTC-FMC to the Hub without this mezzanine is a pair of Fabric Interface channels that run between the Hubs. Note that only the Fabric Interface Channels from the Hub-Module that carries the TTC-FMC mezzanine card are actually active. The TTC Fabric Interface Channels from the Hub-Module without the TTC-FMC are tied Low by that Hub. 4. Base Interface Switch Each Hub-Module provides a 10/100/1000 Base-T Ethernet switch with 19 ports that are connected to the following: 1 connection to the front panel i.e. the "up-link" 1 connection to the ROD (or IPMC) on this Hub 1 connection to the ROD (or IPMC) on the other Hub 1 connection to this Hub's Virtex FPGA 1 connection to the other Hub's Virtex FPGA 1 connection to the Shelf Manager 1 spare front panel connection 12 connections to the "Node" boards in this crate This Hub switch is implemented using 3 Broadcom BCM53118 devices. These 8 port switches include the PHY interface to the BASE-T network connections. Besides providing the advantage of build in PHY interface the BCM53118 can be operated as either a simple unmanaged switch or if managed it can provide advanced switch features. The intent is to provide a prototype Hub switch that is easy for everyone to use but that has advanced features available via remote management if needed. The prototype Hub-Module has 4 RJ45 Ethernet connections on its front panel. These 4 connections are normally used for: the up-link to the external network, two ports for connection to this crate's RODs or Hub IPMCs (depending on whether this is Hub-1 or Hub-2), and a spare front panel ethernet connection. The point of having these connections accessible via front panel RJ45 connectors is to make the prototype Hub-Module easy to uses in various test setups where either one or two Hubs may be used. The RJ45 connections to the Hub allow the switch to be tested without any other ATCA cards in the system. 5. Power Supplies The Hub-Module's power supply system is rather complicated because of the large number of different voltage loads on the card. The power supply system on the Hub-Module is divided into a number of logical and physical blocks. The features in the power entry block on the Hub- Module are defined by the requirements of the ATCA specification. These features include the dual -48V input buses, filtering, holdup, and pre-charge. The power entry block provides isolated power to the Hub's IPMC module and it sends monitoring information to the IPMC. The IPMC provides control signals to the power entry block to tell it when it is OK to power up the Hub-Module loads. The bulk isolated power source on the Hub-Module is an isolated +12 Volt supply. This block provides the bulk +12 Volts to all of the DC/DC converters that that supply the Hub-Module's loads and it provides bulk +12 Volt to the ROD which has its own DC/DC converters. Both the power entry block and the isolated +12 Volt block are stock commercial modules. We have investigated modules up to the 350 Watt power level. Power for the loads on the Hub-Module are provided by a number of commercial non-isolated DC/DC buck converters. These DC/DC converters include those for the Hub's Virtex FPGA loads: core, aux, vco, gthavcc, gthavtt, gthaux and those for other bulk supply loads on the Hub-Module including the TTC-FMC loads. Monitoring of the Hub-Module power supplies for both voltage and current is provided over the Sensor I2C bus to the IPMC and thus to the DCS system. In addition to this all supplies are monitored by a Hi/Low power supply supervisor to provide a 1 bit overall status of the Hub-Module's power system. 6. Hub FPGA Number and type of FPGA and why Configuration of FPGA Enumerate the functions of this FPGA 7. Enumerate connections to ROD 8. Testing of the Prototype Hub-Module stand-alone Ethernet Switch test stand-alone readout data path test stand-alone TTC Clock and Data distribution test stand-alone test --> test using just 2 Hub-Modules 9. A paragraph about each of the 3 line drawings. The intent is to call out what we are trying to show you in each drawing. The TTC Distribution drawing is most complicated. Note that there are more and less versions of this drawing. The more version tries to hint at using a small part of the Hub's FPGA to mix together the TTC data + ROD-1 data + ROD-2 data. Note that some of the lines in this drawing show one way to "use" the Hub TTC data distribution. At the PCB trace level things are fully symmetric between Hub-1 and Hub-2, i.e. in later phases the TCC + ROD-1 data and the TTC + ROD-2 data could be distributed separately to the Node slots. 10. A paragraph about the front panel holes for ROD ATCA required front panel LEDs RJ45s to Ethernet Base Interface Switch holes for the TTC-FMC front panel penetrations Keep this section in the current prototype review document because later versions will of this document will need a place for more front panel details and a front panel drawing. 11. Known Issues and Concerns This has perhaps changed over the last week or so. I think that we now have opinions about everything. Wade instead started a section to call out important points about our Hub (often non-conforming ATCA points) - we can handle only one Shelf Manager - We can not handle slots 15 and 16.