Hub-Module Clock Generation and Distribution ----------------------------------------------- Original Rev. 16-Apr-2015 Current Rev. 26-Jul-2016 The Hub-Module uses 3 different clocks: - Ethernet 25.000 MHz Clock crystal controlled - 40.08 MHz LHC locked clock for FPGA logic - 320.64 MHz LHC locked clock for MGT Reference and FPGA logic Ethernet 25.000 MHz Clock: -------------------------- The 25 MHz clock generation and distribution are shown in drawing #39. This clock is generated by a local crystal and thus is not LHC locked. All other clocks on the Hub Module are LHC locked. The main purpose of the 25.000 MHz clock is to run the Ethernet Switches and the Phys chips. The crystal that is used to generate this clock has better frequency tolerance and jitter specification than are required by either the Switch or Phys chips. The output of this crystal oscillator is fanned out 6 ways by a low jitter TI clock fanout chip. The 25 MHz clock signals from this fanout are all single ended back terminated 3V3 CMOS level. There are separate clock feeds to each of the 3 Switch chips and to both of the Phys chips. There is also a 25.000 MHz clock feed to a Global Clock input in Select I/O Bank 94 of the FPGA. This is in SLR #0 of the UltraScale FPGA. The clock input on the Switch chips requires a high level of 1.7 V min and 3.8 V max (XtalVdd + 0.5V max). The clock input on the Phys chips requires a high level of 2.5 V min and 3.3 V max. So a 3V3 clock is fine for both the Switch and Phys chips. Each of the Phys chips multiplies the 25.000 MHz clock that it receives up to 125 MHz. These Phys chips make their 125 MHz clocks available on output pins in case they are needed by the associated MACs. The 125 MHz clock outputs from the 2 Phys chips have been routed to Global Clock inputs in Select I/O Bank 68 of the FPGA. This is the same I/O Bank that handles all of the RGMII MAC interface connections between the FPGA and the Phys chips. This I/O Bank is in SLR #0 of the UltraScale FPGA. 40.08 MHz LHC Locked Clock: --------------------------- The Hub's 40.08 MHz clock generation is shown in drawing #40A. This 40.08 MHz clock is designed to be locked to the LHC. To make this lock an LHC reference clock is received either as an FELIX Optical TTC signal using one of the Hub's MiniPOD Receivers or it is received from the Other Hub as a differential pair over the ATCA fabric interface backplane. Both the decoding of the optical LHC reference clock and the selection between using the optical reference or using the reference from the Other Hub are implemented in the Hub's FPGA. This is shown in the left-hand side of drawing #40A. In either case, the selected LHC reference clock leaves Bank 68 of the FPGA as an LVDS signal and is routed to an LVDS to single ended receiver U501 that is located right next to the 40.08 MHz crystal controlled PLL U502. This crystal controlled PLL has a low loop frequency so that it removes jitter from the selected LHC reference. When no LHC reference is available the PLL will hold within 50 ppm of the nominal LHC frequency. A signal indicating whether or not this PLL is locked to its reference is sent to a Select I/O input on the Hub's FPGA where it can be monitored in an IPBus visible status register. The 40.08 MHz LVPECL output from this PLL is routed to a set of clock fanouts that are shown in drawing 40B. The First Fanout U503 drives 4 loads and then a Second Fanout U504 drives the 12 FEX cards plus the Other Hub over the Backplane. This two step fanout is necessary because of the two Hub cards in a Shelf only the Hub that receives the FELIX Optical TTC signal will drive its backplane clock lines. Thus the Second Fanout U504 can be controlled from the Hub's FPGA so that it either operates normally or else it holds its output pins static. Only on the Hub that receives the FELIX Optical TTC signal will this Second Fanout be enabled and thus provide a backplane clock to the 12 FEX cards and to the Other Hub. First Fanout U503 a TI CDCLVD1204 drives: - the ROD Mezzanine on This Hub - a Select I/O Global Clock input pair in Bank 71 of This Hub's FPGA for use as a logic clock - a reference to the 320.64 MHz Clock on This Hub - the Second 40.08 MHz Fanout on This Hub Second 40.08 MHz Fanout U504 a TI CDCLVD1216 drives: - the 12 FEX cards over the backplane - the Other Hub over the backplane The 2V5 and 3V3 power for the 40.08 MHz clock generation and distribution components is LC filtered from the bulk supplies and distributed to these components on separate isolated PCB area fills. The ground plane under these components has been partially separated from the rest of the Hub's ground plane with a moat. Drawing #40A includes a note about FPGA differential pin pairs that receive this AC coupled LVDS clock signals. 320.64 MHz LHC Locked Clock for MGT Reference and FPGA Logic: ------------------------------------------------------------- The Hub's 320.64 MHz clock is shown in drawing #41. This is also an LHC locked clock. It uses a fanout copy of the Hub's 40.08 MHz clock as its reference. This 40.08 MHz LVDS reference signal is received by U505 and converted to a single ended CMOS reference signal for U506, the 320.64 MHz PLL. This crystal controlled PLL has a loop frequency that allows it to track changes in the output of the Hub's 40.08 MHz PLL. A signal indicating whether or not this PLL is locked to its reference is sent to a Select I/O input on the Hub's FPGA where it can be monitored in an IPBus visible status register. The LVPECL output from the 320.64 MHz PLL is delivered to U507 which is a MC100LVEP111 10 way LVPECL fanout. - 8 copies of the 320.64 MHz clock from this fanout are delivered to MGT Reference Clock inputs on This Hub's FPGA. These are delivered as AC coupled LVPECL signals. The MGT Reference Clock inputs that receive this clock are listed near the end of the MGT usage file: hub_0_ab_fpga_mgt_transceiver_usage.txt - A copy of the 320.64 MHz clock from this fanout is delivered to a Select I/O Global Clock input pair in Bank 71 of This Hub's FPGA for use as a logic clock. Note that the correct options need to be set in this I/O Block to allow it to receive this AC coupled clock signal. See the note on drawing #41.