Hub-Module Ethernet Line Circuits ------------------------------------ Original Rev. 17-Mar-2015 Current Rev. 29-Apr-2016 This note describes the "line circuits" on the Hub Module that connect its various Ethernet components to the twisted pair lines that run from the card. Here I used "twisted pair" to mean both the ATCA Base Interface Ethernet links and the front panel RJ-45 unshielded twisted pair cable (UTP cable) Ethernet links. The standard transformer coupled Ethernet line circuits are often referred to in engineering slang as "ethernet magnetics". The very end of this note describes the capacitor coupled Ethernet line circuits that are used for the Ethernet connections that have both ends on the same Hub Module. Transformer Coupled Ethernet Line Circuits: ------------------------------------------- At the Ethernet physical layer the Hub Module has three different types of devices that must connect to standard transformer coupled Ethernet line circuits: - 17 line circuits to Broadcom BCM53128 Switch ports. 13 of these run to the backplane Zone 2 Base Interface. 4 of these run to the front panel RJ-45s. - 1 line circuit to the Micrel KSZ9031RNX Phys Chip for this Hub's Virtex FPGA. This circuit runs to the backplane Zone 2 Base Interface. - 1 line circuit to the Micrel KSZ9031RNX Phys Chip for the ROD Virtex FPGA on this Hub card. This circuit runs to a front panel RJ-45 connector. - 1 line circuit to the TI DP83848C Phys Chip on the IPMC mezzanine module. This circuit runs to a front panel RJ-45 connector. Both modern voltage mode and older current mode connections to the Ethernet line circuits are used on the Hub Module. The Broadcom BCM53128 and the Micrel KSZ9031RNX are voltage mode devices. The TI DP83848C Phys Chip in the IPMC is a current mode device. The Ethernet line circuit for the IPMC has the added complication that currently it is a 10/100 Base-T only connection but in the future the IPMC may be supplied as a 10/100/1000 Base-T device that will use 2 additional circuits and will presumable be a voltage mode device. The "magnetics" used on the Hub Module is a SMD device that holds 8 circuits per device, i.e. 2 complete 10/100/1000 Base-T connections per device. It is a Pulse Engineering HX5201NL part that can operate over the -40 to +85 deg. C temperature range. The HX5201NL is of the recommended topology with its isolation transformer next to the Phys port and its common mode choke next to its line port. This is a so called "8 core" per Ethernet connection device and has a 3 wire common mode choke that can be used with Power Over Ethernet. Phys Side Connections: ---------------------- The connection to the Phys side of the magnetics is different for voltage mode and current mode devices. - The voltage mode Phys devices connect to the two outer primary side transformer terminals on each circuit and the primary center tap is tied to logic ground through a 100 nFd capacitor. - The current mode Phys device connects to the two outer primary side transformer terminals as above. In addition each terminal on the Phys has a 50 Ohm resistor that runs to Vcc which is 3.3 Volts for the the DP83848C device on the IPMC. These 50 Ohm resistors are located on the IPMC mezzanine. The Vcc side of these resistors must be bypassed to ground. In addition the center tap of the transformer primary must be bypassed to ground with a 100 nFd cap and tied to Vcc, i.e. 3.3 Volts for the DP83848C device. - So the difference in these circuits on the Hub Module needs to be the inclusion of a jumper from the transformer primary center tap to the 3.3 Volt plane for any circuits that may need to operate in current mode, i.e. the IPMC A/Tx B/Rx circuits. Line Side Connections: ---------------------- In the Hub application (which does not have Power Over Ethernet) the line side of the magnetics is tied to a front panel RJ-45 connector or to a Zone 2 Base Interface connection paying attention to the required strange pinout of the B and "C" circuits in the RJ-45 connector. The center tap of the line side common mode choke is tied to ground through a Bob Smith termination (patented by Robert W. Smith). The intent of this termination of to absorb common mode energy in the line side circuit and thus to help reduce the radiation of this energy. This termination consists of a 75 Ohm series resistor from the common mode choke choke center tap in each of the 4 circuits in a given Ethernet connection. The far side of these 4 resistors are tied together and then tied to ground through a 1 nFd capacitor. For the Base Interface Ethernet connections the ground that this 1 nFd cap is tied to should clearly be the Logic Ground. For the front panel RJ-45 links one could argue that the 1 nFd cap should be tied to the Shelf ground or to the Logic Ground. Being an ATCA card the Hub Module has only a single pin Zone 1 connection to the Shelf Ground and thus this ground connection is likely to be noisy and not a very stiff ground point. Thus for the Hub's front panel RJ-45 Ethernet connections the 1 nFd Bob Smith capacitor is tied to Logic Ground. Ethernet Magnetics Connection Summary: -------------------------------------- In summary, each Ethernet connection (4 circuits) requires the following passive components: - 4x 100 nFd caps from the trans pri center tap to gnd - 2x zero Ohm jumpers to 3V3 for current mode Tx,Rx only - 4x 75 Ohm from common mode choke sec side center tap - 1x 1 nFd from the 75 Ohm resistors to ground Front Panel RJ-45s: ------------------- The front panel RJ-45 connectors used on the Hub Module are TE Connectivity 1888653-x 1x2 condo connectors. These are special condo connectors for ATCA because the standard height RJ-45 condo connector is too tall to fit within the ATCA side 1 component height limitation. The TE 1888653-x connector requires a notch in the front edge of the PCB or else it will stick out to far. On the Hub Module this notch is a minimum of 16.00 mm wide (i.e the connector itself requires a 16.00 mm wide notch and to this one needs to add some clearance and router tool radius). The notch, in theory, needs to be 10.55mm deep. Using a 7.00 mm deep cutout makes a better match to how far the other components (e.g. SFP+) stick out in front of the Hub's front panel. The Hub Module uses the 1888653-4 RJ-45 connector. This connector has the following setup of its LEDs: LEDs 1,4 are in the lower, i.e. PCB level, RJ45 LEDs 2,3 are in the upper RJ45 As installed on our Hub card and with the long axis of the front panel vertical: LEDs 1,2 are above their RJ45s LEDs 3,4 are below their RJ45s. TE Part No. LED_1 LED_2 LED_3 LED_4 ----------- ----- ----- ----- ----- 1888653-4 Green Green Yellow Yellow There are alternative versions of this connector available with different LED arrangements. The 1888653-4 version has the common Green/Yellow LEDs and is available from stock. For the Hub's Ethernet links that do not run through front panel RJ-45 connectors, green and yellow LEDs are provided in a separate front panel LED array. The metal cover shell around the RJ-45 connectors, which makes no electrical connection to the twisted pair cable, is tied to the Shelf Ground via its fingers that touch the front panel. In addition the Shelf Ground net is run to the pins on this connector that attach to its metal cover shell. Capacitor Coupled Ethernet Line Circuits: ----------------------------------------- There are two types of Ethernet connections on the Hub Module were both ends of the connection are on the same module. These are: - The connections between the Broadcom BCM53128 Switch Chips: Chip A <--> Chip B <--> Chip C. - The connection from the Micrel KSZ9031RNX Phys Chip for this Hub's Virtex FPGA to Switch Chip C on the same Hub Module. In these cases both ends of the Ethernet connection are on the same Hub Module and operate with the same ground planes. Thus we do not have a large concern about common mode signals on these links. To save circuit board space capacitor coupling can be used on these links. Both of these devices have modern voltage mode physical layer transmitters which makes it much easier to use capacitor coupling. Specific information about using capacitor coupling with the Broadcom BCM53128 Switch Chip is given on page 29 of the BCM53128 "Layout and Design Guide" including an example of capacitor coupling it to a different LDAC Phys device. Also see the Broadcom note CAP-AN102 "Capacitive Coupling 10/100/1000 Base-T Ethernet Chip-to-Chip and Backplane Applications". Information from Micrel about using capacitor coupling is given in their application note AN-120. Assignment of Ethernet "Magnetics": ----------------------------------- The Pulse Engineering HX5201NL "magnetics" that are used on the Hub have 2 full Ethernet port (i.e. 8 circuits) per module. They are assigned to the various Line Circuits in the following way: Ref. Mag Desig. Sect Usage ------ ----- ------------------------------------- TRNS1 Left RJ1 Lower This Hub's ROD Right RJ1 Upper This Hub's IPMC TRNS2 Left RJ2 Lower Switch Chip "A" Port 6 Right RJ2 Upper Switch Chip "C" Port 6 TRNS3 Left RJ3 Lower Switch Chip "B" Port 6 Right RJ3 Upper Switch Chip "B" Port 7 TRNS4 Left This Hub's FPGA Phy U21 J20 Rw 3&4 to Other Hub Sw Right Chip B Port 5 BI Ch 2 J20 Rw 3&4 to Other Hub FPGA TRNS5 Left Chip C Port 5 BI Ch 3 J23 Rw 7 to Slot 3 FEX Right Chip C Port 4 BI Ch 4 J23 Rw 8 to Slot 4 FEX TRNS6 Left Chip C Port 3 BI Ch 5 J23 Rw 9 to Slot 5 FEX Right Chip C Port 2 BI Ch 6 J23 Rw 10 to Slot 6 FEX TRNS7 Left Chip C Port 1 BI Ch 7 J24 Rw 1 to Slot 7 FEX Right Chip C Port 0 BI Ch 8 J24 Rw 2 to Slot 8 FEX TRNS8 Left Chip A Port 5 BI Ch 9 J24 Rw 3 to Slot 9 FEX Right Chip A Port 4 BI Ch 10 J24 Rw 4 to Slot 10 FEX TRNS9 Left Chip A Port 3 BI Ch 11 J24 Rw 5 to Slot 11 FEX Right Chip A Port 2 BI Ch 12 J24 Rw 6 to Slot 12 FEX TRNS10 Left Chip A Port 1 BI Ch 13 J24 Rw 7 to Slot 13 FEX Right Chip A Port 0 BI Ch 14 J24 Rw 8 to Slot 14 FEX Assignment of Front Panel RJ-45 Connectors: ------------------------------------------- The TE 1888653-x condo RJ-45 connector that are used on the front panel of the Hub Module are assigned in the following way: Ref Desig Connector Usage --------- -------------- ------------------------ RJ1 Lower or Left This Hub's ROD Upper or Right This Hub's IPMC RJ2 Lower or Left Switch Chip "A" Port 6 Upper or Right Switch Chip "C" Port 6 RJ3 Lower or Left Switch Chip "B" Port 6 Upper or Right Switch Chip "B" Port 7 When viewed from the front with the card plugged into a crate and the long axis of its front panel vertical,then RJ1 is nearest the top of the Hub and RJ3 is nearest the bottom edge of the card. The layout of these RJ45 condo connectors is shown in drawing #44. Document / Verify the ROD's RJ45 Ethernet Connection: ----------------------------------------------------- The ethernet connections on the ROD mezzanine card shown below come from pages 3 and 20 of the 4-Sept-2015 ROD print set. ROD's Ethernet Hub Magnetics and RJ45 for the ROD ---------------------------- -------------------------------------- TRNS1 Magnetics Front U11 MegArray MegArray ------------------ Panel ROD KSZ9031 S1 S1 Primary Secondary RJ45 Net Name Pin No. PIN NO. Pin No. Pin No. Pin No. Pin No. -------- ------- ------- -------- ------- ------- ------- TxRxA_P 2 S1-B29 S1-B29 L3 L9 2 TxRxA_N 3 S1-C29 S1-C29 L1 L7 1 TxRxB_P 5 S1-B31 S1-B31 L6 L12 6 TxRxB_N 6 S1-C31 S1-C31 L4 L10 3 TxRxC_P 7 S1-B33 S1-B33 L22 L16 5 TxRxC_N 8 S1-C33 S1-C33 L24 L18 4 TxRxD_P 10 S1-B35 S1-B35 L19 L13 8 TxRxD_N 11 S1-C35 S1-C35 L21 L15 7 For the ROD application (which is a Voltage Mode Phys Chip) the magnetics primary center tap has a 100 nFd capacitor to ground and the secondary center tap has a "Bob Jones" 50 Ohm 1 nFd in series to ground. Note as currently layed out it looks like all 4 pairs are inverted. Document / Verify the IPMC's RJ45 Ethernet Connection: ---------------------- ------------------------------- The current IPMC mezzanine supports 10/100 Mbits/sec Ethernet using a National/Texas DP83848C 10/100 Base-T Phys chip. This Phys chip has an old type Current Mode transmitter connection to the magnetics. The Hub must include jumpers so that it can support either the current IPMC or a future version of the IPMC with a 1000 Base-T Phys chip that would use a Voltage Mode connection the magnetics primary. The IPMC pinout includes pins for all 4 pairs in the standard contemporary Base-T Ethernet. IPMC Mezzanine Hub Magnetics and RJ45 for the IPMC ------------------------- ------------------------------------- Hub TRNS1 Magnetics Front IPMC ----------------- Panel IPMC Socket Primary Secondary RJ45 IPMC Net Name Pin No. Pin No. Pin No. Pin No. Pin No. --------------- ------- ------- ------- ------- ------- Gb_A+ / Eth_Tx+ 171 171 R1 R7 1 Gb_A- / Eth_Tx- 172 172 R3 R9 2 Gb_B+ / Eth_Rx+ 174 174 R4 R10 3 Gb_B- / Eth_Rx- 175 175 R6 R12 6 Gb_C+ 177 177 R24 R18 4 Gb_C- 178 178 R22 R16 5 Gb_D+ 180 180 R21 R15 7 Gb_D- 181 181 R19 R13 8 For the IPMC application (which is currently a Current Mode Phys Chip but could be a Voltage Mode Phys Chip in the future) the magnetics primary center tap has a 100 nFd capacitor to ground and the primary center tap for circuits A (Tx) and B (Rx) also have a 0603 jumpers to the BULK_3V3 supply. All of the secondary center taps have a "Bob Jones" 50 Ohm 1 nFd in series to ground.