ADF-2 Data Path FPGA Configuration ----------------------------------- Original Rev. 27-Sep-2004 Most Recent Rev. 18-Jan-2004 This file describes the configuration process for the Data Path FPGAs on the ADF-2 circuit board using VME access from the L1CAL Trigger Control Computer. Each FPGA can be configured individually or both Data Path FPGAs can be configured simultaneously in parallel. If Both FPGAs are loaded simultaneously, they will automatically receive the same configuration information. Configuring the FPGAs separately makes it possible to send different configuration information to the two Data Path FPGAs of a given ADF-2 Card.As currently planned, all Data Path FPGAs in the RunIIb L1CAL system will use the same device configuration. Register and bit names referred to in this document are described in the Board Control Pal Description document board_control_pal_description.txt in http://www.pa.msu.edu/hep/d0/ftp/run2b/l1cal/hardware/adf_2/general/ Specific actions described below refer to the Function Address (FA) of Registers within the ADF-2 address space. The FA is the address offset off of the base address of the targetted ADF-2. cf. adf_2_vme_addressing.txt in http://www.pa.msu.edu/hep/d0/ftp/run2b/l1cal/hardware/adf_2/general/ Procedure used by TCC to program ADF-2 FPGAs ============================================ 0. Force an appropriate initial programming of the configuration control register write 0x1f to FA 5 i.e. in the Configuration Control Register set bit CNFG_PROG_B_REQ(0) high = do NOT clear configuration of FPGA #0 set bit CNFG_PROG_B_REQ(1) high = do NOT clear configuration of FPGA #1 set bit CNFG_CS_B(0) high = do NOT select FPGA #0 for configuration set bit CNFG_CS_B(1) high = do NOT select FPGA #1 for configuration set bit CNFG_RDWR_B high = select configuration read mode 1. Enable FPGA configuration set bitmask 0x10 high in FA 0 i.e. in the Board Level Control Register #1 (Channel Link & Enables) set bit ENABLE_FPGA_CONFIGURATION high to start configuration activity while leaving other bits of this register unchanged 2. Initiate request to clear the current configuration on the FPGA(s) (to prepare them for a new one). for FPGA#0 : set bitmask 0x01 low in FA 5 for FPGA#1 : 0x02 for both : 0x03 i.e. in the Configuration Control Register set the CNFG_PROG_B_REQ low, then high for the targetted FPGA(s) while leaving other bits of this register unchanged note: the bitmask above is actually used as an argument to the TCC FPGA configuration routines to specify which FPGA(s) should be re-configured. This bitmask is used in this step directly and also later shifted to check other register bitfields during later steps. 3. Set IO direction on FPGAs to write. (this must be done before selecting CS_B) set low bit 0x10 in FA 5 i.e. in the Configuration Control Register set bit CNFG_RDWR_B low while leaving other bits of this register unchanged 4. Complete request to clear the current configuration on the FPGA(s) for FPGA#0 : set bitmask 0x01 high in FA 5 for FPGA#1 : 0x02 for both : 0x03 i.e. in the Configuration Control Register set the CNFG_PROG_B_REQ high for the targetted FPGA(s) while leaving other bits of this register unchanged 5. Wait until device memory is cleared and is ready for configuration data. for FPGA#0 : examine bitmask 0x01 in FA 7 for FPGA#1 : 0x02 for both : 0x03 i.e. in the Configuration Status Register check the CNFG_INIT_B for the targetted FPGA(s) Finding a ZERO on a CNFG_INIT_B bit for an FPGA that is being configured, tells us the FPGA is NOT done clearing its configuration and is NOT ready to receive its first byte of configuration data and TCC should read FA7 again. TCC will read FA7 for a maximum of N_1= 1000 times which corresponds to a maximum wait time of about ~3ms without vertical interconnect or ~5ms with vertical interconnect. Couldn't find any clear description of how long this is expected to take. One table in Virtex-II Platform FPGA Handbook (xylinx document UG002 V1.3, dec 2001) seems to suggest = 1102 configuration frames * 4 us = 4.4 ms. Empirically we measure that it takes about 1 ms. 6. Select FPGA device(s) to be configured. for FPGA#0 : set bitmask 0x04 low in FA 5 for FPGA#1 : 0x08 for both : 0x0c i.e. in the Configuration Control Register set the CNFG_CS_B low for the targetted FPGA(s) while leaving other bits of this register unchanged 7. Write up to 510,324 bytes (4,082,592 bits) of configuration data to targeted Virtex-II XC2V1000 ADF2 Data Path FPGA(s) one byte at a time. Write successive data bytes to FA 6 i.e. in the Configuration Data Register write the next byte's worth of bitstream configuration data as found in the S-Record file given to TCC After every byte written TCC *could* also checks that the FPGA(s) are ready for the next byte for FPGA#0 : examine bitmask 0x04 in FA 7 for FPGA#1 : bitmask 0x08 for both : bitmask 0x0c i.e. in the Configuration Status Register check the CNFG_BUSY for the targetted FPGA(s) Finding a ONE on a CNFG_BUSY bit for an FPGA that is being configured, tells us this FPGA is NOT ready to receive its next byte, but the state of CNFG_BUSY will cannot change until we try writing another byte, so we *could* write the same byte over again. We did this initially while commissioning this procedure. The bistream file is in fact padded with enough leading and trailing bytes that the user may write the whole stream without checking CNFG_BUSY up to a rate of 50MHz which is a factor 100 higher than what TCC could ever do. This saves a factor of 2 in configuration time. TCC does *not* check the state of CNFG_BUSY, and looking for CNFG_DONE at the end of the configuration cycle is proof enough that all bytes were transferred properly. 8. De-select FPGA device(s) set bitmask x0c high in FA 5 i.e. in the Configuration Control Register set both CNFG_CS_B high to deselect device configuration while leaving other bits of this register unchanged 9. Deselect Write Configuratiton mode set bitmask x10 high in FA 5 i.e. in the Configuration Control Register set CNFG_RDWR_B high for read mode while leaving other bits of this register unchanged 10. Check if CNFG_DONE(X) is asserted. for FPGA#0 : examine bitmask 0x10 in FA 7 for FPGA#1 : 0x20 for both : 0x30 i.e. in the Configuration Status Register check the CNFG_DONE for the targetted FPGA(s) Finding a ZERO on a CNFG_DONE bit for an FPGA that is being configured, tells us the FPGA was not yet successfully configured we should check again. TCC will read FA7 for a maximum of N_3 times (to be determined) which corresponds to a maximum wait time of about N_3 * 2us without vertical interconnect or N_3 * 5us with vertical interconnect. I couldn't find any estimate of how long this should take Initially we pick N_3 = 1000 11. Disable FPGA configuration. set bitmask 0x10 low in FA 0 i.e. in the Board Level Control Register #1 (Channel Link & Enables) set bit ENABLE_FPGA_CONFIGURATION low to end configuration activity while leaving other bits of this register unchanged Configuration Time ================== It takes about 1.1 sec to configure a Data Path FPGA when we use a direct bit3 connection to the ADF crate. Whith a Vertical Interconnect link added to the VME access path it should take about twice as long, as the VME IO cycle time will go from about 2.5 us to about 5 us. It takes between 418 and 447 TCC read cycle (also based on direct bit3 access) for the FPGA to clear its content and advertise CNFG_INIT_B high during step #5. The number of read cycles should become about 220 cycles with a Vertical Interconnect link. Notes from first board configuration tests ========================================== Debugging wires ("flags") were soldered on the first board tested. The signals for FPGA#0 were thus accessible to a scope probe. The table below reports the measured logic levels during the configuration steps are named by their numbers (cf. above). After Step # #0 #1 #2 #3 #4 #5 #6 #7 #7 #8 #9 #10 #11 CNFG_RDWR H H L L L L L L L H H CNFG_CS_B H H H H H L L L H H H CNFG_BUSY H H H H H H H-2 L-2 L L L CNFG_PROG_B H L L H H H H H H H H CNFG_DONE L L L L L L L L L-3 L-3 L-3 CNFG_INIT_B H L L H-1 H-1 H H H H H H (1) the transition from low to high happens within milliseconds and we find CNFG_INIT_B high by the time we manage to look at it after step#4. During configuration at full TCC speed, it will take some 430 reads during step #5. (2) CNFG_BUSY can be low after writing the first byte, but then is found low after re-writing the first byte again (or one can write the second byte and ignore the first byte, as padding was automatically included in the bistream file). This measurement was taken during the first configuration cycle after power up. CNFG_BUSY would typically not be found high in step #7 for subsequent configuration cycles. (3) These measurements were taken while the board control PAL was not correctly handling on board VME writes, and thus not allowing proper configuration of the FPGAs. The Virtex II data sheet suggest that CNFG_DONE will go High even before the last byte of the bitsream is written to the device.