Notes on Implementing HSRO in the Trigger Framework ------------------------------------------------------- Original: 29-AUG-2000 Revised: 9-MAR-2001 General Overview ---------------- This is the current understanding of how this should all ultimately work. The VRBC will take care of all of the buffer management for the VRBs. When the VRBC receives an L1 Accept via the SCL, it will provide a buffer number and beam crossing number to the VRBs via the backplane. Upon receipt of this information, each VRB then moves the data from the input FIFO to the specified buffer. When the VRBC receives an L2 Accept via the SCL, it provides the appropriate buffer number and the event number to the VRBs, again via the backplane. When the VRBs receive the buffer number and event number, they move the data to the output FIFOs. The VRBC then informs the VBD that there is an event for it to fetch. The VBD retrieves the data from the FIFOs via VME and passes it to Level 3. For testing purposes, it is possible to do all of the buffer management for the VRBs by hand. Note that the VME addresses given here are of two types: those used by TCC and relative addresses (especially in the case of the VRB where the same card is used in multiple slots). To determine the VME address as seen in the crate, the high order bits 0x19 are dropped from the TCC VME address. Also note that in the Scratch directory of d0tcc1 there are 3 files, vrb_init.vio, vbd_init.vio, and vrbc_init.vio which should provide an example of the initialization for the HSRO crate. At the moment, we are initializing 1) the VRBC 2) the VRB and 3) the VBD. Whether or not this is ultimately the best order remains to be determined. Low Level View -------------- The following cards are read out from the L1 Framework (note: here we also define the mapping to VRB's): Refer to the summary_of_fw_data_readout.txt document in this directory. -------------------------------- VRBC -------------------------------- The VRBC is in M124, middle, slot 14. The documentation gives it a standard I/O base address of 0x480000 which I believe means that TCC uses a base VME address of 0x1948 0000. VRBC Initialization ------------------- The current information for Initializing the VRBC is in: www.pa.msu.edu/hep/d0/ftp/tcc/trics_ii/hsro_crate_initialization.txt According to Daniel Mendoza, we shouldn't have to do anything to setup the VRBC. He did suggest that we reset the VRBC "just to be sure." Experience has shown that it is also necessary to select the primary DAQ path as well. Note that the VRBC has some D16 registers and some D8 registers, it is important to pay attention to this! VME Address ----------- 0x1948 000a Write a 0 to reset the VRBC - D16 0x1948 00b2 Write a 1 to select PDAQ - D8 (PDAQ = Primary DAQ, i.e. through L3) VRBC Monitoring --------------- The register at 0x1948 00d2 (D8) provides information on the status of the SCL: From D. Mendoza email of 31 Jan 2001 : > > The SCL Status register is updated every 20 ns. > The bit assignment is as follows: > > Bit 0 - L1 BZ - ACTIVE HIGH > Bit 1 - L2 BZ - ACTIVE HIGH > Bit 2 - L1 Error - ACTIVE HIGH > Bit 3 - L2 Error - ACTIVE - HIGH > Bit 4 - READOUT BZ* - ACTIVE LOW > Bit 5 - SCAN BZ* - ACTIVE LOW > Bit 6 - FINISHED* - ACTIVE LOW* > Bit 7 - SUCCESS ( When HIGH, Indicates that the previous > event was succesfully transferred to the VBD) > > According to the assignments shown below [above?], the start-up > value of this register is 0x70. Only after reading out > the first event, the value of this register should change > to 0xF0. There are front panel LEDs which indicate when an L1 or L2 Accept has been received, but these don't seem to be available via VME. The L3 transfer number for the event that just received an L2 Confirm is available at 0xc0d2 in A16/D16 (so 0x19ff c0d2 from TCC). There is in principle Event Error and Diagnostic Information available, but this isn't documented. The Crate ID Number (0xc0d0 in A16/D16, 0x19ff c0d0 from TCC) and Serial Number (0x1948 00b4, D8) can be read via VME. The lower 8 bits of the Crate ID reflect what is shown in the front panel selector and the upper 8 bits are set by the on-board dip-switch S1. The Serial Number is entirely hardwired. -------------------------------- VRB -------------------------------- The HSRO crate is in M124, middle. This determines the VME base address that TCC uses to address each VRB: Slot VME Base Address ---- ---------------- 8 0x1908 0000 9 0x1909 0000 10 0x190a 0000 11 0x190b 0000 12 0x190c 0000 13 0x190d 0000 16 0x1910 0000 17 0x1911 0000 17 0x1912 0000 VRB Initialization ------------------ The current information for Initializing the VRB is in: www.pa.msu.edu/hep/d0/ftp/tcc/trics_ii/hsro_crate_initialization.txt Each VRB must have the appropriate channels enabled and then be reset in order for that change to actually take place. In addition, we may supply one 16 bit word to be included in the VRB header block. (This may be written before or after the reset.) For completeness we can specify no Gray coding (this is the default, and we haven't had any problems when we haven't specifically disabled it). The Gray coding disable must also be written before the reset. Slot VME Address write ---- ----------- ----- 8 0x1908 0070 0xff enable all channels 0x1908 0076 0x0 no Gray coding 0x1908 000e user data if desired 0x1908 003c 0x1 reset VRB and VTM 9 0x1909 0070 0xff enable all channels 0x1909 0076 0x0 no Gray coding 0x1909 000e user data if desired 0x1909 003c 0x1 reset VRB and VTM 10 0x190a 0070 0x3f enable all but channels 6 and 7 0x190a 0076 0x0 no Gray coding 0x190a 000e user data if desired 0x190a 003c 0x1 reset VRB and VTM 11 0x190b 0070 0x3f enable all but channels 6 and 7 0x190b 0076 0x0 no Gray coding 0x190b 000e user data if desired 0x190b 003c 0x1 reset VRB and VTM 12 0x190c 0070 0xff enable all channels 0x190c 0076 0x0 no Gray coding 0x190c 000e user data if desired 0x190c 003c 0x1 reset VRB and VTM 13 0x190d 0070 0xff enable all channels 0x190d 0076 0x0 no Gray coding 0x190d 000e user data if desired 0x190d 003c 0x1 reset VRB and VTM 16 0x1910 0070 0x3f enable all channels but 6 and 7 0x1910 0076 0x0 no Gray coding 0x1910 000e user data if desired 0x1910 003c 0x1 reset VRB and VTM 17 0x1911 0070 0x03 enable all channels 0x1911 0076 0x0 no Gray coding 0x1911 000e user data if desired 0x1911 003c 0x1 reset VRB and VTM 18 0x1912 0070 0x0f enable channels 3:0 for now 0x1912 0076 0x0 no Gray coding 0x1912 000e user data if desired 0x1912 003c 0x1 reset VRB and VTM Note that writing a 0x0 to the register 0x3c resets only the VRB. VRB Monitoring -------------- The VRB provides both a Current Status (relative address 0x38) and a Latched Status (relative address 0x3a) register. The implication is that the latched status register is reset by writing a 0 to that register, but this is not explicitly stated. In the current status register, error signals are normally asserted at the completion of readout and remain asserted until the next readout command is received. Status Register Bit --------------- 0 Readout Busy, active when Readout Buffer Number is received, released when all active channels have received the event data 1 Scan Busy, active when Scan Buffer Number is received, released when all data for that event has been read from the VRB Output FIFO via VME 2 Sync Error, indicates a G-Link synchronization error was detected on a data link connected to an active channel 8 Scan Ready, indicates all data from the current output event has been copied to the output FIFO and may be read by the VBD, active when Scan Buffer Number is received, released when all data has been copied to the Output FIFO Another VRB register that is potentially useful for monitoring is the VTM Power register (relative addresses 0x78, 0x7a, 0x7c, 0x7e for Link 0 to 3 respectively). At startup or in response to a VTM control register write, the VRB reads the Finisar optical receiver power. The lowest 8 bits of the register contain the power value read by the A/D converter. Bit 8 is set if the Finisar optical receiver "signal detect" is set. If this is not set, the fiber optic cable may be disconnected or damaged. The VRB also provides a mechanism to "spy" on the data in a specific channel. A monitor event is selected by writing to the Monitor Channel register. The Control Logic will poll this information at the start of each event readout and put the appropriate channel data into the Monitor FIFO. At the completion of event readout, the Control Logic sets the Monitor Status register to a specified (but undocumented) value to indicate that the requested event data is available. (The documentation actually states that it sets the Monitor Channel register to a specified value, but I am guessing it really means Monitor Status register.) A monitor event may be selected at any time, but recording does not begin until the start of the next event readout. It's not entirely clear at what stage the information is transferred from the Monitor FIFO to the Monitor Buffer. The monitoring is a relatively new (or at least newly documented) feature, and we have never used it. Although the monitor information is available via VME, it seems to be primarily intended to be readout through the VRB Auxiliary Port in order to minimize interference with VME data transfers. Details in case we want to try it..... The Monitor Control register (0x4a) determines the type of monitor operation performed. Monitor Control --------------- 0 Idle 1 Reset, i.e. clear Monitor Buffer and reset Monitor Count register 2 Snapshot, i.e. copy data from a single event to the Monitor Buffer 3 Histogram, this is really only useful for SVX I think The Monitor Channel register (0x48) allows the user to select which channel (0-7) will be monitored. The Monitor Count register (0x46) contains the number of bytes in the monitor buffer. In snapshot mode, the count is updated after the data has been copied from the Monitor FIFO to the Monitor Buffer (0x0800-0x1ffe). The Monitor Status register, which is completely undocumented but may indicate when the monitor data is available, is at relative address 0x44. In addition to the previous sources of monitor-type information, there are various static registers (that are probably of limited interest): Address Function Expected Value ------- -------- -------------- 0x0 Module ID 3 0x2 Configuration Setting 5 0x4 Date Code YMDD of firmware version, 828 now 0x6 Module Serial Number individual board serial number 0x8 Module Type probably 0, possibly 1 VRB Buffer Management by Hand ----------------------------- Each VRB for which data is to be processed must be told the buffer number, beam crossing number, etc. 1. begin HSRO readout (i.e. capture and transport HSRO data) 2. move data from input register 0x22 write buffer number FIFO to buffer register 0x26 write BX number 3. move data from buffer register 0x28 write buffer number to output FIFO register 0x2a write event number 4. check word count register 0x32 read 16 bit word 5. read output FIFO register 0x10 read 32 bit long word In practice we have found that steps 1 and 2 can be reversed - if the buffer number has already been supplied, the data goes directly from the input FIFO to the buffer, otherwise it is stored in the input FIFO until the buffer number is available. Various Other VRB Control Registers ----------------------------------- The VRB Control register (0x40) can be used to shut off the VRB control port if there is no VRBC and the VRB is being controlled via VME. The documentation indicates that in the absence of a VRBC, the control port message lines are not driven or terminated so there is a possibility of spurious commands being received by the VRB. Bit 0 on means that the control port is enabled (default), bit 0 off means the control port is disabled. The VTM Control register (0x42) allows various aspects of the optical link to be controlled: Bit --- 0 Read optical power level (A/D) 1 Read optical signal detect 2 Disable GLink clock 3 Enable GLink clock 4 Reset GLink 5-6 VTM Channel number (0-3) This register is reset to 0 as soon as the command is executed. Only one of the command bits (0-4) may be set. -------------------------------- VBD -------------------------------- Notes and Description of using the VBD in Run II with VRB's & VRBC ------------------------------------------------------------------ The VBD occupies two sections in VME address space. One section is for its main memory buffer and the other section is for its control status registers and lists of where it should read data from, i.e. starting addresses and locations where it can read word counts. The VBD is not a VIPA card so it does not have Geographic Addressing. Jumper caps on the VBD control what VME addresses where it will wake up. There are also a number of jumper caps that control "special functions" in the VBD. I do not know how these should be set for the Run II VBD when it is used to readout VRB's. I will look at a working example and then document what I find. The current information for Initializing the VBD is in: www.pa.msu.edu/hep/d0/ftp/tcc/trics_ii/hsro_crate_initialization.txt List of registers in the VBD's control status register section VBDBASE+$0000 VBD Control Status Reg 0 VBDBASE+$0002 VBD Control Status Reg 1 VBDBASE+$0004 VBD Control Status Reg 2 VBDBASE+$0006 VBD Control Status Reg 3 VBDBASE+$0008 Reg for Crate Type for VBD operation VBDBASE+$000A Reg for the Adrs of the Event Number VBDBASE+$000C Reg for the Address of the Crate ID VBDBASE+$0010 Data Read Format Control Reg. VBDBASE+$0012 Parameter Read Format Control Reg. VBDBASE+$0014 Parameter Read Adrs Bits 31-16 VBDBASE+$1000 List of Pointers to Word Counts VBDBASE+$1800 List of Base Adrs's of Data BLK's List of steps to get the VBD ready to run Un-Lock the CSR block load DMA time-out. Load the Crate Type. Load the Event Number Address. Load the Crate ID Address. Load the D Control Register, the format for data reads. Load the P Control Reg., the format for parameter reads. Load the I/O Address high order bits. Load the VBD with the list of pointers to the Word Counts. Load the VBD with the list of data block Base Addresses. Reset the VBD (make these parameters active). Lock the VBD parameters and load DMA time-out. Un-Lock the VBD's CSR block and load DMA time-out. write $000C to register CSR0 Load the Crate Type write $0002 to the Crate Type Register Load the address where the VBD can read the "event number" write $C0D2 to the Event Number Address Register Load the address where the VBD can read the Crate ID write $C0D0 to the Crate ID Address Register Remember that Crate ID == Geographic Section Number the Trig FW is Geo Sect Numb 31 hex 1F the L1 Cal Trig is Geo Sect Numb 16 hex 10 Tell the VBD what format to use to read the event data from the VRB's and to read the Word Counts. We are told to write $BBF9 to the Data Read Format Control Register. I do not understand this value at all thus I do not know what kind of addressing or data width will be used for either of these accesses. Values BB and F9 are not described in the VBD document. So for now we just use this value because it is said to work. Tell the VBD what format to use to read the "parameters" i.e. Event Number and Crate ID. Write $EDED to the Parameter Read Format Control Register This sets up the VBD to read both of these parameters with A16 D16 access. Load the upper address bits that the VBD should use for parameter reads. The Daniel Mendoza stuff says that this should be $FF0A. But this does not fit at all with the format used for parameter read. It should not matter what is in this register. A more natural value would be $0000. For now I guess we should start with his $FF0A value. Load the list of addresses where the VBD can read the word counts write the pointers to the word counts This is the lower 16 bits of the addresses where the VBD will read the word counts for each section of the event. End this list with two blocks of all zeros. Load the list of starting addresses write the starting address for each section of the event that the VBD will read. These are 32 bit values. The high order 16 bits is written first and then at the next higher word address in this list write the lower 16 bits of the starting address is written. Reset the VBD and clear its error FIFO write $00CC to the VBD's CSR0. Writing $00CC is thought to be better than the $0080 shown in the Daniel Mendoza listing. $00CC both Resets the VBD and clears the error FIFO. It also "paints" the rest of the control bits with the values that you need in them instead of changing these values at the instant of reset. Wait about 2 to 5 mill seconds Finally Load the VBD DMA time-out value and lock the VBD Control Memory. write $001C to the VBD's CSR0. Location of Jumper Caps on the VBD top +------------------------------------------------------------------+ | | | +-+ | MJ2 CJ1 | | | DJ1 | | | DJ2 | | P1 | | | | | | | front +-+ | panel | | | | +-+ | MJ6 | | | MJ5 MJ1 DJ3 | | | | | P2 | | | | | | | MJ4 +-+ | | | | | PJ2 | | +-+ | | | | PJ1 | | | | | P3 | VJ1 | | | | | | VJ2 +-+ | | +------------------------------------------------------------------+ CJ1 6 vertical posts. Jumper cap over the lowest two posts. DJ1 An array 4 vertical x 4 horizontal. The left most column DJ2 has 2 jumper caps - one over the upper pair of pins in that column and one over the lower pair of pins in that column. The other 3 columns have one jumper cap each over the center pair of pins. DJ3 An array 2 vertical x 4 horizontal. A single vertical jumper cap over the left most pair of pins. MJ1 An array 13 vertical x 2 horizontal. Horizontal jumper caps over the lower 10 pairs of posts. MJ2 3 vertical posts. Jumper cap over the lower two posts. MJ4 nothing installed on the board - no connections MJ5 3 vertical posts. Jumper cap over the lower two posts. MJ6 3 vertical posts. Jumper cap over the lower two posts. PJ1 An array 5 vertical x 2 horizontal. No jumper caps installed PJ2 An array 6 vertical x 2 horizontal. A horizontal jumper cap connects the bottom pair. VJ1 3 vertical posts. Jumper cap over the lower two posts. VJ2 3 vertical posts. Jumper cap over the lower two posts. I think that the VJx jumpers complete the connection of the Slave Ready and the VBD Done lines to the P3 back plane connector. The MJ1 jumper sets up the base address of the buffer memory. Setup as described above it must select A29, A30, and A31 HI and all others LOW. This is $E000 0000. The problem with this is that we can not reach it with the Vertical Interconnect. A standard Run I place to put the VBD's buffer memory was at $0038 0000. We should be able to pick any 1/2 MByte block in the bottom 24 MBytes to put the VBD. The CJ1 jumper sets up the base address of the VBD's control memory and control status registers. Setup as described above A13, and A14 are HI and all other address lines LOW. This is $6000 as expected. The DJ1 DJ2 compination must pick the Bus Request - Grant level for when the VBD requests bus mastership. We have VBD serial number VBD 00092A1 in the Trig FW To have the data from the main memory buffer make much sense it is best at access it only via D32. A feature of the VBD for Run II is that its Bus Error LED will glow very softly when the VBD is accessing data at a high rate. This is not a problem. If the VBD actually sees a Bus Error then it will illuminate this LED and keep it on until the VBD is reset. ------------------------------------------------------------------ ------------------------------------------------------------------ The VBD is in M124, middle, slot 4. CSR 0 (one of the two base addresses associated with the VBD) is at 0x19ff 6000. Note that the VBD control status registers and lists of pointers only responds to A16 VME cycles. The 0xff are what tell the Vertical Interconnect to generate an A16 cycle in the slave crate. VBD Readout ----------- The VBD handles the readout of the data from the VRB output FIFO but needs to be told 1) the address to read and 2) the number of words to read out. In the case of the VRB, the number of words to read out is stored in relative address 0x32. (The VRB also supplies the number of bytes to read out and the number of longwords to readout if either of these numbers turns out to be more appropriate.) VME Address read ----------- ----- 0x1908 0032 number of words to be read out 0x1908 0018 data from VRB in Slot 19 0x1909 0032 number of words to be read out 0x1909 0018 data from VRB in Slot 9 0x190a 0032 number of words to be read out 0x190a 0018 data from VRB in Slot 10 0x190b 0032 number of words to be read out 0x190b 0018 data from VRB in Slot 11 0x190c 0032 number of words to be read out 0x190c 0018 data from VRB in Slot 12 0x190d 0032 number of words to be read out 0x190d 0018 data from VRB in Slot 13 0x1910 0032 number of words to be read out 0x1910 0018 data from VRB in Slot 16 0x1911 0032 number of words to be read out 0x1911 0018 data from VRB in Slot 17 0x1912 0032 number of words to be read out 0x1912 0018 data from VRB in Slot 18 Note, however, that as described above the VBD ingests this address information in a slightly different format. Note that we are reading out slot 19 first and then slots 9, 10, 11, 12, 13, 16, 17, 18 because we had a problem with slot 8 in this backplane.