Draft Proposal for VX1190A TDC Module Operation ------------------------------------------------- Rev. 27-MAR-2010 This note contains my suggestions about how the VX1190A TDC modules should be used in the HAWC experiment. This note applies to the baseline HAWC design and has the following sections: 1. Notes and Assumptions 2. Configuration Options and Suggestions 3. Module Addressing 4. Readout Data Format per Event per TDC Module 5. Estimated Event Size 6. TDC Module VME Readout Process 7. "Walking" the Event Cross Checks 8. Module Setup Procedure 9. Module Readout Script Assume the baseline HAWC design, i.e. - Reading 8 VX1190A modules all in the same crate - The TDCs operate in "Trigger Matching Mode" with about a 2usec window - All TDC modules are operated "synchronously" from a common isochronous source of: 40 MHz clock, Trigger, CRST, and CLR signals. Notes and Assumptions: - The manual says (implies) that these modules use only the +5V bus and that they draw 5.5 Amps each, i.e. 27.5 Watts each. This is 220 Watts for 8 modules. - We will be sending Differential ECL signals to the TDC module inputs. The built in 110 Ohm terminations is appropriate. The cabling details and the complexity of the front panel cabling need to be worked out. - The 40 MHz Clock, Trigger, CLR, and CRST signals are also sent to the TCS modules as differential ECL signals. These signals go to the modules 16 pin "Control" connector". - The 4 Rotary switches on each module must be setup for the module's VME base address. - Switch "SW1" controls whether the module is uses its Internal Clock or the External Clock Input. SW1 is set to use the External Clock. - Switch "SW2" controls whether or not the Control Bus is terminated. Because we will fan-out an individual run to each VX1190A module, SW2 should be set to terminate the Control Bus. - Header "J13" controls whether the module uses its Standard or its Backup firmware. This is set for Standard. Configuration Options and Suggestions: - Readout Resolution: 195 psec resolution, 102 usec max length, total of 19 bits of timing data - Separate Leading and Trailing Edge measurements - Filter Edges for minimum separation of either 10 or 30 nsec. - Matching Window Width \ Set for a 2 usec data window - Search Window Width | and the appropriate trigger - Matching Window Offset / latency - Enable readout of the TDC Chip's Headers and Trailers - Disable TDC Chip "Group" Headers and Trailers - Enable readout of the TDC Module's Extended Trigger Time Tag - Disable the subtraction of trigger time - Enable readout of all error type packets - Enable all channels - Readout modes: Straight A32 D32 for practice and learning Block transfer of some king for peddle to the metal event readout Module Addressing: For now assume that the 8 TDC modules are packed into slots 1 through 8 of the VME crate. Note that if the chosen VME crate backplane correctly supports IACKIN/IACKOUT jumping then from the point of view of front panel cabling it will be a very BIG advantage to use only every other slot. - We will think in terms of VME A32 addresses only (no A24 addresses) and the MCST/CBLT module address. - The VME base address is set via the 4 rotor switches on the module. - The MCST/CBLT module address is then set via writing this value to the MCST/CBLT address register in each module. The MCST/CBLT Address register offset is 0x1010. - Note that the VX1190A does not pickup its Geographic Address from the backplane pins but rather the Geographic Address must be set by writing it into the GEO address register on each module. The GEO Address register offset is 0x100E. - All TDC modules must have their "First Board", "Last Board", and "Intermediate Board" bits set in their MCST/CBLT Control register. The MCST/CBLT Control register offset is 0x1012. - There is no need to use the base address relocation setup via registers at offsets 0x1004, 0x1006, and 0x1008. Example Address Setup for 8 Modules in Adjacent Slots Rotor Switch Slot VME MCST/CBLT GEO Number Base Address Address FB LB Address ------ ------------ ---------- -- -- ------- 1 0xA1------ 0xAA------ 1 0 0x01 2 0xA2------ 0xAA------ 1 1 0x02 3 0xA3------ 0xAA------ 1 1 0x03 4 0xA4------ 0xAA------ 1 1 0x04 5 0xA5------ 0xAA------ 1 1 0x05 6 0xA6------ 0xAA------ 1 1 0x06 7 0xA7------ 0xAA------ 1 1 0x07 8 0xA8------ 0xAA------ 0 1 0x08 Readout Data Format per Event per TDC Module: All data readout from the TDC Module and TDC Chips is in 32 bit words with the high order 5 bits indicating a "Packet Type". The format of the data from a given TDC Module is: - TDC Module Global Header (01000) contains: 5 bit Module GEO Address and a 22 bit Event Number - Header from TDC Chip #0 (00001) contains: 2 bit TDC Chip Number, 12 bit Event Number, and 12 bit trigger course time - Time Measurements from TDC Chip #0 (00000 for Leading and 000001 for Trailing) which contain: 7 bit Channel Number and 19 bit time measurement 195 psec LSBit - Trailer from from TDC Chip #0 (00011) contains: 2 bit TDC Chip Number, 12 bit Event Number, and 12 bit Word Count - Same format for TDC Chips #1, #2, and #3 - TDC Module Extended Trigger Time Tag (10001) contains: 27 bit Trigger Time Tag with 800 nsec LSBit - TDC Module Global Trailer (10000) contains: 5 bit Module GEO Address, 16 bit Word Count, and 3 bits of Status (triggers lost, buffer overflow, TDC error). The TDC Chips may also inset Error packets (00100) if they have detected and error. Depending on what readout method is used, the TDC modules may also insert Filler packets (11000). Estimated Event Size: With no time measurements there are 11 words (44 bytes) of data per TDC Module per event. For 8 TDC Modules this is 352 bytes of formatting data per event. Assume that the PMT tube noise rate is 20 kHz and that there is a 2 usec window for hit data, and that most hits are low amplitude noise that only makes 2 timing edges, then the average amount of timing information per event from noise will be: 2 usec window x 20 kHz hits/sec x 1000 tubes x 8 bytes/hit which gives 320 bytes of timing information per event from PMT noise. Note that about 40 tubes out of the 1000 have a noise hit during the 2usec window. Estimate that 200 tubes are hit per event from the actual Physics signal and that 20% of these Physics hits are big enough to cause 4 edge hits - then 0.2 x 200 tubes x 16 bytes/hit + 0.8 x 200 tubes x 8 bytes/hit = 1920 bytes/event from the Physics hits. So in summary a typical event is something like: 352 bytes/event formatting 320 bytes/event timing data from noise hits + 1920 bytes/event timing data from Physics hits --------- 2592 bytes total per event. With a 5 kHz trigger rate this makes 13 MBytes/sec readout data. TDC Module VME Readout Process: The VX1190A modules offer many different possibilities for reading them out over the VME Bus. There are major differences in the various readout methods supported by these TDC modules. The following is a list of things that we should keep in mind while designing the TDC readout method that will be used in HAWC. - In baseline HAWC the TDC readout will need to support an average data rate of about 13 MBytes/sec and an average event rate of about 5 kHz. Note that the instantaneous peak data rate of the VX1190A module is 33 MBytes/sec. - From past experience with other DAQ system I have found that it is often the event rate, i.e. the overhead required to setup and handle each event, that causes much more technical challenge than supporting the average VME data flow rate. - A quick quantitative estimate of the time available for the DAQ computer to setup and fully handle each event is: 13 MB/sec / 33 MB/sec says that at a very minimum 40% of the time is required just to ingest the VME data into the DAQ computer's memory. 0.6 sec / 5 kHz says that at the very most 120 usec is available for the DAQ computer to fully setup and handle each event. I have seen contemporary computers with a popular operating system that take about 60 usec to respond to an interrupt, switch context, and begin executing a different thread. 120 usec per event is not a lot of time. - The layout of the raw TDC data in the DAQ computer's memory immediately after it arrives from the VME crate will depend greatly on the VME readout method that is used. Having a raw data layout that makes it easy to check each given event for self consistency and then pass it on to the next steps in the DAQ process will likely favor some VME readout methods over others. - The optimum VME readout method for the TDC modules may depend on how the down stream DAQ components operate. For example, does a given DAQ farm node what to receive one event at a time or a "chunk" of data that contains many events. - The typical HAWC event size, estimated above to be about 2,600 bytes, is pretty small compared to the size of data blocks that modern computers can efficiently handle. - The TDC module's readout FIFO buffer is 32k locations long with each location being a full 4 bytes wide (to match the data packets form the TDC Chips). Thus the HAWC system of 8 TDC modules can hold about 400 events assuming a typical total event size of 2,600 bytes. - DAQ systems that spend lots of time actually copying data from one location in memory to another often have trouble with both high data rates and high event rates. At this time I do not know what VME readout method will work best with the VX1190A modules in the HAWC application. The VX1190A manual provides nice descriptions of the different VME readout methods supported by these modules but it gives little guidance about which method to use in a give application. After we do the work to fully understand these modules and the HAWC application then we may want to talk with the CAEN folks to see what they suggest. Because I have not yet studied the VX1190A module enough to know what VME readout method to suggest, at this time I will just try to sketch two possibilities that illustrate the choices available. TDC Readout Sketch #1 - One event at a time with CBLT64 - Determine that there is at least one event in the TDC modules to be read out. This can be done by reading the Data Ready bit in the Control Register at address offset 0x1000. All 8 TDC modules work together so if there is an event in one module then all 8 modules have an event. - Setup a long Chain Block Transfer that has a long enough word count so that we can be sure that this one CBLT will be able to readout all of this event no matter how big the event is. For example assume that a real Physics event will never be longer than 8 edges in all 1000 TDC channels, i.e. 32k Bytes. - The Chain Block Transfer will start with the left most TDC module (viewed from the front of the crate), readout the data from just one event in that module's Output FIFO Buffer, and then transfer to the next module in the chain. - Note that at each module, only one event's worth of data is readout (i.e. the oldest event's data) and then the CBLT moves to the next module in the chain. - After the last module in the chain reads out its events worth of data, the CBLT completes this cycle by pulling the VME BERR* line. The VME BERR* line moving to is asserted state will cause an interrupt in the DAQ computer. In that way the DAQ computer learns that the transfer of this event, from the 8 TDC modules, into its memory has completed. - Concerns: How long does it take from when the BERR* is asserted until the DAQ computer has responded to the resulting interrupt ? Will the DAQ computer to VME Bus interface (whatever it is, e.g. CAEN PCI <-> VME or SBC Tundra chip set) allow block transfers to have a byte count as big as 32k ? - Nice points: Each event will automatically be layed out contiguously in the DAQ computers memory. Note that the DAQ computer will need to provide 32k Byte buffers to receive event data (even though typically only 2,600 Bytes will be used). Note also that if full CBLT64 is used (vs CBLT32) then at the end of the data from each TDC module, the module *may* have to fill in the last 4 Bytes with a "filler packet" with code (11000). - If some event is too big to fit in this one CBLT cycle, then the cycle will still terminate correctly. The memory buffer in the DAQ computer for this event will not have an overflow. Later in the DAQ process will checking the integrity of this event you will find that it is "broken". In some sense that is OK because this event can not represent real Physics anyway. TDC Readout Sketch #2 - N events at a time per module with BLT32/64 The intent here is to setup block transfers that are larger than just one event per transfer as was used in sketch #1 above. If a significant fraction of the DAQ computer's time is spent setting up each transfer then this should help the system run faster. In this system a given block transfer will all be from just one of the 8 TDC modules. Thus these block transfers must move more than 8 events at a time to gain in performance over the system outlined in sketch #1 above. - Determine that there are greater than N events stored and ready for readout in the first TDC module. Do this by reading the Event Stored Register 0x1020 or by reading the Event FIFO Stored Register 0x103C. - Setup and start a block transfer of event data from this module either by: Read N events worth of control information from the module's Event FIFO and use this information to construct a block transfer of exactly the correct size to fully readout all of the data from these N events in the this TDC module's Output FIFO Buffer. By design, the block transfer will end after reading the last byte of data from the Nth event. Load the BLT Event Number Register 0x1024 with the number of events, i.e. N, that you want to readout from this module. Start a block transfer that you know is bigger than these N events worth of data. After reading out the N events worth of data the module will terminate the transfer and assert the BERR* line which will interrupt the DAQ computer to let it know that this transfer has completed. - All 8 TDC modules work in step with each other - so if the first module had at least N events stored and ready for readout then so do the other 7 TDC modules. Loop through these remaining 7 TDC modules readout N events from each one by either of the methods indicated in the previous step. - So by doing 8 block transfers we now have all the TDC data for N events (where N is assumed to be > 8) stored in the DAQ computer's memory. This data is in 8 contiguous blocks of memory (one block for each module). Note that the data for a given event is not contiguous in the DAQ computer's memory. - Even with the events stored piece wise in the DAQ computer's memory it is still possible to make many of the tests outline below (walking the event) to verify that all of the pieces of all N events are intact. - It is probably too costly in time to actually put the N individual events back together in contiguous blocks of memory in the DAQ computer. Rather it probably makes more sense to transfer all 8 of these blocks of memory, one from each TDC module, as is to a computer in the DAQ farm. Once in the farm node the N individual events can be put together in contiguous memory blocks and any remaining test to verify that a given event is intact can be performed. - The concerns are mostly the same as in sketch #1 above, i.e. can we do single block transfers that are big enough to move N events from one module all in one transfer and how long does the BERR* interrupt response take ? - The hoped for advantage of sketch #2 is that by doing fewer but bigger transfers that the system can run faster than in sketch #1. "Walking" the Event Cross Checks: Once the data for a given event is in the DAQ computer's memory one needs to check it to verify both that all of the individual pieces of this data actually belong to this event and check that no pieces of data are missing from this event. The 18 checks in the following list illustrate the tests that one can make while "walking" the event to verify that the event is self consistent and intact: - Does the data for this event start with a Global Header and end with a Global Trailer ? - Do the Global Headers (01000) and the Global Trailers (10000) from the 8 modules come in pairs as expected ? - From the Global Headers and Trailers, do you see all 8 modules with each module showing up once and only once ? You test this by examining the Module GEO Address bits in each Header and Trailer. Do the 8 modules show up in the expected order ? - From the Global Headers, do all 8 modules show the same 22 bit Event Number ? - From the Global Trailers from the 8 modules, do the 16 bit Word Count values match the amount of data that was actually received from each module ? - From the Global Trailers from the 8 modules, are any of the "Error Bits" (bits 26:24) set ? If so then scan the whole event for "error data packets" (00100) from the TDCs. - Look at the Extended Trigger Time Tags (10001) from the 8 modules - do they all match ? - Examine the TDC Headers (00001), there are 4 of these from each of the 8 modules, i.e. 32 in total: - Do all TDC Headers show the same Event ID ? and does this match the lower order bits of the 22 bit Event Number from the module Global Headers ? - Do all TDC Headers show the same Bunch ID ? and does this match the lower order bits of the Extended Trigger Time Tag from the 8 modules ? - Examine the TDC Trailers (00011), there are 4 of these from each of the 8 modules, i.e. 32 in total: - Do all TDC Trailers show the same Event ID ? and does this match the Event ID in the TDC Headers ? and does this match the lower order bits of the 22 bit Event Number from the module Global Headers ? - Examine the 12 bit Word Count in all TDC Trailers and verify that this matches the amount of data actually received from that TDC chip ? Do the 32 TDC Trailer Word Counts add up to the overall size of the event adjusted for the Global Headers and Trailers ? - Look at all packet type codes (first 5 bits of each 32 bit TDC readout packet. Are there any unexpected packet types ? When the DAQ computer finds a "broken" event what will it do ?? - Mark it as bad and pass it on or throw it away ? How does it mark it ? - Log that it found a broken event including what was wrong with it ? - If the DAQ computer see just one of these (or more than N of these out of M events) should it "Reset" the DAQ system ? - Reseting the DAQ system could mean: Stop all triggers, Run whatever routine is used to cold start the TDC cards and crate, Re-Enable the triggers that had been running, Log that you did this. - If things are bad (defined as some persistent problem) how does the DAQ computer wake up a human ? Module Setup Procedure: The intent of the following pseudo code is to fully go through all the steps to cold start the 8 TDC modules and get them ready for normal Physics data taking operation. The strong intent of this procedure to to make no assumptions about the state of a TDC module before this setup operation takes place. That is, this setup operation wants to be able to "recover" a module no matter what state it has gotten into. We use this "no assumptions about your previous state" approach because: - In the future CAEN could easily change the default state of any number of the control setting on these modules and it would be nice not to need to rewrite this procedure in response to such changes. - We want to use this same procedure to "reset" or "initialize" the TDC modules if something goes wrong with the DAQ system during normal data taking operation. This approach to TDC module setup is accomplished by fully reseting the module and then writing the appropriate data values into all of the modules control registers. Specifically, even if a given control register normally has an OK default value at power up, it needs to be rewritten with the known correct value just in case something went wrong with the data stored in that register. Guidelines for writing this procedure: - Make no assumptions about the previous state of a module. - Reset then write values to all control registers. - Readback all values that you have written to verify that they were properly absorbed by the module and to give some indication of the modules basic health. - Have a way to log any errors or unusual conditions that you find during this procedure (so that a human can learn about any problems with the equipment). - If the modules have previously be setup and placed in operation and you are now running this procedure to recover from some problem with the DAQ system, then it would be good to first readback all control registers that should contain known stable data, before rewriting them. This is done to see if you can identify any problems with the modules while evidence of the problems still exists in the control register's contents. - Some information, e.g. are any TDC channels currently disabled, may need to come from outside of this procedure. There is no point in trying to show all of the data values that would actually be written into the various control registers on the TDC module in order to fully set it up. At this time one can at least list the registers that need to be written to completely define the operation of the module. The following list does not indicate the order in which the actual control value writes will need to be done. Note that some registers control the operation of the overall TDC module, others control just one of the 4 TDC chips, and still other effect just a single TDC channel. To setup the TDC module one must talk with at least: Control Register 0x1000 enable/disable main features Status Register 0x1002 readback only to verify the state of things after the setup ADER 32 Register 0x1004 base relocation register not used ADER 24 Register 0x1006 base relocation register not used Enable Ader Reg 0x1008 enable relocation not used must be 0 Interrupt Level 0x1001 set the interrupt level if used Interrupt Vector 0x100C set the interrupt vector if used GEO Address Reg 0x100E must set unique GEO Adrs very strange usage MCST Base Adrs 0x1010 MSBits of the MCST/CBLT address all same MCST/CBLT Control 0x1012 set for location in MCST/CBLT string Module Reset Reg 0x1014 Same as the Control Bus RESET of module Software Clear Register 0x1016 see pg 78 Software Event Count Reset 0x1018 see pg 78 Software Trigger Register 0x101A will not use Event Counter Reg 0x101C read only confirm zero after setup Event Stored Reg 0x1020 read only confirm zero after setup Almost Full Level 0x1022 word count in output fifo buffer to set status register bit and cause interrupt BLT Event Num Reg 0x1024 set number of complete events in BLT Firmware Revision 0x1026 read only confirm expected revision Out_Prog Control 0x102C control bus output line control register pick: Data_Ready, Full, Almost_Full, Error Event FIFO Stored 0x103C read only confirm zero after setup Event FIFO Status 0x103E read only confirm zero after setup Micro Register 0x102E and Micro Handshake Register 0x1030 These are used to sent commands to and readback information from the microcontroller that mostly takes care of JTAG communications with the 4 TDC ASIC chips on the TDC module. The microcontroller commands required to setup the module include: 00xx Set Trg Match Mode set trigger match mode of readout 02xx Read Acq Mode after setup read this to confirm Trigger Match Mode 03xx Set Keep Token TDC ASIC writes full events We probably do not really benefit from using Save User Config 06xx, Load User Config 07xx, Set Auto Load User Config 08xx. 10xx Set Window Width set trigger match window width 11xx Set Window Offset set trigger match window offset 12xx Set Extra Search set trigger match extra search 13xx Set Reject Margin set trigger match reject margin 14xx Enable Trigger Time Subtraction \ which mode will 14xx Disable Trigger Time Subtraction / HAWC use ? 16xx Read Trigger Config after setup read & confirm configuration 22xx Set Edge Detection we want leading and trailing 23xx Read Edge Detection after setup read & confirm 24xx Set Lead/Trail Resolution set the LSBit resolution 195 psec 26xx Read Resolution after setup read & confirm 28xx Set Dead Time between Hits probably want 10 or 30 nsec 29xx Read Dead Time between Hits after setup read & confirm 30xx Enable TDC Header & Trailer we want TDC header/trailer 32xx Read TDC Header Trailer Status after setup read & confirm 33xx Set Max Hits per Event sets the maximum number hits per event per TDC chip 34xx Read Max Hits per Event after setup read & confirm 35xx Enable TDC Error Marks allow TDC to include error information 38xx Disable Bypass on Global Error still readout TDC chip 39xx Enable TDC Internal Error Types \ for HAWC I think you want 3Axx Read the enabled TDC Error Types / to see all error types after setup read & confirm 3Bxx Set L1 Buffer Size want the full 256 words 3Cxx Read L1 Buffer Size after setup read & confirm 40nn Enable Channel nn \ 41nn Disable Channel nn | Some combination of these 42xx Enable all Channels | must be used to enable the 43xx Disable all Channels | correct set of channels 44xx Write Channel Enable Pattern | and then readback and 45xx Read Channel Enable Pattern | confirm the correct 460n Write Channel Enable Pattern | channels are enabled. 470n Read Channel Enable Pattern / 50xx Set Global Offsets Coarse and Vernier for HAWC zero 51xx Read Global Offsets Coarse and Vernier read and confirm zero 52nn Set Channel nn Offset for HAWC we want all zero offset 53nn Read Channel nn Offset read and confirm all zero offset 600n Read ID Code of TDC n read to confirm the expected value 61xx Read Microcontroller Revision read to confirm expected value Are there things to confirm about the status of the firmware flash memory or the TDC Compensation SRAM after finishing the module setup ? Module Readout Script: The intent of the following pseudo code is to fully show all of the steps that are used during the normal event readout sequence. - ...