HAWC WBS 4.8 Draft Design GPS Time Stamp and Clocks - Timing Control and Fanout ------------------------------------------------------- Original Version 6-JAN-2011 Most Recent Version 13-JAN-2011 This document starts by describing the design of the equipment that is being built to provide the GPS time stamp and clock signals for the HAWC experiment. This it GPS part of WBS item 4.8 A block diagram of this part of WBS 4.8 as originally envisioned is shown in the following drawing: http://www.pa.msu.edu/~edmunds/HAWC/Drawings/gps_clock_and_fanout.pdf Following this discussion this document presents two versions of the Timing Control and Fanout part of WBS 4.8 This document concludes with a brief description of a possible alternative readout method for the GPS time stamp for the Scaler DAQ system. GPS Signal Reception -------------------- The basic GPS receiver is a model CW46 made by a company called NAVSYNC. This is a small unit about 4 or 5 inches on a side that is in a weather proof package designed for outside mounting. A single cable about 3/8 inch in diameter runs up to this receiver. This cable carries the power to the receiver and brings down various timing signals from the receiver. This GPS receiver is basically a model CW25-TIM receiver module that has been mounted in a weather proof package along with an appropriate antenna, power supplies, and line drivers for the timing signals. The CW25-TIM module is made by NAVSYNC and is a modern design specifically made to operate in a stationary location and recover precision timing information from the GPS satellites. The CW46 receiver design was targeted as the clock in cell phone base stations. The CW25-TIM module is in fact the "guts" that is inside some expensive rack mount GPS Clocks. The NAVSYNC company is actually part of a company named Connor Winfield that is located in Aurora Illinois near Fermilab. Various Connor Winfield products have been used as time bases in the Fermilab accelerators and the CW25-TIM module has been used in at least one Fermilab design that I know of. The CW46 receiver was selected based on it being easy to mount outside, its ability to more than meet the HAWC timing accuracy requirements and because it is inexpensive enough so that it is basically just a replaceable part if it is damaged, e.g. by lightening. There is a strong intent to keep the overall GPS clock system inexpensive enough so that they can be used at various development / integration sites so that all systems can be assembled and tested in final form before going up the mountain. The timing accuracy of the CW46 receiver with a clear view of the sky will meet the appropriate European Telecommunications Standards Institute specification. Timing errors are a statistical issue. The output signals from this receiver will typically be within 30 nsec of the absolute time and basically it will never have an error over 1 usec. The CW46 receiver requires only a few Watts of power and is rated for operation in the -40 to +75 deg C temperature range at up to 18,000 ft altitude. The weather proof packaging meets the European IP67 standard (ingress protection against even short term submersion). The cost in small quantity is about $250. The intent is to use only "generic" output signals from the CW46 receiver to control the HAWC clock signals and time stamps. Thus if necessary in the future HAWC could replace this receiver with another unit and not need to change any of the other components in this system. The signals that will be used are the 10 MHz and 1 pulse per second time signals and the NMEA (National Marine Electronics Association) ascii time string messages. Note that the time strings are in UTC time. The CW46 receiver provides these output signals within a few seconds after power up and it reaches the specified accuracy after 10 minutes of operation with a clear view of the sky. The only management of the receiver that is necessary is to tell it what NMEA messages you want it to output. This management is automatically taken care of by the unit that this receiver plugs into. The CW46 receiver provides a number of additional outputs that can be used to confirm its operation when the system is first installed. The 10 MHz and 1 pulse per second time signals and the NMEA ascii time string messages flow from the CW46 receiver to an MSU designed card called H_Clock. H_Clock Card ------------ The fundamental purpose of the H_Clock card is to take the signals mentioned above from the GPS receiver and output the followings set of signals: 40 MHz clock to the TDC DAQ, 100 Hz clock to the Scaler DAQ, and 2 sets of time code pulses to 32 TDC channels that carry the high resolution part of the time stamp. The multiplication up from 10 MHz to 40 MHz is accomplished with a narrow bandwidth now noise phase locked loop (PLL) using a voltage controlled quartz crystal oscillator. Making a low phase noise 40 MHz clock signal for the TDC cards is important because the TDC cards themselves use the 40 MHz clock as the time base for an internal 320 MHz clock. Noise in the 40 MHz signal is equivalently multiplied which if bad enough would result in loss of lock by the TDC chips. This type of narrow bandwidth low noise frequency multiplication is similar to designs that we have made for equipment at Fermilab. Because this frequency multiplication function requires designing something - it will make the cleanest overall layout to build the rest of the GPS portion of WBS item 4.8 on the same card. That is what we have called the H_Clock card. To implement the various logic functions that are necessary the H_Clock card uses a Xilinx Field Programmable Gate Array (FPGA) from their VERTEX family of parts. Such functions as the logic to decode the NMEA ascii time strings, the counters and logic to make time code pulses for the 32 time stamp TDC channels, and logic to implement a VME interface are all rather standard functions to design for FPGA implementation. This FPGA logic is stored in an EEPROM on the H_Clock card and it is loaded into the FPGA automatically when the card is powered up. The H_Clock card connects to the CW46 GPS receiver via a cable that plugs into a front panel connector. This cable carries both power and all the signals both to and from the GPS receiver. The bulk of the space on the H_Clock card is used to provide 170 "general purpose" LVDS I/O signals. The mixture of LVDS inputs and outputs is set when the card is assembled or can be modified afterwards if necessary. All of these general purpose I/O connections use industry standard 17x2 34 pin 0.1" x 0.1" headers for normal 17 pair twist and flat cable. Thus a flat cable from one of these headers can plug directly into 16 channels of a connector on CAEN VX1190A TDC card or a CAEN 1495 FPGA card. The space for these 10 34 pin headers can not be provided on the front panel of the card but is rather provided on the surface of the bottom half of the card. This will require the H_Clock card to use two slots but eliminates the specialized connectors that are used on the CAEN cards. Only 4 of the 10 general purpose I/O connectors are needed to provide the required 2 copies of the TDC times stamp signals. The remaining 6 connectors (up to 102 signals) can provide the 40 MHz and 100 Hz clock outputs. Clearly when implementing only the GPS portion of the WBS item 4.8 much of the I/O signal capability on the H_Clock card will remain unused. Distribution of the 40 MHz clock on the H_Clock card is done with low jitter clock distribution chips that are designed for this function. All logic on the H_Clock card is synchronous logic in one clock domain and uses the 40 MHz as its clock. The VME interface provided on the H_Clock card will use separate bus transceiver chips placed right along the backplane connectors. This bus transceiver location minimizes the VME bus loading and is thus compatible with the fast fancy VME cycles that will be used in the TDC crate for event data readout. The intent is to only implement a simple A24/32 D16/32 slave only VME interface on the H_Clock. For the GPS portion of WBS item 4.8 the VME interface on the H_Clock card is needed only to read a couple of registers to verify the operation of the GPS receiver. None of the "user defined" backplane pin will be used nor will the P0 backplane connector be used. The H_Clock card will use on the +3.3V and +5V power from the backplane and generate the lower FPGA core voltages on the card. A view of the H_Clock card layout study can be seen in the following drawing: http://www.pa.msu.edu/~edmunds/HAWC/WBS_4.8_Design/h_clock_layout_study.pdf Timing Control and Fanout part of the WBS item 4.8 -------------------------------------------------- A view of the Timing Control and Fanout section of the WBS item 4.8 as originally envisioned can be seen in the following drawing: http://www.pa.msu.edu/~edmunds/HAWC/Drawings/timing_and_control.pdf Clearly the H_Clock card has enough logic resources in its Vertex FPGA to implement all of the logic that is required for the Timing Control and Fanout part of WBS item 4.8 But as currently envisioned the H_Clock card does not have either: - the ECL signal level capable that are required for the Control-Bus connections to the CAEN TDC cards - nor does it have enough I/O signals to implement as many copies of the Control-Bus as will be required in some designs of HAWC that involve many TDC cards and many cards in a hardware trigger system. As currently understood the number of cards that will require a Control-Bus could be as small as: 8 TDC card plus 0 trigger cards, or it could be as large as: 16 TDC cards (64 channel TDCs) plus up to 8 hardware trigger cards. The reasons for leaving ECL I/O off of the H_Clock card draft design are the following: putting ECL on it still does not solve the I/O count problem for a HAWC system with many Control-Buses, keeping ECL off of it eliminates a power plane and an on-board power supply and thus helps to keep the design simple. An important thing to note about the Timing Control and Fanout section of WBS item 4.8 is that only the Trigger and Busy (aka Almost_Full or Out_Prog) lines need to be individually controlled or listened to on each copy of the Control-Bus. The 40 MHz clock line and the CLR and CRST lines can fanout to all copies of the Control-Bus all the time. For a given running configuration of HAWC, all that needs to be controlled is: which Control-Bus cables are sent a Trigger signal and from what subset of the various Control-Bus cables is the system paying attention to their Busy lines in making up the over all system Busy that, when asserted, will block the issuing of additional trigger signals. There is plenty of I/O capability on the H_Clock card to provide all the signals described in the above 2 paragraphs for up to 24 copies of the Control_Bus. All that is needed is a "fanout and convert to ECL" card to make up the Control-Bus cables themselves. That is what the CB_Fan card does. This is also a normal 6U by 160 mm 2 slot card but it uses the VME Bus only as a power source. It actively fans out the one copy of: 40 MHz clock, CLR, and CRST that it receives from the H_Clock card and it distributes the individual Trigger and Busy lines. The fanout it done with low skew low jitter chips that are specifically designed for this purpose. Note that the Control-Bus cables run directly off of the CB_Fan card, no specialized patch panel type of setup is required. At this time the designers of the hardware trigger system have requested that their copies of the Control-Bus cables have LVDS signal levels. The selection of how many of the Control-Bus cables have ECL signal levels and how many have LVDS levels is set when the CB_Fan card is assembled (or it can be changed later). For each cable layout pads will be provided for both ECL and LVDS driver/receiver chips. As defined by CAEN there are currently 3 signals on the Control-Bus that are not used. These are the AUX, L2_Rej, and L2_Acpt signals. The CB_Fan card will also receive from the H_Clock card, fanout, and drive these signals onto all copies of the Control-Bus. The intent of this is to both guarantee that all signals are at defined levels and to provide a way to control these lines if CAEN uses them in the future. A view of the CB_Fan card layout study can be seen in the following drawing: http://www.pa.msu.edu/~edmunds/HAWC/WBS_4.8_Design/cb_fan_layout_study.pdf A drawing showing the overall system based on a CW46 GPS receiver, the H_Clock card, and the CB_Fan card is on the web at: http://www.pa.msu.edu/~edmunds/HAWC/WBS_4.8_Design/ cw46_h_clk_cb_fan_blk_diag.pdf Alternate Version of Timing Control and Fanout part of the WBS item 4.8 ----------------------------------------------------------------------- We are also looking at a version of the Timing Control and Fanout section of WBS item 4.8 that would use a CAEN 1495 for its implementation. In this case the 1495 would communicate with the VME Bus so that a control computer could setup for example what subset of the Control-Bus cables would be contributing their Busy signals to the overall Busy signals that stops the issuing of triggers. The advantages of using a CAEN 1495 would include not needing to design and build a CB_Fan card. The disadvantages of using a 1495 for this function may include: needing to make two different FPGA designs (H_Clock and 1495 timing control fanout), no provision on the 1495 for low skew synchronous fanout, and needing to make a patch panel to makeup the actual Control-Bus cables. Extended Readout Functions of the H_Clock Card ---------------------------------------------- Obviously the H_Clock card has on it the accurate GPS time and a VME Bus interface. Thus one could develop the firmware so that the H_Clock card writes into a FIFO the GPS time stamp of each of the 100 Hz readout requests that it sends to the Scaler DAQ system along with an internally generated copy of whatever event ID type information the Struck scalers use. Readout of these GPS time, Event ID data pairs from this FIFO over VME from an H_Clock card in the Scaler DAQ crate would be possible. Possible advantages: - eliminates the need to have a TDC in the scaler crate - may allow a more independent operation of the Scaler DAQ either more independent electrically or more independent at the control computer or VME readout computer level - may allow a more complete integration and testing of the Scaler DAQ before it goes up the mountain What may make such a setup practical to implement is that the Scaler DAQ is relatively slow (100 Hz) and probably uses rather straight forward VME cycles (so the H_Clock card does not need to include a fancy VME interface). On the other hand at this time I do not understand how the Scaler DAQ implements any flow control or how it implements a reset and restart after an event readout mismatch is detected. Thus so far this is not a known doable design idea. At this time I don't thing that it is practical to offer direct VME readout of the GPS time stamp from the H_Clock card to the TDC DAQ because: TDC DAQ must run very fast (10 kHz) and it will probably use some very fancy (almost CAEN proprietary) types of VME cycles. As we know the VX1190A TDC card offers many types of fancy VME cycles, I have heard nothing certain about what types of VME cycles HAWC plans to actually use, and in the time that is available it is not practical to develop the H_Clock's VME interface to simultaneously offer all of these possibilities. Thus I think the time stamp readout system for TDC DAQ with the least risk is to continue to use the agreed upon plan of TDC readout.