CMX review: Jim's Comments > General: > > Do we have a User Requirements Document? > > Was that part of one of the docs for the previous review? We have: - the CMM specification http://www.pa.msu.edu/hep/atlas/l1calo/reference/l1calo/cmm/cmm_project_specification_ver1_8.pdf - the project specification written for the Stockholm review http://www.pa.msu.edu/hep/atlas/l1calo/cmx/specification/1_preliminary_design_review/CMX_Project_Specification_v1.0.pdf - the roundup document created for this review http://www.pa.msu.edu/hep/atlas/l1calo/cmx/specification/3_prototype_review/CMX_Prototype_Review_Material_20130215.pdf > I suppose the previous specification documents is where > it's stated that this is a 9u VME card, for example. We will explicitly add this detail in future documents, but this already implicitly covered under the requirement of being a replacement for the CMM. > Do we have a list of specific responses for the action items > for the previous review? see the review document, > but not a point by point response to the recommendations/findings. We can work on such a list. > 1.5.1 VAT card > > What is current hardware/firmware/software status? Yuri may provide further details as desired. The VAT card was built during the Summer of 2012. The VAT instantiates a version of most of the ancillary functions needed for CMX (i.e. non-real-time data path) using a Spartan FPGA. The CMX is using a bigger version of the same Xilinx Spartan FPGA with more IO pins as we are routing even more information to and through this field programmable device. The Board Support Function FPGA on CMX needs to include ancillary functions in addition to what exists on the VAT card (e.g. the control of the MiniPOD devices) and further firmware development will thus be needed. The VAT is currently used for the initial development of the Board Support Function firmware, for the development of the VME interface firmware on the BF or TP FPGAs, and will be used for the initial development of online software, e.g. to learn how to re-write the content of the Compact Flash card from VME, or to re-learn how to control the TTCdec. > Might results from VAT lead us to change CMX design > between proto and production? We are hoping that all results from VAT are already included in the prototype build of the CMX design. The VAT card will continue to be used as test platform for firmware and online software for CMX, but we are not anticipating that further hardware changes will be generated from this effort. > 1.5.2 Mechanical model > > What is the status of the board? > Where has it been inserted? > At a MSU crate only? > Or in a CMM slot at USA15? This board was sent to CERN and plugged into the MSU l1calo test stand. Pictures are available in http://www.pa.msu.edu/hep/atlas/l1calo/cmx/hardware/mechanical_only_card/ The MSU test stand is understood to be very similar to a l1calo crate but not 100% equivalent. Yuri is also arranging to plug this mechanical model into the official l1calo Test Stand in building 104. The l1calo crate in this test stand is understood to be an exact replica of the online system. We are not planning on disturbing any l1calo crate in USA15 at this time. The MSU l1calo test stand is currently at CERN and will soon be shipped to MSU. > 2. General Description > > Implementation locale questions: (Base vs Topo FPGA, > system vs non-system CMX board): > Sort/zero suppress firmware: all CMX's? > HT partial sums if needed: all Jet CMX's? > CMM functionality: all CMX's? > Extended CMM functionality (More thresholds): all CMX's? > CTP interface with more thresholds: system CMX's only? Sorry... No answer from our hardware-centric point of view. > How many trigger bits does CTP allocate for CMX use > (assuming their Phase 0 baseline)? This could even be divided into two questions/answers: How many trigger bits are currently allocated to each type of CMM? How many additional trigger bits will be needed/allocated to each type of CMX? Sorry... No answer from our hardware-centric point of view. > How many CTP trigger bits does our CMX design support? Each CMX can drive up to two cables with 31 bits of triggering information on each cable for a maximum of 62 bits at 40 Mbps. The CMX would also support sending information to CTP at 80 Mbps. Stephen Hillier had circulated a spreadsheet with the intended usage of these cables. The most recent version that we have for this document has been copied to http://www.pa.msu.edu/hep/atlas/l1calo/cmx/specification/3_prototype_review/Steve_CableFormats_v2_20121213.xlsx > 2.1 > > Is the g-link is an fpga emulation rather than g-link chipset? Yes, that was part of the PDR specification. > Even strictest CMM emulation mode requires updated software because > the register model has changed. Further, the cmm emulation firmware, > even without expanding the number of CTP bits, requires new firmware, > since the application of threshold and the counting of > corresponding TOBs is moved from the CPs and JEMs to the CMXs. > The thresholds now have to be downloaded into the CMX, instead of > the CP/JEMs, though at least the lowest of the thresholds still needs > to be sent to those boards. That depends on the meaning of "CMM emulation mode". If we are still planning on running the CMX as a replacement of the CMM using the current data format with backplane inputs at 40 Mbps, this meaning of "CMM emulation mode" would not require new thresholds to be downloaded into the CMX. But even for this "CMM emulation mode", the map of VME registers and the control software required to program the CMXs will likely be different from the software currently used for programming the CMMs. "CMM emulation function" may describe the section of the Base Function FPGA firmware on a CMX now receiving the new backplane input format at 160 MBps with the purpose of generating counts of objects above thresholds and of sending triggering information to the CTP in a format identical to or expended from what the CMM is currently sending. Wojetk, Pawel and Yuri are working on this firmware and related VME register maps. All VME address lines are routed to all 3 FPGAs (BSPT, BF, TP) thus any arbitrary division of the address map among the 3 FPGAs is possible. > Is there a full list of software tasks > (control/download/monitoring/simulation/analysis-validation) > for the various steps of CMX commissioning? > > Have people been identified for all these tasks? > > Is there a commissioning plan? There is no inclusive list or plan at this time. We are currently focusing on building the CMX. Wojtek, Yuri and Seth are currently learning how to use the test stand and online software in advance of the existence of the CMX prototype. We are new to the l1calo software environment and welcome all guidance and assistance. > Are there any latency issues in applying the threshold bit counting > algorithms in the CMX instead of the CP/JEMs? This seems to require > a cached copy of the inputs in addition to the reprocessed version > sent to L1Topo; does this incur any latency in the path to L1Topo? The Firmware document prepared for this review describes two independent paths for the backplane input data after it has been received and time-demultiplexed, cf http://www.pa.msu.edu/hep/atlas/l1calo/cmx/specification/3_prototype_review/CMX_Firmware_Spec.pdf > 2.4.1 > > If I understand, the current baseline is to use 6 to 12 of the > potential 24 output CMX fibers. Will all 24 be implemented? Yes, all CMX cards will be capable of driving up to 24 fibers. > If we want to make two copies of all (zero-suppressed) outputs > to L1Topo, how large a perturbation is that > for the envisioned firmware? > If one set of outputs is 6 fibers? 12 fibers? This type of flexibility is indeed being incorporated in the Base Function firmware. > 2.4.2 optional TP > > Is this mode essential to initial testing? > It seems to be the only way CMX can receive fiber inputs. The real TP firmware is not needed for initial testing, but a TP FPGA will indeed be required for high speed link tests. > 72 input fibers could accept 6 fibers from each CMX, > i.e. one zero-suppressed copy. > > However, the TP inputs are only 32 fibers each. > Are there realistic use cases able to work with 1/2 of the Topo > input fibers? > Do you imagine using all outputs from half the CMX's? > Or accepting a higher overflow rate by using only 3 fibers > from each CMX? As far as we know there is no clear plan yet regarding if or how the CMX Topo functionality will be used online, but this functionality and flexibility was requested and is being included on the CMX platform. > 3.1.1 > > Do the Crate to System protocols change sufficienty that new firmware, > software, simulation etc are required? In CMM mode? In CMX mode? Yes. The CMM firmware needs to be ported and adapted. E.g. the CMM has two FPGAs, one for the Crate CMM functionality and one for the System CMM functionality. The CMX Base Function FPGA implements both of these aspects. It is our understanding that Pawel has been working on this. > 3.1.2 (see also 2.4.1) > > How critical is an optical patch panel if 6 output fibers are standard? The patch panel would need to split 6 fibers off from each 12-fiber ribbon from the 12x CMX cards and merge them into 2 or more of the 48-fiber ribbons used as inputs to L1topo. (figure 8 and 11 showing "N x 12" fibers sent to L1topo are no longer realistic and should probably say "N x 48" fibers) > Would it be acceptable to simply have the L1Topo ignore fibers 7-12 > or does that waste input resources? We would still need to merge the 12-fiber ribbons into 48-fiber ribbons. Some sort of ribbon splitting patch cables would still be required. One detail: The initial L1topo modules will be equipped with the smaller Virtex-7 XC7VX485T devices supporting "only" 56x GTX transceivers per chip. Later versions of L1topo are planning to use the XC7VX690T with 80x GTH transceivers per chip. > 3.1.3 L1Topo CMX > > Do we have an estimate of the latency penalty (or gain) > of L1Topo CMX compared to L1Topo? Not much effort has been put into any L1topo CMX firmware. The timing performance should presumably be similar, with a reduced input and output bandwidth and reduced available processing logic. > 3.2 > > VRN, VRP add to glossary? oops, sorry. Those are the names of the pair of Select IO pins used for the pair of external resistors optionally available to the Digitally Controlled Impedance Virtex-6 technology to set the input impedance of the 400 backplane inputs. This is a backup feature as the nominal plan is to NOT terminate the 400 processor input signals. > Is the IODELAY algorithm firmware or software? The IODELAY is a Virtex-6 hardware and firmware feature which we will have to use in the Base Function FPGA to de-skew the 25x individual lines within each set of Processor inputs. A control value will need to be downloaded for each processor input signal. This value will control the effective delay applied to that particular input as it is latched using the clock signal accompanying that processor source. Using this feature and determining the appropriate set of IODELAY control values will require new dedicated software to find the range of control values yielding a reliable reception of a known test pattern and then pick e.g. the center of that range as the best operational value. > Is it fixed during a commissioning scan, > or must it be part of the final firmware suite? The IODELAY control values themselves are not part of the firmware. They will all need to be re-determined any time a CMX or one of the processor modules is being replaced. It is assumed that the relative timing of the inputs from the Processor Modules and thus the required set of IODELAY control values will then remain fixed over time and from one power up to the next. These values would need be downloaded by the online control software into each CMX card once after each power up and/or FPGA configuration and would not need to be modified until the FPGA is configured again. > When is the first time a full test with 400 active input lines > on the backplane can be performed at 160MHz? The MSU test stand will only be able to test inputs from one processor module at a time. The CERN test stand in bldg 104 includes more processor modules but does not presently include a full set of modules. This would be the place where such a test would first take place and it might require rounding up additional spare or borrowed modules. > How much modification to the board will be required for 60 Ohms? This involves the trace width, the dialectic thickness and is ultimately specified as a requirement to the manufacturer. It is not a big issue to decrease this number, once we know what value we want. > There has been mention of mini-vias and blind vias, > and full 3-d simulations in the High Speed Demonstrator for efex. > Have these technologies been applied in CMX layout? For all 6.4 Gb traces we will use vias spanning only the top 3 trace layers (i.e. the top 5 physical layers) of the card as described in http://www.pa.msu.edu/hep/atlas/l1calo/cmx/hardware/details/cmx_ab_routing_layer_strategy.txt The 6.4 Gb traces will be fairly short and follow the Xilinx guidelines. We don't think we will be pushing the technology. > 3.5 miniPOD height > > Is 14.5 mm > VME board separation tolerance? > Is it part of the physical board test? > What is the suggested resolution of this issue? This needs to be discussed during the review meeting. We could glue MiniPOD parts to the mechanical model and ship it back to CERN for more tests if desired. > 3.6; 4.1 G-link; Clocks; Appendix H > > The L1Topo board has both 40.08 and 40.00 clocks. > Are both required on CMX to synchronize properly with L1Topo? The link to L1topo only needs to be synchronous with the 40.08 MHz LHC clock. > With CTP? The link to CTP only needs to be synchronous with the 40.08 MHz LHC clock. > The existing RODs? This aspect is unclear and we are soliciting a confirmation. We are hoping that this review can confirm that it is ok for the CMX to use an LHC-derived 40.08 MHz clock for the g-links to the RODs. > Any issues with tracking the LHC clock during ramping? All systems should be tracking the same LHC clock signal during ramping. We have received detailed information about the ramping rates for LHC and they are slower and easier to deal with than the Tevatron ramping rates. All systems should also be resilient and able to re-synchronize to each other if necessary once the LHC frequency has reached its plateau. > 4.2 > > Which CMM LEDs would not be able to be supported? > Which have users found most or least important? Here is our understanding and initial proposal http://www.pa.msu.edu/hep/atlas/l1calo/cmx/hardware/details/cmx_ab_leds_on_front_panel.txt All but one of these LED signals will be controlled by the Board Support FPGA and thus not constrained by the circuit board layout. What is being displayed onthe LEDs can evolve over time (testing vs online) or could even be switched among a set of predefined modes under VME control should such complication proves desirable. > 5. Power > > What is expected height of heat sinks? > Compared to miniPOD or stiffeners? Here are the current notes on heat sinks http://www.pa.msu.edu/hep/atlas/l1calo/cmx/hardware/details/cmx_ab_heat_sinks.txt All proposed heat sinks are within VME component height specification. > 7. Build count > > What is expected functionality for TP FPGA instances of CMX? > Backup/in situ testing? > Sharing algorithms with L1Topo such as HT? > (I assume HT would have to be a base FPGA function since > TP FPGA has only inputs, no outputs?) Sorry... No answer from our hardware-centric point of view. ---------------------------------------------------------------------- Follow up from Jim: > Several "no response"'s: > > For the bit/fiber/cable counts, maybe that's more Sam's responsibility > than yours--he'll say as review leader. But it really does need > to be written down clearly. We are trying to be sure all the hardware > plays together correctly, and these counts are part of the > requirements. Now would seem to be a rather good time to do so. > The sort of thing a review should prod us on. We agree that this needs to be done. We are flagging that we do not have the resources right now to lead this effort and we have a limited view of all constraints involved. We are instead focusing on building a versatile platform able to support all current best guesses as well as future ideas. > A related question for interconnect cables: who is specifying > and buying what? Yes, those are good questions too. > For example, any new cables between CMX and CTP? Fibers between CMX > and L1Topo? and the optical patch panels to match bundles between CMX > and L1Topo. > Will the ROD fibers be usable unaltered? or need to run through > the optical patch panel? All aspects involving the strict replacement of the existing CMMs should, by definition, not require new cables or patch panel. The existing Crate CMX to System CMX cables, the CTP output cables, the DAQ and ROI fiber cables can all be used as they are. New CTP cables will be needed wherever the CMXs send more information than the CMM were sending. By comparing Steve's tables to the CMM documentations, all 4 System CMXs will be sending 2 cables of information to CTP. This means one additional cable from the EM, Jet and Energy System CMX for a total of 3 new CTP cables. Looking at the pictures Chip took earlier http://www.pa.msu.edu/hep/atlas/l1calo/reference/l1calo/cmm/photos/usa15/ it seems like new CTP cables will indeed need to be installed. e.g. a CMM in this crate only has one CTP cable connected (on the right side CMM, presumably the Energy System CMM): http://www.pa.msu.edu/hep/atlas/l1calo/reference/l1calo/cmm/photos/usa15/IMG_1122.jpg The connections between the 12 CMXs and L1topo indeed needs to be understood, then defined in details, then patch fibers with or without a patch panel will need to be specified, purchased and installed. The group needs to identify who will do this. > On section 7, I guess I would still like to hear the rationale > for the chosen build count. Somebody has to take responsibility > and provide rationale for spending. This indeed needs to be examined and either approved or amended by this review panel. These decisions involve the number of spares required, the number of test stand modules required, and the number of CMX TP used in the running system. Not clearly knowing if or how the CMX L1topo functionality will be used is a limitation. Cards initially built without a TP FPGA installed should be able to be upgraded later (for order of $5k each). > L1Calo as a whole needs the software task list. CMX builders have > critical input, as they know where they have modified things so that > "existing software" does not suffice. Again, Sam as project leader > will have to decide how much of this to press now as part of this > review. We agree that the CMX project will need this detailed list but this list does not exist at the time of this review. We have a limited understanding on what software is required, of who should write it, and what it needs to look like. Wojtek, Yuri and Seth have started to investigate this. We welcome outside input to help us understand all aspects involved. ---------------------------------------------------------------------- Update from Yuri: Just to add to the future software task list (if not already added): - SW to set up IODELAY in CMX modules for all 400 input pins from the crate processor modules. I hope, Mainz experience with the BLT can help. - SW to re-program the CF card in the CMX via VME -- via XILINX System ACE chip, again using JEM experience from Mainz, - include CMX in the HDMC (HARDWARE DIAGNOSTICS, MONITORING AND CONTROL). ---------------------------------------------------------------------- Update from Yuri: The Mechanical Model has now been tested in the main L1calo test rig located in CERN building 104 and pictures were added: http://www.pa.msu.edu/hep/atlas/l1calo/cmx/hardware/mechanical_only_card/