Level 1 Trigger Framework Resources programmed by COOR ------------------------------------------------------ Philippe Laurens, MSU -- Tuesday, July 15, 1997 Goal: This is a first working draft towards a Definition of the Communication between COOR and TCC to Program the resources controlled by TCC. The full protocol will include both the messaging protocol and the message contents. This version focuses on the message information content, and in particular we try to list all the concepts and resources that COOR will have to deal with. The messaging protocol includes what software messaging package is used (superset of D0IP?), how and when connection channels are opened, what fixed fields for message ID and message type, what acknowledgment message format, etc. I am (conveniently) trusting that Gordon, Scott, and Stu are leading the attack on messaging protocol, and that TCC/L1/L2 will simply follow along. I listed the L1 Framework resources organized in an hierarchical manner. A similar document will need to exist for the L2 Framework but it is already clear that there will be far fewer L2 Framework Resources. TCC will also control the L1 Calorimeter Trigger, and those resources will also need to be specified in a similar manner. There will however be no or very little change from what resources we had in Run I. It will just need a syntax update (cf. example at bottom of document). It is not yet clear how the programming of the L2 Cal Pre-Processor and L2 Global will transit through TCC. Jim Linnemann and I are involved in this effort, and Jim already has our draft ideas. This is yet another piece of communication that we should push to converge towards an "harmonious" L1/L2/L3 set of protocol. I am proposing a particular syntax in this document in order to be able to be more specific in the description of the resources and protocol. I have no problem with *tuning* this proposed syntax to converge to a common style between L1, L2 and L3. -------------------------------------- Some guidelines: Like in Run I keep simple ASCII/human readable format, but improve keywords and formatting (drop fixed length keywords) Like in Run I keep one acknowledge per message, or per message group Improve error information sent back in acknowledgment. And improve feedback from COOR to TAKER, DAQEXP, and logfile to show offending message and corresponding acknowledgment information. Condense the NUMBER of messages with respect to Run I Maybe simply by making the syntax more flexible and allowing setting several/all features of one common object in one message Group the messages: e.g. one message or group/batch per SpTrg Allow setting several/all properties of a SpTrg in one transaction Still need extra message to enable/disable one or a set of SpTrg Still need extra message to change prescale ratios between runs Condense the LENGTH of messages by using ranges in syntax (e.g. a range of geographic sections 0:127) Carries same information as exhaustive list Actually emphasizes a contiguous sequence and highlights the holes More compact This was used with great success in Run I L1 CT Reference Sets. Minimize the number of SpTrg features to setup but without loosing on functionality and flexibility by clearly defining a default configuration. Implement all message types necessary to access all bells and whistles, But suppress any messages that match standard/normal default settings. Split "INIT" in 2 pieces (because of Run II use of FPGA technology) Download all FPGA configurations (only after power-up, slow = minutes) Reset to default programming (clean restart, faster = seconds) Make use of uppercase-lowercase combination of characters in keywords to help human readality. COOR should use the uppercase-lowercase spelling, but the parser on TCC should be case-NON-sensitive. Since TCC receives on one or separate communication channels messages for (at least) the L1 and L2 Frameworks (and probably L2 Cal PreProcessor and L2 Cal Global), try to make the destination subsystem somewhat apparent in the keywords. ---------------------------- I) Global System Control 1) Initialization a) Power-Up Configure (configure all FPGAs) We will be using Field Programmable Gate Arrays to implement the logic on the cards of the Level 1 and LEvel 2 Frameworks. These Chips wake up blank and need to be configured. One method to program this kind of chips is to have a little serial PROM next to each part that the FPGA knows how to suck its programming out of. We chose instead to have TCC download the configuration to make the implementation easier and more flexible. We have 16 FPGA's per card, and will have 80-100 cards to configure. Most cards have the same programming downloaded in all 16 FPGA's in parallel. This configuration may be part of the TCC booting sequence, or of the Trigger Control Program initialization, but it still makes sense to give COOR the ability to re-configure the hardware for special cases or to get out of a sudden "crisis". Example Syntax: L1FW_Configure L2FW_Configure This configuration will probably take 2-3 minutes, and COOR *MUST* be able to wait that long, at least for this message, even if shorter timeouts are appropriate for all other messages. If this step hasn't completed, or fails, there is no point in thinking the system can then be programmed to operate. The operator might still want to be able to bypass an error or warning acknowledgment after notifying us. b) Initialize (load Default Programming) Example Syntax: L1FW_Initialize L2FW_Initialize This is the more familiar step where TCC programs all the L1 FW resources to make the hardware operational and program all resources to the default-non-allocated-state. This default-non-allocated-state puts all resources in a stand-by state where they do not contribute to triggering. A detailed definition of this default-non-allocated-state will need to be included in the final document. This configuration will probably also take 1-2 minutes, and COOR *MUST* be able to also wait that long for this message, even if shorter timeouts are appropriate for other messages. As for the message above, COOR should raise a red flag if TCC returns an error or warning and prevent further download of run configuration before the problem has been cleared or bypassed. c) Initialize Front-Ends via SCL Example Syntax: SCL_Initialize This might be something that COOR will never need to use. Still it might still be wise to put this access in COOR's emergency toolbox. When the Geographic Sections detect a buffer synchronization problem, they raise an error flag on the Serial Command Link (SCL) that will wake up TCC. TCC will (1) pause the run, (2) locate the error and log its occurence, (3) send Front-End Initialization commands on the SCL to ALL front-ends, (4) wait for all Front-Ends to advertize themselves as ready, and (5) resume the run. There are actually 2 types of errors detected and reported by the front-ends. They can detect "L1 Errors" which are synchronization problems with the "Buffers awaiting L2 Decision". They can also detect "L2 Errors" which are synchronization problems with the "Buffers awaiting transfer to L3". TCC logs these 2 types of errors separately, but the same sequence is used to recover from either type of error. 2) Run Control a) Pause b) Resume Example Syntax: L1FW_Pause L1FW_Resume Just like in Run I, COOR can, with one message, enable or disable ALL the Specific Triggers at once. The typical use of this message is to enable/disable a particular subset of Specific Triggers that were programmed earlier, while another run might be going on. This pause/resume will separate successive runs, but it was also used in Run I, via COOR, as a slow speed event flow throttling mechanism to control the load on Data Logger (I believe). II) Specific Trigger Exposure Groups (0:7) The new concept of Exposure Groups was introduced to allow a flexible, and manageable method of monitoring Specific Trigger Exposure. This is a compromise between allowing more than one Specific Trigger Luminosity requirement and not having to implement scalers for 128 (Specific Triggers) * 159 (Bunches). cf: www.pa.msu.edu/hep/d0/ftp/l1/framework/measuring_luminosity_online.txt An Exposure Group needs to include all aspects of Specific Trigger programming that might produce dead-time correlated to the distribution of the luminosity among the (up to) 159 Bunches in the machine. We believe there are two such sources of correlated deadtime. Some And-Or Term requirements are of that class (e.g. Single Interaction). The other source is the set of Geographic Sections digitized by a particular Specific Trigger which will thus be disabled by the Front-End Busy signal from these same Geographic Sections. The L1 Framework will implement a maximum of 8 Specific Trigger Exposure Groups. One or several Exposure Groups must be defined first, and the subsequent programming of a particular Specific Trigger will then specify that it belongs to one particular Exposure Group already programmed. 1) List of And-Or Terms (0:255) Example Syntax: L1FW_Expo_Group 0 And_Or_List 45 -56 Here we specify which And-Or Term(s) are part of a particular Exposure Group. The Exposure Group(s) identified by the Number(s) following the "L1FW_Expo_Group" keyword is(are) being programmed to require the list of And-Or Terms following the "And_Or_List" keyword. An And-Or Term can be requested to be asserted or negated (veto). A minus sign flags the veto terms (like in Run I). Specifying a new list overwrites any previous list, this is NOT an incremental programming. There could be zero And-Or Terms in the list, but this would be a special case, meaning no And-Or requirement. We can allow specifying ranges of contiguous And-Or Terms (n:m), knowing it will probably not be very useful for this particular type of message. 2) List of Geographic Sections (0:127) Example Syntax: L1FW_Expo_Group 0 Geo_Sect_List 1:56 58:127 Here we specify which Geographic Section(s) are part of a particular Exposure Group. The Exposure Group(s) identified by the Number(s) following the "L1FW_Expo_Group" keyword is(are) being programmed to require the list of Geographic Sections following the "Geo_Sect_List" keyword. Geographic Sections may NOT be negated. Specifying a new list overwrites any previous list, this is NOT an incremental programming. There could be zero Geographic Sections in the list, but this would be a special case, meaning no Front-End Busy requirement. This is a message that benefits from allowing ranges of Geographic Sections. - We can then allow specifying multiple resources in one message. L1FW_Expo_Group 0 And_Or_List 45 -56 Geo_Sect_List 1:56 58:127 III) Specific Triggers (0:127) 1) Andor Terms (0:127) Example Syntax: L1FW_Spec_Trig 0 And_Or_List 45 -56 127 89 Here we specify which And-Or Term(s) are part of a particular Specific Trigger Programming. The Specific Trigger(s) identified by the Number(s) following the "L1FW_Spec_Trig" keyword is(are) being programmed to require the list of And-Or Terms following the "And_Or_List" keyword. An And-Or Term can be requested to be asserted or negated (veto). A minus sign flags the veto terms (like in Run I). Specifying a new list overwrites any previous list, this is NOT an incremental programming. There could be zero And-Or Terms in the list, but this would be a special case, meaning no And-Or requirement. We allow specifying ranges of contiguous And-Or Terms (n:m), but it will probably not be typically useful here. 2) Exposure Group (0:7) Example Syntax: L1FW_Spec_Trig 0 Expo_Group 1 Here we specify which Specific Trigger Exposure Group this particular Specific Trigger is part of. There must be one and only one Exposure Group defined for a particular Specific Trigger. 3) Sources Of Disable a) Global Enable/Disable Example Syntax: L1FW_Spec_Trig 0 1 -2 COOR_Enable Here we specify which Specific Trigger is globally enabled or disabled. The Specific Trigger(s) identified by the Number(s) following the "L1FW_Spec_Trig" keyword is(are) being programmed to be globally enabled or globally disabled by COOR. A minus sign flags Specifc Trigger(s) that are being disabled, while no minus sign means that the Specific Trigger is being enabled The Default-allocated-state of a Specific Trigger is to NOT be globally enabled and COOR does NOT need to issue this message until the Specific Trigger is fully programmed and COOR wants to start a new run including this new Specific Trigger. b) Prescaler Example Syntax: L1FW_Spec_Trig 0 Prescale 1000 Here we specify whether a particular Specific Trigger is being prescaled and what the corresponding prescaler ratio value. The Specific Trigger(s) identified by the Number(s) following the "L1FW_Spec_Trig" keyword is(are) being programmed to require the Prescaler Ratio following the "Prescale" Keyword. An absence of Prescaler Ratio or a prescaler ratio of 1 will remove the prescaling feature. The prescaler Ratio is a 24 bit number (from 1 to 16 million, with the exact upper value defined later). The Default-allocated-state of a Specific Trigger is to NOT be prescaled and COOR does NOT need to issue this message, unless the prescaler feature is needed. c) Obey Front-End Busy Disable Example Syntax: L1FW_Spec_Trig 0 -1 3 Obey_FE_Busy Here we specify which Specific Trigger is programmed to obey its source of Front-End Busy Disable. The Specific Trigger(s) identified by the Number(s) following the "L1FW_Spec_Trig" keyword is(are) being programmed to Obey or Ignore its(their) Front-End Busy Signal. A minus sign flags a Specifc Trigger that is ignoring its Front-End Busy signal, while no minus sign means that the Specific Trigger is obeying its assigned source of Front-End Busy Disable Signal (i.e. defined by the Specific Trigger Exposure Group it belongs to). The Default-allocated-state, and normal mode of operation, of a Specific Trigger is to obey its Front-End Busy Signal, and COOR typically does NOT need to issue this message. d) Auto-Disable i) Auto-Disable Mode Example Syntax: L1FW_Spec_Trig 0 Auto_Disabled Here we specify which Specific Trigger is programmed to use its Auto-Disable feature. The Specific Trigger(s) identified by the Number(s) following the "L1FW_Spec_Trig" keyword is(are) being programmed to Use or Ignore its(their) Auto-Disable feature. A minus sign flags a Specifc Trigger that is NOT using the Auto-Disable Feature, while no minus sign means that the Specific Trigger is automatically disabled after firing. When a Specific Trigger is programmed to use its Auto-Disable feature, it must also be explicitly "re-enabled" as explained below. ii) Re-Enable Example Syntax: L1FW_Spec_Trig 0 Re_Enable Here we specify which Specific Trigger (already programmed to use its Auto-Disabled feature) is being re-enabled. The Specific Trigger(s) identified by the Number(s) following the "L1FW_Spec_Trig" keyword is(are) being programmed to be allowed to fire as soon all its other requirements are met, but it will fire at most once and be automatically disabled again until the next explicit Re-Enable message from COOR (or under TCC direct control in some cases). The Default state of a Specific Trigger that has just been programmed to use the Auto-Disable feature is to NOT be immediately enabled, and COOR will need to issue this message before the Specific Trigger can have a chance to fire. e) Obey Individual External Disable 0:1 f) Obey Global External Disable 0:4 g) Obey Skip next Crossing after L1 Accept h) Obey Global Internal Disable 1:4 Here we specify which Specific Trigger is programmed to obey various sources of Disable. A minus sign flags a Specifc Trigger that is NOT listening to a particular source of Disable, while no minus sign means that the Specific Trigger is obeying the particular Source of Disable Signal. It is not yet totally clear which sources of disable we will have. We reserved 2 sources that are individually routed to each Specific Trigger, 4 sources that are globally routed to all Specific Triggers and come from outside the L1 Trigger Framework, and 4 sources that are globally routed to all Specific Triggers and come from within the L1 Trigger Framework. One known usage of the Global Internal Sources of disable signals is to immediately prevent the Framework from firing on the Beam Crossing directly following a L1 Accept Decsion. The management of these sources of disable is subject to change. A better partitioning of the Global sources of disable might be to identify sources that are correlated to the bunch luminosity and sources that are not. The Default-allocated-state will be what we identify as the normal mode of operation of a Specific Trigger and COOR typically will NOT need to issue these messages. 4) List of L1 Trigger Accept Qualifiers (0:15) Example Syntax: L1FW_Spec_Trig 0 L1_Qualifier 0 2 Each Level 1 Accept Decision is marked by a set of 16 L1 Trigger Accept Qualifiers that is broadcasted to all Geographic Sections on the Serial Command Link along with the L1 Accept Decision. The same 16 L1 Trigger Accept Qualifiers are going to all 128 Geographic Sections. One anticipated usage is to tell some of the L2 Preprocessors whether they need to supply their information, and maybe which of their previously programmed threshold(s) they should use. Another known usage of L1 Accept Qualifier is to flag some events as Mark-and-Force-Pass to the Level 2 system (and the Level 3 system might chose to follow along). The Mark-and-Force-Pass Flag will be generated internally in the Level 1 Framework. This is another resource attached to a Specific Trigger that COOR will need to program (with an MFP ratio), but that is not yet included in this document. Another known usage of a L1 Accept Qualifier will be to flag the events that TCC is going to capture for monitoring every 5 seconds (including the Heartbeat Trigger Events). Each Specific Trigger can be programmed to assert any subset of the 16 Qualifiers, meaning that the corresponding Qualifiers will be set whenever this particular Trigger fires. When several Specific Triggers Fire the sets of Qualifiers associated with each Specific Trigger that has fired are merged (ORed together) to produce the resulting state of assertion of the 16 Qualifiers. The Default-allocated-state of a Specific Trigger is to NOT assert any L1 Tirgger Accept Qualifier and COOR does NOT need to issue this message unless COOR needs to specify a list of Qualifiers. Note that instead of listing which Qualifier(s) is associated with each Specific Trigger this resource could be programmed in the orthogonal way by listing which Specific Trigger(s) is attached to each Qualifier. 5) List of Geographic Sections to digitize In the normal mode of operation the list of Geographic Sections being digitized by a particular Specific Trigger will match the list of Geographic Sections whose Geographic Section Front-End Busy Signal disables this same Specific Trigger. If we think COOR should be able to override this default mode of operation, we can define a message to override it. TCC would first program both list to match the Specific Trigger Exposure Group definition, and COOR could override either list. For the moment, we think that COOR being allowed to force a Specific Trigger to ignore ALL Front-End Busy signals will be sufficient. 6) Deallocate Example Syntax: L1FW_Spec_Trig 0 1 2 Deallocate Here COOR can explicitely release a Specific Trigger and TCC will return the Trigger to its Default-NON-allocated-state. The Specific Trigger(s) identified by the Number(s) following the "L1FW_Spec_Trig" keyword is(are) being deallocated. Note that we did not define an explicit message to "allocate" a Specific Trigger. In Run I TCC was just paying attention to the first time COOR sent any message regarding a new Specific Trigger and would switch the Specific Trigger Programming to the Default-allocated-state. We can continue doing the same thing in run II. 7) syntax extension We allow Grouping several/all Specific Trigger messages into one message, or send them as a block of separate messages in one network transaction. L1FW_Spec_Trig 0 And_Or_List 45 -56 Expo_Group 1 Prescale 10 L1_Qualifier 2 or as a group L1FW_Spec_Trig 0 And_Or_List 45 -56 89 L1FW_Spec_Trig 0 Expo_Group 1 L1FW_Spec_Trig 0 Prescale 100 L1FW_Spec_Trig 0 L1_Qualifier 2 IV) Geographic Sections 1) List of Geographic Sections to detect L1/L2 Error from Example Syntax: L1FW_Geo_Sect 0:45 -46:-54 55:125 Operational As explained in section I.1.c, the L1 Trigger Framework monitors the Geographic Section Buffer Error lines. TCC could implicitely build a list of the superset of all Geographic Sections which are being read out by any of the Specific Triggers. Maybe a more attractive choice would be for COOR to explicitly specify the list of Geographic Sections to watch and ignore. V) Begin-End Run 1) Begin Run 2) End Run 3) Begin Store 4) End Store As in Run I, TCC will be requested to take a snapshot of ALL scalers and write a file. Writing the file could take a number of seconds, or even minutes depending on how responsive the destination file server was. In Run I COOR would ask for this task to be initiated, then COOR would go about talking to other systems (L3 I believe) and come back to synchronize with TCC. In fact TCC would first capture all the information, which is a nearly instantaneous process, then wait on the host to open and transfer the file. A better approach might be for TCC to acknowledge COOR's message as soon as it has captured the information, and report any file server problem, if any, through the Alarm system. This topic needs to be investigated. Example Syntax: L1FW_Snapshot Begin_Run 012345678 L1FW_Snapshot Begin_Store 01234 L2FW_Snapshot Begin_Run 012345678 ----------------------------------------------------------------------------- A quick look at what the L1 Cal Trig Messages could look like using a similar syntax philosophy. L1CT_Ref_Set EM_Et_Ref_Set 0 TT_Eta(-20:20) Phi(1:32) Value 10.0 L1CT_Ref_Set TOT_Et_Ref_Set 0 TT_Eta(-20:20) Value 10.0 L1CT_Ref_Set Large_Tile_Ref_Set 0 TT_Eta(-20:-13) Value 1000.0 L1CT_Ref_Set Large_Tile_Ref_Set 0 TT_Eta(-12:12) Value 22.0 L1CT_Ref_Set Large_Tile_Ref_Set 0 TT_Eta(13:20) Value 1000.0 L1CT_Energy_Threshold TOT_Et Comparator 0 Value 55.5 L1CT_Energy_Threshold TOT_Et Comparator 1 Value 100.0 L1CT_Energy_Threshold Miss_Pt Comparator 0 Value 40.0 L1CT_Count_Threshold EM_Et_Towers Ref_Set 0 Comparator 1 Value 1 L1CT_Count_Threshold TOT_Et_Towers Ref_Set 0 Comparator 1 Value 1 (Large tile counts are directly decoded and wired as And-Or Terms, i.e. no threshold is defined.) We also had messages to specify which specific Triggers were using which reference sets. This was used (mainly in L3) to build a jet list of seeds to run the L3 electron and jet algorithms on. In practice however *all* specific triggers using *any* reference set *always* also included the reference set with the lowest energy which totally defeated the purpose of these messages and the tailoring of the jet list. It is not clear whether we will still need this information, and whether it belongs in L1 or L3.