Draft Draft Draft Draft Draft Draft Draft Draft Draft Draft Draft ------------------------------------------------------------------------------- http://www.pa.msu.edu/hep/d0/ftp/tcc/coor/Coor_To_Tcc_L2Crates_Message_Syntax.txt ------------------------------------------------------------------------------- Level 2 Preprocessor and Level 2 Global Processor Crates Resource programming by COOR ------------------------------------------------------------------ Philippe Laurens, MSU -- April 8, 2002 I. General ========== 1. Introduction --------------- This document defines the Syntax for Communication between COOR and the Level 2 Trigger Control Computer (L2 TCC) to Program the Level 2 Preprocessor and Level 2 Global Processor Crates serviced by L2 TCC (or by the Test Stand TCC). This document also describes the protocol for communication between TCC and the Administrator Alpha Processor of each L2 Trigger Crate. COOR programs the Level 2 Trigger Crates by sending command messages to the L2 Trigger Control Computer (L2 TCC) over an ITC client connection open on TCP port #52165. COOR commands are ASCII messages made of a command ID and a message payload. Other COOR documents describe the overall protocol, and TCC obeys these specifications. cf. http://www-d0.fnal.gov/d0dist/dist/packages/coor/devel/doc/coorover.ps TCC relays the programming information from COOR to the Level 2 Trigger Crates. TCC communicates with the Administrator Alpha Board of each Level 2 Crate. The L2 Crate Administrator is responsible for acting on these commands and propagating the information inside the L2 Crate. TCC and the Administrator exchange information through a Dual Port Memory module attached to the Bit3 PCI to VME Bus Adaptor of each L2 Crate. cf. tcc_to_l2_trigger_communication.txt in http://www.pa.msu.edu/hep/d0/ftp/tcc/vme_access/ and http://www-clued0.fnal.gov/L2/monitoring/dpm_layout.htm The L2 Processors inside the L2 Crate are configured using ASCII commands which are encapsulated within "L2Script" commands from COOR (described below). TCC does not parse or modify the content of these command strings but simply provides the mean to deliver the commands to the Administrator of each L2 crate. This document does not describe or discuss the syntax and details of the L2 configuration syntax. cf. http://www-clued0.fnal.gov/L2/config/L2Parser.html and http://www-d0.fnal.gov/d0dist/dist/packages/coor/devel/doc/coorover.ps This document defines what L2 TCC will do in response to each type of message from COOR. All command Keywords parsed by TCC are interpreted in a non-case-sensitive manner, and COOR is encouraged to use a combination of uppercase and lowercase letters that maximizes readability. L2 TCC controls several L2 Trigger Crates; nominally six: - L2 Global Processor Crate - L2 Central Muon PreProcessor Crate - L2 Forward Muon PreProcessor Crate - L2 Calorimeter PreProcessor Crate - L2 Central Tracking Trigger PreProcessor Crate - L2 Preshower PreProcessor Crate Some COOR commands are targetted at an individual L2 Crate (e.g. for downloading the L2 Global Trigger Script) while others are meant to be distributed to all L2 Crates currently connected (e.g. starting a run). During data taking, the processors in each L2 Crate are busy processing events and L2 TCC should not try to communicate with the Administrator during this time. TCC will thus buffer all L2 Trigger script messages and will only submit them to the Administrators of all crates when no data is flowing through the system. The bulk of the TCC to Admin communication thus happens at the begin and end run messages. The L2 Administrator does not need to put a high priority on checking for commands from TCC while it has events queued up for processing. The Administrator only needs to look for commands from TCC when it is done processing all its events. 2. Common Protocol COOR Messages -------------------------------- L2 TCC follows the "Common Protocol" described in the COOR documentation in section 8.2.1. of coorover.ps. L2 TCC processes all commands immediately and in the order they are received. TCC does not use the "batched command" feature allowed by COOR. The following messages are ignored, and no acknowledgement message is returned to COOR: - Begin_Block - End_Block - Abort The following messages are acknowledged, but no action is taken at the moment: - Configure - Begin_Store - End_Store - Pause_Run - Resume_Run The following common protocol messages are relayed to the L2 Trigger Crates, as described in detail below: - Init - Start_Run - Stop_Run 3. L2 TCC to L2 Admin Communication Protocol -------------------------------------------- 3.1) Shared Memory Layout ------------------------- The Address Range in the Dual Port Memory that is reserved for the communication betwewen COOR and the L2 Crate Administrator for downloading COOR Programming information is 0x00010000 to 0x000fffff cf. http://www-clued0.fnal.gov/L2/monitoring/dpm_layout.htm The L2 Programming Area in the Dual Port Memory Address Range is partitioned as follows: +--------------+-----------+------------------------------------------------+ | Address | Size | Purpose | +==============+===========+================================================+ | 0x00010000 | 4 Byte | L2 TCC to L2 Administrator Post Box | +--------------+-----------+------------------------------------------------+ | 0x00010004 | 4 Byte | L2 Administrator to L2 TCC Post Box | +--------------+-----------+------------------------------------------------+ | 0x00010008 | 4 Byte | Command Buffer Length in bytes | +--------------+-----------+------------------------------------------------+ | 0x0001000c | 4 Byte | Command Count | +--------------+-----------+------------------------------------------------+ | 0x00010010 | 4 Byte | Reserved | | 0x00010014 | 4 Byte | Reserved | | 0x00010018 | 4 Byte | Reserved | | 0x0001001c | 4 Byte | Reserved | +--------------+-----------+------------------------------------------------+ | 0x00010020 | 32 Byte | Optional L2 Administrator Status String | | | | (null terminated) | +--------------+-----------+------------------------------------------------+ | 0x00010040 | almost | Start of Command Buffer | | 0x000fffff | 1 MByte | End of Command Buffer | +--------------+-----------+------------------------------------------------+ 3.2) L2 TCC to L2 Administrator Post Box ---------------------------------------- The TCC to Administrator Post Box is Longword #0 at address 0x00010000. This Longword is written by TCC. The Administrator is allowed to clear this Post Box once it is finished processing the commands in the Command Buffer. The Administrator should poll this location at regular intervals (e.g. at 10 Hz), especially while it is waiting and finds no event in its input queue. Control Values for the "L2 TCC to L2 Administrator Post Box": +--------------+------------------------------------------------------------+ | Value | Meaning | +==============+============================================================+ | 0x00000000 | "Cleared": | | | No command to process. | +--------------+------------------------------------------------------------+ | 0x00000001 | "Wake Up": | | | TCC has placed a set of one or more L2Script | | | or Control commands in the Command Buffer. | +--------------+------------------------------------------------------------+ | 0x00000002 | "Configure" (aka "Reset"): | | | TCC has placed a set of one or more configuration | | | commands in the Command Buffer AND the Administrator | | | is requested to reset and initialize the L2 Crate. | +--------------+------------------------------------------------------------+ This location is initialized by TCC to 0x00000000 at boot time. The Administrator may also want to initialize this location to zero at boot time and whenever it is finished processing a set of commands in the command buffer. 3.3) L2 Administrator to L2 TCC Post Box ---------------------------------------- The Administrator to TCC Post Box is Longword #1 at address 0x00010004. This Longword is cleared by TCC. This Longword is written by the Administrator. Control Values for the "L2 Administrator to L2 TCC Post Box": +--------------+------------------------------------------------------------+ | Value | Meaning | +==============+============================================================+ | 0x00000000 | "Cleared": | | | Command not processed. | +--------------+------------------------------------------------------------+ | 0x00000001 | "Working": | | | The Administator has noticed TCC has submitted a (set of) | | | Command String(s) and is the Administator is now working | | | on processing the commands. | +--------------+------------------------------------------------------------+ | 0x00000010 | "Ok": | | | The Administrator is done processing the (set of) command | | | string(s) and is answering status "Ok". | | | The commands were understood, | | | and the requested actions were performed. | +--------------+------------------------------------------------------------+ | 0x00000020 | "Bad": | | | The Administrator is done processing the (set of) command | | | string(s) and is answering status "Bad". | | | The commands were not understood, | | | or the requested actions could not be performed. | +--------------+------------------------------------------------------------+ 3.4) Command Buffer Length -------------------------- The Command Buffer Length is Longword #2 at address 0x00010008. This Longword is written by TCC only. This is the number of characters (i.e. length in bytes) in the COOR Command Buffer. This includes all the characters embedded in the commands but does not include the final NULL string termination character. 3.5) Command Count ------------------ The Command Count is Longword #3 at address 0x0001000c. This Longword is written by TCC only. This is the number of COOR messages L2 TCC has buffered and concatenated (separated by one character, ascii code 0x0a) into the COOR Command Buffer. This is thus the number of commands the Administrator is expected to process. 3.6) Optional Status String --------------------------- The Optional Status String ranges from address 0x00010020 to 0x0001003f in the Dual Port Memory. This null-terminated string is reset by TCC before a new set of commands is submited to the Administrator. This null-terminated string is optionally filled by the Administrator before replying with an Ok or Bad Status in the Administrator to TCC Post Box. This Optional Status String may be used by the Administrator to qualify its Answer ("Ok" or "Bad") to a set of Commands. The Administrator is NOT required to always write something in the Optional Status String. The Optional Status String can be up to 32 characters long. If the Optional Status String is less than 32 characters long, it MUST be terminated with a NULL character. TCC will overwrite the Optional Status String with a zero length null string right before signaling the Administrator that there are new commands to process. 3.7) Command Buffer ------------------- The Command Buffer String ranges from address 0x00010030 to 0x000fffff in the Dual Port Memory. This null-terminated string is written by TCC only. This is the Command Buffer which consists of the concatenation of all the individual command strings received from COOR. Each individual Command String is terminated with an additional character (i.e. New Line character with ascii code 0x0a). The next Command String starts at the character following this last character. Successive command Strings are thus separated by a single character. The last Command String will be terminated with one NULL characters which can be crosschecked with the information provided in Longwords #2 and/or #3. 3.8) L2 TCC to L2 Admin Command Cycle Protocol ---------------------------------------------- To submit one or more command to the L2 Administrator, L2 TCC will initiate a Command Cycle: a) TCC copies the Command(s) to the dual port memory "Command Buffer" as one single, null-terminated string. Multiple commands are only separated by one character. b) TCC writes the "Command Buffer Length" and the "Command Count" Longwords to their respecitive address locations. c) TCC clears the "Admin to TCC Post Box". (by writing 0x00 as a Longword to its base address). d) TCC clears the Optional Command String (by writing 0x00 as a Longword to its base address). e) TCC writes "Wake Up" (0x01) to the "TCC to Admin Post Box" Then TCC periodically reads the "Admin to TCC Post Box" f) Error: The Admin doesn't wake up. If the "Admin to TCC Post Box" remains cleared (0x00) for longer than 1 second, TCC will consider that the L2 Crate is sick and report to COOR (and the Alarm System). This does not need to be a long timeout since no data is flowing, and the Admin should be able to poll the Post Box and at least advertize that it has received the command. g) Error: The Admin has written an initial "Working" (0x01) status to the "Admin to TCC Post Box" but doesn't complete the Command Cycle. If the "Admin to TCC Post Box" doesn't reach the "Ok" (0x10) or "Bad" (0x20) Status within 1 second, TCC will consider that the L2 Admin has run into trouble processing the command(s). TCC will report to COOR and the Alarm System. h) Success: The Admin completes the Command Cycle by writing "Ok" (0x10) or "Bad" (0x20) to the "Admin to TCC Post Box". TCC will wait until the Admin reports its Command Cycle Completion Status ("Ok" or "Bad"), and will read the Optional Command String to return to COOR. If the completion status was "Bad", TCC will send the "Optional Command String" to the Alarm system (to be defined). Finally TCC will clear the "TCC to Admin Post Box", and the L2 Admin can thus notice (if needed) that TCC is done with this communication cycle. The L2 Admin is allowed to take the initiative and clear the "TCC to Admin Post Box" as soon as it has replied with a completion status to the "Admin to TCC Post Box", corresponding to step (h) above. The Admin will then start polling again the "TCC to Admin Post Box" and wait until it changes value again which will signal the next command cycle. 4) L2 Script Definition Messages -------------------------------- COOR sends a number of "L2Script" messages to one or more L2 Crates (L2 Pre-Processor and/or L2 Global), cf Section II for details. This typically happens during data taking, while other runs are defined and events are flowing. L2 TCC will only parse the first keyword (after the "L2Script" Keyword) of each L2Script message from COOR and expect to find the name of one of the L2 Crates. This keyword must be in the following list: a) list of the L2 Crate Name Keywords used in COOR messages +----------+--------------------------------------------------+ | Keyword | L2 Crate Name | +==========+==================================================+ | L2GBL | L2 Global Processor Crate | +----------+--------------------------------------------------+ | L2CMU | L2 Central Muon Pre-Processor Crate | +----------+--------------------------------------------------+ | L2FMU | L2 Forward Muon Pre-Processor Crate | +----------+--------------------------------------------------+ | L2CAL | L2 Calorimeter Pre-Processor Crate | +----------+--------------------------------------------------+ | L2CTT | L2 Central Tracking Trigger Pre-Processor Crate | +----------+--------------------------------------------------+ | L2PS | L2 Preshower Pre-Processor Crate | +----------+--------------------------------------------------+ L2 TCC will receive the L2Script messages and buffer them separately for each L2 Crate using the above the L2 Crate Name to sort the messages. L2 TCC appends the L2Script command, as defined in I.3.7, including the L2 Crate Name Keyword and the rest of the COOR message without modification, but excluding the L2Script Keyword. L2 TCC does not communicate with the L2 Crate Administrator when a L2Script Message is received. L2 TCC acknowledges the message from COOR with an "Ok" Status. Only if appending the content of the L2Script message was going to overflow the Command Buffer maximum Length would L2 TCC answer with a "Bad" Status (and L2 TCC would not copy the message), which will be an unlikely occurence since the Command Buffer is nearly 1 MByte in length. 5) Begin and End of Run ----------------------- The Command Buffer is transferred to the L2 Crate Administrator only when L2 TCC receives a "start_run" or "stop_run" message from COOR. These two messages are treated identically. L2 Crates for which NO NEW L2Script Message have been received and buffered previously to the start_run or stop_run message are NOT contacted by L2 TCC. Only L2 Crates for which at least one L2 Script Message was previously recieved from COOR and is currently buffered by L2 TCC's will be contacted during the start_run or stop_run message. L2 TCC will perform 3 Command Cycles to the L2 Administrator (each Cycle as described in I.3.8 above): a) Send an "Exit Event Loop Mode" Command L2TCC will send the following single command " ADMIN TCC { COMMAND = "EXIT_EVENTLOOP" }" Where matches the name of the L2 Crate. TCC will keep Track of the Event Loop Mode for each crate and will need to skip this step if the L2 Crate is already out of the Event Loop Mode. b) Send the whole batch of buffered L2 Script Commands c) Send an "Enter Loop Mode" Command L2TCC will send the following single command " ADMIN TCC { COMMAND = "ENTER_EVENTLOOP" }" Where matches the name of the L2 Crate. L2 TCC will abort the procedure as soon as any of the Command Cycles fails or results in a "Bad" Status. 6) Crate Configuration ---------------------- An "init" message from COOR will be immediately processed. For each Crate L2 TCC will: a) dump all previously buffered L2Script Commands. b) read without parsing the content from a dedicated local disk file (e.g. "\l2rs\d0_config\Configure_L2GBL.cfg" for L2 Global). c) start a Command Cycle using the content of the above file as described in (I.3.8 ) with the sole difference that L2 TCC will write the value "Configure" (0x02) to the "TCC to Admin Post Box" instead of "Wake Up" (0x01) in step (e). d) wait for and process the reply from the L2 Crate Administrator as described in (I.3.8 ). 7) Additional Control and Debug Commands ---------------------------------------- The interface provides for additional message types in the style of commands coming from COOR, but which COOR may not need to use on a routine basis, or may not need to ever use. These commands will be available to the L2 Trigger Expert for debugging purposes through the L2 TCC console and windows interface. These messages will generate Command Cycles to the L2 Administrator, and the L2 Administrator is expected to handle them as described in (I.3.8 ) 8) Crate Identity and Crate Availibility Detection -------------------------------------------------- L2 TCC can manage up to 7 Bit3 PCI to VME interfaces. cf. tcc_to_l2_trigger_communication.txt in http://www.pa.msu.edu/hep/d0/ftp/tcc/vme_access/ L2 TCC does not rely on a fixed mapping of which L2 Crate is connected to which Bit3 interface. The Bit3 API (Application Program Interface) does not guarantee a fixed relationship between the physical PCI slot location of a bit3 interface card and the order in which the Bit3 software interfaces are presented by the API. The L2 Crates must thus identify themselves. Each l2 Crate Administrator must write its L2 Crate ID to the first Longword of the Dual Port Memory. cf. http://www-clued0.fnal.gov/L2/monitoring/dpm_layout.htm The L2 Crate ID numbers are assigned to match the L2 Crates respective Geographic Section Number. cf. http://www.pa.msu.edu/hep/d0/ftp/scl/crate_list.txt +-----------+-----------+ | L2 Crate | Crate ID | +===========+===========+ | L2GBL | 0x20 | +-----------+-----------+ | L2CMU | 0x21 | +-----------+-----------+ | L2FMU | 0x22 | +-----------+-----------+ | L2CAL | 0x23 | +-----------+-----------+ | L2CTT | 0x25 | +-----------+-----------+ | L2PS | 0x24 | +-----------+-----------+ At each Initialization request (or manually via TCC's console) L2 TCC will probe again for the existence, availability, and identity of all L2 Crates connected to all existing bit3 interfaces. A L2 Crate will not be classified as "available" a) if it is powered off, b) it its optical connection to the bit3 interface is disconnected, c) or if the Longword at the Base Address of the Dual Port Memory does not match one of the recognized Crate IDs. Only L2 Crates currently considered available by L2 TCC will be contacted and receive any of the commands send by COOR or via TCC's console interface. For a L2 Crate recently brought online to be included, L2 TCC must first be told to look for all crates again, either via an Initialization request from COOR or manually through the L2TCC Console interface. II) Normal Run Control ====================== 1) Initialize ------------- Syntax: Init This message causes L2 TCC to: a) probe for the existence and availibilty of L2 Crates and make a list of which crates are available (cf. I.8) b) for each available L2 Crate: send the configuration information and tell its L2 Administrator to configure the crate (cf. I.6) L2 TCC will contact the available L2 Crates sequentially, and in the Crate ID order 1) L2GBL, 2) L2CMU, 3) L2FMU 4) L2CAL, 5) L2CTT, 6) L2PS. If sequential access proves to be a limiting factor, this can later be changed to a more parallel approach. If any of the L2 Crates determined to be "available" during step (a) replies with a "Bad" Status to the configuration Command Cycle, or if any other fatal problem is encountered, L2 TCC acknowledges COOR with a "Bad" message. L2 TCC will also return a text string along with the acknwoledgement back to COOR. This string will include all of the L2 Crate Optional Status Strings. 2) Level 2 Script Programming ----------------------------- Syntax: L2Script must be one of the L2 Crate Name Keywords from list I.4.a. L2 TCC will only parse the message up to and including the Keyword. The Message Payload is not parsed by L2 TCC. The Message Payload may include one or more character(s). L2 TCC does not contact the targeted L2 Crate when a L2Script message is received and the Dual Port Memory is not accessed at this time. L2 TCC uses the to recognize which of its internal Command Buffer the new L2Script should be appended command to. The "L2Script" Keyword is NOT appended, only the rest of the message (i.e. " ", cf. I.4. COOR may occasionally send empty L2Script Messages (of the form "L2Script"), or L2Script Comment Messages (of the form "L2Script # ") which L2 TCC will be ready to ignore and simply acknowledge. 3) Beginning and End of Run --------------------------- 3.1) Start Run -------------- Syntax: start_run Begin and end of run is the only time when L2 TCC will try to forward the L2Script messages that were previously received from COOR. Only the available L2Crates for which L2 TCC's internal Command Buffer is currently holding messages will be contacted. L2 TCC will perform 3 Command Cycles to the L2 Administrator (as described in I.5 above): a) Send an "Exit Event Loop Mode" Command b) Send the whole batch of buffered L2 Script Commands c) Send an "Enter Loop Mode" Command COOR will guarantee that no events are flowing when L2 TCC receives a Start or Stop Run message. L2 TCC will contact the available L2 Crates sequentially, and in the Crate ID order 1) L2GBL, 2) L2CMU, 3) L2FMU 4) L2CAL, 5) L2CTT, 6) L2PS. If sequential access proves to be a limiting factor, this can later be changed to a more parallel approach. If one of the L2 Crates replies with a "Bad" Status to any of its the 3 L2 Command Cycles, L2 TCC skips to the next L2 Crate. If any of the L2 Crates replies with a "Bad" Status to any L2 Command Cycle, L2 TCC acknowledges COOR with a "Bad" message. L2 TCC will also return a text string along with the acknwoledgement back to COOR. This string will include all of the L2 Crate Optional Status Strings. 3.2) Stop Run ------------- Syntax: stop_run L2 TCC takes exactly the same action as for Start Run, cf. II.3.1. III) Special Control Messages ============================= COOR may not need to use these messages, while they are made available to the L2 Expert through this interface for test or debugging purposes. 1) L2 Crate Reconfigure ----------------------- Syntax: Configure_Crate | All must be one of the L2 Crate Name Keywords from list I.4.a. This message causes L2 TCC to send the configuration information to the named L2 Crate (or all Crates if "All" is specified instead of one Crate Name) and tell the L2 Administrator to configure the crate (cf. I.6) This message is not needed during normal operation, as the underlying action is part of the Full Initialize actions (cf II.1.). 2) Enter and Exit EVENTLOOP Mode -------------------------------- 2.1) Enter "Event Loop" Mode ------------------------- Syntax: Enter_EVENTLOOP |All must be one of the L2 Crate Name Keywords from list I.4.a. L2 TCC will perform a Command Cycle to the L2 Administrator (as described in details in I.3.8 above) with the following command to the L2 Administrator of the named L2 Crate (or all L2 Crates if "All" is specified instead of one Crate Name): ADMIN TCC { COMMAND = "ENTER_EVENTLOOP" } This command tells the L2 Administrator of the designated L2 Crate to switch mode and enter the "Event Loop" Mode and thus be ready to process events. L2 TCC will ensure that the ENTER_EVENTLOOP command is not issued when the L2 Administrator is already in "Event Loop" Mode. This message is not needed during normal operation, as the underlying action is automatically part of the Begin and End Run actions (cf II.3.1.), as needed. 2.2) Exit "Event Loop" Mode ------------------------ Syntax: Exit_EVENTLOOP |All L2 TCC takes the same action as for III.2.1 but with the following command ADMIN TCC { COMMAND = "EXIT_EVENTLOOP" } This command tells the L2 Administrator of the designated L2 Crate to switch mode and exit the "Event Loop" Mode and thus be ready to process Run definition commands (e.g. L2Script). L2 TCC will ensure that the EXIT_EVENTLOOP command is not issued when the L2 Administrator is already out of "Event Loop" Mode. This message is not needed during normal operation, as the underlying action is automatically part of the Begin and End Run actions (cf II.3.1.), as needed. 3) Collect Monitoring Information --------------------------------- Syntax: Collect_Status |All L2 TCC takes the same action as for III.2.1 but with the following command ADMIN TCC { COMMAND = "COLLECT_STATUS" } This command tells the L2 Administrator of the designated L2 Crate to immediately collect Monitoring Information and update the Dual Port Memory. This command can be issued any time, independently of the state of the L2 Administrator with respect to the "Event Loop" Mode. L2 TCC will keep tabs on the frehsness of the Monitoring Data read from the Dual Port Memory and will autonomously send such commands to the L2 Administrator when no fresh Monitoring Data has been received for the last 15 seconds (to be adjusted). One possible case is when no Data is flowing through the system as the L2 Crates rely on a L1 Qualifier to trigger the normal collection of monitoring data. This message is not needed during normal operation, as the underlying action is already part of the L2 TCC Monitoring Relay Service.