##################################################################################################### # # # IPMC and Shelf Manager Guide # # Author: Spencer Lee - leespen1@msu.edu # # # # Rev: September 5th, 2019 # # # ##################################################################################################### Table of Contents: Overview Resources & Links Shelf Manager IPMC ========= Overview: ========= This document will serve as an introduction and guide to the usage of the shelf managers and LAPP IPMC at MSU/CERN. This is a working document. It will be added to and corrected over time. Feel free to contact me with any corrections or requests for new information. What this document currently contains: Description of shelf manager with instructions for basic usage and common operations Instructions for installing/configuring ICARE Instructions for using ICARE to create IPMC firmwares and implementing them Additions to be made: More detailed descriptions of what is happening when configuring ICARE Description of ICARE package structure ================== Resources & Links: ================== ATCA Base Specification https://web.pa.msu.edu/people/edmunds/Do_-_To/va_alpha.pdf Pidgeon Point Shelf Manager User Guide https://web.pa.msu.edu/hep/atlas/l1calo/hub/hardware/components/commercial_atca/Shelf_Manager_UG_Aug_2017.pdf Pigdeon Point Shelf Manager External Interface Reference https://web.pa.msu.edu/hep/atlas/l1calo/hub/hardware/components/commercial_atca/Shelf_Manager_EIR_3_4_0_20140515.pdf 14-Slot ATCA AC/DC Shelf User Manual https://web.pa.msu.edu/hep/atlas/l1calo/hub/hardware/components/commercial_atca/Asis_MaXum_14_Slot_User_Manual_rev1.5.pdf IPMI Specification: https://web.pa.msu.edu/hep/atlas/l1calo/hub/reference/ipmc/ipmi/ipmi_mgt_interface_specification_2ndgen-v2p0-rev1p1_2013_10_01.pdf ============== Shelf Manager: ============== Main Resources: The Shelf Manager User Guide and External Interface Reference are both extremely useful for understanding the shelf manager. Overview: The shelf manager is a device required by the ATCA specification. Its responsibilities include managing the power and cooling of FRUs and the shelf infrastructure. This is done mostly through interactions between the shelf manager and IPMCs (ATCA Spec 3.3). The term shelf manager refers to the role required by the ATCA specification, while the actual piece of hardware that fulfils this role is referred to as the Shelf Management Mezzanine, or ShMM. OS: The ShMMs in use at MSU run some custom, lightweight linux, either Monterery Linux or Pigeon Point Linux, depending on the ShMM variant (User Guide pg 22). As such, they can be ssh'd into like any other linux computer. Furthermore, most standard linux operations work on the shelf manager. However, special care should be taken to consult the user guide and external interface reference on whether performing a certain action will affect shelf manager operations. For example, manual changes in the ShMM network configuration (i.e. using ifconfig) can disrupt Shelf Manager operations. Instead, "clia setlanconfig" should be used to configure the network. System Event Log: Interactions between the shelf manager and the IPMCs on a shelf occur through the generation of IPMI events. For example, an IPMC may report that the temperature of a device is too high by asserting a "Upper Critical (major) - going high" temperature event, which will prompt the shelf manager to increase the fan speed in the corresponding area of the shelf. Interacting with the Shelf Manager: Interaction with the shelf manager is done mostly through clia (command line interpreter agent) commands. They can be executed from the ShMM command line by "clia ". For a list of all clia commands, use "clia help". Some helpful/frequently used commands are: clia sel - Displays the system event log clia sensordata - Get sensor data values clia sendcmd - Sends an arbitrary IPMI command to specified IPMC (IPMI commands can be found in IPMC specification. Generally, clia commands are more conveneient ways of of executing IPMI commands) More information about a command can be displayed by using "clia help ", and for many commands "-v" can be added for verbose output. Detailed descriptions of all clia commands can be found in section 3 of the External Interface Reference. ===== IPMC: ===== ------ ICARE: ------ -------- Overview -------- ICARE stands for Intelligent platform management Controller softwARE. It is created by LAPP and is used for creating LAPP IPMC firmwares and downloading them to LAPP IPMCs. ------------------------------------- Setup - Downloading/Configuring ICARE ------------------------------------- First download the latest ICARE software stack (ICARE_160519.sh at time of writing). It can be found here at the following locations: http://lappwiki.in2p3.fr/twiki/bin/view/AtlasLapp/Informatique https://mydrive.lapp.in2p3.fr/s/SWXlWLJPjsfl03U/download (link directly to download) Once downloaded, move the the software stack to the desired working directory. This directory will be referred to as . Now run the commands: % cd % /ICARE_160519.sh The software stack may not have executable permission set when downloaded. If so, either use chmod to enable execution and use the above commands, or do % sh /ICARE_160519.sh Instructions for setting up ICARE and downloading firmwares can now be found at: /ICARE/releases/ICARE-00-03-01/doc/README Here, we provide a condensed version describing the setup in our specific use cases (on hubdev2 and hubdev3/pcl1c-stf-00). The specified supported operating systems are Scientific Linux CERN 5 and Scientific Linux CERN 6, but we have successfully configured ICARE on hubdev3 and hubdev3, which use Red Hat Enterprise Linux. Furthermore, we were unable to configure ICARE on pcl1c-stf-00, which uses Scientific Linux. Note for STF at CERN: Since we were unable to configure ICARE on the STF gateway machine, pcl1c-stf-00, we instead configured ICARE on hubdev3. We then used pcl1c-stf-00 First follow step 4.1 of the README (installation is redundant if ICARE has already been configured before, but no harm is done by attempting installation again): 4.1) Re-initialize CMT package % cd /ICARE/contrib/CMT/v1r22/mgr % ./INSTALL % source setup.[c]sh CMT is a configuration management environment that will be used throughout the configuration of ICARE. Since we are using the latest ICARE software stack, we skip steps 4.2 and 4.3. Now perform steps 4.4 and 4.4.1 4.4) Re-generate CMT setup scripts in admin package % cd /ICARE/releases/ICARE-00-03-01/admin/cmt % cmt relocate 4.4.1) Setting up of ICARE development environment - [DEFAULT] Set native compiler 'gcc version 4.1.2/4.4.7' and ARM compiler 'arm-elf-gcc version 4.7.0' with debugging flag % source setup.[c]sh <- use this default setting up - Set native compiler 'gcc version 4.1.2/4.4.7' and ARM compiler 'arm-elf-gcc version 4.7.0' with optimization flag % source setup.[c]sh -tag_add=opt Then execute the following: 4.5) Re-generate CMT setup scripts in ICAREPolicy package % cd ../../ICAREPolicy/cmt % cmt relocate 4.6) Re-generate CMT setup scripts in ICARERelease package % cd ../../ICARERelease/cmt % bcmt cmt relocate <- broadcast 'cmt relocate' command to all packages included in the release Now we build the ICARE release. Step 5.1 should ONLY BE EXECUTED ONCE. If changes are made to any ICARE source files, the modified packages should be rebuilt, as in step 5.2. V) Build 5.1) Build release (...THIS TASK IS EXECUTED ONLY ONCE...) % source /ICARE/releases/ICARE-00-03-01/admin/cmt/setup.[c]sh % cd /ICARE/releases/ICARE-00-03-01/ICARERelease/cmt % make release <- broadcast 'make' command to all packages included in the release 5.2) Re-build any package % source /ICARE/releases/ICARE-00-03-01/admin/cmt/setup.[c]sh % cd /ICARE/releases/ICARE-00-03-01//cmt % make clean all This ICARE release will function as the foundation for all IPMC firmwares you will create. For a specific implementation of the IPMC firmware, you will create a project which contains information specific to each board (i.e. what sensors are available and how to read them). Section VI deals with OpenOCD (Open On-Chip Debugger), which is one tool which can be used to download firmware to the IPMC. We will be downloading firmwares over ethernet instead, so this section is unecessary for us and will be skipped. -------------------------- Setup - Creating a Project -------------------------- First, create a directory to store the project in. We will refer to this directory as . Now use: % cd % create_project.sh -a -c -n -p FEX_Hub -v -p specifies the package name. Ours is FEX_Hub, but others exist for other IPMC implementations. -n specifies the host name to run the OpenOCD server on. It is a required field, but is not relevant to us since we will not be using OpenOCD. "localhost" should be a suitable placeholder. -v specifies the project version, and serves no functional purpose - it is purely for organization. As such, any version naming scheme works fine. Then follow step 8.1.2: edit file //config/config/boardconfig.h and change the unspecified PEN to: "#define IANA_MSU_PEN 0x000113" Also change the Manufacturer ID (under user settings in the same file) to: "MANUFACTURER_ID(IANA_MSU_PEN)" The do the following: 8.2) Re-build packages and your project % source /ICARE/releases/ICARE-00-03-01/admin/cmt/setup.[c]sh % cd ///cmt % make rebuild NOTE: This broadcast command re-builds the packages and your project 8.3) Build your project % source /ICARE/releases/ICARE-00-03-01/admin/cmt/setup.[c]sh % cd ///cmt % make bmc ------------------------------------ Setup - Downloading Firmware to IPMC ------------------------------------ We will download firmware to the IPMC via ethernet. To do this, we use fwu (FirmWare Upgrade utility). To enable use of fwu, do: % source /ICARE/releases/ICARE-00-03-01/admin/cmt/setup.[c]sh % cd //cmt % source setup.sh Usage information for fwu can be obtained by running "fwu -h". The IPMC is contains two MCUs (MicroController Units): the IPMC MCU and the IOIF MCU. When you build a project, firmware binaries are created for both MCUs. To implement a firmware in the IPMC, we must download the firmware binaries to the IOIF MCU and the IPMC MCU, then issue an upgrade command for the changes to take effect. IMPORTANT: The IOIF MCU should be upgraded before the IPMC MCU. When upgrading the MCUs, wait at least 1 minute between upgrading the IOIF and IPMC MCUs so that the IPMC MCU upgrade does not interrupt the IOIF MCU upgrade. WARNING: Make sure you download the IOIF MCU firmware to the IOIF MCU, and the IPMC MCU firmware to the IPMC MCU. Configuring the IPMC MCU with the IPMC MCU firmware can put the IPMC in a state where it cannot be talked to over ethernet. Then it will be impossible to change the IPMC MCU firmware back using ethernet. NOTE FOR CERN STF USAGE: At CERN there were issues configuring ICARE on the gateway machine pcl1c-stf-00.cern.ch, but not on hubdev3.cern.ch. However, only the gateway machine has ethernet access to the shelves in the STF (Surface Test Facility). So, our procedure was to build firmwares on hubdev3, then download the binaries using the gateway machine. On the gateway machine, fwu can still be used by specifying the path of fwu. So in the commands below replace "fwu" with "/ICARE/releases/ICARE-00-03-01/fwUpgrade/x86_64_slc7-gcc48-dbg/fwu" So, to implement a firmware in the IPMC, perform the following: Download IOIF MCU firmware % fwu -t IOIF -n -f //arm-gcc-47-dbg/bmc_IOIF.bin Download IPMC MCU Firmware % fwu -t IPMC -n -f //arm-gcc-47-dbg/bmc_IPMC.bin Issue IOIF MCU Upgrade % fwu -t IOIF -n -u (wait at least 1 minute) Issue IPMC MCU Upgrade % fwu -t IPMC -n -u