List of Known instances of VME Readout problem in the M122-Top crate containing the Foreign Per-Bunch Luminosity scalers and the M122-Middle crate containing the Exposure Group Per-Bunch Luminosity scalers. Rev: 02-Jul-2008 Unless specified, the information below comes from the L1TCC logfiles and the MSU logbooks. 01-MAR-2006 aproximate starting date of shutdown between Runs IIA and IIB 03-May-2006 Wednesday During shutdown started at 07:41 ended ? started at LBN = 0x0047a77b failure at slot ?? This was the first VME hang of M122-Top in the TFW since it was installed. 12-May-2006 Friday During shutdown started and ended sometime before 13:40 ended before LBN = 0x004810f2 failure at slot ?? 17-May-2006 Wednesday During shutdown started at 18:43 ended ? started at LBN = 0x00483539 failure at slot #11 which is card SM-31 07-JUN-2006 Wednesday During shutdown started at 22:55 ended ? started at LBN = 0x0048fc52 ended ? failure at slot #14 which is card SM-4 09-JUN-2006 First store of Run IIB 15-JUN-2006 Thursday not in a store started at 04:54 started at LBN = 0x00493188 ended ? failure at slot #14 which is card SM-4 19-JUN-2006 Monday store 4781 started 08:48 ended 09:04 started at LBN = 0x00494e41 (=4804161) ended LBN = 0x00494e55 (=4804181) assessment from the luminosity group: 6/19/06, 9:01, LBNs 4804160 - 4804181 http://www-d0online.fnal.gov/www/groups/tm/daqrates/47XX/store_4781.png failure at slot #11 which is card SM-31 17-JUL-2006 Monday store 4838 started around 7am started at LBN = ? assessment from the luminosity group: 7/17/06, 7:22, LBNs 4854491 - 4854518 http://www-d0online.fnal.gov/www/groups/tm/daqrates/48XX/store_4838.png failure at slot #11 which is card SM-31 28-JUL-2006 Friday store 4859 started at 14:26 ended ? started at LBN = 0x004a8503 (=4883715) assessment from the luminosity group: 7/28/06, 14:52, LBNs 4883714 - 4883736 http://www-d0online.fnal.gov/www/groups/tm/daqrates/48XX/store_4859.png failure at slot #11 which is card SM-50 31-JUL-2006 Monday store 4862 around 1am LBN = ? assessment from the luminosity group: LBNs 4887684 - 4887820 http://www-d0online.fnal.gov/www/groups/tm/daqrates/48XX/store_4862.png failure at slot #14 which is card SM-50 08-AUG-2006 Tuesday store 4895 started around ? ended around 10:00 order of magnitude neighborhood LBN = 0x004af687 (=4912775) assessment from the luminosity group: 8/8/06, 10:13, LBNs 4912722 - 4912763 http://www-d0online.fnal.gov/www/groups/tm/daqrates/48XX/store_4895.png failure at slot #14 which is card SM-50 This is an odd instance. From the logbook: Tuesday morning it appreas that the monitor data VME readout of the M122 Top Crate got hung up again. It was hard to tell what the problem was from the Rolling Rate Plots because the Luminosity in that plot just held constant at its previous value. Brendan says that he just reset M122 top crate to get things running again. He said it was SM-50. Consistency check that this was indeed an M122 error: There was indeed a bit3 timeout recorded at the next initialization on 09-Aug-2006 03:06 09-AUG-2006 Wednesday store 4897 started around 8:45? ended around 09:10 order of magnitude neighborhood LBN = 0x004afce3 (=4914403) assessment from the luminosity group: 8/9/06, 9:22, LBNs 4914380 - 4914400 http://www-d0online.fnal.gov/www/groups/tm/daqrates/48XX/store_4897.png failure at slot #14 which is card SM-50 10-AUG-2006 Install TOM module to delay the AS* by 100 nsec. 09-DEC-2006 Saturday First VME hang in 4 months Right *before* any data taking for store 5115 started at 11:44 ended around 21h ? started at LBN = 0x004e7cb1 (=5143729) assessment from the luminosity group: 12/9/06, 20:58, LBNs ??????? - 5144280 http://www-d0online.fnal.gov/www/groups/tm/daqrates/51XX/store_5115.png failure at slot ? 18-FEB-2007 Sunday not in store (before store 5234) started ? (but before 2am) ended around 2:50am order of magnitude neighborhood LBN = ? failure at slot ? 11-MAR-2007 Sunday store 5269 roughly around 0:45 order of magnitude neighborhood LBN = 0x0050f900 (=5306624) assessment from the luminosity group: 3/11/07, 0:57, LBNs 5306618 - 5306639 http://www-d0online.fnal.gov/www/groups/tm/daqrates/52XX/store_5269.png failure at slot #11 which is card SM-31 12-MAR-2007 Monday store 5274 around 1am order of magnitude neighborhood LBN = 0x0050ff55 (=5308245) assessment from the luminosity group: 3/12/07, 1:13, LBNs 5308166 - 5308187 http://www-d0online.fnal.gov/www/groups/tm/daqrates/52XX/store_5274.png failure at slot #11 which is card SM-31 13:15-MAR-2007 Install TOM module to delay the AS* by 200 nsec. 02-APR-07 Monday not in a store started about 5am until about 10am order of magnitude neighborhood started LBN = 0x00518c98 ended LBN = 0x00518df5 http://www-d0online.fnal.gov/www/groups/tm/daqrates/53XX/store_5315.png http://www-d0online.fnal.gov/www/groups/tm/daqrates/53XX/store_5316.png failure at slot ? 03-APR-07 Install RAY card with the 35 Ohm VME terminator 04-APR-07 Wednesday not in a store early afternoon, not sure exactly when failure at slot ? 10-APR-2007 Tuesday Containment solution: No corrupt luminosity info coming out of L1FW. Switch to new L1 TCC control software TRICS V11.2.A Automatically detect M122-Top VME access problem, and reset M122-Top Crate. Only data read while the crate was verifibly ok is sent to luminosity server. 16-Apr-2007 07:23 Monday LBN= 0x00523a7d run 232214 store 5355 detected before first read attempt 27-Apr-2007 08:46 Friday LBN= 0x0052813c run 232469 store 5379 detected before first read attempt 05-May-2007 15:59 Saturday LBN= 0x0052b631 run 232635 store 5410 detected after first read attempt 05-May-2007 17:24 Saturday LBN= 0x0052b686 same run 232635 same store 5410 detected before first read attempt 06-May-2007 10:59 Sunday LBN= 0x0052bb34 run 232651 store 5411 detected after first read attempt 08-May-2007 19:16 Tuesday LBN= 0x0052ca87 run 232745 store 5413 detected after first read attempt 10-May-2007 01:43 Thursday LBN= 0x0052d2d4 run 232768 store 5415 detected after first read attempt 14-May-2007 21:00 Monday LBN= 0x0052f2d9 run 232945 store 5429 detected after first read attempt 21-May-2007 00:15 Monday LBN= 0x00531baf run 233172 store 5450 detected after first read attempt 22-May-2007 09:03 Tuesday LBN= 0x00532484 run 233211 store 5453 detected before first read attempt 03-Jun-2007 06:17 Sunday LBN= 0x00536fcf run 233378 Store 5470 detected before first read attempt 07-Jun-2007 23:30 Thursday LBN= 0x00538e03 run 233490 store 5480 detected after first read attempt 08-Jun-2007 11:34 Friday LBN= 0x00539142 run 233494 same store 5480 detected after first read attempt 20-Jun-2007 00:45 Wednesday LBN= 0x0053dbb3 run 233736 store 5504 detected before first read attempt 23-Jun-2007 17:43 Saturday LBN= 0x0053f321 run 233798 store 5510 detected before first read attempt 24-Jun-2007 10:27 Sunday LBN= 0x0053f77f run 233809 store 5511 detected after first read attempt 01-Jul-2007 05:58 Sunday LBN= 0x00542459 run 234045 store 5530 detected after first read attempt 04-Jul-2007 16:58 Wednesday LBN= 0x005439a1 run 234100 store 5533 detected after first read attempt 14-Jul-2007 14:47 Saturday LBN= 0x00547872 run 234348 store 5552 detected after first read attempt 17-Jul-2007 10:39 Tuesday LBN= 0x005489c9 run 234398 store 5556 detected after first read attempt ------ begin shutdown 20-Sep-2007 23:08 Thursday LBN= 0x00567a53 run 235992 shutdown detected after first read attempt 21-Sep-2007 12:54 Friday LBN= 0x00567da8 run 235995 shutdown detected after first read attempt 23-Sep-2007 20:21 Sunday LBN= 0x00568b17 run 236016 shutdown detected after first read attempt 26-Sep-2007 20:20 Wednesday LBN= 0x00569fed run 236147 shutdown detected after first read attempt 26-Sep-2007 21:35 Wednesday LBN= 0x0056a03b run 236147 shutdown detected before first read attempt 30-Sep-2007 18:13 Sunday LBN= 0x0056ba61 run 236221 shutdown detected after first read attempt 13-Oct-2007 01:11 Saturday LBN= 0x00573998 run 236847 shutdown detected after first read attempt 13-Oct-2007 22:00 Saturday LBN= 0x00573ee0 run 236870 shutdown detected after first read attempt ------ end of shutdown 03-Nov-2007 23:17 Saturday LBN= 0x0057c625 run 237528 store 5681 M122-Top detected before first read attempt 29-Nov-2007 04:29 Thursday M122-Top LBN= 0x00588cf4 run 238220 not in store M122-Top detected after first read attempt 02-Dec-2007 00:00 Sunday LBN= 0x00589e72 run 238269 not in store M122-Top detected after first read attempt 08-Dec-2007 around 12:40 Saturday starting LBN=5818453 lost luminosity measurement for last ~20E30 5 hours of store #5787 M122-Middle Hangs until the COOR Initialization for next store 08-Dec-2007 20:24 Saturday LBN= 0x0058ca40 run 238453 store 5788 M122-Top detected after first read attempt 09-Dec-2007 21:20 Sunday LBN= 0x0058d09f run 238469 not in store M122-Top detected before first read attempt 13-Dec-2007 12:45 Thursday LBN= 0x00591028 run 238564 not in store detected before first read attempt 14-Dec-2007 start protecting against M122-Middle hangs 21-Dec-2007 19:37 Friday LBN= 0x0059468d run 238815 not in store detected after first read attempt 09-Jan-2008 19:25 Wednesday LBN= 0x0059bd1b run 239155 store 5835 detected after first read attempt 16-Jan-2008 21:08 Wednesday LBN= 0x0059ea1e run 239332 store 5850 detected before first read attempt 04-Feb-2008 20:12 Monday LBN= 0x005a634b run 239850 store 5885 detected before first read attempt 05-Feb-2008 11:29 Tuesday LBN= 0x005a6761 run 239859 not in store detected after first read attempt 12-Feb-2008 06:14 Tuesday LBN= 0x005a91d6 run 239998 store 5901 detected before first read attempt 18-Feb-2008 17:50 Monday LBN= 0x005aba78 run 240149 not in store detected after first read attempt 27-Feb-2008 16:32 Wednesday LBN= 0x005af27a between run 240353 and 240354 at end of store 5934 detected after first read attempt 02-Mar-2008 22:11 Sunday LBN= 0x005b0bdf run 240436 store 5946 detected before first read attempt 17-Mar-2008 14:34 Monday LBN= 0x005b6d47 run 240826 store 5989 detected after first read attempt 09-Apr-2008 16:04 Wednesday LBN= 0x005c00ae run 241505 not in store detected after first read attempt 14-Apr-2008 20:49 Monday LBN= 0x005c210e run 241614 store 6047 detected before first read attempt 30-Apr-2008 04:58 Wednesday LBN= 0x005c7ed6 run 241889 store 6099 detected after first read attempt 13-May-2008 05:37 Tuesday LBN= 0x005cd132 run 242236 store 6135 detected after first read attempt 27-May-2008 06:48 Tuesday LBN= 0x005d2795 run 242510 store 6166 detected after first read attempt 05-Jun-2008 13:11 Thursday LBN= 0x005d60b5 run 242713 store 6195 detected after first read attempt 10-Jun-2008 19:50 Tuesday LBN= 0x005d837f run 242874 store 6214 detected before first read attempt 16-Jun-2008 02:05 Monday LBN= 0x005da67e run 242987 store 6226 (right begin/before) detected before first read attempt 28-Jun-2008 12:05 Saturday LBN= 0x005df636 run 243383 store 6252 detected after first read attempt