A Historic Look at Time Lost on the Keck Telescopes Dr. Robert W. Goodrich W. M. Keck Observatory
Table of Contents INTRODUCTION...3 EXECUTIVE SUMMARY...3 THE DATA...3 TIME LOST ON KECK 1...4 KECK 1 FEATURES...5 BIG PROBLEMS (BP)...5 KECK 2 FEATURES...6 INSTRUMENTS BY SEMESTER...7 FURTHER ANALYSIS...9 Page 2 of 9
Introduction The following contains an analysis of Remedy nightlog summaries from 1996 to November 2002. It relies heavily on the information entered, generally by hand, into the nightlog summaries by the OAs. We should hence keep in mind that the data on which conclusions are based may not be as reliable as the data that will (eventually) be produced by the duty cycle metrics program. Still, statistically the information can be expected to shed some light on some interesting questions about how well we are minimizing the time lost to faults. Realize, too, that many questions can be asked of the data. Each question, while requiring a moment to articulate, can be a large drain on resources to answer. So only a limited set of questions is addressed in the following. Executive Summary The bottom line from the current, limited analysis, is that the only major improvement that can be easily tied to an upgrade is that due to the NIRC P3 software upgrade. The improvement from roughly 18% time lost to 7% time lost implied an increase of 133 hours of time available for NIRC during 1998, or more than 13 extra clear nights! There are probably some effects from LRIS, which has gone through good and bad times, and currently is in a bad state. Major facility upgrades or repairs are not covered in the following analysis. However, it does appear that much of our time lost is due to big problems, that require more than 20% of a night to repair or work around. The Data We used the nightlog summaries to extract the following information: UT date Total dark time Time lost to weather Time spent doing engineering Time lost to faults (labeled as hardware, software, and other in the summaries) Instrument From this we calculate the following: time available = dark time weather loss engineering Page 3 of 9
Then we calculate time lost = time lost to hardware + software + other time lost / time available The goal here is to concentrate on the time that would have been available for science if things beyond our control, such as weather, were with us. Engineering is explicitly removed from the calculation because we do not record faults during engineering nights. Asking the question of how efficiently we use engineering opportunities is hence beyond the scope of this paper. Another set of information was gleaned from memory, with the help of historic telescope schedules. This was the approximate dates of significant milestones in the lives of the two telescopes, such as new instruments beginning science use, important upgrades, etc. Time Lost on Keck 1 The following graph shows the time lost on Keck 1. Each blue point is a single night. Each vertical gridline delineates a calendar year. The red line near the bottom is roughly an 8-week running mean, calculated by summing the numbers of hours of time lost and time available and then calculating the ratio. Milestones are marked as purple triangles near the top. Not all milestones are labeled,and in particular periods in time with facility problems, such as dome problems or Time Lost, Keck I 100% 90% LRIS off NIRC LWS on LRIS: on off blue 8-week average Fraction of available time lost to faults 80% 70% 60% 50% 40% 30% 20% Nightly time lost Important dates 10% 0% 1/1/96 12/31/96 12/31/97 12/31/98 1/1/00 12/31/00 12/31/01 12/31/02 1/1/04 Date (UT) Page 4 of 9
chronic ACS problems, are not shown. Note that many nights are down on the zero line, indicating no time lost during those nights. Nonetheless, there are an alarming number of nights at or near 100% loss. Keck 1 features The running mean shows a significant drop (improvement in our context) around the end of 1997. During the last 2/3 of 2000 there is a rise in fault ratio, and in 2002 we seem to be having more problems than the recent past. Looking at cause and effect, the only obvious correlation is the significant drop at the end of 1997. This was coincident with the advent of the NIRC P3 software upgrade, that was targeted at reducing the number of NIRC software crashes, as well as making NIRC easier to learn, use, and script. During 1998, after the P3 upgrade, there were 1207 hours of time available during NIRC nights. An approximation of the improvement due to the new software is a drop from 18% time lost to 7% time lost, implying that during 1998 we gained back 133 hours of time available, or more than 13 clear nights! At $1/second, a relatively old, conservative rate, that works out to nearly $500,000 in observing time. (At the TSIP rate this would be correspondingly larger.) LRIS, known to have had significant serious problems in the past, does not have a strong effect in the graph. When LRIS was moved from K1 to K2 in late 1996, there is not a noticeable drop in time lost. However, when LRIS was moved back to K1 in early 2000, it may have accounted for the increase in time lost later that year. The case is not clear, however. Instrument-by-instrument analysis is shown later in this paper. Big Problems (BP) The number of points high up in Figure 1 begs the question of whether it would be more efficient to tackle the occasional large problem, rather than the myriad of smaller problems. To this end we removed from consideration any nights that had more than 20% time lost. This is shown as the green line in the following figure. While there may be some residual during 1996 1997, probably from the large number of NIRC software crashes that could add up to more than 20% of a night in bad cases. The rest of the green line is fairly flat, at a comfortable level of 2% 3%, indicating that the peaks in the red 8-week average are predominantly from nights where significant fractions of the night are lost. So tackling the big problems, or at least being better prepared for them, may be a profitable way of decreasing our time lost. Page 5 of 9
Time Lost, Keck I 50% Fraction of available time lost to faults 45% 40% 35% 30% 25% 20% 15% 10% 8-week average Nightly time lost Time lost - BP Important dates 5% 0% 1/1/96 12/31/96 12/31/97 12/31/98 1/1/00 12/31/00 12/31/01 12/31/02 1/1/04 Date (UT) Keck 2 features The Keck 2 graph below has more milestones labeled, due to the greater number of instruments on K2. The left-most part of the 8-week mean is probably spurious, driven more by nightlogs stored in Remedy during K2 commissioning rather than during normal science observing. Note that the LRIS on triangle marks the beginning of science observing, and keep in mind that the 8-week Time Lost, Keck 2 100% 90% LRIS on MAPS/STEPS on RBC2 on NSPEC/KCAM on NSPAO on ESI on RBC2 off LRIS off NIRC2 on DEIMOS on of available time lost to faults Fraction 80% 70% 60% 50% 40% 30% 20% 8-week average Nightly time lost Important dates 10% 0% Page 6 of 9 1/1/96 12/31/96 12/31/97 12/31/98 1/1/00 12/31/00 12/31/01 12/31/02 1/1/04 Date (UT)
mean is backwards looking, averaging the 8 weeks before the date that each point is plotted. The peaks at the beginning of 1998 do not correlate with any of the milestones. The peak in late 2002 may be due to DEIMOS commissioning pains. DEIMOS was pressed into science use very early during its commissioning, perhaps prematurely. Note in particular that the move of LRIS from K2 to K1 in early 2000 did not result in less time lost on K2. Time Lost, Keck 2 50% of available time lost to faults 45% 40% 35% 30% 25% 20% 8-week average Nightly time lost Time lost - BP Important dates Fraction 15% 10% 5% 0% 1/1/96 12/31/96 12/31/97 12/31/98 1/1/00 12/31/00 12/31/01 12/31/02 1/1/04 Date (UT) The graph showing the time lost ignoring nights with big problems is similar to that of Keck 1, albeit a little higher (4% 5%). Instruments by semester We all wonder whether some instruments are better than others, and in fact it is fairly obvious that this is the case. The following color-coded table quantifies this. To calculate these numbers the time line was broken up into observing semesters (Feb. Jul. and Aug. Jan.). The division by observing semester makes more sense than a division by calendar, because often new instruments often begin significant science use at the start of an observing semester. Page 7 of 9
Keck 1 Keck 2 Semester end Semester LRIS NIRC HIRES LWS LRIS NIRSPEC ESI DEIMOS NIRC-2 1/31/96 95B 11.8% 18.0% 5.4% 7/31/96 96A 15.6% 28.9% 5.7% 4.5% 1/31/97 96B 8.1% 16.8% 6.9% 33.5% 17.5% 7/31/97 97A 20.3% 7.5% 13.3% 1/31/98 97B 13.1% 4.7% 8.6% 7/31/98 98A 8.1% 3.2% 6.6% 1/31/99 98B 5.9% 2.3% 7.3% 7/31/99 99A 4.1% 3.1% 8.5% 5.1% 1/31/00 99B 8.0% 2.3% 6.6% 2.7% 13.3% 5.4% 7/31/00 00A 5.8% 3.1% 5.5% 7.1% 8.1% 3.6% 1/31/01 00B 10.7% 9.0% 4.0% 1.6% 8.0% 4.8% 7/31/01 01A 4.7% 5.8% 0.8% 2.0% 5.7% 3.4% 1/31/02 01B 6.8% 3.6% 2.2% 6.9% 6.8% 5.9% 7.3% 7/31/02 02A 13.2% 8.7% 3.3% 11.0% 8.8% 5.0% 1.7% 5.6% 1/31/03 02B 5.4% 5.1% 1.6% 11.5% 16.3% 10.3% 10.1% 5.3% Color coding: > 20% 10%-20% 5%-10% < 5% CAVEATS! The total time lost here includes all faults, not just instrument faults. Some systems, such as those behind AO, may suffer in this comparison because of the increased noninstrument faults (such as AO faults). Of concern are the red and orange semester/instrument combinations. You can see that NIRC was indeed problematic in the early days, before the P3 software upgrade. Since then it has been very well behaved. HIRES has always been wellbehaved, and in fact seems to be less problematic than in the past. (Remember that the problems include telescope problems, not just instrument-related problem, so this improvement could be because of improved telescope performance.) The first incarnation of LWS stands out like a sore thumb, presumably because it was not used for much science before it broke in a spectacular manner. The second incarnation is much more encouraging, although the trend during 2002 is very worrisome. Partly this is due to a problem inside the dewar that was followed by a short sequence of missteps causing new problems on the sky. Hopefully this set of problems is over, and we will learn from our experience in dealing with future problems. LRIS on Keck 1 is problematic and was in the past. On Keck 2 it had initial problem but then settled down, possibly to the large LRIS improvement project. The effects of noninstrument-related problems can be seen by looking behind the apparent poor NIRSPEC performance during the first part of semester 2002B. This shows a fault rate of 16%. However, looking at individual nights we found that ACS was the major problem: 4:31 lost on 8/17, 1:15 on 8/18, another 1:15 on 8/23, 2:15 on 8/24, and 0:12 on 9/2. A guider problem, possibly precipitated by the hard crash of kawa during the day, accounted for all of the time lost on 10/25. By and large, though, NIRSPEC has shown good performance. ESI is likewise well-behaved. DEIMOS had only a small science presence in 2002A, and in the first part of 2002B has suffered from a number of instrument- Page 8 of 9
related problems. NIRC-2 has had only a slightly larger presence, and looks encouraging, especially given that its statistics include AO faults. Further analysis Some further analysis avenues that suggest themselves is to retrieve lists of time lost tickets greater than some floor (30 minutes, 60 minutes, or even 120 minutes) to see what these big hits were and what fraction of time they cost. More finegrained analysis could include subdividing time lost tickets by instrument or system (ACS, DCS, guiders, etc.). What if scenarios can also be investigated, such as What if LRIS could be made as reliable as NIRC, HIRES, ESI, and NIRSPEC? Let s say this is 7.5% time lost. During 2002A we had 134.4 hours lost of 1020 hours of available time (13.2%). If the time lost were 7.5% of this 1020 hours (76.5 hours), this would give us an extra 58 hours of time available to science. (Note that this is not the same as 58 hours of open science shutter time!) For 10 hour nights, this is nearly 6 extra nights of observing per semester, or 12 nights per year! In contrast, large ACS problems (> 1 hr.) during 2002 have cost us only 15.5 hours. (The ACS data are not presented here, but were also gleaned from the Remedy database.) While we would like to avoid such large ACS problems, it appears from this quick and dirty analysis that LRIS is a better target for improvement. Page 9 of 9