QUOTE
Newly planned science activities will resume no earlier than Sol 24 as engineers look into how the spacecraft is handling larger than expected amounts of data.
This sounded alarming, and immediately brought the very scary Spirit sol 18 anomaly to mind. (That anomaly, in brief, had to do with too many files being kept in flash memory, which resulted in Spirit descending into a cycle of continuous reboots that might, if not stopped, have depleted the batteries and killed the rover within a day or two. Through heroic efforts Spirit was recovered and obviously returned to perfect health.)I requested an interview with someone from JPL and am happy to say I just got a call from Barry Goldstein. I'm copying here the entire text of what he said to me. I will be blogging this but am wondering if someone here could help expand a bit on the business about APIDs (Application Process Identifiers) and what part they play in an operating system. I started off by asking for more detail on the problem, and for him to compare and contrast with Spirit Sol 18.
QUOTE ('Barry Goldstein')
When the anomaly happened with Spirit, we lost communication. We never lost communication or control of the vehicle here. It's quite different. On Spirit we had a file management problem that ran amok.
What happened was, at one of the downlinks on sol 22, the engineering housekeeping data was being looked at by the spacecraft team. And they noticed one of the APIDs for a housekeeping data packet, which is normally generated only one to three times every time we do an uplink, was generated 45,000 times. It was a surprise, to say the least. And the reaction of the team was, the obvious which was concern about why the heck did this happen, and the other issue was we were concerned about two things. One, since the APID priority for this data type was very high, would it starve out any of the science data from being saved overnight because it's now so large? And the resolution of that it turned out, yes indeed, it was that large, and we ended up losing very low priority science data from sol 22. But the scientists are not at all concerned about that. The second concern we had yesterday was, we had a restriction on the amount of time it takes for the spacecraft to boot. I can't remember the total value but it's over 60 seconds. If it doesn't boot within a certain amount of time, it will reset and then eventually go over to the B side (it's block redundant, unlike MER). The reason we were concerned is that this data structure, now which is huge because of these 45,000 blocks, it has to pull that out of the flash as part of the boot process. And so we were concerned it would take too long and therefore it would side-swap. So we took some emergency action last night, and I'm happy to say we got the uplinks in due to the following things. Number one, we updated the priority of that APID such that it will restrict the amount of that data type to be saved in flash. Second thing we did is we lost science operations on sol 23. Third thing we did is up the priority of the downlink of that data structure that we generated so often so that we could retrieve what we have so it could help us diagnose the problem. The current state of the spacecraft is as follows.
We have the data down, we have the spacecraft under control, we have the size of the file system in control such that we're no longer worried abou tthe size of the file system growing and keeping us from booting appropriately. The second thing is, the only restriction we put on science activity for sol 24, which the science team is planning right now, is that they can't save the data to the flash because we want to keep the flash small, we don't want this thing to eat us alive. So what the team is doing now is planning sol 24. However, there's a little paradox here. Because we were in this anomalous state, we requested and received a bunch of contingency passes from MRO and Odyssey. So what ends up happening is we told the science team you can do whatever you want, because the only thing we are worried about was flash, we just are not going to save it to flash when we turn off. And we then told them we have all these passes. So as it turns out, what the science team is planning is the most data-rich sol we've had to date, because we have all these extra passes. I was joking with Peter that he should pray for these things more often because he gets more data.
{What other kind of memory is there besides flash?} We execute out of RAM, and every time we turn the vehicle off to save power at night, charge the batteries, we save off the critical data structures which include this file system with the telemetry that has not been marked as received on the ground. And that's what really ate our lunch is the saving of this to the flash. We ran out of room in the flash and that's what caused them to lose the science data, which was low priority. And then it's the time it takes to read it out of flash and get it down on the ground.
{What's generating all these APIDs?} We have a suspect, and I'd prefer not to go into a lot of detail, but the suspect has to do with the packet counter number for each of the packets that are stored. It's been less than 24 hours so I'd like to let the team get a chance to look at this and analyze it completely. At this point it's our prime suspect but that doesn't necessarily mean it will pan out.
Even though we have had this anomaly, the vehicle is under control. We lost a sol of operations, because when this occurred we stopped the uplink for that sol. We have the vehicle under control, we understand the problem, we don't know the root cause, but we've taken preventive measures to make sure it's still functional without risking a problem.
It's much less scary {than Spirit sol 18} but I'll feel a lot better when we know exactly what's going on. All these things are scary to one degree or another. I'd rather have this problem though; not hearing from a vehicle is disconcerting.
--EmilyWhat happened was, at one of the downlinks on sol 22, the engineering housekeeping data was being looked at by the spacecraft team. And they noticed one of the APIDs for a housekeeping data packet, which is normally generated only one to three times every time we do an uplink, was generated 45,000 times. It was a surprise, to say the least. And the reaction of the team was, the obvious which was concern about why the heck did this happen, and the other issue was we were concerned about two things. One, since the APID priority for this data type was very high, would it starve out any of the science data from being saved overnight because it's now so large? And the resolution of that it turned out, yes indeed, it was that large, and we ended up losing very low priority science data from sol 22. But the scientists are not at all concerned about that. The second concern we had yesterday was, we had a restriction on the amount of time it takes for the spacecraft to boot. I can't remember the total value but it's over 60 seconds. If it doesn't boot within a certain amount of time, it will reset and then eventually go over to the B side (it's block redundant, unlike MER). The reason we were concerned is that this data structure, now which is huge because of these 45,000 blocks, it has to pull that out of the flash as part of the boot process. And so we were concerned it would take too long and therefore it would side-swap. So we took some emergency action last night, and I'm happy to say we got the uplinks in due to the following things. Number one, we updated the priority of that APID such that it will restrict the amount of that data type to be saved in flash. Second thing we did is we lost science operations on sol 23. Third thing we did is up the priority of the downlink of that data structure that we generated so often so that we could retrieve what we have so it could help us diagnose the problem. The current state of the spacecraft is as follows.
We have the data down, we have the spacecraft under control, we have the size of the file system in control such that we're no longer worried abou tthe size of the file system growing and keeping us from booting appropriately. The second thing is, the only restriction we put on science activity for sol 24, which the science team is planning right now, is that they can't save the data to the flash because we want to keep the flash small, we don't want this thing to eat us alive. So what the team is doing now is planning sol 24. However, there's a little paradox here. Because we were in this anomalous state, we requested and received a bunch of contingency passes from MRO and Odyssey. So what ends up happening is we told the science team you can do whatever you want, because the only thing we are worried about was flash, we just are not going to save it to flash when we turn off. And we then told them we have all these passes. So as it turns out, what the science team is planning is the most data-rich sol we've had to date, because we have all these extra passes. I was joking with Peter that he should pray for these things more often because he gets more data.
{What other kind of memory is there besides flash?} We execute out of RAM, and every time we turn the vehicle off to save power at night, charge the batteries, we save off the critical data structures which include this file system with the telemetry that has not been marked as received on the ground. And that's what really ate our lunch is the saving of this to the flash. We ran out of room in the flash and that's what caused them to lose the science data, which was low priority. And then it's the time it takes to read it out of flash and get it down on the ground.
{What's generating all these APIDs?} We have a suspect, and I'd prefer not to go into a lot of detail, but the suspect has to do with the packet counter number for each of the packets that are stored. It's been less than 24 hours so I'd like to let the team get a chance to look at this and analyze it completely. At this point it's our prime suspect but that doesn't necessarily mean it will pan out.
Even though we have had this anomaly, the vehicle is under control. We lost a sol of operations, because when this occurred we stopped the uplink for that sol. We have the vehicle under control, we understand the problem, we don't know the root cause, but we've taken preventive measures to make sure it's still functional without risking a problem.
It's much less scary {than Spirit sol 18} but I'll feel a lot better when we know exactly what's going on. All these things are scary to one degree or another. I'd rather have this problem though; not hearing from a vehicle is disconcerting.
).