looking at the MER Pancam tracking website.... it seems none of the images for 612 were received, but images from previous sols were? Is this correct? Is Opportunity having more problems?

Sunspot, I've got the same.
And the same results for sols 611 and 610; only images from previous sols were downlinked.

What does it mean? My guess is that the planned sequences -- the ones we see on the tracking web -- were not executed due to any reason, so the data with less priority gets the opportunity to be downlinked.

Let's wait until either Steve or any of the insiders tell us something.

I remember several times in the past when we also did not hear anything from Oppy for a couple of days ... and yet anything was allright.
And I'm confident that this time it will also turn out to be a benign "send pause" ...

However, what really concerns me is yesterday's remark from pando (who is always very well informed):

[quote]
Unfortunately with all the software issues Oppy is experiencing it seems less and less likely that oppy will ever peek into Victoria
[/qoute]

hope, this refers only to the knwon software issue

Some interesting things planned for Sol 613


Sol Seq.Ver ETH ESF EDN EFF ERP Tot Description
--- -------- --- --- --- --- --- ---- -----------
613 p1420.01 0 0 0 0 0 0 hazcam_sunset_imaging
613 p1591.01 0 0 0 0 0 0 navcam_sunset_imaging
613 p2111.05 0 0 0 0 0 0 pancam_cal_targ_L234567Rall
613 p2396.06 0 0 0 0 0 0 pancam_snoopy_3cx1r_L2R2
613 p2539.13 0 0 0 0 0 0 pancam_foreground_L234567Rall
613 p2540.13 0 0 0 0 0 0 pancam_port_survey_L234567Rall
613 p2541.13 0 0 0 0 0 0 pancam_rear_survey_L234567Rall
613 p2600.07 0 0 0 0 0 0 pancam_tau
613 p2600.07 0 0 0 0 0 0 pancam_tau
613 p2694.05 0 0 0 0 0 0 pancam_dust_devil_hunter_waitfor180_L6
613 p2695.05 0 0 0 0 0 0 pancam_sunset_prepoint_L2R2
613 p2696.05 0 0 0 0 0 0 pancam_pre_sunset_L4567R2467
613 Total 0 0 0 0 0 0

The case may be a little bit different, if you are referring to those times in which no data was seen at Exploratorium/JPL sites.
Now we see three days in a row in which none of the planned sequences were executed; at least that is what the tracking website tells us.
Oppy is not silent, actually she is taking the opportunity to download data from previous sols, so the rover is not on "safe-mode".

Let's wait for the data from sol 613 to see if the situation changes.

can you provide a link to that 'tracking website' ive never seen it. thanks.

http://marswatch.astro.cornell.edu/merweb/merweb.pl

Jim said the tracking website was being a little funny a few days ago,

perhaps that is where the problem lies, not on Mars BUt it does look a bit ominous - 611 was a single navcam image to match a Mini TES observation - perhaps that's put a spanner in the works

The latest flight directors report doesn't mention any problems as of Sol 612

The tracking website states that no images have been received for Sol 613 yet

What EDRs do we have on the ground from sol 613?

Actual number of EDRs by sequence number and image type:

Sol Seq.Ver ETH ESF EDN EFF ERP Tot Description
--- -------- --- --- --- --- --- ---- -----------
613 p1420.01 0 0 0 0 0 0 hazcam_sunset_imaging
613 p1591.01 0 0 0 0 0 0 navcam_sunset_imaging
613 p2111.05 0 0 0 0 0 0 pancam_cal_targ_L234567Rall
613 p2396.06 0 0 0 0 0 0 pancam_snoopy_3cx1r_L2R2
613 p2539.13 0 0 0 0 0 0 pancam_foreground_L234567Rall
613 p2540.13 0 0 0 0 0 0 pancam_port_survey_L234567Rall
613 p2541.13 0 0 0 0 0 0 pancam_rear_survey_L234567Rall
613 p2600.07 0 0 0 0 0 0 pancam_tau
613 p2600.07 0 0 0 0 0 0 pancam_tau
613 p2694.05 0 0 0 0 0 0 pancam_dust_devil_hunter_waitfor180_L6
613 p2695.05 0 0 0 0 0 0 pancam_sunset_prepoint_L2R2
613 p2696.05 0 0 0 0 0 0 pancam_pre_sunset_L4567R2467
613 Total 0 0 0 0 0 0

As I read it using the Pancam website....looking back to Sol 608...

608: A driving day was commanded, and at least some of that drive occured (3 of 5 stumble checks)

609: Remote sensing commanded, and executed and downlinked in full
http://qt.exploratorium.edu/mars/opportuni...JLP2537R7M1.JPG

610: Driving sol commanded - no data recieved at all as yet

611: Commanded a single observation
p1950.23 nav_mtes_rvrAz0_ElN45_1bpp_pri57 but not recieved.

612: Driving sol commanded ( same as 610 ) - but again, nothing recieved.

613: Lots of remote sensing commanded but not recieved

614: Some remote sensing commanded

615: Driving day commanded

that is as of sol 613.9

Now - as I see it, they wouldnt have commanded driving and remote obs 2 sols in advance if they didnt think the rover was healthy - BUT - there clearly was a problem on 610. It might be a different link in the chain of course - just new imagery not filtering thru to the DB or the raw image pages, or even a problem with Odyssey perhaps - who knows.

Possible events - 610: Problem with the drive, 611: Rover checkout, MiniTES trouble? 612: Prognosis healthy - command driving, 613: Lots of remote obs with healthy rover 614: Housekeeping to cure 610 problem a little better, and some remote obs, and 615 on the road again.



Doug

I have a slightly different reading on those tracking website reports.
Doug, you are talking about data not received but I think they were actually not executed.

See, for example, sub-section 12 on the same report for sols 610-613:
-----
12. How does what we requested compare with what the rover executed? (be sure PUL's
know, and if important, the PEL. update the SSF list
/home/mersci/pan/B/ops/Sol_all_seq_list.txt as appropriate)

Data products:
Number Number
Number Number Still not (yet)
Sol Seq.Ver Requested Created on Rover Created Description
--- --------- --------- ------- --------- ----------- -----------
610 p0775.03 20 0 0 20 navcam_5x1_az_306_3_bpp
610 p1201.03 4 0 0 4 front_haz_penultimate_1_bpp_crit16
610 p1214.05 4 0 0 4 front_haz_ultimate_4bpp_pri15
610 p1235.04 4 0 0 4 front_hazcam_stumble_0.5_bpp_pri_56
610 p1235.04 4 0 0 4 front_hazcam_stumble_0.5_bpp_pri_56
610 p1235.04 4 0 0 4 front_hazcam_stumble_0.5_bpp_pri_56
610 p1235.04 4 0 0 4 front_hazcam_stumble_0.5_bpp_pri_56
610 p1311.06 4 0 0 4 rear_haz_ultimate_1_bpp_crit18
610 p1335.03 4 0 0 4 rear_hazcam_stumble_0.5_bpp_pri_56
610 p1335.03 4 0 0 4 rear_hazcam_stumble_0.5_bpp_pri_56
610 p1335.03 4 0 0 4 rear_hazcam_stumble_0.5_bpp_pri_56
610 p1335.03 4 0 0 4 rear_hazcam_stumble_0.5_bpp_pri_56
610 p1675.01 20 0 0 20 navcam_5x1_az_126_1_bpp
610 p2627.01 36 0 0 36 pancam_sky_radiance_thumbs_L457R247
611 p1950.23 4 0 0 4 nav_mtes_rvrAz0_ElN45_1bpp_pri57
612 p0773.03 12 0 0 12 navcam_3x1_az_306_3_bpp
612 p1214.05 4 0 0 4 front_haz_ultimate_4bpp_pri15
612 p1215.02 4 0 0 4 front_hazcam_ultimate_0.5_bpp_pri_15
612 p1235.04 4 0 0 4 front_hazcam_stumble_0.5_bpp_pri_56
612 p1235.04 4 0 0 4 front_hazcam_stumble_0.5_bpp_pri_56
612 p1235.04 4 0 0 4 front_hazcam_stumble_0.5_bpp_pri_56
612 p1235.04 4 0 0 4 front_hazcam_stumble_0.5_bpp_pri_56
612 p1305.03 4 0 0 4 rear_haz_penultimate_0.5bpp_pri18
612 p1335.03 4 0 0 4 rear_hazcam_stumble_0.5_bpp_pri_56
612 p1335.03 4 0 0 4 rear_hazcam_stumble_0.5_bpp_pri_56
612 p1335.03 4 0 0 4 rear_hazcam_stumble_0.5_bpp_pri_56
612 p1335.03 4 0 0 4 rear_hazcam_stumble_0.5_bpp_pri_56
612 p1677.03 28 0 0 28 navcam_7x1_az_126_3_bpp
613 p1420.01 4 0 0 4 hazcam_sunset_imaging
613 p1591.01 4 0 0 4 navcam_sunset_imaging
613 p2111.05 28 0 0 28 pancam_cal_targ_L234567Rall
613 p2396.06 14 0 0 14 pancam_snoopy_3cx1r_L2R2
613 p2539.13 28 0 0 28 pancam_foreground_L234567Rall
613 p2540.13 28 0 0 28 pancam_port_survey_L234567Rall
613 p2541.13 28 0 0 28 pancam_rear_survey_L234567Rall
613 p2600.07 6 0 0 6 pancam_tau
613 p2600.07 6 0 0 6 pancam_tau
613 p2694.05 11 0 0 11 pancam_dust_devil_hunter_waitfor180_L6
613 p2696.05 66 0 0 66 pancam_pre_sunset_L4567R2467
614 p2619.07 117 0 0 117 pancam_skysurvhi_L2345678R345678
614 p2627.01 36 0 0 36 pancam_sky_radiance_thumbs_L457R247
Total 584 0 0 584
-----

Nothing has been received until now because there is nothing to downlink. It is fine for sol 614 but being currently on sol 613.9 and without any data reported to be downloaded for 613 is worrysome.

Another possibility (a good one) would be that those reports are out of synch.

Let's see if any insider tells us something new.

Does that include things like engineering data? If there was a problem, I would have thought they would just want engineerig daata rather than science data?

" you are talking about data not received but I think they were actually not executed."

We dont know that. All we know is what was requested, and what was recieved. We dont know that it didnt happen - I'm assuming they didnt execute, or only partially executed, obviously - but we dont actually KNOW that.

Doug

Just looked at the Pancam tracking site... and no Images have been received from Spirit so far this sol.. although I suppose there is still time for them to come down.

6. What unexpected sequences ran? (that is, sequences we did not enter in the SSF
list file /home/mersci/pan/A/ops/Sol_all_seq_list.txt)

Sol Seq.Ver
--- ---------
634 f0006.00
634 f0006.00
634 f0006.00
634 f0006.00
634 f0006.00

Good News I think:

2. What EDRs do we have on the ground from sol 614?

Actual number of EDRs by sequence number and image type:

Sol Seq.Ver ETH ESF EDN EFF ERP Tot Description
--- -------- --- --- --- --- --- ---- -----------
614 p2111.05 13 3 0 0 1 17 pancam_cal_targ_L234567Rall
614 p2111.05 0 0 0 0 0 0 pancam_cal_targ_L234567Rall
614 p2542.13 13 0 0 0 0 13 pancam_santa_anna_L234567Rall
614 p2543.13 4 0 0 0 0 4 pancam_chinook_L234567Rall
614 p2544.13 0 2 0 0 0 2 pancam_scirocco_L234567Rall
614 p2545.13 0 1 0 0 0 1 pancam_mistral_L234567Rall
614 p2546.13 0 0 0 0 0 0 pancam_ostria_L234567Rall
614 p2600.07 2 2 0 0 2 6 pancam_tau
614 p2600.07 0 0 0 0 0 0 pancam_tau
614 p2619.07 0 0 0 0 0 0 pancam_skysurvhi_L2345678R345678
614 p2627.01 0 0 0 0 0 0 pancam_sky_radiance_thumbs_L457R247
614 Total 32 8 0 0 3 43

EDIT: yayyyyyyy some new pictures at exploratorium

Sol 609:


Sol 614:


Opportunity hasn't moved. So what happened to all those drive commands the last few days?

How come this one from the same download is different, is it an old one? I dont know how to interpret the codes

Obviouosly something went wrong the last few sols...... but at least we know the rover is transmitting data again. Today is another planned drive day.

Also I dont think there were any images transmitted from Spirit on sol 634 - they've only just arrived.

Because Oppy, like most of us, has more than one eye.

knew it had to be something simple

From Steve Squyeres latest update

Ok, those are the bad news.
The good ones are that Oppy has definitely moved on sol 615.

Edited: Quite a few images from the new site are available at Exploratorium but the tracking web reports all drive related navcam and pancam images have been already downlinked.

Edited#2: The move was about 20m Westward.

<glass half full>

Nothing terminal with Oppy.

</glass half full>

Considering the reboots are almost certainly to blame of software bugs, they have nothing to do with age. Aside from a lame wheel-motor and a spent hunk of cesium, Oppy is still spry and exhuberant.

I'm surpized by the outages at DSN; seems like several failures in a row. Maybe they need some refurbishment? That could be a huge deal for a mission like Cassini or Deep Impact, where data buffering on the probe is very limited and the targets are often 1-shot passes.

I guess I should dig up my digerdoo and light a nice fire, like the shaman from All The Right Stuff. Hey, it worked for Spirit's first reboot!

The Spirit is not experiencing any software bug like Opportunity. I am thinking that the software logaritm of Opportunity must be different than Spirit. Maybe, it has an additional logic for sleepage computing and any avoidance manouvers related to that.

Rodolfo

Don't think so - both are running the same version of the flight software. The environments and use cases are different though and there are operational parameters that differe between the two so situations arise on Oppy that don't arise on Spirit (and vice versa).

What is very different between the two is the day to day sequencing. It's very easy to see how a bug might be triggerred by a set of procedures on Oppy that never occurs on Spirit.

...then again the 'software bug' might be precipitated by environmental factors such as voltage irregularities perhaps caused by temperature variations or static charges on weathered components, and these may originate in wires or smaller componenets that arent monitored by the onboard software and so might have nothing at all to do with any software flaw (other than the ability of the software to adapt to such conditions without rebooting, which might not be desireable 'capability' anyway).

Can they monitor all aspects of all parts between critical components? IOt sounds like a lot of extra complexity and weight to include. Theyve got the basics covered but is there a signifficant unknown factor about many smaller componenets? Unless they have ways of determining such things with all parts of the rover, we may be seeing propogated effects of physical breakdown of as-yet unknown components the rover.

The software bug is still unclear up to Sqyures who wrote in his Athena's publication that nobody has clear understand of this case. This is still under an investigation.

Fair enough; that's the way things are supposed to work when the going gets dangerous in this kind of terrain. No sooner did we back out of that, though, than we got hit by another of the mystery reboots that Opportunity has encountered a few times in the past several months. This was different from what hit us on Sol 596... that one we understand and can trace to a simple bug in the software. But this one we don't understand yet, and it cost us a couple more sols..

It is probably not caused by software flawless but by an environment factor....

An fancy idea is that close Erebus might have a strong magnetic field caused by an strange meteoro that might be causing troubles to the electronic devices...

Rodolfo

There's not enough hard data to be sure but the occassional hint indicates that they have enough engineering data to identify the location of shorts, stuck switches, under performing heaters and the like. They certainly have shown the ability to profile current changes that happen when steering actuators get stuck, wheels slip, heaters remain stuck and thereby identify the precise cause of those types of problems. The daily reports show that they were able to identify the "cleaning" events based on a 5% change in power output vs predicted output. I think it's reasonable to assume that they can tell immediately (well by the next downlink) if any of the powered devices are working incorrectly. Overall I'd say that they have a fair idea about the wear and tear but that doesn't mean that there aren't some wild cards - the intermittent problems with the Mini-TES show that they don't have absolute knowledge of the health of every single component.

This is pretty telling though - :

That makes me think that he believes they have enough data now to root cause this too.

Aren't the hazard avoidance routines in Oppy modified for driving across the rippled environment? That could be a critical difference.

--Bill

I'm pretty sure the software on the two rovers is -identical-. However, the various settings are very different for each.

It is very unlikely that a bug in the software is causing the reboots. These reboots are coming late in the life of the rover and are not happening on Spirit. It is far more likely that an aging component is to blame. Probably in the computer some place. Something that the computer can't recover from like an error in a cpu register. There are a thousand components that if flaky would send Opportunity into a brain freeze. This would result in the watchdog circuit rebooting the rover.

We can't rule out a software bug that only is rearing it's ugly head because of something new they are doing. But they've been using these things too long for that to be likely.

Not to be a downer, but these reboots could start becoming more frequent to the point of - dare I say it - End of Mission.

ed

I'll say that it's too early to say for sure whether it's a flaw in the hardware or the software. Opportunity and Spirit, while performing the same basic tasks, are not performing exactly the same tasks. Perhaps some unnoticed, barely important flag is set on Opportunity that is not set on Spirit, or vice versa.

I do think it's unlikely it's a hardware problem - any sort of hardware problem would likely lead to Opportunity being utterly unusable. Unless you get really lucky, a bad gate on a chip will probably make the entire chip useless. At best, perhaps one byte of Opportunity's memory is bad, and it is only being written/read very rarely, but I would think that the engineers at NASA/JPL would be able to easily detect this - didn't they already shut out a chunk of the flash RAM on one of the rovers?

If you forced me to take a bet, I'd wager on a bug in the software.. Hardware can recover from bad software. Software can rarely recover from bad hardware - especially a bad CPU.

Well if it comes down to it, I'm sure the operators could do a "format and reinstall" of the Rover's memory. I'd hope that they've got lots of possible diagnostics routines to do - and heck, look at how bad Spirit was back in the early days, and they got it going just fine.

Doug,

This thread is getting quite OT.
May I suggest a new one and move all this discussion about Oppy sw/hw problems there?

Hi all,

I suggest to continue all discussions about current and/or potential sw/hw problems here.

Amen. Unfortunately, not everyone up the food chain from the Poor Bloody Programmer understands this

Sometimes it is cheaper/easier to fix hardware problems with software, but often it's just exchanging one set of problems for another.

This problem has started recently and is recuring. They are not doing anything that is significantly different than before. The software has not been updated for a long time. There is a chance that it is a subtle bug that they never stepped on before. But that is unlikely. It is more likely that something has changed. If it were a bug, the software would have a good chance of catching it and recording it - even if it could not recover from it.

"Software can rarely recover from bad hardware"

H/w is often intermittent. And a h/w reset will often get you back in action. We all seen crazy PCs that have intermittant problems and resetting them gets us going until the next failure. It keeps happening until that memory stick or offending PCI card is replaced.

The rovers have a hard wired reset that will automatically cycle the computer when the software stops pinging this watchdog circuit. This is common pratice in a system that is unattended.

Here is what we know.
1) It is a recent development.
2) It is reoccuring.
3) The software is not recording the problem.
4) They have not been able to corrolate it to any action being performed.

Computers run on zeros and ones but the circuitry is analog. When a chip becomes marginal, it can randomly change the state of something that it shouldn't. This kind of behavior is common in computers that have been subjected to radiation or thermal cycling. Radiation degrades electronics. They often don't fail straight out when subjected to radiation. Thermal cycling damages solder joints. These can open and close rapidly. It is a common mode of failure in a system the is thermally cycled.

My prediction is that the resets will continue and become more frequent. And oh how I hope I am wrong!

In my experience, an electronic device with any sort of damage will fail regularly and fail spectacularly. However, I agree that there's no way to say for sure - especially since we don't have much information as far as the exact nature of the failures - and what you say about an intermittently flawed device makes sense, particularly given the harsh environs of space.

I imagine that a fatal bug in the rover software will result in a core dump of the operating program, and if there were to be no core dumps, that to me would be a sign of a (bad) hardware problem. If there were to be core dumps, but utterly useless core dumps (no/nonsensical stack trace), that would be a sign of a possible hardware problem, though not necessarily..

Sadly, however, I can not say for certain it's not a hardware problem. I hope it isn't.

From the latest Mission Status at the NASA/JPL site:

http://marsrovers.jpl.nasa.gov/mission/sta...tml#opportunity


And a battery "glitch":



This explains why Oppy's been erratic for the past few Sols...

--Bill

Well.....there doesn't appear to be any data from Sol 629 yet. Maybe another software glitch - or that dust storm !!

re: glitches

wrt hardware problems: it's not rocket science... err wait yes it is
wrt software problems: it's not brain surgery.. umm hang on..

ah ha ha new aphorism regarding problems with the rovers "it's not rocket surgery"

Based on currenlty planned seqs for sols 628-630 (see below) and taking into account that some pancams previously planned for sol 628 just disappeared, I would say that Oppy had another hiccup.

---

628 p0767.03 14 0 0 14 0 28 navcam_7x1_az_288_3_bpp
628 p1663.01 6 0 0 6 0 12 navcam_3x1_az_108_1_bpp
630 p0767.03 14 0 0 14 0 28 navcam_7x1_az_288_3_bpp
630 p1663.01 6 0 0 6 0 12 navcam_3x1_az_108_1_bpp

----

.. or it may be an energy (read dust storm) issue.

I wouldn't call something that travels at 5 mm per second a rocket...

Hey, it is a software problem and not a hardware problem.. I now have further evidence for my 'any real hardware problem causes an utter failure', regardless of said hardware being in the presence of several inter-galactic cosmic particles. Of course, if said inter-galactic cosmic particle(s) somehow hit the gate(s) in the EPROM that control(s) the delay before writing to memory (and didn't break anything else), then I stand corrected.

I dont know, but even the EEPROM might start to lose data unexpectedly this late in mission. The rovers are what, 10 years behind technology we have today? EEPROM onboard might not have the million writes that is common nowdays.