Uh, oh, now the filter wheel is acting up
http://qt.exploratorium.edu/mars/opportuni...KCP2425L7M1.JPG

Well, it's like the 'other Emily L' (Litella) used to say, "it's always something..."

Looks like a wonky L247 sequence, with a real long interval (404 seconds) between the L4 and L7.

Evidently the filter wheel didn't index properly; I'd suspect that they use an encoder to know which filter is where.

We'll check Weekend Update...

--Bill

Opportunity looks to be having some trouble with her left eye filter wheel.

Opportunity woke up and started imaging very early on this particular sol, at around 8:18 in the morning. This is the earliest Opportunity has woken up and taken images since way back on sol 455. From the looks of the images, Oppy was none too happy about getting up this early (maybe she hadn't had her coffee yet).

The first left eye image of the morning, an L8 sun finder, came back as 1 pixel. I'm not sure what this indicates. This could be that the filters weren't set right and the auto finder failed or it could be something else entirely.

The next 3 left eye images appear to not have their filters perfectly centered and the offset from center appears to shift between images as if the filter wheel wasn't turning as much as intended.

The following 3 are very badly offset, and that offset drifts between frames as if the frame wheel still wasn't turning as much as necessary. As Bill noted, there is an unusually long delay between these images. Normally, there is only 20 seconds between an L2 and an L4, and as much as 35 seconds between an L4 and L7. This second set of 3 has almost 2 minutes between the L2 and L4, and 6 and a half minutes between L4 and L7.

Luckily, after that, it appears to have fixed itself. There were more than a dozen+ left eye images later in the sol that look completely normal, and several are already down for the next sol which look fine as well.

L8, 08:18:07 AM

L2, 08:19:30 AM
L4, 08:20:01 AM
L7, 08:20:32 AM

L2, 08:22:21 AM
L4, 08:24:10 AM
L7, 08:30:41 AM

I saw those when I updated today. It was scary to see the offset filter wheel. I don't like to see multiple issues taking place on the same appendage. Could the boosted motor voltage have caused some other problem?

I guess old age is starting to really set in with the poor critter.

It sounds like a temperature dependent problem, maybe. Arthritis, at any rate.
Note that quite decent 2-color images can be taken with the right camera 1 and 2 filters, comparable to 7 and 2 on the left camera.

Just to see, I composited the mis-filtered images as color images.

Most complex systems display failures in a probabalistic curve called the bathtub curve. Initially upon fielding of a system, there are a number of "infant mortality" failures. Testing in the lab actually catches most of them for a system like the rover, but we did see at least two on Mars (Oppy's joint heater and Spirit's wheel). These failures are usually caused by design or manufacturing faults. They form the left side of the bathtub.

Then there is usually a period of adulthood, where there are few failures. The steering failure for Oppy falls into this period though -- which may suggest that it is either a fluke caused by perhaps some debris getting into the mechanical system, or perhaps a latent manufacturing fault. This period is the floor of the bathtub.

As end of life approaches, failures increase, forming the right side of the bathtub. While we have too little data to make predictions at this point, one hard failure followed closely be an unrelated significant anomaly (I don't want to categorize this as a failure yet) is a cause for concern about where we are on the bathtub curve.

Hard to say though. We could have no more failures for the next year or more.

Scott

Spirit's wheel (the Sticky FR wheel) is actually back to nominal, and has been for almost a year. A few months of backwards driving sorted the situation totally.

Doug

Somehow, I didn't ever hear the final resolution of that. Thanks.

The bathtub curve really only applies to mechanical systems that can wear out, not to electronic components that tend to have rather constant failure rates. Of course there are lots of mechanical systems on the MER, and up to now they are the ones that have been showing signs of wear and incipient failure. Remember though that some vital electronic component can also fail at any time.

tty

I wonder what would happen to Pancam if this problem would occur while taking the Sun tracking pictures. If it only would overexpose the image that would be fine, but I'm afraid it might burn out the camera in a worst case scenario.

general
Oppy looks like a Swiss pirate with that eyepatch! cute!
"I'm on MARRRRRRRS - MeridiARRRRRRni PlARRRRRRRnum!"

Are the navcams blocked in any way when they are taking the sun tracking pictures with pancam?

Note there was one early-mission failure, maybe even during cruise stage: The temperature sensor on both rover's MiniTES calibration target failed due to a bonding failure. It was recognized as an expected problem before launch but too minor to delay the launches to fix, if I recall.

Actually, in my experience over the past 20 years dealing with aging systems that are still in use but were built in the late 1960's through late 1980's, I have seen that even for electronics there is a right side wall to the bathtub curve.

I think that for most electronics systems the end-of-life increase in failures is not seen because the systems are usually discarded upon first failure.

The failure curve starts with initial infant mortality, followed by constant fairly low failure rates, and then a significant increase in failures (that at first appears random), with finally a rather catastrophic increase in failures caused by specific issues that affect large numbers of parts. These final failures are traced to things like internal bonding material failures, bimetallic formation leading to bond-wire and other connection failures, migrations of materials (flow), crystal growth, the formation and movement of conductive particulates, seal failures leading to leakage of electrolytes, epoxy degradation with age, shrinkage and hardening of soft goods (in connectors and shock mounts), etc. Most of these processes are greatly exacerbated by environmental extremes and low pressures.

So really what we see are materials and mechanical failures within the electronics components. I have seen the oddest things happen.

I have also become much more concerned about corrosion on Mars lately.

Scott

As my good friend Yogi Berra once said, "It ain't over til it's over."

Nope it isnt over yet, theres more working cameras on the rover. We can still get to see more interesting things even though the perspective would be rather distorted in some cases.

You've had to have a failure of 4 cameras to kill of the driveability of a rover, and even then you've had to kill of FHAZ as well to make it really really impossible.

Given that there's a lot of PC imaging scheduled, I think we can put this down to a brief glitch that they understand or perhaps was the result of a brain-fart in a pancam sequence. Funny images, and they'll make a nice entry into a Mars talk anyway


Doug

Could this happen (acquisition of the image while the filterwheel is not positioned properly) simply because they sent up a faulty command sequence, or is there really something wrong ? I can imagine that there's a built in safety in the camera that makes sure the camera only snaps while wheel is in right place. Is there ? If yes....it obviously failed now !

Dunno. Most of also thought at first a MI sequence wouldn't run if the IDD deployment sequence would fail. A lot of people thought it was just a sequencing error at the day the first "MI in stowed position" images showed up, and that turned out to be a wrong assumption. I'm not placing any bets on this subject

I thought that too, about the MI sequences. I started to realize later that those were the first in a long sequence of images designed to diagnose the arm failure. I think they were looking for small movements of the arm.

They already knew the arm was stuck by the time we started wondering.

I'm encouraged by the fact that the filter wheel seems to be operating normally, now. But I think I am still in DEChengst's camp, or maybe it's SRAnderson's bathtub. I'm not ready to place any bets yet.

This may have been a spurious event, but I'm pretty sure they didn't plan to do this.

That does not fit well with my experience. We had quite a lot of 50’s and 60’s vintage avionics flying until quite recently (some still do) and while we have had a few issues about particular brands of insulation materials deteriorating, there have been no massive reliability problems.

In my experience aging electronics go out of service because:
a spares are no longer obtainable
b operating and maintaining them has become a lost art
c integrating digital and analog systems can be very difficult

Perhaps living up here in the cool and fairly dry north has something to do with it. One thing we have learned in recent years is that dehumidifiers, even if only used intermittently, dramatically improves the reliability of electronics (and almost everything else).

In this respect at least Mars should be a very favorable environment though that still leaves extreme temperature cycles, dust, low pressure and perhaps corrosive chemicals.

tty

If they are using a full encoder on the filter wheel, and the software counts pulses to know when a filter is in place, they could have fat-fingered the number of pulses to count in the rotation sequence. But I would imagine that there are some "fixed" subroutines to handle pulse counting for moves from filter to filter that would make it a no-brainer.

My ST-237 CCD camera uses a simpler system on its filter wheel. Each filter has a single dot beside it that a simple photo detector sees and the software keys off of when responding to commands. Sometimes it does miss and I get a few extra spins before it finds itself again. I haven't seen them get stuck in the middle, though.

Encoders are a robust technology, we've got thousands of robots out there that have been running for many years with few encoder failures - admittedly they're not running in such a harsh environment.

I suspect a sequencing error too...

Hmm....your post sure brought back some memories of meetings where there were heated discussions of whether the problems we were seeing were indicative of ageing or were caused by handling/environments.

Your "b" above is particularly true when you combine it with the fact that young engineers out of college don't want to do patch-up redesign on 40-year-old systems. What finally forced retirement of the system I'm talking about was a microscopic corrosion issue (which was caused by humidity outgassing from materials inside sealed boxes), along with your "b."

As far as the rovers go (after looking at some old briefing charts that I dug up), I think you have convinced me that it is unlikely that we will see any increase in electronics failures during their useful lifetime -- at least for those electronics that are protected in the WEB.

But a random failure could still happen at any time.

Scott

One might suspect that, but we haven't seen any signs of corrosion on any of the imageable parts of the rover so far, have we?

Dragging up an old thread, but I've spoken to Jim Bell about this and the root cause was that the filter wheel wasnt heated enough. It was an early morning sequence, and the thermal system engineers did not think that the wheel would be as cold as it was, so they didnt use the heater. It turned out that it was JUST a little bit too cold and so there was some 'stiction'. The right filter wheel doesnt stick until a tiny bit colder than the left wheel, and just by chance, the temperature at the time was between the two.

No harm done, and the wheel has been fine ever since, they've just adjusted the thermal models and reshot the sequence later.

Doug

Thanks for following up on that issue Doug; good to know it was just a procedural issue after all.

Airbag