Spacecraft Set to Reach Milestone, Reports Technical Glitches
NASA/JPL
February 7, 2007

It's a sad thing, but I can promise you that right now science has not been affected, and the team is doing everything possible to maximize the lifetime of the camera (Referring only to HiRISE).

As a chip guy, I'll be interested to hear whether it's the CCDs or the FPGA look-up tables, or perhaps some memory. The chips that we make these days are not only extremely susceptible to soft errors, but have some pretty wacky methods of working differently as time goes by. You can differentiate mean-time-to-failure in large systems at sea level versus Denver...

From what little I've heard, the problem is almost certainly analog in nature, if not in the CCD output amplifiers then somewhere in the signal chain before digitization.

The problem is analog in nature somewhere, if it was digital, well, it'd have affected all of the CCDs. As to exactly what the problem is, well, it's unknown...

Of course, the MCS could be a digital problem, from what I've heard about it...

I work on boards, and we design for error detection, handling, and recovery at the system level with chip soft errors in mind. In my experience, soft errors from secondary cosmic particle upsets do not increase noticeably over time. Chip degradation over time such as electromigration tend to show up as timing problems as evidenced by increasing sensitivity to supply voltage and temperature variations. Two or more different failure mechanisms.

I don't mean to trivialize the attention that they've already put into it. It's just as it's my job, I tend to blame my own stuff first. But yes, I'm not a device physicist, so I am leery of all effects. It just seems that the magnitude of EM, but also the hot carrier and nbti affects on bleeding edge processes wasn't fully realized until after those chips were shipped. What happens to those same chips when you put them in space probably won't be fully realized until they spend 10e5 hours in space...

And yes, none of this has to do with MRO, likely. It's just me forecasting doom.

It looks like the great galactic ghoul is active in Mars orbit, first MGS ... now MRO

Well, the problem doesn't impact the quality of the images very much. I think that the main concern is degradation ( according to the article ) ...

Mars Reconnaissance Orbiter Glitch Disrupts Data Flow to Earth
By Leonard David
Senior Space Writer, Space.com
posted: 08 February 2007
04:08 pm ET

QUOTE

While HiRISE is acting up, Zurek observed that the instrument’s three months of science gathering to date has yielded roughly 1,000 images—more than 1.5 terabits (1,500 gigabits) of image data.

I haven't done a precise count, but are there "1,000 images" on the HiROC site? If not, then that was what I was alluding to here.

The images are really good, but in terms of frequency of releases, so far it seems that the "People's Camera" public releases have, on average, been occurring at about the same rate as the MOC daily captioned image releases.

How many more images of the northern plains did you want?

<250 by my count

Enough to map every Phoenix landing site target just about in full and any other interesting polar targets before it heads into the dark for the Northern winter.

Doug

Mars Orbiter Imaging Halved
By Craig Covault/AviationWeek.com
February 8, 2007

I'll just make a couple of comments about this:

1. The releases that we are doing now are geared towards a NASA requirement that we have to release a small number of captioned images every week. Also keep in mind that these images aren't our standard science products, these are public releases that just happen to have some degree of scientific utility. Note that we have also released a large batch of uncaptioned images as well....
2. Our long term goal is to try and release our standardized science products as quickly as possible, but we are not there yet, and it's difficult to say at this time just exactly what our real turn around time is going to be between acquiring an image and releasing a standardized science product. There are a number of pieces that need to come together before we can make this happen, such as finalizing the format of our standard data products, jpeg2000 issues, and our interface to the Planetary Data System. Once these issues have been addressed, there's a pretty good chance that the release rate will go up, and the quality of the data released will go up as well.

Agree that some leading edge commercial chips get released only to have customers find problems with them over time. It's the competitive nature of the business. If you extend the characterization and qualification to iron out more latent design issues, the competitors can eat your lunch. It's a judgement call on when to release. Hopefully, some workaround can be found for the problems that show up later.

Chips for space apps undergo much more rigorous environmental qualification tests, and are some years / generations behind the leading edge. I don't work on space HW, but I doubt there will be any bleeding edge _commercial_ processes used in space systems. E.g., you mentioned FPGA LUT's. The predominant commerical ones are based on SRAM configuration. I recall hearing disussions on how space HW would use lower density, lower performance devices based on permanent anti-fuse configuration rather than having to design the system to detect and recover from configuration SRAM corruption and to do it while staying at least partially online. And the MTTF in space is a lot higher than in Denver.

The actual cause of MRO's problems will be interesting to read about when it becomes publicly known. As for doom, when I saw Craig Covault's headline Mars Orbiting Imaging Halved, I read it as half full rather than half empty.

Another thing that is going on is a move to a better web publishing system; we've got a new webmaster (I filled in temporarily in the latter part of '06) preparing that currently. I expect the dams will burst so to speak by the spring....

I'm having a hard time figuring out what the current status of HiRISE is from the myriad of articles currently being written. The NASA/JPL news release refers to "a significant increase in noise, such as bad pixels, in one of its 14 camera detector pairs." while the Nature story refers to "Seven of HiRISE's 14 detectors are sending back spurious data, the mission team reports, and one of the four colour detectors has stopped working completely."

Which color detector is lost? It would seem to be less damaging, from a public release standpoint, if it turned out to be one of the two infrared detectors, since the two blue/green detectors are what we have seen used for color images. Also, is the nature story exaggerating by referring to the color detector as completely stopped or is the JPL release euphemistically telling us that same thing?

A little disambiguation might go a long way here.

I have no idea what the actual situation with HiRISE is, but it would be nice to get an official and unambiguous statement. The media reports do make it sound pretty bad.

From the Nature story: "A previous high resolution NASA camera was lost with the demise of MGS. The best camera currently in orbit around the red planet is on Mars Express, which arrived in December 2003."

Given that the SRC system on MEx hasn't performed to spec, I think that CTX on MRO at 6 meters/pixel wins the prize for next best camera, at least in terms of resolution.

This is more than a bit disconcerting; now I'm not even sure if we have an evolving situation or a fixed set of known problems. The MRO program office needs to issue some clarification, ASAP.

That is a large exaggeration. Channel 1 of IR10 almost has completely stopped working, but channel 0 is working just fine. Looking at the articles that have been posted, I would point you to the space.com article and state that it seems to be the most informative and accurate, except for it's title. Quoting two parts of it which are totally true:





These two statements are 100% correct. Just thought I'd point that out.

Well I hope they get a picture of the possible Beagle 2 and MPL crash sites before the camera dies. They should probably move them up the target list.

Whoa... there are some black thoughts here, Sunspot.

Why move them up the list? Even if HiRISE were known to be about to expire ( and we know that actually, the media is overhyping this enormously and in actual fact, HiRISE is not about to expire ) - why reprioritise targets? The most important targets are the most important targets if HiRISE is fine, unwell or broken.

Personally - if (and we know this is not the case ) there were 'X' shots left in the HiRISE lifespan - I would give one shot each to B2 and MPL (the supposed B2 target and the previous MPL target ) and then move on to geologically interesting targets instead of duck-shoots for targets of unknown locations.

The point is moot anyway - MPL couldn't be imaged until the southern Summer and we're not there yet - hence the large numbers of northern polar imaging targets

Doug

Some questions to better understand the situation:

tuvas said channel 1 of IR10 (Why 10, there are only two IR detectors?) is almost dead, channel 0 working fine. Are these channels redundant and we are using the backup now? Or are they complementary and something is lost already?

How does the warming happen? Are there heaters or is this done just by operating the detectors?

Analyst

I believe that they number the CCDs from 1 to 14, with a prefix that indicates which filter it has. So there are two IR sensors, called IR10 and IR11. See http://hirise.seti.org/epo/hirise_lesson1.htm and http://marsoweb.nas.nasa.gov/HiRISE/papers...ISE_InstDev.pdf

I surmise that each CCD has two output channels that are separately processed. It's pretty typical for linear CCDs to have such arrangements: the MOC NA, for example, had two channels for its single CCD (one for even pixels, one for odd). These are not redundant; if you lost one you'd have to interpolate every other pixel.

From what I've read now on this forum from UofA, the media accounts are fearmongering to some degree.

http://isis.astrogeology.usgs.gov/Applicat...issance_Orbiter might give some insight into the processing required for HiRISE images.

Disclaimer: this information is all from public sources.

mcaplinger, IIRC you are involved with the MRO CTX camera. Could you provide us with a small comfort by verifying that CTX is working correctly. Not seeing any images on MSSS since October makes me wonder how it's doing.

These points are almost correct. The CCDs actually are numbered 0 to 13, RED0-9, IR10,11, and BG12,13. Each of them has two channels which are individually processed, as you stated. They aren't even/odd pixels, but rather one half of the channel followed by the other. With the exception of the focal plane stuff, there is redundancies on HiRISE, exactly what I couldn't tell you.

I can tell you that HiRISE is working quite well at the moment, http://hiroc.lpl.arizona.edu/images/PSP/di...PSP_002281_2115 is a picture that was only taken 21-Jan-2007, well after the problems had first appeared. In fact, to even detect any problems at all with 3 of the CCDs, we had to chill the camera to lower-than-normal temperatures, two others barely show any signs at all. HiRISE still has a lot more time, the degradation on the newest CCDs to show is very minimal, I beleive the level hasn't changed reasonably since first noticed.

As to what would happen if HiRISE was found to only have x number of pictures, I would bet that as many as can be taken of MSL potential landing sites would be taken, as much as HiRISE would like to be used as a geographical instrument, where it really comes in handy is for photographic potential landing sites to search for a safe place to land.

Can you confirm the Nature report that one of the color detectors has failed? I'm not reading anything near as severe as that elsewhere, so I suspect that this was an error in this report.

Also, when you say things are redundant, are the two channels in the detectors redudancies? I.e., losing one still allows you to read the data from the detector array through the other channel?

Tuvas, thanks for the clarification. It's good to know the detector isn't completely lost, and I did breathe a slight sigh of relief to hear that it is the IR-color channel that is having the most problems. Not to belittle its contribution, but if I'm not mistaken, we've never seen a public release that even shows data from the 2 IR ccds. If there were one to lose, I suppose that would be it.

As a remote sensing scientist (although of things that have chlorophyll), I really value IR channels. There may well be much more scientific value in the IR channels than in any of the other "colors".

I think all the 'colour' strips of HiRISE we've seen will use all three - the BG, R and nIR

Doug

I'm could certainly be wrong about this, heck I probably am, but my impression from looking at the Spirit and Opportunity images was that the color stripes we were seeing are a composite of the R and B/G (visible as 2 distinct bands from the slight misalignment errors).

You could assume from the published reports that the instruments not mentioned were performing normally. Or did you think that CRISM had failed since they hadn't released anything either?

http://www.msss.com/msss_images/2007/02/12/index.html -- The top middle image is a CTX image from 23 November 2006.

One of the 4 IR channels has failed, IR10_1, or the leftmost channel. The channels are not redundant, rather they each handle half of the CCD, channel 1 handling the leftmost part and channel 0 the rightmost part. The samples from that channel at it's worst look like a black field with random bright pixels, no usable data anywhere.

As to the first picture of Victoria Crater, well, taken from it's release page: "Enhanced-color image generated from images acquired by the HiRISE camera in the red and blue-green filters." You are correct, the IR channel was omitted, as I recall there was a nice gap in the IR channel right over the rover... At least, something like that, anyways, it wasn't usable, that much I know. Most of the 3 channel release images (There are several) either were one that the IR channel behaved (In the beginning of PSP it was working fine), or the 3 channels.

New Scientist reports that HiRISE is currently "stable". (The article also mentions the fate of MGS, but no real news.)
(This is reported to have come out of LPSC. Would this have been at NASA Night?)

Does anyone know if HiRSE is going to continue taking only half the number of images per week that it did at the start of PSP, to keep the CCDs and associated electronics stable? Tuvas? Anybody?

The short answer is, nope. That's at least not the plan. The half-picture phase is basically a stop-gap till we figure out what's really going on, it buys us a bit more time.

Given the constantly increasing MRO data transmission rate, I hope the spacecraft operations are flexible enough to increase data returns from CRISM, CXT, etc. to use up extra spacecraft memory freed up by decreasing HiRISE images taken. You know, given enough mission extensions, CXT could map all of Mars at 6 meter resolution. This could be an opportunistic time to increase CXT coverage (you know, if you're given lemons, make lemonade).

Actually, as of right now, HiRISE is using fairly near it's data transmission rate, we are just taking bigger pictures. And I would imagine that a solution will be found sometime in the next few months, these things just take some time. Secondly, I don't doubt that while CTX/CRISM/etc. enjoy our pictures and such, they wouldn't complain for a bit more bandwidth (Rarely is that the case...)

Thanks, tuvas. It's always nice to know what's going on with HiRISE.

Speaking of which, Tuvas, do you have the scoop on MRO's recent safe mode event? Noticed that Spirit had a comm loss because of same...

http://www.azstarnet.com/allheadlines/174972

All I can really say is that MRO has left safe mode. I don't know why Spirit would be affected, AFAIK, there isn't any relaying done by MRO (Well, once in a great while for testing purposes, but...)

This is the explanation from Spirit updates.

Spirit is healthy but had to sit out a Martian day waiting to send data to Earth while the Mars Reconnaissance Orbiter was in safe mode. Both the rover and the orbiter share the same X-band frequency with Earth and must coordinate communications.
http://marsrovers.jpl.nasa.gov/mission/sta...ll.html#sol1132

Ah...I get it now, thanks, guys! Anybody else think it's high time to upgrade the DSN...?

If I remember correctly, the reason the two spacecraft share the same frequency is that MRO uses a spare X-band communications transponder left over from the Mars Rover program, identical to the transponder on Spirit, and it never occurred to anyone that Spirit would still be roving when MRO began operations in orbit around Mars.


TTT

MRO's a real spares-thief... MOI was done using engines taken from the orig. 2001 lander

Doug

It's got nothing on Phoenix though... But I now understand, safe mode spacecraft always have precedence over normal mode spacecraft... I know we've lost communications at least twice due to another spacecraft being in safe mode, guess it happens alot.

Phoenix IS the spares cupboard Magellan did well on spares - and Stardust used some optics from Voyager.

Anyway - I digress - MGS's demise certainly stole DSN time from elsewhere, it's only right that missions that have the most need get the most attention.

Hopefully the money will soon be found to do a large array of modest sized dishes for the DSN so that in such a case, the array could be sliced up appropriately to get some sort of coverage for the safe mode AND keep in touch with other missions at the same time.

Doug