The article says this:

I find it incredible that they don't have any kind of simulator. I wouldn't even try to support a mission without at least a software simulator.

There's a little FDS information in Chapter 6 of https://ntrs.nasa.gov/api/citations/1988006...19880069935.pdf but all of the specifics are in JPL internal reports.

I have no trouble envisioning systems running a simulator (or other GSE) becoming unmaintainable, and operations budget becoming too small to pay for port and validation of old software to a new platform.


Interesting doc. Thanks for sharing.

Sure, but if they had tools that ran on some 60s-70s platform, they might be operable with a simulator. A slightly newer example -- all of the tools for the MOC on MGS ran on a Microvax. By the end of the mission, I was running them on a simh software simulator since the Microvax hardware was long dead.

The biggest problem could be media -- it's progressively harder to read 9-track tapes -- but I'm sure there's a solution somewhere.

That FDS history is better than anything else I found a few weeks back. And James Wooddell might be unhappy to hear them say that so many people of import have passed. It appears that he might be turning 91 before the month is out. The text cites a graduate paper from USC in 1974 as having a lot of documentation on the architecture.

That text does make it clear that the bigger issue is... what's the software on there, and if a bit's become stuck, what do you do about it. There, Edgar Blizzard is (maybe) 89. Richard Rice is a more generic name, but I can't see that he's necessarily passed, either.

I did sort of glean that the "registers" might simply be specific words in the memory, and "bad register" might be what means the other FDS was abandoned. Which might make a bad bit in a register harder to work around vs. like it was in a specific word of instruction memory like in the hard error in 1985. Yes, there are other registers, but rewriting that much of the code...

Oh, and it says a full FDS software load took 4hrs in 1984, which was then considered real slow because of lower bandwidth.

I'm not sure why you take issue with the simple claim that a lot of people who worked on the project are dead. And even if they're not, they may not be able to help for other reasons.

I note that the document talks about a software simulator that ran on an Interdata system. The 8/32 has SIMH emulator support (likely due to its use in the early development of Unix) so maybe there's some hope there. https://en.wikipedia.org/wiki/Interdata_7/32_and_8/32

FWIW, I reached out to the Voyager team and mentioned all of these possibilities.

Oh, I'm not taking any truck with it, just wondering about particular players. And actually surprised that I came up 0/3 on, well, obituaries for those three, and I was looking for that mainly to get a sense of when they'd have stopped being involved in things. All retired in the 90s, maybe. But I dunno.

It sort of describes the shape of the problem to me. Because it sure looks like a fix will mean changing to FDS firmware, and how lost to history that process might be. So here's hoping it doesn't.

Oh, and maybe I can square my prior "internal redundancy" comment with the Ars report that 1 of 2 FDS on V1 is dead-- your text does say the processor could address memory in either unit. So the system is semantically two intertwined units, one of which is dead on V1. So some higher level block diagrams might call that intertwined unit "the FDS" which internally has lost some redundancy.

The Voygers were launched in 1977 and their computers date to the pre Intel microprocessor era. The 8080 came out in 1974 and had 7 8 bit registers. The 8085 came out in 1976 and also had only 7 key registers. The 8086 came out in 1978 and had about 14 registers. What the registers do in each of these computer is explained in Wikipedia. Registers are specific memory locations often with very specific functions such as stack pointer, program counter, status register flags and a limited number of Main register. Most every instruction in 8080 and 8085 code referenced registers A thru E. If one of these registers died, you don't just rewrite your program using the 4 remaining registers. A register was the accumulator, so used in math operations. You lose any of the registers and you are dead. Someone who know how many registers were in the Voyager computer can speak up, but I assume it was no more powerful than the early Intel chips and had similar limitations. My first 8080 computer had 16 K of 8-bit memory. Programs usually occupied 4-8 K leaving 8 to 12 K of memory (minus 2K for screen memory).

I'm 77 and did machine programing for these chips in the late 1970's and early 1980's. Anyone with better memory of early computers please correct my misstatements.

The Voyager FDS was designed several years before the first microprocessors. It has some unusual architectural features, including 128 general-purpose registers (mapped from the main RAM and not as separate logic entities) and a six-clock basic instruction cycle operating on 4-bit values per clock.

I'm not sure how the Voyager team is proceeding. If I were faced with this problem, I would try to build the smallest possible software load that would send useful telemetry to the transmitter. And to support that, I would build a software simulator of the system and make sure the behavior of existing loads was understood. The FDS memory (8K, I think) is loaded through the CCS, so it should be possible to experiment a bit with new FDS loads without the possibility of bricking everything, assuming of course that the CCS keeps working.

What I gathered from Ars was simply that they're going to try to command it into "encounter mode" or some such other modes. Which make sense, try that before software reload, see what happens.

Floyd -- the very long PDF linked by mcaplinger above has lots of details, including that tidbit about the massive amount of registers. The images basically look like the "8K" memory is probably something like 256 discrete CMOS ICs -- making each 32 bits, so if each register is 4bits, maybe there are 16 chips for the "registers" and 240 for the rest of the memory. All that to say -- the reason the registers are mapped from "main" memory is because the MPU itself is a collection of discrete ICs on a huge board with probably hundreds of SRAM ICs... the memory bottlenecks aren't analogous to CPUs.

In any event, if there really were 128 x 4 = 512 bits of registers over 16 separate chips, simple programs probably don't need to use all 128. So I was thinking a bad register would be hypothetically easy to work around, esp since there's not image processing happening. There is some text on page 187 of the PDF about how DMA instructions take the same time that all other instructions take... or I don't quite follow. Probably the biggest distinction between the regs and the other memory words was that the regs were addressable with 7 bits of a 16-bit instruction, while the memory addresses were 4k/16=8bits? So separate instructions were needed to access the "lower 4k" instruction memory vs. upper 4k scratch vs. upper 4k scratch in other unit? Also not clear to me how a 16-bit memory word would load into 4-bit registers, but this is a special ISA, so perhaps 4 registers load at a time. The other thing indicated is that arithmetic would be slow... a several cycle operation because it was doing 4-bits a cycle. Perhaps a more common operation was to simply forward along specific ranges of data, or MSBs, etc.

All that to say, a bit flip in one of the registers shouldn't be more fatal than a bit flip in instruction memory... just even harder to work around at this stage, because you have to change and reload the firmware. And sure, if a 16-bit word has to load into 4 registers, maybe there are effectively 32 and not 128 registers, when it comes to programs loading from memory. But 32 --> 31 should still be manageable.

I don't think the registers are 4 bits, I think the ALU does arithmetic 4 bits at a time on presumably 16-bit registers. This is called bit-slicing and was a fairly common design technique at the time: https://en.wikipedia.org/wiki/Bit_slicing

So, does anyone have a prognosis here? How likely does it seem that we'll get Voyager back?

My gut feeling, is sadly no.

I wouldn't count the team out, but not being able to get any useful debugging telemetry back makes it much harder to diagnose than the last problem. And it's not clear they have the resources it would take to either recreate the problem on the ground or start over with a new FDS software image.

I wrote a post on the situation here: https://crookedtimber.org/2024/02/19/death-lonely-death/

Written by a non-technical person for a non-technical audience, so please be kind to any errors.


Doug M.

Nice post, but I feel compelled to nit-pick a little, sorry.


They had all of that for Voyager, they just don't have it today, probably because the hardware died and couldn't be repaired. Missions I've worked on (MGS for example) have been severely challenged late in the mission to keep those resources going. The situation has been improving, but I wouldn't call it "solved". And it is rarely if ever a "perfect duplicate" -- on MGS our MOC ground hardware was a bare circuit board mounted to a big sheet of plywood. The spacecraft simulator is usually what's called a "flatsat" -- a collection of boards on tables or in racks, hardly a complete spacecraft.

And good luck finding any of that Pathfinder hardware even today, much less in the near future of "The Martian". All of that stuff was likely scrapped shortly after the mission ended.

What about the Pathfinder models that JPL occasionally trots out for "group portrait" photo ops to show the evolution of Mars rovers from, e.g., Pathfinder to Perseverance? Are those dummies without the electronic innards?

Pathfinder wasn't a rover, do you mean Sojourner?

It's possible that they still have a working flight-like Sojourner, maybe. But the last time I saw it in person actually driving was nearly 20 years ago now.

<groan>

Naturally I meant to refer to the Sojourner component of the Pathfinder mission.

A few ... well, quite a few years ago, Doug Ellison put up some interesting posts on twitter showing how he had borrowed a Sojourner-shaped object from JPL and taken it into his home shop, to spruce it up for a JPL open house.

I guess it's unlikely to have been a drivable model.

The only flight-like rover testbeds in the wild are Marie Curie (Sojourner testbed rover, at one point destined to fly on the cancelled 2001 lander) and 'Dusty' ( MER testbed ) which are both now at the Air and Space museum. There were others 'driveable' MER and Sojourner testbeds that were significantly lower fidelity - more like the 'Scarecrow' rover used for Perseverance and Curiosity. The Perseverance and Curiosity Vehicle System Test Beds are both in the garage at the Mars Yard at JPL. Perseverance and Curiosity also share an avionics testbed (MSTB) which is more analogous to what testbeds are usually like for missions that are not rovers/landers - the 'flat-sat' Mike mentions above.

This is Marie Curie, Dusty, and Maggie - the Sojourner, MER and MSL testbeds...
https://mars.nasa.gov/resources/3792/three-...s-in-mars-yard/

I don't know if Voyager has an equivalent to the MSTB functioning right now. I would be surprised if it does. I will say, keeping our testbeds up and running is VERY non-trivial. Having spares to fix them when things break gets harder and harder with age - and can reach a point where it's simply not possible to get the parts to do it. Older missions get parts poached from their testbeds in support of newer missions etc etc. If there IS a Voyager testbed, I suspect in terms of technicians certified to maintain it or operate it, the engineer trained by the engineer trained by the engineer trained by the engineer who built it probably got laid off 2 weeks ago.

I would have expected nothing less!




Really! TIL.

And as for Voyager... they just tossed it? I mean, Voyager's not that big, and storage is cheap... But on the other hand, 46 years is a very long time, I guess.

Storage is not as plentiful as you might think on the JPL campus. It could easily be in some off-site storage where no one can find it (think the last scene of RAIDERS OF THE LOST ARK) and probably non-functional if it could be found.

From https://ntrs.nasa.gov/api/citations/1988006...5_Optimized.pdf



You want to bet on the chances of an Interdata computer from 1975 still being in working order? I did find emulator support for the Interdata and data files for the Interdata operating system, but finding and reading the unique Voyager software on old tapes if they still exist might be a challenge. Even if anyone knows how those tools work any more.

It is a little surprising that they didn't plan ahead for this a bit better, but it's understandable.

mcaplinger the PDF you linked is amazing reading. Thank you for posting.

<GRIN>Yikes, that brings back memories of my father bringing home giant spools from an IBM 360 to work on over the weekend, and me as a little kid looking up references for him in a 4-inch thick IBM binder.
"Mom, is COBOL a curse word?" "Sometimes dear, sometimes when your father uses it..."

Yes. But the "open in case of emergency" kit that I'm envisioning prob still wouldn't have a working emulator. Might be something more like several "phone home" FDS OS versions that each used subsets of the FDS registers/SRAM, that could be loaded relatively quickly, and tried.

(I am actually somewhat surprised that the "died in 1981" side of the FDS wasn't better diagnosed in that sort of way. I guess in 1981 they presumed they would still have the expertise to so something like develop a modified OS if they needed to cross that bridge, and never did. Or maybe it was diagnosed, and plans to workaround whiteboarded, and forgotten.)

Also, I second what Floyd wrote above. The book from the NTRS site is great reading. So thanks again.

It might still be garbage (probably is), but DSS-63 Madrid is receiving at 40 bps from Voyager 1 right now.

NPR had some more color on this, yesterday:
https://www.npr.org/2024/03/06/1236033493/n...rth-are-worried

No news except perhaps reconfirmation that they're trying mild commanding well before anything drastic, so perhaps expect this to take a long long while.

It looks like we have encouraging progress (sorry for the very long link) : https://www.pasadenastarnews.com/2024/03/09...9o_t4ctexhFHAQM

From the article: "Today, the Voyager team consists of only 12 full-time employees."

Interesting. Somehow I had gotten the notion that Voyager was no longer a full-time job for anyone, except, of course, when there's a problem. I guess that's incorrect?

Either the article is wrong or your notion was, yes.

I don't find it hard to believe that they have 12 FTEs just at JPL for routine ops for the two spacecraft. They have to manage DSN scheduling, build uplink, process downlink, generate PDS archives, etc. And I presume that there are other people at the instrument institutions involved also, though maybe not full-time.

Well they still need a permanent team to deal with the dwindling power supply, even if it takes years, the measurements to extend their lifespans are probably extremely well calculated. And you need at least some guys to maintain base knowledge of the probe.

Voyager costs $5-7M per year according to the 2024 NASA budget request https://www.nasa.gov/nasa-fiscal-year-2024-budget-request/
I expect most of that is for labor, not sure how DSN time is accounted for.

Also it's worth noting - the phrase " 12 full time employees " may not mean 12 people who are full time on Voyager. FTE is a unit of currency for work like this. 12 'full time employees' might be......4 people working 100% on Voyager, 8 people working 50% on Voyager, and 16 people working 25% on Voyager.

As an example - I know the power lead for both MSL and M20 also works on Voyager.

That makes sense (and I realize that it's an example, not the actual roster of the Voyager team).

I knew that there are people "permanently" assigned to Voyager, but I assumed that all of them -- or all but a few -- also had regular responsibilities for other missions or projects.

Well, this is looking more hopeful than I was expecting. V1 has provided a full memory read out of the FDS, presumably missing framing / sync so it took a bit of work to pull it out of the data stream. https://blogs.nasa.gov/sunspot/?fbclid=IwAR...3K_Q5h4e4iY3tyc

There's a 2022 documentary about the team that's very informative. Not sure if anyone else here has watched it, but good for insight into the current goings-on.

I gather a bare bones version of the FDS software was loaded and thats how comms has been restored. I assume its now a case of running tests to find the faulty area of memory.

Excellent and available on Prime Video. I seem to have something in my eye...

It's hard to tell since the blog has been so dumbed down.

Doesn't sound like new software to me, but it might be a patch of some kind.

In some old computer languages, the peek command was used to read a location in memory and a poke command to write to memory. No idea if they poked some addresses in memory to program the peek into memory that was returned????

Thanks Explorer1 for suggesting It's Quieter in the Twilight. It's available for streaming from PBS, and it's an interesting documentary, quite moving, as mcaplinger noted upthread.

And I now recall reading a good New York Times piece (gift link below) about the same team.


https://www.nytimes.com/2017/08/03/magazine...;smid=url-share
The article contains a passage which I had obviously forgotten when I posted yesterday:


I wonder to what extent the team has been able to rely on additional resources outside of their team during the current crisis.

My speculation is that the "dump memory" routine was in instruction memory that they decided to command the FDS to try. I would not guess they changed the instruction memory, because then it'd be running the program they expected, not "eureka! it's a core dump." (Commanding might simply be "start program at this instruction address" but it's not really worth clarifying that because it'll come out in time.)

As forecast on Feb 2:



... so what I'm speculating is that one of those "modes" had a "dump memory" routine and they didn't expect it.

My comment about a new FDS software isnt from the blog article.

Then where is it from?

A NASA source, and it would be unfair of me to say who.

Media reports of a DSN engineer having a eureka moment are a poor game of telephone in that case. But, we've seen that before. Time will tell.

Going by the latest SFOS this looks very hopeful. FDS memory update was sent Friday. Upon the response V1 is getting close to 24hrs coverage! I'll be checking the DSN website to see if we get a data rate of 160bps (cruise mode) :-) https://voyager.jpl.nasa.gov/pdf/sfos2024pd..._04_15.sfos.pdf

Thought I'd post an update as clearly Voyager 1 did not return to science / cruise mode following the update. I gather the update however fully restored telemetry, hence the 24hr track afterwards. This article seems to confirm it is a memory failure and they are having to rewrite / move code to avoid the failed area. https://spacenews.com/nasa-optimistic-about...mputer-problem/

This makes me wonder if the "died in 1981" side of the FDS has more working memory words in the lower region they use as instruction memory. In an earlier post, I wondered how well documented what the bad bits or words were from that 1981 failure.

That failure meant that when the FDS glitched this time, they could not just switch to the other side. (And the sides are intertwined, but there's a lower region of memory that's dedicated to the processor logic on either side, while either side can retrieve data from the upper memory of the other side)

Because if the 1981 failure was also bad memory, they're now contemplating what they didn't do then... which is to rewrite the programs to avoid the bad addresses. And that makes me wonder which side is more functional now that they're both bad.

https://blogs.nasa.gov/voyager/2024/04/04/e...ng-on-solution/

NASA suspecting entire SRAM chip is bad. Saying 3% of memory, which I suspect is 256/8192=0.03125. That might make one chip 256bytes: 2048bits, 16bit word, 128 addresses. 128x16 is what we'd call it.

If it's instruction memory, that's 6% of instruction memory, but the FDS isn't processing many images these decades.

We’re back !

Very impressive work! Congratulations!
https://blogs.nasa.gov/voyager/2024/04/04/e...ng-on-solution/