I wrote a shell script that allows you to download all images from a given sol.:

http://paranoid.dechengst.nl/files/MSLget.sh

There are two rm commands in the script to clean up the tempdir. If you're afraid my script screws up your system replace them with rm -i commands. If you want any features added please let me know and I'll see what I can do.

EDIT1:

Added leading zeros in the directory names for easy sorting after a request for it on IRC.

EDIT2:

I see a lot of people downloading the script. To be clear: This is a script for *NIX systems and won't run on a standard Windows box. To get it running under Windows you need to install a *NIX environment like Cygwin. Your other option would be to install something like Ubuntu in VirtualBox.

EDIT3:

As RoverDriver pointed out you need to have wget on your system to use this script. The script now handles this gracefully.

I looked over and this should work also on Macs. Your script relies upon wget being installed on your system which is not always the case. That is the only dependency I see. DEChengst, this is not a criticism, just trying to be helpful to anyone who might try using your script and found a problem. Wget for Mac can be found here and it is a cool tool that you should get (no pun intended) anyway. I use it all the time to download large stuff, like HiRISE images, DTMs.

Paolo

As an example of using wget to download the HiRISE images, here's my script that downloads the non-map projecte, the map projected and label files to your current working directory:



Paolo

I've written a small too to help me download the "RAW" Curiosity images from the mars.jpl.nasa.gov site. Originally, I wrote it for my own use, but since some people here might find it useful I thought I should make it public.

The system requirement to run it is .NET 4.0.

It's important to note that this is just a quick and dirty tool I whipped up this morning, so the code quality isn't exactly great. There's not a lot of fault tolerance in the code, for example.
Another thing to keep in mind is that it's dependant on the current appearance of the mars.jpl.nasa.gov site. If that changes significantly, the tool will stop working. If that happens, and there's interest, I will hopefully have time to update the tool to make it work again.

Anyway, here it is, Curiosity RAW Get. It works for me up to Sol 38 data.
Click to view attachment

EDIT:
Since I don't have a web site of my own, I had initially intended to provide public support for the tool on this board, releasing new version here and making sure this post always pointed to the latest version of the software, and handling support in the followup posts.
After all posts related to software were merged into one thread, that is unfortunately no longer possible, so we'll go to plan B instead:
The version included in this post might in a few days or weeks get old. To get new versions or support, contact me off this board through email instead (My email address is included in the About dialog of the program.).

CHANGELOG:
0.16.38 - 2012-09-14
Added subframe support.

0.15.12 - 2012-08-18
Added thumbnail browsing mode.
Added installer.

0.14.10 - 2012-08-16
Added support for the Sol 10 version of the Curiosity web site.
Enabled threaded downloading.

0.13.3 - 2012-08-11
Switched from Microsoft's HTML parser to the HtmlAgilityPack parser.

0.12.3 - 2012-08-11
Fixed a problem where the Microsoft HTML parser didn't return parent element for some elements.

0.11.3 - 2012-08-11
Fixed version dependency on the Microsoft HTML parser.

0.10.3 - 2012-08-11
First release.

For windows users, I've been using Star Downloader Pro. Its a link leecher that Scans and follows links on webpages based on what filtered file extention you input. Then it filters and downloads them. In this case .jpg files. There is a 30 day free trial to try it out, works like a charm for me, so I bought it.

http://www.stardownloader.com/

I'm also working in a phyton script that automate big parts of the downlading process, but as I started teaching myself programming the first week of july it won't be so automatic and will require input from the user, at least until I improve my programming skills. The good part is that anyone will be able to use it as python runs in Windows, Linux and Mac.

Also a shell script using wget initially mentioned in another thread. Several related posts appear right afterward - might be nice to consolidate in this thread.

MOD NOTE: Good idea- done!

Sounds good - the script is now post #1 by DEChengst in this thread.

I would also suggest the DownloadThemAll plugin for Firefox for those who wish. http://www.downthemall.net/

It is not at powerful as wget, but it has served me well for six to seven years now.


-- Pertinax

Hm... actually... The reason why I posted my tool in a new thread was so that the link to it could be in the first post (and thus easily updated when future versions are released), and then use the rest of the thread for support and change logs (compare this to how other message boards [for example doom9] dealing with software releases work).

Is there any chance we could break this out again into one thread per tool? As it is now, I fear it won't be obvious to anyone using the tool when a new version is released. In fact, the first post in the thread called "Curiosity RAW Get" isn't even about the tool called "Curiosity RAW Get" anymore.

MOD EDIT: Thread title changed to reflect a more general theme. Please don't start threads for each tool; that would rapidly become very difficult for the UMSF all-volunteer admin/mod team to manage.

I installed it, but when I try to run it, nothing appears to happen. Actually, not nothing. I get an info window with some disclaimers, but that is it. I have .NET 4.0 running under win7. I tried double-clicking the icon and also running it from a command line.

Do you have any idea what I may be doing wrong?

Okey, that's too bad. I'm afraid supporting the tool in the middle of a generic software thread is too much work for me to manage, instead, so I'll take support off this board instead. I'll update the post to reflect that.

It seems that it might be useful to create a subforum under MSL similar to the "Tech, General, and Imagery" section under MER. That would seem to allow for a little more flexibility with regard to specialized topics like specific software applications.

I'm building a webportal for easier access to the raw images, but I'm highly time-constrained, so it's going slowly.
But the first step is there:

I've put an automatically up-to-date JSON listing of all RAW images online, as a more convenient listing for (web)developers or as input for wget scripts.
(With the lack of folder listings in the raw-images CDN, I've done the same HTML parsing as all of you:)

You can find the JSON list at: http://msl-raw-images.appspot.com/landing
There are a couple of parameters:

counts -> Only list image counts per sol/camera.
cam -> Only list specific camera
sol -> Only list specific sol

Examples:
http://msl-raw-images.appspot.com/landing?...m=MD&counts (will list the count of descent images: 1522)
http://msl-raw-images.appspot.com/landing?sol=1&cam=NRA (will list all right Navcam images for sol 1)

The site will automatically update every 30 minutes, so expect a ~15 minutes delay compared to the original site.
Hope this portal will be usefull already. I'm planning on making a webpage similar to the MER raw images as soon as possible. (hopefully within a day or two)

Greetings,
Ludo.

On previous missions when it's been tough to parse raw images (Phoenix being a major example) I've posted easier-to-browse indexes on the Society's site. But I'm no programmer; I always did this manually. There appear to be lots of different solutions brewing here. I would LOVE it if when one of these gets fleshed out, it could be dropped in to a subsite on the Planetary Society website. With lots of praise and credit to whomever developed the solution. I can handle making it play nicely with our site, but I can't do automated image retrieval/indexing.

I think it'll be fairly easy to get this "fleshed out". I see mainly two problems:

• First, that the tools need to be able to run on a variety of strange and wonderful systems, which means a lot of compatibility testing to start with. It's not difficult, it just takes time, and for a while the tools will embarrass their programmers with really, really stupid bugs.
• Second, and most important, the tools will need to be constantly updated to reflect any changes in the web sites from which they gather data.

To simply get a list of all the raw images for a given sol, I put up a web widget--access it with wget as follows, for example for sol 4:

wget -O sol4files.txt http://curiositymsl.com/cgi-bin/getfiles.cgi?sol=4

I.e., just set the sol=N parameter and you get a text file back with all the raw images currently on the MSL public web site

For a really basic web interface to same, see:

http://curiositymsl.com/rawlist

Thanks, this is great! Note that with Navcam and Hazcam images it's indexing thumbnail versions of the raws (*-thm.jpg) whereas with the MSSS cameras the raws are linked directly.

You're right--thanks! It's fixed now.

Any chance that Midnight Mars Browser can be modified by the code wizards to do the trick for quick MSL downloads? I love that program! It sends photos of each camera to their own folders, even folders for separates sols. Very nice way to keep track. (Crossing fingers).

Believe me, Mike knows very well we want that, and no amount of pestering will make it more likely to happen (or happen any faster), though compliments are probably always welcome. As with pretty much all of you guys, that isn't his day job. We just have to wait and hope he can find time somehow!

Again, many thanks. I downloaded everything available so far, did some light filing (separated thumbs from fulls), and posted it here (30 MB):

https://planetary.s3.amazonaws.com/data/msl...ru_sol_0004.zip

That is how Curiosity RAW Get works as well. It also keeps track of the images you've previously downloaded.

Anyone had any luck decoding the filenames on the raw images?

SSSSIICCCCCCCCCCT1_DXXX

SSSS = sol number (good to see 4 digits here)
II = is the instrument (ML = Mastcam left)
CCCCCCCCCC = ????, is this a serial number? It doen't appear to be a time (see below)
T = image size I or E
1_DXXXX = ????

Looking at a couple of examples:
0003ML0000139000I1_DXXX taken at 2012-08-09 06:01:57 UTC
0003ML0000134000I1_DXXX taken at 2012-08-09 05:59:34 UTC

Those timestamps are 2 min 23 seconds apart. the numbers in the filename do not seem to refer to time, Earth or Mars.


Also, some of the mastcam images ending in I1_DXXX also have higher resolution versions ending in E1_DXXX but aren't listed on the mars.jpl.nasa.gov site. Are they listed somewhere else?

Noticed a little bug on the JPL raw images web site--for full frame MAHLI images they show a thumbnail with a link to the full frame image, but it turns out the thumbnail IMG SRC is actually the full frame image. The html code sets the thumbnail width to 160 so it looks like a thumbnail, but it's a waste of bandwidth as all the full frame MAHLI images for a given sol will be downloaded when just looking at the thumbnail page.

Here's an example (MAHLI images with 'E' in the 17th character of the filename are full-frame, 'I' for thumbnail):

That sounds like a useful bug report, but I have no idea who's in charge of the website. I'll ask around. Maybe Doug knows?

I sent a feedback message under the category "site problem"--we'll see if there's a response. I notice that the same holds true for mastcam images. Most of the mastcam images so far have been thumbnails, but there are a few intermediate ("subframe" 1200x1200) versions which have the non-thumbnail as the IMG SRC. That could be a problem when the big mastcam images start rolling in.

I reported the same to Veronica McGregor a few days ago (who is not in charge, but obviously can talk to whomever is) and she said that they officially do not care about bandwidth since everything is stored in the cloud. Now, that's fine for them but it'd be better for the user if they didn't have to download so much data. But I don't want to hassle them about this detail since it's the least of our problems with accessing data. If I'm going to bug them I want to bug them about things like getting metadata (filter information, pointing information etc) and faster distribution of the raws hopefully by automating things. As of the time I left JPL, as far as anybody knew it was a manual process. Priorities! Other people have been begging Mike Howard to make a version of MMB for Curiosity. Even if he wanted to, he couldn't right now, because they're not posting enough metadata.

That's not a useful response IMHO. The camera thumbnails are lovingly antialiased onboard to be as good as they can be at their minimal size, so just tossing them seems really lazy, and it's contributing to make the pages load really slowly even if you have a fast connection.

As for metadata, if I recall correctly this was provided in a separate file for MER. There are no technical barriers to doing the same thing for MSL...

Let me emphasize, just my own opinion and not an official position, etc.

Let's remember that the mission is not obliged to give us access to these image at all. At least until the PDS release.
It's early days in the mission and the priority is the mission.

All of the so-called "issues" with getting quick and easy access to the images will sort itself out in due course.

Remember that we are privileged to get access to this data, but they are not obligated to provide it.
The fact that they do provide it at all is highly appreciated and while frustrating to some that it's not quicker and easier, we need to remember that their people and resources are not unlimited. Give them time.

UMSF is a great place to provide useful suggestions and ways to access the imagery.
It is of no value to complain about it.

I'm on the camera team and I'll complain about it if I want to, thanks.

That said, this is kind of an interesting article.
http://www.informationweek.com/government/...web-s/240005286

Although I had some problems with the first version, the new one (0.13) is working quite well for me. It seems to be a good option for Windows users who want to download MSL images in mass. The application allows you to apply various filters to your selection, and it automatically creates a nice directory structure.

Michelle Viotti

In this case, the fix is dead simple--the thumbnails do exist. If they just change the 'E' to 'I' in the img src file names they'd be done. Straight bug fix, not a feature request. Until that happens, anyone scraping the web pages for URLs should be aware.

I guess Amazon Web Services would give a big thumbs up to the current situation though!

That's very good to know! I guess that means I can safely enable the threaded downloader in Curiosity RAW Get without risking to hurt their servers, then.

I have my AlgorimancerPG rangefinder/photogrammetry tool updated to work with the MSL navcam images -- I hope to post it to the website tonight (it's not there yet). I'll announce the availability here when I do (this seems the closest topic match in the MSL section).

One thing it is lacking is the correct sol/time calculation; it will display a value, but it's not going to be a correct value. If someone could clarify how to get the sol/time from the image title, that would be much appreciated.

Another thing lacking is camera orientation. For MER, this could be obtained from the Pancam Tracking Database. I'm not aware of any such database as yet for MSL.

Support for the Mastcam's higher resolution cameras may or may not happen for MSL, as this will require rather more time than I happen to have at the moment. If there's someone with experience programming with Microsoft Visual C++ (Visual Studio 2010 professional) who would like to take-on the project, particularly someone familiar with imagery and vectors and quaternion transformations, I would be happy to share the code. This may take anywhere from a few days to a couple of weeks of programming time. Considering that we have high resolution images and topography from orbiters already, I'm not sure there would be much value in adding Mastcam capability, but it is an option.

For Navcam the file name has the SCLK value for the image time, I think. For MMM it appears there is no SCLK in the filename and metadata would be needed to figure out the time. I'm not sure what the time on the raw image website means, I haven't tried to correlate it.

The NAIF website (http://naif.jpl.nasa.gov/naif/data_mars.html) has some information about how to map SCLK to UT and LMST and may be a source for pointing data (I'm not sure what the CK files there look like or what their plan is to update it.)

I think it's just SCET.

I've checked it--for example the website has FLA_397506083EDR_F0010008AUT_04096M_ with a time of 2012-08-06 06:23:34 UTC.

From the file name, the et is 397506083, which according to et2utc_c() translates to 2012 AUG 06 06:20:16 UTC. So there's a discrepancy of about three minutes (3:18). Possibly related to processing on the rover?

Just for another example, here's one from sol 2, NLA_397682534EDR_F0020000AUT_04096M_--the date given is 2012-08-08 07:24:27 UTC and 397682534 translates to 2012 AUG 08 07:21:07 UTC, similarly a 3:20 difference.

My guess is that this is not ET but SCLK, which drifts from ET. You need to use scs2e (or sct2e) to get this into ET.

Thank you, I think that did the trick.

EDIT: I've run into some difficulties with the installer (upgrading the development environment from VS 2005 to VS 2010 has been a real headache). Hopefully I'll be able to fix it tomorrow and get it posted.

Sounds reasonable--but isn't SCLK measured in 'ticks'?

You don't get the subseconds in the filename, so you will be off by a second minus a tick worst-case.

Trying to develop a handier web interface to the raw images, here's a first go if anyone would like to help beta test it:

http://curiositymsl.com/table/view

It shows the raw images in table form, sortable by date taken, release date, etc. by clicking on the column headers. The default sort, by release date, is the handiest I think, since it shows the most recent uploads to the JPL site. It's hard otherwise to keep track of the newest files because they can come in under any sol.

Open to suggestions as to how it could be more useful. One thing that comes to mind is to be able to filter by camera, because with all the images in one big list, there are a lot of pages, mostly due to the large number of MARDI images.

First: AWESOME!

It would be great to filter by camera.

I think image dimensions would be more useful than file size.

I'm amused by the "lag" column, though it's not exactly fair, since it folds in both transmission lag (when did it arrive on Earth?) and raw posting lag.

I'd prefer smaller thumbnails and more images per page, but don't know if that opinion is shared.

This takes more logic, but: it would be great if it did not show *both* thumbnails *and* fulls of the same image. It would be lovely if it could just show the highest-resolution one returned to Earth so far.

Works great on a Samsung Galaxy Nexus smartphone, for what its worth... -Excellent job!

Click to view attachment

Allow me to say OMAGERD THANK YOU

Joe, you are an incredible source of amazing products! You asked for some suggestions, here they are:

1) yes selecting by camera would be great and probably necessary at some point in the mission

2) the dialog at the bottom with the sol numbers should be at the top in a separate, static frame so you can jump to any given sol without having to go to the top or bottom.

3) the dialog dialog box with the sol number should highlight the sol with the most recent images available.

4) if at some point in time they will make the image header available, or get telemetry for each image I suggest that the image name be a link to the telemetry.

One question: why keeping track of when the image was received and, most notably, why keeping track of the "lag"? I'm not sure we keep track of that, but if there is a reason I would like to know and do it as well. Never stop learning!

Paolo

True, that was just a matter of expediency. I can find out the size of the files remotely with a "curl -sI <url>" command, but to get the image dimensions I would need to download all the files as they come in. That could be done.



Haha, I thought that was potentially snarky too, but still, it's of interest to know how "live" the images are. There was a press conference last week where a guy from "Nasa Watch" was insinuating that if they happened to find a picture of stromatolites they would withhold it (which Dr. Grotzinger of course denied). Anyway, looking at earth-shattering photos like NRA_397586934EDR_F0010008AUT_04096M_ which was "withheld" for almost a week, it would seem that real life is less intriguing.

Thanks for the other good suggestions.

I'm thinking maybe some check boxes at he top for each camera to be included (not 17 check boxes, but 1 for each major type!).



The navigation buttons do seem inconvenient down there. Note that they are simply page numbers, not sol numbers (20 entries per page).



Absolutely--did they ever do that for MER?



No pressing reason for tracking the lag other than I thought it might be interesting to see a pattern--see reply above. I realize a lot of time is spent waiting in the relay queue.

Fantastic, Joe!

My one suggestion has to do with the curiositymsl.com home page. Please either A) link to the basic index page from there, or configure your site to automatically forward to the index page. That way I just have to remember "curiositymsl.com" when I get inspired to check for new images, no matter what computer I'm at.

Of course, I can navigate to UMSF, find this thread, click to the right page, and then click the link, but ...