Hi all,

Here the long overdue continuation of the "Alien Landscapes" series. This time based on 3D
DEMs generated with "Shape from Shading" from single HiRISE images.
Enjoy

Click on Images for larger version.

Detail views from PSP_002172_1410 (large gully system)





Detail view of Gullies from PSP_001376_1675





Detail of gully system in PSP_002022_1455



Dune Views from PSP_004339_1890





Detail from PSP_001834_1605





Here is some background info on the making of the images:

"Shape from Shading" (SFS) i.e. the possibility to extract shape information from a single image has always been a fascinating topic for me.
Now I found the time to implement a prototype for a new SFS algorithm based on some ideas that I've been thinking about for a long time.
The problem with existing SFS approaches (see here for a survey is that they either tend to over-smooth the details (due to the regularization constraint) or suffer from excessive noise in the high-frequency components of the reconstructed surface. Another problem is the large demand on CPU ressources which would make them very challenging to apply to large scale input data, such as HiRISE orbiter images.

So for a long time I was rather sceptical as to the potential of SFS and it was my impression that Methods based on multiple images (stereo) must be far superior to single-image SFS.

However, after a long time of experimenting, combining existing approaches with some new ideas, I got the following quite promising first results that I'd like to share:

All of the images were generated from a single HiRISE image (no depth information was used from stereo or laser altimeter data).
Also, no texturing or additional coloring/shading was applied when rendering the surface.
Every detail visible is real 3D down to the pixel-level...
For rendering I used a very simple model based on lambertian reflection with gouraud shading.

The resolution of the images is still moderate: that is downsampled details crops in the order of 0.5-1 Megapixels.
However, despite the heavy math machinery that drives the core of the algoritm (several systems of equations with millions of unknowns)
the processing time is still moderate (about 15 Minutes per med-res image, using about 2 Gigs main mem) such that the application to full-res HiRISE images should be possible

The following image shows an example to illustrate the general principle (click to enlarge).


On the left hand side the 2D input image (simple noisy JPEG from the Web with unknwon light source direction). On the right hand side shows the recovered 3D surface re-lighted under a different light source direction.
Note that one problem of the current implementation of the algorithm is it's vulnerability to notable distortions in the low frequency components (i.e. large scale variations) of the generated surface. However I'm confident that this can be overcome by an improved version or by adding the large-scale depth information from stereo-based DEMs or altimeter data (MOLA) where available.

WOAH!

I want to climb those dunes..... amazing!

Mind-blowing.

The images are just amazing, fantastic job.

Those are just astonishing Bernhard, great work!

Amazing and beautiful stuff, as always. I don't know about everyone else but my mind is spinning as I ponder the implications of this process. I can see demand for a self-contained software product for this process, or at least a plug-in for one of the popular 3-D utilities.

Another UMSF classic, it's truly a joy to come here and view this wonderful work,
thanks for posting that...

Bernhard,
This is fabulous progress report!
I hope to see you write this up in a journal soon. The images are stunning
(I look forward to the day I can use your technique!) Cheers

O

M

F

G

Unbelievable!!!
I want to climb one of those hills.

In a word:

WOW!!!

Are you draping the original image over the DEM or is the DEM itself this detailed?

Brilliant Bernhard

As I was looking at those pictures I was wondering what Doug would think of them... then he posted.
Nuff said.

Roy

Just freakin' amazing, is all! What immediacy these images have.

There is no texture mapping or smoothing involved: every pixel is real 3D - The DEM itself is detailed down to the single pixel level.

That's the really interesting thing with this new techinque: The absence of the strong smoothing process that gives many conventional DEM-images that somewhat "washed", unnaturally soft appearance.

(however: while it works perfectly for the higher frequency bands, there can be distortions in the lower frequency components ...
so the DEMs in their current form could probably not be used for exact inference about variations of absolute terrain heights over larger scales (that's where MOLA and conventional stereo based techniques have advantages: so it seems natural to combine the techniques) ...

P.S.: I also forgot to mention that in most images the vertical scale is exaggerated by a factor of about 1.5 to 2.

Can your amazing new process create renders with no vertical exaggeration? I'd love to see those!

Me too. I always prefer to see landscapes without the 'victorian-painter-in-the-highlands' effect. Very nice work tho.

I suspected this from the first message but wanted to be sure. This makes this DEM even more amazing.


It would be really interesting to see DEMs for bodies like Europa and Enceladus. Also I have a global DEM of Rhea so a comparison to my results there would be interesting. As you mentioned a DEM from your software would be far more detailed than mine but low frequency variations would be less accurate (combining results from these two approaches might be interesting for big impact basins like Tirawa).

Thanks Björn,

i too am interested in combing the various DEM approaches but at the moment, unfortunately lacking the time (still working on the improvement to the single-image SFS method)

here is another image (a close up of gully wall) that illustrates the high detail preservation (it's not even rendered at full resolution)

(click for larger version)


And for Stu and and ngunn: no vertical (i.e. profile) exaggeration this time :-)

...how sedimentary can you get?

Excellent! Now I can make an informed choice of footwear.

Please excuse a very basic question. If these are made from single images how do you prevent albedo differences from masquerading as relief?

of course there is no general solution to deal with varying albedo in "shape-from-single-image", because, as you pointed out, any difference in luminance will be interpreted as difference in local surface orientation. (The same problem also occurs for cast shadows that will be interpreted just as flat areas)
However, experience shows that the underlying assumption of constant albedo works surprisingly well for large parts of the martian surface where there is very little variance in surface color and albedo.

Furthermore, the general idea with the SFS approach is to incorporate additional depth information from stereo or altimeter based measurements, so those techniques nicely complement each other

If you want a good test - try the Pathfinder observations

And for an Earthly analog here is the Emperor Face of Mt. Robson, the highest peak in the Canadian Rockies. (If it's anything similar Nigel I wouldn't just worry about what you are wearing on your feet but which implements you are carrying in your hands as well.)

...what everyone else has said!

I know very little of the field, but isn't this somewhat ground-breaking, revolutionary work that will have significant applications, like, all over the place? Including earth observations? (As a somewhat random example, there's an appeal out for high-res DEMs of Haiti : http://www.boingboing.net/2010/01/15/haiti...#comment-688501 )

Forgive a lay-person's question: would it be possible to drape a texture map or surface image over the DEM, without it looking really ugly and pixellated in places?



Ah, that's just what I was wondering; those dunes looked a little steep.

I would not use such big words like "revolutionary" (this would be waaay too much honor )

The field of single-image "Shape-from-Shading" (SfS) has been a classical area of computer vision since the 1970s (in the planetary science community also known under the name photoclinometry) and it has also been widely applied already for planetary data sets. See for example the seminal work done by Randolph Kirk of USGS in the field.

Among other notable applications, SfS/photoclinometric methods based on MGS/MOC imagery were involved with the MER landing site selections.

For a general survey on the field, see for example this paper

The current new method that I'm experimenting with at the moment, does seem to have the potential to improve on existing approaches in particular with respect to the better preserving of high frequency (i.e. detail) variations, whereas on the other hand it too suffers from some of the (fundamental) problems inherent to any single-image method, namely the constant-albedo, shadows, convex-concave-ambiguity, low-frequency distortions to name a few (thats were it could be complemented with the already mentiond combination with stereo- and altimeter-based methods )

However, at this time I'm at a very early stage of experimenting and it's way too early for a final conclusive judgment. I just wanted to share the first results, because I for myself was surprised that they came out rather promising. But there is still a lot of work to do before any definite conclusions can be drawn.
Its more of an evolutionary than revolutionary process

just for fun (and comparison with the Mars DEMs): here is a quick try of the SFS method to the earthly Mt. Robson (unoptimized low-res, just from the noisy JPEG image as attached to ElkroveDan's post above)

Here are some views of the resulting 3D DEM:

Click to view attachment

Click to view attachment

Bernhard, your work is "estupendo" ! "Stupéfiant" ! Brilliant ! ...I'm just lacking of words to express my enthusiasm about what you did.
BRAVO !

You are waaay too modest -- these are extraordinary. I never expected to hover off a martian crater wall in a helicopter.

Bernhard, after Olivier's statement, knowing what he does himself, I only can add: "Salut l'Artiste".

Bernhard,
What could your technique do with a fairly bland area, such as Meridiani?
Would you be able to discern the small rise that fresh crater (conception) sits upon? Cheers

Earlier in this thread I mentioned Enceladus and Europa. If it's not too much work I would be very interested in seeing what a DEM of the attached image of Enceladus looks like. I have a DEM for a fairly big part of it (not near the bottom though) so a comparison would be interesting.

Click to view attachment

ok, no problem: here I generated some views of it in 3D

Click to view attachment
Click to view attachment
Click to view attachment
Click to view attachment
Click to view attachment
Click to view attachment

P.S.: one nice thing about the algorithm is that one does not need to specify any external calibration parameters, like the light source (sun incidence and azimut) direction.( because those are estimated simultaneously with the recoverered 3D surface )

So it can be applied to any arbitrary 2D image without knowledge about the exact viewing conditions.

OK - yeah - do you think you'll be able to share the software and technique at some point? I have enough ideas for this that would keep you on your toes till April

*Jaw drops to the floor*

This is awesome, probably the most interesting UMSF-member-developed software I have seen in the history of UMSF so I can simply repeat what Doug said above (actually I could probably keep you busy till April *next* year or longer ;-). If I could combine output from SFS with the output from my steromatcher I'd get DEMs of probably something like 10 times higher resolution than I have ever dreamt of. Combining DEMs like this is probably not difficult. If you manage to get this to work the way you want to there are lots of people here who would love to test it.

And I would make my stereomatcher available - would need to spruce it up a lot though and it still has some bugs and quirks. Combining results from these two approaches opens up *lots* of possibilities. In particular, the low frequency variations from my software are pretty accurate (how accurate depends largely on how accurate the viewing geometry information is) while high frequency details are a problem and frequently 'disappear'.


In contrast my stereomatcher needs accurate viewing geometry information, field of view etc. Inaccurate viewing geometry information typically manifests itself as a 'tilted' DEM.

Things I want to try ( in no particular order )

Topography of rocks using MER MI and Pancam imagery
Height maps of MER sites using vertical projection mosaics
Produce DEM's from HiRISE of the Mars Pathfinder landing site, and the full Opportunity site ( from Eagle all the way to Victoria ) as well as Phoenix and Viking sites

Not to be dismissive of this, but as Nirgal pointed out himself, SFS/photoclinometry has been used in the planetary science community for a long time but just isn't very accurate in a lot of cases. While it may be sufficient for pretty visualizations, do you have any sense for how truly accurate your algorithm is?

Since we have a lot of examples of places with very detailed and accurate DEMs it would be interesting to try this process on single images of those areas and compare the two methods. Columbia Hills comes to mind. I think we have flyover DEM of that around here somewhere

That's right: the main application of the algorithm (at least in its current form) is clearly the area of plausible visualization in near-photorealistic image resolution. (and that's the application that I'm personally most interested in)
In this area it seems to be a very interesting tool to visually explore and display the huge space imaging datasets, particularly when there is no other 3D cue like altimeter or stereo data available.
The question of actual physical accuracy however, is much more difficult to answer.
In this regard, there are clear limitations to any shape-from-single-image methodology as already pointed out.
This is where the already mentioned combination with low-res but more accurate data (altimeter, stereo) comes into play ...
So in summary: yes, this project (in its current state) is intended primarily as a means of visualization, not so much as a scientific measuring instrument (of which there are already many other and better possibilities).

For me, it has been an old childhood dream to literally "wander into" and "bring to live" the images that were sent back by our space probes ... images to fuel the imagination of space exploration

ASTOUNDING ! AMAZING ! You just did a WONDERFUL work Nirgal
Absolument magnifique !

I want to view more about that

You keep sharing your "pretty visualisations" Nirgal; 1000% accurate or not, we're all loving them!

I have a list of requests too, but they'll keep. I'm sure there's a queue!

There are lots and lots of things I'd be interested in testing but I'm posting just a single image to test if it's not too much trouble. What I'm posting here has been carefully selected to ensure I shouldn't have to post more stuff to test, at least not in the near term (although testing how the SFS software performs with the sun almost directly overhead would be interesting as well, my stereomatcher performs poorly in that case for obvious reasons).

This is a Cassini image of Rhea with the Tirawa impact basin partially visible in the upper right quadrant:

Click to view attachment

I would be very interested in seeing a DEM from this image. What would be most interesting, if possible, would be a 1024x1024 pixel 16 bit grayscale DEM as a tiff or png file. If this is possible I'll attempt to merge it into my global DEM of Rhea that I made using stereo pairs. That would combine the best of both worlds and I would post the results here. Otherwise some test renders would be nice.

This is a portion of my DEM of Rhea showing most of the image above (it's slightly cropped):

Click to view attachment

Here black represents -6.320 km and white 6.176 km relative to an ellipsoid. Unlike the Cassini image above, this 'image' is in simple cylindrical projection and covers longitude 150.25 to 187.875 degrees and latitude 8.625 to 41.625 degrees (including the black stuff surrounding the DEM).

Exactly the same needs for me !
=> Is your software working also on a Windows environment (to be more easily carried on a portable home computer when travelling) ?
Bravo ! Encore !

I'd VERY much like to produce a detailed DEM from the Magellan Venus radar images, such as Maxwell Montes: http://photojournal.jpl.nasa.gov/catalog/PIA00149
I think that small scale topography could be recovered by SFS / photoclinometry algorithms and integrated into the low resolutions altimetry data we already have. That would reduce artifacts introduced by surface "albedo" changes, as we would only be recovering small scale local terrain features. Anyone tried something like it?

I'm not sure this would work on Radar.

It should work on radar, as there's the concept of "radar illumination".
See here: http://www.ccrs.nrcan.gc.ca/glossary/index_e.php?id=2844
Of course, having the "light" coming from an orbiting (moving) spacecraft is not the same as from a single distant light source, but you still have higher or lower reflections according to the surface angle. So in theory you could recover some information from those radar images, at least better than the global altimetry we have.

On the Magellan project this was known as "radarclinometry" (do a google search.) It didn't work all that well IIRC.

More recent applications are described in http://www.lpi.usra.edu/meetings/lpsc2009/pdf/1071.pdf

*cough* Titan..? (Edit - sorry, I didn't spot Mcaplinger's post before posting.)

Also, I wonder if hidden craters like Scamander might show up at Gusev?

The software wouldn't work on radar images without some rewriting, because foreshortening of topography works oppositely between radar and visible imaging. In radar, high-elevation things are closer to the spacecraft than low-elevation things, so their echoes are received earlier, so mountain peaks are apparently displaced toward the spacecraft rather than away from it as they are with visible images.

Mcaplinger: Although I think you're right that radarclinometry wasn't as effective as photoclinometry on, say, Viking or LO images, I think in many cases it was better than the quality of the Magellan altimetric data. I worked really closely with that altimetric data and over any place where there was any significant topography its quality was really poor, with lots of blatantly incorrect elevations, great big holes represented by one altimetric footprint in the middle of ridges and the like. Apparently Peter Ford's group developed an improved version of the global topographic map internally, correcting some of these issues, but as far as I know it has never been published.

When I was working on Magellan we were using radar stereogrammetry (see http://www.lpi.usra.edu/meetings/lpsc2009/pdf/1253.pdf for some recent work.) Stereogrammetry is always preferred to photoclinometry if stereo is available.

Bernhard, just an idea...
Because this 3D HI-Res DEM rendere is a huge work of yours that is to be acknowledged, what if you share your software to UMSF members versus a financial participation (or a donation) ?