Okay, I see where this is going and I guess as an seperated art that uses a photo as start it is okay. In a photographic sense it is not, its not better than any given smartphone hallucinating depth of field when the mono-aperture lense does not has it.
As someone who has a photography walk every few months to years I draw the hard line in the moment its not deterministic anymore, and if I understand this right..its some kind of machine learning modell interferring the depth by looking at it. The data isnt really there, in short, the depth is hallucinated. Same when you use KI fill to touch up your photos...changing exposure, even in zones, boosting colors from raws, all fine..but the moment you non-deterministicly touch a photo it ceases to be a photo and starts to become an AI image that happens to have a photo as starting point.
That all said, on this very topic, is there a camera that actually captures distance in any meaningful resolution? There are fun ToF sensons and 3d voxel clouds of places, but those are usually not something you can snap in milliseconds.
Is "deterministic" really the right line to draw? On the one hand, this is "deterministic", in the sense that (given a fixed model) the same photo will produce the same depth estimation. On the other hand, even film isn't deterministic, with its random grain. But at least random grain looks random. Perhaps the issue at play is really "indexicality", where perceptible signal should be caused by the scene. But even that is fuzzy. What, qualitatively, is the difference between "touching up" and AI enhancement? It's all just algorithms that combine a starting photo with some knowledge of how humans like photos to look. Again, not even film is free of this - when chemically developing, the choice of how strong a developer to use and how long to leave it in has as much effect on the image as the exposure settings chosen by the photographer. There's nothing "deterministic" about a lab technician eyeballing your photos and tweaking the development process.
this is not what I meant. If I take my pro-sumer camera, and take a pictures, photons from reality hit the sensor after passing through a few layers of hopefully high quality glass. So far a lot of messy physics is involved and then there is the bayer array business and math is used to torture those poor photons into pixels. But then. I got a RAW file, maybe 10bit depth, maybe 12, sometimes even 14, more than a jpeg.
Then I go and do things with exposure zones, define highlights, increase the lumina of certain ranges and so on. A lot functions that result in one final receipt. If I take the same receipt with the same raw I reasonable expect the exact same result. The resulting, although manipulated picture, is deterministic in a sense. As soon as an LLM gets involved it is not.
This is the point where I cannot really be in the conversation anymore, I dont understand how the Mariegold Transformer works. Like, I dont know how repeatable it is. I cannot refute or confirm your words. The only thing I can do is nitpick, I am quite sure its not an algorithm as it is a diffusor model as far as I understand it
I feel like you are saying "deterministic", when what you really mean is "predictable". They're all algorithms, just a bunch of multiplies and adds on the image data really. But the algorithms you don't mind are the ones where you feel you have a good intuitive grasp of what they'll do. The ones that seem eerie and break your sense of indexicality are the ones where 1) you're not sure what it will do, and relatedly 2) seem to introduce a substantial amount of new information, perhaps drawn from some database of how images look generally.
To sum up: Marigold V2 is just as deterministic as Bayer demosaicing. But Marigold has about 20 billion magic numbers in it, thoroughly dwarfing the source image. It feels less like transforming the image with Marigold, and more like steering Marigold with the image. And who knows what's in that thing or how it works...
(note: I see the same confusion often in conversations about LLMs. People call them "nondeterministic", as if that were some new category of software. But, with temp 0, they're no less deterministic than any other pile of floating point math - and nobody complains that fluid simulations aren't deterministic. No, what they are is unpredictable.)
This was the core premise of your original comment!
I'm not sure how you define algorithms, but I have a feeling it's not a universal definition. I would argue that diffusion is an algorithm like HDR compensation. Both describe well defined and repeatable processes that can be broken down into low-level components with different potential implementations.
How does your HDR algorithm select a exposure curve for tone-mapping? How does it handle noise rejection? Does it apply any non-local corrections or smoothing? Does the existence of these questions mean that HDR is less of an algorithm? Where do we draw that line?
Stereo cameras/camera arrays can have very large resolutions. An ML model is necessary for optimal results with stereo cameras, but it's more grounded in reality and verifiable versus a mono capture. With a large enough camera array, you can get precise, high resolution depth without ML, especially if you accept Gaussian splats, which use gradient descent but I wouldn't call them generative models in the same sense. Large, featureless, textureless planes are still a weakness however, while generative models can guess/interpolate values.
You can use a hybrid method with speckle projection/ToF/LIDAR (the first two behind mainly usable indoors) if you want to stay fully grounded though.
> I draw the hard line in the moment its not deterministic anymore
Photography never was. It has been copy&pasted together from multiple exposures, different photos and such in the darkroom for well over a hundred years. And once digital became common place the manipulation went completely into overdrive. The photo you see has little to do with the photo that the camera captured.
If we cared about fully deterministic captures, we'd all be using Lytro lightfield cameras, stereo cameras or 360 cameras or other contraptions that capture more light information than the classic camera.
Check the photo with the moped rider next to the escalators (subway station exit?): the depth map shows structure for the shadows of the supports of the glass roof. There is no true depth. Only fake.
Ya, when we specifically ask for hallucinated data we call it _inferred_ data. As I want "good" pictures, I give the system permission to add/remove/infer/hallucinate whatever data it can to achieve a "good" image. Very soon we will all look like supermodels... at least in photos.
Not everyone is a socmed influencer. Some of us want to take pictures that match reality.
In that case, most of the information in every single color digital photo is "hallucinated", because each sensor pixel only captures red, green, or blue, in a checkerboard pattern called a Bayer array; we get uniform RGB photos by applying heuristic interpolation algorithms to (shall we say?) "infer" the missing data.
The selective censorship of faces in the photos but not the depth maps strongly undermines their assertion that they believe they are "detailed enough to become the artwork" and "a new way to approach taking pictures". Ironic, because many of the faces can indeed be resolved in the depth maps. A curious blind spot.
Yes and no. We've been able to generate messy models from photos for a while, but the geometry cleanup process is tedious enough that it's often easier to start from scratch.
Is it just me or does the depth map also include color/brightness? The vatograph images shown seem to also be mostly about subject recognition, not depth. How accurate are these inferred depth maps?
It's not accurate at all. Measuring accuracy would require some sort of calibration. If this was using some kind of stereographical setup or lidar data to calibrate the inferred depth could probably be calibrated, but this is not the case.
Not to say it's useless, its perfect for subject isolation and it would be very difficult for someone to tell the difference with a scientifically accurate equivalent
Lol. The author seems to think that people only deserve privacy when in color. Faces are blocked in the original color image but not in the black-and-white depth image.
Okay, I see where this is going and I guess as an seperated art that uses a photo as start it is okay. In a photographic sense it is not, its not better than any given smartphone hallucinating depth of field when the mono-aperture lense does not has it.
As someone who has a photography walk every few months to years I draw the hard line in the moment its not deterministic anymore, and if I understand this right..its some kind of machine learning modell interferring the depth by looking at it. The data isnt really there, in short, the depth is hallucinated. Same when you use KI fill to touch up your photos...changing exposure, even in zones, boosting colors from raws, all fine..but the moment you non-deterministicly touch a photo it ceases to be a photo and starts to become an AI image that happens to have a photo as starting point.
That all said, on this very topic, is there a camera that actually captures distance in any meaningful resolution? There are fun ToF sensons and 3d voxel clouds of places, but those are usually not something you can snap in milliseconds.
Is "deterministic" really the right line to draw? On the one hand, this is "deterministic", in the sense that (given a fixed model) the same photo will produce the same depth estimation. On the other hand, even film isn't deterministic, with its random grain. But at least random grain looks random. Perhaps the issue at play is really "indexicality", where perceptible signal should be caused by the scene. But even that is fuzzy. What, qualitatively, is the difference between "touching up" and AI enhancement? It's all just algorithms that combine a starting photo with some knowledge of how humans like photos to look. Again, not even film is free of this - when chemically developing, the choice of how strong a developer to use and how long to leave it in has as much effect on the image as the exposure settings chosen by the photographer. There's nothing "deterministic" about a lab technician eyeballing your photos and tweaking the development process.
this is not what I meant. If I take my pro-sumer camera, and take a pictures, photons from reality hit the sensor after passing through a few layers of hopefully high quality glass. So far a lot of messy physics is involved and then there is the bayer array business and math is used to torture those poor photons into pixels. But then. I got a RAW file, maybe 10bit depth, maybe 12, sometimes even 14, more than a jpeg.
Then I go and do things with exposure zones, define highlights, increase the lumina of certain ranges and so on. A lot functions that result in one final receipt. If I take the same receipt with the same raw I reasonable expect the exact same result. The resulting, although manipulated picture, is deterministic in a sense. As soon as an LLM gets involved it is not.
> The resulting, although manipulated picture, is deterministic in a sense. As soon as an LLM gets involved it is not.
Monocular depth estimation algorithms are deterministic in the same sense: using the same recipe on the same input produces the same result.
This is the point where I cannot really be in the conversation anymore, I dont understand how the Mariegold Transformer works. Like, I dont know how repeatable it is. I cannot refute or confirm your words. The only thing I can do is nitpick, I am quite sure its not an algorithm as it is a diffusor model as far as I understand it
I feel like you are saying "deterministic", when what you really mean is "predictable". They're all algorithms, just a bunch of multiplies and adds on the image data really. But the algorithms you don't mind are the ones where you feel you have a good intuitive grasp of what they'll do. The ones that seem eerie and break your sense of indexicality are the ones where 1) you're not sure what it will do, and relatedly 2) seem to introduce a substantial amount of new information, perhaps drawn from some database of how images look generally.
To sum up: Marigold V2 is just as deterministic as Bayer demosaicing. But Marigold has about 20 billion magic numbers in it, thoroughly dwarfing the source image. It feels less like transforming the image with Marigold, and more like steering Marigold with the image. And who knows what's in that thing or how it works...
(note: I see the same confusion often in conversations about LLMs. People call them "nondeterministic", as if that were some new category of software. But, with temp 0, they're no less deterministic than any other pile of floating point math - and nobody complains that fluid simulations aren't deterministic. No, what they are is unpredictable.)
This was the core premise of your original comment!
I'm not sure how you define algorithms, but I have a feeling it's not a universal definition. I would argue that diffusion is an algorithm like HDR compensation. Both describe well defined and repeatable processes that can be broken down into low-level components with different potential implementations.
How does your HDR algorithm select a exposure curve for tone-mapping? How does it handle noise rejection? Does it apply any non-local corrections or smoothing? Does the existence of these questions mean that HDR is less of an algorithm? Where do we draw that line?
Stereo cameras/camera arrays can have very large resolutions. An ML model is necessary for optimal results with stereo cameras, but it's more grounded in reality and verifiable versus a mono capture. With a large enough camera array, you can get precise, high resolution depth without ML, especially if you accept Gaussian splats, which use gradient descent but I wouldn't call them generative models in the same sense. Large, featureless, textureless planes are still a weakness however, while generative models can guess/interpolate values.
You can use a hybrid method with speckle projection/ToF/LIDAR (the first two behind mainly usable indoors) if you want to stay fully grounded though.
> I draw the hard line in the moment its not deterministic anymore
Photography never was. It has been copy&pasted together from multiple exposures, different photos and such in the darkroom for well over a hundred years. And once digital became common place the manipulation went completely into overdrive. The photo you see has little to do with the photo that the camera captured.
If we cared about fully deterministic captures, we'd all be using Lytro lightfield cameras, stereo cameras or 360 cameras or other contraptions that capture more light information than the classic camera.
> Photography never was.
Yup, and it's a fundamental constraint. Photons cannot be captured deterministically. Lightfield imaging won't fix the noise floor.
Phones (Google Pixels at least) use sensor data from inertial sensors and slight hand movement as part of their depth calculation: https://research.google/pubs/depth-from-motion-for-smartphon...
It looks like this research applies specifically to AR video streams, leveraging motion to combine data across frames.
I do not believe this technique is used for photographs.
As a small niche like color key or IR photography or anything else everyone else did once :D
No, no, the depth is _inferred_, not hallucinated. See? There's a difference.
Check the photo with the moped rider next to the escalators (subway station exit?): the depth map shows structure for the shadows of the supports of the glass roof. There is no true depth. Only fake.
Ya, when we specifically ask for hallucinated data we call it _inferred_ data. As I want "good" pictures, I give the system permission to add/remove/infer/hallucinate whatever data it can to achieve a "good" image. Very soon we will all look like supermodels... at least in photos.
Not everyone is a socmed influencer. Some of us want to take pictures that match reality.
In that case, most of the information in every single color digital photo is "hallucinated", because each sensor pixel only captures red, green, or blue, in a checkerboard pattern called a Bayer array; we get uniform RGB photos by applying heuristic interpolation algorithms to (shall we say?) "infer" the missing data.
The selective censorship of faces in the photos but not the depth maps strongly undermines their assertion that they believe they are "detailed enough to become the artwork" and "a new way to approach taking pictures". Ironic, because many of the faces can indeed be resolved in the depth maps. A curious blind spot.
Utter ignorance to begin with.
I wanted to do street photography too, I never did.
I'm big on photography and thought this was a really neat use of what is normally a transient image into the point of the creation.
related discussion from a year ago:
https://news.ycombinator.com/item?id=44183591
More things on there about ways to capture depth maps instead of generating them from a single image
Could this be used to generate 3D models for games and such?
Yes and no. We've been able to generate messy models from photos for a while, but the geometry cleanup process is tedious enough that it's often easier to start from scratch.
Why is the nearby bicycle same distance as far away building?
The black and white images are artistic manipulations of the underlying data.
This is the only depth map with a bicycle from the article:
https://vathography.com/assets/images/vathography-gem-depth....
Is it just me or does the depth map also include color/brightness? The vatograph images shown seem to also be mostly about subject recognition, not depth. How accurate are these inferred depth maps?
It's not accurate at all. Measuring accuracy would require some sort of calibration. If this was using some kind of stereographical setup or lidar data to calibrate the inferred depth could probably be calibrated, but this is not the case.
Not to say it's useless, its perfect for subject isolation and it would be very difficult for someone to tell the difference with a scientifically accurate equivalent
All text on the page is AI slop. Cool effect though.
Lol. The author seems to think that people only deserve privacy when in color. Faces are blocked in the original color image but not in the black-and-white depth image.