Holy mother of god. Wow!
Either matterport takes and runs with this or this is a startup waiting to disrupt Realestate.
I can’t believe how smooth this ran on my smartphone.
Feedback: if there was a mode to use the phone compass and gyro for navigation, it’d feel natural. Felt weird to navigate with fingers and figure how to move in xyz dimension.
As others have said, VR mode would be epic.
It runs impressively well on my 2yo s21fe. It was super impressive how it streamed in more images as I explored the space. The tv reflections in the Berlin demo were super impressive.
My one note is that it look a really long time to load all the images - the scene wouldn't render until all ~40 initial images loaded. Would it be possible to start partially rendering as the images arrive, or do you need to wait for all of them before you can do the first big render?
Wow. Some questions:
Take for instance the fulllivingroom demo. (I prefer fps mode.)
1) How many images are input?
2) How long does it take to compute these models?
3) How long does it take to prepare these models for this browser, with all levels, etc?
4) Have you tried this in VR yet?
"Researchers create open-source platform for Neural Radiance Field development" (2023) https://news.ycombinator.com/item?id=36966076
NeRF Studio > Included Methods, Third-party Methods: https://docs.nerf.studio/#supported-methods
Neural Radiance Field: https://en.wikipedia.org/wiki/Neural_radiance_field
I'm following this through two minutes paper and I'm looking forward to using it.
My grandpa died 2 years ago and in hindsight I took pictures for using them as in your demo.
Awesome thanks:)
This is __really__ stunning work, huge, huge, deal that I'm seeing this in a web browser on my phone. Congratulations!
When I look at the NYC scene in the highest quality on desktop, I'm surprised by how low-quality ex. the stuff on the counter and shelves is. So then I load the lego model, and see that's _very_ detailed, so it doesn't seem inherent to the method.
Is it a consequence of input photo quality, or something else?
There is a market here for Realtors to upload pictures and produce walk-throughs of homes for sale.
Does the an open source toolchain exist for capturing, processing, and hosting navigable 3D walkthroughs like this (e.g. something like an open-source Matterport)?
What I'm seeing from all of these things is very accurate single navigable 3D images.
What I haven't seen anything of is feature and object detection, blocking and extraction.
Hopefully a more efficient and streamable codec necessitates the sort of structure that lends itself more easily to analysis.
Why is there a 300m^2 footprint limit if the sub-models are dynamically loaded. Is this constrained by training, rasterizing, or both?
When might we see this in consumer VR? I'm surprised we don't already but I was suspecting it was a computation constraint.
Does this relieve the computation constraint enough to run on Quest 2/3?
Is there something else that would prevent binocular use?
Wow! What am I even looking at here? Polygons, voxels, or something else entirely? How were the benchmarks recorded?
Is there any relation between this class of rendering techniques and the way the BD scenes in Cyberpunk 2077 were created? The behavior of the volume and the "voxels" seem eerily similar.
Very impressive! Any information on how this compares to 3D Gaussian splatting in terms of performance, quality or data size?
this looks really amazing. i have a relatively old smartphone (2019) and its really surprisingly smooth and high fidently. amazing job!
Great work!!
Question for the authors, are there opportunities, where they exist, to not use optimization or tuning methods for reconstructing a model of a scene?
We are refining efficient ways of rendering a view of a scene from these models but the scenes remain static. The scenes also take a while to reconstruct too.
Can we still achieve the great look and details of RF and GS without paying for an expensive reconstruction per instance of the scene?
Are there ways of greedily reconstructing a scene with traditional CG methods into these new representations now that they are fast to render?
Please forgive any misconceptions that I may have in advanced! We really appreciate the work y'all are advancing!
Can you recommend a good entry point into the theory/math behind these? This is one of those true "wtf, we can do this now?" moments, I'm super curious about how these are generated/created.
Is there a relatively easy way to apply these kinds of techniques (either NeRFs or gaussian splats) to larger environments even if it's lower precision? Like say small towns/a few blocks worth of env.
Any plans to do this in VR? I would love to try this.
Are radiance fields related to Gaussian splattering?
I wonder since this runs at real time framerate if it would be possible for someone to composite a regular rasterized frame on top of something like this (with correct depth testing) to make a game
For example a 3rd person game where the character you control and the NPCs/enemies is raster but the environment is all radiance fields
Im not sure why this demo runs so horribly in Firefox but not other browsers..anyone else having this?
Been following Jon Barron et al’s work for a little while now. The speed of improvement given the complexity of these types of systems is mind-boggling.
I wonder how long it takes before Google street view gets replaced by some NeRF variant.
Get this on a VR headset and you have a game changer literally.
How long until you can stitch Street View into a seamless streaming NeRF of every street in the world? I hope that's the goal you're working towards!
I'm curious how the creators would compare this to the capabilities of Unreal Engine 5 (as far as the display technology goes.)
Impressive is not a big enough statement! This is incredibly smooth on my phone and crazy good on a desktop pc. Keep it up!
Since you're here @author :) Do you mind giving a quick rundown on how this competes with the quality of zip-nerf?
Amazing, impressive, almost unbelievable :O
How long until I can make my own?
I had read about a competing technology that was suggesting NeRF's were a dead end
but perhaps that was biased?
Any plans to release the models ?
This is very impressive but given its by Google, will some code ever be released?
>Google DeepMind Google Research Google Inc.
What a variety of groups! How did this come about?
Will there be any notebooks or other code released to train our own models?
What kind of modes does the viewer cycle through when I press the space key?
Just ran this on my phone through a browser, this is very impressive
Very impressive demo.
memory efficient? It downloaded 500meg!
[dead]
[dead]
[flagged]
Hope this doesn't come as snarky, but does Google pressure researchers to do PR in their papers? This really is cool, but there is a lot of self-promotion in this paper and very little discussion of limitations (and the discussion of them is bookended by qualifications why they really aren't limitations).
It makes it harder for me to trust the paper if I feel like the paper is trying to persuade me of something rather than describe the complete findings.
The mirror on the wall of the bathroom in the Berlin location looks through to the kitchen in the next room. I guess the depth gauging algorithm uses parallax, and mirrors confuse it, seeming like windows. The kitchen has a blob of blurriness as the rear of the mirror intrudes into kitchen, but you can see through the blurriness to either room.
The effect is a bit spooky. I felt like a ghost going through walls.