Really cool, I am also working on a port of gaussian-splatting [0] but to WebGPU.
Like all the other implementations I have seen so far, this also makes the same mistake when projecting the ellipsoids in a perspective: First you calculate the covariance in 3D and then project that to 2D [1]. This approach only works with parallel / orthographic projections and applying it to perspectives leads to incorrect results. That is because perspective projections have three additional effects:
- Parallax movements (that is the view plane moves parallel to the ellipsoids) change the shape of the projected ellipse. E.g. a sphere only appears circular when in center of the view, once it moves to the edges it becomes stretched into an ellipse. This effect is manually counter balanced by this matrix I believe [2].
- Rotating an ellipse can change the position it appears at, or in other words creates additional translation. This effect is zero if the ellipse has one of its three axes pointing straight at the view (parallel to the normal of the view plane). But, if it is rotated 45°, then the tip of the ellipse that is closer to the view plane becomes larger through the perspective while the other end becomes smaller. Put together, this slightly shifts the center of the appearance away from the projected center of the ellipsoid.
- Conic sections can not only result in ellipses but also parabola and hyperbola. This however is an edge case that only happens when the ellipsoid intersects with the view plane and can probably be ignored as one would clip away such ellipsoids anyway.
The last two effects are not accounted for in these calculations in any of the implementations I have seen so far. What would be correct to do instead? Do not calculate the 3D covariance. Instead calculate the bounding cone around the ellipsoid which has its vertex at the camera position (perspective origin). Then intersect that with the view plane and the resulting conic section is guaranteed to be the correct contour of the perspective projection of the ellipsoid.
[0]: https://github.com/graphdeco-inria/gaussian-splatting [1]: https://github.com/antimatter15/splat/blob/3695c57e8828fedc2... [2]: https://github.com/antimatter15/splat/blob/3695c57e8828fedc2...
When you zoom out there's lots of visible polygon edges that don't look like they should really be there, as if it's trying to draw soft 'blobs' but the texture coords aren't quite right? Is that a bug or an intentional part of the technique?
So far I have only seen gaussian splatting used on photographic data. Would it make sens to use it for other graphics data, too. Or in other words, does it have potential to be used in games?
Would this technique work for video? The readme of the inria work[1] seems to imply a model is trained per static scene, does that rule out video?
What am I looking at?
Does this use the method proposed by Kerbl and Kopanas at SIGGRAPH 2023?
This is beyond cool. Point clouds are one thing but this… this is amazing. Kudos and great job. It even runs on my work Lenovo at 60fps.
Wow this is insanely cool.
If you make it work within ThreeJS, you're going to leave a trace in the history of 3D on the web with that stuff!
I've never experienced this set of mouse controls for a 3D view ever before and was highly confused for a bit.
Very impressive! Curious what the frame rate would be like for stereoscopic rendering of the same scene on the same hardware. Are there optimizations to be had past the halfway mark?
Fancy! I like that on mobile I can drag to move around!
Gotta try on a pc for some reason on iOS the cloudiness feels more like nerf than Gaussian to me for some reason, gotta try it on pc later
Is it possible to increase the number of points (resolution) with some setting? I want to see more refined view on a higher end machine.
Click through to the github for a list of the controls (I didn't think to try spacebar!) and links to other example scenes.
Gaussian Spatting is the new sensation of the sumer in the 3D Scanning Field. Will it live to its expectation ?
Wow. I was literally just working on my own implementation. You beat me to it! Great work!
why the hell is it that anyone who makes these "clever" demos provides the world's shittiest camera that adds unwelcome rolls.
late 90s bedroom me is shaking his head.
Can't wait to pull this up on my desktop tomorrow.
Runs fine on my 2016 iPhone SE. kudos
Last sentence of the readme…!
This is really cool! The control scheme is confusing though. Instead of the typical WASD for moving and using the mouse to look around, dragging the mouse moves forwards and backwards and orbits around some point, A and D strafe, while W and S look up and down.
EDIT: Looks like a full list of controls is in the readme: https://github.com/antimatter15/splat#controls