Rendering gaussian splats is cool and all but you know what would get me really exited?
The ability to generate a gaussian splat from a 3d Scene.
These days 80% of my income comes from realtime stuff.
I’d love to setup a scene in vray, setup some sort of region, press a button wait for, (probably ages) and get a ready made gaussian splat out of it.
That would turn my workflow for everything upside down.
I guess I could render a few thousand frames from a scene and generate the splat outside of vray. Which would be cool too. I kinda shy away from the setup time for that though. Is there reserch in that direction from you guys? Is it even possible?
Im still wrapping my head around this tech.
So I dont know how much sense stuff that I say might make.
If capured from the real world you’d obviously have to walk around objects in order to get them represended from all sides. Idealy you’d do that from different angles too.
So, by that you define the “explorable” area. Still, because you’ll also capture the background while doing that, it will also be included but kind of “as seen” from the explorable area. So you can’t go there without it falling apart.
Thats what I have in mind when I say:
So the way I imagine this is as following:
We can put a gizmo or even better a custom object in the scene which defines the exporable area. Objetcs inside of that area are captured from mutlible different Angles*. But the rest of the scene outside of that area would be also included, but only to be viewn from the exploreable area if you know what I mean.
*As this will be very important in terms of rendertimes in vray and complexity/size of the gaussian spalt scene in the end (i.e realtimyness) I guess it would be good to have some sort of parameter to influrence that.
What I hope to get is the ablility to get a scene or the static part of it into a realtime engine and be able to use all the tools Max/Forestpack/Vray have to offer.
So I can put background trees with millions of poligons and 3d leafes in. I can use displacement, and where I can, avoid having to have Uvs. All be lit with GI and have raytraced reflections (that obviosliy won’t reflect any additional object I’d put in, in the realtime engine.)
I would have used that for many of my projects in the last 5 years.
But of course for it to be usefull, the rendertimes would have to be managable. And the quality would have to be good enough while keeping the file size (because I guess VRAM is the limiting factor here) in reasonable I.e realtime ready bounds.
By “rendertimes would have to be managable”
I mean
1: I’d have to have parameters to balance between quality and rendertimes.
And
2: while it renders be able to view or output the scene so I can judge is everything is going in the right direction.
Much like the resumeable rendering feature which I can set to save its state every n amount of time.
Thats how I manage to do Matterport style 360° tours with 12x2k 6:1 Cubemaps
If I’ve got those two things in place I’d be able to use it in production even if I’d have to wait for a week or two for it to be done. I guess distributed rendering wouldn’t harm :-Dthough;)
If I could create a gaussian splat from a 3d model in vray and load it into Unity3d to view in a Quest, that could potentially be huge. Currently I have a very complex process of baking lighting onto the geometry, and importing the geometry into Unity3d. Which loses some of the reflections/materiality.
Gaussian splatting would be a great tool for real-time rendering of complex interiors such as airplanes or trains. It takes an enormous amount of effort to simplify a heavy 3D model for real-time application and to bake the materials and lighting. For many 3D applications such as Rhino3D, this is not even available yet. With Gaussian splat, you could skip baking for these tools and go straight to the better-looking Gaussian splatting.
To start with, I imagine that the user defines a camera area volume in which they could later move around with a special viewer. This area would be good for generating the data and would later help to ensure that the calculated space is not left when viewing.
I suspect that it is not too complicated to generate a point cloud with raytracing. Wasn’t that already available for the irradiance map and/or light cache calculation?
The method photographers use to generate Gaussian splatting models is far too cumbersome for 3D artists and would cause unnecessary duplicate calculations. For example, the exact spatial point of the surfaces in the 3D model is already known, which should prevent floating splats. The diffuse lighting of a point in space also does not need to be unnecessarily recalculated for hundreds of frames; the good old light cache should help here. …
I would love to see this integrated in Vantage. I work in 3DVista to set up virtual tours for future shopping malls to present all the single commercial spaces to potential tenants. Now I have to render several panorama’s and animations to give a complete presentation of the whole center and the single spaces. Having a PLY, which I can load in 3DVista, generated from the model I use for animations would make things far more spectacualr and insightful.
I saw a video of them supporting gaussian splats, but can’t find a sample project, was looking to invest in it because I was trying to code something myself in Playcanvas but for a fixed non subscription price this would be a no brainer. Recently I found this although the UI looks horrible, haven’t tried it yet, but also looks nice as they combine guassian splats with video and stills. https://viz4d.com/
Yes I know it can be done, do it should be possible to integrate this in Vantage, so it can generate a PLY somehow. If I can get the quality of the interior demo and stream the PLY locally this could be the solution. External hosting and subscription for this service is just a nightmare. I already spend too much time up- and downloading stuff to AI-services, hoping their output somehow matches my wishes in acceptable time. I wonder why the “I” in AI is for Intelligence…it would better be Investigation. 100’s of engines, each with their strong AND weak points.
It was paused because we wanted to improve the quality. Since then, there have been new papers that can be used in order to help with this process, and we will investigate them further.
It would help us a lot if we could collect potential use cases in which you would like to use the feature. And what would you like to control as an output?
Best use case - being able to load the splat into a VR headset. Great way to explore a space after having done all the work in 3dsmax/Vray. And not have to redo work in a game engine or do a lot of work to optimize geometry, bake lighting, etc.
