3D Gaussian Splatting in V-Ray

[kosso_olli]

Hey guys,
this year at Siggraph some really cool technology has been introduced. There already are some implementations for Blender and Unreal, so it would be cool to have in V-Ray as well.
You can think of this as photogrammetry on steroids.

https://repo-sam.inria.fr/fungraph/3...ian-splatting/
https://arxiv.org/pdf/2308.04079.pdf
https://github.com/graphdeco-inria/gaussian-splatting

[M.Max]

Yeah I've seen some amazing videos about it recently and I really wish there is something like that in Max.
I've seen some blender users render different cameras to cover a full scene then take that to a tool that can convert t to 3D Gaussian Splatting thing .
the splatting preview kinda reminded me of the light cache/irradiance map when viewed inside the Irradiance map viewer so that might help maybe?

other use case was taking clips from movies to convert them to a 3D Gaussian Splatting scenes then put that into unreal to re-render it from different view and add 3d elements to it ,
https://www.youtube.com/watch?v=8CdL...ecisionsStudio ( Sorry for the over-reacting and cringe jokes in advanced )
and here is a tool that can do that online in addition to few interactive examples to see how cool such tech can be ( especially with capturing reflection and transparency)
https://poly.cam/gaussian-splatting

[kosso_olli]

Yeah, I closely followed PolyCam. Somehow their results look worse than what is possible with the viewer from GitHub.
Creating scenes with 3D Gaussian Splatting and then placing objects in them (well, cars in our case), would be very handy.

[vlado]

Yes, I was just actually thinking about this the other day... we did some experiments with NeRFs but that didn't work out too well, hopefully Gaussian splats will work better. My own results from converting videos to Gaussian splats have been quite disappointing but maybe the renderer can provide better input 3D data.

Best regards,
Vlado

[kosso_olli]

Hey Vlado,
great to hear!
The researchers uploaded scene files along with the input data, the results are very promising. I have various sets of images intended for photogrammetry sessions, those should work well with 3D Gaussian Splatting, too. Hope I can find some spare time to test this.

There are hundreds of videos already on Youtube. The first one is amazing, the scene is actually reconstructed in the reflection of the pond, too.

https://www.youtube.com/watch?v=mD0oBE9LJTQ

https://www.youtube.com/watch?v=kShNYOuDnlI

[threedguy]

Yeah, gaussian splatting integration would be great!

I've been testing GSplats, here are examples of a few scenes using VRay for compositing vehicles into the GS/NeRF environments.
And yes, those are Cosmos assets

This was done with a traditional compositing workflow (GS-background rendered in Unity).
Would be fantastic to do this all within 3ds Max.

The mega bonus would be additional integration of GSplats with Phoenix and/or tyflow:
Manipulating GS objects or entire scenes with particle effects, wow

[kosso_olli]

Yeah, gaussian splatting integration would be great!

I've been testing GSplats, here are examples of a few scenes using VRay for compositing vehicles into the GS/NeRF environments.
And yes, those are Cosmos assets

This was done with a traditional compositing workflow (GS-background rendered in Unity).
Would be fantastic to do this all within 3ds Max.

The mega bonus would be additional integration of GSplats with Phoenix and/or tyflow:
Manipulating GS objects or entire scenes with particle effects, wow

Very very cool, thanks for that!
One question regarding the car: The reflections in the car paint is based on the G-Splat volumes or did you use an HDRI or roughly blocked-out geometry for that?

[threedguy]

Thanks for the feedback!
The first part of radiance field 3d reconstructions is estimating the camera poses using photogrammetry.
During this process one can also extract a mesh of the POI. I used this textured mesh as shadow, occlucion and reflection matte and added an hdri.

I'm currently testing Unreal Engine, because it supports GSplats, so the process should be easier (no mattes).
Houdini now also has GSplat support.

Additional info: NerfStudio has just released a version which creates GSplats (Inria's licensing has restrictions for commercial use).

[asen.atanasov]

Yeah, gaussian splatting integration would be great!

I've been testing GSplats, here are examples of a few scenes using VRay for compositing vehicles into the GS/NeRF environments.
And yes, those are Cosmos assets

This was done with a traditional compositing workflow (GS-background rendered in Unity).
Would be fantastic to do this all within 3ds Max.

The mega bonus would be additional integration of GSplats with Phoenix and/or tyflow:
Manipulating GS objects or entire scenes with particle effects, wow

Hi,

the video looks great! I see no GS artifacts. What is your approach for creating the GS model in such a high quality?

We are working on a V-Ray plugin for ray-tracing Gaussian splats which will render reflections and refractions correctly.

Best,
Asen

[threedguy]

Thanks, and your plugin sounds interesting!

Concerning GSplat quality: that's a rather broad topic and probably a bit off topic for this messaging board.
I've sent you a PM.

[kosso_olli]

Hi,

the video looks great! I see no GS artifacts. What is your approach for creating the GS model in such a high quality?

We are working on a V-Ray plugin for ray-tracing Gaussian splats which will render reflections and refractions correctly.

Best,
Asen

Hey Asen, this sounds great!
Can you please elaborate on the reflections/refractions? Do you mean the Gaussian Splatting scene will be reflected/refracted in the objects we render with V-Ray? If so, can the GS also cast GI?

[asen.atanasov]

Hey Asen, this sounds great!
Can you please elaborate on the reflections/refractions? Do you mean the Gaussian Splatting scene will be reflected/refracted in the objects we render with V-Ray? If so, can the GS also cast GI?

Hi!

Yes, exactly! The GS will be something like pre-baked environment which can be seen in camera or through reflections and refractions from the scene objects. We plan to publish the plugin here for testing. At the moment, we haven't considered GI, but we will pay attention to the relevant use cases and the feedback from you!

Best,
Asen

[kosso_olli]

Hi!

Yes, exactly! The GS will be something like pre-baked environment which can be seen in camera or through reflections and refractions from the scene objects. We plan to publish the plugin here for testing. At the moment, we haven't considered GI, but we will pay attention to the relevant use cases and the feedback from you!

Best,
Asen

That's fantastic, thank you so much!

[ultrasonic]

Hey Asen,
first of all thanks for looking into GS for Vray - I can't wait to test this! Even without support for reflections or refractions!
Will this work in Maya and/or Houdini?

Could this be brought to USD land as well?

Cheers
Oli

[kosso_olli]

Some more inspiration:

https://gaussiantracer.github.io/
https://arxiv.org/abs/2407.07090

This is useful for combining 3D with Gaussian Splatting techniques