SDK for 3dsMax: VRenderInstance best practices?

[tysonibele]

I've implemented VRay render instancing in my plugin and it works fine to a degree, but it's slower than expected and uses more memory than expected, so I want to know if I'm doing things right.

I'm doing things the same way that they're done in the VrayInstancer sample project: at render time, new instances are created with calls to VRenderObject::newRenderInstance. Once those instances are created, they're passed to the renderer with calls to RenderBegin and FrameBegin, and when a render finishes, they're cleaned up with FrameEnd/RenderEnd.

However, when creating multiple millions of instances, these initialization/cleanup times can take extra-ordinarily long times. Several minutes. Testing other plugins (like Thinkbox's Frost) using their VRay Instance mode takes only a few seconds for the same number of instances. Also RAM seems to bloat in my implementation....10 million instances of some lowres geospheres can require 50+gigs of RAM in some cases. sizeof(VR::VRenderInstance) returns 1320 bytes, so at minimum 10m instances would require about 13 gigs of RAM excluding any extra allocations during initialization...but the amount of usage I'm seeing is still much higher than what's returned when using Frost for the same particle count, so I'm not sure what I'm doing wrong.

Since all of my instances use the same geometry, my thought is that maybe I'm doing things wrong. Do I need a new VRenderInstance for each actual instance? Or is there a way to pass VRay a single VRenderInstance object along with a list of transforms for it that represent the total number of desired instances or something? There's not much documentation on how these things work under the hood, so I'm hoping a developer can shed some light.

Thanks!

[joconnell]

This conversation makes me happy

As an aside Tyson, how did you go about learning c++ or did you have a background in computer science before vfx?

[tysonibele]

No background in it, went from MAXScript to C# for Unity to C++ for MaxSDK. Took many years of trial and error

[joconnell]

I've been talking about learning c for years and kept waiting for the best way to learn it, I'd be making stuff at this stage if I'd just jumped in

Congrats on TyFlow, it's looking seriously impressive!

[tysonibele]

Hey thanks! It's almost ready for release...just a couple more features and a few more pesky details (like this) to iron out.

[joconnell]

Oh there'll be a lot of people in here very excited about that, we love having very elegant quick and dirty tools available to us for when the fx department gets a bit too tied up! What's the oldest release of max it's running in? We're still on 2016 here for a while longer and no doubt there's a few of the thinking particles folks that'd love to have some power back! I hope you're selling it to recoup all of the development time!

[tysonibele]

Unfortunately it's 2017 and above....is Thinking Particles no longer supported by 2016 either?

[joconnell]

It's more the rental idea that's annoying, pflow / tyflow is far more accessible for people that haven't used max much either!

[yavor.rubenov]

Hello,

First of all some questions:
- what type of geometry do you want to instance - is it some fixed list of preset geometries like box,sphere, etc or do you allow users to specify custom objects
- do you plan some material variation options - like different instances to use different materials
- same for the various object properties/V-Ray properties - will all instances have the same options or there will be variation

Best regards,
Yavor

[tysonibele]

Hi Yavor,

1) I allow users to specify custom objects, but I also have a list of preset objects that they can choose from. However, for the sake of discussion, let's say I limit users' choices to a single node that will be instanced.
2) By default all instances will get the same material, but I also allow users to choose override materials for groups of instances. But let's say users will only be able to assign the same material to all instances.
3) All instances get the same object properties.

[yavor.rubenov]

The full VRenderInstance for each instance is probably the easiest way for custom objects - it will handle regular geometry but also special ones like VRayProxy, hairs etc.. And also you can easily do different materials or properties per instance. But as you've seen - it requires quite some memory..

For the simpler case where you have some fixed preset geometry instanced many times you can go with only one custom instance and some special geometry in it. Something similar is done in the vraymeshgeom sample in the SDK. The sample creates some spheres at random locations. Internally it uses only one custom VRenderInstance.

By the way - if you prefer we can continue by email

[tysonibele]

Hey Yavor,

I haven't studied the VrayMeshGeom sample project too closely yet in order to figure out what it's doing, but it still seems to have pretty high memory usage. For example, if I set num spheres to 1 million, radius to 0.1 and num segments to 100, memory usage during a render (with default Vray settings, except GI is disabled and render mode is set to bucket) peaks at over 80 gigs (before I cancelled the render to avoid stalling my machine). It also only got through about 2 buckets in 5 minutes. So it rendered extremely slowly.

The same settings on a VrayInstancer object (1 million spheres with 100 segments) has a peak RAM usage of 8 gigs and the whole frame rendered in 5 seconds (after about 5 minutes of 'Preparing Instances'). So it seems like in VRayMeshGeom, it's not actually instancing the spheres...merely generating them procedurally on the fly and combining them into a giant mesh at rendertime? I'm not sure....

For reference, the same settings on a Thinkbox Frost object (1 million spheres with 100 segments set to render with Vray Instancing turned on) renders in 6 seconds with peak RAM usage of 2 gigs (their 'preparing instancs' phase takes about a second), and in my own plugin the same settings render in 4 minutes with a peak RAM usage of 8 gigs (my plugin uses a method identical to the VRayInstancer, with some additional multithreading in a couple of places).

So in summary:

1 million spheres, .1 radius, 100 segments:

• VRayMeshGeom: 80+ gigs of RAM, potentially hours to render based on the number of buckets that completed in 5+ minutes.
• VRayInstancer: 8 gigs of RAM, 5 minutes to render.
• My plugin: 8 gigs of RAM, 4 minutes to render.
• Frost: 2 gigs of RAM, 6 seconds to render.

Obviously Frost's code is not public so we can't know what they're doing differently....but surely there must be a faster way to go about creating instances than either the VRayMeshGeom or VRayInstancer way of doing things? Either that or the Frost devs managed to implement time travel...

For some further information, I noticed that in my own plugin, the "compiling geometry" phase of instance preparation takes a long time (this is where compileGeometry is called on all the instances)....but why is that phase even necessary for each instance? Shouldn't they all be using the same geometry? The same phase exists in the VRayInstancer plugin (instance.cpp, line 364) and is probably contributing to its slow speed as well. Could you shed some light on why compileGeometry needs to be called on all instances?

[yavor.rubenov]

Hello,

You are right about the VrayMeshGeom plugin not really instancing the geometry - it creates each sphere separately. I'll play a bit with the sample to get it to use instancing and will share it.

As for the compiling geometry phase in the VRayInstancer plugin. The VRayInstancer has an option to instance different geometry for every particle of the source. Also it can instance various custom geometries like the VRayProxy etc.. To handle those custom stuff and not have to write a lot of code VRayInstancer creates a full VRenderInstance for each particle and calls all the required methods (compileGeometry). But if you need millions of simpler geometries - then there are better options

[tysonibele]

Thanks Yavor, I'm looking forward to it!

[yavor.rubenov]

I've made some quick modifications to the code to create only one real sphere mesh and the rest are instances of it with some different transform.
Also a thing I didn't notice initially in the example - for some variance it creates a Multi/Sub-Object material with different material for each sphere - for a few spheres that looks nice but for a million it takes a lot of time and consumes ~10 gigabytes of memory
With those changes now the rendering takes a lot less memory - close to the Frost figure.
Rendering is also much faster but I'll check a few more things to see if can easily be optimized

vraymeshgeom.zip

I'm attaching the original source and the changed version - don't know which Max and V-Ray are you using so that I can also attached a compiled version..