Instancing workflow? Performance issues.

[Monkeybrother]

I've been struggling with getting good performance from Houdini and VRay and now, after months of learning Houdini (not just the scattering part, though), I'm wondering if I'm missing something fundamental. In Max, I can render fairly heavy scenes using VRay, but Houdini/VRay just fills up 64GB of ram, freezes and crashes after a few thousand objects. It occurred to me that maybe VRay in Houdini doesn't recognize packed geos as instances? If I have a bunch of packed geos, what is the correct workflow to get VRay to load them as instances? I'm scattering points and "instancing" them in a for loop, nothing weird. I have googled this and haven't found any straightforward answers.

Or, if VRay actually does recognize packed geo as instances, why is it so heavy/crashing, what am I missing?

Thanks.

[david.anastacio]

I've been struggling with getting good performance from Houdini and VRay and now, after months of learning Houdini (not just the scattering part, though), I'm wondering if I'm missing something fundamental. In Max, I can render fairly heavy scenes using VRay, but Houdini/VRay just fills up 64GB of ram, freezes and crashes after a few thousand objects. It occurred to me that maybe VRay in Houdini doesn't recognize packed geos as instances? If I have a bunch of packed geos, what is the correct workflow to get VRay to load them as instances? I'm scattering points and "instancing" them in a for loop, nothing weird. I have googled this and haven't found any straightforward answers.

Or, if VRay actually does recognize packed geo as instances, why is it so heavy/crashing, what am I missing?

Thanks.

Hi Monkeybrother,

V-Ray has no problem recognising packed instances, I remember setting a test scene with 100000 trees (1 instanced tree mesh) and I took only 4gb of ram to render. We usually use the instance node for this, also works with regular copy to points methods.
Are you by any chance using displacement? If yes, check this post: https://forums.chaosgroup.com/forum/...en-subdiv-gone

Can you post an example scene of your scene that goes crazy with the ram?

Best,
david.

[Monkeybrother]

Hi Monkeybrother,

V-Ray has no problem recognising packed instances, I remember setting a test scene with 100000 trees (1 instanced tree mesh) and I took only 4gb of ram to render. We usually use the instance node for this, also works with regular copy to points methods.
Are you by any chance using displacement? If yes, check this post: https://forums.chaosgroup.com/forum/...en-subdiv-gone

Can you post an example scene of your scene that goes crazy with the ram?

Best,
david.

Thanks for answering. So, it wasn't displacement, but you did put me on the right path. I had transferred a lot of assets from Max to Houdini using VRMesh, which I then unpacked, edited materials and repacked. It turns out that something in the materials that VRay in Houdini creates ("unpack material" in the VRMesh node) was doing something weird. When I try to render a scatter with objects coming from a VRMesh, I got "evaluating python" -> ram fills up -> crash -> repeat -> think about trying octane -> buying a one month octane license -> suddenly solve the problem.
When I imported megascans with octane materials from quixel bridge, the material was so much cleaner and simpler, and on a hunch I tried rebuilding the vray materials by hand and then everything started to work. So, now I just have to redo a few hundred materials, I guess.

Thanks, you pointed me in the right direction.

[glacierise]

Yeah unfortunately automatic material translations across platforms are far from perfect. We end up recreating materials manually.

[bdancer]

> It turns out that something in the materials that VRay in Houdini creates ("unpack material" in the VRMesh node) was doing something weird.

I'm not sure what VRMesh node is because we don't have such... Do you mean VRayScene?
I'm also not sure what "unpack material" could "weirdly" do with materials up to "evaluating python", because all it does is creating VOP nodes...
Is there any change to look at the vrscene that you are trying to import and causing crashes?

[Monkeybrother]

> It turns out that something in the materials that VRay in Houdini creates ("unpack material" in the VRMesh node) was doing something weird.

I'm not sure what VRMesh node is because we don't have such... Do you mean VRayScene?
I'm also not sure what "unpack material" could "weirdly" do with materials up to "evaluating python", because all it does is creating VOP nodes...
Is there any change to look at the vrscene that you are trying to import and causing crashes?

Yes, VRayScene of course, it was late. I'm new to Houdini so I have no idea what could be wrong, all I know is that if I recreate clean materials by hand, the scenes render like they should. I can now render several hundred thousand instances without a problem, where before it would choke with a few thousand. No sign of "evaluating python" since I remade the materials. I'm sorry I can't send you any files, they are all work stuff and commercial assets, but it seems to happen with all meshes I've imported from Max via VRayScene. Max 2020/Vray 5, Houdini 18.5/Vray 5.

[david.anastacio]

Hi again Monkeybrother,

Thanks, you pointed me in the right direction.

Glad I could help.

Yes, VRayScene of course, it was late. I'm new to Houdini so I have no idea what could be wrong, all I know is that if I recreate clean materials by hand, the scenes render like they should. I can now render several hundred thousand instances without a problem, where before it would choke with a few thousand. No sign of "evaluating python" since I remade the materials. I'm sorry I can't send you any files, they are all work stuff and commercial assets, but it seems to happen with all meshes I've imported from Max via VRayScene. Max 2020/Vray 5, Houdini 18.5/Vray 5.

I also ran into issues with vrscenes, this might be related to this: https://forums.chaosgroup.com/forum/...71#post1094571

Best,
David.

[glacierise]

David, are you guys using vrscenes as part of lighting assemblies? We're thinking about it in regard of environment builds that have many instances, if we write it in a vrscene once for the instancing setup, we can avoid writing the points in the lighting vrscene at every render and every frame. But I haven't figured a good way to override shaders in these vrscenes yet, basically to assign shaders to objects inside the vrscene in a rule based way.

[david.anastacio]

David, are you guys using vrscenes as part of lighting assemblies? We're thinking about it in regard of environment builds that have many instances, if we write it in a vrscene once for the instancing setup, we can avoid writing the points in the lighting vrscene at every render and every frame. But I haven't figured a good way to override shaders in these vrscenes yet, basically to assign shaders to objects inside the vrscene in a rule based way.

Hi Hristo,

We are using vrscenes just for transferring some old hard surface assets from 3dsMax to Houdini.
With the point instancing everything goes to the render vrscenes, but we very rarely make render scenes per frame, instead we generate for the whole frame sequence, so for static instancing the points are only written once.
That technique sounds interesting of caching the full instance setup to an intermediate vrscenes.

Best,
David.

[glacierise]

Good to know - when we made the call for single frame per scene, it was because of parallel export, more reliability in case some frame breaks, smaller files. When all points on all frames get dumped into the vrscene, that's quite a wait. And you render many times, dumping all the points on all renders is making things slow and bloated. Doesn't look like there's many practical ways forward besides keeping the points inside the alembic, which would be best, or if nested vrscenes are used - options to override shapes and shaders there, which doesn't really exist now.

[glacierise]

We often do destruction shots where debris are simulated as points or simple shapes, and ideally they get shapes overriden in lighting - that's our main scenario for this many animated alembic points.

[david.anastacio]

We often do destruction shots where debris are simulated as points or simple shapes, and ideally they get shapes overriden in lighting - that's our main scenario for this many animated alembic points.

We are in a different situation then, lighting scenes are usually composed of hero assets made of static alembics, deforming alembics, some volumes and every so often some points. So for the majority of render scene generation (upwards of 95% I would say) are sequence based, as they take a single engine license and they only have to start hython once, so a 100-200 frame sequence is like 2-10 minutes.
For per-frame render scenes (we really try to avoid these) they will cook using multiple engine licenses and each will start hython and that alone adds over an hour on a 100 frame sequence (hython takes a while to load due to pipeline).

When it comes to issues from having all frames in 1 render scene, I would say that we are stress free on that since almost the beginning I think, issues on the render farm are extremely rare (this is limited to our usage scenario, were we are like hawks making sure everything is only linked into the vrscenes), the only thing that comes to mind is still the volume cooking time when generating vrscenes.

Best,
David.

[glacierise]

We keep everything in the alembics too and that's why the instancing points are such a pain - they need to go to the vrscene now. Can be swallowed with static points but too much of our stuff is animated With geometry in the alembic VRScene bake time is really short, we do bursts of 20 per task which are done in a few seconds, unless we have the volume issue you mentioned too. Keeping the instance points in the alembics would be the solution.

[david.anastacio]

We keep everything in the alembics too and that's why the instancing points are such a pain - they need to go to the vrscene now. Can be swallowed with static points but too much of our stuff is animated With geometry in the alembic VRScene bake time is really short, we do bursts of 20 per task which are done in a few seconds, unless we have the volume issue you mentioned too. Keeping the instance points in the alembics would be the solution.

Points in the Alembics would be ideal yes, we talked about this before. But Andrei mentioned the issues with that, they still need to put the info (material assignment) of each instance somewhere, plus for the copy to points or instance node to work that alembic needs to be unpacked in Houdini which is the start of our problems....

[Dubbie99]

Thanks for the great thread guys. It’s really interesting to read about how other studios are dealing with this. I think it would be good if Chaos could have a long and detailed manual page on the topic, as it is very important and takes a huge amount of work to figure out for yourself.