unlimited amount of proxies; the proxy container

[zoobadoo]

large amounts of proxies are a problem inside max. scripting 100K or more instances of instanced proxy chokes max, even with the new proxy point mode (although it helps).

it will render fine, but opening, editing and saving scenes is a nightmare. sometimes scene's simply won't unload causing jam's inside the renderfarm after rendering the scene nicely. 3dsmax simply can't handle large amounts of individual objects.

having this said, i think it's fundamentily wrong to instance large amounts of proxies around as 3dsmax objects. What you really want is a big container in which you store only the locations (3D transform data) for the instanced proxies. At rendertime this array is used to actually render the proxies.

in the workflow you might call it a 'proxy-container'. instance proxies around as you'd normally do, are script them around. But if you're done simply store them inside the container (the container wil read out all the relevant data form the proxies, and deletes the actual proxies resulting in a leightweight container object). maybe inverse-scatter is a better name. script positions directly inside the container would be cool too...

proxies store large amounts of mesh. proxy containers store large amounts of proxies. makes sense doesn't it?

[re:FORM]

hoooooyeaaaa. That's a nice idea.

[Aldaryn]

Several months ago I posted almost the same idea along with some others considering massive instancing (not thousands, but billions), and the main concept was to handle the positions and transform separately from max (or any 3d application), to consider that as a "proxy" as well, read from disk during rendertime, and with a generic model, so that you could stack multiple "container" layers on top of another to speed the lookup of individual instances.

Back than, Vlado said it was an interesting idea, so I'm still hoping. It is pretty sad, that when it comes to rendering insane amounts of instances it is max what fails to handle it in the first place.

Best regards,
A.

[trick]

I really would like to see this integrated in VRay too.

I use GroundWiz and love it for its way to spray lots of VRay-proxies (from a pool) around. However on a 16 GB system you will have a hard time to render over 500000 proxies; but over lots of projects I hit that border only once !! The alternative is VRayScatter, but even then it's hard to do over 5M instances; its also expensive for only this feature in that it requires you to buy a new license (both workstation and rendernode) each time you need to install it on a new PC. In Lightwave I have a plugin that handles instances volumetrically (HDInstance) and is so able to do instances per area based on weight maps so you can go infinite. This gives endless possibilities !!

I always thought that the problem for this is in Max and not in VRay

[zoobadoo]

Several months ago I posted almost the same idea along with some others considering massive instancing (not thousands, but billions), and the main concept was to handle the positions and transform separately from max (or any 3d application), to consider that as a "proxy" as well, read from disk during rendertime, and with a generic model, so that you could stack multiple "container" layers on top of another to speed the lookup of individual instances.

Back than, Vlado said it was an interesting idea, so I'm still hoping. It is pretty sad, that when it comes to rendering insane amounts of instances it is max what fails to handle it in the first place.

Best regards,
A.

i missed that one... if Vlado thinks it's interesting we might have a chance! any progress on this...?

[zoobadoo]

I use GroundWiz and love it for its way to spray lots of VRay-proxies (from a pool) around. However on a 16 GB system you will have a hard time to render over 500000 proxies; but over lots of projects I hit that border only once !! The alternative is VRayScatter, but even then it's hard to do over 5M instances; its also expensive for only this feature in that it requires you to buy a new license (both workstation and rendernode) each time you need to install it on a new PC.

when it's integrated and accessable through maxscript we can leave the limited scatter and groundviz third party stuff behind.

if this day arrives i'll start a 'how to script unlimited amounts of proxies inside your daily scenes' thread.

[duke2]

Or just add support for Particle flow so you could use particles to control your proxy positions without necessarily even loading the proxies in, and then VRay swaps them out at rendertime. I do understand that Max has problems with large particle counts though - maybe something like Krakatoa where you can have it add extra particles (and therefore proxy positions) at rendertime by interpolating between particles, so you'd have a particle multiplier.

[trick]

when it's integrated and accessable through maxscript we can leave the limited scatter and groundviz third party stuff behind...

GroundWiz is in no way limited, except for the max.amount of instances. Even if Max/VRay could handle unlimited amounts of proxies, it would really take a huge script to be able to put all those instances exactly where you want them with all the variety you need. Of course I would like to see Vue, GroundWiz, VRayScatter, etc integrated in VRay, but realistically I would be happy already if the max.instances limitations were solved.. Generally the big advantage for good 3rd party stuff is that they are actively developed. So it is with VRay and so it is with GroundWiz. That is not something I can say from some stuff integrated in Max. But in the end, all good stuff will be absorbed by AD...

[^Lele^]

The number of objects can be of concern, but then why not proxy out big chunks?
After all, the proxy is a voxel format, and will load only the parts needed, so it's unlikely that it will have to load just every byte of the proxied file before render start, unless some form of heavy multi-bounce GI is used, like photon mapping or LC, and/or the scene is an interior shot, and the standard 3 bounces of brute force end up looking for every part of the proxy.

For an outdoor, with IBL (1 bounce),i'd say it's safe to assume that the proxy will be extremely efficient regardless of its size on disk.

[zoobadoo]

The number of objects can be of concern, but then why not proxy out big chunks?
After all, the proxy is a voxel format, and will load only the parts needed, so it's unlikely that it will have to load just every byte of the proxied file before render start, unless some form of heavy multi-bounce GI is used, like photon mapping or LC, and/or the scene is an interior shot, and the standard 3 bounces of brute force end up looking for every part of the proxy.

For an outdoor, with IBL (1 bounce),i'd say it's safe to assume that the proxy will be extremely efficient regardless of its size on disk.

True. As an example I once made a big max scattered object for hedges and other greenery like flowers on balconys and stuff. It spread out across the whole scene, and killed the LC calculations. My feeling was this happened because the same big proxy was visible thoughout the whole scene. It was heavy to create, and heavy to export.

Making smaller proxies and repeating them is much better for robust GI calculation which we use all the time, also regarding animation. Secondly I think it's more elegant to create smaller proxy variations and have those distributed across the scene.

This discussion might lead to 'Massive Proxy Handling' vs 'Unlimited Proxy Scattering'. Building a large mesh works, but technically it's more a workaround than a solution imho. Think of how much bigger/detailed we could go with a 'vray container workflow', or the less resources it takes in a normal setup! It's easy to translate scripted, scattered or sprayed points, meshes or whatever to a cloud of 3D transforms neatly stored in a container object.

The 'Vray_Proxy_Cloud'.. has a nice ring to it..

almost christmas, any thoughts on this Vlado?

[re:FORM]

I assume what your suggesting with the 'cloud' idea is to have the proxy-container storing all the transforms of the proxies, but then instead of treating the container as a single mesh at render time, to break it up into a specified number of chunks, so ensuring efficient bounding boxes for LC calculations?

Sounds good to me!

[^Lele^]

I'm in the middle of something much similar to what you describe as being an issue.
Just, likely, on a much bigger scale.
I chose to proxy out entire chunks of distributed geometry, versus distributing proxy instances.
The bottleneck is NOT in the proxy being one versus a thoushand meshes stored in it: that's handled correctly by the dynamic memory limit.
The issue is to have them scattered in the first place.
Having a "rendertime" multiplier doesn't help either: Krakatoa is just as fast as max allows it to, and changing seeds without knowing where your stuff will exactly be can work only in peculiar situations (ie. nice, soft, chaotic particle effects).
Sure, a separate file referencing PRS for each proxy might be nice, but what if i have 50 types of proxies to reference? Or 500? What if my base instances get a different noise mod pattern on them, making each different to each other?
I am still of the idea that leveraging the voxel-based storage of data inside the proxy file format is a better way about it.
Also, LC speed issues are only due to 2 causes: a big proxy file to be loaded at the start of the calculation (that's due to disk/network speed), and in case a small dynamic mem limit (if it keeps being slow throughout, using little or no cpu).
So one always has the chance to be slow in opening max (with hundred of thoushand objects) and quicker rendering (if your proxy is small, fits in the dyn mem limit neatly, and is instanced many a time in an identical fashion), or the other way around.
With big proxies, though, and the growing amount of available ram, the performance difference is going to get smaller and smaller at rendertime.
Or so i hope.
I'll surely be able to let you know how it went soon enough...

[zoobadoo]

I assume what your suggesting with the 'cloud' idea is to have the proxy-container storing all the transforms of the proxies, but then instead of treating the container as a single mesh at render time, to break it up into a specified number of chunks, so ensuring efficient bounding boxes for LC calculations?

Sounds good to me!

the cloud simply tells vray where the proxie should be positioned at rendertime. not to actually position them there, because of the limitations of 3dsmax

[zoobadoo]

The issue is to have them scattered in the first place.

exactly, and for a lot of stuff like greenery it's easy to script out the positions. light and fast. i already do so for creating fields of high grasses and such. compare it to scattering proxies in point mode. when scattering with preview ON this will slow down max as usual. But when finished the preview can be OFF, so it's there only at rendertime in a much lighter state using much lesser memory.

Having a "rendertime" multiplier doesn't help either: Krakatoa is just as fast as max allows it to, and changing seeds without knowing where your stuff will exactly be can work only in peculiar situations (ie. nice, soft, chaotic particle effects).

it's not about random seeds or where your proxies are, it's about storing your proxies in a different manner. Doesn't matter how you want to create them, do as you like. But afterward you might replace the scattered instances (consuming lots of max memory, slowing viewports, slowing filesaving/opening) by a singly proxy and a container that has the position's stored inside. this way vray could render them without max interfering.

[zoobadoo]

Sure, a separate file referencing PRS for each proxy might be nice, but what if i have 50 types of proxies to reference? Or 500? What if my base instances get a different noise mod pattern on them, making each different to each other?
I am still of the idea that leveraging the voxel-based storage of data inside the proxy file format is a better way about it.

It's not about a seperate file, the container resides as an helper inside max. so no trouble creating different containers for different proxies.

The basic idea is not to have a different system for creating instances of proxies.The idea is how to store these instances (that's why they are instances, no?) in a more efficient manner.