Backburner render, proxies disappear randomly

[dreidesign]

Hey guys,

we got a weird problem rendering some animations via backburner. VRay proxy files, are disappearing on some machines. The strange thing is, that its in one shot 1 or 2 machines which got this error. But in another shot its a different machine.
I thought it is somekind of network issue, but xrefs and textures are getting loaded normally without any issues.

Sadly I cant really reproduce the issue and therefor dont really know a way how to troubleshoot the problem. Does maybe anyone got a hint what could be wrong?

The machines all using Win10, Max 2019, VRay 4.2

[Martin.Minev]

Hello,

We had sent you a reply to your ticket regarding the same issue. Would it be possible to send us there the scene with the assets, so we can test it here in our environment.

[dreidesign]

The scene is quite big with all assets. I will try to get a scene ready, but I have the feeling that this is something that cant be reproduced on your site.
We got some other projects rendering, and they all got the same issue. What I observed is that vray proxies which are scattered via forest pack, are rendering normal. Its just happening to "normal" vray proxy files.

I also have the feeling that its some kind of "memory leak thing". In the first animation the frame render times were about 10mins, and the proxies start to randomly disappear around 10-20 frames rendered. Yesterday I rendered a more heavy scene, which take around 1 hour per frame, and there the proxy disappeard after 2 frames.

[pg1]

Hey,
Had a similar thread https://forums.chaosgroup.com/forum/...y-falling-over
I will stop posting there and will post here.

We are running into some issues using vray 3.6.002 deadline 9 and 3ds2016 where proxy are randomly falling over and it is killing us.
A machine will render 5 frames correctly and then drop the proxy for the next 10 frames.

I believe the issue is less of a vray problem and more to do with our network but am really not sure at this point hoping to get some insight or help if anyone has any recommendations.
The proxies are stored on a qnap centralised server location.
The weird thing is that the frame will still render any textures that are in the same location but the vray proxies will fail to render.
Is there some special reason Vray would do this? i.e. treat proxies and how they are loaded differently?

Is it possible to get Vray to fail to render i.e. generate an error if the vray proxy is not linked succesfully?




I have also had one of our software devs start to look at the problem, see below.

We are experiencing an intermittent issue where Vray Proxies are disappearing in seemingly random frames when frames are submitted across the our farm.
We're observing the same machine creating successful and unsuccessful frames in a single job.

See attached logs for Frames 195 and Frames 197, as well as the opening section of the Vray Log.
Also attached are the output renders. The top 3 text objects in the image are vray proxies, that have been mapped either using a mapped drive (mapped text) unc path (unc text) or the proxy is local on each render node (local) the mesh text is in the scene

Frame 195 renders first, and has a successful output.
Frame 197 renders directly after in the vray log, and the UNC text is missing.

I've performed a Winmerge diffs of logs and they are identical, except for the times being rendered and the time taken to render each frame.
We can't understand why this text would appear in one render, but disappear in the subsequent frame when rendered on the same machine in succession.
Can you provide any insights as to what happens behind the scenes with Vray during a render like this? Could it be that the UNC path to the Vrayproxy is inaccessible, causing the mesh to disappear?

Additionally, is there anywhere else we should be checking for log errors? We can't see anything in Deadline or 3DS max that would indicate an issue.

Thanks

[pg1]

The scenes are large and take up a lot of ram.
Does vray put the proxies out to paging file?
Do you need to enable bitmap paging for that to work?

[dreidesign]

Ok, this is quite weird. I thought that this is related to like Vray Next and Max 2019 with some specific system configurations. Because we didnt had this issues when using vray 3 with Max 2017 or Max 2019.
The things we changed since this time is:
- Switch to Vray Next,
- Upgrade to 10G network
- Newer qnap nas with 10G support

So when reading your issues, this has to be like a windows or network issue. Are you using Win10 on your machines?

[Martin.Minev]

We have tested a simple scene having a V-Ray Proxy stored on a UNC path and a V-Ray Proxy stored on mapped drive. We rendered the scene using Deadline and several render nodes and all the proxies render as expected.
Latest official V-Ray Next, Windows 10, 10G network and 3ds max 2019 used.
If the mapped drive and the UNC path are at the same location and the file was unassessable, the V-Ray Proxy object should occur in both cases. The page file will be used, if there is not enough RAM, regardless what the RAM is used for.
pg1 Could you also try leaving only the Text VRay Proxy objects and render like that. If the issue is still reproducible, could you send us the scene, so we can reproduce your issue in our environment. If it is not, we will need the whole scene.
dreidesign If you could also send us a scene it will help, since I could not reproduce the issue on a simple scene and it could differ from the issue pg1 is having.

[dreidesign]

I got the file zipped (2.7g) and uploading it right now. Will send you an email with the download link.

[pg1]

Hey dreiddesign,
Might be useful to understand what similarity we have in network structure and plugins.
Besides deadline, vray and 3dsmax 2016, we use a fairly extensive script/plugin called renderpass manager to handle our batch rendering.
Do you guys use it?

We also have qnap storage. I don't have the specs to hand but will get them. I suspect it maybe something to do with the storage getting too many requests, the disks being too slow to hand out the file or the storage somehow locking the file at random time frames.
Weirdly it only seems to affect .vrmesh files textures on the same drive render fine.

[pg1]

Mapping the proxy to UNC path seems to be more stable although I am still finding some issues with UNC pathing also.

Using check for missing files in the system panel will only crash the render if the file is missing when the max file opens, not when the render starts.
If max opens and the proxy is there and then renders a few frames at which point the mapped drive or nas drive falls over, vray doesn't check between frames and will assume that everything is still there when in fact it isnt.

[dreidesign]

We using pretty standard stuff: Backburner, ForestPack, Railclone, PSD Manager, Multitexture, Bercontile. All assets are linked to UNC paths.

[Macker]

Just spotted this thread; I'm experiencing the exact same issue (but without backburner/deadline): https://forums.chaosgroup.com/forum/...roxies-missing

[pg1]

Hey Macker,
Can I suggest you re-post your issue here and we can consolidate all the requests together.
Can you post what your server hardware is? Both myself and Dreiddesign are using Qnap's
Maybe it is an issue with Qnap's themselves.
We have a sync process that is syncing files between 1 qnap and another and Im wondering if this process is somehow interfering with the render.

Like you machines are rendering correctly at first and at some point they lose access to only vray proxies. I have done extensive testing also cannot figure it out.

Remapping the files from mapped drives to use unc seems to have made the issues a bit better but hasnt fixed it, although as the problem is intermittent they maybe coincidence.

I can suggest some short term fixes but cannot understand the real issue.
This is only useful in scenes where proxies are not needed for memory.

Go to the selection filter (see attached image)
In the pop down menu select combos.
This will bring up another dialogue box. In the bottom left is class id (very useful

Scroll down to Vray proxy and click add.
This will add vray proxys to the selection filter.
Hit Ctrl A and in the bottom left corner of max where the little maxscript listener window is type
$.display = 4
This will change all proxies to display as mesh. Then right click and convert to edit mesh/poly. This will make all the vray proxies into regular mesh and they will render correctly.
If you need the meshes to remain instanced then instead of convert to edit mesh, put an edit poly on top and right click collapse to, will keep instancing.

While they will sort it, it doesn't resolve the underlying issue.

In our case the scenes are too large and we have to use proxies.
Our solution has been to write a script that will copy selected files locally and then repoint.

So we copy all proxies to each machine user/public/documents
This will work and confirms the issues is definitely network/server related but is a bad idea overall as you now have lots of files to manage.

[Macker]

Hi PG1,

Helpful post, and we are also on a QNAP server - so that's interesting. Curious though that this problem has only arisen now, and not years ago (we've been on a QNAP for a long, long time).

Having all the meshes in our scene just isn't possible because as you said the files are huge.

If someone from ChaosGroup could weigh in on this, that would be super helpful: I am planning on rendering this out on the VRayCloud and am rapidly running out of time to do so - could someone confirm that my credits will be refunded if the proxies go missing?

Also if it is a server issue why don't all of the assets go missing? Everything we are rendering is on this QNAP yet the rest of it renders just fine.


[Edit] Have just spoken with one of our IT guys who says we had an issue some time ago whereby our backup QNAP wasn't talking to the main storage QNAP - and now this has been fixed so they are working as they should; could this be the problem? Proxies going missing when the server is backing up?

[Macker]

Anyone from chaosgroup? vlado ^Lele^ ?