Backburner render, proxies disappear randomly

[dreidesign]

Hey guys,

we got a weird problem rendering some animations via backburner. VRay proxy files, are disappearing on some machines. The strange thing is, that its in one shot 1 or 2 machines which got this error. But in another shot its a different machine.
I thought it is somekind of network issue, but xrefs and textures are getting loaded normally without any issues.

Sadly I cant really reproduce the issue and therefor dont really know a way how to troubleshoot the problem. Does maybe anyone got a hint what could be wrong?

The machines all using Win10, Max 2019, VRay 4.2

[Martin.Minev]

Hello,

We had sent you a reply to your ticket regarding the same issue. Would it be possible to send us there the scene with the assets, so we can test it here in our environment.

[dreidesign]

The scene is quite big with all assets. I will try to get a scene ready, but I have the feeling that this is something that cant be reproduced on your site.
We got some other projects rendering, and they all got the same issue. What I observed is that vray proxies which are scattered via forest pack, are rendering normal. Its just happening to “normal” vray proxy files.

I also have the feeling that its some kind of “memory leak thing”. In the first animation the frame render times were about 10mins, and the proxies start to randomly disappear around 10-20 frames rendered. Yesterday I rendered a more heavy scene, which take around 1 hour per frame, and there the proxy disappeard after 2 frames.

[pg1]

Hey,
Had a similar thread Vray proxy falling over - General - Chaos Forum
I will stop posting there and will post here.

We are running into some issues using vray 3.6.002 deadline 9 and 3ds2016 where proxy are randomly falling over and it is killing us.
A machine will render 5 frames correctly and then drop the proxy for the next 10 frames.

I believe the issue is less of a vray problem and more to do with our network but am really not sure at this point hoping to get some insight or help if anyone has any recommendations.
The proxies are stored on a qnap centralised server location.
The weird thing is that the frame will still render any textures that are in the same location but the vray proxies will fail to render.
Is there some special reason Vray would do this? i.e. treat proxies and how they are loaded differently?

Is it possible to get Vray to fail to render i.e. generate an error if the vray proxy is not linked succesfully?

I have also had one of our software devs start to look at the problem, see below.

We are experiencing an intermittent issue where Vray Proxies are disappearing in seemingly random frames when frames are submitted across the our farm.
We’re observing the same machine creating successful and unsuccessful frames in a single job.

See attached logs for Frames 195 and Frames 197, as well as the opening section of the Vray Log.
Also attached are the output renders. The top 3 text objects in the image are vray proxies, that have been mapped either using a mapped drive (mapped text) unc path (unc text) or the proxy is local on each render node (local) the mesh text is in the scene

Frame 195 renders first, and has a successful output.
Frame 197 renders directly after in the vray log, and the UNC text is missing.

I’ve performed a Winmerge diffs of logs and they are identical, except for the times being rendered and the time taken to render each frame.
We can’t understand why this text would appear in one render, but disappear in the subsequent frame when rendered on the same machine in succession.
Can you provide any insights as to what happens behind the scenes with Vray during a render like this? Could it be that the UNC path to the Vrayproxy is inaccessible, causing the mesh to disappear?

Additionally, is there anywhere else we should be checking for log errors? We can’t see anything in Deadline or 3DS max that would indicate an issue.

Thanks
VraylogOpen.txt (107 KB)
VraylogFrame197.txt (522 KB)
VraylogFrame195.txt (522 KB)

[pg1]

The scenes are large and take up a lot of ram.
Does vray put the proxies out to paging file?
Do you need to enable bitmap paging for that to work?

[dreidesign]

Ok, this is quite weird. I thought that this is related to like Vray Next and Max 2019 with some specific system configurations. Because we didnt had this issues when using vray 3 with Max 2017 or Max 2019.
The things we changed since this time is:
- Switch to Vray Next,
- Upgrade to 10G network
- Newer qnap nas with 10G support

So when reading your issues, this has to be like a windows or network issue. Are you using Win10 on your machines?

[Martin.Minev]

We have tested a simple scene having a V-Ray Proxy stored on a UNC path and a V-Ray Proxy stored on mapped drive. We rendered the scene using Deadline and several render nodes and all the proxies render as expected.
Latest official V-Ray Next, Windows 10, 10G network and 3ds max 2019 used.
If the mapped drive and the UNC path are at the same location and the file was unassessable, the V-Ray Proxy object should occur in both cases. The page file will be used, if there is not enough RAM, regardless what the RAM is used for.
pg1 Could you also try leaving only the Text VRay Proxy objects and render like that. If the issue is still reproducible, could you send us the scene, so we can reproduce your issue in our environment. If it is not, we will need the whole scene.
@dreidesign If you could also send us a scene it will help, since I could not reproduce the issue on a simple scene and it could differ from the issue pg1 is having.

[dreidesign]

I got the file zipped (2.7g) and uploading it right now. Will send you an email with the download link.

[pg1]

Hey dreiddesign,
Might be useful to understand what similarity we have in network structure and plugins.
Besides deadline, vray and 3dsmax 2016, we use a fairly extensive script/plugin called renderpass manager to handle our batch rendering.
Do you guys use it?

We also have qnap storage. I don’t have the specs to hand but will get them. I suspect it maybe something to do with the storage getting too many requests, the disks being too slow to hand out the file or the storage somehow locking the file at random time frames.
Weirdly it only seems to affect .vrmesh files textures on the same drive render fine.

[pg1]

Mapping the proxy to UNC path seems to be more stable although I am still finding some issues with UNC pathing also.

Using check for missing files in the system panel will only crash the render if the file is missing when the max file opens, not when the render starts.
If max opens and the proxy is there and then renders a few frames at which point the mapped drive or nas drive falls over, vray doesn’t check between frames and will assume that everything is still there when in fact it isnt.

[dreidesign]

We using pretty standard stuff: Backburner, ForestPack, Railclone, PSD Manager, Multitexture, Bercontile. All assets are linked to UNC paths.

[Macker]

Just spotted this thread; I’m experiencing the exact same issue (but without backburner/deadline): Distributed rendering proxies missing - Issues - Chaos Forum

[pg1]

Hey Macker,
Can I suggest you re-post your issue here and we can consolidate all the requests together.
Can you post what your server hardware is? Both myself and Dreiddesign are using Qnap’s
Maybe it is an issue with Qnap’s themselves.
We have a sync process that is syncing files between 1 qnap and another and Im wondering if this process is somehow interfering with the render.

Like you machines are rendering correctly at first and at some point they lose access to only vray proxies. I have done extensive testing also cannot figure it out.

Remapping the files from mapped drives to use unc seems to have made the issues a bit better but hasnt fixed it, although as the problem is intermittent they maybe coincidence.

I can suggest some short term fixes but cannot understand the real issue.
This is only useful in scenes where proxies are not needed for memory.

Go to the selection filter (see attached image)
In the pop down menu select combos.
This will bring up another dialogue box. In the bottom left is class id (very useful

Scroll down to Vray proxy and click add.
This will add vray proxys to the selection filter.
Hit Ctrl A and in the bottom left corner of max where the little maxscript listener window is type
$.display = 4
This will change all proxies to display as mesh. Then right click and convert to edit mesh/poly. This will make all the vray proxies into regular mesh and they will render correctly.
If you need the meshes to remain instanced then instead of convert to edit mesh, put an edit poly on top and right click collapse to, will keep instancing.

While they will sort it, it doesn’t resolve the underlying issue.

In our case the scenes are too large and we have to use proxies.
Our solution has been to write a script that will copy selected files locally and then repoint.
​​​​​​​
So we copy all proxies to each machine user/public/documents
This will work and confirms the issues is definitely network/server related but is a bad idea overall as you now have lots of files to manage.

[Macker]

Hi PG1,

Helpful post, and we are also on a QNAP server - so that’s interesting. Curious though that this problem has only arisen now, and not years ago (we’ve been on a QNAP for a long, long time).

Having all the meshes in our scene just isn’t possible because as you said the files are huge.

If someone from ChaosGroup could weigh in on this, that would be super helpful: I am planning on rendering this out on the VRayCloud and am rapidly running out of time to do so - could someone confirm that my credits will be refunded if the proxies go missing?

Also if it is a server issue why don’t all of the assets go missing? Everything we are rendering is on this QNAP yet the rest of it renders just fine.

Have just spoken with one of our IT guys who says we had an issue some time ago whereby our backup QNAP wasn’t talking to the main storage QNAP - and now this has been fixed so they are working as they should; could this be the problem? Proxies going missing when the server is backing up?

[Macker]

Anyone from chaosgroup? @vlado @_Lele ?

[dreidesign]

I’m in contact with the chaos group support and they are investigating the scene I sent them. I’ve got a question to you guys, as it seems one thing we got in common is the qnap. Do you know which kind of configuration you are running? We got a RAID5 of hdds and an RAID10 SSD cache for reads and writes. I had the idea that its maybe related to the ssd cache. Are you guys using it?

[jhall.nz]

Always a sigh of relief realizing you’re not the only one. In return I may have a sigh of relief for you all with a possible solution.

So here’s my scenario which I believe is identical to all of yours.

Long Version:

Send a Vray animation via Backburner to our 20+, Win 10 & Win 7 node render farm, check some completed frames, everything’s working perfectly, head home and sleep soundly. Next day, check the frames and…all the proxy’s are gone after about 4/5 successful frames from each node, “What?”. First time I’ve ever seen anything like this before, “Why just the proxy’s, and not the textures in the same network location?”.

Now this is an impressively heavy scene, massive max files, a tonne of xref’s, hundreds of proxy’s, so my mind goes to optimization, maybe the nodes ran low on RAM. Spend some time optimizing, get things lighter and moving quicker, send it off again, few success frames, head home, sleep less soundly. Next morning, exact same problem. “OK, lets start checking things off the list”. Spend some time making sure all software versions are the same, they are. All plugins are installed correctly, they are. All network locations are accessible, they are.

“OK, now it’s time to start ruling things out”. Convert everything to UNC paths, no change same problem. Collapse proxy’s to mesh, scene crashes way too heavy that’s why they’re proxy’s in the first place, undo…undo…undo. Copy all proxy’s locally, success everything works perfectly, this is however a work around not a solution. So with network being the possible issue I go back and check which computers had the issue and what they have in common, Windows 10!

Now this is where I leave my area of expertise and our IT Supervisor steps in. I lay out the problem to him and after “turning it off and on again?”, he does some serious dark web digging and comes up with the following -

“In your GPO try to replace the action on the faulty network drive. In my case, I was using the action “Update”, I changed it for “Replace” and it works.” Now, I have no idea what that meant but who cares, it worked, onward with the renders!

So it seems Windows 10 rolled out an auto update that broke the mapped drives system. From this I’m assuming proxies are loaded every frame, where as textures load once and live in RAM. So after a few frames, Windows 10 decides it’s had enough of mapped drives disconnecting them, and all of a sudden Vray can’t see any proxy’s anymore and doesn’t render them.

Long post, but it was a long process.

Short Version:

Proxy’s fail to render after a few frames on Windows 10 machines, do the following: In your GPO try to replace the action on the faulty network drive. In my case, I was using the action “Update”, I changed it for “Replace” and it works.

Further reading here: https://social.technet.microsoft.com/Forums/en-US/f23947ae-88b2-4049-964c-9ee2da79f33f/mapped-drives-showing-disconnect-after-feature-1809-update?forum=win10itprosetup

Hope that helps folks.

[dreidesign]

Thanks for your detailed explanation. Which kind of storage are you using if I may ask? The fix you mentioned does only apply when you are using mapped drives, but we are only working with UNC paths. So that GPO change wouldnt do a thing, if I’m not mistaken.

[jhall.nz]

As I mentioned a little out of my league on the network end, so I’m passing on the info from our IT legend.

“We use mapped drives for our assets/outputs and that solution worked for us as per the tech net thread. Our storage back-end is a Synology NAS (all flash). Try upgrading to the latest build of windows.”

“It’s a known issue in the build we’re using, but if you update to Build 1903, it fixes it…allegedly”.

Hope that helps.

[pg1]

Thanks jhall.nz will get onto IT and give it a try.
Although as dreiddesign mentioned this would only work with mapped drives, no?

The problem has been there for at least 6 months. I have told the production team to not use proxies as a result.
We had to use them on a recent project and ran into problems again.

I ran more tests. You can see attached that I clone the text UNC.vrmesh and linked that proxy using mapped drive (T: ), unc (\\ucqnap03) path and unc (\\ucqnap03.uc.local) path

Rendered fine for 300 plus frames and on frame 335 it dropped the mesh that was mapped and the mesh that was using \\ucqnap03.uc.local but not \\ucqnap03
Am at the edge of my IT knowledge also but this makes no sense to me as I understand \\ucqnap03.uc.local and \\ucqnap03 to be exactly the same thing, right?

It seems to happen less when you use unc path over mapped drives.

Hopefully the group policy change will fix it. Will check back in if it does.