Backburner render, proxies disappear randomly

[Macker]

Update on our situation:

We shifted proxies to a windows PC and created a drive share which we then mapped the proxies to - same issue, which rules out the QNAP - almost.

We're noticing all of this renders absolutely fine during the day, and only has issues at night, leading me to believe it could be to do with network bandwidth or server backups... Although it's never at the same time every night.

Anyone else have the same pattern?

[Morbid Angel]

Hi Macker, I had used two qNap servers before and never had the problem with proxies dropping but I used mapped drives not unc, but that really should not matter. What model qnap is it? Unless its some heavy duty industrial grade model, it is quite possible that it cannot handle the traffic. One of the solutions is to sync proxies locally but it can be a pain, a better way to go is to switch to a new nas server like synology (which is what I use)

[Macker]

Hi Dmitry,

Yeah it's a pretty big unit - the TS-1635AX, with about 49TB of storage, being backed up to a slightly older QNAP.

I'm leaning towards it being to do with network issues after hours (and again presuming it's something to do with the backup process that happens purely because I can't think of anything else that would be using that much bandwidth?)

That said, our IT guy says it's unlikely to be a bandwidth problem because:

"In an IP network each packet of data is requested, if a failure occurs it will re send, the application will wait for the resource and then continue once it arrives, so in theory the network shouldn't effect a render in that way - providing the render software is not buggy.

Backups do happen over night and do take up lots of resources which could slow down other services but wouldn't stop them."

So, I'm at a loss.

[Edit] Forgot to mention we ran tests over the weekend with all the proxies loaded onto local machines - rendered perfectly.

[Macker]

My colleague was adamant he'd has this problem a long time ago but thought nothing of it because the next day when he hit render (during working hours), everything was fine. We went back and checked the old frames and it was exactly the same issue; all the proxies disappearing. This was back in November 2016.

[pg1]

Hey Macker,
Can confirm that it seems to occur overnight.
That could be because of backup on the qnap and also a lot of our workstations are used in the renderfarm so when everyone goes home the number of computers rendering and therefore accessing the qnaps increases dramatically.
I have given up testing the qnaps and moved the proxies to a windows based server so far so good.
Not the best solution but better than copying files locally.

I would really like to get to the root of the problem as we have 4 large qnap's ,30tbs, in the company that are looking likely to be chucked out and replaced at considerable cost in time and money.

[gadzooks]

Are you using UNC paths? I have had this issue when rendering over the farm, proxies and objects not appearing in my renders. Mapping the paths to UNC fixes it every time. Worth a try

[Martin.Minev]

Hello,

We are working on error checking with V-Ray Proxies and there will be more information about them in the V-Ray log. We will post an update here when we have a build, which could be tested.
If some of you have an opportunity to test the build and please share with us the results.

[jhall.nz]

Quick update from our scenario. My previous post outlined a solution that seemed to work in our studio, as at the time it solved the proxy's disappearing on the job I was rendering. Few weeks on, new project, time to render and guess what rared it's ugly head again, proxy started vanishing again! Now this time I had no time to investigate so I just went straight to copying proxy's locally, annoying but at least a working solution. Our current line of thinking is leaning toward proxy's created in older versions of Max/Vray, but am yet too test this theory.

[Martin.Minev]

Hello,

Could you download the latest stable nightlie build and make some tests using a scene having V-Ray Proxies stored on a network location. There is an improvement included, which will provide more details in the V-Ray Log when a V-Ray proxy is missing.
This could give us more hints about the disappearing V-Ray Proxy behavior. Please tell us how it goes.

Thank you for your feedback.

https://nightlies.chaosgroup.com/mai...e_4.2/20190911

[mak11887]

We are having the same problem!!!!
It started to appear exactly when we switch to a new set of servers and got Forest Pack. We have no idea if it's a problem about Forest Pack or the servers (including the network).

The problem is still persisting.

[jhall.nz]

Greetings Martin.Minev, thanks for your input. In order to correctly test a new nightly build I would need to install on wide sample size of machines (Win 10, 7, workstation, render node), then get a scene rendering for many hours to get the error to maybe occur or maybe not occur. Remember this error occurs after some initial frames have rendered correctly, then seemingly occurs on random machines and at random times of the day. Unfortunately I don't have the time or resources to confidently test this theory.

Mak11887, from my experience it's not Forest Pack causing this issue as I've had it occur on scenes with and without Forest Pack. For the scenes I had with Forest Pack the entire forest scatter would disappear randomly, I'm assuming this is because the proxy is no longer "available" due to this proxy error.

[dreidesign]

Hello,

Could you download the latest stable nightlie build and make some tests using a scene having V-Ray Proxies stored on a network location. There is an improvement included, which will provide more details in the V-Ray Log when a V-Ray proxy is missing.
This could give us more hints about the disappearing V-Ray Proxy behavior. Please tell us how it goes.

Thank you for your feedback.

https://nightlies.chaosgroup.com/mai...e_4.2/20190911

Thx for the nightly. Sadly I cant install the nightly on our production used workstations/rendermachines. Like I've already written, we could eliminate the issue by copying the proxies onto another NAS. But maybe you can implement the more detailed logging into the next stable vray update.

[mx63]

Hi Everyone

Just found this thread.
We are facing the same problem


Has Chaos group found any solution?

Yes, we also use a Qnap, not a low end version. TVS 872 XT, on a 10GbE network cable and 10GbE switch.
We do have a live sync of files happening to another Qnap (which I will now try to disable as someone mentioned that could be the culprit)
We use UNC always.

Our team seems to feel that the first 10s of network renders are ok. (up to 20 or so). And thereafter we get random missing proxies on random servers.

A workaround we are trying now is to send our (for example) 300 frame job, into batches of 20 frames. Since the problem seems to occur only after rendering some frames corrrectly

[dreidesign]

We didnt found a way to fix that QNAP related problem. We did at first hand move the proxies to a different storage device which isnt a QNAP, as a "workaround".
A few months ago we switched completely from QNAP to a self built NAS using "Freenas". No problems so far.

[joaquim_montserrat1]

Hello,

We had the same issue on a QNAP NAS, and didn't have the problem on other WIndows server based storages.

We found the solution to this problem. QNAP has "OPlocks" activated by default on shared folders. Oplocks are supposed to give a up to 30% networks speed improvement, but that seem to create an access queue when multiple clients access the same file at the same moment. I think 3dsmax then timeout, and skips the texture / proxy etc.. That's very obvious on scenes with hundreds of assets, rendered simultaneously by dozens of rendernodes.

On your shared folder, edit the properties and uncheck "Lock Files (Oplocks)".

Disabling the Oplocks instantly solves the problem.

https://docs.qnap.com/operating-syst...1C3ADADD6.html

Note that we have several QNAP machines. On some machines there is a little question mark next to the Oplocks options that recommends to disable the Oplocks if the machine is accessed by multiple users simultaneously.

More reading about this:
https://www.smallnetbuilder.com/nas/...oogle_vignette
https://docs.netapp.com/a/ontap/7-mo...For-7-Mode.pdf

However, if multiple clients are trying to read or write to the same file, then oplocks is a huge performance loser. The problem is that the first client
gets the oplock granted by the server and begins work.
Then up comes the second client. The server sees the possible conflict and sends a Revoke lock operation to the first client and lets both of them know that
oplock requests will not work on this file due to multiple concurrent clients trying to access it. As this point all locking operations are sent to, and handled by, the server.