[V-Ray] Cannot create output image file Issue

[Donald2B]

Over the past few months, I've been running into this strange issue when rendering with the following software:
3DS Max 2020.3
VRay Next Update 3.1
Backburner 2019 Update 5 (Latest version)

So the problem is I'm getting the "[V-Ray] Cannot create output image file" error message. Only this is not due to permissions. The animation will run fine for a random amount of time and then I will start getting this error...which doesn't make sense, because the previous frame(s) rendered out correctly. So I started looking closer at the backburner log files and notice the problem.

When trying to render frame 16 of the animation, VRay is trying to save the output VRIMG file to frame 0001...which is already being rendered on another render node. This is why I'm getting the permission error as the 0001 file is locked while being rendered on another render node.

So why is VRay trying to save frame 16 with a 0001 frame pad? This has been haunting me for the past 4-6 months at random times...and I can't figure out where the problem is coming from. 3DS Max? VRay? Backburner?

I'm using VRay ray image file output within the Frame Buffer settings for setting the file output.
I've included a snippet of the backburner log file showing the frame being rendered as 15 and VRay trying to output frame 0001.vrimg.

Any ideas would be greatly appreciated!
I'm trying NOT to update any software if at all possible due to being in the middle of a large project that has been going for an extended amount of time. I don't want to break anything else, if it can be avoided.)



Backburner Log:
2021/01/14 16:52:53 INF Receiving new job from 172.20.0.12
2021/01/14 16:52:53 INF Job 'D2077_RR_1107F_Shot_05b_Full Color' received and ready
2021/01/14 16:52:53 INF Launching 'Autodesk 3ds Max 2020 (64-bit)'
2021/01/14 16:54:40 INF 3dsmax adapter : Executing command line : C:\Program Files\Autodesk\3ds Max 2020\3dsmax.exe "-_pipe:102828562" -server "C:\Users\MStudio\AppData\Local\backburner\Ser verJ ob\D2077_RR_1107F_Shot_05b_Full Color.max".
2021/01/14 16:54:40 INF [ServerMessage]ProcessStarted processId=30508
2021/01/14 16:54:40 INF 3dsmax adapter : Product version: 3ds Max 2020.3 Update (22.3.0.3165)
2021/01/14 16:54:40 INF 3dsmax adapter : Data collection and use is 'ON'. Change your participation anytime in the Help menu of 3ds Max.
2021/01/14 16:54:40 INF 3dsmax adapter : Autodesk 3ds Max Security Tools is active via configuration settings
2021/01/14 16:54:40 INF 3dsmax adapter : ALC Security Tool 1.0 loaded. (c:\program files\autodesk\3ds max 2020\applicationplugins\securitytools\contents\scr ipts\ALC_SecurityTool.ms)
2021/01/14 16:54:40 INF 3dsmax adapter : CRP Security Tool 1.0 loaded. (c:\program files\autodesk\3ds max 2020\applicationplugins\securitytools\contents\scr ipts\CRP_SecurityTool.ms)
2021/01/14 16:54:40 INF 3dsmax adapter : ADSL Security Tool 1.0 loaded. (c:\program files\autodesk\3ds max 2020\applicationplugins\securitytools\contents\scr ipts\ADSL_SecurityTool.ms)
2021/01/14 16:54:40 INF 3dsmax adapter : Autodesk 22.3 is Ready.
2021/01/14 16:54:40 INF New task assigned: 16
2021/01/14 16:54:44 INF 3dsmax adapter : BEGIN FRAME 16; TASK 17 OF 60; (NO SAVE);
2021/01/14 16:54:44 INF 3dsmax adapter : END FRAME 16; TASK 17 OF 60; (NO SAVE); 0.985s; 0 faces; MEM 4272 MB;
2021/01/14 16:54:44 INF Adapter returned unexpected code -2 for "C:\Program Files\Autodesk\3ds Max 2020\maxadapter.adp.exe" "-o" "NewTask" "-l" "Info" "-j" "C:\Users\MStudio\AppData\Local\backburner\Ser verJ ob\D2077_RR_1107F_Shot_05b_Full Color.xml.details" "-s" "16" "-n" "1" "-u" "mstudio" "-a" "C:\Users\MStudio\AppData\Local\backburner\Ser verJ ob\D2077_RR_1107F_Shot_05b_Full Color.zip" "-d" "utf-8"
2021/01/14 16:54:44 ERR Task error: 3dsmax adapter error : Autodesk 22.3 reported error: [V-Ray] Cannot create output image file "\\172.20.0.13\FRAMES\D2027 RR 1107F Benefits\Frames\05_Section_03\Shot_05\Full_Color\D 2077_RR_1107F_Shot_05b_0001.vrimg": Permission denied (13)

2021/01/14 16:54:44 ERR 3dsmax adapter error : Autodesk 22.3 reported error: [V-Ray] Cannot create output image file "\\172.20.0.13\FRAMES\D2027 RR 1107F Benefits\Frames\05_Section_03\Shot_05\Full_Color\D 2077_RR_1107F_Shot_05b_0001.vrimg": Permission denied (13)

2021/01/14 16:54:44 INF Application is down
2021/01/14 16:54:57 INF 3dsmax adapter : Autodesk 22.3 shutdown.

[hermit.crab]

Does this happen on a simple scene? Are you certain that all slave machines have an identical V-Ray version installed? Check this topic in the Autodesk forums for possible solutions. If the issue persists, do send the scene in question via the contact form so we can take a look. Mention this thread in the message.

[Donald2B]

This only happens when using the VFB area for file output for saving out EXRs / VRIMGs.

It's sporadic, in the sense that not every single animation job sent will have this issue...currently approximately 80% of the jobs fail due to this error.

All workstations and render slaves are identical installations. (We only update once the large jobs are finished, so it can be 8-10 months between updates unless there is catastrophic software issue)

I have reviewed the Autodesk forum thread, but my issue doesn't appear to be a writing permissions issue on the server, as permissions are good. Our issue is when rendering an animation, all 20+ render nodes will try to write to the exact same file name regardless of the animation frame that has been assigned to the render slave

So the first slave that gets frame 1, will render without issue to the filename_0001.vrimg, while the rest of the animation slaves getting frames 2-899 will all try to save their output to filename_0001.vrimg as well. Creating the error writing out the file. Since frame 2 should be filename_0002.vrimg....it's trying to output to filename_0001.vrimg which is already being written by another slave.

Since the number padding is created by the rendering software upon rendering, this is where I'm lost in trying to solve the issue.

Unfortunately, I can't send you this latest project, due to heavy NDAs. Any animation sent to the farm will have this issue. Even a simple sphere animation.

[vanpixelguy]

Did you find a workaround for this issue? We're seeing up to 8 failures per day. It just started happening when we moved to V-Ray Next. We did not see the issue with our prior version (3.60). Since we render our frames starting at 1000, and render across multiple computers simultaneously to the same output folder on a shared network drive. Each computer is rendering a distinct part of the frame range, so at no time should we ever be rendering from 0001 yet we see these errors in the log file. Unfortunately these errors are impacting our downstream processes because these 0001 files are zero-bytes and thus not valid image files.

Questions:

1. is frame 0001 outside the frame range you are rendering at the time? That's the case with us.
2. We're on Windows 10, 64 bits, writing to a shared network drive. Is your environment similar?

Thanks.

[Donald2B]

1. Our frame ranges can vary, but it didn't matter what frame was being rendered...every single machine would try to save to the exact same 0001 frame number. It's as though VFB couldn't pull the frame number information from the current frame being rendered and therefore defaulted to 0001.

2. We are in a Windows environment, yes.
There are 30+ dedicated render nodes running Windows 7 Pro x64
Artist Workstations are running Windows 10 Pro x64
File servers are running Windows Server 2016 x64

Unfortunately, we were in crunch mode trying to deliver a few shots, so I didn't document all the changes made. So it was a shotgun approach of change everything and hopefully something will make this thing work. I don't remember everything but here are a few things I changed:
- converted back to using UNC file conventions for saving files (Even if the drive was already mapped to a drive letter)
- Upgraded VRayNext to version 4.30.02 Build 0001
- Upgraded 3ds Max 2020 to latest service pack (ALWAYS sketchy doing this in the middle of a project...but I was desperate and didn't have a ton of time for messing with this)
- full windows update on all machines (again, sketchy in the middle of a project, NOT recommended under normal circumstance without proper testing first)

For the most part, this has remedied the situation for us. We do still get a few hiccups every now and then...but the farm usually works itself out and doesn't kill a job or leave just a single node running on a long animation.

Once we finish with the last few jobs...we will be upgrading....and hopefully we don't run into this issue again. This one kind of snuck up on me, so that's why I think it was just the version of VRayNext we were using, as we didn't have any problems early on with VRayNext...it was later builds where I think it popped up...but we were doing a bunch of still shots and didn't render out too much animation for a good chunk of time while we were between animation jobs.

I hope that helps...I'll try and look at all my notes when I get into the office in the morning and see if I can find any more details.

[vanpixelguy]

Thank you for the detailed reply.
We are already using UNC paths and are on 4.30.00, just looking into whether 4.30.02 might resolve the issue. As we know, the issue is not reproducible, but happens often enough to have an impact on production, especially downstream processes such as de-noise, compositing, etc because the 0001 image file is corrupt (in our case its zero bytes). We're also locked in a production schedule and it would be a while before we could safely ponder an upgrade, although 4.30.00 to 4.30.02 might be safe.

So we're still examining how to resolve the issue and we might have to resort to a post-process that looks for and deletes these spurious image files.
Thanks.

[Donald2B]

We too were on version 4.30.00 and having issues....so you may want to start with submitting a test animation to a single node updated to the latest 4.30.02 and see if that solves the problem.

[vanpixelguy]

OK, I've downloaded 4.30.02 and will see what happens...

Thanks for your help on this!

[Donald2B]

I hope it helps....let me know how it works out for you.