One server is freezing at "Writing Output" -- very sporadically.

[davision]

I have one render server that freezes up at random times while rendering backburner jobs. It will render a frame completely, it will go through denoising, and then it just freezes up at the very end when the status says "Writing Output ..."

If I reboot, and add it back into the job queue it often will start right up again. I can't figure out the cause and don't know where to begin troubleshooting.

I am saving the VrayFrameBuffer and not using the standard file save option in case that matters. What could be happening at this point? Is there any type of detailed log that would tell me exactly where/when the freeze is occurring? I can't believe it would be file or folder access issues because the server works about 98% of the time without fail.

[LeePuznowski]

If you have multiple servers rendering and writing, it may occur that more than one are trying to save the same file out at the same time and thus aren't allowed to write over the other. This a BUG that happens with 3.6 that saves the same filename.fileformat for every frame of a sequence along with the actual sequence filename (0001, 0002, etc.). Although this usually gives a write output error with Backburner, but it's hard to tell without knowing if you're rendering a sequence and are using 3.6.

[davision]

When you say 3.6 -- you are referring to the very latest Vray build 3.6.04? Of do you mean any of the 3.6 builds?

[Zdravko.Keremidchiev]

Do you use Separate render channels option?

[davision]

Yes I use separate channels.

[jblessing]

If you have multiple servers rendering and writing, it may occur that more than one are trying to save the same file out at the same time and thus aren't allowed to write over the other. This a BUG that happens with 3.6 that saves the same filename.fileformat for every frame of a sequence along with the actual sequence filename (0001, 0002, etc.). Although this usually gives a write output error with Backburner, but it's hard to tell without knowing if you're rendering a sequence and are using 3.6.

We are having this same issue. 3ds Max 2018/Vray 3.60.04 (official build). We are using separate render channels. Our servers crash at random maybe once every couple of weeks

[Zdravko.Keremidchiev]

Are you both rendering a sequence and which mode is the denoiser set to? Also would it be possible to provide BB log (%localappdata%backburner\backburner.log)?

[davision]

I am rendering a sequence of frames but as I said, usually it's just the one server that will sometimes hang on a frame. Denoiser is set to denoise elements separately and store result in separate channel.

Also, not sure it's relevant but the server in question is a Threadripper so it's the only AMD device in my render farm, and has more cores than any other computer.

I believe the span of the backburner log here between 11:58 and 12:20 shows when the error occurred. (Well, I know there was an error here but I'm not 100% sure it was the "writing Output" issue... although I'm fairly confident.


2018/01/30 10:43:53 INF Receiving new job from 192.168.1.18
2018/01/30 10:43:54 INF Job '17115 Engage Base_2099361714L' received and ready
2018/01/30 10:43:54 INF Launching 'Autodesk 3ds Max 2018 (64-bit)'
2018/01/30 10:44:44 INF 3dsmax adapter : Executing command line : C:\Program Files\Autodesk\3ds Max 2018\3dsmax.exe "-_pipe:344087578" -server "C:\Users\MattD\AppData\Local\backburner\Serve rJob \17115 Engage Base_2099361714L.max".
2018/01/30 10:44:44 INF [ServerMessage]ProcessStarted processId=81316
2018/01/30 10:44:44 INF 3dsmax adapter : Product version: 3ds Max 2018.4 Update (20.4.0.4254)
2018/01/30 10:44:44 INF 3dsmax adapter : Data collection and use is 'ON'. Change your participation anytime in the Help menu of 3ds Max.
2018/01/30 10:44:44 INF 3dsmax adapter : Autodesk 20.4 is Ready.
2018/01/30 10:44:44 INF New task assigned: 1001
2018/01/30 11:57:49 INF 3dsmax adapter : BEGIN FRAME 1001; TASK 2 OF 60; (NO SAVE);
2018/01/30 11:57:49 INF 3dsmax adapter : END FRAME 1001; TASK 2 OF 60; (NO SAVE); 4.38e+03s; 0 faces; MEM 2525 MB;
2018/01/30 11:57:49 INF 3dsmax adapter : Autodesk 20.4 is Done with : Frame 1001.
2018/01/30 11:57:49 INF 3dsmax adapter : Autodesk 20.4 is Done with task : Frame 1001 to Frame 1001.
2018/01/30 11:57:49 INF Block of 1 task(s) completed
2018/01/30 11:57:55 INF New task assigned: 1011
2018/01/30 12:18:28 INF Adapter returned unexpected code 1073807364 for "C:\Program Files\Autodesk\3ds Max 2018\maxadapter.adp.exe" "-o" "NewTask" "-l" "Info" "-j" "C:\Users\MattD\AppData\Local\backburner\Serve rJob \17115 Engage Base_2099361714L.xml.details" "-s" "1011" "-n" "1" "-a" "C:\Users\MattD\AppData\Local\backburner\Serve rJob \17115 Engage Base_2099361714L.zip" "-d" "utf-8"
2018/01/30 12:18:28 ERR Task error: Unknown Error
2018/01/30 12:18:28 ERR Unknown Error
2018/01/30 12:18:28 INF Application is down
2018/01/30 12:18:28 INF Adapter returned unexpected code -1073741502 for "C:\Program Files\Autodesk\3ds Max 2018\maxadapter.adp.exe" "-o" "ShutDown" "-l" "Info"
2018/01/30 12:18:28 ERR C:\Users\MattD\AppData\Local\Temp\adp82451: No such file or directoryC:\Users\MattD\AppData\Local\Temp\adp8245 0: No such file or directory
2018/01/30 12:21:52 INF <AdapterInfo>
2018/01/30 12:21:52 INF <Name>Autodesk 3ds Max 2018 (64-bit)</Name>
2018/01/30 12:21:52 INF <Version>300</Version>
2018/01/30 12:21:52 INF <Description>Autodesk 3ds Max 2018 (64-bit) Renderer Adapter</Description>
2018/01/30 12:21:52 INF </AdapterInfo>
2018/01/30 12:21:52 INF <AdapterInfo>
2018/01/30 12:21:52 INF <Name>Command Line Tool</Name>
2018/01/30 12:21:52 INF <Version>300</Version>
2018/01/30 12:21:52 INF <Description>Command Job Adapter v2018</Description>
2018/01/30 12:21:52 INF </AdapterInfo>

[LeePuznowski]

When you say 3.6 -- you are referring to the very latest Vray build 3.6.04? Of do you mean any of the 3.6 builds?

Yes any 3.6 build including the latest stable nightlies. Although I'm only using RT GPU so this may not be the case for Advanced. Maybe try changing your output path to a local folder on each node to test whether it saves without error. If a redundent file (ie. filename.fileformat) is being saved along any sequences (0001,0002, 0003, etc.), this might be a clear sign that it's this problem.