3dsmax / vray gpu / deadline

[nicolas_fuminier]

I've been trying to figure out what's going on with our production renders on 3DSMAX / VRAY GPU / DEADLINE for the past 2 months.

I have searched, all the options I knew, all the tests I knew how to do and I managed to "circle" the problem but impossible to understand what causes it. There is very strong chance that the problem is more on the side of DEADLINE than VRAY / 3DSMAX, but I try my luck sometimes that my problem face echo here.

Software :
3DSMAX 2023.2.2
VRAY GPU 6.00.20
DEADLINE 10.1.23.6
NVIDIA 512.59

We are working on a project with jewelry (as often in the studio) and the jewels arrive in 3DSMAX in .ABC from animations and rigs with position constraints made in MAYA.
The .ABC are composed of multiple sub-objects corresponding to each element of the jewel (in this case links of chains among others)

The loading of .ABC in 3DSMAX is done without worries, then shading and lighting in 3DSMAX, then we send all that on DEADLINE for the rendering. Classic.

Our concern is that when 3DSMAX will load for the first time the project on deadline order, one of the sub-objects of the ABC can load badly and be positioned at the first frame of the ABC.

Example:
Computer 10 arrives in the DEADLINE and receives the order to render frame 43 because the other 42 frames are already rendered or being rendered by other computers.
Computer 10 loads the scene (without any apparent error on DEADLINE) and renders frame 43 (without any apparent error on DEADLINE)
In this precise case (first time that the computer loads the scene) it is "possible" that one of the sub-objects of the ABC loads the position of this sub-object as if it renders the frame 00 (we are not 100% sure of this analysis, is it the timing of the ABC, is it the position contriantes that jump ...).
And we are talking here about a single sub-object (always a single one, never 2), any one of them (no matter size, polycount ...), on any computer (we have 10).

When we restart the task manually to correct the problem on DEADLINE, the frame is restarted by an available computer and the dice are rerun. It is quite possible that this frame has the same problem but on another subobject of the ABC. Still only one object, but any object. It is also possible that the frame renders well and that the problem does not arise.

When the computer moves on to the second frame of its list, there are no more errors of this kind, until it finishes its job, but without ever leaving the queue. If for some reason it has to reload the scene via DEADLINE, the dice are rolled again and the error can return on any of the sub-objects.

Things got even more murky for me when we discovered that this could also affect an animated VRAY light.
The scene can have 10 VRAY lights, just the animated one can have a one frame rollback of a single light when 3DSMAX loads the scene via DEADLINE. According to exactly the same rules as the problem with ABCs.

We have tried a LOT of solutions, none of them worked for the moment.
There is no other third party plugin needed to render these frames.

We were in 3DSMAX 2021, VRAY 5XXXX and DEADLINE 10.1.20.3, we updated everything, it did not change anything.
We made multiple tests with our weak knowledge in DEADLINE. All our tests proved to be unsuccessful for the moment.
We thought of a network problem, but DEADLINE not indicating ANY related error, it seems to me a weak hypothesis.
We made tests in CPU and it seems that the problem does not exist (same scene, same setup, just switch to CPU).
We made tests in GPU with a local render (no DEADLINE involved) and the render is perfectly OK

Unfortunately, I can't share in public the scenes in question as I am under NDA for this project. I can eventually transmit the scene via the contact form if needed. I have attached a small piece of rendering to illustrate my point.

I am well aware that the concern seems to be at the level of DEADLINE and I would share a similar post on their forum in parallel of this one, but not having apparent concerns in CPU, I say to myself that there is perhaps something to dig with you.

We have already made multiple 3DSMAX / VRAY GPU / DEADLINE projects without encountering similar problems.
We have found no echo of our issue on any forum we search on (CHAOS forum, AUTODESK forum, THINKBOX forum)

Thanks in advance for your time and your answers,
Nicolas

[nicolas_fuminier]

Here is the job log of a bad rendered frame, if someone see something that shouldn't be there (there is a couple of warning but they dont looklike involved in the issue)

[nicolas_fuminier]

Hello again,

I try to add as much detail as possible as we research, test and understand the problem internally.
We imported an old ABC from another project to serve as a test more freely and simply than the production files we were using for testing.


I confirm that on the same new simple scene, with the same DEADLINE settings, the same sources and the basic VRAY

- No worries on the CPU version
- No problems on an ABC loaded via a VRAY PROXY (unfortunately it's more complicated for us to use this method, because of our shading and lighting piepline)
- ABC problems with a ABC imported in the classical way directly via 3DSMAX import.

In the videos attached (very fast test on a sample scene) we ask DEADLINE to reload the scene after each frame in order to generate a maximum of bug (this option is checked in the three renders).

I cannot for the moment provide the basic MAX scene for creative properties reason of the 3D models but I can share the 3D video renders (there is a small normal issue on the chain but it's not relevant here).
Linked in this post (https://we.tl/t-AEnVkXtaXk) :
- TEST_ABC_GPU_VRAYPROXY.mp4 (abc file loaded in a Vrayproxy rendered in GPU)
- TEST_ABC_CPU.mp4 (abc file loaded via 3DSMAX > import > .abc rendered in CPU)
- TEST_ABC_GPU_V01.mp4 (abc file loaded via 3DSMAX > import > .abc, V01 is exactly the same .max file as V02 rerendered a second time to show that the bugs a not sowing in the same frame with the same objet, it's totally random. Rendered in GPU)
- TEST_ABC_GPU_V02.mp4

Has it already happened to someone? I can't find out if it's unique to our setup or if the error has already been encountered by someone else.
(same thread / posts has been sent to the DEADLINE forum)

Any help is welcome,
Thanks in advance,
Nicolas

[Muhammed_Hamed]

Thank you for the detailed information Nicholas, I will test this on my farm locally on the weekend(with a similar ABC setup I have). I will write here with a conclusion around Monday

Best,
Muhammed

[nicolas_fuminier]

Thanks for your time Muhammed_Hamed,

Let me know if you need a share of some of our scene.
If you want to "force the odd" to see more bugs you have to enable : "Restart Renderer Between Frames" in the Render Tabs. It will force DEADLINE to see each new frame renderer as a new computer joining the render pool.



I could confirm that the problem came from DEADLINE and not from VRAY. People have similar problems with MAYA and REDSHIFT via DEADLINE.

The only solution we found is to do without DEADLINE and to choose the rendering management solution of a competitor.

In our case (3DSMAX) we found a small software that seduced us instantly.
https://www.pulze.io/
Easy to install, managing almost all the rendering engines compatible with 3DSMAX and a good number of plugins. The developers are very active and the prices are quite acceptable for us.
Of course (otherwise I wouldn't present it as a solution), no worries on the alembics and VRAY GPU, no worries on the rendering anywhere in fact. Everything works fine.
Certainly less rich in options than DEADLINE but much more up to date and especially focused on 3DSMAX.

We keep DEADLINE for security but I think we will not return on it. No return of their share on our problem, of the almost non-existent updates and the software in itself begins to date a little.


Best,
Nicolas