My team and I have been working on some tour visuals for the past month with some pretty crazy Houdini sims and now we’re trying to render out the 7,000 frame sequence and in order to do so we’re using Vray + Houdini on AWS using Deadline’s AWS Portal system.
It took us about a week to get to where we are now but our render times on a g3.8xlarge are still over an hour and a half, with 75%+ of that time spent with nothing happening. In the Vray logs, I can see that while rendering this frame it spent 40 minutes doing nothing:
2019-10-10 04:55:02: 0: STDOUT: VFH V-Ray: Prefiltering light cache… 100%
2019-10-10 05:37:09: 0: STDOUT: VFH V-Ray: Average rays per light cache sample: 7.51 (min 1, max 398 )
On top of that, I’d love to be able to take advantage of hybrid rendering since we have 32 cores sitting there (and billing us per core hour), but the issue is that Vray says C++ CUDA implementation needs to be “set explicitly” to use the processor but we don’t have matching machines in our office that we can submit with in order to have that checkbox checked off.
I’m under a lot of pressure to get this done (we’re already a week late due to rendering issues), and would appreciate any help I can get!
Attached is a full render log for one frame where you can see the big slowdown.
I’m doing a blend of the 5.12, 5.18, and 5.24s. Deadline is not good about doing 50x of the same machine, even though we got our limit increased. Ended up with 300+ errors and a lot of wasted compute + credits for stalled workers because Deadline kept killing and restarting my nodes for some reason. (with 50 nodes at once… this has proven to be very expensive)
But if I let Deadline make a pool that combines the three instance types, it does much better and I haven’t really had any stalled workers recently.
Shit, we had a similar issue. We had amazon/thinkbox tech support trying to solve that for us too. they were completely stumped and said it shouldn’t be happening - when monitoring it through remote desktop it would suddenly pop up with a ‘windows is shutting down’ message as if someone hit the power switch and deadline would lock up it’s updates for 10 minutes before it realized the machine was not running and kill it.
We didnt think of making a pool that combined 3 instance types… had 2 tech support guys from amazon claim it should not be happening and it was so intermittent we just worked around it. Once it started we dropped to 10 machines or so, after 12hrs we found we could safely ramp back up to 40+ for another half day.