Hi,
I recently set up a render farm for us consisting of (total) 3 workstations and 3 render nodes. Now when I got DR up and running everything seemed to work fine, but I noticed one thing:
When rendering with all available machines, it often happens that the buckets of one or several machines get rendered correctly, but not immediately getting assigned “new buckets” if that makes sense. It seems they render their assigned buckets, then have a little delay of a couple of seconds, then get assigned new buckets, and not how I expected they render one bucket, and as soon as they are finished with one bucket, directly continue rendering the next one.
Could this be some kind of overhead issue? Is it a known issue? Its a relatively simple image, but large in pixel dimensions (10k), also its writing a vrimg on the fly onto the network. I can provide a scene, although I don’t think it would make much sense.
It looks like a network issue, but can’t be sure about that. What is the speed of your local network?
Is it possible to run some tests with bigger bucket size and smaller resolution ? Maybe without writing the render into a file ?
And this is Max 2016 like shown in your signature ?
Update: Do you have any proxy file or displacement geo in this scene?
1) The speed of our network is 1 GBit, via a HP 1820-48G Switch.
2) I can run some tests with the changed parameters you mentioned. Will do and report back.
3) Yes, it is 3ds max 2016 with VRay 3.60.03
4) The only “special” stuff in the scene is a simple multiscatter. It’s just a test scene, with a teapot, textured with a texture on the network, and multiscattered boxes on top of the teapot. Nothing else.
I hope the issue can be solves without disabling the vrimg write, since I would love to have it to be able to resume a rendering.
Alright, I made some more tests. As for your edit: No there’s no proxies or displacement involved. It is just a teapot, a domelight and a groundplane.
Reducing output resolution to 5k and increasing bucket size from 32 to 128 did not make any difference. Yet disabling file save altogether did make a difference, yet the behaviour is still present, although to a much smaller degree.
I monitored network load and it averaged around 20% network load on the 1GBit connection, with bottoms of 5% and short peaks of 80%. Here’s a video of the behaviour along with network load:
With file saving: --deleted--
The points where it happens are evident. After gradually losing buckets, they all jump back in at 1:04
Without file saving: --deleted--
Way less of the behaviour, yet still there, for example at 0:50 where the blob of new buckets appear.
The network server is a DiskStation DS1815+ with 8x WD-WCC4E0023240 HDDs (ext4) running as RAID-5. I will test writing the image locally and report back.
Tested this one more time and I’m getting the same behavior for the DR slave buckets in a little bit higher resolution. Here is what happens: https://ftp.chaosgroup.com/support/screenshots/2018-04-04\_09-08-23.gif
There is a big difference when the file is not saved to the disk during the rendering process so we’ll look into it and try to make some optimizations.
Great to hear, hopefully you can make some optimizations, in the end you can have another changelog entry stating “Up to 20% faster rendering speed under certain conditions” ;-). Just kidding - thanks for the confirmation!
Also if you don’t need to see them anymore, I’d take down the youtube videos. Feel free to download them for reference, and/or tell me when it’s fine to take them down.
