Distributed bucked - slave ignores GI pass?

Heya

So I had enought of this issue and decided to post it out.

It apears that my slave is not rendering Global Illumination for some reason - when I look at render layers all are fine except this1 GI. Buckets are black. (might be wrong I havent used it for a while but when IRR is calculating the buckeds done by slave are darker/black…)

Did any1 else had this issue ? How can I fix it ? :confused:

Maps are accessible..

Thanks, bye.

Since Vray1.5/win7 x64 DR is broken for us. It allways works on small scenes, but after tracking all this on ports, with packets sniffers etc, on big scenes with heavy GI files, all DR calculate the GI ok, they send it back to the main machine, the main machine merge it, but then it begin to send back to full GI to DRs machines and some never get it, some start to get it but don’t get it fully, etc … We now use DR a lot less, and allways stop it after gi pass, save the gi files, and render the AA pass with GI from files.
Cumbersome, but the heavier the GI, the more useless DR is :frowning:

Wow I’m happy I’m not alone.

Would be nice to have an option to not use DR for GI pass… ;s or script ;s

Interesting; can you get me a scene where you get this problem to vlado@chaosgroup.com? How many machines do you use for DR? How fast is your network?

Best regards,
Vlado

Workstation + 1 slave
10mb/s transfer

It ignorest buckets at random on any scenes… its quite random :confused:

Vlado, it bug with any scene when GI is heavy. Usual debug testing done with one host and four DRs, around i7-920, everything gigabit network, win7 x64pro, all files on server2008R2. Entire network render with backburner without problems.
Test with a complex interior with lot of details, IR/lightmap. GI prepass take around 30min to 1h with DRs (5 machines).

Using a packet sniffer was open minded. All DR render prepass fine, send their GI to the main, the main merge the GI and send it back to DRs. Here comes the problems. I Can see all packets going to DRs, and some stop receiving packets before they got all the GI file. When a DR stop receiving the GI packets, it has no connectivity problem (ping -t for example give no error).

Thanks for the detailed information; I will check to see if I can find anything.

Best regards,
Vlado

Ok Vlado, let me know if you can reproduce the bug, identified it, and working on a fix. Here or by email, like you want.

Regards,
Kib

Well, while I couldn’t quite figure out what may be causing the bug on your end (tried DR with 8 machines, and quite a large irradiance map files and I could not reproduce it), at least the slaves will now detect such a problem, cancel the render on their side and attempt to join again the client machine. (before, they just went on with the rendering). In this way, at least you won’t get wrong buckets.

Best regards,
Vlado

Nice if we have no more wrong buckets indeed !
We have an example there today, when even sending with GI from file doesn’t work. The irradiance map is around 180Mo on the disk.

Did you turn on the “check for missing files” option in the system rollout?

Best regards,
Vlado

I never check it - would that force slaves to check untill they get file or cancel render on their side ? :o

Also any chance to have an option to render Gi on main machine only and then distribute it to slaves?

Is update available in nightly ?

Thanks.

Originally it would cause the slave to refuse to render the scene at all.

Best regards,
Vlado

Now they will properly cancel the render, instead of trying to re-join it.

Best regards,
Vlado

Vlado, we never check ‘check for missing files’. All our machines are installed and managed via scripts and all have exact same network units and max path for models, maps, etc.

Ok then. I’ve modified the code for the render slaves so that if they fail to load an irradiance map file, or fail to receive it from the client machine, they will at least cancel the render so that they do not contribute wrong buckets.

Best regards,
Vlado

Yes nice, but could you find/fix why it doesn’t receive entire file ? Not having wrong bucket is better indeed, but having good bucket is even better and faster, which is the purpose of DRs (faster)

No, I could not reproduce it unfortunately, although I tried quite hard (using 8 machines from our blade server with quite large irradiance maps), and I did not see an obvious problem in the code (of course, it’s probably non-obvious :-)). I added a message to print out some error codes in case this happens, so it might help to track the issue if it happens again on your side.

Best regards,
Vlado

Well, it happens everytime we try to use DRs. We usually use one host and 4 DRs, on complexe interior scenes (I can send one if you need).
If you need any beta version or debug utility to be run here, let me know, I can deploy it, do some test to report errors to you, and deploy back actual version. We really need to find a way to use DRs (we are looking for alternative solution, but not convinced yet), so if I can help you to track the bug, I will do.

You can email me to vlado@chaosgroup.com and I’ll get you access to the nightly builds; then we can work from there - I could add some debug code to track down the issue as needed.

Best regards,
Vlado