really weird network rendering problem

[j.galang]

Hi. Has anyone experienced this sort of network rendering problem. We do mostly arch vis stuff in our office and the usual technique is the saved irradiance map from an incremental add to with low AA settings thing. The weird problem is this. Some frames turn out different, what I mean by this is that it is a bit darker than the other frames. Sometimes some lights even look that they were turned off. Sometimes the thin elements of the rendering (thin geometry) look darker than the rest. I was able to narrow it down to one machine, so I thought that it was just something wrong with that slave because re-rendering the frames in the other slaves corrected the problem. Now after sometime I tried using that malfunctioning slave again. Surprisingly it went well in the beginning (2 render jobs of 600 frames distributed). On the third job it acted up again rendering darker frames, NOW the weird thing was is this time it wasn't just that computer that acted up but also another computer.

Just for reference, all computers are running the same version of Vray (1.09.02U). All have the same plugins (network rendering won't work on a machine if that machine is missing a plugin) and map paths. Has anyone experienced this problem. Its becoming a really annoying since now it seems to be appearing randomly (sometimes ok, sometimes not) and now its happening on 2 of my dual processor machines (a waste of resource). Any help would be much appreciated.


Joel

[vlado]

Do you have the plugins installed locally on each slave or in a common network folder? If you have them in a shared network folder, try to copy them locally and see if this changes anything.

Best regards,
Vlado

[j.galang]

RE

the plugins are installed locally on each slave.

joel

[nagboy]

Cheers,

There is a bug in the network rendering submittion thingy, the "Ignore Scene Path" does not actually force the render nodes to absolutely positively ignore the scene path, thus if there are several different versions of a map with the same name some machines which have this map in their local path will use it instead of the map at the server. This is a problem if you use your workstations as render nodes as well, i cant count how many times we have scratched our heads why one machine renders with minute changes compared to the other ones only to find out someone had editied a map locally a little bit just to try out something new and that map ended up being used in network rendering.
Have you tried submitting the jobs with all the maps included to make sure they all have access to exactly the same map? Do you render with a framestamp which includes the vray version (just to be 100% sure its all the same one)? Also remember that if you have a big renderfarm and the server runs out of concurrent connections some machines may or may not get access to maps etc. if this should happen to be for example a reflection map it might have a fair impact on you rendering. Also if some of your maps are avi files make sure that all render nodes uses the same version of the avi codec. Is the different nodes using exactly the same max instsallation (which you have installed on just the one machine and copied it to the other nodes?) Its a network render nono to use the workstation max installations for both work and rendering, a bit more cumbersome to set up two different max'es for work/rendering with max5, but definatly worth the effort. hope you find out whats wrong

[little_mikey]

nagboy,

can u elaborate more abt the 'big network rendering no no to use max installation as work / render" bit?

u mean we shld have 2 instances of max installed in the computer?

[nagboy]

Cheers,

Basicly yes you need a network render max and a seperate work max. The reason to do this is to make absolutly sure that all rendernodes uses exactly the same versions of all scripts, plugins, dll's etc. Different animators install different plugins and scripts in different versions and its a complete disaster if all these nodes uses different things to render images. Just having different preferences in max is enough to tamper with network rendering. Max4 is dead easy to configure to do this:

1. Install max4 minimum install on a clean machine. Install to for example "C:\Netmax42" (Apply all pathces, no need to authorize)

2. Make an "R42" dir on the server and make the following directories within: "Plugins", "PluginsAdded", "LocalPath", "NetMaps".

3. Edit plugin.ini in "C:\Netmax42\", modify it so that all plugins apart from the Standard plugins are loaded from the server, lets just call the server for "server" from now on.

Additional MAX plug-ins=\\Server\R42\Plugins
NetAdditional MAX plug-ins=\\Server\R42\PluginsAdded

4. Move all the plugins from "C:\Netmax42\Plugins" to "\\Server\R42\Plugins\" and copy all 3rd party plugins needed (like vray etc.) to "\\Server\R42\PluginsAdded"

5. Edit 3dsmax.ini in the "C:\Netmax42", remove all bitmap dirs and add:

[BitmapDirs]
Dir1=\\Server\R42\NetMaps

6. Add "\\Server\R42\LocalPath" to the path of all your rendernodes, do this by rightclicking "my computer", properties, advanced, environment variables, system variables, path. add ";\\Server\R42\LocalPath" to the string. Copy all files that max requires to be locally into this dir (for example all the extra vray files) (You also need to either log out and in or reboot the machine to initiate the change in the path variable)

7. Run ServerApp.exe and configure.

8. Copy the entire "C:\Netmax42\" directory to all rendernodes and workstations, from now on you only use the ServerApp.exe in the C:\NetMax42\ Dir to network render, never the one in your work max.

9. Make a shortcut to ServerApp.exe and copy it to your system startup.

10. some machines will need "MSVCP60.DLL" copy that from a machine that has got it (all your workstations has it) to (in usual cases) "C:\Winnt\System32\"

This is a basic setup that allows you to:
1. Update all plugins at once to all rendernodes.
2. Update all other locally needed files to all rendernodes in one go.
3. All maps for all jobs go into the netmaps folder, all machines render with these maps and no other. (Always submit _without_ include maps)
4. Ensures that all nodes uses the same, plugins, vray files and maps

I hope i havent forgotten anything that should prevent this from working, also you can scale up this installation as much as you like and move basicly every dir in the max installation to the server, just remember to edit the paths in the 3dsmax.ini accordingly. The only things you wont be able to move in this way is "Scripts" and "UI", for some reason you only get error messages when rendering if these are on the server. The only readson ive split plugins into two groups "plugins" and "pluginsadded" on the server is just so you never mess with the default plugins and have complete control over what plugins you have added.

The only annoying thing with this setup is that vray wont work in DR mode with this, it seems that in DR mode the Spawner has the path to the vray plugins hardcoded into it. i hope vlado will change this to use the plugin path(s) in the plugin.ini to find the vray plugins.

Max5 setup is a bit more tricky to set up, but not much if anyone wants a step by step setup for that aswell give word.

Cheers again,

[j.galang]

Wow. Thanks for all the anwers. I will try them soon one by one to see what I can do to get rid of my problem. Thanks again for the great answers.

Joel

[eltioseba]

Hi, tnx for the complet instruction


Can you do a 5.x one?
And just a simple one, I dont undestand in wich machine you have to make those install. Server are suposed to be the EXTRAS machine, and client the one I`m working. Thats aply to your instructions? Im confused in wich machine a have to set your instruction, and to wich machine I have to point.

Tnx for your patience, Im new in distributed rendering.