Render Farm extremely slow

[jacksc02]

bobby, you don't need to buy full vray licenses just render node licenses which are roughly USD$250 each but cheaper if you bundle it in with a license upgrade

[glorybound]

bobby, you don't need to buy full vray licenses just render node licenses which are roughly USD$250 each but cheaper if you bundle it in with a license upgrade

I did not know that, that changes things.

[jacksc02]

Also if you have any old licenses of Vray 2.X you can upgrade with additional render licenses for a pretty good price.

[EricE]

Is there any truth to this video. that vray 2.4 works better on a single processor and slower on dual.
https://www.youtube.com/watch?v=KfsaZxi887s

[Joelaff]

That tests looks like DR, which has a lot of overhead. Only good for long renders or smaller scene files where the overhead of transferring the data can be made up in time. For animations I assume you are rendering whole frames.

Bobby, for render power look at building yourself a 5960X machine for around $2-3k tops with a rackmount case and a good cooler. Overclocked it should be almost as fast (or possibly faster) as those Boxx dual machines for half the price.

As far as textures slowing things down... Are the textures being pulled over the network, or local? Never had too much trouble with non-tiled textures being pulled over the network, just a little slower loading time when tins of machines try to load the scene at once. With tiled formats(and vRay proxies) rendering is often a lot slower if the assets are on a server. For larger scenes where we use these formats we mirror our assets to the render nodes with GoodSync. Mild pain to sync when you make changes, but worth the speed up in those cases.

When you look at the task manager are the machines spending a lot of time doing single threaded operations (not using all the cores?) That is the biggest drawback to Xeons, poor single threaded performance due to low clock speeds.

You are timing renders on the workstation and the render nodes both using 3dsmaxcmd.exe from the command line? This is the best way to compare apples to apples. Could some of the slowdown just be in loading the scene? Or processing single threaded things like pFlow?

[theedge]

But how do you set the nodes to use the local assets instead of trying to grab them from the server ?

[Joelaff]

Create a local directory that is the same on all nodes. (C:\maps\)

Then share that directory from each node. Then use one machine with Good Sync to sync files from your workstation to that directory on each node. We make it a habit to keep the assists on the server, and then sync them to the workstation first. On the workstation you have to use the asset manager in Max to change the assets to use C:\maps. If you happen to forget for one or two assets they are still pulled off the server.

Keeping them in sync is a little bit of a pain. We use GoodSync like that as it syncs each one in sequence, never connecting to the workstation more than once at a time (windows limits to connections to non server OSes). For smaller scenes we just leave assists on the server. Never seemed to make much difference without a lot of maps or tiled images or proxies.

I suppose windows offline files might work as well, but that would still be stating each file to see if it had changed. Never had great luck with offline files.

[theedge]

Thanks for the detailed explanation. It is indeed a bit complicated. It would be awesome if v-ray could handle this. There's already an option to transfer missing assets. It must not be that hard to extend the option to all assets.

[Joelaff]

Once you set it up, it is not too bad. But not as simple as a checkbox.

Of course any auto option would still have to stat all the files (read their file info like size, and mod time) both locally and on the server. Stat-ing files (I call it that because of the Linux function stat()) is actually a hell of a lot slower than you would think it should be, and on a fast file server or drive may be almost as slow as copying a small file. So doing it manually is the most efficient, but also the most work.

Did I mention GoodSync makes an awesome backup tool as well? It can save multiple older versions of files (and remove them after a time). Pretty slick tool, though not the fastest thing on the planet.

As far as the OP goes, I would just watch the processor and network usage and see where things are slowing down. Also keep in mind many scene will indeed render faster on a fast i7 than a dual Xeon machine. Two processors on one motherboard is never as fast as one faster processor (often only 50% better than a single proc of the same speed), and megahertz matters a LOT more than marketing people want you to think. The $300 or whatever it is per vRay license is far less than the extra cost of a dual proc machine. Also, more machines means a smaller percent of the farm is down when one machine goes down (and they will, usually the power supply, especially with a branded machine vs. a home built, as the brand name companies use the cheapest, lowest output, power supply that gets the job done vs. buying a good supply yourself. I would never put less than 800watts in a render node, ideally more. We have burned up far too many 550w supplies in the farm. I keep a spare PSU at all times to swap out.).

It all depends on your needs and how you do your rendering.

[Dman3d]

Not to hijack your thread but we have been experiencing a similar problem. In our case you can watch an image finish and just sit there for an additional hour or minutes before it saves and ends the job. Others are just incredibly slow to complete vs rendering locally on a workstation. This seems to be a problem with mentalray and vray. Running SP3 on 2015 by chance?
Our nodes are dual 2680v2's and most workstations are 4770-4690 single quad/six core.

I am having IT torch a machine so we can clean install 2015 with no service packs and walk it through that way for testing.

Are you running backburner or deadline? May want to set the max concurrent jobs to a lower number, something like 3-4. I cant remember the default in backburner to make sure only X machines are starting the job at once so you are not overwhelming servers with sending scene files and loading textures all at once on all the machines. Depends of course on your network and how its setup as to whether that would be an issue or not.

I have a vague memory of the HT issues. Also having enough memory per core. Something like 2GB min or your processors suffer memory starvation. X cores / X RAM
So 32GB on dual 8 (16 physical/ 16 virtual) 32 logical processors/32GB RAM = 1GB each but I am not sure how the memory allocation is for HT threads.
I am sure someone can get me sorted on that.

[Joelaff]

Not familiar with that slowdown at render end, but could be the OP's problem.

Not sure where the whole 2GB per core thing that gets thrown around a lot comes from. How is more main memory related to more cores? It is not as though each core uses its own chunk of main memory.

Yes, they all fight over the same memory, which is why fewer, faster cores (comparable) will always be faster than more slower cores. The data is not copied into multiple places in main memory once per core (would be an interesting speed optimization for a developer to try, but not really practical given how much memory that would require).

I think that whole per core myth may be from Intel suggesting it for virtualization, where indeed each VM gets its own chunk of main memory. You either have enough memory to fit your project in RAM, or you don't. More memory is not going to speed things up unless you are then increasing your BSP tree or caching more textures in RAM or whatnot. But VRay just loads texture into RAM (at the beginning of the frame for bitmap nodes, when needed for VRayHDRI node). If you are using tiled textures you could increase the tile cache, but that is per instance of VRay, not per core.

[Dman3d]

Not familiar with that slowdown at render end, but could be the OP's problem.

Not sure where the whole 2GB per core thing that gets thrown around a lot comes from. How is more main memory related to more cores? It is not as though each core uses its own chunk of main memory.

Yes, they all fight over the same memory, which is why fewer, faster cores (comparable) will always be faster than more slower cores. The data is not copied into multiple places in main memory once per core (would be an interesting speed optimization for a developer to try, but not really practical given how much memory that would require).

I think that whole per core myth may be from Intel suggesting it for virtualization, where indeed each VM gets its own chunk of main memory. You either have enough memory to fit your project in RAM, or you don't. More memory is not going to speed things up unless you are then increasing your BSP tree or caching more textures in RAM or whatnot. But VRay just loads texture into RAM (at the beginning of the frame for bitmap nodes, when needed for VRayHDRI node). If you are using tiled textures you could increase the tile cache, but that is per instance of VRay, not per core.

Gotcha. See someone cleared it up

Thanks Joelaff