NUMA System. For Vlado and his team.

[AndresAhumada]

Hello ...

For Vlado and his team.

I need to know if the latest build of-vray (2.40) is integrated NUMA recognition. Currently I have two computers Intel dual xeon e5-2687w and would like to make the best performance for vray.

Greetings and thanks for your answers.

[tashko.zashev]

Hi,

We have some improvements made for the internal V-Ray builds on that regard. Please contact us (vray@chaosgroup.com) and we will provide you with further assistance.
You are using this machine as a render slave, is that right?

P.S. Just include this thread as a note in the mail.

[vlado]

I need to know if the latest build of-vray (2.40) is integrated NUMA recognition. Currently I have two computers Intel dual xeon e5-2687w and would like to make the best performance for vray.

It might just work fine; Intel processors seem to handle NUMA better than AMD processors. We did some changes to the nightly builds, but you will have to test if they actually help.

Best regards,
Vlado

[AndresAhumada]

Each workstation is for a CG Artist and each independent mind, are connected to a farm of 10 dual xeon e-5670.
We have two identical farms.

That kindof assistance in the mail I request you exhibit? there a way to improve performance vray base to better understand NUMA environments?

Best regards and thanks for your help.

[AndresAhumada]

I wish I could help in the development of the following versions of v-ray. Especially in environments that everyday numa become more frequent.

Best regards and thanks for your help.

[tashko.zashev]

Thanks for your efforts it is really appreciated.
Did you experience any problems with the latest V-Ray build and these workstations?
You can email us (vray@chaosgroup.com) in order to test a nightly build.

[AndresAhumada]

I have no problem.

Always ... looking for more performance.
If there is any way to improve the use of resources would appreciate it.

Thank you for your help.

[peakyfreak80]

Can I ask what NUMA recognition is and if it might help with rendering performance? I currently have two 2x2687W machines and am always looking for a performance boost if it's out there.

Thanks

[Dariusz Makowski (Dadal)]

Same questions as above, about to build 16+core rig on intel.

Thanks, bye.

[vasil.minkov]

Hello,

One possible approach to take an advantage of a NUMA system is to start several processes, running on different NUMA node(s) each.

Recent V-Ray versions include some functionality that helps to restrict the number of CPU cores used by the process:

- it is now possible to restrict the number of computation threads by setting the VRAY_NUM_THREADS=N environment variable.
By default V-Ray creates one computational thread per CPU core.
- VRaySpawner now can launch several slave processes, running on different NUMA nodes.
- several slave processes can run on the same server machine, using range of listening ports. The port range is selectable on the render client.

I'll give more details on these in the next posts.

[vasil.minkov]

Running render servers on NUMA

VRaySpawner.exe is now extended with the following command line options:

• -numa[=N]
• -node=node1[,node2][,node3]...
• -port=port1[,port2][,port3]...

If "-numa" or "-node" options are not given, vrayspawner acts as usually - creates one slave process that uses all available CPU cores.

When "-numa" and/or "-node" options are present on the command line, vrayspawner launches several slave processes, such as
every process runs on its own NUMA node(s) and listens to different slave port.

Ideally, one slave process per NUMA node could be used for best performance. Running too many slave processes on the same
machine may eat all of your memory, so it may be useful to run less processes, distributed on the available NUMA nodes.


Description of the options:

• -port=port1[,port2][,port3]...
  select listening ports for the slave process(es). If the number of processes is greate than
  the number of ports given, the last port is auto incremented. The default listening port is 20204 for 3ds max.
  The render client should be set to use the corresponding port range.
• -node=node1[,node2][,node3]...
  select NUMA nodes that will be used for the slave process(es). Default - use all available NUMA nodes.
• -numa[=N]
  select the number of slave processes to start. If N is not given, N=0 or N>=M, one process per NUMA node will be created.
  Here "M" is the number of selected nodes using "-node", or the number of all available NUMA nodes if no "-node" is used.

Examples (considering you use 3ds max 2012 and a system with 8 NUMA nodes):

• vrayspawener2012.exe -numa
  spawns eight 3dsmax.exe processes, every running on single NUMA node and using listening ports 20204-20211
• vrayspawener2012.exe -numa=4
  spawns four 3dsmax.exe processes, every running on two NUMA nodes and using listening ports 20204-20207
• vrayspawener2012.exe -node=3,5,6 -ports=30000,40000
  spawns three 3dsmax.exe processes, running on NUMA nodes 3,5 & 6 and using listening ports 30000, 40000 & 40001
• vrayspawener2012.exe -node=2,3,4,5,6 -numa=3
  spawns three 3dsmax.exe processes, running on nodes (2,3) (4,5) & (6) and using listening ports 20204,20205 & 20206

[vasil.minkov]

Running 3ds max as render client on a NUMA machine

One possible approach to use 3ds max as a client on a NUMA machine is to start several DR slaves on the same machine, using VRaySpawner as described in he previous post.
The following examples assume 8 NUMA nodes with 6 CPU cores each:

1. Using single-threaded client
vrayspwaner2012.exe -numa
or
vrayspwaner2012.exe -numa -node=N - to restrict the number of processes and reduce the memory usage
Then launch 3ds max and make the following settings:

• turn off multithreading (Customize -> Preferences -> Rendering -> Multi-threading)
• turn on DR and select DR servers: Rendering -> Settings -> Distributed rendering -> Settings. Add "localhost" or "127.0.0.1" server with a port range of 20204-20211 for 8 DR slave processes.

In this case the client machine will use single computation thread in order to not interfere too much with the render servers.

2. Using multi-threaded client
Another approach could be to launch the client process on a dedicated NUMA node. As with the example above first start vrayspawner:
vrayspwaner2012.exe -numa -node=1,2,3,4,5,6,7 - reserve node 0 for the client process and spawn 7 slave processes
or
vrayspwaner2012.exe -numa=3 -node=1,2,3,4,5,6,7 - reserve node 0 for the client process and spawn three slave processes using nodes (1,2,3), (4,5) & (6,7)
Then launch 3dsmax.exe - from the command line, or maybe using a batch file:
set VRAY_NUM_THREADS=6
start /node 0 /affinity 3f 3dsmax.exe
This will start 3ds max using six computation threads and running on the first six CPU cores. If Node0 on your system use different number and/or topology of the CPU cores, the "VRAY_NUM_THREADS=6" and "/affinity 3f" should be modified.

Finally, VRaySpawner launching could be automated using simple windows shortcut. There is also ability to use .ini configuration file instead of command line for VRaySpawner.

[tecsun]

So is that mean now AMD 4x Opteron system can work well now?

One possible approach to use 3ds max as a client on a NUMA machine is to start several DR slaves on the same machine, using VRaySpawner as described in he previous post.
The following examples assume 8 NUMA nodes with 6 CPU cores each:

1. Using single-threaded client
vrayspwaner2012.exe -numa
or
vrayspwaner2012.exe -numa -node=N - to restrict the number of processes and reduce the memory usage
Then launch 3ds max and make the following settings:

• turn off multithreading (Customize -> Preferences -> Rendering -> Multi-threading)
• turn on DR and select DR servers: Rendering -> Settings -> Distributed rendering -> Settings. Add "localhost" or "127.0.0.1" server with a port range of 20204-20211 for 8 DR slave processes.

In this case the client machine will use single computation thread in order to not interfere too much with the render servers.

2. Using multi-threaded client
Another approach could be to launch the client process on a dedicated NUMA node. As with the example above first start vrayspawner:
vrayspwaner2012.exe -numa -node=1,2,3,4,5,6,7 - reserve node 0 for the client process and spawn 7 slave processes
or
vrayspwaner2012.exe -numa=3 -node=1,2,3,4,5,6,7 - reserve node 0 for the client process and spawn three slave processes using nodes (1,2,3), (4,5) & (6,7)
Then launch 3dsmax.exe - from the command line, or maybe using a batch file:
set VRAY_NUM_THREADS=6
start /node 0 /affinity 3f 3dsmax.exe
This will start 3ds max using six computation threads and running on the first six CPU cores. If Node0 on your system use different number and/or topology of the CPU cores, the "VRAY_NUM_THREADS=6" and "/affinity 3f" should be modified.

Finally, VRaySpawner launching could be automated using simple windows shortcut. There is also ability to use .ini configuration file instead of command line for VRaySpawner.

[vlado]

So is that mean now AMD 4x Opteron system can work well now?

If you already got one of these (by mistake or not), this approach might help to bring the performance to the expected level. But it's definitely not the most straightforward thing.

Best regards,
Vlado

[Macker]

Can someone recapitulate this thread in laymans terms please?