NUMA machines. Updated!

[ChaosGroup]

Reposting additional info from our developers posted in this thread:

- Chaos Forums

@vlado

It might just work fine; Intel processors seem to handle NUMA better than AMD processors. We did some changes to the nightly builds, but you will have to test if they actually help.

Best regards,
Vlado

@vlado

Hello,

One possible approach to take an advantage of a NUMA system is to start several processes, running on different NUMA node(s) each.

Recent V-Ray versions include some functionality that helps to restrict the number of CPU cores used by the process:

- it is now possible to restrict the number of computation threads by setting the VRAY_NUM_THREADS=N environment variable.
By default V-Ray creates one computational thread per CPU core.
- VRaySpawner now can launch several slave processes, running on different NUMA nodes.
- several slave processes can run on the same server machine, using range of listening ports. The port range is selectable on the render client.

I’ll give more details on these in the next posts.

@ivaylo.ivanov

VRaySpawner.exe is now extended with the following command line options:

• -numa[=N]
• -node=node1[,node2][,node3]…
• -port=port1[,port2][,port3]…

If “-numa” or “-node” options are not given, vrayspawner acts as usually - creates one slave process that uses all available CPU cores.

When “-numa” and/or “-node” options are present on the command line, vrayspawner launches several slave processes, such as
every process runs on its own NUMA node(s) and listens to different slave port.

Ideally, one slave process per NUMA node could be used for best performance. Running too many slave processes on the same
machine may eat all of your memory, so it may be useful to run less processes, distributed on the available NUMA nodes.

Description of the options:

• -port=port1[,port2][,port3]…
       select listening ports for the slave process(es). If the number of processes is greate than
       the number of ports given, the last port is auto incremented. The default listening port is 20204 for 3ds max.
       The render client should be set to use the corresponding port range.
• -node=node1[,node2][,node3]…
       select NUMA nodes that will be used for the slave process(es). Default - use all available NUMA nodes.
• -numa[=N]
       select the number of slave processes to start. If N is not given, N=0 or N>=M, one process per NUMA node will be created.
       Here “M” is the number of selected nodes using “-node”, or the number of all available NUMA nodes if no “-node” is used.

Examples (considering you use 3ds max 2012 and a system with 8 NUMA nodes):

• vrayspawener2012.exe -numa
          spawns eight 3dsmax.exe processes, every running on single NUMA node and using listening ports 20204-20211
• vrayspawener2012.exe -numa=4
          spawns four 3dsmax.exe processes, every running on two NUMA nodes and using listening ports 20204-20207
• vrayspawener2012.exe -node=3,5,6 -ports=30000,40000
          spawns three 3dsmax.exe processes, running on NUMA nodes 3,5 & 6 and using listening ports 30000, 40000 & 40001
• vrayspawener2012.exe -node=2,3,4,5,6 -numa=3
          spawns three 3dsmax.exe processes, running on nodes (2,3) (4,5) & (6) and using listening ports 20204,20205 & 20206

@swisscad

One possible approach to use 3ds max as a client on a NUMA machine is to start several DR slaves on the same machine, using VRaySpawner as described in he previous post.
The following examples assume 8 NUMA nodes with 6 CPU cores each:

1. Using single-threaded client
vrayspwaner2012.exe -numa
or
vrayspwaner2012.exe -numa -node=N - to restrict the number of processes and reduce the memory usage
Then launch 3ds max and make the following settings:

• turn off multithreading (Customize → Preferences → Rendering → Multi-threading)
• turn on DR and select DR servers: Rendering → Settings → Distributed rendering → Settings. Add “localhost” or “127.0.0.1” server with a port range of 20204-20211 for 8 DR slave processes.

In this case the client machine will use single computation thread in order to not interfere too much with the render servers.

2. Using multi-threaded client
Another approach could be to launch the client process on a dedicated NUMA node. As with the example above first start vrayspawner:
vrayspwaner2012.exe -numa -node=1,2,3,4,5,6,7 - reserve node 0 for the client process and spawn 7 slave processes
or
vrayspwaner2012.exe -numa=3 -node=1,2,3,4,5,6,7 - reserve node 0 for the client process and spawn three slave processes using nodes (1,2,3), (4,5) & (6,7)
Then launch 3dsmax.exe - from the command line, or maybe using a batch file:
set VRAY_NUM_THREADS=6
start /node 0 /affinity 3f 3dsmax.exe
This will start 3ds max using six computation threads and running on the first six CPU cores. If Node0 on your system use different number and/or topology of the CPU cores, the “VRAY_NUM_THREADS=6” and “/affinity 3f” should be modified.

Finally, VRaySpawner launching could be automated using simple windows shortcut. There is also ability to use .ini configuration file instead of command line for VRaySpawner.