VIDEO of using vrayspawner.exe on 72 Threads Xeon E5 - Slight set-up problem..

[Pete3d]

Hi,

I recently upgraded my workstation to 2xXeons 2696v3 (36 cores total/ 72 threads) which is the equivalent of the v2699v3.

I started doing tests what OS and workflows are best to utilize 100% of the processor power for vray renderings. The OSes to go against each other are Windows 2012 Server vs Centos 7.1.
Lets start with Windows for now.


WINDOWS 2012 Server
--------------------
Windows seems to put my cpus into two groups I noticed in the Task Manager. -> group0 and group1 with each 36 threads. Total it displays 72 threads though. All good there

I noticed in the couple of days of testing that somehow application processes tend to be assigned to either one of those groups to perform things. Vray in Maya seems to render by default on only one of those cpu's at 100 percent while the other is idle. What a disappointment right there , right ?

Mr. petrov from this forum recently gave me a hint to an existing forum article which mentioned that one can utilize vrayspawner.exe under windows to utilize all the xeon cpus ( numa nodes) and get maximum performance. So I did. But I noticed some small annoyance when setting up the vrayspawner. Not sure If I am doing something wrong here... ???
Have a look at my video recording ... it demonstrats the usage of the vrayspawner and my issues with it

https://youtu.be/wJDh4wr1JtQ

Overall the rendering of the testframe in my video under Windows 2012 Server was around 1min 7 sec in the end. I cut the video short, cos its boring watching buckets.

Centos 7.1
--------------------
I noticed that linux is not havnig this group0 , 1 crap and lists all the cpus and threads as ONE unit! Great I thought.
I rendered the same image as above (sorry no video here for now and found that 100 % of the CPU are being used ( so basically all 72 Threads are at 100 %) . I thought -> WOW -> great! initially - but somehow the image only finshes rendering around 2 min. HOW COME ??? Any ideas why it takes so long ? I expected around 1 min ....

thanks
Peter

[ivaylo.ivanov]

The vrayspawner has an option to launch the rendering on specified processor group, so you don't need to do this affinity mask manually. But AFAIK, it's not in 3.0

http://docs.chaosgroup.com/display/V...andlineoptions

[t.petrov]

Can you tell us the version of V-Ray you are using for both OSes?

We've made some fixes to the recent Linux builds that might affect scaling.

[Pete3d]

Can you tell us the version of V-Ray you are using for both OSes?

We've made some fixes to the recent Linux builds that might affect scaling.

HI there,
For Windows 2012 I am using 3.00.1 which is the official release.
For Linux I was using the latest nightly build from 2 days ago and the official 3.00.1 as well.

What I am really shocked to see was the linux performance of that render which I used in the video. It was 2 min on Centos VS 1 min on windows.
That kinda left me baffled...
Btw, I checked if centos was throttling the speed (read in some forums that linux likes to do it on occasions), but it wasn't. All threads were performing at around 2.8Ghz Turbo. So yeah, no idea there. If I won't find solution for Linux, I will stick to windows.

[Pete3d]

The vrayspawner has an option to launch the rendering on specified processor group, so you don't need to do this affinity mask manually. But AFAIK, it's not in 3.0

http://docs.chaosgroup.com/display/V...andlineoptions

hmm...the thing is that its quiet random what group (either group0, or group1) maya is on when running. Its usually not fixed by default.
Only if I know that maya is on group0 I guess I could fix assign the spawner to group1 in my example.

[t.petrov]

Interesting. Generally the linux builds are a bit slower because on linux we use GCC which generates a bit slower code, but the difference should not be 2x.

Could you please do some tests so we can check what is the scaling on you machine?

For the tests you need to export the scene as vrscene file, to make it easier for testing.
The scene shouldn't be very high resolution (720p would probably be best) and should render for at least 1-2 mins with all cores.
Then you should execute the following commands:

1. <path to vray>/vray -sceneFile=<path to your vrscene>
2. <path to vray>/vray -sceneFile=<path to your vrscene> -numThreads=36
3. <path to vray>/vray -sceneFile=<path to your vrscene> -numThreads=18
4. numactl -N 0 <path to vray>/vray -sceneFile=<path to your vrscene> -numThreads=36
5. numactl -N 0 <path to vray>/vray -sceneFile=<path to your vrscene> -numThreads=18

Run the commands several times to make sure that there is not much variation.
Post the times here.

I'm running some tests on our 32 thread machine and I'm seeing linear scaling with the current build from 16 to 32 threads.
Tomorrow I'll run some tests on the beefiest machine we have (40 threads) and also on another 32 thread windows machine.

[t.petrov]

hmm...the thing is that its quiet random what group (either group0, or group1) maya is on when running. Its usually not fixed by default.
Only if I know that maya is on group0 I guess I could fix assign the spawner to group1 in my example.

We think that the easiest solution for this problem will be to create a bat file that will start both Maya and V-Ray Standalone in the proper groups.
You can do this by using the start command. You should pass the /node 0 or /node 1 parameter for Maya and V-Ray Standalone respectively.

Hope it helps.

[Pete3d]

V-Ray Standalone, version 3.00.01 for x64
V-Ray core version is 3.05.07

1. vray -sceneFile=/home/user/Downloads/vrscene/myscene.vrscene
-> System Monitor CPU shows 100% utilization

[2015/Jun/23|04:39:09] Starting frame 1270.
[2015/Jun/23|04:39:10] Preparing scene for frame...: done [ 0h 0m 0.1s]
[2015/Jun/23|04:39:10] Compiling geometry...: done [ 0h 0m 0.1s]
[2015/Jun/23|04:39:10] Building Embree static raycaster...
[2015/Jun/23|04:39:10] -- Embree BVH type is bvh4.spatialsplit
[2015/Jun/23|04:39:10] -- Embree triangle representation is triangle4
[2015/Jun/23|04:39:10] Embree static BVH stats:
[2015/Jun/23|04:39:10] warning: Performance loss: number of rendering threads (72) is greater than number of light cache passes (64); some threads will be idle.
[2015/Jun/23|04:39:10] Tracing 1000000 image samples for light cache in 64 passes.
[2015/Jun/23|04:39:10] Building light cache...: done [ 0h 0m 0.4s]
[2015/Jun/23|04:39:13] Merging light cache passes...: done [ 0h 0m 2.3s]
[2015/Jun/23|04:39:13] Light cache contains 6936 samples.
[2015/Jun/23|04:39:13] Prefiltering light cache...: done [ 0h 0m 0.0s]
[2015/Jun/23|04:41:30] Rendering image...: done [ 0h 2m 16.9s]
[2015/Jun/23|04:41:30] Number of raycasts: 1461522642 (87.11 per pixel)
[2015/Jun/23|04:41:30] Camera rays: 523416176 (31.20 per pixel)
[2015/Jun/23|04:41:30] Shadow rays: 527400408 (31.44 per pixel)
[2015/Jun/23|04:41:30] GI rays: 426143078 (25.40 per pixel)
[2015/Jun/23|04:41:30] Reflection rays: 0 (0.00 per pixel)
[2015/Jun/23|04:41:30] Refraction rays: 0 (0.00 per pixel)
[2015/Jun/23|04:41:30] Unshaded rays: 0 (0.00 per pixel)
[2015/Jun/23|04:41:30] Number of light evaluations: 444785596 (26.51 per pixel)
[2015/Jun/23|04:41:30] Number of intersectable primitives: 327819
[2015/Jun/23|04:41:30] SD triangles: 327818
[2015/Jun/23|04:41:30] MB triangles: 0
[2015/Jun/23|04:41:30] Static primitives: 0
[2015/Jun/23|04:41:30] Moving primitives: 0
[2015/Jun/23|04:41:30] Infinite primitives: 1
[2015/Jun/23|04:41:30] Static hair segments: 0
[2015/Jun/23|04:41:30] Moving hair segments: 0
[2015/Jun/23|04:41:59] Successfully written image file "/home/user/maya/projects/default/images/tmp/testrender2015.png"
[2015/Jun/23|04:41:59] Frame took 169.79 s.
Another run -> 167 sec, 169 sec, 171 sec

2. vray -sceneFile=/home/user/Downloads/vrscene/myscene.vrscene -numThreads=36
-> System Monitor CPU shows 50% utilization

Starting frame 1270.
[2015/Jun/23|04:03:05] Preparing scene for frame...: done [ 0h 0m 0.1s]
[2015/Jun/23|04:03:06] Compiling geometry...: done [ 0h 0m 0.1s]
[2015/Jun/23|04:03:06] Building Embree static raycaster...
[2015/Jun/23|04:03:06] -- Embree BVH type is bvh4.spatialsplit
[2015/Jun/23|04:03:06] -- Embree triangle representation is triangle4
[2015/Jun/23|04:03:06] Embree static BVH stats:
[2015/Jun/23|04:03:06] Tracing 1000000 image samples for light cache in 64 passes.
[2015/Jun/23|04:03:07] Building light cache...: done [ 0h 0m 0.7s]
[2015/Jun/23|04:03:08] Merging light cache passes...: done [ 0h 0m 1.6s]
[2015/Jun/23|04:03:08] Light cache contains 6936 samples.
[2015/Jun/23|04:03:08] Prefiltering light cache...: done [ 0h 0m 0.0s]
[2015/Jun/23|04:05:18] Rendering image...: done [ 0h 2m 10.1s]
[2015/Jun/23|04:05:19] Number of raycasts: 1460965149 (87.08 per pixel)
[2015/Jun/23|04:05:19] Camera rays: 522858646 (31.16 per pixel)
[2015/Jun/23|04:05:19] Shadow rays: 527400426 (31.44 per pixel)
[2015/Jun/23|04:05:19] GI rays: 426143105 (25.40 per pixel)
[2015/Jun/23|04:05:19] Reflection rays: 0 (0.00 per pixel)
[2015/Jun/23|04:05:19] Refraction rays: 0 (0.00 per pixel)
[2015/Jun/23|04:05:19] Unshaded rays: 0 (0.00 per pixel)
[2015/Jun/23|04:05:19] Number of light evaluations: 444785620 (26.51 per pixel)
[2015/Jun/23|04:05:19] Number of intersectable primitives: 327819
[2015/Jun/23|04:05:19] SD triangles: 327818
[2015/Jun/23|04:05:19] MB triangles: 0
[2015/Jun/23|04:05:19] Static primitives: 0
[2015/Jun/23|04:05:19] Moving primitives: 0
[2015/Jun/23|04:05:19] Infinite primitives: 1
[2015/Jun/23|04:05:19] Static hair segments: 0
[2015/Jun/23|04:05:19] Moving hair segments: 0
[2015/Jun/23|04:05:49] Successfully written image file "/home/user/maya/projects/default/images/tmp/testrender2015.png"
[2015/Jun/23|04:05:49] Frame took 163.70 s.
Another run -> 166.95 sec


3. vray -sceneFile=/home/user/Downloads/vrscene/myscene.vrscene -numThreads=18
-> System Monitor CPU shows 25% utilization

Starting frame 1270.
[2015/Jun/23|04:07:08] Preparing scene for frame...: done [ 0h 0m 0.1s]
[2015/Jun/23|04:07:08] Compiling geometry...: done [ 0h 0m 0.1s]
[2015/Jun/23|04:07:08] Building Embree static raycaster...
[2015/Jun/23|04:07:08] -- Embree BVH type is bvh4.spatialsplit
[2015/Jun/23|04:07:08] -- Embree triangle representation is triangle4
[2015/Jun/23|04:07:08] Embree static BVH stats:
[2015/Jun/23|04:07:09] Tracing 1000000 image samples for light cache in 64 passes.
[2015/Jun/23|04:07:10] Building light cache...: done [ 0h 0m 1.2s]
[2015/Jun/23|04:07:11] Merging light cache passes...: done [ 0h 0m 0.9s]
[2015/Jun/23|04:07:11] Light cache contains 6936 samples.
[2015/Jun/23|04:07:11] Prefiltering light cache...: done [ 0h 0m 0.0s]
[2015/Jun/23|04:09:59] Rendering image...: done [ 0h 2m 48.4s]
[2015/Jun/23|04:10:00] Number of raycasts: 1460686316 (87.06 per pixel)
[2015/Jun/23|04:10:00] Camera rays: 522579843 (31.15 per pixel)
[2015/Jun/23|04:10:00] Shadow rays: 527400409 (31.44 per pixel)
[2015/Jun/23|04:10:00] GI rays: 426143093 (25.40 per pixel)
[2015/Jun/23|04:10:00] Reflection rays: 0 (0.00 per pixel)
[2015/Jun/23|04:10:00] Refraction rays: 0 (0.00 per pixel)
[2015/Jun/23|04:10:00] Unshaded rays: 0 (0.00 per pixel)
[2015/Jun/23|04:10:00] Number of light evaluations: 444785606 (26.51 per pixel)
[2015/Jun/23|04:10:00] Number of intersectable primitives: 327819
[2015/Jun/23|04:10:00] SD triangles: 327818
[2015/Jun/23|04:10:00] MB triangles: 0
[2015/Jun/23|04:10:00] Static primitives: 0
[2015/Jun/23|04:10:00] Moving primitives: 0
[2015/Jun/23|04:10:00] Infinite primitives: 1
[2015/Jun/23|04:10:00] Static hair segments: 0
[2015/Jun/23|04:10:00] Moving hair segments: 0
[2015/Jun/23|04:10:28] Successfully written image file "/home/user/maya/projects/default/images/tmp/testrender2015.png"
[2015/Jun/23|04:10:28] Frame took 199.89 s.

[Pete3d]

Continuation...

V-Ray Standalone, version 3.00.01 for x64
V-Ray core version is 3.05.07

4. numactl -N 0 ./vray -sceneFile=/home/user/Downloads/vrscene/myscene.vrscene -numThreads=36
-> System Monitor CPU shows 50% utilization

[2015/Jun/23|04:20:45] Starting frame 1270.
[2015/Jun/23|04:20:45] Preparing scene for frame...: done [ 0h 0m 0.1s]
[2015/Jun/23|04:20:45] Compiling geometry...: done [ 0h 0m 0.1s]
[2015/Jun/23|04:20:45] Building Embree static raycaster...
[2015/Jun/23|04:20:45] -- Embree BVH type is bvh4.spatialsplit
[2015/Jun/23|04:20:45] -- Embree triangle representation is triangle4
[2015/Jun/23|04:20:45] Embree static BVH stats:
[2015/Jun/23|04:20:46] Tracing 1000000 image samples for light cache in 64 passes.
[2015/Jun/23|04:20:46] Building light cache...: done [ 0h 0m 0.8s]
[2015/Jun/23|04:20:47] Merging light cache passes...: done [ 0h 0m 0.8s]
[2015/Jun/23|04:20:47] Light cache contains 6936 samples.
[2015/Jun/23|04:20:47] Prefiltering light cache...: done [ 0h 0m 0.0s]
[2015/Jun/23|04:22:44] Rendering image...: done [ 0h 1m 56.4s]
[2015/Jun/23|04:22:44] Number of raycasts: 1460965111 (87.08 per pixel)
[2015/Jun/23|04:22:44] Camera rays: 522858627 (31.16 per pixel)
[2015/Jun/23|04:22:44] Shadow rays: 527400415 (31.44 per pixel)
[2015/Jun/23|04:22:44] GI rays: 426143087 (25.40 per pixel)
[2015/Jun/23|04:22:44] Reflection rays: 0 (0.00 per pixel)
[2015/Jun/23|04:22:44] Refraction rays: 0 (0.00 per pixel)
[2015/Jun/23|04:22:44] Unshaded rays: 0 (0.00 per pixel)
[2015/Jun/23|04:22:44] Number of light evaluations: 444785611 (26.51 per pixel)
[2015/Jun/23|04:22:44] Number of intersectable primitives: 327819
[2015/Jun/23|04:22:44] SD triangles: 327818
[2015/Jun/23|04:22:44] MB triangles: 0
[2015/Jun/23|04:22:44] Static primitives: 0
[2015/Jun/23|04:22:44] Moving primitives: 0
[2015/Jun/23|04:22:44] Infinite primitives: 1
[2015/Jun/23|04:22:44] Static hair segments: 0
[2015/Jun/23|04:22:44] Moving hair segments: 0
[2015/Jun/23|04:23:13] Successfully written image file "/home/user/maya/projects/default/images/tmp/testrender2015.png"
[2015/Jun/23|04:23:13] Frame took 147.88 s.

5. numactl -N 0 ./vray -sceneFile=/home/user/Downloads/vrscene/myscene.vrscene -numThreads=18
-> System Monitor CPU shows 25% utilization

[2015/Jun/23|04:24:45] Starting frame 1270.
[2015/Jun/23|04:24:45] Preparing scene for frame...: done [ 0h 0m 0.1s]
[2015/Jun/23|04:24:45] Compiling geometry...: done [ 0h 0m 0.1s]
[2015/Jun/23|04:24:45] Building Embree static raycaster...
[2015/Jun/23|04:24:45] -- Embree BVH type is bvh4.spatialsplit
[2015/Jun/23|04:24:45] -- Embree triangle representation is triangle4
[2015/Jun/23|04:24:45] Embree static BVH stats:
[2015/Jun/23|04:24:45] Tracing 1000000 image samples for light cache in 64 passes.
[2015/Jun/23|04:24:46] Building light cache...: done [ 0h 0m 1.2s]
[2015/Jun/23|04:24:47] Merging light cache passes...: done [ 0h 0m 0.5s]
[2015/Jun/23|04:24:47] Light cache contains 6936 samples.
[2015/Jun/23|04:24:47] Prefiltering light cache...: done [ 0h 0m 0.0s]
[2015/Jun/23|04:27:23] Rendering image...: done [ 0h 2m 35.9s]
[2015/Jun/23|04:27:24] Number of raycasts: 1460686254 (87.06 per pixel)
[2015/Jun/23|04:27:24] Camera rays: 522579810 (31.15 per pixel)
[2015/Jun/23|04:27:24] Shadow rays: 527400391 (31.44 per pixel)
[2015/Jun/23|04:27:24] GI rays: 426143073 (25.40 per pixel)
[2015/Jun/23|04:27:24] Reflection rays: 0 (0.00 per pixel)
[2015/Jun/23|04:27:24] Refraction rays: 0 (0.00 per pixel)
[2015/Jun/23|04:27:24] Unshaded rays: 0 (0.00 per pixel)
[2015/Jun/23|04:27:24] Number of light evaluations: 444785590 (26.51 per pixel)
[2015/Jun/23|04:27:24] Number of intersectable primitives: 327819
[2015/Jun/23|04:27:24] SD triangles: 327818
[2015/Jun/23|04:27:24] MB triangles: 0
[2015/Jun/23|04:27:24] Static primitives: 0
[2015/Jun/23|04:27:24] Moving primitives: 0
[2015/Jun/23|04:27:24] Infinite primitives: 1
[2015/Jun/23|04:27:24] Static hair segments: 0
[2015/Jun/23|04:27:24] Moving hair segments: 0
[2015/Jun/23|04:27:49] Successfully written image file "/home/user/maya/projects/default/images/tmp/testrender2015.png"
[2015/Jun/23|04:27:49] Frame took 183.92 s.

[Pete3d]

V-Ray Standalone, version 3.25.01 for x64
Build 25931 from Jun 23 2015, 02:43:50
V-Ray core version is 3.25.01

Note: I had a warning message with the newest nightly build and my vrscene.
-> warning: Unknown property "embreeHighPrec" in object "vraySettingsRaycaster"


1. vray -sceneFile=/home/user/Downloads/vrscene/myscene.vrscene
-> System Monitor CPU shows 100% utilization

[2015/Jun/23|05:10:10] Starting frame 1270.
[2015/Jun/23|05:10:10] Preparing scene for frame...: done [ 0h 0m 0.1s]
[2015/Jun/23|05:10:10] Compiling geometry...: done [ 0h 0m 0.1s]
[2015/Jun/23|05:10:10] Using embree ray tracing.
[2015/Jun/23|05:10:10] Building embree static trees took 244 milliseconds
[2015/Jun/23|05:10:10] Building embree static accelerator ...: done [ 0h 0m 0.3s]
[2015/Jun/23|05:10:11] warning: Performance loss: number of rendering threads (72) is greater than number of light cache passes (64); some threads will be idle.
[2015/Jun/23|05:10:11] Tracing 1000000 image samples for light cache in 64 passes.
[2015/Jun/23|05:10:12] Building light cache...: done [ 0h 0m 1.1s]
[2015/Jun/23|05:10:13] Merging light cache passes...: done [ 0h 0m 0.3s]
[2015/Jun/23|05:10:13] Light cache contains 6875 samples.
[2015/Jun/23|05:10:13] Light cache takes 5.2 MB.
[2015/Jun/23|05:10:13] Prefiltering light cache...: done [ 0h 0m 0.0s]
[2015/Jun/23|05:10:13] Average rays per light cache sample: 122.46 (min 1, max 470)
[2015/Jun/23|05:11:59] Rendering image...: done [ 0h 1m 46.6s]
[2015/Jun/23|05:12:00] Number of raycasts: 1493356092 (89.01 per pixel)
[2015/Jun/23|05:12:00] Camera rays: 533206645 (31.78 per pixel)
[2015/Jun/23|05:12:00] Shadow rays: 539686353 (32.17 per pixel)
[2015/Jun/23|05:12:00] GI rays: 436907880 (26.04 per pixel)
[2015/Jun/23|05:12:00] Reflection rays: 0 (0.00 per pixel)
[2015/Jun/23|05:12:00] Refraction rays: 0 (0.00 per pixel)
[2015/Jun/23|05:12:00] Unshaded rays: 0 (0.00 per pixel)
[2015/Jun/23|05:12:00] Light cache utilization: 78.48%
[2015/Jun/23|05:12:00] Number of light evaluations: 456672841 (27.22 per pixel)
[2015/Jun/23|05:12:00] Number of intersectable primitives: 327819
[2015/Jun/23|05:12:00] SD triangles: 327818
[2015/Jun/23|05:12:00] MB triangles: 0
[2015/Jun/23|05:12:00] Static primitives: 0
[2015/Jun/23|05:12:00] Moving primitives: 0
[2015/Jun/23|05:12:00] Infinite primitives: 1
[2015/Jun/23|05:12:00] Static hair segments: 0
[2015/Jun/23|05:12:00] Moving hair segments: 0
[2015/Jun/23|05:12:26] Successfully written image file "/home/user/maya/projects/default/images/tmp/testrender2015.png"
[2015/Jun/23|05:12:26] Frame took 135.86 s.
Another run 143sec


2. vray -sceneFile=/home/user/Downloads/vrscene/myscene.vrscene -numThreads=36
-> System Monitor CPU shows 50% utilization

[2015/Jun/23|05:13:43] Starting frame 1270.
[2015/Jun/23|05:13:43] Preparing scene for frame...: done [ 0h 0m 0.1s]
[2015/Jun/23|05:13:43] Compiling geometry...: done [ 0h 0m 0.1s]
[2015/Jun/23|05:13:43] Using embree ray tracing.
[2015/Jun/23|05:13:43] Building embree static trees took 308 milliseconds
[2015/Jun/23|05:13:43] Building embree static accelerator ...: done [ 0h 0m 0.3s]
[2015/Jun/23|05:13:44] Tracing 1000000 image samples for light cache in 64 passes.
[2015/Jun/23|05:13:45] Building light cache...: done [ 0h 0m 1.1s]
[2015/Jun/23|05:13:45] Merging light cache passes...: done [ 0h 0m 0.3s]
[2015/Jun/23|05:13:45] Light cache contains 6875 samples.
[2015/Jun/23|05:13:45] Light cache takes 5.2 MB.
[2015/Jun/23|05:13:45] Prefiltering light cache...: done [ 0h 0m 0.0s]
[2015/Jun/23|05:13:45] Average rays per light cache sample: 122.46 (min 1, max 470)
[2015/Jun/23|05:15:26] Rendering image...: done [ 0h 1m 40.6s]
[2015/Jun/23|05:15:27] Number of raycasts: 1492798547 (88.98 per pixel)
[2015/Jun/23|05:15:27] Camera rays: 532649081 (31.75 per pixel)
[2015/Jun/23|05:15:27] Shadow rays: 539686367 (32.17 per pixel)
[2015/Jun/23|05:15:27] GI rays: 436907889 (26.04 per pixel)
[2015/Jun/23|05:15:27] Reflection rays: 0 (0.00 per pixel)
[2015/Jun/23|05:15:27] Refraction rays: 0 (0.00 per pixel)
[2015/Jun/23|05:15:27] Unshaded rays: 0 (0.00 per pixel)
[2015/Jun/23|05:15:27] Light cache utilization: 78.48%
[2015/Jun/23|05:15:27] Number of light evaluations: 456672843 (27.22 per pixel)
[2015/Jun/23|05:15:27] Number of intersectable primitives: 327819
[2015/Jun/23|05:15:27] SD triangles: 327818
[2015/Jun/23|05:15:27] MB triangles: 0
[2015/Jun/23|05:15:27] Static primitives: 0
[2015/Jun/23|05:15:27] Moving primitives: 0
[2015/Jun/23|05:15:27] Infinite primitives: 1
[2015/Jun/23|05:15:27] Static hair segments: 0
[2015/Jun/23|05:15:27] Moving hair segments: 0
[2015/Jun/23|05:15:55] Successfully written image file "/home/user/maya/projects/default/images/tmp/testrender2015.png"
[2015/Jun/23|05:15:55] Frame took 132.21 s.
Another run 142.52 sec

3. vray -sceneFile=/home/user/Downloads/vrscene/myscene.vrscene -numThreads=18
-> System Monitor CPU shows 25% utilization

[2015/Jun/23|05:19:46] Starting frame 1270.
[2015/Jun/23|05:19:46] Preparing scene for frame...: done [ 0h 0m 0.1s]
[2015/Jun/23|05:19:46] Compiling geometry...: done [ 0h 0m 0.1s]
[2015/Jun/23|05:19:46] Using embree ray tracing.
[2015/Jun/23|05:19:47] Building embree static trees took 267 milliseconds
[2015/Jun/23|05:19:47] Building embree static accelerator ...: done [ 0h 0m 0.3s]
[2015/Jun/23|05:19:47] Tracing 1000000 image samples for light cache in 64 passes.
[2015/Jun/23|05:19:49] Building light cache...: done [ 0h 0m 2.1s]
[2015/Jun/23|05:19:50] Merging light cache passes...: done [ 0h 0m 0.2s]
[2015/Jun/23|05:19:50] Light cache contains 6875 samples.
[2015/Jun/23|05:19:50] Light cache takes 5.2 MB.
[2015/Jun/23|05:19:50] Prefiltering light cache...: done [ 0h 0m 0.0s]
[2015/Jun/23|05:19:50] Average rays per light cache sample: 122.46 (min 1, max 470)
[2015/Jun/23|05:22:18] Rendering image...: done [ 0h 2m 28.2s]
[2015/Jun/23|05:22:18] Number of raycasts: 1492519827 (88.96 per pixel)
[2015/Jun/23|05:22:18] Camera rays: 532370311 (31.73 per pixel)
[2015/Jun/23|05:22:18] Shadow rays: 539686406 (32.17 per pixel)
[2015/Jun/23|05:22:18] GI rays: 436907901 (26.04 per pixel)
[2015/Jun/23|05:22:18] Reflection rays: 0 (0.00 per pixel)
[2015/Jun/23|05:22:18] Refraction rays: 0 (0.00 per pixel)
[2015/Jun/23|05:22:18] Unshaded rays: 0 (0.00 per pixel)
[2015/Jun/23|05:22:18] Light cache utilization: 78.48%
[2015/Jun/23|05:22:18] Number of light evaluations: 456672859 (27.22 per pixel)
[2015/Jun/23|05:22:18] Number of intersectable primitives: 327819
[2015/Jun/23|05:22:18] SD triangles: 327818
[2015/Jun/23|05:22:18] MB triangles: 0
[2015/Jun/23|05:22:18] Static primitives: 0
[2015/Jun/23|05:22:18] Moving primitives: 0
[2015/Jun/23|05:22:18] Infinite primitives: 1
[2015/Jun/23|05:22:18] Static hair segments: 0
[2015/Jun/23|05:22:18] Moving hair segments: 0
[2015/Jun/23|05:22:46] Successfully written image file "/home/user/maya/projects/default/images/tmp/testrender2015.png"
[2015/Jun/23|05:22:46] Frame took 179.42 s.
Another run 179.60 sec

[Pete3d]

Continuation

V-Ray Standalone, version 3.25.01 for x64
Build 25931 from Jun 23 2015, 02:43:50
V-Ray core version is 3.25.01

4. numactl -N 0 ./vray -sceneFile=/home/user/Downloads/vrscene/myscene.vrscene -numThreads=36
-> System Monitor CPU shows 50% utilization

[2015/Jun/23|05:27:15] Starting frame 1270.
[2015/Jun/23|05:27:15] Preparing scene for frame...: done [ 0h 0m 0.1s]
[2015/Jun/23|05:27:15] Compiling geometry...: done [ 0h 0m 0.1s]
[2015/Jun/23|05:27:15] Using embree ray tracing.
[2015/Jun/23|05:27:15] Building embree static trees took 327 milliseconds
[2015/Jun/23|05:27:15] Building embree static accelerator ...: done [ 0h 0m 0.3s]
[2015/Jun/23|05:27:16] Tracing 1000000 image samples for light cache in 64 passes.
[2015/Jun/23|05:27:17] Building light cache...: done [ 0h 0m 1.2s]
[2015/Jun/23|05:27:17] Merging light cache passes...: done [ 0h 0m 0.2s]
[2015/Jun/23|05:27:17] Light cache contains 6875 samples.
[2015/Jun/23|05:27:17] Light cache takes 5.2 MB.
[2015/Jun/23|05:27:17] Prefiltering light cache...: done [ 0h 0m 0.0s]
[2015/Jun/23|05:27:17] Average rays per light cache sample: 122.46 (min 1, max 470)
[2015/Jun/23|05:29:09] Rendering image...: done [ 0h 1m 51.7s]
[2015/Jun/23|05:29:10] Number of raycasts: 1492798613 (88.98 per pixel)
[2015/Jun/23|05:29:10] Camera rays: 532649093 (31.75 per pixel)
[2015/Jun/23|05:29:10] Shadow rays: 539686396 (32.17 per pixel)
[2015/Jun/23|05:29:10] GI rays: 436907911 (26.04 per pixel)
[2015/Jun/23|05:29:10] Reflection rays: 0 (0.00 per pixel)
[2015/Jun/23|05:29:10] Refraction rays: 0 (0.00 per pixel)
[2015/Jun/23|05:29:10] Unshaded rays: 0 (0.00 per pixel)
[2015/Jun/23|05:29:10] Light cache utilization: 78.48%
[2015/Jun/23|05:29:10] Number of light evaluations: 456672869 (27.22 per pixel)
[2015/Jun/23|05:29:10] Number of intersectable primitives: 327819
[2015/Jun/23|05:29:10] SD triangles: 327818
[2015/Jun/23|05:29:10] MB triangles: 0
[2015/Jun/23|05:29:10] Static primitives: 0
[2015/Jun/23|05:29:10] Moving primitives: 0
[2015/Jun/23|05:29:10] Infinite primitives: 1
[2015/Jun/23|05:29:10] Static hair segments: 0
[2015/Jun/23|05:29:10] Moving hair segments: 0
[2015/Jun/23|05:29:36] Successfully written image file "/home/user/maya/projects/default/images/tmp/testrender2015.png"
[2015/Jun/23|05:29:36] Frame took 141.62 s.
Another run 141.80 sec


5. numactl -N 0 ./vray -sceneFile=/home/user/Downloads/vrscene/myscene.vrscene -numThreads=18
-> System Monitor CPU shows 25% utilization

[2015/Jun/23|05:34:02] Starting frame 1270.
[2015/Jun/23|05:34:02] Preparing scene for frame...: done [ 0h 0m 0.1s]
[2015/Jun/23|05:34:02] Compiling geometry...: done [ 0h 0m 0.1s]
[2015/Jun/23|05:34:02] Using embree ray tracing.
[2015/Jun/23|05:34:02] Building embree static trees took 291 milliseconds
[2015/Jun/23|05:34:02] Building embree static accelerator ...: done [ 0h 0m 0.3s]
[2015/Jun/23|05:34:02] Tracing 1000000 image samples for light cache in 64 passes.
[2015/Jun/23|05:34:05] Building light cache...: done [ 0h 0m 2.1s]
[2015/Jun/23|05:34:05] Merging light cache passes...: done [ 0h 0m 0.2s]
[2015/Jun/23|05:34:05] Light cache contains 6875 samples.
[2015/Jun/23|05:34:05] Light cache takes 5.2 MB.
[2015/Jun/23|05:34:05] Prefiltering light cache...: done [ 0h 0m 0.0s]
[2015/Jun/23|05:34:05] Average rays per light cache sample: 122.46 (min 1, max 470)
[2015/Jun/23|05:36:26] Rendering image...: done [ 0h 2m 21.5s]
[2015/Jun/23|05:36:27] Number of raycasts: 1492519745 (88.96 per pixel)
[2015/Jun/23|05:36:27] Camera rays: 532370304 (31.73 per pixel)
[2015/Jun/23|05:36:27] Shadow rays: 539686345 (32.17 per pixel)
[2015/Jun/23|05:36:27] GI rays: 436907880 (26.04 per pixel)
[2015/Jun/23|05:36:27] Reflection rays: 0 (0.00 per pixel)
[2015/Jun/23|05:36:27] Refraction rays: 0 (0.00 per pixel)
[2015/Jun/23|05:36:27] Unshaded rays: 0 (0.00 per pixel)
[2015/Jun/23|05:36:27] Light cache utilization: 78.48%
[2015/Jun/23|05:36:27] Number of light evaluations: 456672835 (27.22 per pixel)
[2015/Jun/23|05:36:27] Number of intersectable primitives: 327819
[2015/Jun/23|05:36:27] SD triangles: 327818
[2015/Jun/23|05:36:27] MB triangles: 0
[2015/Jun/23|05:36:27] Static primitives: 0
[2015/Jun/23|05:36:27] Moving primitives: 0
[2015/Jun/23|05:36:27] Infinite primitives: 1
[2015/Jun/23|05:36:27] Static hair segments: 0
[2015/Jun/23|05:36:27] Moving hair segments: 0
[2015/Jun/23|05:36:54] Successfully written image file "/home/user/maya/projects/default/images/tmp/testrender2015.png"
[2015/Jun/23|05:36:54] Frame took 172.40 s.
Another run 174.08 sec

[Pete3d]

I put the values into Excel.
It looks like the threads don't scale well beyond 32 threads. Actually they become worse! It takes longer to render on 72 threads compared to 36 threads

[t.petrov]

Very unpleasant.

Can you try one last scenario:

$ numactl -N 1 vray -server -numThreads=36
$ numactl -N 0 vray -sceneFile=<path_to_vrscene> -distributed=1 -renderHost=localhost -numThreads=36

Also can you share the scene, so I can try to render it on our machines?

[Pete3d]

V-Ray Standalone, version 3.25.01 for x64
Build 25931 from Jun 23 2015, 02:43:50
V-Ray core version is 3.25.01

$ numactl -N 0 ./vray -sceneFile=/home/user/Downloads/vrscene/myscene.vrscene -distributed=1 -renderHost=localhost -numThreads=36

[2015/Jun/23|12:48:07] Starting frame 1270.
[2015/Jun/23|12:48:07] Scene transferred to localhost.localdomain:20207.6s] [ 0h 0m 0.0s est]
[2015/Jun/23|12:48:07] Preparing scene for frame...: done [ 0h 0m 0.1s]
[2015/Jun/23|12:48:07] Server 127.0.0.1:20207: Scene loaded; starting renderm 0.0s est]
[2015/Jun/23|12:48:07] Compiling geometry...: done [ 0h 0m 0.1s]
[2015/Jun/23|12:48:07] Using embree ray tracing.
[2015/Jun/23|12:48:07] Server 127.0.0.1:20207: Starting frame 1270
[2015/Jun/23|12:48:07] Building embree static trees took 284 milliseconds
[2015/Jun/23|12:48:07] Building embree static accelerator ...: done [ 0h 0m 0.3s]
[2015/Jun/23|12:48:08] Tracing 1000000 image samples for light cache in 64 passes.
[2015/Jun/23|12:48:09] Building light cache...: done [ 0h 0m 1.2s]
[2015/Jun/23|12:48:10] Merging light cache passes...: done [ 0h 0m 0.2s]
[2015/Jun/23|12:48:10] Light cache contains 6875 samples.
[2015/Jun/23|12:48:10] Light cache takes 5.2 MB.
[2015/Jun/23|12:48:10] Prefiltering light cache...: done [ 0h 0m 0.0s]
[2015/Jun/23|12:48:10] Average rays per light cache sample: 122.46 (min 1, max 470)
[2015/Jun/23|12:49:14] Rendering image...: done [ 0h 1m 4.9s]
[2015/Jun/23|12:49:15] Number of raycasts: 826062723 (49.24 per pixel)
[2015/Jun/23|12:49:15] Camera rays: 314345802 (18.74 per pixel)
[2015/Jun/23|12:49:15] Shadow rays: 287513571 (17.14 per pixel)
[2015/Jun/23|12:49:15] GI rays: 232556499 (13.86 per pixel)
[2015/Jun/23|12:49:15] Reflection rays: 0 (0.00 per pixel)
[2015/Jun/23|12:49:15] Refraction rays: 0 (0.00 per pixel)
[2015/Jun/23|12:49:15] Unshaded rays: 0 (0.00 per pixel)
[2015/Jun/23|12:49:15] Light cache utilization: 79.81%
[2015/Jun/23|12:49:15] Number of light evaluations: 242852414 (14.48 per pixel)
[2015/Jun/23|12:49:15] Number of intersectable primitives: 327819
[2015/Jun/23|12:49:15] SD triangles: 327818
[2015/Jun/23|12:49:15] MB triangles: 0
[2015/Jun/23|12:49:15] Static primitives: 0
[2015/Jun/23|12:49:15] Moving primitives: 0
[2015/Jun/23|12:49:15] Infinite primitives: 1
[2015/Jun/23|12:49:15] Static hair segments: 0
[2015/Jun/23|12:49:15] Moving hair segments: 0
[2015/Jun/23|12:49:42] Successfully written image file "/home/user/maya/projects/default/images/tmp/testrender2015.png"
[2015/Jun/23|12:49:42] Frame took 95.04 s.

[Pete3d]

Your scenario worked perfectly as desired.

How difficult do you think is it to fix the scaling issue beyond 32 threads ? Do you guys need a machine for that ? I am willing to buy you guys a machine ..

(just kidding heheh :P)

Very unpleasant.

Can you try one last scenario:

$ numactl -N 1 vray -server -numThreads=36
$ numactl -N 0 vray -sceneFile=<path_to_vrscene> -distributed=1 -renderHost=localhost -numThreads=36

Also can you share the scene, so I can try to render it on our machines?