Hi all,
I've installed VRay Standalone on a cluster based on Ubuntu.
I've bought "10 VRay Standalone Pack" licenses, so I have 10x10 licenses for DR.
How does my system work?
In a loop I run a script which renders frame by frame in following steps:
1. Take 10 nodes for rendering the current frame.
2. The 1st one is master, other 9 are slaves.
3. Run on each slave vray -server.
4. Run vray -distributed=1 (with other params) on master node.
5. Rendering image on 10 hosts.
6. After all work killall -9 vray on each slave.
What is happen when I start rendering 6 frames?
ad. 1. Script reserves 60 nodes (6x10) for my jobs.
ad. 3. Script runs 54 vray -server which take 54 licences from my licensing server (I can see IPs of current nodes).
ad. 4. Script runs 6 vray in master mode which take 6 licences.
ad. 5. Here I have a problem.
The first job has in its log:
Preparing scene for rendering...: 0.00% [ 0h 0m 0.3s] [ 0h 0m 0.0s est]
[2013/Feb/7|15:17:18] Preparing scene for rendering...: done [ 0h 0m 0.0s]
[2013/Feb/7|15:17:18] dr host: 192.168.10.216
[2013/Feb/7|15:17:18] dr host: 192.168.10.217
[2013/Feb/7|15:17:18] dr host: 192.168.10.218
[2013/Feb/7|15:17:18] dr host: 192.168.10.219
[2013/Feb/7|15:17:18] dr host: 192.168.10.220
[2013/Feb/7|15:17:18] dr host: 192.168.10.221
[2013/Feb/7|15:17:18] dr host: 192.168.10.222
[2013/Feb/7|15:17:18] dr host: 192.168.10.223
[2013/Feb/7|15:17:18] dr host: 192.168.10.224
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.216:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.217:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.218:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.219:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.220:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.221:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.222:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.223:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.224:20207
[2013/Feb/7|15:17:18] Using 9 hosts for distributed rendering.
[2013/Feb/7|15:17:18] ========================
[2013/Feb/7|15:17:18] Starting frame 0.
The second job:
Preparing scene for rendering...: 0.00% [ 0h 0m 0.2s] [ 0h 0m 0.0s est]
[2013/Feb/7|15:17:18] Preparing scene for rendering...: done [ 0h 0m 0.0s]
[2013/Feb/7|15:17:18] dr host: 192.168.10.164
[2013/Feb/7|15:17:18] dr host: 192.168.10.165
[2013/Feb/7|15:17:18] dr host: 192.168.10.181
[2013/Feb/7|15:17:18] dr host: 192.168.10.182
[2013/Feb/7|15:17:18] dr host: 192.168.10.183
[2013/Feb/7|15:17:18] dr host: 192.168.10.184
[2013/Feb/7|15:17:18] dr host: 192.168.10.185
[2013/Feb/7|15:17:18] dr host: 192.168.10.186
[2013/Feb/7|15:17:18] dr host: 192.168.10.187
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.164:20207
[2013/Feb/7|15:17:18] Host 192.168.10.165:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.181:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.182:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.183:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.184:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.185:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.186:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.187:20207 added to waiting list
[2013/Feb/7|15:17:18] Using 1 hosts for distributed rendering.
[2013/Feb/7|15:17:18] ========================
[2013/Feb/7|15:17:18] Starting frame 0.
And the others:
Preparing scene for rendering...: 0.00% [ 0h 0m 0.3s] [ 0h 0m 0.0s est]
[2013/Feb/7|15:17:18] Preparing scene for rendering...: done [ 0h 0m 0.0s]
[2013/Feb/7|15:17:18] dr host: 192.168.10.127
[2013/Feb/7|15:17:18] dr host: 192.168.10.128
[2013/Feb/7|15:17:18] dr host: 192.168.10.129
[2013/Feb/7|15:17:18] dr host: 192.168.10.130
[2013/Feb/7|15:17:18] dr host: 192.168.10.146
[2013/Feb/7|15:17:18] dr host: 192.168.10.147
[2013/Feb/7|15:17:18] dr host: 192.168.10.148
[2013/Feb/7|15:17:18] dr host: 192.168.10.149
[2013/Feb/7|15:17:18] dr host: 192.168.10.150
[2013/Feb/7|15:17:18] Host 192.168.10.127:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.128:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.129:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.130:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.146:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.147:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.148:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.149:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.150:20207 added to waiting list
[2013/Feb/7|15:17:18] Using 0 hosts for distributed rendering.
[2013/Feb/7|15:17:18] ========================
[2013/Feb/7|15:17:18] Starting frame 0.
And when I reserve only 5 nodes per job (one frame), I'll get:
1st frame: Using 4 hosts for distributed rendering.
2nd frame: Using 4 hosts for distributed rendering.
3rd frame: Using 2 hosts for distributed rendering.
4th and others: Using 0 hosts for distributed rendering.
In licensing server I can see 40 nodes.
Could somebody explain what I'm doing or maybe I don't understand how license server works. I'll be much grateful.
PS. I've been thinking...
I've got license server on one host (for instance 10.0.0.1).
The nodes of the cluster are in a private network (192.168.10.255) and the cluster has one public IP (for instance 10.10.10.2).
On the list of my license server there are:
Sessions list for VRay Standalone/Adv 2.0 GUI RENDER DR Last access
192.168.10.217 0 0 0 1
192.168.10.153 0 0 0 18
192.168.10.128 0 0 0 18
192.168.10.216 0 0 0 1
etc.
Is it possible the license server sees all these master nodes (which run vray -distributed=1) as one machine with the external IP 10.10.10.2 and thus only 10 licences work in DR in the same time?
Notice the other master nodes render frames as well, but without slaves.
PS2. One (maybe) important thing more. The all nodes have shared home directory, where VRay is installed. The nodes see home folder as their own and they simply run it. To clarify. Let's say I have 200 nodes. I had to install VRay only once, but it could be run 200 times.
I've installed VRay Standalone on a cluster based on Ubuntu.
I've bought "10 VRay Standalone Pack" licenses, so I have 10x10 licenses for DR.
How does my system work?
In a loop I run a script which renders frame by frame in following steps:
1. Take 10 nodes for rendering the current frame.
2. The 1st one is master, other 9 are slaves.
3. Run on each slave vray -server.
4. Run vray -distributed=1 (with other params) on master node.
5. Rendering image on 10 hosts.
6. After all work killall -9 vray on each slave.
What is happen when I start rendering 6 frames?
ad. 1. Script reserves 60 nodes (6x10) for my jobs.
ad. 3. Script runs 54 vray -server which take 54 licences from my licensing server (I can see IPs of current nodes).
ad. 4. Script runs 6 vray in master mode which take 6 licences.
ad. 5. Here I have a problem.
The first job has in its log:
Preparing scene for rendering...: 0.00% [ 0h 0m 0.3s] [ 0h 0m 0.0s est]
[2013/Feb/7|15:17:18] Preparing scene for rendering...: done [ 0h 0m 0.0s]
[2013/Feb/7|15:17:18] dr host: 192.168.10.216
[2013/Feb/7|15:17:18] dr host: 192.168.10.217
[2013/Feb/7|15:17:18] dr host: 192.168.10.218
[2013/Feb/7|15:17:18] dr host: 192.168.10.219
[2013/Feb/7|15:17:18] dr host: 192.168.10.220
[2013/Feb/7|15:17:18] dr host: 192.168.10.221
[2013/Feb/7|15:17:18] dr host: 192.168.10.222
[2013/Feb/7|15:17:18] dr host: 192.168.10.223
[2013/Feb/7|15:17:18] dr host: 192.168.10.224
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.216:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.217:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.218:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.219:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.220:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.221:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.222:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.223:20207
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.224:20207
[2013/Feb/7|15:17:18] Using 9 hosts for distributed rendering.
[2013/Feb/7|15:17:18] ========================
[2013/Feb/7|15:17:18] Starting frame 0.
The second job:
Preparing scene for rendering...: 0.00% [ 0h 0m 0.2s] [ 0h 0m 0.0s est]
[2013/Feb/7|15:17:18] Preparing scene for rendering...: done [ 0h 0m 0.0s]
[2013/Feb/7|15:17:18] dr host: 192.168.10.164
[2013/Feb/7|15:17:18] dr host: 192.168.10.165
[2013/Feb/7|15:17:18] dr host: 192.168.10.181
[2013/Feb/7|15:17:18] dr host: 192.168.10.182
[2013/Feb/7|15:17:18] dr host: 192.168.10.183
[2013/Feb/7|15:17:18] dr host: 192.168.10.184
[2013/Feb/7|15:17:18] dr host: 192.168.10.185
[2013/Feb/7|15:17:18] dr host: 192.168.10.186
[2013/Feb/7|15:17:18] dr host: 192.168.10.187
[2013/Feb/7|15:17:18] Connected to render host 192.168.10.164:20207
[2013/Feb/7|15:17:18] Host 192.168.10.165:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.181:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.182:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.183:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.184:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.185:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.186:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.187:20207 added to waiting list
[2013/Feb/7|15:17:18] Using 1 hosts for distributed rendering.
[2013/Feb/7|15:17:18] ========================
[2013/Feb/7|15:17:18] Starting frame 0.
And the others:
Preparing scene for rendering...: 0.00% [ 0h 0m 0.3s] [ 0h 0m 0.0s est]
[2013/Feb/7|15:17:18] Preparing scene for rendering...: done [ 0h 0m 0.0s]
[2013/Feb/7|15:17:18] dr host: 192.168.10.127
[2013/Feb/7|15:17:18] dr host: 192.168.10.128
[2013/Feb/7|15:17:18] dr host: 192.168.10.129
[2013/Feb/7|15:17:18] dr host: 192.168.10.130
[2013/Feb/7|15:17:18] dr host: 192.168.10.146
[2013/Feb/7|15:17:18] dr host: 192.168.10.147
[2013/Feb/7|15:17:18] dr host: 192.168.10.148
[2013/Feb/7|15:17:18] dr host: 192.168.10.149
[2013/Feb/7|15:17:18] dr host: 192.168.10.150
[2013/Feb/7|15:17:18] Host 192.168.10.127:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.128:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.129:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.130:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.146:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.147:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.148:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.149:20207 added to waiting list
[2013/Feb/7|15:17:18] Host 192.168.10.150:20207 added to waiting list
[2013/Feb/7|15:17:18] Using 0 hosts for distributed rendering.
[2013/Feb/7|15:17:18] ========================
[2013/Feb/7|15:17:18] Starting frame 0.
And when I reserve only 5 nodes per job (one frame), I'll get:
1st frame: Using 4 hosts for distributed rendering.
2nd frame: Using 4 hosts for distributed rendering.
3rd frame: Using 2 hosts for distributed rendering.
4th and others: Using 0 hosts for distributed rendering.
In licensing server I can see 40 nodes.
Could somebody explain what I'm doing or maybe I don't understand how license server works. I'll be much grateful.
PS. I've been thinking...
I've got license server on one host (for instance 10.0.0.1).
The nodes of the cluster are in a private network (192.168.10.255) and the cluster has one public IP (for instance 10.10.10.2).
On the list of my license server there are:
Sessions list for VRay Standalone/Adv 2.0 GUI RENDER DR Last access
192.168.10.217 0 0 0 1
192.168.10.153 0 0 0 18
192.168.10.128 0 0 0 18
192.168.10.216 0 0 0 1
etc.
Is it possible the license server sees all these master nodes (which run vray -distributed=1) as one machine with the external IP 10.10.10.2 and thus only 10 licences work in DR in the same time?
Notice the other master nodes render frames as well, but without slaves.
PS2. One (maybe) important thing more. The all nodes have shared home directory, where VRay is installed. The nodes see home folder as their own and they simply run it. To clarify. Let's say I have 200 nodes. I had to install VRay only once, but it could be run 200 times.
Comment