Cannot load vrmesh file @ 1001th vrmesh

[Fredrik Averpil]

Hi,

We have a huge amount of vrmeshes in a scene here. When attempting to render this on Windows, it works just fine. On Linux (CentOS 7) however, the first 1000 vrmeshes are loaded fine but then we get "V-Ray warning: Could not load mesh file" for all of the following vrmeshes. Please note, we do *not* get the "Max opened files" error.

So in my /etc/security/limits.conf on the farm machines, I double checked, and I've got the following set:

*         hard    nofile      500000
*         soft    nofile      500000
root      hard    nofile      500000
root      soft    nofile      500000
IRUser    hard    nofile      500000
IRUser    soft    nofile      500000

The user "IRUser" is the one performing renders and it's supposed to be able to load all these vrmeshes, as far as I know. We actually hit the "Too many files opened" a while back on CentOS 6, and fixed it by upping the values in this config file. But now, it seems this setting isn't sufficient.

Do you have any idea what I'm missing?
I'm not aware of any other way of upping the max number of allowed opened files in CentOS 7.

[vlado]

We have a CentOS 7 install here; let us do a few tests.

Best regards,
Vlado

[Fredrik Averpil]

Great, thanks. This happened with 3.30.02 stable build 26678 for Maya 2015 Extension 1 Service Pack 6 on CentOS Linux, 7.2.1511, Linux-3.10.0-327.13.1.el7.x86_64-x86_64-with-centos-7.2.1511-Core.

Talk about a versions report!

[t.petrov]

All guides I could find on the internet mention that you have to add this

fs.file-max = 100000

to /etc/sysctl.conf and then run sysconf -p
Have you done this?
Also how are you starting V-Ray? If it is as a systemd service then you need to modify the service script, too.

Here is an example how to do it for mysql - https://naveensnayak.wordpress.com/2...imit-centos-7/

[Fredrik Averpil]

No I haven't tried that. Seems like something new with CentOS 7?
I'm giving this a try now.

[Fredrik Averpil]

Unfortunately, setting fs.file-max in /etc/sysctl.conf didn't help

[Fredrik Averpil]

how are you starting V-Ray? If it is as a systemd service then you need to modify the service script, too.

We're rendering via mayabatch. So I'm not using V-Ray standalone. Anything I need to do in this case?

[t.petrov]

Have you restarted all your shells?
What is the output of ulimit -a in a terminal?

[Fredrik Averpil]

Just rebooted, to be sure.

[root@bs01gen04 ~]# ulimit
unlimited
[root@bs01gen04 ~]# ulimit -n
500000
[root@bs01gen04 ~]# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256849
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 500000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 500000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I did change some things to /etc/security/limits.conf prior to the reboot:

*         hard    nproc       500000
*         soft    nproc       500000
*         hard    nofile      500000
*         soft    nofile      500000

I still cannot load more than exactly 1000 vrmeshes before triggering "could not load mesh file".

[t.petrov]

And if you run mayabatch from this same console/terminal it won't be able to open all the needed files?

[Fredrik Averpil]

What the?!
When running the render command in that shell, all vrmeshes load up just fine.

[t.petrov]

Then again my question: how are you starting mayabatch?
Deadline, some other render manager, custom system?

[Fredrik Averpil]

I'm starting mayabatch via Pixar's Tractor. Mayabatch is not started directly, but through a Python wrapper script.

So, Tractor makes the Linux machine execute "python script.py " with a bunch of arguments such as which renderer to use and some custom config.

And the executable which is launched by Python is called "Render", not "mayabatch".

[Fredrik Averpil]

Just ran this as a test from my wrapper script:

import resource

# the soft limit imposed by the current configuration
# the hard limit imposed by the operating system.
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
print 'Soft limit is ', soft
print 'Hard limit is ', hard

During the failing render, this showed:
Soft limit is 1024
Hard limit is 4096

...so obviously the system wide setting does not apply in that shell for some reason.

[Fredrik Averpil]

Thanks guys for helping out. This is obviously a Pixar Tractor issue. I'll report back when I solve this. Sounds like I need to configure the Tractor client to allow a larger amount of opened files (via ulimit perhaps).