Cached scene being rendered on farm

[Fredrik Averpil]

Hi,

On our render farm (Linux based), I've noticed some machines sometimes render old versions of a V-Ray 3 scene. I'm not sure if this is required to reproduce this behavior, but we've only noticed this when mayabatch region rendering (splitting up one image onto multiple machines);
render ... -reg 1641 2187 0 413 ...

I've also noticed that when this happens, the machine has been running two instances of maya.bin, which is not normal here. We limit machines to run 1 instance of Maya/V-Ray. This leads me to think that Maya has crashed and that our wrapper scripts have not properly dealt with that. Perhaps the locally cached vrscene file in /tmp has not been removed after a successful render and is later being re-used?

So, as a solution to this issue (so that it won't happen again), I'm thinking I could always make sure there are no "vrscene_*.txt" files and no "*.mel" files residing in the /tmp folder prior to launching Maya's Render executable. But perhaps this should be done by V-Ray automatically?

[t.petrov]

Newer version of V-Ray already deal with this issue.
They append the PID of the process to the filename.

The stable 3.0 build should have this feature. Also 3.0 night builds where rev >=24721.

[Fredrik Averpil]

We are already running a new-ish stable build 25208 of V-Ray 3. But I was unaware of that the PID was appended to the process name. I'm going to look into that.

But would it be useful if V-Ray indeed removed any vrscene_*.txt files from the /tmp folder prior to rendering?


Thanks!

[t.petrov]

But would it be useful if V-Ray indeed removed any vrscene_*.txt files from the /tmp folder prior to rendering?

I'm not sure this is a good idea, because other processes might need them.

[Fredrik Averpil]

Hm, yes. You're probably right... we're not the only ones running only 1 process of V-Ray on the same machine...

So basically, what I could do is when my wrapper script is finished executing the V-Ray render, it could simply delete any vrscene_[PID].txt files which match the PID of the process it just ran. Right?
I'm hoping that even if Maya crashes, the wrapper script will continue to run and be able to perform this file deletion.

[t.petrov]

In fact it is not the PID we're appending, but some random stuff.
Sorry for misleading you.

Is this problem easy to reproduce on your side?
And these vrscene file are important only if you're using V-Ray's DR.

[Fredrik Averpil]

I noticed that this was not a PID just now... but if these vrscene_*.txt files are only needed for DR, then it doesn't matter in this case as this was not a DR render.

Does V-Ray at all store any kind of local copy of the scene file to be rendered somewhere and is it possible that during a Maya crash, this scene file can be "left behind", and re-used by a new Maya instance?

EDIT: I really think this is related to some crashed maya.bin process which kept on rendering bogus files into the render directory... even if that indeed sounds a bit strange.

[tashko.zashev]

Does V-Ray at all store any kind of local copy of the scene file to be rendered somewhere and is it possible that during a Maya crash, this scene file can be "left behind", and re-used by a new Maya instance?

Usually a .Vrscene file is saved in the "tmp" folder.