multiple access to vrmeshes (locked files), production in danger

[CGEV]

@Vlado
Hi sorry for late reply, we were busy modifying tools and testing.
Here is what we got:
We've modified the proxy export process so we now have 1 vrmesh per frame (now we have 500x500=250000 vrmeshes )
Using this method, each render loads different vrmeshes so parallel rendering becomes possible.
We are not totally sure if the lock is disappeard using "by frame" proxies or if it is just harder to notice as load a 300kb file is very fast.
Anyway, when we'll launch multiple scenes rendering that rely on the same vrmeshes (elemnts that are the basis of a crowd), it is likely that sometimes same frame/vrmesh will be required by 2 render nodes at the same moment.

I've setup an additionnal storage server outside the production for testing, and tweaking additionnal parameters like samba lock, but no success.
I've also opened a ticket on storage system provider explaining the the issue and asking for aditionnal tips to try and debug ( our storage is pretty closed in terms of debugging functions accessible to customer), we'll see if they can help troubleshooting on their side.

In the meanwhile, we were wondering what really happens when vray "links" to a vrmesh, does it load the file in memory ? ( with 500 vrm of 100Mo, unlikely to happen, 500 of 300k, possible), does it seek in the file to access required frame at the rendering step ? does it make a difference if it is one big vrmesh vs 1 frame vrmesh ?
When you've tested on your side, were you using vrmeshes with multiple frames inside or vrmeshes "by frame" ?

Many thanks

@bigbossfr
we'll test that but it requires copying a very large amount of data to local hdd,and will just proove that a local filesystem behaves differently that a nas, if Vlado tested , it probably is not just a vray issue, but a combination of parameters, anyway thanks for the help


Pierre Billet CTO at CGEV, Paris

[vlado]

In the meanwhile, we were wondering what really happens when vray "links" to a vrmesh, does it load the file in memory?

When this happens, V-Ray opens the file for reading and reads from it an index table that generally describes where the actual data is located in the file, for each frame. The table is relatively small and usually located at the end of the .vrmesh file. The file is kept open for reading during the rendering process, but V-Ray doesn't read any actual data until it's needed.

does it seek in the file to access required frame at the rendering step?

Yes, V-Ray will only read the data that it needs for rendering a particular frame; it will not read the entire file.

does it make a difference if it is one big vrmesh vs 1 frame vrmesh?

Ideally it shouldn't make any difference at all.

When you've tested on your side, were you using vrmeshes with multiple frames inside or vrmeshes "by frame"?

I was testing multiple frames inside one file.

Best regards,
Vlado

[CGEV]

OK thanks for that fast and precise answer

In the 1st phase of the job, when vray is "linking" to proxies, if i got it well, it is just reading the index, which should be pretty fast (it is for the first machine to render), then in the rendering process, it will seek into the file and read required data.

Is there any chance that the way it opens file is misinterpreted by samba and opens it with write attribute , which would explain why the second fails doing so ?

In per frame vrmeshes, is there an index too, or is it useless?

Just to make sure we compare on same testing, our your side, you used multiple frames per vrmeshes, were you under windows using samba network storage?
May I ask what storage you've got on the other end ?

[vlado]

Is there any chance that the way it opens file is misinterpreted by samba and opens it with write attribute , which would explain why the second fails doing so ?

V-Ray uses the following Windows call to open the file (for more info see http://msdn.microsoft.com/en-us/libr...(v=vs.85).aspx ):

handle = CreateFileA(name, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, NULL);

Perhaps this can help in some way.

In per frame vrmeshes, is there an index too, or is it useless?

Yes, there is an index, as V-Ray may still divide the geometry data in chunks for easier handling.

Just to make sure we compare on same testing, our your side, you used multiple frames per vrmeshes, were you under windows using samba network storage? May I ask what storage you've got on the other end ?

I was rendering on Windows, yes. I'm not sure exactly what type of samba server we are using though, but I think it's some version of FreeBSD. I can ask for more details if you need them.

Best regards,
Vlado

[bigbossfr]

@bigbossfr
we'll test that but it requires copying a very large amount of data to local hdd,and will just proove that a local filesystem behaves differently that a nas, if Vlado tested , it probably is not just a vray issue, but a combination of parameters, anyway thanks for the help
Pierre Billet CTO at CGEV, Paris

The test that i want to do is just :
1) Anim a cube on 10 frames. Export the cube as a proxy. Import the proxy in a scene with high settings (to do not render fastly the cube).
2) Copy the cube to the NAS. Render with 2 computers and see if the problem is here
3) Copy the cube in c:\ and render the scene with 2 Maya (on the same machine).

If the problem is in 2) : there is a problem with the NAS
If the problem is in 2) and 3) : Vray problem
If the problem is in 3) : local problem (but this is not the problem here)

Not a very large amount of data to copy , just a cube ^^