We are trying to render scenes that link many vrmeshes files through our renderfam (windows 7 |maya 2013 render.exe|royal render |vray 2.31).
We’ve noticed that once a render thread has started loading vrmeshes, no other process ( maya in GUI or any other instance of render) can access the vrmeshes files.
We’ve immediately thought of file locks that would be put in an inappropriate way , and tried several things.
We know that the render process locks files in write (ie can’t overwrite nor delete) , but windows can still read and copy the files.
We’ve tried removing write rights to everyone to make sure no write lock should be put, but it results in : cannot load vrmesh file.
There seem to be a limitation or problem in the way maya render + vray for maya lock the vrmesh files.
Can you please help, we are in the big rush of a production, and can’t render on more that one machine at a time, nor work using with the same vrmeshes while render runs.
EDIT: I forgot to mention that each vrmesh contains all the frames of the shot, for one character. (not 1 vrmesh per frame per character)
+
we’ve just tested exporting in vrscene and rendering using standalone, same issue.
Hmm, actually now that you mention it, this has never been an issue and many many customers over the years have been using .vrmesh files on shared network locations without any issues.
Can you show me the exact error messages printed out by V-Ray?
There is no error message by default, just that we notice we are locked and can’t use the vmreshes anymore:
If we submit a render, even with just 1 render node, and we try to import the vrmeshes in maya gui, it remains freezed until job is finished or killed.
I we submit job on multiple nodes and chek the logs with attention, we see that the first node that starts rendering links to the vrmeshes very quickly, then all the other render nodes seem to be waiting, and the logs show us that linking the vrmeshes takes huge additionnal time (like 20 minutes or so). ( in the storage logs and graphs, there seem to be absolutely no issue of load or bandwidth).
We didn’t mention that each vrmesh was about 100/200MB and there are 500 of them in the scene. (each vrmesh contains all the frames for one character).
We were thinking of rendering vrmeshes per frame per character ( it will end up to approx 250000 small files instead of 500 big), what are your thoughts about that ?
For your particular case this might work better. Still, we don’t really lock the files, but I will run a few tests with the latest stable 2.3 release just to make sure.
Yes, first thing we tried, rendering the same command (render.exe -vray …) from a cmd commandline on a workstation, same issue, one rendering prevents access for any user.
tomorrow we’ll rewrite our batch proxy exports to render frame by frame (though is is less efficient in generation), and test.
TO be more precise about the locking, here is what we see:
1/when a render is running on one node or more, windows explorer can read and copy to another location the vrmeshes.
windows explorer cannot delete or move the file stating that is in use ( which is true )
maya through gui cannot import the vrmeshes (freeze)
any other render seems to struggle to get acces to vrmeshes and waits.
2/ when render is finished or disabled, windows explorer can read, copy , moce delete the vrmeshes
maya can import vrmeshes
If we change the file rights so they are read-only (which possibly prevent to put a write lock), vray fails to load the vrmeshes ( like it requires a read-write access to the files).
I’ve tried to disable handling of locks on my file server ( which is hihgly unrecommended to prevent file corruption) and it changes nothing.
it behaves like both vray rendering and proxy/import/loading in gui requires more locking than expected ( we would expect a file read lock or something like that)
Hm, a quick test here with a read-only .vrmesh file works as expected (V-Ray can render it just fine). But this is with a newer V-Ray version; will try 2.3 tomorrow.
Can you show me the error message that you get in that case?
The best to know if it is a VRay problem or not is, on 1 computer :
- create a .vrmesh into c:\ (it is important, not on shared folders or network folders)
- open 2 Maya, render at the same time (on the same computer obviously)
Yes; unfortunately (or fortunately, I don’t know) everything works as expected here; even when the file is read-only on a shared network path, we are able to successfully open and render it in several instances of Maya on different machines.
@Vlado
Hi sorry for late reply, we were busy modifying tools and testing.
Here is what we got:
We’ve modified the proxy export process so we now have 1 vrmesh per frame (now we have 500x500=250000 vrmeshes )
Using this method, each render loads different vrmeshes so parallel rendering becomes possible.
We are not totally sure if the lock is disappeard using “by frame” proxies or if it is just harder to notice as load a 300kb file is very fast.
Anyway, when we’ll launch multiple scenes rendering that rely on the same vrmeshes (elemnts that are the basis of a crowd), it is likely that sometimes same frame/vrmesh will be required by 2 render nodes at the same moment.
I’ve setup an additionnal storage server outside the production for testing, and tweaking additionnal parameters like samba lock, but no success.
I’ve also opened a ticket on storage system provider explaining the the issue and asking for aditionnal tips to try and debug ( our storage is pretty closed in terms of debugging functions accessible to customer), we’ll see if they can help troubleshooting on their side.
In the meanwhile, we were wondering what really happens when vray “links” to a vrmesh, does it load the file in memory ? ( with 500 vrm of 100Mo, unlikely to happen, 500 of 300k, possible), does it seek in the file to access required frame at the rendering step ? does it make a difference if it is one big vrmesh vs 1 frame vrmesh ?
When you’ve tested on your side, were you using vrmeshes with multiple frames inside or vrmeshes “by frame” ?
Many thanks
@bigbossfr
we’ll test that but it requires copying a very large amount of data to local hdd,and will just proove that a local filesystem behaves differently that a nas, if Vlado tested , it probably is not just a vray issue, but a combination of parameters, anyway thanks for the help
When this happens, V-Ray opens the file for reading and reads from it an index table that generally describes where the actual data is located in the file, for each frame. The table is relatively small and usually located at the end of the .vrmesh file. The file is kept open for reading during the rendering process, but V-Ray doesn’t read any actual data until it’s needed.
does it seek in the file to access required frame at the rendering step?Yes, V-Ray will only read the data that it needs for rendering a particular frame; it will not read the entire file.
does it make a difference if it is one big vrmesh vs 1 frame vrmesh?Ideally it shouldn’t make any difference at all.
When you’ve tested on your side, were you using vrmeshes with multiple frames inside or vrmeshes “by frame”?I was testing multiple frames inside one file.
In the 1st phase of the job, when vray is “linking” to proxies, if i got it well, it is just reading the index, which should be pretty fast (it is for the first machine to render), then in the rendering process, it will seek into the file and read required data.
Is there any chance that the way it opens file is misinterpreted by samba and opens it with write attribute , which would explain why the second fails doing so ?
In per frame vrmeshes, is there an index too, or is it useless?
Just to make sure we compare on same testing, our your side, you used multiple frames per vrmeshes, were you under windows using samba network storage?
May I ask what storage you’ve got on the other end ?
In per frame vrmeshes, is there an index too, or is it useless?Yes, there is an index, as V-Ray may still divide the geometry data in chunks for easier handling.
Just to make sure we compare on same testing, our your side, you used multiple frames per vrmeshes, were you under windows using samba network storage? May I ask what storage you’ve got on the other end ?I was rendering on Windows, yes. I’m not sure exactly what type of samba server we are using though, but I think it’s some version of FreeBSD. I can ask for more details if you need them.
The test that i want to do is just :
1) Anim a cube on 10 frames. Export the cube as a proxy. Import the proxy in a scene with high settings (to do not render fastly the cube).
2) Copy the cube to the NAS. Render with 2 computers and see if the problem is here
3) Copy the cube in c:\ and render the scene with 2 Maya (on the same machine).
If the problem is in 2) : there is a problem with the NAS
If the problem is in 2) and 3) : Vray problem
If the problem is in 3) : local problem (but this is not the problem here)
Not a very large amount of data to copy , just a cube ^^
)
, just a cube ^^