Textures and geometry drop on certain frames

[pandapanther]

I'm heading up a fairly large project & we're having issues with textures and geometry dropping from rendered images on random frames. It's costing us lots of time/money & I'd love to get it solved. Some context:

We're Windows 7
We're using Maya 2012 Service Pack 2
We're using VRay 2.20.01
We're using Smedge 2011 Update 1 as our render farm manager
Our file server is on CentOS 6.2
We have about 200 machines, so there is a fare amount of network traffic

It happens with jpg/tiff/tga texture files -- pretty much any texture file we test. It seems to happen on the same geo per scene. I originally thought there was something wrong w/the particular models/texture files involved, but can't find anything wrong with them (normals are okay, no overlapping geo etc). Our scenes are high memory usage scenes (usually between 10 and 16 gigs), but our renderfarm machines have 32G of ram. There is no warnings or errors in the logs on frames that have missing textures/geo (besides warnings about large scene size, but we get that everywhere). Our "Message Level" is set to "all messages." Some of our machines were not running Maya Service Pack 2 & when we upgraded those machines the dropping geo went away (so far) but not the dropping textures.

To be more specific: it looks like the textures darken to almost black. ie, we can see some detail in the textures still. The issue is hard to reproduce. I'll render a frame locally on my machine & the issue will appear. Then I'll render the same frame again & the issue will be gone. Smedge in that case was not part of the loop. It happens apparently at random per machine / per frame, but more often with high ram usage jobs. We're using baked GI in most of our scenes, multi-frame incremental and animated. Our scene scales are quite large. Characters flying from one mountain to another in real world units, that sort of thing.

Has anything like this been reported before? I've looked through the forums & haven't found anything that sounds like this problem.

Any help appreciated.

Cheers,
Raphael

[vlado]

I haven't come across that issue, so I'll need some more info. Can you show me some of the bad frames and some of the correct frames? (you can email the images to vlado@chaosgroup.com if you don't want to post here)

Are you rendering with Maya batch or through the V-Ray Standalone?

Does the geometry in question have render time subdivision?

Can you get me any way to reproduce this here?

Best regards,
Vlado

[sharktacos]

We are experiencing this too periodically, so if you could post a solution once you find it, that would be awesome

[pandapanther]

Hi Vlado,

Thanks for the quick response. We're in the thick of production, so apologies for my delayed reply. I've attached a couple images here -- you can see the bg hills going black in the "bad" one.

Yes, we are adding the Vray "Subdivision" and "Subdivision and Displacement Quality" attrs to our geo. We're using Maya batch. I'll keep trying to get you a reproducible bug scene, but it's hard to pin it down.

[vlado]

Ok, some more questions. Are you using the <UDIM> or <UVTILE> tags? Is it always the same texture(s) that are missing? Have you tried a more recent stable build of V-Ray? (If you don't have access to the stable builds, please email me to vlado@chaosgroup.com)

Best regards,
Vlado

[pandapanther]

It's always the same geo/texture per shot that's missing.. but we get different textures acting up in different shots. & like I mentioned below, we've tried switching the texture & changing the format (tiff, jpg, tga) but that doesn't seem to make a difference -- which seems to point to an issue w/the geo.. but then the geo renders fine in another shot & another texture acts up on a dif piece of geo.. & around & around we go. The only workaround we have is to render shaders without textures connected to them (flat colors) on problematic assets. We can get away with it in some cases, but not others. For example w/the bg hills in the posted images we'll probably just render them flat green.

We're not using any tags in our texture file paths. I know what UDIM is having helped bug-check Mari when it was in dev, but am not familiar with UVTILE -- I'm guessing that's mudbox or some other software's version of udim? In any case, all our uv layouts are between 0 & 1. I took a quick look at the UV layout on the geo in question (the bg hills in the images I posted here) and they do have a vertices very close to the edge of the tile. I'll shrink it a bit & test, maybe they overlap a tiny bit.

At this point in production, I don't think we can switch versions of VRay.. but I'll give it a shot in our dev environment. I'm still crossing my fingers that we're doing something wrong in our scenes.. or perhaps a render setting of some kind could save the day.

Other things I've tested today w/no luck:

* changed the camera clipping planes to avoid possible precision errors
* removed the input to bump (thought maybe the bump was somehow being exaggerated & flipping the normals inside out)
* rendered without subdiv attrs on the geo

Cheers,
Raphael

[vlado]

You could perhaps try to turn off filtering for the textures... In any case I don't have any more questions - we have to try and reproduce this here, so if you have any way to get me a scene or part of a scene that shows the problem, even if this is random - it will be very helpful.

Best regards,
Vlado

[hellobard]

I just wanted to chime in and say that I have experienced the exact same problem in a scene on snow leopard version 2.20.01. One piece of geometry / texture just disappears in one specific frame. I'd send my scene, but it's got a crazy amount of referenced objects and cached particle systems.

[vlado]

I'd send my scene, but it's got a crazy amount of referenced objects and cached particle systems.

Even so; we can get you an ftp to upload it. Without a reproducible case I really don't know what to be looking for.

Best regards,
Vlado

[hellobard]

I was just about to wrap up my files and send them to you, but I tested the same frame again now and it was not producing an error anymore. It seems to be a very temperamental bug, I've made changes and re-cached my particle systems since I last tested the frame with the problem and now the error is not appearing anymore.

[vlado]

Nevertless, it would still be useful to take a look

Best regards,
Vlado

[pandapanther]

Okay, I've got a scene packed up for you -- it's 750megs. I'll email you for ftp info.

[hellobard]

I did the same with my files, hopefully you'll be able to figure this out so PandaPanther can go back to making their awesome work

[pizzaia]

I've been having the same issue with the standalone, during different frames some textures disappear. There's any way to solve this?

[Morbid Angel]

one thing I can suggest is to sync all textures to the render machines and re-target paths to be from "c:\temp" or whatever your sync local location is. We had to do this for our setup; even though our server is an isilon, we would have 40-50 machines pull 30 gb of texture data at once, which would just overload the entire network and though they could render the frames, we would get dropped textures as well as the machines would not be able to write to network at the time of rendering.
By copying the textures locally, we solved all those issues.
One other thing I could suggest, is to take each of the textures (shaders) and put them on the plane, one plane for each shader in the screen view and render that scene (no actual geometry) just planes with shaders/textures and see if this does happen still.
It could be as simple as having a certain texture in a certain location on the server that is causing a problem.