When V-Ray uses displacement or subdivision like in your case, quite a lot of triangles are generated. These are added to a special dynamic memory pool, which has a maximum size ( 500 MB by default). When more triangles need to be generated, but there is no memory in the pool, some of the least recently used triangles are removed from the pool to make place for the new ones. Depending on the scene, newly generated rays will need the triangles that were just evicted from the pool, so they’ll be generated again, but something else removed. Since not everything that is needed to render the scene can fit in the dynamic memory pool at one time, the rendering cannot finish.
Beauty. Vladimir, thanks. Really thanks a lot for explain the things so easy and so understandable.
Let me one more question about:
Its possible to know the amount memory needed by render the subidivided/displaced meshesh (supposing all the subdivided/displaced meshes gonna appear directly or indirectly on the final render)?. I mean, do you got some kind of equivalence, like 2 million poligons = 700 mb destinated to dynamic memory?
The amount of triangles generated and thus the memory needed depends on how close the displaced/subdivided mesh is to the camera, as at least by default the edges of the subdivided triangles are made to be a certain number of pixels long. V-Ray does compute the memory needed internally in order to decide what needs to be evicted from memory, but this computation is done on the fly as triangles are generated. My guess is that you can safely set the “Dynamic memory limit” to at least two thirds of your memory, if it is more than needed, V-Ray won’t actually allocate the rest, so it’ll be available for other stuff, like texture images.