ok ive managed to get it to render by reducing the number of trees from 40,000 to 30,000.. odd id have assumed it would not be limited unless the ram was full. could it have been some coplanar faces amongs the intersecting trees? id not imagine that would throw an error either.
anyway next problem:
i got a decent render out in about 10 minutes, so i thought id try a few frames overnight to see how the gpu behaves, and check the shimmering in the foliage.
i set 200 frames going, first frame completed ok, using about 4gb on gpu and vray.exe using about 7gb
it saved the frame, moved onto frame 2 and…stopped. before the gpu even got involved.
ram usage rose again on vray rt up to about 12gb, then stopped rising… i waited 15 minutes then cancelled.
max is half crashed (i cant close render dialog, which wont cancel, but i can save the file and spin the viewport.)
any ideas? so far everythng im trying on rt beyond the benchmark scene seems to have issues 
-ive wetransferred latest scene to vlado.
-update:
with 95% tree reduction, the scene renders as expected.. 50% reduction it still locks after first frame.
-2nd update:
with 85% reduction in trees i can see perhaps the “crash” may just be an extremely long pause.. with 85% reduction, i wait about a minute for the next frame to start, but it does start.
HOWEVER, i can see even after the 3rd frame, that the ram is not being cleared after each frame. each frame adds a gb to the ram usage on the host pc. not good! at that rate id only be able to render about 28 frames before my ram maxes out.
considering with 20k trees, it added 6gb before starting the second frame, id only be able to render a handful of frames before game over. assuming i could be bothered to actually wait for the 15+minutes to see if it actually starts the next frame at all.
3rd update:
opencl has same issue as cuda. i let it do 9 frames and my ram usage climbed from 7gb to 12.3 over the course of those frames.
4th update:
even with only *ONE* proxy tree in the scene, i -still- get the ram climbing after each frame. obviously only by a smaller amount, but after 50 frames, its gone up a few GB.
5th update: at least on the ram increase between frames, it seems to be related to the tree material. if i put a flat grey vraymtl on the trees, the ram usage stays constant. i tried just removing the 2sided from the leaves, but still had the issue.. i also found a “forest colour” map in the leaf diffuse, but removing that didnt help either.