Error building Embree static ray caster, error message: [EmbreeRayServer::doBuildSDTree] 3: Setting vertex buffer failed

[simon_kotowicz]

Hello, I'm a little stuck with this error message "Error building Embree static ray caster, error message: [EmbreeRayServer::doBuildSDTree] 3: Setting vertex buffer failed (number triangles 441194591)"

I've searched the forum and seen about three others have this error, but nobody has explained how to fix it or what's causing it. I'd appreciate some help, please, and perhaps a little pointer to the correct instructions the fix relates.

Here's what happened. I have a scene that I've been working on for a while. It rendered easily unitl I changed two lights from VRAY Spheres to VRAY Targetted Discs. With the lights set to targetting discs I get the error message. I'll attach the full log. If I go back to the save before I switched the lights the scene will render.

During the render the Task Manager shows it's using 159GB of RAM, with plenty to spare.

My system is a Threadripper Pro, 256GB RAM, RTX A6000.
The scene is being rendered in 3DS Max 2024, VRAY 6.1 - CPU Render. Bucket Rendering. Brute Force / Brute Force in the GI.

NDA prevents me posting the scene, so I'll need some guidence as to what to try.

Thanks!

Simon

[hermit.crab]

Hard to say without the scene. Usually, this type of error is related to RAM usage. There's no error if you revert the target lights to sphere lights?

[simon_kotowicz]

Hi Mr Crab,

Yes, the error disappears when I revert the lights back to spheres.

HOWEVER, my current renders are 1920x1080. The final resolution needs to be 4K and yesterday's 4k test render caused the same error with the spheres.

The other posts have mentioned it's related to Displacement - there are displacements in my scene.

Perhaps there is a technique to tackle this? None of the others posted a solution and I'm fairly new to VRAY. As for the RAM, I see no indication that it's ran out. Does the error log I attached to the post say that?

-- UPDATE --

If I turn off the Displacement, in the Global Switches of the Render Settings, the scene will render at the larger resolution. So it's clearly a Displacement problem. But I do need the displacement.

So, what settings do I tinker with to coerce it into rendering? What would the big studios do? Surely they don't just say, "Well, it's the RAM. We're giving up"?

I already have 256GB of RAM, filling all 8 memory slots. There's no room for more. There's no money to replace it all with larger (they're slower too) DIMMS. Task Manager is telling me I'm only using 159GB of RAM, anyway.

Can anyone suggest anything to try? Anything?

I've already tried using Region Render, with the aim of rendering the image in smaller blocks, but it still won't render with Displacement activated.

Really need some pointers, please. Anyone?

Cheers!

Simon

[Vizioen]

Some more information about your scene might be handy. First of all which displacement are you using? In the material or with a displacement modifier (2D or 3D?) Which other settings?
Did you try using conserve memory option in Vray Render Settings > Settings > System
Did you try to render it with Light Cache? If you're using BF / BF and your lighting is set to Adaptive you will use Light Tree instead. Did you try settings this to full light evaluation?
Can you maybe post a screenshot or draft render?

[simon_kotowicz]

Hi Vizioen,

Thanks for giving me some options to try. TLDR is that it didn't help.

Conserve Memory
This option was not switched on. I ticked the box and rendered again, but it still bombs out. I left the option ticked.

Light Cache
I switched over to Brute Force / Light Cache and tried rendering again The VFB returns the same error message and fails to render.
I also tried Light Cache / Light Cache, but still no joy.

Full Light Evalution
I switched the Global Switch over to "Full Light Evaluation" and tied again. Still has the problem.


As for the rest of the information:

Every object has UVs, some of them are UDIMs.
Every texture is a VRAYmtl with PBR textures plugged in (Diffuse, Normal, Roughness, etc)
The PBR Textures are EXR files, 16bit float, from Substance Painter. ACEScg.
All the lights in the scene are VRAY lights.
The camera is a Vray Camera
The objects that have dispacement have a VrayDisplacementMod at the top of the stack.
The stacks are all very short - base object ediable poly, that's been converted to Edit Poly after using Reset XRef. Then a VrayDisplacementMod added to it.
The displacements are small, 1 or 2 mm. Scene unit is CM.
Displacement type is 3D Mapping
3D Mapping/Subdivision is set to "Use Object Material"
The Height map for displacement is piped into the VRAYmtl, into the Displace socket.

Every render returns the same error message:-
16:43:39.864] Error : Error building Embree static ray caster, error message: [EmbreeRayServer::doBuildSDTree] 3: Setting vertex buffer failed (number triangles 43988647)


Once again, thanks for trying. If you, or anyone else, has any more ideas I'll definitely give them a try. The docs.chaos.com tell me what the buttons do, but aren't really good at the nuances, foilbles and combinations. I'm litterally having to guess what to change to make this go away. Any wisdom is warmly welcome.

Thanks Vizioen!

Meanwhile, I'm going to remove Displacement from an object or two to see if I can persuade it to render for me.

All the best,

Simon

[Vizioen]

As the problems only occur when working with the Vray Disc Lights: what are your settings there, and did you try to mess with them, like directionality, sampling or decay?

Also maybe try raising the edge length of the displacement to something bigger like 10 or 20 pixels, just for the sake of trying to get it to render. As I'm not sure you need 3D displacement because I don't know your scene and objects, are you sure you need 3D, or can you get away with 2D?

It must be some combination of displacement and these lights that throw this error. All in all I encountered this error a couple of times over the years, ofcourse only when there's tight deadlines and no time to mess around so I don't recall if I was able to fix it or if just worked around it. But I'm quite sure it didn't have anything to do with using disc lights in combinations with displacement.

[Joelaff]

You watch your ram usage when rendering does it max out your memory?

You could turn off view dependent for the displacement and set the edge length based on scene units. Be careful, lower it slowly and watch the ram usage. It is easy to consume all memory with displacement , even on this 512GB box.

If you have view dependent on it will user more memory when getting closer to the camera. It displaces the entire object at that level (even parts out of view) so cut off the backs of objects if you can and make them separate non displaced objects.

Turning off pretessellated can save memory, but render slower. This sometimes faster, lol.

You can also convert textures to tiled format if they are big.

[simon_kotowicz]

Thanks Vizioen, Thanks Joelaff.

So, where I'm up to:

When rendering at 1280x720 everything works.
When rendering at 1920x1080 everything works.
When rendering at 2460x1440 it doesn't work.
I need to go up 5500x5500, single frame, for print. The scene will eventaully be animated and rendered at 1920x1080, but the 5500x5500 is really important.

I think the disc/spherical lights thing is a mis-direction. Here's why - I started to delete an object and hit render. It would fail. I would delete another object and render again. It would fail. And so on. I was trying to find which object is causing the render to stop working, because it doesn't say which one in the error log, despite setting the log to ALL Level 4.
I rendered the disc lights with the biggest object with a displace on, that takes up most of the frame, everything else deleted and it rendered, so it's not an error in those disc lights.

I tried adjusting the Edge Length, and turned off View Dependent, and set it to 10cm. I only need 1 or 2 milimetre displacement, but it still wouldn't render at Edge Length 10.

As for the RAM, I'm not getting anywhere near my RAM limit. Most I've seen it fill in Task Manager is 159GB. If I turn on Conserve Memory the RAM usage drops to something like 39GB.

I'm using EXR textures, which I thought were already a tilable format. Is that right? If not, how do I convert them? It's not that awful Arnold TX fileformat, is it?

I'm going to try cutting off the back of the object and removing its displace and see if that works. But I'm not even sure what object is causing the problem. If I delete the VrayDisplacementMod and the Height texture from that main object it still doesn't render.

I shall also try turning off pretessellated and see what happens.


Meanwhile, I've spent the entire day trying things and nothing is working. Is there a Checklist to follow methodically when things like this occur? Is there a way to get the log to tell me what object is causing the problem? Or are we literally left to try random things until we fix it?

I'll report back.

Thank guys!

Simon

[Vizioen]

Now I'm really curious actually. There's not really a checklist but if I have problems I try to render without all the fancy stuff (like displacement, scatterers, override material etc, but you already figured out that it's displacement that causes the issues).

EXR can be tiled, as well as tiff and tx afaik. But not sure if you use EXR it's a tiled texture whatsoever. You can use the image to tiled multiresolution EXR converter that comes with VRay to be sure.

Some other things to try: replace the displacement map with a checker map for instance, to see if it's not a problem with the texture being maybe corrupt, too big, etc,

If all fails you can try and send it over to chaos, they can troubleshoot your scene but not sure how fast you'll get an answer as it's weekend. I'm also willing to give it a shot but understand if it's "classified".

[simon_kotowicz]

Okay, I have news. I have finally found the object that's causing the problems and it's currently rendering 3840x2160. Yippee!

So, first of all it was a Displacement that was causing me the headaches. I removed the Height Map from the VRAYmtl and the VrayDisplacementMod from the object and it just started working. Except now the lighting is all wrong, which is weird.

I've spent most of this week tinkering with the lights and now the Environment is too strong. I opened the file again, tried it at lower res with the problematic displacement, lighting was as I set it. I removed the problem DIsplacement and the Environment kicks in x10.

So, the object that was causing problems was a window frame and its glass. The scene is through a shop window. The shop's window display is what we're looking at and needed its displacement. But the Window Frame and its glass had a displacement of 1mm, to make the old flakey paint pop-out a bit. I can live without it, as the normals do almost all of the work. It's just where I had paint on the glass, you know, where it's been repainted year after year and they've gone over onto the glass, I really wanted to sell that paint effect and make it stand out a little. Never mind.

So, what do I think it was? Well, I think it was the glass. Using Substance Painter, the window frame was painted with the glass attached as part of the object and I used an opactiy channel, so I could paint onto the glass. Then, back in 3ds Max, I seperated the glass into its own object, seperate from the frame, and added a VRAYmtl set to glass, pumped into a composite node with the window frame texture composited over it, so that the paint is on the glass. Not sure if it was because it was a compsite or just because it's the closest object to the camera that disappears out of frame, like Joelaff mentioned. I suppose the displacement on a composite over glass material caused it. Not sure why it alters the environment levels, but they're easily tweaked again.

THE FIX

So, to present a fix for anybody else that's having this issue - I litterally had to select an object, remove its displacement and try to render. If that wasn't it, load the scene again so the deleted stuff is put back and select the next object, deleting its displacement and trying to render. Eventually you'll find the object which is crashing the render, so you'll atleast know what you've got to work on. The Error Log the VFB generates is no use at helping you know which is the problem object.


What would I like Chaos to do?
Perhaps add a debug line which states which object the Error Code relates to? That would have saved me a couple of days of trying random things. It would be cool to see it said in the log "Window Frame - Error building Embree static ray caster, error message: [EmbreeRayServer::doBuildSDTree] 3: Setting vertex buffer failed". But go easy on me, I'm not a developer so I don't know if it can be done


So thank you very much for all your help - I've learned some stuff and you've got me rendering again.

Also, once the project is out in the wild, I'll try to remember to come back to this post and upload the image.

Cheers Everyone! You've all taught me so much today.

Simon.

[simon_kotowicz]

Here's some more information, for future artists struggling with this error.

The Edge Length Parameter in the "Render Settings - Settings - Default Displacement" has zero effect if you have a VrayDisplacementMod in the stacks of your scene objects.

I've spent the past two days trying to solve this error and the instructions aren't clear enough. What I've found is that you have to change each model's VrayDisplacementMod "Edge Length" individually for it to have any affect on the outcome of the render.

Hope this helps!

[Joelaff]

Yes, that overall edge length is only if you use the displacement map connected directly to VRayMtl and do not have a VrayDisplacementMod on the object.

Personally I always use the modifier, and never connect anything to the displacement map of the VRayMtl. It is more clear that way, and you get more, and per object control.

[Vizioen]

Well it is also quite clearly explained in the docs: Default Displacement - V-Ray for 3ds Max - Global Site (chaos.com)

[simon_kotowicz]

Hi Joelaff,

Yes, I agree, the VrayDisplacementMod gives much more control, especailly if some objects need centimetres of displacement and some only need a millimetre or two. But I prefer to use it in combination with a displacement texture map pumped into the VRAYmtl, because I usually have two or three different texture libraries for each scene. That way I can swap a material and not have to mess with the VrayDisplacementMod, or even forget to do it. It's all tied into the material and then saved in a library.

Well it is also quite clearly explained in the docs: Default Displacement - V-Ray for 3ds Max - Global Site (chaos.com)

Thanks for trying Vizioen, but unfortunately, I beg to differ. I always read the instructions first, but I cannot see where it says that button won't work if you have the VrayDisplacementMod applied to the object.

I think the instructions are too brief, assume you alreay know and are simply coming back for a refresher. For someone new to Vray, like me, I don't think the instructions are good enough or contain enough detail.

I also think an earlier reply where someone said, "It's the RAM" didn't help. At no point did I ever get to the limit, or even near the limit, of my RAM. I think what's really going on is a limitation within VRAY that we all keep hitting. It's possible those users that hit this limitation first DIDN'T have enough RAM and that's what support have concluded. I worked in support for a different company in the 90s and support-types are always looking for easy ways to stop their customers hassling them Also, the debug log doesn't contain enough information. It's no wonder the support team can't see what the problem is. I'd like to see a more thorough debug log, it would have really helped me.

Anyway, let's keep helping each other out, we all love VRAY as it does amazing things. By coming here and asking questions and then giving detailed replies and finished threads will help the next person who hits a wall.

All the best!

Simon

[Vizioen]

Thanks for trying Vizioen, but unfortunately, I beg to differ. I always read the instructions first, but I cannot see where it says that button won't work if you have the VrayDisplacementMod applied to the object.

Unless I'm not following anymore but it states in the first sentence: The settings in this rollout control the displacement of objects with displacement materials that do not have a VRayDisplacementMod modifier applied.