Cuda error 719

V-Ray V-Ray for 3ds Max Issues
1

Anyone know what this error code refers to?

2

This is what the CUDA driver reports when the GPU kernel that does the rendering encountered an error (no matter what error).

Best,
Blago.

3

Yeah, I’ve been getting this error on a couple render nodes on my last render job.

./src/ocl_tracedevice.cpp(2041):CUDA error 719:exception occurred on the device while executing a kernel

just now 2 nodes reported the error simultaneously, when I first started a new job. Using rt 3.60.02.

4

Best thing to do in this case is to send the scene so we can check here what is going on.

Best,
Blago.

5

Now, all nodes reported the error…maybe I have a problem with my scene…debugging now…

6

OK, when I get some time, I’ll strip the scene down (without proprietary models) and send over

7

Only change I made was removing gpu low priority to avoid the node render hang issue.

8

Thanks!

Best,
Blago.

9

Now, two nodes that have 2 gpu’s each, lost connection to one of their gpu’s each. Rebooting the nodes made the lost gpu come back online.

Restarting the job on the nodes, reported the cuda error, when starting the 1st task through backburner, but then the task restarted ok.

now another node that had been working on a frame for about 30 minutes, failed with cuda error 719.

I think I’ll go back to low gpu priority back on and see if that helps

10

nope that didn’t help…I think this is on my end, like maybe the ocl cuda kernel build thing…maybe it got corrupted…

going to try going back a version in vray…

11

went back to 36001 didn’t help

12

OK, I can’t seem to debug on my end.

I created a stripped down scene, but I cannot reproduce with the stipped down scene so I don’t think sending that over will help.

Maybe it is something with my scene, maybe the camera is passing through a transparent object or something. Don’t have time now to debug, maybe rendering outside of BB will help me get past those trouble frames, if that’s it…gonna try that.

13

How do you rebuild the ocl cuda kernel thing?

14

maybe a driver issue…when I got the cuda error on a node, I lost a gpu on that node.

15

tried reinstalling the driver on one of the problem nodes…it appears to have fixed it…am trying other nodes now

maybe changing low gpu priority went thru all nodes and corrupted the drivers…

16

looks like reinstalling the driver is fixing the issue…wierd how the driver would go out on 3 nodes at the same time…wondering if changing the low gpu priority may have triggered that.

17

These are not connected in any way.

Best,
Blago.

18

ok…just trying to brainstorm on what caused the issue and how to fix.

it seems weird that its affected 4 nodes. I was rendering this job yesterday without error, then rendering today, same job, 4 nodes went haywire.

just trying to think what I did differently between the two days that may be the culprit

19

ok , the issue came back

20

yep, problems back…2 nodes failed…one can’t get past pass 1, the other rendered for an hour before failing right before writing the file