Cuda error 719

[biochemical_animations]

Anyone know what this error code refers to?

[savage309]

This is what the CUDA driver reports when the GPU kernel that does the rendering encountered an error (no matter what error).

Best,
Blago.

[biochemical_animations]

Yeah, I've been getting this error on a couple render nodes on my last render job.

./src/ocl_tracedevice.cpp(2041):CUDA error 719:exception occurred on the device while executing a kernel

just now 2 nodes reported the error simultaneously, when I first started a new job. Using rt 3.60.02.

[savage309]

Best thing to do in this case is to send the scene so we can check here what is going on.

Best,
Blago.

[biochemical_animations]

Now, all nodes reported the error...maybe I have a problem with my scene...debugging now...

[biochemical_animations]

Best thing to do in this case is to send the scene so we can check here what is going on.

Best,
Blago.

OK, when I get some time, I'll strip the scene down (without proprietary models) and send over

[biochemical_animations]

Only change I made was removing gpu low priority to avoid the node render hang issue.

[savage309]

Thanks!

Best,
Blago.

[biochemical_animations]

Now, two nodes that have 2 gpu's each, lost connection to one of their gpu's each. Rebooting the nodes made the lost gpu come back online.

Restarting the job on the nodes, reported the cuda error, when starting the 1st task through backburner, but then the task restarted ok.

now another node that had been working on a frame for about 30 minutes, failed with cuda error 719.

I think I'll go back to low gpu priority back on and see if that helps

[biochemical_animations]

nope that didn't help...I think this is on my end, like maybe the ocl cuda kernel build thing...maybe it got corrupted...

going to try going back a version in vray...

[biochemical_animations]

went back to 36001 didn't help

[biochemical_animations]

OK, I can't seem to debug on my end.

I created a stripped down scene, but I cannot reproduce with the stipped down scene so I don't think sending that over will help.

Maybe it is something with my scene, maybe the camera is passing through a transparent object or something. Don't have time now to debug, maybe rendering outside of BB will help me get past those trouble frames, if that's it...gonna try that.

[biochemical_animations]

How do you rebuild the ocl cuda kernel thing?

[biochemical_animations]

maybe a driver issue...when I got the cuda error on a node, I lost a gpu on that node.

[biochemical_animations]

tried reinstalling the driver on one of the problem nodes...it appears to have fixed it...am trying other nodes now

maybe changing low gpu priority went thru all nodes and corrupted the drivers...