I really wish this option was available. I know you guys would like us to use deadline integrated submitter but it also does not have this function, and is more bugged then your vrayDeadline submitter (which we use).
Reason for this is simple: When rendering through houdini on deadline we have to use vray standalone to do the actual rendering to avoid having as many houdini engine licenses as we have render nodes. Therefor we export a houdni export job first then render the frames. With the frames per task set to anything higher then 1, we have 2 problems. 1 submitter automatically is trying to enter ${FBATCH} into the name (which we don’t want, but then deadline submitter will abort the submission. We are using a different custom string there: /${HIPNAME}_${OS}_${F4}.vrscene and it does not seem to work if frames per task is set to higher then 1. If we do accept that setting, then both export job and render job end up with the same frames per task setting. So if we are exporting a job where frames export fast, but render long this is not an acceptable way.
In my ideal world, we would have the frames per task for export job set to 10 (without any custom additional ${FBATCH}, then have the render tasks set to 1.
Hello @Morbid_Angel Thank you for taking the time to share your idea with us.
First of all I would like to let you know that VRayDeadline node was meant to provide support for Deadline in times when Houdini did not have that node themselves. Now, as they have a native one, we recommend using it and we have stopped updating ours.
Nevertheless, you could log your idea to the Chaos Ideas Portal. This will allow us to track the interest toward it. You can find a link for it here: V-Ray Ideas Portal
Like I said before, native deadline submitter also does not have this option. Additionally it seems not to work correctly at all when exporting vray export/render jobs. We are in the same exact spot as with vrayDeadline.
> I know you guys would like us to use deadline integrated submitter
If we would we’d never invest time in our custom submitter…
> ${FBATCH} into the name (which we don’t want
Well, I haven’t found a way how to export data in multiple vrscene files and then render those in batch by a single job.
Thats why FBATCH was added which allows export frames chunk into a vrscene file and render by a single job.
If we use export string like this:
${HIP}/vrscene/${HIPNAME}/${HIPNAME}_${OS}/${HIPNAME}_${OS}_${F4}.vrscene
then each vray scene file is written as a single frame. This is preferred method for majority of our workflow, since we can deal with large scenes and each frame can be 1Gb in size (or more), so exporting lets say 25 frames chunk would be a file size of 25Gb. I don’t have a big issue with using this other than the strain on the IO during write/read, however when vrayDeadline submitter then submits a dependency job to render those frames, if we were to use ${FBATCH}, it will make 1 render machine render all 25 frames, same chunk as the exported chunk. While the export can be somewhat fast, the render of a frame can be hours per frame. So rendering 25 frames in one chunk on one machine is not a good way to utilize farm resources. It results in us having to always go in and manually set the export chunk to 25, render chunk to 1. There by it brings me to original post. I’d like to export X number of vrscene frames (write them as single frame .vrscenes) and automatically during submission make the dependency job only render 1 frame per task (1 frame per machine).
I’m using the latest nightly, and vraySubmitter. What I need ideally is:
1 vrscene per frame (which is around 1gb now)
1 task per frame in deadline
I’m unable to split the vrscene into several frames. It forces me to use FBATCH which does a single file. And when i’m forced to do this I end with a 100GB file, and having 20 render machines reading the same file at the same time, it takes almost 15-20min just for the render to start.
So is there any way currently to split the vrscene into several files and submitted to deadline?
(i tried forcing $F and manually export, but then the vraySubmitter submits a houdini license dependent job, and not a vray standalone job…)
Hi mori4h, this should be working now but it’s a bit finicky. Here is a screenshot of how we export things:
You have to manually type in the output string like this, where at the end you have _${F4}.vrscene, which will export 1 scene per 1 frame it should not force you to use ${FBATCH} in this case.
${HIP}/vrscene/${HIPNAME}/${HIPNAME}_${OS}/${HIPNAME}_${OS}_${F4}.vrscene
In render job options we specify frames per task as 1, if you set this value higher, it may error, I believe by default its set to 0, which also produces an error during submission.
Btw I also gave it some thought, you should not end up with large vrscene files in general, if you do it means you are not caching your node graph into alembics and reading that back in for rendering. Doing this you can substantially reduce file size and also export times.
Now i have an issue which i think is realated to this… AOV binds only render in deadline on the first frame… and i think it has to do with this fbatch.
EDIT: standard deadline submitter is working this way, and the bug is gone… however it’s not working in vrayDeadline and it’s bugging AOV binds due to single vrscene file.
I opened your hip and submitted it, it exported with 1 vrscene per frame out of the box. I see you are using vray 7.010, which version of deadline are you using? The only difference is that we are on houdini 19.5 but why would that impact the export tag I don’t know.
