Rendering irradiance map on multiple machines (not distributed rendering)

[Fredrik Averpil]

Hi,

I'm setting irradiance map mode to "Incremental add to current map" and I'm auto saving to /some/file/path/renderLayerImapCache.####.vrmap but this results in several machines trying to write to a file that's called /some/file/path/renderLayerImapCache.####.vrmap rather than /some/file/path/renderLayerImapCache.0001.vrmap.
What am I doing wrong?


Why am I doing this? (read below)

In some scenes we are currently just flying through a frozen environment. Here, we have one machine go over N frames to incrementally add to an irradiance map, which is much faster than using the "Animation (prepass)" which does not take every Nth frame.

However, I'd like to expand this into having multiple machines being able to render out each frame's irradiance map and when the whole job is complete I'll use a script to take all the irradiance maps and bake them into a single one - which will be used for final rendering.

[vlado]

We don't support '#' symbols in the irradiance map name. For the time being, you'll have to generate unique names for the maps (or just save them locally and pick them up later on).

Best regards,
Vlado

[Fredrik Averpil]

Aha, okay.

Right now I set it up with the "Animation (prepass)" mode. Is it valid to render out one irradiance map per frame using the "Animation (prepass)" mode and then merge them into one single irradiance map (for typical flythrough of frozen environments)?

The reason I'm asking is I saw this in the docs on this mode:

Note that in this mode you have to render one map for each frame (i.e. you cannot render every Nth frame)

- or maybe that's just referring to using it in conjunction with "Animation (rendering)"?

EDIT: Ok, I noticed using the vrmaps from an Animation prepass won't work when trying to load "From file"...

[Neilg]

You can do that, yes. I've done it to spread Ir maps across the entire farm when in a rush.

Do it every 20-30 frames, depending on camera speed.
I've taken to keeping the gap pretty wide and doing another one with 'use camera path' selected on a single machine to merge that in too. fills in any gaps or corners you may miss doing it every 30f.

Bear in mind it doesnt give you much of a speed increase. When using incremental modes, the IRmap only calculates the extra detail it needs, so each frame it does is much faster than the one before.
You're far better off cutting the path up into 4/5 incremental maps and merging those together at the end.

[vlado]

If you are doing fly-through's (no animation except the camera) there's no need to use the Animation modes. Instead, use "Incremental add to current map" to generate several maps for different segments of the animation (or you can render every Nth frame). Then you can merge those maps into a single one that you can load with the "From file" mode.

From the first post, it seems like this is what you are doing anyways, no? You either use Incremental add + From file, or Animation(prepass)+Animation (rendering); you can't mix and match them. The first approach is for fly-through animations, the second one is for moving objects/lights.

Best regards,
Vlado

[Fredrik Averpil]

If you are doing fly-through's (no animation except the camera) there's no need to use the Animation modes. Instead, use "Incremental add to current map" to generate several maps for different segments of the animation (or you can render every Nth frame). Then you can merge those maps into a single one that you can load with the "From file" mode.

This is for flythroughs, yes. So basically you're saying I shouldn't use Animation (prepass) to generate a bunch of maps, then merge them into one map and have it read using the "From file" mode?
Is the "Animatoin (prepass)" only to be used in conjunction with "Animation (rendering)"?
EDIT: To answer my own question: Yes.

From the first post, it seems like this is what you are doing anyways, no? You either use Incremental add + From file, or Animation(prepass)+Animation (rendering); you can't mix and match them. The first approach is for fly-through animations, the second one is for moving objects/lights.

Yes, kind of. I was doing "Incremental add to current map" for every Nth frame (one render job), and using a single machine (which takes some time) which would then render e.g. framerange 1-20 in one take (just 1 chunk). Then, during actual final image rendering, I used "From file" to read it. Easy-peasy but the irradiance map creation phase takes quite long on some very long camera flythroughs.

What I would like to do now is to have multiple machines work on the irradiance cache calculation, rather than just one machine – and from the sound of it I should use "Incremental add to current map" to generate one .vrmap for each frame (and then merge them) ... however, I can't figure out how to make each machine write to myIrradianceMap.####.vrmap with the frame numbering in there since the "Auto Save File" in Render Globals is not scriptable and does not understand ####. Hence my first post/question.

Since we render overnight I cannot pick the files up manually and rename them etc... and I need to generate this on the fly. You mentioned I could generate unique names. How?

I am working in Maya but then I export .vrscenes (one for the irradiance calculation phase/write vrmap and one for the final rendering phase/read vrmap).

I'm grateful for any advice on the matter because I feel I'm stuck right now unless I can use the "Animation (prepass)" mode to generate the vrmap files.

[Neilg]

You dont need to do an IRmap every frame for a fly through. You can send frames 0-90, every frame 30 as one IR map set to incremental (so it only renders 3 frames), send 90-150 as another (changing the name manually), then merge them together afterwards. It's clumsy, but does mean you can break it up over more machines.

[Fredrik Averpil]

Hi cubiclegangster,

Your method is fine when you want to manually change filenames. However, that's exactly what I don't want to do (hence why I started this thread).

Right now the only thing that prohibits me from having a completely scripted/automatic workflow is that I don't know how to change the vrmap filename or filepath on a per-frame basis (making each machine write a uniquely named .vrmap file name or file path).

[tashko.zashev]

Hi,

Maybe you have already have it mind, but you can send a job via Backburner with specified frame numbers and pre-calculate the Irr Maps on all of your render nodes.
In order to get the Vrmap per machine, you can set C:\... folder for the output and every machine will write to his own drive.

Other way is to create independent job for every frame with different .Vrmap file name.

[Fredrik Averpil]

Hi Tashko,

I understand what you are suggesting, but I think that this (and previous approaches) are very messy when you are trying to automate this process and keep a firm version control of your renderings, server side.

In our studio we have in-house scripts and systems to chain together and automate processes throughout most software we use. This limits user error to a minimum and lets the users focus on artistry rather than technical issues and render failures.

So far all suggestions on being able to render out the irradiance cache file using multiple machines (before initiating the actual final render job) do not at all fit into this type of systematic idea of a workflow, in my humble opinion - Instead it suits very well if you are a single person working in an environment without rules and automation and where you manually move files around and rename them the way you want. Your suggestion on rendering the vrmaps to C:/ is the best suggestion yet, and I might have to go with this which entails creating render job post scripts to check whether the job was successful, rename the file accordingly and then copy the vrmap over to the actual server-side render folder before our auto vrimg2exr merges all maps into one huge vrmap.

However, I would get very happy if you would consider looking into improving this functionality so that it would indeed be possible to render out the vrmap files in a more straightforward way, into the same folder on the server without less steps on the way that may fail. A feature request, that is

I would be super happy if any of the below would be considered:
- Input box "vraySettings.imap_autoSaveFile" is made scriptable, I could add in the frame numbers via an expression.
- Input box "vraySettings.imap_autoSaveFile" auto-converts any occurance of #### or %04d into the current frame number.
- An additional checkbox "Insert frame number into .vrmap file name"
- A commandline argument (V-Ray for Maya & V-Ray standalone) which makes it possible to define the name of the vrmap filepath (this way I can send different filenames for each frame)

[vlado]

What I would like to do now is to have multiple machines work on the irradiance cache calculation, rather than just one machine – and from the sound of it I should use "Incremental add to current map" to generate one .vrmap for each frame (and then merge them)

While you can do that, it's also a waste of CPU cycles as the same information will be computed over and over again for each frame. It might make more sense to give each machine a segment of the animation to render in one go, f.e. one machine to render out a map for frames 1-100, another for frames 101-200, a third one frames 201-300 etc and then merge the resulting irradiance map files.

... however, I can't figure out how to make each machine write to myIrradianceMap.####.vrmap with the frame numbering in there since the "Auto Save File" in Render Globals is not scriptable and does not understand ####. Hence my first post/question.

Why would it not be scriptable? From what I can see, something like

setAttr -type "string" vraySettings.imap_autoSaveFile "imap_segment_0001.vrmap";

works just fine.

I'm grateful for any advice on the matter because I feel I'm stuck right now unless I can use the "Animation (prepass)" mode to generate the vrmap files.

Actually if only the camera is moving, you can do that without problems. You can even render every N-th frame. But it might still be slower than doing incremental add.

Best regards,
Vlado

[vlado]

- A commandline argument (V-Ray for Maya & V-Ray standalone) which makes it possible to define the name of the vrmap filepath (this way I can send different filenames for each frame)

You can execute an arbitrary pre-render script with Maya batch. There is no problem for this script modify any V-Ray settings in any way.

Best regards,
Vlado

[vlado]

I would be super happy if any of the below would be considered:
- Input box "vraySettings.imap_autoSaveFile" is made scriptable, I could add in the frame numbers via an expression.
- Input box "vraySettings.imap_autoSaveFile" auto-converts any occurance of #### or %04d into the current frame number.
- An additional checkbox "Insert frame number into .vrmap file name"

So if you are rendering a sequence of frames in incremental add mode, which frame should be used for the file name? The first one? The last one? Or should we write a new file for each frame even though the last one contains all information from the previous frames?

Best regards,
Vlado

[vlado]

- Input box "vraySettings.imap_autoSaveFile" is made scriptable, I could add in the frame numbers via an expression.

It *is* scriptable (insofar as you can modify it with a pre-render script). However we can't make it support expressions.

- An additional checkbox "Insert frame number into .vrmap file name"

Nope; there are too many options already

- Input box "vraySettings.imap_autoSaveFile" auto-converts any occurance of #### or %04d into the current frame number.

#### is dangerous as it is a valid file/directory name character and believe or not, people use it in folder names (we've gotten into trouble for that reason before).

That leaves %0nd as a possible variant, but again, it is somewhat ambiguous as to what should happen. I think it would work best if %0nd is replaced with the first frame that gets rendered in the particular rendering process. For example, if you put something like imap_%03d.vrmap and render frames 50 through 70, you'll get one irradiance map file named imap_050.vrmap that contains the baked map for the entire sequence (frames 50 through 70). Would that work for you?

Best regards,
Vlado

[Neilg]

#### is dangerous as it is a valid file/directory name character and believe or not, people use it in folder names (we've gotten into trouble for that reason before).

Hah! that's amazing.