[HOU-1458] Writing Deepdata on the farm

[GoodbyeKansas]

When rendering locally we can generate Deepdata but on the farm the file that comes out has no deepdata in it.
Is there some extra flag needed for command line rendering?

Command looks like:

$VRAYHOUDINI_BIN -display=0 -noFrameNumbers=0 -frames=1001 -sceneFile=${NS_TMP}/1001.vrscene -imgFile=/path/asdfasdfasdf/foobar.#.exr

EDIT: Was a bunch of mumbojumbo in here first because I had a brainfart.

-- Erik

[GoodbyeKansas]

Inspecting the vrscene file I can see img_deepFile=0; compared to img_deepFile=1; when generating it locally.

-- Erik

[GoodbyeKansas]

Nevermind. It was a part of our internal code screwing it up.

-- Erik

[GoodbyeKansas]

Or maybe not. Fixed so that img_deepFile=1 in the vrscene file but still get a file without deepdata.

-- Erik

[GoodbyeKansas]

first 10000 chars of vrscene file:

/////////////////////////////////////////
// Exported by V-Ray ADV Plugins Exporter
// V-Ray core version is 4.30.00 built at Nov 14 16:15:48 2019 13b27e3
// Exported at Wed Dec 11 21:11:59 2019
// V-Ray for Houdini [fb6521b]
/////////////////////////////////////////

SettingsPhotonMap settingsPhotonMap {
  bounces=10;
  max_photons=30;
  prefilter=0;
  prefilter_samples=10;
  mode=0;
  auto_search_distance=1;
  convex_hull_estimate=0;
  dont_delete=1;
  auto_save=0;
  retrace_bounces=10;
  store_direct_light=1;
  search_distance=20;
  multiplier=1;
  retrace_corners=0;
  max_density=0;
  file="";
  auto_save_file="";
}

// Frame at time 1002

SettingsRTEngine vraySettingsRTEngine {
  trace_depth=5;
  gi_depth=3;
  cpu_bundle_size=64;
  cpu_samples_per_pixel=4;
  gpu_bundle_size=192;
  gpu_samples_per_pixel=16;
  coherent_tracing=0;
  stereo_mode=0;
  stereo_eye_distance=0;
  stereo_focus=0;
  opencl_texsize=512;
  opencl_resizeTextures=0;
  opencl_textureFormat=2;
  progressive_samples_per_pixel=1;
  undersampling=4;
  disable_render_elements=0;
  max_render_time=0;
  max_sample_level=10000;
  noise_threshold=0.01;
  enable_mask=0;
  max_draw_interval=0;
  min_draw_interval=0;
  enable_bucket_sampler=0;
}

SettingsEXR vraySettingsEXR {
  compression=3;
  bits_per_channel=16;
  extra_attributes="task_id=59555dda-f4d4-11e9-8bae-000c29353009";
  auto_data_window=0;
  write_integer_ids=0;
  multipart=0;
  deep_alpha_colors=0;
}

SettingsOutput vraySettingsOutput {
  img_width=1280;
  img_height=720;
  img_pixelAspect=1;
  img_file="foobar.<frame04>.exr";
  img_dir="/path/to/stuff/";
  img_file_needFrameNumber=0;
  img_separateAlpha=0;
  img_noAlpha=0;
  img_deepFile=1;
  img_rawFile=1;
  img_rawFileVFB=1;
  img_clearMode=0;
  anim_start=1002;
  anim_end=1002;
  frame_start=1002;
  frames_per_second=1;
  frames=List(1002);
  rgn_left=0;
  rgn_width=1280;
  rgn_top=0;
  rgn_height=720;
  bmp_width=1280;
  bmp_height=720;
  r_left=0;
  r_width=1280;
  r_top=0;
  r_height=720;
  relements_separateFolders=0;
  relements_separate_rgba=0;
  film_offset_x=0;
  film_offset_y=-0;
}

SettingsOptions vraySettingsOptions {
  geom_displacement=1;
  geom_doHidden=0;
  light_doLights=1;
  light_doDefaultLights=1;
  light_doHiddenLights=0;
  light_doShadows=1;
  light_onlyGI=0;
  light_disableSelfIllumination=0;
  gi_dontRenderImage=0;
  mtl_reflectionRefraction=1;
  mtl_limitDepth=0;
  mtl_maxDepth=5;
  mtl_doMaps=1;
  mtl_filterMaps=1;
  mtl_SSSEnabled=1;
  mtl_filterMapsForSecondaryRays=1;
  mtl_transpMaxLevels=50;
  mtl_transpCutoff=0.001;
  mtl_glossy=1;
  mtl_uninvertedNormalBump=0;
  geom_backfaceCull=0;
  ray_bias=0;
  gi_texFilteringMultiplier=5;
  misc_abortOnMissingAsset=0;
  misc_transferAssets=0;
  misc_useCachedAssets=1;
  dr_assetsCacheLimitType=0;
  dr_assetsCacheLimitValue=0;
  dr_overwriteLocalCacheSettings=0;
  ray_max_intensity_on=1;
  ray_max_intensity=20;
  probabilistic_lights_on=2;
  num_probabilistic_lights=8;
  probabilistic_vol_on=0;
  probabilistic_vol_samples=20;
  probabilistic_vol_gi_samples=3;
  misc_lowThreadPriority=0;
  misc_consistentLightingElements=1;
}

SettingsImageSampler vraySettingsImageSampler {
  type=1;
  min_shade_rate=6;
  dmc_minSubdivs=1;
  dmc_maxSubdivs=24;
  dmc_threshold=0.01;
  subdivision_minRate=-1;
  subdivision_maxRate=2;
  subdivision_jitter=1;
  subdivision_threshold=0.15;
  subdivision_edges=0;
  subdivision_normals=0;
  subdivision_normals_threshold=0.1;
  progressive_minSubdivs=1;
  progressive_maxSubdivs=100;
  progressive_threshold=0.01;
  progressive_maxTime=0;
  progressive_bundleSize=128;
  progressive_showMask=0;
  progressive_progNoiseThreshold=30;
  progressive_effectsUpdate=20;
  render_mask_mode=0;
}

SettingsUnitsInfo vraySettingsUnitsInfo {
  meters_scale=1;
  photometric_scale=0.002;
  scene_upDir=Vector(0, 1, 0);
  seconds_scale=0.03333334;
  frames_scale=30;
}

SettingsColorMapping vraySettingsColorMapping {
  type=6;
  affect_background=1;
  dark_mult=1;
  bright_mult=1;
  gamma=2.2;
  subpixel_mapping=0;
  clamp_output=0;
  clamp_level=1;
  adaptation_only=1;
  linearWorkflow=0;
}

SettingsGI vraySettingsGI {
  on=1;
  refract_caustics=1;
  reflect_caustics=1;
  saturation=1;
  contrast=1;
  contrast_base=0.5;
  primary_engine=2;
  primary_multiplier=1;
  secondary_engine=3;
  secondary_multiplier=1;
  ray_distance_on=0;
  ray_distance=100;
  ao_on=0;
  ao_amount=0.8;
  ao_radius=10;
  ao_subdivs=8;
}

SettingsIrradianceMap vraySettingsIrradianceMap {
  min_rate=-3;
  max_rate=0;
  subdivs=50;
  interp_samples=20;
  calc_interp_samples=15;
  interp_frames=2;
  color_threshold=0.3;
  normal_threshold=0.1;
  distance_threshold=0.1;
  detail_radius=60;
  detail_subdivs_mult=0.3;
  detail_scale=0;
  randomize_samples=1;
  interpolation_mode=1;
  lookup_mode=3;
  mode=0;
  dont_delete=1;
  show_samples=0;
  show_calc_phase=1;
  show_direct_light=1;
  multiple_views=0;
  multipass=1;
  check_sample_visibility=0;
  auto_save=0;
}

SettingsDMCGI vraySettingsDMCGI {
  subdivs=8;
  depth=3;
}

SettingsLightLinker settingsLightLinker {
  ignored_lights=List(
    List(|obj|vrayLightDome|light|87FCD5C, |obj|sphere_object1|mesh|E5A942CC|node)
  );
  ignored_shadow_lights=List(
    List(|obj|vrayLightDome|light|87FCD5C, |obj|sphere_object1|mesh|E5A942CC|node)
  );
  include_exclude_light_flags=ListIntHex("ZIPC040000000C000000eAFjZGBgAAAACAAC");
  include_exclude_shadow_flags=ListIntHex("ZIPC040000000C000000eAFjZGBgAAAACAAC");
}

[psanitra]

I went trough this whole craziness of editing vrscene files, deep vs elements vs depths around half year ago. To save you some time - adding img_deepFile is not enough, you need to add vrayOutputDeepWriter plugin too, something like this:

OutputDeepWriter vrayOutputDeepWriter {
file="/mnt/render-share/out/shot010/LOTR_TR_0010-cg_lighting_RENDER_v003_BgRL.beauty.deep.exr";
float_colors_mode=1;
exr_compression=2;
}

[GoodbyeKansas]

That don't seem to do anything here :/

-- Erik

[Gosho.Genchev]

Hey, Erik,

we export the following when Deep is enabled for the image output.

img_deepFile=1;
img_rawFile=1;

Those are flipped for hython when the export mode is set to Export&Render but seem correct if set to Export only. We'll fix this, of course. Scratch that, my mistake.I still have no clue why this happens for you but I'll keep digging.

The issue you're having on the farm seems to be caused by the imgFile flag.The dev team has been notified and I hope we'll fix this ASAP but until then, you can use the -parameterOverride option to specify the output folder and output image name. E.g.

$VFH/ -display=0 -noFrameNumbers=0 -frames=1001 -sceneFile=path/exportOnly_deepTrue.vrscene
-parameterOverride="SettingsOutput::img_dir=Path/to/dir"
-parameterOverride="SettingsOutput::img_file=output .<reName>.<frame04>.exr"


Please also note the <reName> and <frame04> flags in the image file name - those are the equivalent to $AOV and $F4 in the Houdini UI.


Cheers!

[GoodbyeKansas]

Just to recap so the only thing I need to do to get it going is to add <reName> to our output path filename if deep is enabled?

-- Erik

[bdancer]

No, there is an issue with "-imgFile=" command line option and deep writer.
So, instead of using "-imgFile=" command line option use "-parameterOverride=" command line argument to override SettingsOutput::img_dir and SettingsOutput::img_file.

So, instead of this:

$VRAYHOUDINI_BIN -display=0 -noFrameNumbers=0 -frames=1001 -sceneFile=${NS_TMP}/1001.vrscene -imgFile=/path/asdfasdfasdf/foobar.#.exr

Use this:

$VRAYHOUDINI_BIN -display=0 -noFrameNumbers=0 -frames=1001 -sceneFile=${NS_TMP}/1001.vrscene -parameterOverride="SettingsOutput::img_dir=/path/asdfasdfasdf/" -parameterOverride="SettingsOutput::img_file=foobar.<frame04>.exr"

Will try to fix "-imgFile=" asap.

[GoodbyeKansas]

Cool cool. Is the ETA like sometimes next week or further in the future? Wondering if its worth modifying our farm submitter?

-- Erik

[bdancer]

> Is the ETA like sometimes next week

Yes, smth like "sometimes next week"...

[GoodbyeKansas]

Then i'll just chill but for your info I added those overrides and then it did not write out any file at all even tho it says overridden successfully in the log.

[2019/Dec/12|15:15:45] [48 MB] Parameter 'vraySettingsOutput.img_dir' overridden successfully.
[2019/Dec/12|15:15:45] [48 MB] Parameter 'vraySettingsOutput.img_file' overridden successfully.

-- Erik

[bdancer]

Fido Film, could I see the full command line? And what the log says.

[blented]

We're having this same issue, I can confirm that it's not writing img_depFile=1 when outputing the .vrscene