V-Ray Scene Optimisation Script

[^Lele^]

Script update to 1.45R

I hoped to be able to post the scene with the new version, but alas, here's the script.
This will not touch your DR settings (or rather, restore them to what they were before the script was run) and the max ray intensity (turning it off is tied to something else i'm working on, but it's creating more problems than it solves, i can see,f or most people.).
It's got a checkbox to turn off all and any VFB operations (so with the checkbox unticked, your last VFB image will be CLEARED as vray resets to defaults. Save it yourself, if you so wish.)
It also has a way to control the LightCache sample rate and sample size the way i described in a previous post: the default of 1.0 will create one sample per pixel (so, no matter what the render size is, the script will take care of your subdivs), and the sample size is expressed in pixels, rather than linear fraction of the image size.
the presets have been updated to include the LC sampling, but other than that, no change has been made tot heir general sampling behaviour.

Did i forget any of the requests?

EDIT: please be kind and let me know if something's misbehaving.

RE-EDIT: I took the script down. Why? 'cause i'm a dolt. Someone did write about the AA threshold being 0.1, and i read it as 0.01. Only at ten to five in the morning i realised it was indeed 0.1, after spending most of the night trying to figure out why V-Ray was misbehaving (lesson i ought to have learned: the issue is oftentimes between keyboard and chair. And that's where i ought to have looked first.)
On the plus side, i shall post the new version quite soon, and hopefully with a few sample images to follow (as the renders get done).
The new version will have an optimised ArchViz preset, which will part from some speed to allow for more precision in the final image render, namely by letting the LC be filtered at rendertime, rather than pre-filtered, so to take care of any appearing light leaks through thin geo (which are a lot less of an issue in the VFX world).

[^Lele^]

Allright, here it goes.
Contrary to what i stated before, the ArchViz preset will NOT filter the LC at rendertime,as that would be overkill in terms of speed.
I found that oversampling (ie. shooting loads more LC photons, in the specific case, 16 per pixel) with a decent enough radius (10 pixels, but it's not cast in stone: experiment) plus PRE-filtering will result in just as clean a GI (no leaks, splotches, thin details missed), but with a minimum of speed impact.
To give you an idea of the difference, just turning the LC to rendertime filtering would kick the rendertimes to three times longer (on my puny machine, for the test scene, from ~1 to ~3 hours.), while the shooting and pre-filtering, altogether, would only add three minutes to the render completion.

Compared to the VFX-Std preset, the ArchViz renders for around 50% to 75% longer, however i believe the results will be quite pleasing to most: hard to light interiors (like those images i posted), with sub-pixel detail and plenty of glossy inter-reflections are EXTREMELY clean in all the elements: from direct lighting, to GI , to reflections, to geometry definition.

As usual, give it a whirl if you want, and post results if you please.
It still won't work (nor it ever will.) with anything but BF/LC, so please do not attempt to render with an IRMap again.

A post with images showing the test render and its relevant elements will follow in a bit.

[^Lele^]

Render Results from the ArchViz Preset

Here are the images i tested the archviz preset with.
The rendertime on my puny AMD FX8350 @ 4.4 Ghz was 1h30mins.
That's translate in around 30 minutes on a dual xeon (the same for which the first post's benchmark was 22minutes odd).
Of particular note is the sample rate element: you will notice the AA steps up only at geometric edges.

The scene and maps will follow in further posts for anybody interested to give it a try with various setup approaches.

[^Lele^]

Sample scene for benchmarking

The scene:
https://drive.google.com/file/d/0B1P...it?usp=sharing

The maps (unpack in the same folder as the scene to avoid re-linking. the garlic plant maps are standard with the max distribution):
https://drive.google.com/file/d/0B1P...it?usp=sharing

Have a ball.

[^Lele^]

Unit Tests showing the script core ideas and mechanics

I ought to have started with this post, but I was busy with something else, and didn't quite find the time to prepare these.
The goal of this post is to show, with images AND numbers (sorry. it won't be very hard maths, as I am incapable of any.) why I devised this type of approach rather than relying on a generic "Universal Settings" one.
To keep things as easy and followable as possible, i will tackle one part of the V-Ray shading engine at a time, under extremely simple, and extremely controlled scenarios, or what would be called Normalised conditions.

The screen is a 10,000px image, or 100x100 pixels (this to keep numbers decent).
The geometry is a simple plane, parallel to the ground, located at the origin, with a purely diffuse, 0.5 float white shader applied, under linear workflow conditions.
Lighting is done with a white, 1.0 float, dome light, or what in general Lighting terms is called GanzField Illumination (German for "Entire Field"): light comes at the same intensity from all directions.
For this unit test, I will choose to work with Fixed AA so to do away with any form of adaptivity and smart algorithm from the part of V-ray, and in turn to make sure the maths is as straightforward as possible.
Further, although it's not immediately of any use, I set the noise threshold, the adaptive amount and the light and shader cutoff values to 0.

So, if we chose a fixed, 1 AA mode, and had the dome light with 1 subdivision, we should get V-Ray cast 10,000 camera rays, and 10,000 shadow rays.
Sure enough, the log shows

[2014/Aug/9|19:08:31]         Number of raycasts: 20000
[2014/Aug/9|19:08:31]           Camera rays: 10000
[2014/Aug/9|19:08:31]           Shadow rays: 10000

the image looks like this:

Seemingly uninteresting, but it proves the point.

Let's now suppose we set the AA to a fixed 10, or 100 (10^2) rays per pixel, leaving the dome at 1 subdivision.
This ought to cast a hundred more camera rays, which will multiply the shadow rays, resulting in a hundred more shadow rays per pixel, as each camera ray spawns a new shadow ray.
Indeed, here's what the log says (slight error aside):

[2014/Aug/9|19:14:33]         Number of raycasts: 1999999
[2014/Aug/9|19:14:33]           Camera rays: 1000000
[2014/Aug/9|19:14:33]           Shadow rays: 999999

Here's the rendered image:


the image now looks a LOT cleaner, and it's no mistake, as the numbers show.

Just how much cleaner, you ask, and can it even be evaluated?
It can indeed be theorised, and measured, and maybe you even guessed the factor: it's 100 times cleaner.
In fact the original picture had a variance of + or - 0.5 (as in this scenario, the rendered colour is simply the result of the diffuse value times the light value, or 0.5 * 1.0), with pixels close to 1.0 float, and others close to 0.0.
In the second image, the colour variance is of + or - 0.005, proving the above point.
You will need to render from within V-Ray's framebuffer to validate the numbers, as the PNG format will quantize the values to 8 bits (making these uploaded images here for show and clarity, not for double-checking of the results)

Let's now play with settings a bit, reverting to a fixed 1 AA, but raising the dome to 10 subdivisions.
This should give us 10,000 camera rays, plus 10^2, or 100, shadow rays per pixel, for a total of (10,000*100) 1 million shadow rays, as we will only cast one camera ray per pixel.

[2014/Aug/9|19:56:15]         Number of raycasts: 1009999
[2014/Aug/9|19:56:15]           Camera rays: 10000
[2014/Aug/9|19:56:15]           Shadow rays: 999999

And the image looks like this:

The variance in pixel colour is STILL + or - 0.005 (albeit we have no geometric anti-aliasing, of course)
However this image took one third of the rendertime, for around half of the total rays cast.
Of course, the image taking tenths of a second to render, it's not an accurate measurement of performance.
We will get deeper into performance analysis later on, with bigger image sizes which will lead to more significant benchmark values.
However, shading-wise, you will be hard pressed to disagree on one key fact: the last two images are identical, down to the third decimal digit of colour values.


I shall end the unit test here (there's a limit on postable images, and I will need more in the next part).
In the next post, supported by the evidence of this test, we will start taking a look at the Adaptive DMC AA behaviour.

[^Lele^]

The Adaptive (DMC) AA

The scene setup is identical to the previous one, but for a few key differences in the DMC sampler and AA settings: we will return to the default values for both, or in other words a 1-4 adaptive AA a noise threshold of 0.01 and an adaptive amount of 0.85.
This is to allow V-Ray to use its adaptation routines which will decide for the user when to stop tracing, something very useful to avoid the so called "ray explosion".
With the default of 1-4, we're asking V-Ray to trace 1 to (4^2) 16 camera rays per pixel.
As we have seen in the previous post, for each camera ray, the rendered would be forced to cast ALL the shadow rays (any type of shading ray, for that matter. in this test we only have the two types, so to keep things nice and simple), were it not for a pretty clever trick Vlado introduced.
In fact, let's demonstrate the approach with two renders.
In the first, I will leave the AA to fixed-1, and the light subdivs to 4 (which, I'm sure it's second nature by now, will render 160000 shadow rays and 10000 camera rays).
The image looks like this:

Now, let's revert to the 1-4 adaptive AA.
The image should render cleaner, right?
Here it is:

Wrong!
Is the AA working, at all?
If we had geometric edges visible we'd know right away it is, but since we haven't I'll show you what the sample rate element looks like:

It's red, meaning V-Ray has indeed decided (as it's not down to us, right now, but to the noise threshold and adaptive amount values) to AA at the maximum rate the whole image.

This had me baffled, way back in 2008, and since then, I tried to find out more about this seemingly odd behaviour.

Let's take a look at what the render log says for both images:

[2014/Aug/9|20:40:44]         Number of raycasts: 170000
[2014/Aug/9|20:40:44]           Camera rays: 10000
[2014/Aug/9|20:40:44]           Shadow rays: 160000

[2014/Aug/9|20:42:36]         Number of raycasts: 320000
[2014/Aug/9|20:42:36]           Camera rays: 160000
[2014/Aug/9|20:42:36]           Shadow rays: 160000

Notice one key thing: the number of shadow rays stays EXACTLY the same for both images, but the number of camera rays reflects what we could infer from the sample rate: it's casting 160k (4^2*10,000) camera rays, anti-aliasing the whole image region.
So we can deduce what the trick is: the MAX AA Subdivs value DIVIDES the Shading Subdivs on lights and shaders, so to keep a consistent number of rays per pixel cast by lights and shaders, regardless of the AA settings.
In one way, we get clean shading AND clean geometric anti-aliasing, nearly for free.
Were it not for this, what would happen to our renders?
In V-Ray 3.0 we can test this, thanks to the "Divide Shading Subdivs" option, which when turned off will stop the magic trick from happening.

For this test only, i will also turn back adaptive amount and noise threshold to 0, so to avoid the adaptation routines which may skew the numbers beyond (easy) recognition.
As usual, let's first try and find out what to expect: we ought to have the same camera rays as the previous render, but 16 times more shadow rays, as each additional camera ray will spawn the full set of shadow rays (just as in the fixed-AA scenario from the previous post).
So, for the math, 160k camera rays, and 2,560,000 shadow rays, for a substantially cleaner image.
So here's the render, and corresponding log:

[2014/Aug/9|20:56:29]         Number of raycasts: 2719999
[2014/Aug/9|20:56:29]           Camera rays: 160000
[2014/Aug/9|20:56:29]           Shadow rays: 2559999

Lo and behold, we did get the maths right, and the correspondingly cleaner image.
The Shading Rate element looks bright red, as expected.
Timing wise, the first image took the least time to render, obviously, as it had the least amount of overall rays cast, the second took around twice the time, and the latter took around four times the time.
However, as stated before, this is just too quick a render to infer any relevant benchmarking, and in the next post we'll make sure to be able to do just that.

[Paul Oblomov]

Oh how nice to read all this math
May I force the testing scene, to smth corona users are used to ? I have it converted to vray, if you ask.

[^Lele^]

Licence: Permission is granted to use the scene for personal learning purposes. It is strictly forbidden to sell the scene or any of its parts, or to claim the scene or its renders as own. Name of the original author (Ludvik Koutny) must be attached to any renders of the original or modified scene.

Umh, are you sure?
I'd rather get something of original make.
I can see trouble, especially if for any reason we manage a similar look but faster and cleaner ...

[LudvikKoutny]

Actually,

i am the author of that scene, and you are free to use it even for comparison purposes. Only thing in reward i would like is to keep the comparisons fair and objective. That means for example not comparing irradiance cache to path tracing as primary GI.

I would have no problem with Vray being faster than Corona. I use Vray at work daily and i am fully aware of it's strong points, good irradiance caching being one of them. So we all know, that IM+LC combo in Vray would obliterate Corona in terms of speed/quality ratio in this particular scene. But in case of BF+LC, results may be a bit more even.

Also, in case you would be using sky portals in Vray, then you should not forget Corona has them too, so it would again be appropriate to use them in Corona as well.

That's all... feel free to download that scene and use it for comparison purposes

[^Lele^]

Camera Rays versus Specialised Rays: simple benchmarking.

Continuing on the line of thought from the previous posts, it's now time to find out if there is any difference in cleaning an image through the ubiquitous camera rays (ubiquitous in that without those, you wouldn't get an image: they are always cast, in order to "find" what to shade.), rather than cleaning it with the so called "specialised" rays (shadow, reflection, refraction, GI rays, to be more specific).
To do so, i'll make the image size ten times bigger, at 1000*1000 pixels, which will give us a more representative time with which to calculate the speeds, one "control" render with AA at fixed-1 to have the least camera rays possible, and two renders which should look identical, and cast the same amount of shadow rays (remember we only have those and the camera rays, in our over-simplified setup), but a different amount of camera rays: one image will then be better anti-aliased than the other, geometric edges-wise, but the amount of color variance in the pixels, or image cleanliness, will have to be the exact same.
We will once again turn off any adaptivity (adaptive amount and noise threshold to 0, cutoffs on lights and shaders to 0) in order to have the engine cast exactly as many rays as we set in the various parts of the scene.
To get a clean image under our present conditions, I set the dome light to 16 subdivisions, or 256 (16^2) rays per pixel.
This will cast 256 million shadow rays, 1 million (1000*1000) camera rays.
The variance per pixel, in this case, is + or - 0.001 float (ie. our pixels range from 0.499 to 0.501 float, when we set the shader to be 0.500), so i suppose we can all agree the image is to all intents and purposes, noiseless.

Minor rounding to the side, we ended up casting the 257 million (256 + 1) rays we expected, in 13.6 seconds.

[2014/Aug/10|14:09:08]         Number of raycasts: 256999728
[2014/Aug/10|14:09:08]           Camera rays: 1000000
[2014/Aug/10|14:09:08]           Shadow rays: 255999728
[...]
[2014/Aug/10|14:09:08]         Region rendering: 13.6 s

Since we want to have the same shadow rays cast, we will keep the "divide shading subdivs" active, and switch AA to fixed 4.
We ought to get the same shadow rays, and 16 times more camera rays, for an identical looking (albeit anti-aliased) shaded patch.

Which we do (are we not solid in our predictions, now? ), but watch what happens to the render times:

[2014/Aug/10|14:12:25]         Number of raycasts: 271999728
[2014/Aug/10|14:12:25]           Camera rays: 16000000
[2014/Aug/10|14:12:25]           Shadow rays: 255999728
[...]
[2014/Aug/10|14:12:25]         Region rendering: 23.1 s

This does make sense, of course, as we are casting so many more camera rays (we will get to the performance numerical comparisons later on, not to worry.) and getting our edges anti-aliased.
Now for the second anti-aliased image: a fixed-16, "divide" on.
We ought to get 256 million camera rays, and 256 million shadow rays.
Let's put this to the test:

We have the 512 million (minus small change) rays expected, but our render times are a whole lot bigger: 175.7 seconds

[2014/Aug/10|14:22:25]         Number of raycasts: 511999682
[2014/Aug/10|14:22:25]           Camera rays: 256000000
[2014/Aug/10|14:22:25]           Shadow rays: 255999682
[...]
[2014/Aug/10|14:22:25]         Region rendering: 175.7 s

The pixel variance in all three stays precisely the same, on count of the same(ish) number of shadow rays cast, but the times are vastly different.
Let's normalise the time to account for the number of camera rays: since the shadow rays stay the same, we should be able to subtract the fixed-1 render time from our calculations, although we will incur in a little bit of precision waste as even the fixed-1 casts 1 million camera rays.
Let's benchmark those as well, as they get cast even if the scene is empty.
I'll hide the plane, and render again, for 0.7 seconds, which is the time we will subtract from the fixed-1 image region render time.
Our base time to cast the shadow rays is then 12.9 seconds (13.6-0.7).

16 million camera rays, then, took 23.1-12.9 seconds, or 10.2 seconds, or 0.0000006375 seconds each (i know: the accuracy here is QUITE on the loose side.)
256 million camera rays took 175.7-12.9 seconds, or 162.8 seconds, or 0.0000006359 seconds each.
Given our accuracy being limited, we can infer they are indeed being cast at the same speed (how could they not, after all?).
How fast are our shadow rays, instead?
256 million shadow rays, in the first image, took, we calculated earlier, 12.9 seconds, which leads to 0.00000005039, or around TEN TIMES LESS than the camera rays (one just need to count the zeroes after the dot).
This behaviour is common across all specialised rays.


So here comes the, by now obvious, I hope, conclusion: in any type of scene setup you will ever face, if speed is a concern, you will want to use the camera rays (AA rays, if you please) as sparingly as you can possibly afford.
My overarching suggestion (and the script aids in PRECISELY this department) is this: FIRST set up the AA so that your geometric edges and high frequency textures are resolved cleanly (if there is motion blur or depth of field at play, it makes no conceptual difference, you will just have to use more camera rays to get those effects clean) with the LEAST amount of camera rays possible, THEN increase the specialised rays so that your shading effects look clean under those anti-aliasing conditions.

There, now I am out of a job, as I won't be needed to speed up other people's renders.
Talk about an own goal. ^^

If anyone is marginally interested, I may delve even deeper into the inner workings of V-Ray's adaptive tracing routines, and show (I loathe arbitrary opinions if they aren't grounded in facts. and I sincerely hope i stuck to facts alone, with these posts. Should I have made mistakes, please let me know!) why, when and how to reduce their influence on your renders.

[Companioncube]

i'm going to have to read a few times to fully understand, just wanted to thank you for sharing

[Paul Oblomov]

If you guys are willing to compare - here we go. I tried to make it very similar, every material converted by hands
Credits are inside modified frame stamp. No worries.
http://rghost.ru/57395214

[^Lele^]

Well, thank you Ludvik and Paul.
I gave the scene a whirl, but made a couple of modifications to what you prepared, Paul, which i'll list here:
*Converted all the bitmap loaders to VRayHDRI, to make use of the better and faster filtering on offer.
*Reverted all the bitmaps filtering "blur" to 1.0 (I won't go into details as to why, here.)
*Shelled the roof and walls on the outside, to avoid any light leaks (not that i saw any, but it's a standard practice for me.)
*Turned off "use LC for glossy rays" as i wanted to render with direct calculation of glossies
*Put the self illuminated plane (outside of the window) material into a VrayMtlWrapper to make sure it didn't cast any GI, as the sky and sun are already present
*Instanced the dome and rotated it 180 degrees on Z, and turned off "full dome", as this allows for faster speeds (again, i won't go into the details as to why that is, here.)
*I hid, and didn't use, any light portal.

I then ran my script, selected the archviz preset, after making ONE change to the direct lighting subdivs (10 instead of 8 )for a total min of 1600SPP

this is the link to the modified scene:
https://drive.google.com/file/d/0B1P...it?usp=sharing
this is the link to the full complement of render elements' EXR:
https://drive.google.com/file/d/0B1P...it?usp=sharing

FYI, i ain't too happy about the noise in the reflection and spec channel, but I didn't want to re-render it cleaner as my workstation is on the snail-side... ^^
I am guessing adding another 10 minutes to my render time would give us spotless reflections and speculars.
For those which paid attention to my earlier claim that no matter the contents (and given a comparable amount of light fixtures, i should have noted! then again, with "probabilistic lights" active, it may well matter less how many fixtures one has in the scene...) renders of the same size ought to take around the same time to complete, you will find that that claim holds true in this case, when compared to the previous test scene.
This is around 8% smaller a screen size, and took around 15% less, thanks chiefly to a much cleaner (ie. less AA stepping up) samplerate element, due to less thin geometric detail.

Here are some of the relevant channels as PNGs:

[^Lele^]

for comparions, and outside of the boundaries of this thread, this is what the progressive sampler rendered on my AMD in 3.5 minutes with a Min Sample Rate of 8.



And again, 3.5 minutes, but with a MSR of 32:



Those thin specular lines on the floor, without any image filtering, definitely require more camera rays.

[AlexP]

Lele Ive PMd you a dropbox link to a bunch of stuff from over the weekend.
Some bits will take a bit of time to upload.