Bucket and Progressive Samplers' Measured Performance

[^Lele^]

Thanks to Olli's scene we could profile a very detailed, production grade car, at very high resolutions (around 60 MPx.).
So we now have a real-world measured performance chart for bucket versus progressive, with and without (many) Render Elements, and at increasing resolutions.

Tests conducted in max 2017.
The render was set up as a 1-100, 0.025 N.T.
The reason for the max AA subdivs is to make sure the noise threshold was met, and we didn't hide from stuck buckets or long sampling.
The higher Noise Threshold, instead, was chosen so to finish the profiling in a reasonable time, and should carry absolutely no conceptual difference from the same test carried with a lower Threshold.
The image resolution was 8961 x 6723 Px (~ 60.2 MPx), while the 16 Render Elements (when on) consisted of the usual set of beauty elements, and nine multi-mattes.

Sampler Type	Render Elements Active	Resolution	Render Time (Seconds)	Slower than Quickest Task	Slower than Bucket Sampler
Bucket	No	Quarter (~2k)	4195.27	0.00%	0.00%
Bucket	No	Half (~4k)	16558.8	0.00%	0.00%
Bucket	No	Full (~8k)	64265.5	0.00%	0.00%
Bucket	Yes	Quarter (~2k)	4589.23	8.58%	0.00%
Bucket	Yes	Half (~4k)	17591.5	5.87%	0.00%
Bucket	Yes	Full (~8k)	68197.3	5.77%	0.00%
Progressive	No	Quarter (~2k)	4289.12	2.19%	2.19%
Progressive	No	Half (~4k)	18955.7	12.64%	12.64%
Progressive	No	Full (~8k)	67315.7	4.53%	4.53%
Progressive	Yes	Quarter (~2k)	4671.05	10.19%	1.75%
Progressive	Yes	Half (~4k)	18301.9	9.52%	3.88%
Progressive	Yes	Full (~8k)	85519.2	24.85%	20.25%

So, the first three rows are our "zero", the fastest way to get to an RGBA image: bucket sampler, and no Render Elements.
The next three rows are again the bucket sampler, but with render element active: we can see the slack is between 5 and 9 percent, give or take decimals.
Now, the first thre progressive sampler rows are calculated without Render Elements, and the performance slack with the bucket sampler doing the same job is between 2 and 4%. The middle result, showing a 12% difference, should be currently treated as an outlier, as i believe windows updated mid-job (don't ask...) (EDIT: I think it was windows defender working on a schedule.).
The last triplet of rows in the table above has the worst penalty for the progressive sampler, where compared to a bucket *without* REs, it's 10% or so slower, until the resolution is very big, and then it jumps to a 24%.
Notice the differences between Samplers (instead of tasks and samplers) are *much* more contained, for all but the two outlier values.

The test is being re-run to validate the results, so for now take with a pinch of salt.

[kosso_olli]

Lele, thanks for taking the time to profile these settings! It's good to see a comparison with tangible numbers. I think for the amount of given render elements and the resolution, the hit in render time has to be accepted. If it could be further improved though, I would be happy.
Did you also compare the rendered output? I remember the progressive sampler handled the out of focus highlights much better, which I like. Bucket tended to produce a noisy result in these areas.
Also, the settings for the ray bundle size were at default of 128? And one question regarding the noise threhsold: It really was 0.025? Did you miss a zero was that on purpose to make the test reasonably quick?

[^Lele^]

Mind you, i am really not convinced the two outliers can be taken at face value.
Those would be the half res progressive without REs, and the full res progressive with the REs.
The issue was that windows defender ran once a day on the -very full of data on disks- slave.
So the jobs which took the longer may have incurred into a one or two system scans, skewing the results.
I since deactivated the task in the task scheduler, and the new batch of tests is running as we speak, so we'll see then.
Should we still have doubts after a rerun, i'll try and get hold of one of the many-cored machines in the office, so to rerun these tests a few times, but much more quickly.

RBS was at the default, yes.
I used 0.025 on purpose, yes, to finish in a decent time.
For the resolution these are at, should they go to print, it'd be mighty fine i'd reckon.

I'll do proper image analysis in a little bit, but there are things i learnt in the while about how V-Ray goes about noise threshold checking which are cool to share, i think.
A) The progressive sampler's noise threshold checking is fully multithreaded, and done in a way which retains performance (and i'd say the numbers tend to confirm that. Rado's no slouch...).
B) The bucket sampler isn't as naive in Noise Checking as i had assumed: it'll do a very competent job across buckets too, if of course not always as comprehensive as the progressive. If detail seems amiss, particularly with DoF or Moblur (but very fine geo would count as well), just raise min AA samples (2, 3 subdivs.), and there should be noticeably more parity between the two Samplers' results.

[M.Max]

Thank you very much for doing these tests! very interesting results here

[JD3D_CGI]

I would be curious to see some results with notorious stuck buckets.

[^Lele^]

The render was set up as a 1-100, 0.025 N.T.
The reason for the max AA subdivs is to make sure the noise threshold was met, and we didn't hide from stuck buckets or long sampling.

There was plenty of "stuck" material in the scene, so you can consider the test comprehensive in that respect.
If there was a stuck bucket somewhere, it eventually came unstuck (notice the render times...).

[jasonhuang1115]

subscribed. Thanks for posting the tests, Lele.