Is there any hack to eliminate the absolutely awful halo around everything that makes ‘denoise alpha’ a pointless exercise?
I have noisy shadows that I would love to fix.
Not sure exactly what you are doing, but perhaps you could use a white override material (or duplicate object) and basically catch the shadows as a separate render (invert to get a mask to apply CC in post).
That way your shadow matte is in RGB, not alpha.
I went without denoiser in the end and just upped the frame time, then rendered smaller and upscaled, equalling out the total render time v quality.
The scene is a load of objects falling and interacting with a back wall and floor plane, with the problem being large noisy diffused shadows on the wall, that were ofc sharper on the floor as contacts,
so initially I wanted to denoise everything, but couldn’t because of this awful issue.
The denoise alpha is on by default, which is a UI mistake, as it’s clearly broken/useless so should come with a warning 
Regardless, the next rabbit hole was trying to find a simple method of extracting only the shadows on the wall/floor. I couldn’t, as all the elements I looked at contained the shadows of all else.
So I thought to overlay the object select element with the matte/shadow pass, however it turns out that the object select element doesn’t keep the vfb corrections, so that wasted even more time.
The main issue was that of course the deadline was insane, the goalposts had a life of their own and each change propagated to each element of complexity I would have had to introduce in the render/s to try to
work around the Vray issues. Such is life,…anyway it’s almost done and looks fine 
what vfb corrections did you need to reproduce in post?
I think there was just Hue/sat, maybe curves, but regardless, I just wanted a simple pipeline that didn’t need me adding other steps each time, as they kept/keep changing things, so iterating and remembering to apply
extra steps to match output gets annoying and/or confusing, when basically it was just the shadows causing issues, if you see what I mean.
any corrections to render elements make compositing more difficult. if things keep changing - frame buffer is not a place to make any decisions.
also depending on render time denoising is most likely not a good idea.
Ah no, you misunderstand. The CC thing only mattered if I opted to use object selects. So I was going to render the undenoised version, try to improve that afterwards, then overlay the object selects.
The CCs were all fixed and approved, so there was no issue with that. The changes were only to animation aspects, - timings and physics stuff.
Actually, reassessing it now, the blurring I was considering wouldn’t have worked well, as it would have blurred the objects’ alphas much like the denoiser would, so it seems there’s no really ideal way to do this, which is infuriating.
Anyway it all became annoying to troubleshoot, so as I say I went for overall longer render time, smaller frames, upscaled and and no denoiser. That works adequately.
object select saving without vfb corrections should never be an issue. Raw VRay output, without anything from vfb should always be the first thing you save and use in post.
and denoising just alpha in post should be straightforward.
Ok, so the standalone denoiser does the job on the alpha, although what should be pure white is made grey in the process, with a white halo, which seems wrong but at least progress. Thanks for your contribution Marcin, as otherwise I’d likely not have tried it
Luckily for me the render’s done for this one and I have to move to other jobs, but I’ll revisit this when I have a moment to see if I can fix the workflow.
if there is nothing else on hand I would just use the compositor in Blender - this would be OIDN denoiser with albedo and bump normals input for added accuracy. it does not have any frame blending mechanism like native VRay denoiser but I was using similar setup in Fusion without issues.
I have close to zero experience with the standalone denoiser tool. are you denoising exr/vrayimg or some integer format?
I rarely look at the denoiser either, so it’s only during this exchange that it actually got an airing. I’m outputting .exr as that’s the only way to get the vfb image looking exactly the same on import to AE.
I do then tend to export PNGs for size, so if Blender can denoise those then all good.
The example frame I tested earlier, with the strange grey alpha result, was from a vrimg.
I’ll have a look at that compositor, maybe it can be a bridge, so thanks for the tip.
If it can work for animations without any bad issues then it should come in useful for one thing that’s just reared its head again.
That took me 72 of rendering last summer and I don’t want to go through that again, lol, so it’s small/noisy/denoise/upscale if I can get away with it from now on.
exrs can be smaller than pngs. half float, piz or even dwab compression. not sure about AE but exr playback is faster than png in general.
I don’t think there is any colour correction in VRay’s frame buffer that can not be done in AE. even filmic tone mapping can be 1:1 recreated with csp luts.
Yeah I ‘could’ do all the CC in AE but the thing is that most of the time I’m in Max and screenshotting the vfb for feedback, while they play the ‘left a bit, right a bit’ game, so to have to continually go between Max and AE comps is
a total pain tbh and wastes too much time. If it’s nailed in the vfb then the end render is simpler and the comp less bulky, especially as I need to deliver an AE comp that they can output for various formats, so I thought to make it as basic as possible.
With exrs I did have a problem with the last sequence that had a bunch of elements, resulting in AE having the memory allocation error, which I never found a way around and forced me to quickly reconfigure the render output in any case.
I thought to use Fusion but of course after rabbit-holing that (no prev exp) I discovered crytomattes are behind the paywall for it…so sad 
the answer to big exr is multipart exr - PS supports them just fine so I guess AE should too. in that case only one part (so for example RGBA or zdepth) is loaded and not the whole thing. there might be issues with cryptomattes though - I think Blender’s cryptomatte does not work with multipart exrs (it was definitely the case with older Fusion with Reactor cryptomatte plugin).
I always setup a comp in Nuke before showing anything to the clients because I don’t want to “chase the Polaroid” with trying to match VFB corrections later.
if cryptomatte is the only thing you are missing from Studio version I’m relatively confident cryptomatte Reactor plugin works with free version.
you can transfer corrections 1:1 from vfb with a hald image (the one used for creating luts) but it is not straightforward (and it would be a .csp lut to cover any tone mapping and be usable in PS/affinity) .
what might be helpful is just a bunch of decent luts in the ocio config (bit clunky) to scroll through when needed.
Yeah I looked at the Reactor plugin but it no longer functions, sadly…can’t remember what they did to disable its use but it wasn’t an option.
I am quite busy at the moment, so I’ll assess funds and maybe get a seat of Nuke and make the switch from AE…or maybe take a holiday, which I probably need more
just get Resolve Studio. hard to beat the value. but it will also make you a bit sad about all the time wasted doing 3D with AE for years..
AE was pretty much the only option available decades ago, so I just fell into it, due to the people I did stuff for all doing broadcast tv and using it exclusively.
Very little 3D stuff - 1000s of roto shots and other video trickery mainly, then when we started doing a lot of 3D it was already baked in as the choice unfortunately.
Anyway, I can look into Studio, though any reason why it’s prefered over Nuke, or just the price?
only experience with Nuke I have is Natron heh..
I can see indie licence for not-a-gazylion-credits now. might be a better deal.