To reproduce the crash you need Windows 10 Pro and 3Ds Max 2016 + V-Ray 3.20.02
Open 3Ds Max, load a teapot and a vray light and hit render.
It might take a long while with such a simple scene, or open anything you created and made sure did not had any problems before. Observe how “system” process loads more and more memory each render.
After sometime 3Ds Max will crash or there will be a Windows pop-up suggesting to shutdown 3Ds Max because there is not enough memory.
If Max crashes Watch how the memory is still loaded in the “system” process.
Go to the “vrayspawner2016” process and restart it.
You will see the “system” process is now releasing memory until it hits 12MB.
I search a little and it is probably causes by a Windows 10 “feature” called “compression store”
Why Is Windows 10's "System" Process Using So Much RAM? | Lifehacker
I will try disabling it somehow and report back if there any improvements.
Right after that