Why should they? They still affect the way V-Ray performs adaptive sampling. This was the purpose for adding the “don’t affect colors” option - people wanted the color mapping to affect the sampling (i.e. the noise levels), but not to influence the final colors.
Best regards,
Vlado