include/exclude in lights render slowdown

[frodopytlicek]

Hi,
I know that including or excluding object from lights can slow down render, but 2h render can become 6 hour render only because of this? Is there some way how to fix this and why its so render time heavy?