Xrefs should not have any impact (maybe a little, the xref update time if you render in DR or remotely) on rendertime, since they’re “merged” into the scene before render. Xref related bugs are different, but annoying nevertheless. 
Having very bright and small geometry, such as lightbulbs in the scene can cause splotches. The GI engine will find these hotspots, and try to take their intensity into account when calculating the solution, but thanks to how gi in VRay currently works “object” lights like these tend to need very, very large amount of samples to render smooth. This is a reason why you use direct lights for point like lightsources.
However, an ies light has no physical volume, it’s not present in the scene in that way, so people tend to put light emitting geometry where the light bulb should be to have convincing reflections, or to see the actual light source itself. But since you don’t need these lights to actually illuminate the scene, it’s best to keep them out of the GI solution.
Splotches go wild when “GI caustics” somes into the picture, so if you’ve got refractive shaders in there, and a small geometric lightsource contributing energy to the gi solution, splotches will start to appear. Just think of an average light fixture, chromes and glasses all over, multiple specular transmittance, put in a small high energy object, turn on gi caustics, and voila, you’ve got yourself a disco ball.
Hope it makes any sense, had to type, again, fast, hell just broke loose here in our office.
And yes, it’s such a pity that the max implementation of ies lights is such a bloody bruteforce approach. Quite far from “fast” I must say. Let’s hope Vlado adds ies support into the MAX build of Vray too.
Oh, and don’t worry about the Physical cam, has very little to do with your rendertime if you set the exposure correctly. Just don’t under or overexpose it. Under will bring undersampled areas, so splotches, noise, etc, and overexposed will mean longer rendertimes (oversampled areas).
Best regards,
A.