Cylindrical (ortho) camera for panorama camera match ?

Anyone ever tried matching a building on to a stitched together panorama image ?
if so is there a clever way of doing this ?

(short of matching the building onto each part of the panorama image before stitching it)

I was thinking about using the cylindrical vray camera, but have not yet seen the light ..

so any help/suggestions would be very welcome =)