Your mesh is not connected. This is the issue of how it was exported.
You have two options:
Fix the export
Repair the mesh afterwards
I was able to fix your model by first merging all the relevant mesh parts together using Mesh → Conversion → Connect Objects and then by welding the vertices using the Connect modifier (see the attached image). I also had to delete the phong tags, which were incorrect for some reason. The displacement size of 2cm was also a bit too high, I think, so I lowered it to 0.5cm.
I’m afraid I cannot help here. This is a matter of correct setup in your exporter and you mention Rhino or Archicad. I have zero experience in any of those two.
@houska: I tried to reproduce your solution, but I noticed it creates the same problem of strangely rounded geometry that I receive with using Optimize. Can you tell what is the problem here? Thanks a lot!
I noticed that disabling the highlighted conversion option during import and using Optimize on the complete geometry also resolves the issue completely. Thanks again for your help.