Maybe I can help out here, I gave it a try with our newest Version under Vectorworks and I was able to invert Spacemouse movement there using the 3dconnexion application.
For me it looks like this, I click on the Enscape renderer window and get the Enscape3D_Vectorworks application focused in the 3dconnexion application:
For 4.0, there was a bugfix for our Spacemouse implementation under Vectorworks, so that might be where different behavior is coming from. We are now correctly using the Enscape-under-Vectorworks config file that is shipped as the 3DxWare driver (so in case there are some issues that persist, it could also help updating the 3DxWare driver to the newest version, available here: https://3dconnexion.com/drivers/).
Does this help with your issue? Or is there a different inversion checkbox that you are referring to that I maybe missed?