Have you tried playing around with the “luminance center” setting in the displacement? Are you using the modifier or piping a map into the material? In the material there is a min- and max-level setting. Try setting the min level to a negative value.
It depends on your displacement map. Can you show it?
Basically it’s very simple, black colour don’t displace at all and white colour displaces by amount set in max level. If your map is constructed in a way that mid grey (127, 127, 127) shouldn’t displace and blacks and whites should displace inwards and outwards respectively, then set min and max levels to oppositely equal amounts (-1 and 1, -5 and 5, and so on) in displacement settings.
For displacement it doesn’t matter if it’s RGB or greyscale bitmap. You won’t get good results from those textures anyway. Displacement requires specially created map and it’s practically impossible to get good displacement map from photographs, unless you taking them with some kind of light field camera :]
Or you could try to use sth like Pix plant for creating displacement maps. But anyway those would not be as good as sculpted geometry converted to heighmap.
And other thing. If it is converted phototexture they you should always combine displacement (for structure and large detais) and normalmap in bumpslot for fine details.