Cesium Terrain generated using DSM is way higher than the base map

[overide]

Is there any way I can clamp the terrain to the base-map's height? Is the problem with the vertical datum for the data. I tried changing ellipsoid's radius but I learnt that the height is relative to the ellipsoid's base height.

Playing around with terrainExaggeration, usually setting it 0.1 brings it closer to the base layer, but I'm expecting that it would cause issues while measuring heights. Also, it appears extremely flat [undesirable]

Is it due to TIFF being in different CRS, or vertical datum being off? Or is it something that I can correct using cesium options for viewer/terrain provider or something which can be corrected while generating mesh using cesium-terrain-builder?

Link to GIS stack exchange question - https://gis.stackexchange.com/questions/376270/cesium-terrain-generated-using-dsm-is-way-higher-than-the-base-map