Quantized-mesh output support

[ahuarte47]

Provides some new features:

• New quantized-mesh format ouput (-f Mesh), optionally you could add VertexNormals with -N option.
• New metadata file output.
• New cesium-friendly option to automatically create missing root tiles (level 0).
• Consider nodata values when creating tiles.
• Catch GDAL 'Integer overflow' error when creating tiles of low levels.
• Refactoring of code to easily write new output formats.

Details:
https://www.linkedin.com/pulse/fast-cesium-terrain-rendering-new-quantized-mesh-output-alvaro-huarte/

Update 2018/08/08:
Available Cesium Terrain Builder Docker thanks to @BWibo

[bjpirt]

I've just built this branch and generated quantized mesh tiles for some elevation data I've been working with and can confirm all seems to work well. Would be good to get it merged in

Thanks to @ahuarte47 for a quick fix and @markerikson for help in getting it built

[ahuarte47]

Thanks @bjpirt , my apologies to everybody, I wasn't able to reopen the original PR #55 to keep saved all comments already written

[LHolst]

i successfully build a terrain as Meshformat with Normals plus Cesiumfriendly option. Its working great @ahuarte47 , to use the normal information in cesium i had to set the "extensions": ["octvertexnormals"] in the layer.json

maybe that could be done by ctb-tile when the layer.json is generated by setting the -l option

[ahuarte47]

Thanks @LHolst for the feedback! I will add the "extensions" entry.

[LHolst]

i have noticed some inconsistency between tiles normals at the edges.

it could be an issue in cesium...

[LHolst]

The extensions entry works. Thanks @ahuarte47

When using the cesiumFriendly option, the missing toplevel 0.terrain file is copied, but not reflected in the layer.json by the first entry in available. Therefore cesium only loads half of the globe, because the startX and endX of z=0 misses the copied file.

Lastly, it would be nice when using 'Normal' and 'CesiumFriendly' together, the normals of the copied missing 0.terrain were flipped.

see wrong normals on the left, i modified the layer.json by setting startX=0

[ahuarte47]

Thanks @LHolst for your feedback and advices! About flipping normals in that copied missing root tile, it needs some more of work

[LHolst]

i really appreciate your help, root tiles working now 👍

[ahuarte47]

Hi @LHolst, finally I have changed how CTB creates the missing root tile, now it is created from a temporary empty elevation that spatially overlaps the missing tile. I hope it works

[LHolst]

I am getting an bounds error Exception when using -l layer only option

ctb-tile -C -N -o . -f Mesh -l  /data/dtm_converted.tif                                                                                                                                                                                                     
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
terminate called after throwing an instance of 'ctb::CTBException'
  what():  The value is greater than the maximum X value
Aborted (core dumped)

gdalinfo /data/dtm_converted.tif
Driver: GTiff/GeoTIFF
Files: /data/dtm_converted.tif
       /data/dtm_converted.tif.aux.xml
Size is 8750, 6000
Coordinate System is:
PROJCS["CH1903+ / LV95",
    GEOGCS["CH1903+",
        DATUM["CH1903+",
            SPHEROID["Bessel 1841",6377397.155,299.1528128,
                AUTHORITY["EPSG","7004"]],
            TOWGS84[674.374,15.056,405.346,0,0,0,0],
            AUTHORITY["EPSG","6150"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4150"]],
    PROJECTION["Hotine_Oblique_Mercator_Azimuth_Center"],
    PARAMETER["latitude_of_center",46.95240555555556],
    PARAMETER["longitude_of_center",7.439583333333333],
    PARAMETER["azimuth",90],
    PARAMETER["rectified_grid_angle",90],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",2600000],
    PARAMETER["false_northing",1200000],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH],
    AUTHORITY["EPSG","2056"]]
Origin = (2685625.000000000000000,1287000.000000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  ( 2685625.000, 1287000.000) (  8d34'51.96"E, 47d43'45.82"N)
Lower Left  ( 2685625.000, 1284000.000) (  8d34'49.86"E, 47d42' 8.69"N)
Upper Right ( 2690000.000, 1287000.000) (  8d38'21.90"E, 47d43'43.70"N)
Lower Right ( 2690000.000, 1284000.000) (  8d38'19.69"E, 47d42' 6.58"N)
Center      ( 2687812.500, 1285500.000) (  8d36'35.85"E, 47d42'56.21"N)
Band 1 Block=8750x1 Type=Float32, ColorInterp=Gray
  Min=399.590 Max=679.660 
  Minimum=399.590, Maximum=679.660, Mean=547.999, StdDev=62.302
  NoData Value=-9999
  Metadata:
    STATISTICS_MAXIMUM=679,65997314453
    STATISTICS_MEAN=547,99864003316
    STATISTICS_MINIMUM=399,58999633789
    STATISTICS_STDDEV=62,30161802239

without the layers option tiles get generated
is there a way to generate the layer.json along with the tiles?

[LHolst]

when i use my old layer.json, the normals are improved.

[ahuarte47]

Hi @LHolst, I fixed a bug, I hope it works now

[LHolst]

ahuarte47@4bb78ba generates layer.json correctly, thanks @ahuarte47 👍 🎉

also #15 (comment) did the trick to run gdal 2.2

[geoportallux]

for anyone who wants to test this PR using docker:

FROM debian
RUN apt-get update && apt-get -y install cmake build-essential gdal-bin git libgdal-dev
RUN cd /root && git clone https://github.com/ahuarte47/cesium-terrain-builder.git && cd cesium-terrain-builder && git checkout master-quantized-mesh && mkdir build && cd build && cmake .. && make install . && ldconfig

works great for us. many thanks

[winsento]

Generate terrain from dem.tif EPSG:4326

ctb-tile  -f Mesh -s 24 -e 0 -C -c 4 -l --output-dir /terrain /dem.tif

But when set CesiumTerrainProvider in Cesium - result is empty. Example here:
https://jsfiddle.net/winsent/7kp7y9k2/6/

[LHolst]

@winsento you need to run the command without -l to generate the tiles, and with to generate the layer.json

[BWibo]

Today I created an Alpine Linux based Docker image: tum-gis/cesium-terrain-builder-docker#5

• Alpine v3.10
• Gdal: v2.4.2

Get it here:
docker pull tumgis/ctb-quantized-mesh:alpine

This has not been properly tested jet. I'm happy for any feedback.

[bibuzz]

@ahuarte47

I have used your Quantized-mesh builder to created tiles of the world tiff file. But the world tiff file contains bathymetery data and on viewing the tiles on Cesium using new Cesium.CesiumTerrainProvider , the land is below the oceans, like in a trench. ( the land 3d works well )
I used the command
ctb-tile -s 1 -e 0 /backup/data/tilesets/terrain/world /backup/global_gebco.tif

[

]

I asked in cesium forum and they mentioned that nothing can be done from the cesium end and it has to be changed at how the data is generated.

CesiumGS/cesium#9642

I saw in your limitations and To-Do that you consider all tiles as land. Is that why the ocean is above? How is the positive altitude coming from?

[adam-ce]

are you still working on this?
I was trying to get it to work, but i'm getting segfaults. I was able to track it down to out of bounds access. The following (new) assert is triggered:

index 9bab8ff..390cca9 100644
--- a/src/HeightFieldChunker.hpp
+++ b/src/HeightFieldChunker.hpp
@@ -1,6 +1,8 @@
 #ifndef HEIGHTFIELDCHUNKER_HPP
 #define HEIGHTFIELDCHUNKER_HPP
 
+#include <cassert>
+
 /*******************************************************************************
  * Copyright 2018 GeoData <[email protected]>
  *
@@ -226,6 +228,7 @@ public:
 
   /// Return the array-index of specified coordinate, row order by default.
   virtual int indexOfGridCoordinate(int x, int y) const {
+    assert((y * m_size) + x < m_size * m_size);
     return (y * m_size) + x;
   }
   /// Return the height of specified coordinate.

m_size is 256, y is 256, but the array the code is reading and writing is only 256x256. so clearly, the addresses are out of bounds.
I can imagine, that this is something simple (increasing the allocated memory and change the addressing?), but without understanding the algorithm and what's going on, i can't be confident in how to fix it. Any hints?

edit: I was able to fix it. the branch in my profile appears to work for me. In addition, it contains some code modernisation, and I ran it through address sanitizer. i'm still having trouble with rendering as tiles disappear when using a perspective camera (but not when using an orthographic one). i'm looking into it..

edit2: fixed the disappearing tiles. the quantised mesh branch was definitely broken here, but probably previous versions of cesium didn't care. fixed in my branch as well. still needs a bit of cleaning though..

[v8gx]

i have noticed some inconsistency between tiles normals at the edges.

it could be an issue in cesium...

Hi! Was the normal seam issue ever solved? I am processing terrain using the tum-gis docker image and getting seams like this:

[v8gx]

I have looked into the normal issue a bit and at level 0 already, the normals between the 2 "half-globes" already fail:

[BWibo]

@ahuarte47 @ShawayL I just reflected your changes in the https://github.com/tum-gis/cesium-terrain-builder-docker Docker image.

[SHIHUAMarryMe]

wow! great!!!!!!!!!!!

[gfrontera]

I have looked into the normal issue a bit and at level 0 already, the normals between the 2 "half-globes" already fail:

I believe that the problems with normals at the perimeter of tiles are intrinsic to how the tiles are built. The normal for each vertex is computed as a weighted average of the normals of all the triangles containing such vertex. That can be computed for all vertices in the tile except for those vertices in the perimeter, as you can only access some of the triangles that contain them.

One solution would involve generating the tiles without the per-vertex normal extension, and then on a second stage (when the vertices for all tiles have been computed), compute and append the normals for each tile. Does this sound right, @ahuarte47?

[ahuarte47]

One solution would involve generating the tiles without the per-vertex normal extension, and then on a second stage (when the vertices for all tiles have been computed), compute and append the normals for each tile.

Maybe the issue is related with what neighbour vertices are used to calculate the normals. I think that row of wrong normals can be located in 0 longitude. Normals of triangles located in the left and in the right side of that X/lon were calculated with a different subset of vertices. It would need work to validate this theory and a fix to resolve the issue.

Unfortunatelly I can not maintain this effort, any help is welcome.

[gfrontera]

Is this issue what is preventing this MR to be merged?

[mhaberler]

the fix by #64 (comment) is really important - without it --profile mercator crashes with the same symptoms described above

same goes for https://github.com/tum-gis/cesium-terrain-builder-docker

is this project still maintained?

[ahuarte47]

Hi @mhaberler I am so sorry, I am not maintaining this branch, maybe other fork is beeing improved.

[BWibo]

Hey @mhaberler,
I am still maintaining tum-gis/cesium-terrain-builder-docker, but only the Docker part. I can't do anything about CTB itself.
The image is built from https://github.com/ahuarte47/cesium-terrain-builder/tree/master-quantized-mesh.
So any commits to that branch end up in the image.

[kakadiyaAnkit]

why do generated layer json has bounds
link
of "bounds": [ 0.00, -90.00, 180.00, 90.00 ]"
and not "bounds": [ -180.00, -90.00, 180.00, 90.00 ]