Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add optional pre triangulation for polygons when creating MLT #383

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
2 changes: 2 additions & 0 deletions java/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,13 @@ dependencies {
implementation 'org.apache.orc:orc-core:1.8.1'
implementation 'com.github.davidmoten:hilbert-curve:0.2.3'
implementation 'com.carrotsearch:hppc:0.10.0'
implementation "io.github.earcut4j:earcut4j:2.2.2"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already depend on JTS, which has a few built-in polygon triangulation utilities: https://github.com/locationtech/jts/tree/master/modules/core/src/main/java/org/locationtech/jts/triangulate/polygon - and I think PolygonEarClipper is similar to earcut. Do you have any quality/performance comparison of those vs. earcut4j? If earcut4j performs better then adding a new dependency seems fine, but if not I'd default to the dependencies we already have.

testImplementation 'org.junit.jupiter:junit-jupiter-api:5.10.3'
testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.10.3'
testImplementation 'org.junit.jupiter:junit-jupiter-params:5.10.3'
testImplementation 'org.openjdk.jmh:jmh-core:1.37 '
testImplementation 'org.openjdk.jmh:jmh-generator-annprocess:1.37'
testImplementation "org.mockito:mockito-core:3.+"
}

test {
Expand Down
5 changes: 4 additions & 1 deletion java/src/jmh/java/com/mlt/BenchmarkUtils.java
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,10 @@ public static void encodeTile(
.collect(Collectors.toMap(l -> l, l -> optimization));
var encodedMltTile =
MltConverter.convertMvt(
encodedMvtTile.getRight(), new ConversionConfig(true, true, optimizations), metadata);
encodedMvtTile.getRight(),
new ConversionConfig(true, true, optimizations),
metadata,
false);
encodedMltTiles.put(z, encodedMltTile);
}

Expand Down
2 changes: 1 addition & 1 deletion java/src/main/java/com/mlt/cli/Encode.java
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ public static void main(String[] args) {
var conversionConfig =
new ConversionConfig(includeIds, useAdvancedEncodingSchemes, optimizations);
Timer timer = new Timer();
var mlTile = MltConverter.convertMvt(decodedMvTile, conversionConfig, tileMetadata);
var mlTile = MltConverter.convertMvt(decodedMvTile, conversionConfig, tileMetadata, false);
if (willTime) timer.stop("encoding");
if (willOutput) {
Path outputPath = null;
Expand Down
16 changes: 12 additions & 4 deletions java/src/main/java/com/mlt/converter/MltConverter.java
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,8 @@ public static MltTilesetMetadata.TileSetMetadata createTilesetMetadata(
public static byte[] convertMvt(
MapboxVectorTile mvt,
ConversionConfig config,
MltTilesetMetadata.TileSetMetadata tilesetMetadata)
MltTilesetMetadata.TileSetMetadata tilesetMetadata,
boolean triangulatePolygons)
throws IOException {
var physicalLevelTechnique =
config.useAdvancedEncodingSchemes()
Expand All @@ -192,7 +193,12 @@ public static byte[] convertMvt(

var result =
sortFeaturesAndEncodeGeometryColumn(
config, featureTableOptimizations, mvtFeatures, mvtFeatures, physicalLevelTechnique);
config,
featureTableOptimizations,
mvtFeatures,
mvtFeatures,
physicalLevelTechnique,
triangulatePolygons);
var sortedFeatures = result.getLeft();
var encodedGeometryColumn = result.getRight();
var encodedGeometryFieldMetadata =
Expand Down Expand Up @@ -266,7 +272,8 @@ private static byte[] encodePropertyColumns(
FeatureTableOptimizations featureTableOptimizations,
List<Feature> sortedFeatures,
List<Feature> mvtFeatures,
PhysicalLevelTechnique physicalLevelTechnique) {
PhysicalLevelTechnique physicalLevelTechnique,
Boolean triangulatePolygons) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use a primitive boolean here

/*
* Following simple strategy is currently used for ordering the features when sorting is enabled:
* - if id column is present and ids should not be reassigned -> sort id column
Expand Down Expand Up @@ -294,7 +301,8 @@ private static byte[] encodePropertyColumns(
new GeometryEncoder.SortSettings(
isColumnSortable && featureTableOptimizations.allowIdRegeneration(), ids);
var encodedGeometryColumn =
GeometryEncoder.encodeGeometryColumn(geometries, physicalLevelTechnique, sortSettings);
GeometryEncoder.encodeGeometryColumn(
geometries, physicalLevelTechnique, sortSettings, triangulatePolygons);

if (encodedGeometryColumn.geometryColumnSorted()) {
sortedFeatures =
Expand Down
99 changes: 78 additions & 21 deletions java/src/main/java/com/mlt/converter/encodings/GeometryEncoder.java
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

import com.mlt.converter.CollectionUtils;
import com.mlt.converter.geometry.*;
import com.mlt.converter.triangulation.VectorTileConverter;
import com.mlt.metadata.stream.*;
import java.util.*;
import java.util.function.Function;
Expand Down Expand Up @@ -33,12 +34,15 @@ private GeometryEncoder() {}
public static EncodedGeometryColumn encodeGeometryColumn(
List<Geometry> geometries,
PhysicalLevelTechnique physicalLevelTechnique,
SortSettings sortSettings) {
SortSettings sortSettings,
boolean triangulatePolygons) {
Copy link
Collaborator

@msbarry msbarry Nov 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we make it possible this take the triangulated polygons as an input to this function instead of embedding the triangulation inside of it?

On planetiler I'm considering some options for fixing invalid polygons that produce triangulated geometries as an intermediate step and I'd like for it to at least be possible to pass those triangulated polygons into this routine without forcing them to be computed again.

var geometryTypes = new ArrayList<Integer>();
var numGeometries = new ArrayList<Integer>();
var numParts = new ArrayList<Integer>();
var numRings = new ArrayList<Integer>();
var vertexBuffer = new ArrayList<Vertex>();
var numTrianglesPerPolygon = new ArrayList<Integer>();
var indexBuffer = new ArrayList<Integer>();
var containsPolygon =
geometries.stream()
.anyMatch(
Expand Down Expand Up @@ -74,6 +78,11 @@ public static EncodedGeometryColumn encodeGeometryColumn(
var polygon = (Polygon) geometry;
var vertices = flatPolygon(polygon, numParts, numRings);
vertexBuffer.addAll(vertices);
if (triangulatePolygons) {
var triangulatedPolygon = new VectorTileConverter(polygon);
indexBuffer.addAll(triangulatedPolygon.getIndexBuffer());
numTrianglesPerPolygon.addAll(triangulatedPolygon.getNumTrianglesPerPolygon());
}
break;
}
case Geometry.TYPENAME_MULTILINESTRING:
Expand Down Expand Up @@ -102,6 +111,11 @@ public static EncodedGeometryColumn encodeGeometryColumn(
var vertices = flatPolygon(polygon, numParts, numRings);
vertexBuffer.addAll(vertices);
}
if (triangulatePolygons) {
var triangulatedPolygon = new VectorTileConverter(multiPolygon);
indexBuffer.addAll(triangulatedPolygon.getIndexBuffer());
numTrianglesPerPolygon.addAll(triangulatedPolygon.getNumTrianglesPerPolygon());
}
break;
}
case Geometry.TYPENAME_MULTIPOINT:
Expand Down Expand Up @@ -161,6 +175,21 @@ public static EncodedGeometryColumn encodeGeometryColumn(
Arrays.stream(zigZagDeltaVertexBuffer).boxed().collect(Collectors.toList()),
physicalLevelTechnique,
false);

var encodedIndexBuffer =
IntegerEncoder.encodeIntStream(
indexBuffer,
physicalLevelTechnique,
false,
PhysicalStreamType.OFFSET,
new LogicalStreamType(OffsetType.INDEX));
var encodedNumTrianglesBuffer =
IntegerEncoder.encodeIntStream(
numTrianglesPerPolygon,
physicalLevelTechnique,
false,
PhysicalStreamType.OFFSET,
new LogicalStreamType(OffsetType.INDEX));
// TODO: should we do a potential recursive encoding again
var encodedVertexDictionary =
IntegerEncoder.encodeInt(
Expand Down Expand Up @@ -251,11 +280,16 @@ public static EncodedGeometryColumn encodeGeometryColumn(
Arrays.stream(zigZagDeltaVertexBuffer).boxed().collect(Collectors.toList()),
physicalLevelTechnique);

return new EncodedGeometryColumn(
numStreams + 1,
ArrayUtils.addAll(encodedTopologyStreams, encodedVertexBufferStream),
maxVertexValue,
geometryColumnSorted);
var encodedGeometryColumn =
new EncodedGeometryColumn(
numStreams + 1,
ArrayUtils.addAll(encodedTopologyStreams, encodedVertexBufferStream),
maxVertexValue,
geometryColumnSorted);
return triangulatePolygons
? buildTriangulatedEncodedGeometryColumn(
encodedGeometryColumn, encodedIndexBuffer, encodedNumTrianglesBuffer)
: encodedGeometryColumn;
} else if (dictionaryEncodedSize < plainVertexBufferSize
&& dictionaryEncodedSize <= mortonDictionaryEncodedSize) {
var encodedVertexOffsetStream =
Expand All @@ -269,13 +303,18 @@ public static EncodedGeometryColumn encodeGeometryColumn(
encodeVertexBuffer(
Arrays.stream(zigZagDeltaVertexDictionary).boxed().collect(Collectors.toList()),
physicalLevelTechnique);

return new EncodedGeometryColumn(
numStreams + 2,
CollectionUtils.concatByteArrays(
encodedTopologyStreams, encodedVertexOffsetStream, encodedVertexDictionaryStream),
maxVertexValue,
false);
var encodedGeometryColumn =
new EncodedGeometryColumn(
numStreams + 2,
CollectionUtils.concatByteArrays(
encodedTopologyStreams, encodedVertexOffsetStream, encodedVertexDictionaryStream),
maxVertexValue,
false);

return triangulatePolygons
? buildTriangulatedEncodedGeometryColumn(
encodedGeometryColumn, encodedIndexBuffer, encodedNumTrianglesBuffer)
: encodedGeometryColumn;
} else {
var encodedMortonVertexOffsetStream =
IntegerEncoder.encodeIntStream(
Expand All @@ -292,14 +331,20 @@ public static EncodedGeometryColumn encodeGeometryColumn(
zOrderCurve.coordinateShift(),
physicalLevelTechnique);

return new EncodedGeometryColumn(
numStreams + 2,
CollectionUtils.concatByteArrays(
encodedTopologyStreams,
encodedMortonVertexOffsetStream,
encodedMortonEncodedVertexDictionaryStream),
maxVertexValue,
geometryColumnSorted);
var encodedGeometryColumn =
new EncodedGeometryColumn(
numStreams + 2,
CollectionUtils.concatByteArrays(
encodedTopologyStreams,
encodedMortonVertexOffsetStream,
encodedMortonEncodedVertexDictionaryStream),
maxVertexValue,
geometryColumnSorted);

return triangulatePolygons
? buildTriangulatedEncodedGeometryColumn(
encodedGeometryColumn, encodedIndexBuffer, encodedNumTrianglesBuffer)
: encodedGeometryColumn;
}
}

Expand Down Expand Up @@ -426,4 +471,16 @@ private static byte[] encodeVertexBuffer(

return ArrayUtils.addAll(encodedMetadata, encodedValues);
}

private static EncodedGeometryColumn buildTriangulatedEncodedGeometryColumn(
EncodedGeometryColumn encodedGeometryColumn,
byte[] encodedIndexBuffer,
byte[] encodedNumTrianglesPerPolygon) {
return new EncodedGeometryColumn(
encodedGeometryColumn.numStreams + 2,
CollectionUtils.concatByteArrays(
encodedGeometryColumn.encodedValues, encodedIndexBuffer, encodedNumTrianglesPerPolygon),
encodedGeometryColumn.maxVertexValue,
encodedGeometryColumn.geometryColumnSorted);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
package com.mlt.converter.triangulation;

import java.util.ArrayList;
import java.util.List;

public class TriangulatedPolygon {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I'm reading build.gradle correctly, it looks like we're compiling against java 17. If that's right could this class just be record TriangulatedPolygon(List<Integer> indexBuffer, int numTrianglesPerPolygon) {} ?

private final int numTrianglesPerPolygon;

private final ArrayList<Integer> indexBuffer;

TriangulatedPolygon(ArrayList<Integer> indexBuffer, int numTriangles) {
this.numTrianglesPerPolygon = numTriangles;
this.indexBuffer = indexBuffer;
}

public Integer getNumTrianglesPerPolygon() {
return numTrianglesPerPolygon;
}

public List<Integer> getIndexBuffer() {
return indexBuffer;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
package com.mlt.converter.triangulation;

import earcut4j.Earcut;
import java.util.ArrayList;
import java.util.List;
import org.apache.commons.lang3.ArrayUtils;
import org.locationtech.jts.geom.Coordinate;
import org.locationtech.jts.geom.MultiPolygon;
import org.locationtech.jts.geom.Polygon;

public class TriangulationUtils {
private TriangulationUtils() {}

public static TriangulatedPolygon triangulatePolygon(Polygon polygon) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A Polygon can have holes, and a MultiPolygon just contains many Polygons (which may each have holes). Can we change this to handle those cases and use the poly.getExteriorRing() poly.getNumInteriorRing() poly.getInteriorRingN(int n) helpers? Let's add some tests for polygons with holes as well.

var convertedCoordinates = convertCoordinates(polygon.getCoordinates());

List<Integer> triangles = Earcut.earcut(convertedCoordinates, null, 2);

ArrayList<Integer> indexBuffer = new ArrayList<>(triangles);
var numTriangles = triangles.size() / 3;

return new TriangulatedPolygon(indexBuffer, numTriangles);
}

public static TriangulatedPolygon triangulatePolygonWithHoles(MultiPolygon multiPolygon) {
var holeIndex = 0;

ArrayList<Double> multiPolygonCoordinates = new ArrayList<>();
ArrayList<Integer> holeIndices = new ArrayList<>();

for (int i = 0; i < multiPolygon.getNumGeometries(); i++) {
// assertion: first polygon defines the outer linear ring and the other polygons define its
// holes!
if (i == 0) {
holeIndex = multiPolygon.getGeometryN(i).getCoordinates().length;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each call to getCoordinates may be transforming a packed double[] CoordinateSequence to a Coordinate[] array. Is it possible to just extract the CoordinateSequence once and use double getX(int idx), double getY(int idx) and double getZ(int idx) (only when coordinateSequence.getDimension() > 2)

holeIndices.add(holeIndex);
} else if (i != multiPolygon.getNumGeometries() - 1) {
holeIndex += multiPolygon.getGeometryN(i).getCoordinates().length;
holeIndices.add(holeIndex);
}

var coordinates = multiPolygon.getGeometryN(i).getCoordinates();
for (Coordinate coordinate : coordinates) {
multiPolygonCoordinates.add(coordinate.x);
multiPolygonCoordinates.add(coordinate.y);
if (!Double.isNaN(coordinate.z)) {
multiPolygonCoordinates.add(coordinate.z);
}
}
}

var doubleArray = ArrayUtils.toPrimitive(multiPolygonCoordinates.toArray(new Double[0]));
List<Integer> triangleVertices =
Earcut.earcut(doubleArray, holeIndices.stream().mapToInt(Integer::intValue).toArray(), 2);

ArrayList<Integer> indexBuffer = new ArrayList<>(triangleVertices);
var numTriangles = triangleVertices.size() / 3;

return new TriangulatedPolygon(indexBuffer, numTriangles);
}

private static double[] convertCoordinates(Coordinate[] coordinates) {
ArrayList<Double> convertedCoordinates = new ArrayList<>();

for (Coordinate coordinate : coordinates) {
convertedCoordinates.add(coordinate.x);
convertedCoordinates.add(coordinate.y);
if (!Double.isNaN(coordinate.z)) {
convertedCoordinates.add(coordinate.z);
}
}
Double[] array = convertedCoordinates.toArray(new Double[0]);

return ArrayUtils.toPrimitive(array);
}
}
Loading