Move basemap profile into openmaptiles org?

[msbarry]

Talking with some folks from OpenMapTiles, they are interested in moving the planetler-basemap module from planetiler into a separate repo under https://github.com/openmaptiles so that planetiler ends up being the tool schemas can be built on, and openmaptiles schema depends on that.

Some benefits of this move:

• OpenMapTiles team can bring the basemap/openmaptiles schema to full compatibility and start adopting that internally instead of the current openmaptiles-tools postgis stack for faster, lower-cost updates.
• OpenMapTiles team will keep the schema up-to-date with the latest improvements, so planetiler developers (me) don't need to spend time on PRs like Upgrade Basemap profile to openmaptiles 3.13.1 #216
• Planetiler licensing becomes more clear: It's just Apache 2.0, and all of the the BSD/CC-BY stuff is in a separate repo
• Versioning is cleaner: openmaptiles schema release 3.13.1 depends on planetiler 0.4.0, etc.. (instead of today where planetiler 0.4.0 bundles 3.13.1 internally)

I'd like to gauge the feeling from the community on potential drawbacks to make sure those concerns are addressed?

[msbarry]

One potential drawback is that planetiler should be able to generate a default schema out-of-the-box.

To address this, we would make openmaptiles schema repo a submodule dependency of planetiler so we could ensure it works in CI on planetiler changes, and build planetiler releases against it?

[klokan]

Yes, same as now. No change.

[msbarry]

Another concern is that it will be slower to iterate on planetiler API/capabilities then use those in the openmaptiles schema when they aren't together in a mono-repo. For example make improvements to line merging in planetiler, and generate openmaptiles schema to see the result.

One solution to this would be to wait until the API has solidified more before the move, but openmaptiles team seems OK to do the move sooner then deal with potential upgrades later.

[msbarry]

I'm not sure what the workflow would look like to make changes to planetiler and run openmaptiles immediately against that if it's a separate repo/submodule? Maybe we could have a second pom.xml in openmaptiles-planetiler repo that lets it be linked in as either a child module or a separate project, like the planetiler-examples module?

[klokan]

You do the changes localy the same as until now - only you have the changes in two different repos.

BTW we should set up CI on the OMT side too - to validate pull request and performance (the same way as it is done on omt/omt with omt-tools now).

[msbarry]

What would the new workflow look like to customize openmaptiles schema? Some people have planetiler forks with custom behavior, and we've added some custom stuff that differs from openmaptiles schema to the basemap profile that you can opt into with a command-line flag. Would we continue letting people add custom behavior with command-line flags to openmaptiles-planetiler schema? or would the only option be to fork that project?

[klokan]

What are the examples of such command line flags?

[msbarry]

Here are the openmaptiles-specific arguments currently in use:

• --building-merge-z13=false disables the z13 building merge, this cuts ~30% off of planet generation time, but some people (i.e. americana) do it based on style preference.
• --boundary-country-names=false disables adding adm0_l/adm0_r tags to country borders
• --transportation-z13-paths=true shows all paths at z13 - useful for outdoor maps, but less of a concern now that important paths are shown at z12/13.

And some knobs to tweak transportation name line merging so that we show more road names at lower zooms. These should be less necessary when I implement more robust line merging outlined in #54 (which will also help waterways, and transportation layer)

• --transportation-name-brunnel to exclude brunnel tag from transportation name layer, since it breaks up the lines, brings them below min length and they get excluded
• --transportation-name-size-for-shield to lower the min length limit for transportation_name segments and show shorter ones for styles might just render a shield
• --transportation-name-limit-merge limit merge so we don't combine different relations to help merge long highways (@boldtrn has been experimenting with this)
• --transportation-name-minor-refs to include refs on "minor road networks"

There are also some that configure default behavior like --min-feature-size and --simplify-tolerance but planetiler supports those automatically.

[klokan]

While turning the openmaptiles into submodule none of this needs to change immediately. No change right now.

Some ideas (open for discussion with the community):

In case people want to use planetiler as a general tiling software in future - these options should be perhaps more general (now they are very much bound directly to current OpenMapTiles schema maxzooms, terminology and building generalization implementation on z13, etc). This implies (perhaps?) one of a more universal approaches:

1. a single option for parameters of the schema - passing these to the schema implementation code
2. prefixing these - so it is clear they are omt specific
3. or a documented proper way how to customize the OpenMapTiles schema on your own for these popular(?) use-cases

In case people want Planetiler to be the high performance super optimized generator of OpenMapTiles (as it is now) just better - then we could:

• move the entire project into openmaptiles org on github
• set up proper governance rules (similar as done now on MapLibre)
• continue to develop it there as a community - without making a submodule.

This is very much welcome - and let's talk about how to make it happen. I would be glad to form the formal OpenMapTiles Steering Committee and start with regular monthly meetings - if there are people out there willing to invest their time into this direction.

[shermp]

Would it be worth waiting for #160 to mature, and then deprecate planetiler-basemap? OpenMapTiles support could then just be another configuration file that they (OpenMaptTiles) can manage.

[klokan]

This is the goal in the end. The submodule is intermediate solution - which can gradually migrate to configuration instead of pure submodule code.

There is a lot of advanced functionality which need to be configurable for a complex schema (or in Planetiler terminology profile) like OpenMapTiles... On stable OpenMapTiles you have the full power of SQL. In Planetiler this has been ported to Java code 1 to 1. There will very probably always need to be Java functions linked / called / referenced from the config - as some things cannot be even defined - but that's for discussion under #160.

[ZeLonewolf]

As the author of #160, I can say -- once that's merged, it's only going to have a fraction of the functionality that's needed to make OpenMapTiles boiled down to just a config file. There's a lot of complexity in OMT, including parts that I don't even fully understand myself. So the plan is to get out an initial configuration schema that works for basic cases and then iterate over time to add additional functionality.

[wipfli]

Sounds like a good idea to me to separate planetiler from openmaptiles.

[wipfli]

Openmaptiles is my favourite schema and I make my maps with it. Planetiler is great for making omt tiles. But it can also do other schemas and I think Planetiler should be independent of omt. Anyway, I really admire @msbarry for what he has created with planetiler and agree with @ZeLonewolf that Mike should decide under which umbrella Planetiler will live.

[ZeLonewolf]

I don't have a position on where the parts of planetiler should live, as @msbarry poured heart and soul into making it a reality and I think he is the one that has to be comfortable with where it ends up and where it's governed. So whatever the community decides I'll support.

I do agree that the planetiler-basemap module needs to be separated from the rest of the planetiler modules due to the differences in licensing, with the understanding that we'll probably deprecate it eventually. Just to level everyone, below is a diagram of the current setup, as it will be after #160 is merged.

What I think we would all like to get to is something like this:

This would be a huge boon for OpenMapTiles, as it would significantly simplify the process of schema development and schema extension. The OpenMapTiles project could focus building out schemas that make beautiful maps, and the infrastructure around it that ensures that tiles sizes are appropriate and so forth. The current setup of SQL and python is certainly daunting for new contributors and this would allow us to better focus on the core mission of schema development and alleviate the current contention that we have with forked variants of OpenMapTiles.

Planetiler is 100% configurable today, IF you're a Java programmer. It has a rich API that can be used to generate arbitrary schema outputs. If we are successful in this movement to configuring planetiler via config files rather than Java code, we might be able to fully deprecate the current planetiler-basemap entirely (although there's a TON of work between where we are now and that vision).

It still remains to be seen whether OpenMapTiles can be generated purely by configuration or if there are some things that are so complex that code is absolutely required, or if we need to move off of a declarative schema as I've started, and move to a more programmatic model such as using lua. I don't have any of those answers and we're going to discover them as we work through supporting the demands of the current OpenMapTiles schema, one core feature at a time.

Should planetiler-basemap move to OpenMapTiles, and if the community supports the direction I've been going with in #160, what would be useful I think is to set up a new project under the openmaptiles organization to hold the configuration-based schema for openmaptiles as well as the scripts needed to run it through planetiler to pop out an .mvt. That way there's a home for configured openmaptiles to grow over time and lets us work incrementally to catch up to the "main" openmaptiles repo. It also lets us track via github tickets what functionality is still needed to deprecate planetiler-basemap/openmaptiles-tools, as well as hold any custom Java code for special functionality should we determine that code solutions are necessary.

[klokan]

Thanks for a great post @ZeLonewolf - it explains very well the situation exactly as I have understood it.

Very happy to create a separate config repo at openmaptiles org - and granting you admin - so it could be used for prototyping the config based approach of the future - while having there config specific issues (and milestones) to this approach... any suggestion for a suitable name of such repo? Would it be "openmaptiles-profile" - in the terminology of planetiler/imposm3 where profile is the config for schema - right @msbarry?

It would indeed remove the block of to change a map well - if you must be a good cartographer (with OSM knowledge), and you must be a good Java (or Python/SQL) programmer too - to modify the project. It would make the project significantly more accessible to the broader community of users.

[ZeLonewolf]

openmaptiles-profile seems reasonable to me.

[wipfli]

For hot-air ballooning, powerlines are important to have on the map. I customized openmaptiles to have an extra layer for powerlines, see https://gist.github.com/wipfli/036aceb4a8008a10253b34ac1ded728f it felt strange to do it this way and it would be great if a repo split made customization easier.

[nyurik]

I agree that planetiler should be a generic tilegen tool, not a OMT-specific generator. That said, I agree with @msbarry that it would significantly hamper progress if it is split up right now, as it would require a stable interface between the core and the OMT-specific java code. I don't believe planetiler is mature enough to know exactly what that API would need to look like.

Thus, I second @shermp opinions that it might make far more sense to implement @ZeLonewolf 's #160 and similar configuration first - reducing the need/scope of the plugin API, and once that config schema is stable, move the OMT-specific stuff elsewhere.