[FEAT] Read tag-wahoo.xml and tags-to-keep.json from user directory f…

…irst (from python module as fallback) (#170) * new function to access files from user repo or PyPI installation * access tag-wahoo xml from user dir in favor of PyPI installtion * read tags-to-keep from .json instead of constants.py - incl. changing unittests to not mock variables but files and cleaning up the file * access tags-to-keep .json from user dir in favor of PyPI installtion * force using .json file from repo for unittests for unittesting it makes not sense to read the user-dir. The result would not be equal in all cases * delete tag-wahoo .xml files I am not aware of what they do they are coming from #38 and #34 but might most probably not be used actually. With this PR, users can have their own tag-wahoo .xml file in the user-directory * move initialization of dirs and version adjustments up - initialization needed for windows tooling. Needs to be before check of required programs * correction of user-dir vs. repo-dir - incl. usage of constant for user config directory - incl. creation of user config directory on start * new init function to copy files from python package to user dir * fix pylint findings * insert user-dir to docu and shorten README * insert table of arguments
treee111 · Nov 13, 2022 · fd23b8a · fd23b8a
1 parent a29fd99
commit fd23b8a
Show file tree

Hide file tree

Showing 18 changed files with 422 additions and 521 deletions.
diff --git a/.vscode/launch.json b/.vscode/launch.json
@@ -149,6 +149,14 @@
                 "-h"
             ]
         },
+        {
+            "name": "cli.init",
+            "type": "python",
+            "request": "launch",
+            "module": "wahoomc.init",
+            "console": "integratedTerminal",
+            "args": []
+        },
         {
             "name": "x/y: 133/88 (germany&france)",
             "type": "python",

diff --git a/README.md b/README.md
@@ -12,19 +12,16 @@
         <a href="#sponsors" alt="Contributions welcome">
             <img src="https://img.shields.io/badge/contributions-welcome-orange.svg" /></a>
     </p>
-    <h1>Wahoo Maps Creator</h1>
+    <h1>wahooMapsCreator</h1>
 </div>
-A tool to create up-to-date maps for your Wahoo BOLTv1 and BOLTv2, ROAM, ROAMv2 and ELEMNT!
+wahooMapsCreator is a tool to create up-to-date maps for your Wahoo BOLTv1 and BOLTv2, ROAM, ROAMv2 and ELEMNT!
 
 It runs on Windows, macOS as well as on Linux!
 
-## Basic Overview
-WahooMapsCreator is a tool to create maps based on the latest OSM data for your Wahoo devices. You can generate maps for the countries you like and you can control which OSM-tags are included.
+You can generate maps for the countries you like and you can control which OSM-tags are included based on latest OSM data.
 
 The maps of your device may be old because Wahoo did not release a newer version in the last years.
 
-OSM maps are constantly updated. With this program, the updated maps can be used on our Wahoo.
-
 # Get it running
 The instructions are intended to be suitable for beginners.
 
@@ -33,6 +30,9 @@ If anything is unclear or seams wrong, write an [:pencil2: issue](https://github
 ## Brand-New: Get POIs displayed on your Wahoo!
 [:cookie: here](docs/USAGE.md#pois---points-of-interest)
 
+## Brand-New: Control OSM tags to be included in your maps!
+[:wrench: here](docs/USAGE.md#user-specific-configuration)
+
 ## Download and Install required programs
 Using Anaconda to setup a virtual Python environment is the fastest way to get wahooMapsCreator running!
 

diff --git a/docs/TAGS_ON_MAP_AND_DEVICE.md b/docs/TAGS_ON_MAP_AND_DEVICE.md
@@ -1,18 +1,32 @@
 # OSM-tags during map creation and on your device <!-- omit in toc -->
 
 #### Table of contents <!-- omit in toc -->
+- [Order of generating maps](#order-of-generating-maps)
 - [Zoom levels and scale](#zoom-levels-and-scale)
-- [File tag-wahoo.xml](#file-tag-wahooxml)
-  - [Attribute "zoom-appear"](#attribute-zoom-appear)
+- [Files used in map processing](#files-used-in-map-processing)
+  - [File tags-to-keep.json](#file-tags-to-keepjson)
+  - [File tag-wahoo.xml](#file-tag-wahooxml)
+    - [Attribute "zoom-appear"](#attribute-zoom-appear)
+    - [Combination of tags-to-keep.json and tag-wahoo.xml](#combination-of-tags-to-keepjson-and-tag-wahooxml)
 - [Device-Theme](#device-theme)
   - [Attribute "zoom-min"](#attribute-zoom-min)
   - [Copy the theme to the device](#copy-the-theme-to-the-device)
   - [Use the theme in cruiser](#use-the-theme-in-cruiser)
-- [Combination of tag-wahoo.xml and device-theme](#combination-of-tag-wahooxml-and-device-theme)
+  - [Combination of tag-wahoo.xml and device-theme](#combination-of-tag-wahooxml-and-device-theme)
 - [Adjustments of the device themes](#adjustments-of-the-device-themes)
   - [mapsforge-bolt.xml](#mapsforge-boltxml)
 
 
+# Order of generating maps
+There are mainly 4 steps to process maps from raw OSM data to the final maps for the Wahoo device. You'll find them in the following list with links to the files you can control yourself.
+
+1. Filter relevant OSM tags from raw OSM country files - [tags-to-keep.json](#file-tags-to-keepjson)
+2. Create different map increments per tile: OSM map, land and sea
+3. Merge these map increments into a merged .osm.pbf file
+4. Create .map file while applying tag-wahoo.xml to the merged .osm.pbf file - [tag-wahoo.xml](#file-tag-wahooxml)
+
+For a OSM tag in the map to be displayed on your Wahoo device, it needs to be rendered using the [device theme](#device-theme).
+
 # Zoom levels and scale
 | Zoom level | Scale |
 | ---------- | :---: |
@@ -26,15 +40,24 @@
 
 That information were derived from cruiser on macOS. It needs to be checked if the zoom level and scale on cruiser and Wahoo device is exactly the same or if there is a difference.
 
-# File tag-wahoo.xml
+# Files used in map processing
+## File tags-to-keep.json
+The `tags-to-keep.json` file controls which tags and name-tags will stay on the map. This happens to keep the file of the generated maps low by filtering out out all other information from the downloaded OSM maps.
+
+## File tag-wahoo.xml
 The `tag-wahoo.xml` files defines how to proceed wit OSM-elements during map generation.
 The OSM-tags defined in the tag-wahoo.xml are stored in the map file.
 
 E.g. roads, locations, ...
 
-## Attribute "zoom-appear"
+### Attribute "zoom-appear"
 Each entry has a "zoom-appear" attribute, which defines from which zoom level onwards the element will be stored in the map. If zoom-appear is set to 13, the OSM-tag will be stored in the 500m, 200m and 100m zoom levels and therefore could be rendered.
 
+### Combination of tags-to-keep.json and tag-wahoo.xml
+To bring a certain OSM tag to your generated maps they have to be included in both files. You can have a look at existing OSM tags to get 
+1. `tags-to-keep.json` to not be filtered out
+2. `tag-wahoo.xml` to be included in the generated maps
+
 # Device-Theme
 The device theme defines, which OSM-tags are rendered on the device. In cruiser, you can also apply the device theme to preview generated maps on your computer.
 
@@ -60,7 +83,7 @@ See [here](COPY_TO_WAHOO.md#Copy-device-theme)
 You should always use the corresponding theme in cruiser if you preview generated maps on your computer.
 Because the theme kind of determines what you're gonna see, you want to preview what later on will be visible on the device.
 
-# Combination of tag-wahoo.xml and device-theme
+## Combination of tag-wahoo.xml and device-theme
 This zoom-appear in combination with the settings in the theme on the device (which can also be applied in cruiser) controls when certain elements are shown on our BOLT/ROAM etc (zoom-min).
 
 If a element is included in the map beginning from zoom level 10 (zoom-appear in tag-wahoo.xml) but on the device only displayed beginning with zoom level 12 (zoom-min in mapsforge-bolt.xml), the element is only displayed beginning zoom level 12.

diff --git a/docs/USAGE.md b/docs/USAGE.md
@@ -1,14 +1,15 @@
 # Usage of wahooMapsCreator <!-- omit in toc -->
 #### Table of contents <!-- omit in toc -->
 - [Usage of wahooMapsCreator](#usage-of-wahoomapscreator)
-    - [Always activate environment first](#always-activate-environment-first)
+  - [Always activate environment first](#always-activate-environment-first)
   - [Run wahooMapsCreator for your country](#run-wahoomapscreator-for-your-country)
   - [GUI (Graphical User Interface)](#gui-graphical-user-interface)
   - [CLI (Command Line Interface)](#cli-command-line-interface)
   - [Advanced CLI-Usage](#advanced-cli-usage)
     - [Main arguments](#main-arguments)
     - [Examples](#examples)
   - [POIs - Points of Interest](#pois---points-of-interest)
+  - [User specific configuration](#user-specific-configuration)
 
 # Usage of wahooMapsCreator
 wahooMapsCreator can be used in two different ways:
@@ -17,7 +18,10 @@ wahooMapsCreator can be used in two different ways:
 
 Both ways support the same arguments to be used for the map-creation process. You can choose the arguments via GUI or as [CLI-arguments](#advanced-cli-usage).
 
-### Always activate environment first
+## Always activate environment first
+```
+conda activate gdal-user
+```
 
 ## Run wahooMapsCreator for your country
 It might be a good idea to run wahooMapsCreator first for a small country e.g. Malta to check if everything is running fine.
@@ -42,10 +46,27 @@ Examples:
 - for Ireland: `python -m wahoomc cli -co ireland`
 
 ## Advanced CLI-Usage
-The script supports many arguments via command line.
+wahooMapsCreator supports many arguments via command line.
 For a list of all supported arguments, run:
 - `python -m wahoomc cli -h`
 
+
+wahooMapsCreator provides following arguments.
+
+| Option    | Description                                                                                                     | Mandatory | Default Value       |
+| :-------- | :-------------------------------------------------------------------------------------------------------------- | :-------- | :------------------ |
+| gui / cli | Start graphical user interface to select options or run the tool via command line interface.                    | X         | -                   |
+| -co / -xy | Country to create maps or X/Y coordinates to create maps for. X/Y as `133/35`.                                  | X         | -                   |
+| -md       | Maximum age of source maps and other files in days.                                                             | -         | 24                  |
+| -nbc      | Do not process border countries of tiles involving more than one country. Only useful when processing a country | -         | false               |
+| -fd       | Force download of files. Download all files new.                                                                | -         | false               |
+| -fp       | Force processing of files. Create all files new.                                                                | -         | false               |
+| -c        | Save uncompressed maps for Cruiser.                                                                             | -         | false               |
+| -tag      | File with tags to keep in the output.                                                                           | -         | `tag-wahoo-poi.xml` |
+| -gt       | Calculate tiles based on geofabrik index-v1.json file.                                                          | -         | false               |
+| -z        | Zip the country (and country-maps) folder.                                                                      | -         | false               |
+| -v        | Output debug logger messages.                                                                                   | -         | false               |
+
 ### Main arguments
 **Create maps for a country**
 - `python -m wahoomc cli -co <country>`
@@ -79,3 +100,14 @@ Actually, wahooMapsCreator includes fuel stations, backeries, cafes and railway
 3. Activate VTM rendering if needed
 - [see here](COPY_TO_WAHOO.md#activate-vtm-rendering)
 - see also: https://github.com/treee111/wahooMapsCreator/wiki/Enable-hidden-features
+
+## User specific configuration
+You can control popular configuration with your own files in the home directory `wahooMapsCreatorData/_config`.
+
+Actually, it is possible to control which OSM tags will stay on the map while filtering tags ([documentation](TAGS_ON_MAP_AND_DEVICE.md#file-tags-to-keepjson)) with `~/wahooMapsCreatorData/_config/tags-to-keep.json` . Furthermore, you can control which and how OSM tags will be included in the generated maps with tag-wahoo .xml ([documentation](TAGS_ON_MAP_AND_DEVICE.md#file-tag-wahooxml)) in directory `~/wahooMapsCreatorData/_config/tag_wahoo_adjusted/` and specify the file via `-tag` argument or via GUI.
+
+You can start the tool with `python -m wahoomc.init` to copy the files from the python module to the user directory. You'll have a structure and can change the files to your needs.
+
+The tool searches for configuration in your home directory first and secondly (fallback) in the python installation. If a file is in both directories with the same name, the file of the user directory will be used.
+
+The structure of `~/wahooMapsCreatorData/_config/tag_wahoo_adjusted/` is analogue to the one in the repo: `wahooMapsCreatorwahoomc/resources`.
diff --git a/tests/test_constants.py b/tests/test_constants.py
@@ -3,7 +3,7 @@
 """
 import os
 import unittest
-from mock import patch
+import mock
 
 
 from wahoomc.constants_functions import translate_country_input_to_geofabrik, translate_tags_to_keep, \
@@ -43,92 +43,74 @@ def test_translated_countries_no_mapping(self):
         self.assertEqual(expected, transl_c)
 
 
-tags_universal_simple = {
+tags_universal_simple = {"TAGS_TO_KEEP_UNIVERSAL": {
     'access': '',
     'area': 'yes'
-}
+}}
 
-tags_universal_adv = {
+tags_universal_adv = {"TAGS_TO_KEEP_UNIVERSAL": {
     'access': '',
     'area': 'yes',
     'bicycle': '',
     'bridge': '',
     'foot': ['ft_yes', 'foot_designated']
-}
-
-tags_universal_full = {
-    'access': '',
-    'area': 'yes',
-    'bicycle': '',
-    'bridge': '',
-    'foot': ['ft_yes', 'foot_designated'],
-    'highway': ['abandoned', 'bus_guideway', 'disused', 'bridleway', 'byway', 'construction', 'cycleway', 'footway', 'living_street', 'motorway', 'motorway_link', 'path', 'pedestrian', 'primary', 'primary_link', 'residential', 'road', 'secondary', 'secondary_link', 'service', 'steps', 'tertiary', 'tertiary_link', 'track', 'trunk', 'trunk_link', 'unclassified'],
-    'natural': ['coastline', 'nosea', 'sea', 'beach', 'land', 'scrub', 'water', 'wetland', 'wood'],
-    'leisure': ['park', 'nature_reserve'],
-    'railway': ['abandoned', 'bus_guideway', 'disused', 'funicular', 'light_rail', 'miniature', 'narrow_gauge', 'preserved', 'rail', 'subway', 'tram'],
-    'surface': '',
-    'tracktype': '',
-    'tunnel': '',
-    'waterway': ['canal', 'drain', 'river', 'riverbank'],
-    'wood': 'deciduous'
-}
-
-name_tags_universal_full = {
-    'admin_level': '2',
-    'area': 'yes',
-    'mountain_pass': '',
-    'natural': '',
-    'place': ['city', 'hamlet', 'island', 'isolated_dwelling', 'islet', 'locality', 'suburb', 'town', 'village', 'country']
-}
+}}
 
 
 class TestTranslateTags(unittest.TestCase):
     """
     tests for translating tags-constants between the universal format and OS-specific formats
     """
 
-    @patch('wahoomc.constants.TAGS_TO_KEEP_UNIVERSAL', tags_universal_simple)
-    def test_translate_tags_to_keep_simple_macos(self):
+    @ mock.patch("wahoomc.file_directory_functions.json.load")
+    @ mock.patch("wahoomc.open")
+    def test_translate_tags_to_keep_simple_macos(self, mock_open, mock_json_load):  # pylint: disable=unused-argument
         """
         Test translating tags to keep from universal format to macOS
         """
         tags = ['access', 'area=yes']
+        mock_json_load.return_value = tags_universal_simple
 
         transl_tags = translate_tags_to_keep()
         self.assertEqual(tags, transl_tags)
 
-    @patch('wahoomc.constants.TAGS_TO_KEEP_UNIVERSAL', tags_universal_simple)
-    def test_translate_tags_to_keep_simple_win(self):
+    @ mock.patch("wahoomc.file_directory_functions.json.load")
+    @ mock.patch("wahoomc.open")
+    def test_translate_tags_to_keep_simple_win(self, mock_open, mock_json_load):  # pylint: disable=unused-argument
         """
         Test translating tags to keep from universal format to Windows
         """
         tags_win = 'access= area=yes'
+        mock_json_load.return_value = tags_universal_simple
 
         transl_tags = translate_tags_to_keep(sys_platform='Windows')
         self.assertEqual(tags_win, transl_tags)
 
-    @patch('wahoomc.constants.TAGS_TO_KEEP_UNIVERSAL', tags_universal_adv)
-    def test_translate_tags_to_keep_adv_macos(self):
+    @ mock.patch("wahoomc.file_directory_functions.json.load")
+    @ mock.patch("wahoomc.open")
+    def test_translate_tags_to_keep_adv_macos(self, mock_open, mock_json_load):  # pylint: disable=unused-argument
         """
         Test translating tags to keep from universal format to macOS
         """
         tags = ['access', 'area=yes', 'bicycle',
                 'bridge', 'foot=ft_yes, foot_designated']
+        mock_json_load.return_value = tags_universal_adv
 
         transl_tags = translate_tags_to_keep()
         self.assertEqual(tags, transl_tags)
 
-    @patch('wahoomc.constants.TAGS_TO_KEEP_UNIVERSAL', tags_universal_adv)
-    def test_translate_tags_to_keep_adv_win(self):
+    @ mock.patch("wahoomc.file_directory_functions.json.load")
+    @ mock.patch("wahoomc.open")
+    def test_translate_tags_to_keep_adv_win(self, mock_open, mock_json_load):  # pylint: disable=unused-argument
         """
         Test translating tags to keep from universal format to Windows
         """
         tags_win = 'access= area=yes bicycle= bridge= foot=ft_yes =foot_designated'
+        mock_json_load.return_value = tags_universal_adv
 
         transl_tags = translate_tags_to_keep(sys_platform='Windows')
         self.assertEqual(tags_win, transl_tags)
 
-    # @patch('wahoomc.constants.TAGS_TO_KEEP_UNIVERSAL', tags_universal_full)
     def test_translate_tags_to_keep_full_macos(self):
         """
         Test translating tags to keep from universal format to macOS // all "tags to keep"
@@ -141,31 +123,29 @@ def test_translate_tags_to_keep_full_macos(self):
                 'leisure=park, nature_reserve', 'railway=rail, tram, station, stop',
                 'surface', 'tracktype', 'tunnel', 'waterway=canal, drain, river, riverbank', 'wood=deciduous']
 
-        transl_tags = translate_tags_to_keep()
+        transl_tags = translate_tags_to_keep(use_repo=True)
         self.assertEqual(tags, transl_tags)
 
-    # @patch('wahoomc.constants.TAGS_TO_KEEP_UNIVERSAL', tags_universal_full)
     def test_translate_tags_to_keep_full_win(self):
         """
         Test translating tags to keep from universal format to Windows // all "tags to keep"
         """
         tags_win = 'access= area=yes bicycle= bridge= foot=ft_yes =foot_designated amenity=fuel =cafe =drinking_water shop=bakery =bicycle highway=abandoned =bus_guideway =disused =bridleway =byway =construction =cycleway =footway =living_street =motorway =motorway_link =path =pedestrian =primary =primary_link =residential =road =secondary =secondary_link =service =steps =tertiary =tertiary_link =track =trunk =trunk_link =unclassified natural=coastline =nosea =sea =beach =land =scrub =water =wetland =wood landuse=forest =commercial =industrial =residential =retail leisure=park =nature_reserve railway=rail =tram =station =stop surface= tracktype= tunnel= waterway=canal =drain =river =riverbank wood=deciduous'
 
-        transl_tags = translate_tags_to_keep(sys_platform='Windows')
+        transl_tags = translate_tags_to_keep(
+            sys_platform='Windows', use_repo=True)
         self.assertEqual(tags_win, transl_tags)
 
-    # @patch('wahoomc.constants.NAME_TAGS_TO_KEEP_UNIVERSAL', name_tags_universal_full)
     def test_translate_name_tags_to_keep_full_macos(self):
         """
         Test translating name tags to keep from universal format to Windows // all "name tags to keep"
         """
         names_tags = ['admin_level=2', 'area=yes', 'mountain_pass', 'natural',
                       'place=city, hamlet, island, isolated_dwelling, islet, locality, suburb, town, village, country']
 
-        transl_tags = translate_tags_to_keep(name_tags=True)
+        transl_tags = translate_tags_to_keep(name_tags=True, use_repo=True)
         self.assertEqual(names_tags, transl_tags)
 
-    # @patch('wahoomc.constants.NAME_TAGS_TO_KEEP_UNIVERSAL', name_tags_universal_full)
     def test_translate_name_tags_to_keep_full_win(self):
         """
         Test translating name tags to keep from universal format to macOS // all "name tags to keep"
@@ -174,7 +154,7 @@ def test_translate_name_tags_to_keep_full_win(self):
         names_tags_win = 'admin_level=2 area=yes mountain_pass= natural= place=city =hamlet =island =isolated_dwelling =islet =locality =suburb =town =village =country'
 
         transl_tags = translate_tags_to_keep(
-            name_tags=True, sys_platform='Windows')
+            name_tags=True, sys_platform='Windows', use_repo=True)
         self.assertEqual(names_tags_win, transl_tags)
 
 

diff --git a/wahoomc/__main__.py b/wahoomc/__main__.py
@@ -4,4 +4,4 @@
 from wahoomc import main
 
 if __name__ == '__main__':
-    main.run()
+    main.run("run")