OpenTrepWrapper

Table of Content (ToC)

Table of contents generated with markdown-toc

Overview

OpenTREP aims at providing a clean API, its corresponding C++ implementation and Python extension, for parsing travel-/transport-focused requests. It powers the https://transport-search.org Web site (as well as its newer version, https://www2.transport-search.org).

As part of the OpenTREP releases, there is a Python extension, named opentrep and published on PyPi. That Python extension comes as a so-called binary wheel only for MacOS (which makes it very easy to install on that platform, then). Contributions are welcome to also build wheels for other platforms (e.g., Linux, including WSL on MS Windows). So far, for those other platforms, the Python extension has to be built from the source distribution, also available on PyPi in the same place.

OpenTrepWrapper (this project) provides a simple Python wrapper around the OpenTREP Python extension, and is also released on PyPi.

Refefences

OpenTREP project: https://github.com/trep/opentrep
This GitHub project: https://github.com/trep/wrapper
PyPi artifacts:
- OpenTREP Python extension: https://github.com/trep/opentrep/pulls
- OpenTREP simple Python wrapper: https://pypi.org/project/OpenTrepWrapper/
OpenTravelData (OPTD) project: https://github.com/opentraveldata/opentraveldata
How to install pyenv and pipenv
How to setup, on your favorite Linux distribution (e.g., Ubuntu LTS, Debian or CentOS), C++-, Python- and database-related development tools: https://github.com/cpp-projects-showcase/docker-images
Twine utility

Configuration

Installation of `pyenv` and `pipenv`

See https://github.com/machine-learning-helpers/induction-python/tree/master/installation/virtual-env for the details.
Install Python:

$ pushd ~/.pyenv && git pull && popd # if pyenv is in ~/.pyenv
$ pyenv install 3.8.5 && pyenv global 3.8.5 && \
  pip install -U pip pipenv && pyenv global system

Clone this Git repository:

$ mkdir -p ~/dev/geo/trep && \
  git clone https://github.com/trep/wrapper.git ~/dev/geo/trep-wrapper

Install the Python virtual environment:

$ cd ~/dev/geo/trep-wrapper
$ pipenv --rm && rm -f Pipfile.lock && \
  pipenv install && pipenv install --dev

Installation of OpenTREP

This module does not install OpenTREP itself; the overview gives the general procedure on how to do it.

Also, on RedHat/CentOS/Fedora, OpenTREP is packaged and can therefore easily be installed with the native packager manager (dnf or yum).

Alternatively, on most of the platforms, it can be installed by following the instructions in the OpenTREP README.md file.

Basically, a few C++-, Python- and database-related development tools have to be installed. The Docker images for C++ projects GitHub repository gives all the details in the Dockerfile of the corresponding Linux distribution folders, e.g., ubuntu2004/Dockerfile for Ubuntu 20.04 LTS (Focal Fossa).
And then the OpenTREP Python extension may simply be installed with pip (either invoked through the native Python interpreter or through pipenv with a Python interpreter installed by pyenv).
Note that the OpenTREP Python extension is not referred to in the Pipfile file, in order to have a minimum viable Python virtual environment installed by pipenv. If opentrep were referred to in the Pipfile file, the installation of the Python virtual environment may just stop in error after an attempt to compile the OpenTREP Python extension, and it would be hard to debug, without evenhaving a minimal viable Python virtual environment.
Once the Python virtual environment has been installed with pipenv, The installation of the OpenTREP Python extension should go something like:

user@laptop$ pipenv shell
Launching subshell in virtual environment…
(trep-wrapper-FkiBIXof) ✔ python -V
Python 3.8.5
(trep-wrapper-FkiBIXof) ✔ pip install -U opentrep
Collecting opentrep
  Using cached opentrep-0.7.7.post6-cp38-cp38-macosx_10_15_x86_64.whl (10.3 MB)
Installing collected packages: opentrep
Successfully installed opentrep-0.7.7.post6
(trep-wrapper-FkiBIXof) ✔ exit

Check that the OpenTREP Python extension (opentrep) has been installed properly:

user@laptop$ pipenv run python -mpip freeze|grep opentrep
opentrep==0.7.7.post6
$ pipenv run python -mpip show opentrep
Name: opentrep
Version: 0.7.7.post6
Summary: Simple Python wrapper for OpenTREP
Home-page: https://github.com/trep/opentrep
Location: ~/.local/share/virtualenvs/trep-wrapper-FkiBIXof/lib/python3.8/site-packages

Set a few environment variables accordingly:

$ export TREP_VENV="${HOME}/.local/share/virtualenvs/trep-wrapper-FkiBIXof"
$ export PYTHONPATH="${TREP_VENV}/lib:${TREP_VENV}/lib/python3.8/site-packages/pyopentrep"
$ export LD_LIBRARY_PATH="${TREP_VENV}/lib"
$ export PATH="${TREP_VENV}/bin:${PATH}"

Alternatively, if OpenTREP, including its Python extension, has been installed from the sources, the environment variables would rather be:

$ export TREP_DIR="${HOME}/dev/deliveries/opentrep-latest"
$ export PYTHONPATH="${TREP_DIR}/lib/python3.8/site-packages/pyopentrep:${TREP_DIR}/lib"
$ export LD_LIBRARY_PATH="${TREP_DIR}/lib"
$ export PATH="${TREP_DIR}/bin:${PATH}"

As a reminder, on MacOS,
- OpenTREP may be installed with pip, without any virtual environment (as it does not play well with the MacOS Python so-called framework):

$ python3 -mpip uninstall -y opentrep
$ python3 -mpip install -U opentrep

The envrionment variables become:

$ export PYTHONPATH=/usr/local/lib/python3.8/site-packages/pyopentrep:/usr/local/lib
$ export DYLD_LIBRARY_PATH=/usr/local/lib

For the remaining of this document, it is assumed that OpenTREP has been installed by pip, either with the native Python framework (MacOS) or with pipenv (Linux, including WSL on MS Windows)

Download the OpenTravelData (OPTD) data and index them

Download the OpenTravelData:

>>> import opentraveldata
>>> myOPTD = opentraveldata.OpenTravelData()
>>> myOPTD.downloadFilesIfNeeded()
>>> myOPTD
OpenTravelData:
	Local IATA/ICAO POR file: /tmp/opentraveldata/optd_por_public_all.csv
	Local UN/LOCODE POR file: /tmp/opentraveldata/optd_por_unlc.csv
>>> myOPTD.fileSizes()
(44079255, 4888752)
>>>

Initialize the Xapian index with the -i option of pyopentrep.py, so as to index the full OpenTravelData (OPTD) POR (points of reference) data file
- On Linux:

$ pipenv run python ${TREP_VENV}/lib/python3.8/site-packages/pyopentrep/pyopentrep.py -p /tmp/opentraveldata/optd_por_public_all.csv -i

On MacOS:

$ DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib ASAN_OPTIONS=detect_container_overflow=0 /usr/local/Cellar/python\@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python /usr/local/lib/python3.8/site-packages/pyopentrep/pyopentrep.py -p /tmp/opentraveldata/optd_por_public_all.csv -i

OPTD-maintained list of POR (points of reference): '/tmp/opentraveldata/optd_por_public_all.csv'
Xapian-based travel database/index: '/tmp/opentrep/xapian_traveldb0'
SQLite database: '/tmp/opentrep/sqlite_travel.db'
Perform the indexation of the (Xapian-based) travel database.
That operation may take several minutes on some slow machines.
It takes less than 20 seconds on fast ones...
Number of actually parsed records: 1,000, out of 103,394 records in the POR data file so far
...
Number of actually parsed records: 20,000, out of 122,394 records in the POR data file so far
Done. Indexed 20335 POR (points of reference)

Test of the OpenTREP Python extension

Start the Python interpreter on Linux:

$ pipenv run python

Start the Python interpreter on MacOS:

$ DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib ASAN_OPTIONS=detect_container_overflow=0 /usr/local/Cellar/python\@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python

Test the OpenTREP Python extension:

>>> import pyopentrep
>>> openTrepLibrary = pyopentrep.OpenTrepSearcher()
>>> initOK = openTrepLibrary.init ('/tmp/opentraveldata/optd_por_public.csv', '/tmp/opentrep/xapian_traveldb', 'sqlite', '/tmp/opentrep/sqlite_travel.db', 0, False, True, True, 'pyopentrep.log')
>>> initOK
True
>>> openTrepLibrary.search('S', 'nce sfo')
'NCE/0,SFO/0'

Test the wrapper application

In the following Python examples, it is assumed that an interactive Python Shell has been launched:

$ pipenv run python
Python 3.8.5 (default, Jul 24 2020, 13:35:18) 
>>>

Or, on MacOS:

$ ASAN_OPTIONS=detect_container_overflow=0 \
 DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib \
 /usr/local/Cellar/python\@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python
Python 3.8.5 (default, Jul 24 2020, 13:35:18) 
>>>

Import the module and initialize the Python extension:

>>> import OpenTrepWrapper
>>> otp = OpenTrepWrapper.OpenTrepLib()
>>> otp.init_cpp_extension(por_path=None,
                           xapian_index_path="/tmp/opentrep/xapian_traveldb",
                           sql_db_type="nodb",
                           sql_db_conn_str=None,
                           deployment_nb=0,
                           log_path="test_OpenTrepWrapper.log",
                           log_level=5)
>>>

Index the OPTD data file:

>>> otp.index()

Search
- Short output format:

>>> otp.search(search_string="nce sfo", outputFormat="S")
([(89.8466, 'NCE'), (357.45599999999996, 'SFO')], '')
------------------

Full output format:

>>> otp.search(search_string="nce sfo", outputFormat="F")
search_string: nce sfo
Raw result from the OpenTrep library:
1. NCE-A-6299418, 8.16788%, Nice Côte d'Azur International Airport, Nice Cote d'Azur International Airport, LFMN, , FRNCE, , 0, 1970-Jan-01, 2999-Dec-31, , NCE|2990440|Nice|Nice|FR|PAC, PAC, FR, , France, 427, France, EUR, NA, Europe, 43.6584, 7.21587, S, AIRP, 93, Provence-Alpes-Côte d'Azur, Provence-Alpes-Cote d'Azur, 06, Alpes-Maritimes, Alpes-Maritimes, 062, 06088, 0, 3, 5, Europe/Paris, 1, 2, 1, 2018-Dec-05, , https://en.wikipedia.org/wiki/Nice_C%C3%B4te_d%27Azur_Airport, 43.6627, 7.20787, nce, nce, 8984.66%, 0, 0
2. SFO-C-5391959, 32.496%, San Francisco, San Francisco, , , USSFO, , 0, 1970-Jan-01, 2999-Dec-31, , SFO|5391959|San Francisco|San Francisco|US|CA, CA, US, , United States, 91, California, USD, NA, North America, 37.7749, -122.419, P, PPLA2, CA, California, California, 075, City and County of San Francisco, City and County of San Francisco, Z, , 864816, 16, 28, America/Los_Angeles, -8, -7, -8, 2019-Sep-05, SFO, https://en.wikipedia.org/wiki/San_Francisco, 0, 0, sfo, sfo, 35745.6%, 0, 0

------------------

JSON output format:

>>> otp.search(search_string="nce sfo", outputFormat="J")
search_string: nce sfo
Raw (JSON) result from the OpenTrep library:
{
    "locations": [
        {
            "iata_code": "NCE",
            "icao_code": "LFMN",
            "geonames_id": "6299418",
            "feature_class": "S",
            "feature_code": "AIRP",
            ...
        },
        {
            "iata_code": "SFO",
            "icao_code": "",
            "geonames_id": "5391959",
            "feature_class": "P",
            "feature_code": "PPLA2",
            ...
        }
    ]
}

------------------

Interpreted format from JSON:

>>> otp.search(search_string="nce sfo", outputFormat="I")
search_string: nce sfo
JSON format => recognised place (city/airport) codes:
NCE-LFMN-6299418 (8.1678768123135352%) / NCE: 43.658411000000001 7.2158720000000001; SFO--5391959 (32.496021550940505%) / SFO: 37.774929999999998 -122.41942; 
------------------

End the Python session:

>>> quit()

On MacOS, if there is an issue with the interceptors:

$ ASAN_OPTIONS=detect_container_overflow=0 \
 DYLD_INSERT_LIBRARIES=/Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib \
 /usr/local/Cellar/python\@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/Resources/Python.app/Contents/MacOS/Python -mpytest test_OpenTrepWrapper.py
====================== test session starts =====================
platform darwin -- Python 3.8.5, pytest-5.4.3, py-1.8.1, pluggy-0.13.1
rootdir: /Users/darnaud/dev/geo/trep-wrapper
plugins: dash-1.12.0
collected 3 items
test_OpenTrepWrapper.py ...    [100%]

====================== 3 passed in 1.35s =======================

Release OpenTrepWrapper to PyPi

Build the Python artifacts for OpenTrepWrapper:

$ rm -rf dist build */*.egg-info *.egg-info .tox MANIFEST
$ pipenv run python setup.py sdist bdist_wheel
$ ls -lFh dist
total 56
-rw-r--r--  1 user  staff   7.7K Mar  2 11:14 OpenTrepWrapper-0.7.7.post1-py3-none-any.whl
-rw-r--r--  1 user  staff   7.3K Mar  2 11:14 OpenTrepWrapper-0.7.7.post1-py3.8.egg
-rw-r--r--  1 user  staff   8.4K Mar  2 11:14 OpenTrepWrapper-0.7.7.post1.tar.gz

Publish to PyPi:

$ pipenv run twine upload -u __token__ dist/*

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

OpenTrepWrapper

Table of Content (ToC)

Overview

Refefences

Configuration

Installation of `pyenv` and `pipenv`

Installation of OpenTREP

Download the OpenTravelData (OPTD) data and index them

Test of the OpenTREP Python extension

Test the wrapper application

Release OpenTrepWrapper to PyPi

Files

README.md

Latest commit

History

README.md

File metadata and controls

OpenTrepWrapper

Table of Content (ToC)

Overview

Refefences

Configuration

Installation of pyenv and pipenv

Installation of OpenTREP

Download the OpenTravelData (OPTD) data and index them

Test of the OpenTREP Python extension

Test the wrapper application

Release OpenTrepWrapper to PyPi

Installation of `pyenv` and `pipenv`