Improve database management, files storage management and sub-process management

[gschwind]

Overview

This patch series include many change to have a even better handle of sub-process. The patch series include:

• Change the management of end-points, now the service provide /wps, /status, /files URL
• Refactor of Process and Services
• Rewrite of Storage Back-ends
• Dynamic status files (Implement dynamic status response and data response #354)
• Implement optional file lock for the database access, after several test the current lock is not safe.

Missing peace:

• Implementation of the back-end for S3 storage

More detail on patches series

Storage Back-end rewrite:
The rewrite of back-end storage, simplify the API of back-end storage by only allowing read and write and does not provide particular API for one or another Storage type (for example I removed the copy/move optimisation). Now files in the back-end storage may be exported. If they are exported they can be downloaded via /files endpoint using it's identifier. I did not implemented the S3 back-end because I can't test it, but I think there is nothing that blocking the implementation of the new API.

Multiple end-point:
This allow to implement dynamic status, and to manage files output via Storage back-end. There is no more direct file access, all files download goes through /files URL.

New locks scheme for databases.
The current lock scheme does not work, the filelock scheme is 100% safe in some setup, in particular one WPS server that access to the database. The filelock will not work if some process that is not aware of file lock try to access to the database or if the database is shared within several sever. I did not found a 'standard' lock method that is shared accross several database back-end thus I implemented this one.

I currently use this implementation on our server.

Best regards

Related Issue / Discussion

Additional Information

This contribution is supported by MINES ParisTech.

Contribution Agreement

(as per https://github.com/geopython/pywps/blob/master/CONTRIBUTING.rst#contributions-and-licensing)

• I'd like to contribute [feature X|bugfix Y|docs|something else] to PyWPS. I confirm that my contributions to PyWPS will be compatible with the PyWPS license guidelines at the time of contribution.
• I have already previously agreed to the PyWPS Contributions and Licensing Guidelines

[coveralls]

Coverage decreased (-0.07%) to 80.978% when pulling f6fa83f on gschwind:fix-status-failed-pull-request into 85ca819 on geopython:main.

[huard]

I tried testing with https://github.com/bird-house/emu (a test and demo servier) and all processes failed.

[gschwind]

I will have a look,

Thanks for feed back :)

[gschwind]

Hello, after a first look I gather two issue.

The first one is fixed by gschwind/pywps-emu@6e4e3aa and it's due to change that make request url ends with /wps when wps is made, to separate /files or /status request. I may change that by using service query parameter instead, as exemple.

The second issue that I found is related to the new Storage handle, I do not store file immediately in the storage when we use code such as response.outputs["output"].file = "/some/path". and it's seems that during test temporary file vanish before I try to put them into the store. If I put the into a directory that do not vanish it's work. In test outputs files are stored in process.workdir, if I put them directly in "/tmp" every thing goes well. I will investigate futher.

[gschwind]

Hello,

I did fixed more test regarding previous comment here : https://github.com/gschwind/pywps-emu/commits/update-test

I have also some update to fix outputs storage, but I need to fix issue for inputs. I will try to fix it and update the branch once I got something that I'm confident with it.

And I still have the test test_wps_ncml.py that I cannot fix, and that is invalid test in my opinion, it tries to retrieve data from not-existing server. Thus the test should be rethinked using client.get(...) instead of d3.retrieveData()

Best Regards.

[gschwind]

Hello,

I working on replacement of this patch suite, the first part is in #667

Best regards