Cache downloaded data #43
Assignees
Comments
pmayd
added a commit
that referenced
this issue
Aug 1, 2022
pmayd
added a commit
that referenced
this issue
Aug 9, 2022
felixschmitz
added a commit
that referenced
this issue
Sep 19, 2022
pmayd
added a commit
that referenced
this issue
Oct 8, 2022
* added clean_cache function function that removes all files from cache directory or only one, specified by name. Likely to be moved to cache.py, once this is on dev * Added ToDos, Unified query param comparison * Updated destatis status check #45 Incorporated further response codes and response type checks. Output for Code 0 tbd * Merged updates from personal branch * Added http 5xx error check, added destatis status tests * Updated urls, unified Destatis type-style * Added generic response creation for generic http_helper test * Updated type hints, made contents rely on functionality from http_helpers * merged current dev into feature branch. * Merged changes from draft branch, including pylint fixes, Exception and Error clarification. Also moved clean_cache function. Some ToDos still left, #45. * fixed linting issues, including inconsistent returns. * hotfix of overlooked error #45. * narrowed the catched exception * Added notes w\ Felix regarding automatic job support, #40. * Notebook for interaction & workflow with jobs * integrating jobs into http_helper.py * Simplified if selections, added TODO comments, added catches for fail cases, #40 * splitting up jobs function and writing first tests for user input * work in progress * changes in jobs params for faster tests * Changes in job due to http_helper #43 * refactor: Updated & added documentation, identified Windows input issue, #40. * Commit before merge of dev * commit before PR * last working commit, #40. * seem to have fixed issue with destatis response, however, currently am timed out and can not finally check, #40. * Fixed new implementation and updated jobs notebook. Code needs clean-up & tests need to be updated (for input), #40. * changes with respect to first PR * Removed user input, always starting job, test update, currently on timeout * pylint changes * commit before merge * Code quality http_helper.py and tests * pylint uand black changes * removing _generic_status_dict() * Changes due to bandit * Changes due to mypy * refactor load_data to handle jobs * fix tests * Removed one old comment, #40 Co-authored-by: MarcoHuebner <[email protected]> Co-authored-by: Michael Aydinbas <[email protected]>
pmayd
added a commit
that referenced
this issue
Oct 12, 2022
* add flake8 config to pyproject.toml, add flake8-docstrings plugin to repo, add .pylintrc config from Google * add flake8 config, remove flake9 and use normal flake8 with own .flake8 config file * first functions for downloading cube files * update * no verification * pylint fixes * add simple test cases for _handle_status_code * update line length from 80 to 120, change indentation from 2 to 4 * activate more rules * fix pylint errors * fix pylint errors * fix missing dependencies * fix mypy errors * remove unncessary code * fix pylint issues * fix pylint config issues * add missing dependencies to .toml * fix import error * fix jsondecode error * add demo notebook for tablefile endpoint * fix return type issue * adjust error message for http_helper * fix column header issues * remove unncessary cut footer function * integrate PR feedback * update unit tests * refactor http_helper module * set area to optional parameter * fix typo * debug failing unit tests * debug failing unit tests * add .env * restore config_loader.py * integrate feedback from PR * integrate PR feedback * remove dotenv, use configparser instead (standard library) implement a user config using configparser module upload created doc as artifact update module docstring * merge tablefile and cubefile PR and use latest config module * [#43] Cache downloaded data; Implement feature with decorator; added tests * fix pylint errors; refactor code so that common code for downloading data is now in data.py module and table.py and cube.py only hold specific parsing logic * remove helper methods from get_data, move logic to the individual moduls * added clean_cache function function that removes all files from cache directory or only one, specified by name. Likely to be moved to cache.py, once this is on dev * Added ToDos, Unified query param comparison * Updated destatis status check #45 Incorporated further response codes and response type checks. Output for Code 0 tbd * Merged updates from personal branch * Added http 5xx error check, added destatis status tests * Updated urls, unified Destatis type-style * Added generic response creation for generic http_helper test * Updated type hints, made contents rely on functionality from http_helpers * Merged changes from draft branch, including pylint fixes, Exception and Error clarification. Also moved clean_cache function. Some ToDos still left, #45. * fixed linting issues, including inconsistent returns. * hotfix of overlooked error #45. * narrowed the catched exception * Updated clean_cache, updated http_helper * Fix lint issue with assert * mypy fixes for ci/cd pipeline #45 * Updated clean_cache function, added a first version of a test * Fixed linting issues, #45 * updated with remarks from pull request #45, #46 * updated logging due to linting errors W1203 * Improved test coverage, update temporary test directory for user names with spaces and non-latin characters, added mocked API call, added mocked create_settings call, moved test_http_helper, added TODOs for the future, #45 * Added mock as dependency (as otherwise CI/CD understandably fails), #45 * Also update project.lock, forgot it before, #45 * Fixed missing config dict error (in remote CI/CD) by mocking it, moved load_config into function, #45 * Fixed missing config dict error (in remote CI/CD) by mocking it, moved load_config into function, #45 * Fixed safety issues (mako and dparse) by updating poetry.lock, #45 * Fixed linting issue: timeout for requests.get added, #45 * Feat/update cube format (#67) * update cube module to handle cubes with more than one value variable in DQI. Add correct types to value columns so they are no longer of type object. * introduce Table und Cube class for coherent and simplified user interface; both classes download data with get_data() and provide access to data via .data and access to metadata via .metadata. Cube has .cube in addition for special cube metadata * reimplement cache for download data from endpoint * use compression for caching data * new tests for cube module * add tests for cache decorator that is called with functions that don't use the data endpoint (expect no caching) * find endpoint (#66) * simple example of Statistic class * first version of parsing cube; added pandas and notebook to dependencies * adding find and hierarchy * adding find and hierarchy * adding find and hierarchy * adding find and hierarchy * created results class * created results class * created results class * updated Results class with get_metadata for statistics and variables * updated string to f-strings according to pylint * solved pylint issues * solved pylint issues * solved pylint issues * several improvements based on feedback for PR #66 * several improvements based on feedback for PR #66 * several improvements based on feedback for PR #66 * modified docstring * modified docstring Co-authored-by: Michael Aydinbas <[email protected]> * Feat/refactor cache (#68) * refactor cache; no longer a decorator function but part of the new `load_data` function. This function checks if there is a hit in cache and loads the data from there, or load the data per request from Destatis. The new cache function stores data under `<name>/<endpoint>/<method>/hash(<request-params>)/<date-today>.zip` so different params will be cached differently `load_data` has a new parameter `as_json`, which defaults to `False`, so that the response is returned as raw text. However, setting `as_json=True` will return the response as a parsed JSON dict. * Feature/#48 hello world and profile endpoint (#69) * added clean_cache function function that removes all files from cache directory or only one, specified by name. Likely to be moved to cache.py, once this is on dev * Added ToDos, Unified query param comparison * Updated destatis status check #45 Incorporated further response codes and response type checks. Output for Code 0 tbd * Merged updates from personal branch * Added http 5xx error check, added destatis status tests * Updated urls, unified Destatis type-style * Added generic response creation for generic http_helper test * Updated type hints, made contents rely on functionality from http_helpers * merged current dev into feature branch. * Merged changes from draft branch, including pylint fixes, Exception and Error clarification. Also moved clean_cache function. Some ToDos still left, #45. * fixed linting issues, including inconsistent returns. * hotfix of overlooked error #45. * narrowed the catched exception * Updated clean_cache, updated http_helper * Fix lint issue with assert * mypy fixes for ci/cd pipeline #45 * Updated clean_cache function, added a first version of a test * Fixed linting issues, #45 * updated with remarks from pull request #45, #46 * updated logging due to linting errors W1203 * Added most of the methods from helloworld and profile endpoint, without tests so far, #48 * Modified pre-commit hook (set black profile for isort) to avoid isort and black conflicting changes * Added most of the methods from helloworld and profile endpoint - this time for real, without tests so far, #48 * Merged current dev into feature branch, #48 * Updated line-length argument for isort, added tests for helloworld and profile, #48 * merged dev again and fixed merge conflicts for automatic merge, #48 * Implemented removeresult + test, #48. * updated profile password test, #48, still isort issues seem ominous. * fixed isort in run tests to same line-length as used in pre-commit-config to avoid reoccuring conflicts in profile, #48, see also #60. * Fix W0707 raise ... from, #48. * Fix mypy type-hinting error, #48. * Minor requested updates, #48. * fixed mypy type, #48. * isort toml configuration commit (check line-length specification at only one point) * fixed line-length -> line_length * Added notebooks for helloworld and profile endpoint functions, updated get_data_from_endpoint to load_data as requested, #48. * fixed profile function text outputs, fixed tests, #48. * Added casting for mypy/ ci/cd, #48 * Feat/improve cache (#71) * refactor cache to work properly for jobs: no longer cache endpoint and method but only name as identifier, so a cache hit happens when the unique name is found in the cache data, irrelevant of the endpoint or method, thus working for data/resultfile and data/tablefile * normalize an object name so we can strip off the job-id * fix cicd * fix cicd * fix isort settings * fix mypy issues * add comments * refactor all tests to use python-mock (#72) * refactor all tests to use python-mock * update mocks * Feature/#40 automatic job query (#70) * added clean_cache function function that removes all files from cache directory or only one, specified by name. Likely to be moved to cache.py, once this is on dev * Added ToDos, Unified query param comparison * Updated destatis status check #45 Incorporated further response codes and response type checks. Output for Code 0 tbd * Merged updates from personal branch * Added http 5xx error check, added destatis status tests * Updated urls, unified Destatis type-style * Added generic response creation for generic http_helper test * Updated type hints, made contents rely on functionality from http_helpers * merged current dev into feature branch. * Merged changes from draft branch, including pylint fixes, Exception and Error clarification. Also moved clean_cache function. Some ToDos still left, #45. * fixed linting issues, including inconsistent returns. * hotfix of overlooked error #45. * narrowed the catched exception * Added notes w\ Felix regarding automatic job support, #40. * Notebook for interaction & workflow with jobs * integrating jobs into http_helper.py * Simplified if selections, added TODO comments, added catches for fail cases, #40 * splitting up jobs function and writing first tests for user input * work in progress * changes in jobs params for faster tests * Changes in job due to http_helper #43 * refactor: Updated & added documentation, identified Windows input issue, #40. * Commit before merge of dev * commit before PR * last working commit, #40. * seem to have fixed issue with destatis response, however, currently am timed out and can not finally check, #40. * Fixed new implementation and updated jobs notebook. Code needs clean-up & tests need to be updated (for input), #40. * changes with respect to first PR * Removed user input, always starting job, test update, currently on timeout * pylint changes * commit before merge * Code quality http_helper.py and tests * pylint uand black changes * removing _generic_status_dict() * Changes due to bandit * Changes due to mypy * refactor load_data to handle jobs * fix tests * Removed one old comment, #40 Co-authored-by: MarcoHuebner <[email protected]> Co-authored-by: Michael Aydinbas <[email protected]> * update dependencies * rename package to pystatis * import all main functions into pystatis main namespace * fix failing tests * add authors and maintainers * fix pyproject.toml * Feat/56 57 final readme (#73) * Updating README; add sections about Installation and first setup * updated readme with main features, how to use and how to contribute. * fix tests * added new functions cube, table Co-authored-by: dpleus <[email protected]> Co-authored-by: frederik9 <[email protected]> Co-authored-by: codehering <[email protected]> Co-authored-by: MarcoHuebner <[email protected]> Co-authored-by: Daniel Pleus <[email protected]> Co-authored-by: Felix Schmitz <[email protected]> Co-authored-by: dpleus <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: