Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Zimfarm readyness #14

Merged
merged 29 commits into from
Aug 29, 2023
Merged

Zimfarm readyness #14

merged 29 commits into from
Aug 29, 2023

Commits on Aug 21, 2023

  1. Rework CLI arguments + upgrade to Zimscraperlib 3.1.1

    @mdp @benoit74
    mdp authored and benoit74 committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    e9e404b View commit details
    Browse the repository at this point in the history

  2. Revert name change

    @benoit74
    benoit74 committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    c081320 View commit details
    Browse the repository at this point in the history

  3. Use toptal .gitignore

    @benoit74
    benoit74 committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    63e1bee View commit details
    Browse the repository at this point in the history

  4. Adapt components names

    @benoit74
    benoit74 committed Aug 21, 2023
    Configuration menu
    Copy the full SHA
    a042cb7 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2023

  1. Rework to simplify CLI arguments + align with common openzim scrapers… 

    … behavior
    @benoit74
    benoit74 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    75acfee View commit details
    Browse the repository at this point in the history

  2. Adopt Python bootstrap conventions + other small fixes

    @benoit74
    benoit74 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    15ebea1 View commit details
    Browse the repository at this point in the history

  3. Add utility function to compute/check ZIM descriptions based on defau… 

    …lt and user provided values
    @benoit74
    benoit74 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    fa31089 View commit details
    Browse the repository at this point in the history

  4. Switch to official NodeJS image instead of devcontainer one

    @benoit74
    benoit74 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    249ca7a View commit details
    Browse the repository at this point in the history

  5. Remove devcontainer since it did not worked

    @benoit74
    benoit74 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    5662d2b View commit details
    Browse the repository at this point in the history

  6. Return error code when scrapper has to stop

    @benoit74
    benoit74 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    929e4c3 View commit details
    Browse the repository at this point in the history

  7. Install Python dependencies early to benefit for Docker build cache 

    In order to benefit from Docker build cache, we install Python dependencies
separately from the scraper itself.

We do it early in the build pipe so that this layer is modified only if dependencies
are updated. Otherwise, only layers linked to scraper source code or zimui are updated,
leading to a much faster build + using less resources since we do not update a big layer
again and again for only a small scraper source code modification
    @benoit74
    benoit74 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    e6a25b4 View commit details
    Browse the repository at this point in the history

  8. Update deps

    @benoit74
    benoit74 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    49af8e6 View commit details
    Browse the repository at this point in the history

  9. Switch pyright mode to basic for now

    @benoit74
    benoit74 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    4a447aa View commit details
    Browse the repository at this point in the history

  10. Fix workflow for Docker test

    @benoit74
    benoit74 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    fac9ab0 View commit details
    Browse the repository at this point in the history

  11. Ignore B108 issue about /tmp

    @benoit74
    benoit74 committed Aug 22, 2023
    Configuration menu
    Copy the full SHA
    8720c37 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2023

  1. Delete unmaintained files 

    - Makefile is not in the suite of file maintained at openzim
- pypi-readme.rst provides less value than README.md and is more difficult / error prone to maintain
    @benoit74
    benoit74 committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    b66a27a View commit details
    Browse the repository at this point in the history

  2. Enhance README.md following review comments

    @benoit74
    benoit74 committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    4f8a45b View commit details
    Browse the repository at this point in the history

  3. Move scraper logic to specific class + prepare everything asap + othe… 

    …r minor fixes
    @benoit74
    benoit74 committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    aea2a4f View commit details
    Browse the repository at this point in the history

  4. Add tests for scraper checks at initialization

    @benoit74
    benoit74 committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    12c5e16 View commit details
    Browse the repository at this point in the history

  5. Replace pypi-readme.rst by README.md

    @benoit74
    benoit74 committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    791bb58 View commit details
    Browse the repository at this point in the history

  6. Remove TODO since it can't be done for now

    @benoit74
    benoit74 committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    ba102f2 View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2023

  1. Revert Dockerfile to regular way of building

    @benoit74
    benoit74 committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    90f434a View commit details
    Browse the repository at this point in the history

  2. Use date instead of datetime + add missing ruff ignore

    @benoit74
    benoit74 committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    4a50085 View commit details
    Browse the repository at this point in the history

  3. Do not ignore .map files which are not generated by Vite anymore

    @benoit74
    benoit74 committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    a930123 View commit details
    Browse the repository at this point in the history

  4. docstring

    @benoit74
    benoit74 committed Aug 24, 2023
    Configuration menu
    Copy the full SHA
    5195ecb View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2023

  1. Remove useless parenthesis

    @benoit74
    benoit74 committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    1251432 View commit details
    Browse the repository at this point in the history

  2. Use more meaningfull args value 

    - Docker can set its specific values based on environment variables
- Default values are now ok for developer/Python usage
    @benoit74
    benoit74 committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    d64c514 View commit details
    Browse the repository at this point in the history

  3. Move debug-only options to environment variables

    @benoit74
    benoit74 committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    7b5bef0 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2023

  1. Simplify stuff following review

    @benoit74
    benoit74 committed Aug 29, 2023
    Configuration menu
    Copy the full SHA
    2fd2853 View commit details
    Browse the repository at this point in the history