Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace webpub manifest parser with pydantic (PP-1367) #2163

Merged
merged 2 commits into from
Nov 20, 2024

Conversation

jonathangreen
Copy link
Member

@jonathangreen jonathangreen commented Nov 13, 2024

Description

Replace our custom web manifest parser with Pydantic models.

I replace the validation we were doing via a patched json schema with the pydantic parser, since it gives better errors and more closely matches what we are expecting in Palace.

Edit: I broke this one into two PRs to make it a little easier to review. This PR actually implements the new models in our existing code, #2168 adds the models.

Motivation and Context

JIRA: PP-1367

Note: After this one goes in we should cut a new release of https://github.com/ThePalaceProject/webpub-manifest-parser and then mark it as archived, since we won't be using it anymore.

How Has This Been Tested?

  • Ran unit tests
  • Imported some feeds locally:
    • Test Palace Marketplace feed
    • Palace Bookshelf feed

Checklist

  • I have updated the documentation accordingly.
  • All new and existing tests passed.

@jonathangreen jonathangreen force-pushed the feature/replace-webpub-manifest-parser branch from 6f7438a to 2ae2600 Compare November 13, 2024 16:40
Copy link

codecov bot commented Nov 13, 2024

Codecov Report

Attention: Patch coverage is 93.33333% with 9 lines in your changes missing coverage. Please review.

Project coverage is 91.00%. Comparing base (5a49b70) to head (bed6d49).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/palace/manager/core/opds2_import.py 90.12% 4 Missing and 4 partials ⚠️
src/palace/manager/api/odl/importer.py 96.55% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2163      +/-   ##
==========================================
+ Coverage   90.87%   91.00%   +0.12%     
==========================================
  Files         359      359              
  Lines       41263    41136     -127     
  Branches     8883     8842      -41     
==========================================
- Hits        37498    37436      -62     
+ Misses       2476     2424      -52     
+ Partials     1289     1276      -13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:

@jonathangreen jonathangreen changed the title Replace webpub manifest parser with pydantic Replace webpub manifest parser with pydantic (PP-1367) Nov 13, 2024
@jonathangreen jonathangreen marked this pull request as ready for review November 13, 2024 18:04
@jonathangreen jonathangreen added the feature New feature label Nov 13, 2024
@jonathangreen jonathangreen requested a review from a team November 13, 2024 18:04
@jonathangreen jonathangreen force-pushed the feature/replace-webpub-manifest-parser branch from 4a9279d to 50fc11c Compare November 14, 2024 17:14
@jonathangreen jonathangreen changed the base branch from main to feature/add-pydantic-opds-models November 14, 2024 17:14
@jonathangreen jonathangreen force-pushed the feature/add-pydantic-opds-models branch from 399768d to 1ca1dce Compare November 19, 2024 14:52
@jonathangreen jonathangreen force-pushed the feature/replace-webpub-manifest-parser branch from da3e71a to 8101188 Compare November 19, 2024 14:53
Base automatically changed from feature/add-pydantic-opds-models to main November 19, 2024 18:06
@jonathangreen jonathangreen force-pushed the feature/replace-webpub-manifest-parser branch from 8101188 to bed6d49 Compare November 19, 2024 18:08
Copy link
Contributor

@tdilauro tdilauro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks ready to roll! 🍣


contributor_metadata = ContributorData(
sort_name=contributor.sort_as,
display_name=contributor.name,
display_name=str(contributor.name),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm. Maybe just food for future thought here, but this makes me realize that, at least for some properties (like contributors), it might not make sense for name to be a LanguageMap. But maybe it does (e.g., in the case of corporate authors, which might need a translation).

I note also that the schema calls for a string or a language map.

@jonathangreen jonathangreen merged commit 71ef9d4 into main Nov 20, 2024
21 checks passed
@jonathangreen jonathangreen deleted the feature/replace-webpub-manifest-parser branch November 20, 2024 16:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants