Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

716 updating pydantic version #721

Open
wants to merge 94 commits into
base: main
Choose a base branch
from
Open

Conversation

Antonyjin
Copy link
Member

@Antonyjin Antonyjin commented Nov 11, 2024

Pydantic need to be updated to v2

Jira tickets : ALEPH-150

Self proofreading checklist

  • The new code clear, easy to read and well commented.
  • New code does not duplicate the functions of builtin or popular libraries.

Changes

Upgrading to pydantic 2.

  • Removing every deprecated functions
  • Adapt the parsing with pydantic 2 which is more strict (model_dump / model_dump_json)
  • Modify the usage of dotenv to adapt with pydantic 2

Antonyjin and others added 15 commits October 31, 2024 20:30
Started the migration using the tool dump-pydantic.
It only do some parts, but there are mch to do, not finished.
Now, have to change and update manually.
Typing is more strict in pydantic v2
…dation support

Validation was failing with Pydantic v2 due to stricter date formats and unhandled JSON errors.

Adapted the exceptions so we handle format/validation with changes bringed by
pydantic v2
The migration to Pydantic 2 bringed some warnings and deprecated
funtions.

Fixing them all manually
The return value for url was string but now pydantic is strict and need a good
type

Changing str to HttpUrl for url
…l_dump_json`

In Pydantic v2, the `.json()` method has been replaced with `.model_dump_json()` for serializing json
Updated the mock in `test_check_internet_wrong_result_code` to use `.model_dump_json`
instead of `.json` to align with this change.
Pydantic 2 add more fields while returning error such as `url`, `ctx` or `input`
Adapting the test to the new format given by pydantic
Adding a / with the url makes the url containing two slashes which break the url

Removing it
@Antonyjin Antonyjin linked an issue Nov 11, 2024 that may be closed by this pull request
Changed @root_validator to @model_validator(mode="after") for checking data after the model is created.
Updated values to use a more flexible type for Pydantic v2.
Accessed values as a dictionary (values["payload"]) instead of using values.payload.
Copy link

codecov bot commented Nov 12, 2024

Codecov Report

Attention: Patch coverage is 85.90604% with 21 lines in your changes missing coverage. Please review.

Project coverage is 62.45%. Comparing base (0241c04) to head (5143f3d).

Files with missing lines Patch % Lines
src/aleph/vm/conf.py 88.00% 1 Missing and 8 partials ⚠️
src/aleph/vm/storage.py 57.14% 3 Missing ⚠️
src/aleph/vm/orchestrator/views/authentication.py 94.59% 1 Missing and 1 partial ⚠️
src/aleph/vm/controllers/qemu/instance.py 0.00% 1 Missing ⚠️
src/aleph/vm/orchestrator/reactor.py 0.00% 1 Missing ⚠️
src/aleph/vm/orchestrator/run.py 50.00% 1 Missing ⚠️
src/aleph/vm/orchestrator/tasks.py 50.00% 1 Missing ⚠️
src/aleph/vm/orchestrator/views/__init__.py 50.00% 1 Missing ⚠️
src/aleph/vm/orchestrator/views/operator.py 0.00% 1 Missing ⚠️
src/aleph/vm/utils/__init__.py 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #721      +/-   ##
==========================================
- Coverage   62.50%   62.45%   -0.05%     
==========================================
  Files          70       70              
  Lines        6232     6246      +14     
  Branches      506      515       +9     
==========================================
+ Hits         3895     3901       +6     
- Misses       2179     2180       +1     
- Partials      158      165       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Antonyjin and others added 9 commits November 19, 2024 20:46
- Improved error handling in `get_signed_pubkey` to align with stricter
JSON validation and date format requirements in Pydantic v2.
- Addressed warnings and deprecated functions raised during the migration to Pydantic v2.
- Updated type annotations to comply with Pydantic's stricter type checks
(`url` now requires proper type validation).
- Refactored test mocks to use `.model_dump_json()` instead of `.json()` for serialization
- Adjusted test cases to account for new fields in validation errors (`url`, `ctx`, and `input`)
introduced in Pydantic v2.
- Applied linting for style consistency across tests and codebase.

fix: yamlfix and pydantic are incompatible

Fix: HttpUrl no longer need scheme value
Remove the counter way to assign a vm_id as it didn't work reliably

Jira ticket: ALEPH-272

That method was broken when persitent instances were loaded at start up.
Since the "new" feature that allow persistent instance across aleph-vm reboot
if one was started then aleph-vm was stopped and restarted the counter method
could reassign the ip and break the existing vm's.

Secundary reason was that the feature wasn't working properly with the default
settings, as `2**available_bits` returned 1. So that code path was only used if
the node owner tweaked some undocumented settings making it hard to identify and
debug in prod nodes.
* Feature: Implement new EVM chains.

* FIX: Update Makefile with new dependency.

* Fix: Updated to proper released package version of aleph_message dependency.

---------

Co-authored-by: Andres D. Molins <[email protected]>
* Feature: allow IPv6 DNS

Problem IPv6 DNS were automatically filtered when detected from resolvectl

Solution: Nameservers are now split into ipv4 and ipv6 and can be passed to the VM accordingly
At the moment we pass them if the ipv6 parameter is present on the tap interface but we need a
more robust detection method


* Display proper env conf
Symptom:
The CustomError from the diagnostics VM was printed even if if IGNORE_TRACEBACK_FROM_DIAGNOSTICS was set to True (the default)

Analysis:
This was caused by the refactoring of the fastapi_example/main.py file
done in fe9235a
Which changed the output used to detect the error to catch

Solution:
Fix detection string
Jira ticket: ALEPH-298

some CRN failed on any action with error OSError: [Errno 24] Too many open files:

Solution:
Properly close stream to journald when the VM is stopped
Pydantic's validation system now enforces strict checks, we cannot
leave fields marked as required empty
@Antonyjin Antonyjin force-pushed the 716-updating-pydantic-version branch from 90f4b27 to 907cd6d Compare November 19, 2024 11:51
- Updated the display() method to use self.model_config.env_prefix instead of self.Config.env_prefix.
The previous implementation caused an AttributeError because Config is no longer supported in Pydantic v2.

Adjusted the method to align with the new Pydantic v2 configuration approach, which uses model_config for settings validation and customization.
@Antonyjin Antonyjin force-pushed the 716-updating-pydantic-version branch from bd06710 to 950d1cd Compare November 19, 2024 12:03
Antonyjin and others added 12 commits November 19, 2024 13:04
Previously, `os.makedirs` was called directly on variables like `MESSAGE_CACHE`
or `CODE_CACHE`, which could be `None`. This caused issues with `mypy` and potential
runtime errors since `os.makedirs` does not handle `None`.

The fix ensures these variables are checked for `None` before calling `os.makedirs`,
preventing invalid operations and aligning with type checks.
Previously, `os.makedirs` was called directly on variables like `MESSAGE_CACHE`
or `CODE_CACHE`, which could be `None`. This caused issues with `mypy` and potential
runtime errors since `os.makedirs` does not handle `None`.

The fix ensures these variables are checked for `None` before calling `os.makedirs`,
preventing invalid operations and aligning with type checks.

fix: wrong name
@Antonyjin Antonyjin force-pushed the 716-updating-pydantic-version branch from f8ce889 to d96e9ca Compare November 22, 2024 17:28
@Antonyjin Antonyjin marked this pull request as ready for review November 22, 2024 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Updating Pydantic version
3 participants