Refactoring tests to be more faster, more better, more stronger #141
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In my environment I do not have a useful or workable docker-compose setup, so this allows pointing the tests towards and existing Kafka broker(s) and S3 endpoint. Currently overriding the docker-compose workloads for Azure tests is not working.
…` flags This will make development iteration times and binaries much smaller, which is an improvement
This code is copied and pasted so we need to have it made more common
rtyler
force-pushed
the
test-flexibility
branch
2 times, most recently
from
cc7d756 to
09c9aeb
Compare
This looks like it was just as a copy/paste error
…ests
There's a test failure I'm seeing in GitHub Actions that I'm hoping this
resolves:
hread 'when_both_workers_started_simultaneously' panicked at 'called `Result::unwrap()` on an `Err` value: LoadCheckpoint { source: Storage { source: Generic { store: "S3", source: GetRequest { source: Error { retries: 0, message: "Received redirect without LOCATION, this normally indicates an incorrectly configured region", source: None }, path: "emails_s3-33758618-2217-4969-9a91-dbd6b973ba04/_delta_log/_last_checkpoint" } } } }', tests/helpers/mod.rs:602:66
This change includes a lot of minor changes to compile against newer deltalake libraries Closes delta-io#156
This is being done in init_logger() which means its typically as early as possible for most tests
rtyler
force-pushed
the
test-flexibility
branch
from
b64938b to
1e3dda8
Compare
Versions of the parquet crate after 38 don't perform direct writes to the underlying writer, but instead create rowgroup writers, so the mechanism for understanding the number of bytes "written"
rtyler
force-pushed
the
test-flexibility
branch
from
1e3dda8 to
91172ac
Compare
rtyler
added a commit
to rtyler/kafka-delta-ingest
that referenced
this pull request
Dec 17, 2023
Built on delta-io#141, should be merged after that
mightyshazam
approved these changes
Dec 18, 2023
| @@ -30,21 +30,25 @@ jobs: | |||
| profile: default | |||
| toolchain: stable | |||
| override: true | |||
| - name: build and lint with clippy | |||
| run: cargo clippy | |||
| - name: Build and lint for S3 | |||
There was a problem hiding this comment.
Consider using
matrix:
feature: ["s3", "azure"]
| .map(|dt: DateTime<Utc>| Value::Number((dt.timestamp_nanos() / 1000).into())) | ||
| parsed.ok().map(|dt: DateTime<Utc>| { | ||
| Value::Number( | ||
| (dt.timestamp_nanos_opt() |
There was a problem hiding this comment.
We use this logic pretty regularly. It might make sense to extract it into a method or implement into
impl<i64> Into<i64> for DateTime<Utc> {
fn into(self) -> i64 {
self.timestamp_nanos_opt()
.expect("Failed to convert timezone to nanoseconds")
/ 1000
}
}
Implementing Into might be too much, though, so a utility method may be beter.
rtyler
added a commit
to rtyler/kafka-delta-ingest
that referenced
this pull request
Dec 18, 2023
Built on delta-io#141, should be merged after that
rtyler
added a commit
that referenced
this pull request
Jan 2, 2024
Built on #141, should be merged after that
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I started this work just trying to make it easier to re-use an existing Minio and Kafka configuration for the integration tests, and in the process started finding more and more that should be cleaned up in
tests/