Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/128 improve performance #140

Merged
merged 10 commits into from
Mar 4, 2024
Merged

Feature/128 improve performance #140

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
6620cf4
Replace take_swap with mem::replace
chmp Mar 4, 2024
a2b0781
Remove manual Vec::fill impl
chmp Mar 4, 2024
e1e04ff
Simplify cache_key comp
chmp Mar 4, 2024
a227a10
Avoid deserializing the key during struct serialization
chmp Mar 4, 2024
42356fe
Merge remote-tracking branch 'origin/main' into feature/128-improve-p…
chmp Mar 4, 2024
db12779
Add outer sequence builder to remove unused list builder
chmp Mar 4, 2024
e054669
Hard code that the outer element is always a struct
chmp Mar 4, 2024
333bfab
Fix clippy + comments
chmp Mar 4, 2024
57e3c06
Update workflows
chmp Mar 4, 2024
a7d5e68
Update benchmarks
chmp Mar 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
},
"steps": [
{
"uses": "actions/checkout@v3"
"uses": "actions/checkout@v4"
},
{
"name": "rustc",
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
"runs-on": "ubuntu-latest",
"steps": [
{
"uses": "actions/checkout@v3"
"uses": "actions/checkout@v4"
},
{
"name": "rustc",
Expand Down
34 changes: 17 additions & 17 deletions Readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ write_chunk(
)?;
```

### Usage from python
### Usage from python

The written file can now be read in Python via

Expand Down Expand Up @@ -171,37 +171,37 @@ The detailed runtimes of the [benchmarks](./serde_arrow/benches/groups/) are lis

| label | time [ms] | arrow2_convert: | serde_arrow::to | serde_arrow::to | arrow_json::Rea |
|------------------------------|-----------|-----------------|-----------------|-----------------|-----------------|
| arrow2_convert::TryIntoArrow | 50.25 | 1.00 | 0.33 | 0.30 | 0.13 |
| serde_arrow::to_arrow2 | 152.02 | 3.03 | 1.00 | 0.92 | 0.41 |
| serde_arrow::to_arrow | 164.94 | 3.28 | 1.09 | 1.00 | 0.44 |
| arrow_json::ReaderBuilder | 375.15 | 7.47 | 2.47 | 2.27 | 1.00 |
| arrow2_convert::TryIntoArrow | 51.73 | 1.00 | 0.32 | 0.28 | 0.13 |
| serde_arrow::to_arrow2 | 160.55 | 3.10 | 1.00 | 0.88 | 0.41 |
| serde_arrow::to_arrow | 183.29 | 3.54 | 1.14 | 1.00 | 0.47 |
| arrow_json::ReaderBuilder | 391.25 | 7.56 | 2.44 | 2.13 | 1.00 |

### complex_common_serialize(1000000)

| label | time [ms] | arrow2_convert: | serde_arrow::to | serde_arrow::to | arrow_json::Rea |
|------------------------------|-----------|-----------------|-----------------|-----------------|-----------------|
| arrow2_convert::TryIntoArrow | 475.17 | 1.00 | 0.32 | 0.30 | 0.13 |
| serde_arrow::to_arrow2 | 1464.57 | 3.08 | 1.00 | 0.92 | 0.40 |
| serde_arrow::to_arrow | 1584.39 | 3.33 | 1.08 | 1.00 | 0.43 |
| arrow_json::ReaderBuilder | 3656.23 | 7.69 | 2.50 | 2.31 | 1.00 |
| arrow2_convert::TryIntoArrow | 509.11 | 1.00 | 0.33 | 0.26 | 0.14 |
| serde_arrow::to_arrow2 | 1534.82 | 3.01 | 1.00 | 0.80 | 0.42 |
| serde_arrow::to_arrow | 1921.37 | 3.77 | 1.25 | 1.00 | 0.53 |
| arrow_json::ReaderBuilder | 3653.99 | 7.18 | 2.38 | 1.90 | 1.00 |

### primitives_serialize(100000)

| label | time [ms] | arrow2_convert: | serde_arrow::to | serde_arrow::to | arrow_json::Rea |
|------------------------------|-----------|-----------------|-----------------|-----------------|-----------------|
| arrow2_convert::TryIntoArrow | 13.12 | 1.00 | 0.40 | 0.29 | 0.10 |
| serde_arrow::to_arrow2 | 32.46 | 2.47 | 1.00 | 0.71 | 0.25 |
| serde_arrow::to_arrow | 45.47 | 3.47 | 1.40 | 1.00 | 0.35 |
| arrow_json::ReaderBuilder | 130.77 | 9.97 | 4.03 | 2.88 | 1.00 |
| arrow2_convert::TryIntoArrow | 13.96 | 1.00 | 0.47 | 0.33 | 0.11 |
| serde_arrow::to_arrow2 | 29.42 | 2.11 | 1.00 | 0.70 | 0.23 |
| serde_arrow::to_arrow | 41.94 | 3.01 | 1.43 | 1.00 | 0.33 |
| arrow_json::ReaderBuilder | 128.67 | 9.22 | 4.37 | 3.07 | 1.00 |

### primitives_serialize(1000000)

| label | time [ms] | arrow2_convert: | serde_arrow::to | serde_arrow::to | arrow_json::Rea |
|------------------------------|-----------|-----------------|-----------------|-----------------|-----------------|
| arrow2_convert::TryIntoArrow | 151.80 | 1.00 | 0.44 | 0.32 | 0.11 |
| serde_arrow::to_arrow2 | 344.82 | 2.27 | 1.00 | 0.73 | 0.25 |
| serde_arrow::to_arrow | 473.56 | 3.12 | 1.37 | 1.00 | 0.34 |
| arrow_json::ReaderBuilder | 1403.08 | 9.24 | 4.07 | 2.96 | 1.00 |
| arrow2_convert::TryIntoArrow | 133.19 | 1.00 | 0.47 | 0.31 | 0.09 |
| serde_arrow::to_arrow2 | 284.80 | 2.14 | 1.00 | 0.67 | 0.20 |
| serde_arrow::to_arrow | 427.89 | 3.21 | 1.50 | 1.00 | 0.30 |
| arrow_json::ReaderBuilder | 1408.58 | 10.58 | 4.95 | 3.29 | 1.00 |

<!-- end:benchmarks -->

Expand Down
8 changes: 4 additions & 4 deletions serde_arrow/src/arrow2_impl/api.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ use crate::{
internal::{
error::Result,
schema::{GenericField, SerdeArrowSchema},
serialization_ng::ArrayBuilder,
serialization_ng::OuterSequenceBuilder,
source::deserialize_from_source,
},
};
Expand Down Expand Up @@ -58,7 +58,7 @@ use crate::{
/// # Ok(())
/// # }
/// ```
pub struct Arrow2Builder(ArrayBuilder);
pub struct Arrow2Builder(OuterSequenceBuilder);

impl std::fmt::Debug for Arrow2Builder {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
Expand All @@ -74,13 +74,13 @@ impl Arrow2Builder {
///
pub fn new(fields: &[Field]) -> Result<Self> {
let schema = SerdeArrowSchema::from_arrow2_fields(fields)?;
Ok(Self(ArrayBuilder::new(&schema)?))
Ok(Self(OuterSequenceBuilder::new(&schema)?))
}

/// Add a single record to the arrays
///
pub fn push<T: Serialize + ?Sized>(&mut self, item: &T) -> Result<()> {
self.0.extend(&[item])
self.0.push(item)
}

/// Add multiple records to the arrays
Expand Down
4 changes: 2 additions & 2 deletions serde_arrow/src/arrow2_impl/serialization.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@ use crate::{
common::MutableBitBuffer,
error::{fail, Result},
schema::GenericField,
serialization_ng::ArrayBuilder,
serialization_ng::{ArrayBuilder, OuterSequenceBuilder},
},
};

impl ArrayBuilder {
impl OuterSequenceBuilder {
/// Build the arrow2 arrays
pub fn build_arrow2_arrays(&mut self) -> Result<Vec<Box<dyn Array>>> {
let fields = self.take_records()?;
Expand Down
8 changes: 4 additions & 4 deletions serde_arrow/src/arrow_impl/api.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ use crate::{
internal::{
error::Result,
schema::{GenericField, SerdeArrowSchema},
serialization_ng::ArrayBuilder,
serialization_ng::OuterSequenceBuilder,
source::deserialize_from_source,
},
};
Expand Down Expand Up @@ -58,7 +58,7 @@ use crate::{
/// # Ok(())
/// # }
/// ```
pub struct ArrowBuilder(ArrayBuilder);
pub struct ArrowBuilder(OuterSequenceBuilder);

impl std::fmt::Debug for ArrowBuilder {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
Expand All @@ -74,13 +74,13 @@ impl ArrowBuilder {
///
pub fn new(fields: &[Field]) -> Result<Self> {
let schema = SerdeArrowSchema::from_arrow_fields(fields)?;
Ok(Self(ArrayBuilder::new(&schema)?))
Ok(Self(OuterSequenceBuilder::new(&schema)?))
}

/// Add a single record to the arrays
///
pub fn push<T: Serialize + ?Sized>(&mut self, item: &T) -> Result<()> {
self.0.extend(&[item])
self.0.push(item)
}

/// Add multiple records to the arrays
Expand Down
4 changes: 2 additions & 2 deletions serde_arrow/src/arrow_impl/serialization.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@ use crate::{
internal::{
common::MutableBitBuffer,
error::{fail, Result},
serialization_ng::ArrayBuilder,
serialization_ng::{ArrayBuilder, OuterSequenceBuilder},
},
};

impl ArrayBuilder {
impl OuterSequenceBuilder {
pub fn build_arrow_arrays(&mut self) -> Result<Vec<ArrayRef>> {
let fields = self.take_records()?;
let arrays = fields
Expand Down
Loading