Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chore/broswerfs deprecated issue 127 #93

Draft
wants to merge 65 commits into
base: master
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
20f1fdf
Update license, package.json, and add basic notice to the Readme (#3)
wilwade Apr 27, 2021
5d03d1b
Bloom r us (#2)
enddynayn May 5, 2021
5547ad4
GitHub Actions NPM Release (#5)
wilwade May 6, 2021
c887f10
Setup package.json to support the dist directory instead of just incl…
wilwade May 10, 2021
7d3e6dc
Build before publishing (#7)
wilwade May 10, 2021
ce24b50
chore(README): update to include bloom filter (#9)
enddynayn May 11, 2021
4099f37
use xxhash-wasm (#8)
shannonwells May 12, 2021
2ccc4eb
Update package org from unfinishedlabs -> dsnp (#12)
wilwade May 18, 2021
a8ddc96
Chore/pre release 0.1.0 (#13)
wilwade Jun 7, 2021
a4f425a
Don't publish types for now (#14)
wilwade Jun 7, 2021
ba60652
feat(fetch): add browser/node http client (#15)
enddynayn Jun 18, 2021
b35942b
Parquetjs browser support (#17)
shannonwells Jun 30, 2021
f605856
fix bug caused by side-effect of change in xxhasher. regen package-lo…
shannonwells Jul 1, 2021
ac8ebb2
update BSON due to vulnerability, minor fixes (#20)
shannonwells Jul 1, 2021
81e38e4
add cjs and ejm builds (#21)
acruikshank Jul 13, 2021
98871a2
Clean up dependencies and remove lodash (#22)
wilwade Aug 17, 2021
44d9f3c
Added test cases for types and fixed bigint precision (#24)
waylandli Oct 21, 2021
a82975b
Chore/typescript conversion/27 (#33)
waylandli Nov 30, 2021
7c7bbb9
updated typescript and esbuild. Fixed typescript issue (#35)
waylandli Dec 2, 2021
6fb5fcd
Convert lib/schema.js to typescript 1/3 (#37)
waylandli Feb 22, 2022
f1bee25
Converting shred.js to typescript 2/3 (#40)
waylandli Feb 24, 2022
14bbb97
Convertinging util.js to typescript 3/3 (#39)
waylandli Feb 25, 2022
ce1f98b
Reader conversion (#44)
waylandli Mar 16, 2022
eb029bd
[Issue #41] - Slimmed down FixedTFramedTransport class (#46)
dopatraman Mar 16, 2022
d3b2089
Removed redundant reference to callback (#48)
dopatraman Apr 12, 2022
a1bc22e
fix for typed array BYTE_ARRAY (#23) (#49)
aramikm Apr 13, 2022
4bd382e
removed force32 (#51)
aramikm Apr 19, 2022
91ac192
Removed ts-ignored lines (#50)
dopatraman Apr 19, 2022
8f30383
Add test coverage for Buffer Reader (#52)
dopatraman Apr 22, 2022
b4ca349
Support async iteration in the reader (#55)
wilwade Apr 26, 2022
33c80fe
Export the types into the package (#54)
wilwade Apr 26, 2022
0b027a3
Removed async declaration (#59)
dopatraman Apr 29, 2022
5a94d60
Chore/ts writer take2 (#60)
mehtaishita May 10, 2022
555af72
Upstream bug fixes for RLE encoding (#61)
dopatraman May 10, 2022
cadf486
Performance improvements using a cursor instead of a shift (#57)
wilwade May 10, 2022
2f150a0
Update packages (#63)
wilwade May 10, 2022
b8bfc07
Remove baseUrl config option (#65)
wilwade May 11, 2022
c86f490
Chore/fix entry file (#66)
wilwade May 11, 2022
d382d0f
Types Cleanup (#67)
wilwade May 11, 2022
73cf70d
Better handling of millis and micros (#68)
wilwade May 12, 2022
a2cd4ff
Bug/browser types bug #70 (#71)
wilwade Jun 22, 2022
a62db08
Fix typo in comment (#74)
JasonYeMSFT Dec 13, 2022
a011a2e
Feature - collect and report multiple field errors (#75)
dgaudet Mar 8, 2023
2c733b5
Add ability to read decimal columns (#79)
dgaudet Apr 26, 2023
b5698e4
Feature: Parquet Schema from JSON Schema (#82) with
wilwade May 25, 2023
17cb5ed
update to node 16, FIX HASHER BUG (#84)
shannonwells May 25, 2023
fa1865b
Ensure Buffer objects are returned by compression functions (#88)
JasonYeMSFT Jun 21, 2023
19f3ffa
Decimal Writer Support (#90)
wilwade Jun 23, 2023
43732c5
Add null pages and boundary order (Fixes #92) (#94)
wilwade Jul 13, 2023
ac5257d
Feature: Timestamp support for JSONSchema generated schemas (#95)
noxify Jul 25, 2023
c07e7e8
Add support to byte array decimal fields (#97)
YECHUNAN Aug 14, 2023
bda4e3f
upgrade to nodejs 18.18.2 (#101)
shannonwells Nov 17, 2023
19707ef
Support bloom filters in lists/nested columns (#105)
shannonwells Nov 28, 2023
e0f1ebd
Update docs and add simple test for readme encoding examples (#107)
wilwade Jan 9, 2024
0a42955
Use the Double primative for JSON Schema "number" type (#111)
mpotter Jan 11, 2024
6fdb9da
Update index.ts to support RLE_DICTIONARY (#112)
saritvakrat Jan 18, 2024
8d34ac1
Reference Tests and Breaking Change: Optional nullable fields are now…
wilwade Jan 19, 2024
2622ff1
allow typeLength to come from opts.column when decoding FIXED_LEN_BYT…
j4ys0n Jan 19, 2024
117e5a5
Feat/support aws s3 v3 (#115)
shannonwells Jan 30, 2024
91fc71f
Update Deprecated Function Calls (#118)
wilwade Feb 10, 2024
3de7eea
fix: Fix incorrect primitive type detection (#122)
JasonYeMSFT Mar 13, 2024
5db9db4
Swap brotli dependency (#123)
wilwade Mar 14, 2024
2435994
Dependency updates (#128)
wilwade May 14, 2024
6396d26
Linting and Formatting (#133)
wilwade Jul 1, 2024
d6b64e0
chore(*): replace browserfs for zenfs
enddynayn Jul 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Performance improvements using a cursor instead of a shift (#57)
wilwade authored May 10, 2022
commit cadf48649dbbb543873fff40971aa8e12b9bdaee
8 changes: 6 additions & 2 deletions lib/reader.ts
Original file line number Diff line number Diff line change
@@ -43,6 +43,7 @@ class ParquetCursor {
columnList: Array<Array<unknown>>;
rowGroup: Array<unknown>;
rowGroupIndex: number;
cursorIndex: number;
/**
* Create a new parquet reader from the file metadata and an envelope reader.
* It is usually not recommended to call this constructor directly except for
@@ -56,14 +57,15 @@ class ParquetCursor {
this.columnList = columnList;
this.rowGroup = [];
this.rowGroupIndex = 0;
this.cursorIndex = 0;
}

/**
* Retrieve the next row from the cursor. Returns a row or NULL if the end
* of the file was reached
*/
async next() {
if (this.rowGroup.length === 0) {
if (this.cursorIndex >= this.rowGroup.length) {
if (this.rowGroupIndex >= this.metadata.row_groups.length) {

return null;
@@ -76,9 +78,10 @@ class ParquetCursor {

this.rowGroup = parquet_shredder.materializeRecords(this.schema, rowBuffer);
this.rowGroupIndex++;
this.cursorIndex = 0;
}

return this.rowGroup.shift();
return this.rowGroup[this.cursorIndex++];
}

/**
@@ -87,6 +90,7 @@ class ParquetCursor {
rewind() {
this.rowGroup = [];
this.rowGroupIndex = 0;
this.cursorIndex = 0;
}
};