Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

size RangeError with RLE encoding on INT32 field #100

Open
nicwaller opened this issue May 18, 2020 · 2 comments
Open

size RangeError with RLE encoding on INT32 field #100

nicwaller opened this issue May 18, 2020 · 2 comments

Comments

@nicwaller
Copy link

I'm trying to use RLE encoding with this library, but I always get this error regardless of how I tweak the configuration parameters.

The value "NaN" is invalid for option "size" at Function.alloc (buffer.js:371:3)

(node:74887) UnhandledPromiseRejectionWarning: RangeError [ERR_INVALID_OPT_VALUE]: The value "NaN" is invalid for option "size"
    at Function.alloc (buffer.js:371:3)
    at encodeRunBitpacked (/myproject/node_modules/parquetjs/lib/codec/rle.js:8:20)
    at Object.exports.encodeValues (/myproject/node_modules/parquetjs/lib/codec/rle.js:81:31)
    at encodeValues (/myproject/node_modules/parquetjs/lib/writer.js:287:34)
    at encodeDataPageV2 (/myproject/node_modules/parquetjs/lib/writer.js:346:19)
    at encodeColumnChunk (/myproject/node_modules/parquetjs/lib/writer.js:420:18)
    at encodeRowGroup (/myproject/node_modules/parquetjs/lib/writer.js:483:22)
    at ParquetEnvelopeWriter.writeRowGroup (/myproject/node_modules/parquetjs/lib/writer.js:201:18)
    at ParquetWriter.close (/myproject/node_modules/parquetjs/lib/writer.js:116:33)
    at test (/myproject/src/parquetify.js:112:18)

Here is my simple test case. Am I doing something wrong, or is this a real bug?

var person_schema = new parquet.ParquetSchema({
    name: { type: 'UTF8' },
    age: { type: 'INT32', encoding: 'RLE', bitWidth: 7 },
});
var writer = await parquet.ParquetWriter.openFile(person_schema, 'person.parquet');
await writer.appendRow({
    name: 'John Smith',
    age: 0,
});
await writer.close();

I'm using parquetjs 0.11.2 from https://registry.npmjs.org/parquetjs/-/parquetjs-0.11.2.tgz

@samsullivan
Copy link

The docs are wrong; you want to use typeLength instead of bitWidth.

See:

parquetjs/lib/writer.js

Lines 310 to 316 in 455ff9f

let valuesBuf = encodeValues(
column.primitiveType,
column.encoding,
values, {
typeLength: column.typeLength,
bitWidth: column.typeLength
});

@MollsReis
Copy link

Blast from the past, thanks for this @samsullivan I was having the same issue

wilwade added a commit to LibertyDSNP/parquetjs that referenced this issue Jan 9, 2024
Problem
=======
Docs were wrong from a long time ago per
ironSource#100

Solution
========
Updated the docs and added a simple test

Closes #106
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants