[Feature] In-Memory / no persistence storage runtime

[touma-I]

Search before asking

• I searched the issues and found no similar issues.

Component

Library/core

Feature

In certain environment there is a need not to rely on persistence storage (local disk, S3, etc.). Currently, when a transform is initialized, it defines the folders where the input and output for the transform reside. It would be desirable to identify an alternative approach where the referenced input/output are in-memory objects.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[touma-I]

@touma-I to investigate how this is currently done by the datasift team.

[daw3rd]

Isn't this a call to transform() with an arrow Table or transform_binary() with byte array and file name (dummy perhaps) that indicates the file extension? If not, can you suggest an API?

[daw3rd]

This may be satisfied with the in-memory transform pipeline PR #424

[daw3rd]

This may be duplicate of issue #374

[daw3rd]

Closing and moving design discussion to #374