Batch processing from FS file lister node

[MateRadz]

Hi,

It would be great if that node would work with simply file name as input without the UI widget. I have tens of CSV files each 200 MB in length I need to process to extract some data from each file. Now I need to manually choose file by file using UI widget.

[Alkarex]

Hello,
I believe that would require making a node that reads from a list of local files with back-pressure mechanism and sequences.
Are the files on the same machine than where your Node-RED instance runs?

[MateRadz]

Yes. they are in that case. For the file size of 200 MB I actually managed to feed your node with standard file-in node set as buffer and then while processing your node properly releases lines on msg.ticks. So if the file is on that scale the issue can be closed:-). However going forward I would love to directly process Azure blob files - but for that I guess I need to write my own....

[Alkarex]

@MateRadz Do you by any chance use PostgreSQL? If yes, I have just made a new related node that supports back-pressure for reading and writing large datasets in Postgres. Feedback welcome there https://github.com/alexandrainst/node-red-contrib-postgresql :-)