zero-copy rom source data possible ?

[makapuf]

A use case for heatshrink is storing data in flash compressed and decompressing it to a RAM buffer as needed (the original blog post seems to show the library came from this very need). However, the current design implies copying data to RAM input buffer. this is necessary if the data is in a file, but not if it's in ROM/ MCU flash : in this case the data is readily available to use, however the implementation stores the input buffer right next to decompression window buffer.

Would it be possible to provide separate source data as a pointer to ROM + decompression scratch data - avoiding sink interruptions, unnecessary copies and RAM consumption or is copying and proximity of those buffers necessary by the algorithm ?

Thanks, makapuf

[silentbicycle]

It could be adapted to use ROM for the input buffer, but it might be a fairly deep change. The buffering impacts the algorithm - it needs to have a full buffer to do (de)compression, because it needs to have a fixed amount of the recent context.

It'd be worth a try - adding a variant of sink() to the API that updates a pointer to a const uint8_t region would probably be the way to go.

[silentbicycle]

The API would probably look like:

enum HSE_set_input_res {
    HSER_SET_INPUT_OK = 0,
    HSER_SET_INPUT_ERROR_NULL = -1,
};

enum HSE_set_input_res heatshrink_encoder_set_input(heatshrink_encoder *hse,
    const uint8_t *input, size_t input_size);

enum HSE_step_input_res {
    HSER_STEP_INPUT_MORE = 0,
    HSER_STEP_INPUT_DONE = 1,
    HSER_STEP_INPUT_ERROR_NULL = -1,
    HSER_STEP_INPUT_ERROR_MISUSE = -2,
};

enum HSE_step_input_res heatshrink_encoder_step_input(heatshrink_encoder *hse,
    uint8_t *out_buf, size_t out_buf_size, size_t *output_size);

and essentially the same for the decoder. A #define could compile out either the sink/poll API or the set/step input API.

[sutaburosu]

Fwiw, I just grabbed the low hanging fruit: duplicate the sink() function, and modify it to read directly from Flash. It elimates a memcpy() for each buffer fill with minimal changes to heatshrink. https://github.com/sutaburosu/scintillating_heatshrink/blob/master/heatshrink_decoder.ino#L140-L159

Thanks for an awesome library, Scott. It has helped me to achieve things on AVR that I didn't think I would be able to do without a bigger platform.

[silentbicycle]

Great!

Duplicating sink is probably the right call there. I don't know if there's a good way to simultaneously satisfy constraints on the pointer address space (P, in this case) and alignment.

[silentbicycle]

Another developer and I have been discussing this, reworking the IO model to accommodate zero-copy will be a more substantial change, we are planning it as one of the major features for 0.5.