Data compression

[aydar-viv]

Hi there! I'm trying to understand how storage data compression works. Here is storage use graph:

Older data is ingested from different seq instance using clef tool. Recent data (from April onvards) is ingested directly to seq instance, and obviously takes much more space. Amount of evens is within same range, so I'm guessing imported/older data is heavily compressed. How compression works? Is there a way for force compression for anything older than X days to save disk space?

[liammclennan]

Hi @aydar-viv

The compression is the same across the entire stream. You can use the histogram on the events page to see how event volume has changed over time, or the 'Space by event type' built-in query to check the data size by event type. If the event volume is the same then it must be that the newer events carry more data. The two tools you have available to manage disk usage are:

• set a filter on the API key to reduce ingestion
• use retention policies to delete events when they are no longer needed

[aydar-viv]

Interesting... Asking because I noticed this:

• data on server A was taking let's say 150MB per day
• I exported data to server B using clef tool and it is taking 15MB per day

[liammclennan]

Next step is probably to export the data from both and do a text diff.

[aydar-viv]

Yeah I think I messed up with initial data import from different server and it was only partially transferred. Thanks!

[liammclennan]

Glad you found a solution.