Tag pruning limits

[collimarco]

From your documentation:

Note that this metadata is only generated when the cardinality of the field is less than 1 000. Tag pruning is notably useful on multi-tenant datasets.

Does that mean that we can only have 1000 tenants?

I was thinking about using the tenant ID as a tag...

[fmassot]

Hi @collimarco, the tag works if you don't have too many tenants indeed.

There are different use cases for multitenancy (example: a few very big tenants, many many very small ones) and the right solution for each one is not always obvious.
Our current plan for handling many tenants is to have one index per tenant and be able to index many tenants per indexer node. It's not currently possible to have an indexer which indexes on 1000 different indexes, we are currently optimizing the indexer and it will be shipped in the 0.4.

How many tenants do you have?

[fulmicoton]

@collimarco You can have one index with a lot of tenants. If you do you will have to pool your tenants into different sources.

For instance, upstream you could route you tenants data into different kafka topics.

This sharding is more dynamic than regular sharding.
Upstream it is perfectly fine for you to move one tenant from one shard (for instance because it got bigger).
It will not impact anything in quickwit's operations.

Unfortunately one piece of the puzzle is missing to let you operate such a system.
With such a system, you need to make sure that Quickwit does not merge spltis coming from different sources,
and Quickwit 0.3 does not offer any knobs for you to configure split merges.

This is not a big change, so if you have an important multitenant use case, we are happy to discuss it with you.

Quickwit also have other solutions to make this work. Some of them not shipped yet (supporting many indexes efficiently - i'm working on it right now), and some of them are shipped but undocumented (demux).
If you can discuss your use case, the best would be to identify together what the best approach is for your use case. We can prioritize missing features in our roadmap accordingly.