Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add best practice for custom partition in tag bucket #5606

Open
appletreeisyellow opened this issue Sep 20, 2024 · 0 comments
Open

Add best practice for custom partition in tag bucket #5606

appletreeisyellow opened this issue Sep 20, 2024 · 0 comments
Labels
InfluxDB v3 InfluxDB v3.0 (platforms using IOx)

Comments

@appletreeisyellow
Copy link
Contributor

Describe the issue here.

This request applies to both Cloud Dedicated and Clustered, but I will just to keep the information here for Cloud Dedicated to reduce duplicated words.

I'd like to add the following best practice for tag bucket definition:

Content

Best practice for number of tag buckets: While InfluxDB supports up to 1,000 tag buckets, the higher the number of tag buckets you configure, the worse your query performance may become, depending on how you define custom partition templates for a database or table.

For example: if a table's custom partition templates are defined as follows:

influxctl table create \
  --template-timeformat '%Y-%m-%d' \
  --template-tag-bucket customerID,1000 \
  DATABASE_NAME \
  example-table

The values of customerID tag are bucketed into 1,000 distinct "buckets" each day, which will generate 1,000 partitions per day for this table. Therefore, one month of data will generate 30k partitions (1,000 buckets/day * 30 days/month). If the retention
policy of this database is one month or longer, such a high number of partitions will significantly show down query performance. We recommend keeping the total number of partitions around 1,000 or below for this table by:

  1. Reducing the number of tag buckets.
  2. Setting the time part to a longer duration, e.g. '%Y-%m'.
  3. Shortening the retention policy (i.e. keeping the data for a shorter period).
Relevant URLs

Add the information to this page under "Tag bucket part templates": https://docs.influxdata.com/influxdb/cloud-dedicated/admin/custom-partitions/partition-templates/#tag-bucket-part-templates

Or "Partitioning best practices" page: https://docs.influxdata.com/influxdb/cloud-dedicated/admin/custom-partitions/best-practices/

Whatever you think is the best!

@jstirnaman jstirnaman added the InfluxDB v3 InfluxDB v3.0 (platforms using IOx) label Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
InfluxDB v3 InfluxDB v3.0 (platforms using IOx)
Projects
None yet
Development

No branches or pull requests

2 participants