Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate the performance regression in ordered append for partially compressed chunk #6032

Open
akuzm opened this issue Aug 31, 2023 · 5 comments
Open

Investigate the performance regression in ordered append for partially compressed chunk #6032

akuzm opened this issue Aug 31, 2023 · 5 comments
Labels
bug

Comments

@akuzm
Copy link
Member

akuzm commented Aug 31, 2023

We should investigate this before releasing 2.12. Introduced in #5812
@akuzm akuzm added the bug label Aug 31, 2023
@akuzm akuzm added this to the TimescaleDB 2.12 milestone Aug 31, 2023
@konskov konskov self-assigned this Sep 1, 2023
@horzsolt horzsolt removed this from the TimescaleDB 2.12 milestone Sep 14, 2023
@konskov
Copy link
Contributor

konskov commented Sep 18, 2023

Looking into this now. For the first case, I see that two queries marked as eligible for chunk append:

SELECT * FROM ht_metrics_partially_compressed ORDER BY time_bucket('1d', time) DESC, device LIMIT 1;
SELECT * FROM ht_metrics_partially_compressed WHERE device = 3 ORDER BY device, time DESC LIMIT 1;

do not use a chunk append plan.
Before the commit that introduced the regression, the plan was a gather merge on top of parallel append, with this commit the plan is a merge append.

For the second case, I actually see no difference between the produced plans or the execution times when running the benchmark locally.

@akuzm
Copy link
Member Author

akuzm commented Sep 18, 2023

I started benchmarks of the "fix ordered append" commit and the previous one again, maybe it was some simultaneous change in test tables which looked as a regression.

@akuzm
Copy link
Member Author

akuzm commented Sep 19, 2023

New benchmark:
https://grafana.ops.savannah-dev.timescale.com/d/fasYic_4z/compare-akuzm?orgId=1&var-branch=All&var-run1=2737&var-run2=2736&var-threshold=0.05&var-use_historical_thresholds=false

Old benchmark:
https://grafana.ops.savannah-dev.timescale.com/d/fasYic_4z/compare-akuzm?orgId=1&var-branch=All&var-run1=2664&var-run2=2665&var-threshold=0.05&var-use_historical_thresholds=false

Looks like there are some regressions that are reproducible.

@konskov
Copy link
Contributor

konskov commented Oct 6, 2023 •
edited
Loading

it seems the following queries get slower:

  1. SELECT * FROM space_part ORDER BY a, time DESC LIMIT 1;
  2. SELECT * FROM space_part ORDER BY time_bucket('2d', time) DESC, a LIMIT 1;
  3. SELECT * FROM ht_metrics_partially_compressed ORDER BY time_bucket('1d', time) DESC, device LIMIT 1;
  4. SELECT min(device), max(value) FROM ht_metrics_partially_compressed;

1. SELECT * FROM space_part ORDER BY a, time DESC LIMIT 1;

Plan Before: Parallel, Gather Merge

explain SELECT * FROM space_part ORDER BY a, time DESC LIMIT 1;
                                                     QUERY PLAN                                                      
---------------------------------------------------------------------------------------------------------------------
 Limit  (cost=49873.81..49873.92 rows=1 width=20)
   ->  Gather Merge  (cost=49873.81..1184421.78 rows=9724018 width=20)
         Workers Planned: 2
         ->  Sort  (cost=48873.78..61028.81 rows=4862009 width=20)
               Sort Key: _hyper_3_13_chunk.a, _hyper_3_13_chunk."time" DESC
               ->  Parallel Append  (cost=0.04..24563.74 rows=4862009 width=20)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_13_chunk  (cost=0.04..6.59 rows=159000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_37_chunk  (cost=0.00..6.59 rows=159 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_14_chunk  (cost=0.04..6.59 rows=159000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_38_chunk  (cost=0.00..6.59 rows=159 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_15_chunk  (cost=0.04..6.59 rows=159000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_39_chunk  (cost=0.00..6.59 rows=159 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_34_chunk  (cost=0.04..10.99 rows=299000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_58_chunk  (cost=0.00..10.99 rows=299 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_35_chunk  (cost=0.04..10.99 rows=299000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_59_chunk  (cost=0.00..10.99 rows=299 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_36_chunk  (cost=0.04..10.99 rows=299000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_60_chunk  (cost=0.00..10.99 rows=299 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_16_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_40_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_17_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_41_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_18_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_42_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_19_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_43_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_20_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_44_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_21_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_45_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_22_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_46_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_23_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_47_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_24_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_48_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_25_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_49_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_26_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_50_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_27_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_51_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_28_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_52_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_29_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_53_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_30_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_54_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_31_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_55_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_32_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_56_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Custom Scan (DecompressChunk) on _hyper_3_33_chunk  (cost=0.04..11.05 rows=305000 width=20)
                           ->  Parallel Seq Scan on compress_hyper_4_57_chunk  (cost=0.00..11.05 rows=305 width=152)
                     ->  Parallel Seq Scan on _hyper_3_13_chunk  (cost=0.00..1.01 rows=1 width=20)
                     ->  Parallel Seq Scan on _hyper_3_14_chunk  (cost=0.00..1.01 rows=1 width=20)
(56 rows)

After plan: LIMIT on top of Merge Append

explain SELECT * FROM space_part ORDER BY a, time DESC LIMIT 1;
                                                  QUERY PLAN                                                  
---------------------------------------------------------------------------------------------------------------
 Limit  (cost=396.04..396.07 rows=1 width=20)
   ->  Merge Append  (cost=396.04..333234.59 rows=11676002 width=20)
         Sort Key: _hyper_3_13_chunk.a, _hyper_3_13_chunk."time" DESC
         ->  Sort  (cost=1357.70..2032.70 rows=270000 width=20)
               Sort Key: _hyper_3_13_chunk.a, _hyper_3_13_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_13_chunk  (cost=0.03..7.70 rows=270000 width=20)
                     ->  Seq Scan on compress_hyper_4_37_chunk  (cost=0.00..7.70 rows=270 width=152)
         ->  Sort  (cost=1.02..1.02 rows=1 width=20)
               Sort Key: _hyper_3_13_chunk.a, _hyper_3_13_chunk."time" DESC
               ->  Seq Scan on _hyper_3_13_chunk  (cost=0.00..1.01 rows=1 width=20)
         ->  Sort  (cost=1357.70..2032.70 rows=270000 width=20)
               Sort Key: _hyper_3_14_chunk.a, _hyper_3_14_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_14_chunk  (cost=0.03..7.70 rows=270000 width=20)
                     ->  Seq Scan on compress_hyper_4_38_chunk  (cost=0.00..7.70 rows=270 width=152)
         ->  Sort  (cost=1.02..1.02 rows=1 width=20)
               Sort Key: _hyper_3_14_chunk.a, _hyper_3_14_chunk."time" DESC
               ->  Seq Scan on _hyper_3_14_chunk  (cost=0.00..1.01 rows=1 width=20)
         ->  Sort  (cost=1357.70..2032.70 rows=270000 width=20)
               Sort Key: _hyper_3_15_chunk.a, _hyper_3_15_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_15_chunk  (cost=0.03..7.70 rows=270000 width=20)
                     ->  Seq Scan on compress_hyper_4_39_chunk  (cost=0.00..7.70 rows=270 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_16_chunk.a, _hyper_3_16_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_16_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_40_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_17_chunk.a, _hyper_3_17_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_17_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_41_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_18_chunk.a, _hyper_3_18_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_18_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_42_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_19_chunk.a, _hyper_3_19_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_19_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_43_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_20_chunk.a, _hyper_3_20_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_20_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_44_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_21_chunk.a, _hyper_3_21_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_21_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_45_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_22_chunk.a, _hyper_3_22_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_22_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_46_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_23_chunk.a, _hyper_3_23_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_23_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_47_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_24_chunk.a, _hyper_3_24_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_24_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_48_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_25_chunk.a, _hyper_3_25_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_25_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_49_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_26_chunk.a, _hyper_3_26_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_26_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_50_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_27_chunk.a, _hyper_3_27_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_27_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_51_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_28_chunk.a, _hyper_3_28_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_28_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_52_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_29_chunk.a, _hyper_3_29_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_29_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_53_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_30_chunk.a, _hyper_3_30_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_30_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_54_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_31_chunk.a, _hyper_3_31_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_31_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_55_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_32_chunk.a, _hyper_3_32_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_32_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_56_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2608.19..3905.69 rows=519000 width=20)
               Sort Key: _hyper_3_33_chunk.a, _hyper_3_33_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_33_chunk  (cost=0.03..13.19 rows=519000 width=20)
                     ->  Seq Scan on compress_hyper_4_57_chunk  (cost=0.00..13.19 rows=519 width=152)
         ->  Sort  (cost=2553.08..3823.08 rows=508000 width=20)
               Sort Key: _hyper_3_34_chunk.a, _hyper_3_34_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_34_chunk  (cost=0.03..13.08 rows=508000 width=20)
                     ->  Seq Scan on compress_hyper_4_58_chunk  (cost=0.00..13.08 rows=508 width=152)
         ->  Sort  (cost=2553.08..3823.08 rows=508000 width=20)
               Sort Key: _hyper_3_35_chunk.a, _hyper_3_35_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_35_chunk  (cost=0.03..13.08 rows=508000 width=20)
                     ->  Seq Scan on compress_hyper_4_59_chunk  (cost=0.00..13.08 rows=508 width=152)
         ->  Sort  (cost=2553.08..3823.08 rows=508000 width=20)
               Sort Key: _hyper_3_36_chunk.a, _hyper_3_36_chunk."time" DESC
               ->  Custom Scan (DecompressChunk) on _hyper_3_36_chunk  (cost=0.03..13.08 rows=508000 width=20)
                     ->  Seq Scan on compress_hyper_4_60_chunk  (cost=0.00..13.08 rows=508 width=152)
(105 rows)

2. SELECT * FROM space_part ORDER BY time_bucket('2d', time) DESC, a LIMIT 1;

3. SELECT * FROM ht_metrics_partially_compressed ORDER BY time_bucket('1d', time) DESC, device LIMIT 1;

The plan is different before and after:

Before commit 373c556 (PR #5812)

Produced plan: parallel gather merge

``` explain analyze select * FROM ht_metrics_partially_compressed ORDER BY time_bucket('1d', time) DESC, device LIMIT 1; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Limit (cost=254871.03..254871.15 rows=1 width=28) (actual time=3562.470..3563.649 rows=1 loops=1) -> Gather Merge (cost=254871.03..2860589.69 rows=22333172 width=28) (actual time=3562.466..3563.644 rows=1 loops=1) Workers Planned: 2 Workers Launched: 2 -> Sort (cost=253871.01..281787.47 rows=11166586 width=28) (actual time=3547.759..3547.765 rows=1 loops=3) Sort Key: (time_bucket('1 day'::interval, _hyper_1_1_chunk."time")) DESC, _hyper_1_1_chunk.device Sort Method: top-N heapsort Memory: 25kB Worker 0: Sort Method: top-N heapsort Memory: 25kB Worker 1: Sort Method: top-N heapsort Memory: 25kB -> Result (cost=0.07..198038.08 rows=11166586 width=28) (actual time=0.045..2121.360 rows=8928003 loops=3) -> Parallel Append (cost=0.07..58455.75 rows=11166586 width=20) (actual time=0.041..1166.239 rows=8928003 loops=3) -> Custom Scan (DecompressChunk) on _hyper_1_1_chunk (cost=0.07..40.41 rows=541000 width=20) (actual time=0.049..63.924 rows=916000 loops=1) -> Parallel Seq Scan on compress_hyper_2_7_chunk (cost=0.00..40.41 rows=541 width=88) (actual time=0.005..0.158 rows=920 loops=1) -> Custom Scan (DecompressChunk) on _hyper_1_6_chunk (cost=0.07..70.65 rows=965000 width=20) (actual time=0.058..161.449 rows=1636010 loops=1) -> Parallel Seq Scan on compress_hyper_2_12_chunk (cost=0.00..70.65 rows=965 width=88) (actual time=0.006..0.522 rows=1640 loops=1) -> Custom Scan (DecompressChunk) on _hyper_1_2_chunk (cost=0.07..259.47 rows=3547000 width=20) (actual time=0.028..455.151 rows=6028000 loops=1) -> Parallel Seq Scan on compress_hyper_2_8_chunk (cost=0.00..259.47 rows=3547 width=88) (actual time=0.002..1.177 rows=6030 loops=1) -> Custom Scan (DecompressChunk) on _hyper_1_3_chunk (cost=0.07..259.47 rows=3547000 width=20) (actual time=0.049..472.343 rows=6028000 loops=1) -> Parallel Seq Scan on compress_hyper_2_9_chunk (cost=0.00..259.47 rows=3547 width=88) (actual time=0.003..1.177 rows=6030 loops=1) -> Custom Scan (DecompressChunk) on _hyper_1_4_chunk (cost=0.07..259.47 rows=3547000 width=20) (actual time=0.041..164.879 rows=2009333 loops=3) -> Parallel Seq Scan on compress_hyper_2_10_chunk (cost=0.00..259.47 rows=3547 width=88) (actual time=0.003..0.388 rows=2010 loops=3) -> Custom Scan (DecompressChunk) on _hyper_1_5_chunk (cost=0.07..259.47 rows=3547000 width=20) (actual time=0.041..436.828 rows=6028000 loops=1) -> Parallel Seq Scan on compress_hyper_2_11_chunk (cost=0.00..259.47 rows=3547 width=88) (actual time=0.002..0.899 rows=6030 loops=1) -> Parallel Seq Scan on _hyper_1_1_chunk (cost=0.00..245.65 rows=11765 width=20) (actual time=0.003..1.370 rows=20000 loops=1) -> Parallel Seq Scan on _hyper_1_2_chunk (cost=0.00..245.65 rows=11765 width=20) (actual time=0.002..1.379 rows=20000 loops=1) -> Parallel Seq Scan on _hyper_1_3_chunk (cost=0.00..245.65 rows=11765 width=20) (actual time=0.003..1.363 rows=20000 loops=1) -> Parallel Seq Scan on _hyper_1_4_chunk (cost=0.00..245.65 rows=11765 width=20) (actual time=0.002..1.437 rows=20000 loops=1) -> Parallel Seq Scan on _hyper_1_5_chunk (cost=0.00..245.65 rows=11765 width=20) (actual time=0.002..1.370 rows=20000 loops=1) -> Parallel Seq Scan on _hyper_1_6_chunk (cost=0.00..245.65 rows=11765 width=20) (actual time=0.013..2.224 rows=20000 loops=1) Planning Time: 2.372 ms Execution Time: 3563.894 ms (31 rows) ```

After PR #5812

Produced plan: Merge Append

``` explain analyze select * FROM ht_metrics_partially_compressed ORDER BY time_bucket('1d', time) DESC, device LIMIT 1; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Limit (cost=4427.02..4427.05 rows=1 width=28) (actual time=8326.498..8326.510 rows=1 loops=1) -> Merge Append (cost=4427.02..954411.99 rows=26800000 width=28) (actual time=8326.495..8326.506 rows=1 loops=1) Sort Key: (time_bucket('1 day'::interval, _hyper_1_1_chunk."time")) DESC, _hyper_1_1_chunk.device -> Sort (cost=4644.20..6944.20 rows=920000 width=20) (actual time=296.704..296.706 rows=1 loops=1) Sort Key: (time_bucket('1 day'::interval, _hyper_1_1_chunk."time")) DESC, _hyper_1_1_chunk.device Sort Method: top-N heapsort Memory: 25kB -> Result (cost=0.05..44.20 rows=920000 width=20) (actual time=0.052..149.768 rows=916000 loops=1) -> Custom Scan (DecompressChunk) on _hyper_1_1_chunk (cost=0.05..44.20 rows=920000 width=20) (actual time=0.050..63.894 rows=916000 loops=1) -> Seq Scan on compress_hyper_2_7_chunk (cost=0.00..44.20 rows=920 width=88) (actual time=0.012..0.313 rows=920 loops=1) -> Sort (cost=428.00..478.00 rows=20000 width=20) (actual time=6.908..6.908 rows=1 loops=1) Sort Key: (time_bucket('1 day'::interval, _hyper_1_1_chunk."time")) DESC, _hyper_1_1_chunk.device Sort Method: top-N heapsort Memory: 25kB -> Seq Scan on _hyper_1_1_chunk (cost=0.00..328.00 rows=20000 width=20) (actual time=0.016..3.480 rows=20000 loops=1) -> Sort (cost=30434.30..45509.30 rows=6030000 width=20) (actual time=1879.827..1879.828 rows=1 loops=1) Sort Key: (time_bucket('1 day'::interval, _hyper_1_2_chunk."time")) DESC, _hyper_1_2_chunk.device Sort Method: top-N heapsort Memory: 25kB -> Result (cost=0.05..284.30 rows=6030000 width=20) (actual time=0.045..996.643 rows=6028000 loops=1) -> Custom Scan (DecompressChunk) on _hyper_1_2_chunk (cost=0.05..284.30 rows=6030000 width=20) (actual time=0.044..424.280 rows=6028000 loops=1) -> Seq Scan on compress_hyper_2_8_chunk (cost=0.00..284.30 rows=6030 width=88) (actual time=0.010..1.980 rows=6030 loops=1) -> Sort (cost=428.00..478.00 rows=20000 width=20) (actual time=6.826..6.827 rows=1 loops=1) Sort Key: (time_bucket('1 day'::interval, _hyper_1_2_chunk."time")) DESC, _hyper_1_2_chunk.device Sort Method: top-N heapsort Memory: 25kB -> Seq Scan on _hyper_1_2_chunk (cost=0.00..328.00 rows=20000 width=20) (actual time=0.018..3.442 rows=20000 loops=1) -> Sort (cost=30434.30..45509.30 rows=6030000 width=20) (actual time=1863.252..1863.253 rows=1 loops=1) Sort Key: (time_bucket('1 day'::interval, _hyper_1_3_chunk."time")) DESC, _hyper_1_3_chunk.device Sort Method: top-N heapsort Memory: 25kB -> Result (cost=0.05..284.30 rows=6030000 width=20) (actual time=0.042..990.892 rows=6028000 loops=1) -> Custom Scan (DecompressChunk) on _hyper_1_3_chunk (cost=0.05..284.30 rows=6030000 width=20) (actual time=0.040..419.105 rows=6028000 loops=1) -> Seq Scan on compress_hyper_2_9_chunk (cost=0.00..284.30 rows=6030 width=88) (actual time=0.011..1.989 rows=6030 loops=1) -> Sort (cost=428.00..478.00 rows=20000 width=20) (actual time=6.826..6.826 rows=1 loops=1) Sort Key: (time_bucket('1 day'::interval, _hyper_1_3_chunk."time")) DESC, _hyper_1_3_chunk.device Sort Method: top-N heapsort Memory: 25kB -> Seq Scan on _hyper_1_3_chunk (cost=0.00..328.00 rows=20000 width=20) (actual time=0.018..3.422 rows=20000 loops=1) -> Sort (cost=30434.30..45509.30 rows=6030000 width=20) (actual time=1863.692..1863.693 rows=1 loops=1) Sort Key: (time_bucket('1 day'::interval, _hyper_1_4_chunk."time")) DESC, _hyper_1_4_chunk.device Sort Method: top-N heapsort Memory: 25kB -> Result (cost=0.05..284.30 rows=6030000 width=20) (actual time=0.045..991.920 rows=6028000 loops=1) -> Custom Scan (DecompressChunk) on _hyper_1_4_chunk (cost=0.05..284.30 rows=6030000 width=20) (actual time=0.043..420.526 rows=6028000 loops=1) -> Seq Scan on compress_hyper_2_10_chunk (cost=0.00..284.30 rows=6030 width=88) (actual time=0.011..2.544 rows=6030 loops=1) -> Sort (cost=428.00..478.00 rows=20000 width=20) (actual time=7.041..7.041 rows=1 loops=1) Sort Key: (time_bucket('1 day'::interval, _hyper_1_4_chunk."time")) DESC, _hyper_1_4_chunk.device Sort Method: top-N heapsort Memory: 25kB -> Seq Scan on _hyper_1_4_chunk (cost=0.00..328.00 rows=20000 width=20) (actual time=0.043..3.600 rows=20000 loops=1) -> Sort (cost=30434.30..45509.30 rows=6030000 width=20) (actual time=1864.865..1864.866 rows=1 loops=1) Sort Key: (time_bucket('1 day'::interval, _hyper_1_5_chunk."time")) DESC, _hyper_1_5_chunk.device Sort Method: top-N heapsort Memory: 25kB -> Result (cost=0.05..284.30 rows=6030000 width=20) (actual time=0.042..991.010 rows=6028000 loops=1) -> Custom Scan (DecompressChunk) on _hyper_1_5_chunk (cost=0.05..284.30 rows=6030000 width=20) (actual time=0.040..419.622 rows=6028000 loops=1) -> Seq Scan on compress_hyper_2_11_chunk (cost=0.00..284.30 rows=6030 width=88) (actual time=0.010..2.173 rows=6030 loops=1) -> Sort (cost=428.00..478.00 rows=20000 width=20) (actual time=6.883..6.883 rows=1 loops=1) Sort Key: (time_bucket('1 day'::interval, _hyper_1_5_chunk."time")) DESC, _hyper_1_5_chunk.device Sort Method: top-N heapsort Memory: 25kB -> Seq Scan on _hyper_1_5_chunk (cost=0.00..328.00 rows=20000 width=20) (actual time=0.020..3.472 rows=20000 loops=1) -> Sort (cost=8277.40..12377.40 rows=1640000 width=20) (actual time=516.734..516.735 rows=1 loops=1) Sort Key: (time_bucket('1 day'::interval, _hyper_1_6_chunk."time")) DESC, _hyper_1_6_chunk.device Sort Method: top-N heapsort Memory: 25kB -> Result (cost=0.05..77.40 rows=1640000 width=20) (actual time=0.049..268.715 rows=1636010 loops=1) -> Custom Scan (DecompressChunk) on _hyper_1_6_chunk (cost=0.05..77.40 rows=1640000 width=20) (actual time=0.047..114.312 rows=1636010 loops=1) -> Seq Scan on compress_hyper_2_12_chunk (cost=0.00..77.40 rows=1640 width=88) (actual time=0.009..0.533 rows=1640 loops=1) -> Sort (cost=428.00..478.00 rows=20000 width=20) (actual time=6.914..6.914 rows=1 loops=1) Sort Key: (time_bucket('1 day'::interval, _hyper_1_6_chunk."time")) DESC, _hyper_1_6_chunk.device Sort Method: top-N heapsort Memory: 25kB -> Seq Scan on _hyper_1_6_chunk (cost=0.00..328.00 rows=20000 width=20) (actual time=0.016..3.511 rows=20000 loops=1) Planning Time: 4.904 ms Execution Time: 8326.839 ms (65 rows) ```

4. SELECT min(device), max(value) FROM ht_metrics_partially_compressed;

Plan before: Finalize Aggregate -> Gather -> Partial Aggregate -> Parallel Append

explain select min(device), max(value) from ht_metrics_partially_compressed ;
                                                      QUERY PLAN                                                      
----------------------------------------------------------------------------------------------------------------------
 Finalize Aggregate  (cost=115288.90..115288.91 rows=1 width=12)
   ->  Gather  (cost=115288.68..115288.89 rows=2 width=12)
         Workers Planned: 2
         ->  Partial Aggregate  (cost=114288.68..114288.69 rows=1 width=12)
               ->  Parallel Append  (cost=0.07..58455.75 rows=11166586 width=12)
                     ->  Custom Scan (DecompressChunk) on _hyper_1_1_chunk  (cost=0.07..40.41 rows=541000 width=12)
                           ->  Parallel Seq Scan on compress_hyper_2_7_chunk  (cost=0.00..40.41 rows=541 width=40)
                     ->  Custom Scan (DecompressChunk) on _hyper_1_6_chunk  (cost=0.07..70.65 rows=965000 width=12)
                           ->  Parallel Seq Scan on compress_hyper_2_12_chunk  (cost=0.00..70.65 rows=965 width=40)
                     ->  Custom Scan (DecompressChunk) on _hyper_1_2_chunk  (cost=0.07..259.47 rows=3547000 width=12)
                           ->  Parallel Seq Scan on compress_hyper_2_8_chunk  (cost=0.00..259.47 rows=3547 width=40)
                     ->  Custom Scan (DecompressChunk) on _hyper_1_3_chunk  (cost=0.07..259.47 rows=3547000 width=12)
                           ->  Parallel Seq Scan on compress_hyper_2_9_chunk  (cost=0.00..259.47 rows=3547 width=40)
                     ->  Custom Scan (DecompressChunk) on _hyper_1_4_chunk  (cost=0.07..259.47 rows=3547000 width=12)
                           ->  Parallel Seq Scan on compress_hyper_2_10_chunk  (cost=0.00..259.47 rows=3547 width=40)
                     ->  Custom Scan (DecompressChunk) on _hyper_1_5_chunk  (cost=0.07..259.47 rows=3547000 width=12)
                           ->  Parallel Seq Scan on compress_hyper_2_11_chunk  (cost=0.00..259.47 rows=3547 width=40)
                     ->  Parallel Seq Scan on _hyper_1_1_chunk  (cost=0.00..245.65 rows=11765 width=12)
                     ->  Parallel Seq Scan on _hyper_1_2_chunk  (cost=0.00..245.65 rows=11765 width=12)
                     ->  Parallel Seq Scan on _hyper_1_3_chunk  (cost=0.00..245.65 rows=11765 width=12)
                     ->  Parallel Seq Scan on _hyper_1_4_chunk  (cost=0.00..245.65 rows=11765 width=12)
                     ->  Parallel Seq Scan on _hyper_1_5_chunk  (cost=0.00..245.65 rows=11765 width=12)
                     ->  Parallel Seq Scan on _hyper_1_6_chunk  (cost=0.00..245.65 rows=11765 width=12)
(23 rows)

Plan after: Limit on top of Merge Append

explain select min(device), max(value) from ht_metrics_partially_compressed ;
                                                                                   QUERY PLAN                                                                                   
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Result  (cost=6995.99..6996.00 rows=1 width=12)
   InitPlan 1 (returns $0)
     ->  Limit  (cost=2568.93..2568.95 rows=1 width=4)
           ->  Merge Append  (cost=2568.93..620377.90 rows=26800000 width=4)
                 Sort Key: _hyper_1_1_chunk.device
                 ->  Custom Scan (DecompressChunk) on _hyper_1_1_chunk  (cost=0.13..119.00 rows=920000 width=4)
                       Filter: (device IS NOT NULL)
                       ->  Index Scan using compress_hyper_2_7_chunk__compressed_hypertable_2_device__ts_me on compress_hyper_2_7_chunk  (cost=0.28..119.00 rows=920 width=8)
                             Index Cond: (device IS NOT NULL)
                 ->  Sort  (cost=428.00..478.00 rows=20000 width=4)
                       Sort Key: _hyper_1_1_chunk.device
                       ->  Seq Scan on _hyper_1_1_chunk  (cost=0.00..328.00 rows=20000 width=4)
                             Filter: (device IS NOT NULL)
                 ->  Custom Scan (DecompressChunk) on _hyper_1_2_chunk  (cost=0.12..701.50 rows=6030000 width=4)
                       Filter: (device IS NOT NULL)
                       ->  Index Scan using compress_hyper_2_8_chunk__compressed_hypertable_2_device__ts_me on compress_hyper_2_8_chunk  (cost=0.28..701.50 rows=6030 width=8)
                             Index Cond: (device IS NOT NULL)
                 ->  Sort  (cost=428.00..478.00 rows=20000 width=4)
                       Sort Key: _hyper_1_2_chunk.device
                       ->  Seq Scan on _hyper_1_2_chunk  (cost=0.00..328.00 rows=20000 width=4)
                             Filter: (device IS NOT NULL)
                 ->  Custom Scan (DecompressChunk) on _hyper_1_3_chunk  (cost=0.12..701.50 rows=6030000 width=4)
                       Filter: (device IS NOT NULL)
                       ->  Index Scan using compress_hyper_2_9_chunk__compressed_hypertable_2_device__ts_me on compress_hyper_2_9_chunk  (cost=0.28..701.50 rows=6030 width=8)
                             Index Cond: (device IS NOT NULL)
                 ->  Sort  (cost=428.00..478.00 rows=20000 width=4)
                       Sort Key: _hyper_1_3_chunk.device
                       ->  Seq Scan on _hyper_1_3_chunk  (cost=0.00..328.00 rows=20000 width=4)
                             Filter: (device IS NOT NULL)
                 ->  Custom Scan (DecompressChunk) on _hyper_1_4_chunk  (cost=0.12..701.50 rows=6030000 width=4)
                       Filter: (device IS NOT NULL)
                       ->  Index Scan using compress_hyper_2_10_chunk__compressed_hypertable_2_device__ts_m on compress_hyper_2_10_chunk  (cost=0.28..701.50 rows=6030 width=8)
                             Index Cond: (device IS NOT NULL)
                 ->  Sort  (cost=428.00..478.00 rows=20000 width=4)
                       Sort Key: _hyper_1_4_chunk.device
                       ->  Seq Scan on _hyper_1_4_chunk  (cost=0.00..328.00 rows=20000 width=4)
                             Filter: (device IS NOT NULL)
                 ->  Custom Scan (DecompressChunk) on _hyper_1_5_chunk  (cost=0.12..701.50 rows=6030000 width=4)
                       Filter: (device IS NOT NULL)
                       ->  Index Scan using compress_hyper_2_11_chunk__compressed_hypertable_2_device__ts_m on compress_hyper_2_11_chunk  (cost=0.28..701.50 rows=6030 width=8)
                             Index Cond: (device IS NOT NULL)
                 ->  Sort  (cost=428.00..478.00 rows=20000 width=4)
                       Sort Key: _hyper_1_5_chunk.device
                       ->  Seq Scan on _hyper_1_5_chunk  (cost=0.00..328.00 rows=20000 width=4)
                             Filter: (device IS NOT NULL)
                 ->  Custom Scan (DecompressChunk) on _hyper_1_6_chunk  (cost=0.12..199.73 rows=1640000 width=4)
                       Filter: (device IS NOT NULL)
                       ->  Index Scan using compress_hyper_2_12_chunk__compressed_hypertable_2_device__ts_m on compress_hyper_2_12_chunk  (cost=0.28..199.73 rows=1640 width=8)
                             Index Cond: (device IS NOT NULL)
                 ->  Sort  (cost=428.00..478.00 rows=20000 width=4)
                       Sort Key: _hyper_1_6_chunk.device
                       ->  Seq Scan on _hyper_1_6_chunk  (cost=0.00..328.00 rows=20000 width=4)
                             Filter: (device IS NOT NULL)
   InitPlan 2 (returns $1)
     ->  Limit  (cost=4427.02..4427.04 rows=1 width=8)
           ->  Merge Append  (cost=4427.02..619411.99 rows=26800000 width=8)
                 Sort Key: _hyper_1_1_chunk_1.value DESC
                 ->  Sort  (cost=4644.20..6944.20 rows=920000 width=8)
                       Sort Key: _hyper_1_1_chunk_1.value DESC
                       ->  Custom Scan (DecompressChunk) on _hyper_1_1_chunk _hyper_1_1_chunk_1  (cost=0.05..44.20 rows=920000 width=8)
                             Filter: (value IS NOT NULL)
                             ->  Seq Scan on compress_hyper_2_7_chunk compress_hyper_2_7_chunk_1  (cost=0.00..44.20 rows=920 width=36)
                 ->  Sort  (cost=428.00..478.00 rows=20000 width=8)
                       Sort Key: _hyper_1_1_chunk_1.value DESC
                       ->  Seq Scan on _hyper_1_1_chunk _hyper_1_1_chunk_1  (cost=0.00..328.00 rows=20000 width=8)
                             Filter: (value IS NOT NULL)
                 ->  Sort  (cost=30434.30..45509.30 rows=6030000 width=8)
                       Sort Key: _hyper_1_2_chunk_1.value DESC
                       ->  Custom Scan (DecompressChunk) on _hyper_1_2_chunk _hyper_1_2_chunk_1  (cost=0.05..284.30 rows=6030000 width=8)
                             Filter: (value IS NOT NULL)
                             ->  Seq Scan on compress_hyper_2_8_chunk compress_hyper_2_8_chunk_1  (cost=0.00..284.30 rows=6030 width=36)
                 ->  Sort  (cost=428.00..478.00 rows=20000 width=8)
                       Sort Key: _hyper_1_2_chunk_1.value DESC
                       ->  Seq Scan on _hyper_1_2_chunk _hyper_1_2_chunk_1  (cost=0.00..328.00 rows=20000 width=8)
                             Filter: (value IS NOT NULL)
                 ->  Sort  (cost=30434.30..45509.30 rows=6030000 width=8)
                       Sort Key: _hyper_1_3_chunk_1.value DESC
                       ->  Custom Scan (DecompressChunk) on _hyper_1_3_chunk _hyper_1_3_chunk_1  (cost=0.05..284.30 rows=6030000 width=8)
                             Filter: (value IS NOT NULL)
                             ->  Seq Scan on compress_hyper_2_9_chunk compress_hyper_2_9_chunk_1  (cost=0.00..284.30 rows=6030 width=36)
                 ->  Sort  (cost=428.00..478.00 rows=20000 width=8)
                       Sort Key: _hyper_1_3_chunk_1.value DESC
                       ->  Seq Scan on _hyper_1_3_chunk _hyper_1_3_chunk_1  (cost=0.00..328.00 rows=20000 width=8)
                             Filter: (value IS NOT NULL)
                 ->  Sort  (cost=30434.30..45509.30 rows=6030000 width=8)
                       Sort Key: _hyper_1_4_chunk_1.value DESC
                       ->  Custom Scan (DecompressChunk) on _hyper_1_4_chunk _hyper_1_4_chunk_1  (cost=0.05..284.30 rows=6030000 width=8)
                             Filter: (value IS NOT NULL)
                             ->  Seq Scan on compress_hyper_2_10_chunk compress_hyper_2_10_chunk_1  (cost=0.00..284.30 rows=6030 width=36)
                 ->  Sort  (cost=428.00..478.00 rows=20000 width=8)
                       Sort Key: _hyper_1_4_chunk_1.value DESC
                       ->  Seq Scan on _hyper_1_4_chunk _hyper_1_4_chunk_1  (cost=0.00..328.00 rows=20000 width=8)
                             Filter: (value IS NOT NULL)
                 ->  Sort  (cost=30434.30..45509.30 rows=6030000 width=8)
                       Sort Key: _hyper_1_5_chunk_1.value DESC
                       ->  Custom Scan (DecompressChunk) on _hyper_1_5_chunk _hyper_1_5_chunk_1  (cost=0.05..284.30 rows=6030000 width=8)
                             Filter: (value IS NOT NULL)
                             ->  Seq Scan on compress_hyper_2_11_chunk compress_hyper_2_11_chunk_1  (cost=0.00..284.30 rows=6030 width=36)
                 ->  Sort  (cost=428.00..478.00 rows=20000 width=8)
                       Sort Key: _hyper_1_5_chunk_1.value DESC
                       ->  Seq Scan on _hyper_1_5_chunk _hyper_1_5_chunk_1  (cost=0.00..328.00 rows=20000 width=8)
                             Filter: (value IS NOT NULL)
                 ->  Sort  (cost=8277.40..12377.40 rows=1640000 width=8)
                       Sort Key: _hyper_1_6_chunk_1.value DESC
                       ->  Custom Scan (DecompressChunk) on _hyper_1_6_chunk _hyper_1_6_chunk_1  (cost=0.05..77.40 rows=1640000 width=8)
                             Filter: (value IS NOT NULL)
                             ->  Seq Scan on compress_hyper_2_12_chunk compress_hyper_2_12_chunk_1  (cost=0.00..77.40 rows=1640 width=36)
                 ->  Sort  (cost=428.00..478.00 rows=20000 width=8)
                       Sort Key: _hyper_1_6_chunk_1.value DESC
                       ->  Seq Scan on _hyper_1_6_chunk _hyper_1_6_chunk_1  (cost=0.00..328.00 rows=20000 width=8)
                             Filter: (value IS NOT NULL)
(111 rows)

@akuzm akuzm mentioned this issue Jan 19, 2024
Closed
@akuzm
Copy link
Member Author

akuzm commented Dec 9, 2024 •
edited
Loading

To sum up, the problem is that the "limit 1 after sort" plan we have for minmax is chosen over the usual "aggregation" plan, because it has lower cost. Locally for me, the aggregation plan is also slower, so it seems correct. I can still force it by setting enable_sort = off.

The upcoming changes in chunkwise aggregation planning don't seem to fix this: https://grafana.ops.savannah-dev.timescale.com/d/fasYic_4z/compare-akuzm?orgId=1&var-branch=All&var-run1=3968&var-run2=3979&var-threshold=0.02&var-use_historical_thresholds=true&var-threshold_expression=2%20%2A%20percentile_cont%280.90%29&var-exact_suite_version=false&from=now-2d&to=now (tested with #7025)

Also lowering the costs for vectorized aggregation might help. Currenlty we vectorize it only after all the planning is done (post-planning hook), so it's not accounted for in the costs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@horzsolt @akuzm @konskov