Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compactor container crashed #18054

Closed
PhanSon95 opened this issue Aug 15, 2024 · 2 comments
Closed

Compactor container crashed #18054

PhanSon95 opened this issue Aug 15, 2024 · 2 comments
Assignees
@Li0k
Labels
type/bug Something isn't working
Milestone
release-2.0

Comments

@PhanSon95
Copy link

Describe the bug

I was deployed the Risingwave with helm chart into Kubernetes cluster
Helm chart link: https://github.com/risingwavelabs/helm-charts/tree/main
Version: 0.1.67
Here is the components after I deployed
image
Now, I'm getting crash with compactor POD

Error message/log

launching `compactor`
opentelemetry tracing will be exported to `http://risingwave-distributed-meta-headless.risingwave-distributed.svc:5690/` if enabled
2024-08-15T09:59:33.347021174Z  INFO risingwave_rt::panic_hook: coredump on panic is likely to be enabled limit=18446744073709551615
2024-08-15T09:59:33.349117182Z  INFO risingwave_compactor: Compactor node options: CompactorOpts { listen_addr: "0.0.0.0:6660", advertise_addr: Some("100.64.9.106:6660"), port: None, prometheus_listener_addr: "0.0.0.0:1260", meta_address: LoadBalance(http://risingwave-distributed-meta-headless.risingwave-distributed.svc:5690/), compaction_worker_threads_number: None, config_path: "/risingwave/config/risingwave.toml", metrics_level: None, async_stack_trace: None, heap_profiling_dir: None, compactor_mode: None, proxy_rpc_endpoint: "" }
2024-08-15T09:59:33.349279248Z  INFO risingwave_compactor: meta address: load-balance+http://risingwave-distributed-meta-headless.risingwave-distributed.svc:5690/
2024-08-15T09:59:33.349597669Z  INFO risingwave_compactor:  address is 100.64.9.106:6660
2024-08-15T09:59:33.350058629Z  INFO risingwave_compactor::server: Starting compactor node
2024-08-15T09:59:33.350091371Z  INFO risingwave_compactor::server: > config: RwConfig { server: ServerConfig { heartbeat_interval_ms: 1000, connection_pool_size: 16, metrics_level: Info, telemetry_enabled: true, heap_profiling: HeapProfilingConfig { enable_auto: true, threshold_auto: 0.9, dir: "./" }, grpc_max_reset_stream: 200, unrecognized: {} }, meta: MetaConfig { min_sst_retention_time_sec: 86400, full_gc_interval_sec: 86400, collect_gc_watermark_spin_interval_sec: 5, periodic_compaction_interval_sec: 60, vacuum_interval_sec: 30, vacuum_spin_interval_ms: 10, hummock_version_checkpoint_interval_sec: 30, enable_hummock_data_archive: false, min_delta_log_num_for_hummock_version_checkpoint: 10, max_heartbeat_interval_secs: 60, disable_recovery: false, disable_automatic_parallelism_control: false, parallelism_control_batch_size: 10, parallelism_control_trigger_period_sec: 10, parallelism_control_trigger_first_delay_sec: 30, meta_leader_lease_secs: 30, dangerous_max_idle_secs: None, default_parallelism: Full, enable_compaction_deterministic: false, enable_committed_sst_sanity_check: false, node_num_monitor_interval_sec: 10, backend: Mem, periodic_space_reclaim_compaction_interval_sec: 3600, periodic_ttl_reclaim_compaction_interval_sec: 1800, periodic_tombstone_reclaim_compaction_interval_sec: 600, periodic_split_compact_group_interval_sec: 10, move_table_size_limit: 10737418240, split_group_size_limit: 68719476736, cut_table_size_limit: 1073741824, unrecognized: {}, do_not_config_object_storage_lifecycle: false, partition_vnode_count: 16, table_write_throughput_threshold: 16777216, min_table_split_write_throughput: 4194304, compaction_task_max_heartbeat_interval_secs: 30, compaction_task_max_progress_interval_secs: 600, compaction_config: CompactionConfig { max_bytes_for_level_base: 536870912, max_bytes_for_level_multiplier: 5, max_compaction_bytes: 2147483648, sub_level_max_compaction_bytes: 134217728, level0_tier_compact_file_number: 12, target_file_size_base: 33554432, compaction_filter_mask: 6, max_sub_compaction: 4, level0_stop_write_threshold_sub_level_number: 300, level0_sub_level_compact_level_count: 3, level0_overlapping_sub_level_compact_level_count: 12, max_space_reclaim_bytes: 536870912, level0_max_compact_file_number: 100, tombstone_reclaim_ratio: 40, enable_emergency_picker: true }, hybird_partition_vnode_count: 4, event_log_enabled: true, event_log_channel_max_size: 10, developer: MetaDeveloperConfig { cached_traces_num: 256, cached_traces_memory_limit_bytes: 134217728, enable_trivial_move: true, enable_check_task_level_overlap: false, max_trivial_move_task_count_per_loop: 256, max_get_task_probe_times: 5 }, enable_dropped_column_reclaim: false }, batch: BatchConfig { worker_threads_num: None, developer: BatchDeveloperConfig { connector_message_buffer_size: 16, output_channel_size: 64, chunk_size: 1024 }, distributed_query_limit: None, max_batch_queries_per_frontend_node: None, enable_barrier_read: false, statement_timeout_in_sec: 3600, unrecognized: {}, frontend_compute_runtime_worker_threads: 4, mask_worker_temporary_secs: 30, redact_sql_option_keywords: ["credential", "key", "password", "private", "secret", "token"] }, streaming: StreamingConfig { in_flight_barrier_nums: 10000, actor_runtime_worker_threads_num: None, async_stack_trace: ReleaseVerbose, developer: StreamingDeveloperConfig { enable_executor_row_count: false, connector_message_buffer_size: 16, unsafe_extreme_cache_size: 10, chunk_size: 256, exchange_initial_permits: 2048, exchange_batched_permits: 256, exchange_concurrent_barriers: 1, exchange_concurrent_dispatchers: 0, dml_channel_initial_permits: 32768, hash_agg_max_dirty_groups_heap_size: 67108864, memory_controller_threshold_aggressive: 0.9, memory_controller_threshold_graceful: 0.8, memory_controller_threshold_stable: 0.7, enable_arrangement_backfill: true, high_join_amplification_threshold: 2048 }, unique_user_stream_errors: 10, unsafe_enable_strict_consistency: true, unrecognized: {} }, storage: StorageConfig { share_buffers_sync_parallelism: 1, share_buffer_compaction_worker_threads_number: 4, shared_buffer_capacity_mb: None, shared_buffer_flush_ratio: 0.8, imm_merge_threshold: 0, write_conflict_detection_enabled: false, cache: CacheConfig { block_cache_capacity_mb: None, block_cache_shard_num: None, block_cache_eviction: Lru { high_priority_ratio_in_percent: None }, meta_cache_capacity_mb: None, meta_cache_shard_num: None, meta_cache_eviction: Lru { high_priority_ratio_in_percent: None } }, block_cache_capacity_mb: None, meta_cache_capacity_mb: None, high_priority_ratio_in_percent: None, prefetch_buffer_capacity_mb: None, max_prefetch_block_number: 16, disable_remote_compactor: false, share_buffer_upload_concurrency: 8, compactor_memory_limit_mb: None, compactor_max_task_multiplier: 3.0, compactor_memory_available_proportion: 0.8, sstable_id_remote_fetch_number: 10, data_file_cache: FileCacheConfig { dir: "", capacity_mb: 1024, file_capacity_mb: 64, device_align: 4096, device_io_size: 16384, flushers: 4, reclaimers: 4, recover_concurrency: 8, lfu_window_to_cache_size_ratio: 1, lfu_tiny_lru_capacity_ratio: 0.01, insert_rate_limit_mb: 0, catalog_bits: 6, compression: "none", unrecognized: {} }, meta_file_cache: FileCacheConfig { dir: "", capacity_mb: 1024, file_capacity_mb: 64, device_align: 4096, device_io_size: 16384, flushers: 4, reclaimers: 4, recover_concurrency: 8, lfu_window_to_cache_size_ratio: 1, lfu_tiny_lru_capacity_ratio: 0.01, insert_rate_limit_mb: 0, catalog_bits: 6, compression: "none", unrecognized: {} }, cache_refill: CacheRefillConfig { data_refill_levels: [], timeout_ms: 6000, concurrency: 10, unit: 64, threshold: 0.5, recent_filter_layers: 6, recent_filter_rotate_interval_ms: 10000, unrecognized: {} }, min_sst_size_for_streaming_upload: 33554432, max_sub_compaction: 4, max_concurrent_compaction_task_number: 16, max_preload_wait_time_mill: 0, max_version_pinning_duration_sec: 10800, compactor_max_sst_key_count: 2097152, compact_iter_recreate_timeout_ms: 600000, compactor_max_sst_size: 536870912, enable_fast_compaction: true, check_compaction_result: false, max_preload_io_retry_times: 3, compactor_fast_max_compact_delete_ratio: 40, compactor_fast_max_compact_task_size: 2147483648, compactor_iter_max_io_retry_times: 8, unrecognized: {}, mem_table_spill_threshold: 4194304, object_store: ObjectStoreConfig { object_store_set_atomic_write_dir: false, retry: ObjectStoreRetryConfig { req_backoff_interval_ms: 1000, req_backoff_max_delay_ms: 10000, req_backoff_factor: 2, upload_attempt_timeout_ms: 8000, upload_retry_attempts: 3, streaming_upload_attempt_timeout_ms: 5000, streaming_upload_retry_attempts: 3, read_attempt_timeout_ms: 8000, read_retry_attempts: 3, streaming_read_attempt_timeout_ms: 3000, streaming_read_retry_attempts: 3, metadata_attempt_timeout_ms: 60000, metadata_retry_attempts: 3, delete_attempt_timeout_ms: 5000, delete_retry_attempts: 3, delete_objects_attempt_timeout_ms: 5000, delete_objects_retry_attempts: 3, list_attempt_timeout_ms: 600000, list_retry_attempts: 3 }, s3: S3ObjectStoreConfig { object_store_keepalive_ms: Some(600000), object_store_recv_buffer_size: Some(2097152), object_store_send_buffer_size: None, object_store_nodelay: Some(true), retry_unknown_service_error: false, identity_resolution_timeout_s: 5, developer: S3ObjectStoreDeveloperConfig { object_store_retry_unknown_service_error: false, object_store_retryable_service_error_codes: ["SlowDown", "TooManyRequests"], use_opendal: false } } } }, unrecognized: {} }
2024-08-15T09:59:33.350337967Z  INFO risingwave_compactor::server: > debug assertions: off
2024-08-15T09:59:33.350346543Z  INFO risingwave_compactor::server: > version: 1.9.1 (4fa6c8b9d3afdb76ca9267725626f032aff02369)
2024-08-15T09:59:33.350359137Z  INFO risingwave_rpc_client::meta_client: register meta client using strategy: load-balance+http://risingwave-distributed-meta-headless.risingwave-distributed.svc:5690/
2024-08-15T09:59:33.361894072Z  INFO risingwave_rpc_client::meta_client: Connect to meta server http://risingwave-distributed-meta-headless.risingwave-distributed.svc:5690/ successfully
2024-08-15T09:59:33.372183468Z  INFO risingwave_rpc_client::meta_client: new meta leader http://risingwave-distributed-meta-0.risingwave-distributed-meta-headless.risingwave-distributed.svc:5690/ discovered
2024-08-15T09:59:33.40079399Z  INFO risingwave_compactor::server: Assigned compactor id 256
2024-08-15T09:59:33.419143932Z  INFO risingwave_compactor::server: Compactor non_reserved_memory_bytes 3272854732 meta_cache_capacity_bytes 134217728 compactor_memory_limit_bytes 3138637004 sstable_size_bytes 268435456 block_size_bytes 65536
2024-08-15T09:59:33.422098811Z  INFO risingwave_common_metrics::monitor::connection: monitoring connector hyper_rustls::connector::HttpsConnector<hyper::client::connect::http::HttpConnector> with type S3
2024-08-15T09:59:33.436206835Z  INFO risingwave_rt::logger: opentelemetry tracing disabled
2024-08-15T09:59:33.444373934Z  INFO risingwave_common_heap_profiling::profiler: Auto memory dump is disabled.
2024-08-15T09:59:33.447775706Z  INFO risingwave_compactor: Server listening at 0.0.0.0:6660
2024-08-15T09:59:33.447945437Z  INFO risingwave_common::telemetry::report: start telemetry reporting
2024-08-15T09:59:33.448250482Z  INFO risingwave_common_service::metrics_manager: Prometheus listener for Prometheus is set up on http://0.0.0.0:1260
2024-08-15T09:59:33.503024618Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=0 pull_task_ack=false pending_pull_task_count=4
2024-08-15T09:59:33.563944676Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Ready to handle task: 44480 compact_task_statistics CompactTaskStatistics { total_file_count: 43, total_key_count: 50, total_file_size: 18402, total_uncompressed_file_size: 18015 } compression_algorithm 0  parallelism 1 task_memory_capacity_with_parallelism 93146634, enable fast runner: false, Compaction task id: 44480, group-id: 2, type: Dynamic, target level: 0, target sub level: 6973425938923520 watermark: 6977970254643200, target_file_size: 33554432, splits: 1
Input: 
Level 0 ["[id: 581, obj_id: 581 0KB stale_ratio 0]"] 
Level 0 ["[id: 450, obj_id: 450 0KB stale_ratio 0]", "[id: 435, obj_id: 435 0KB stale_ratio 0]", "[id: 571, obj_id: 571 0KB stale_ratio 0]"] 
Level 0 ["[id: 60, obj_id: 60 0KB stale_ratio 0]", "[id: 434, obj_id: 434 0KB stale_ratio 0]", "[id: 449, obj_id: 449 0KB stale_ratio 0]"] 
Level 0 ["[id: 448, obj_id: 448 0KB stale_ratio 0]", "[id: 420, obj_id: 420 0KB stale_ratio 0]"] 
Level 0 ["[id: 419, obj_id: 419 0KB stale_ratio 0]", "[id: 447, obj_id: 447 0KB stale_ratio 0]"] 
Level 0 ["[id: 446, obj_id: 446 0KB stale_ratio 0]", "[id: 418, obj_id: 418 0KB stale_ratio 0]"] 
Level 0 ["[id: 445, obj_id: 445 0KB stale_ratio 0]", "[id: 59, obj_id: 59 0KB stale_ratio 0]", "[id: 433, obj_id: 433 0KB stale_ratio 0]"] 
Level 0 ["[id: 417, obj_id: 417 0KB stale_ratio 0]", "[id: 444, obj_id: 444 0KB stale_ratio 0]"] 
Level 0 ["[id: 443, obj_id: 443 0KB stale_ratio 0]", "[id: 416, obj_id: 416 0KB stale_ratio 0]"] 
Level 0 ["[id: 58, obj_id: 58 0KB stale_ratio 0]", "[id: 432, obj_id: 432 0KB stale_ratio 0]", "[id: 442, obj_id: 442 0KB stale_ratio 0]", "[id: 415, obj_id: 415 0KB stale_ratio 0]"] 
Level 0 ["[id: 57, obj_id: 57 0KB stale_ratio 0]", "[id: 431, obj_id: 431 0KB stale_ratio 0]", "[id: 441, obj_id: 441 0KB stale_ratio 0]"] 
Level 0 ["[id: 56, obj_id: 56 0KB stale_ratio 0]", "[id: 40, obj_id: 40 0KB stale_ratio 0]", "[id: 410, obj_id: 410 0KB stale_ratio 0]"] 
Level 0 ["[id: 39, obj_id: 39 0KB stale_ratio 0]", "[id: 55, obj_id: 55 0KB stale_ratio 0]", "[id: 409, obj_id: 409 0KB stale_ratio 0]"] 
Level 0 ["[id: 414, obj_id: 414 0KB stale_ratio 0]"] 
Level 0 ["[id: 413, obj_id: 413 0KB stale_ratio 0]"] 
Level 0 ["[id: 408, obj_id: 408 0KB stale_ratio 0]"] 
Level 0 ["[id: 412, obj_id: 412 0KB stale_ratio 0]"] 
Level 0 ["[id: 422, obj_id: 422 0KB stale_ratio 0]"] 
Level 0 ["[id: 421, obj_id: 421 0KB stale_ratio 0]"] 
Level 0 ["[id: 407, obj_id: 407 0KB stale_ratio 0]"] 
Level 0 ["[id: 390, obj_id: 390 0KB stale_ratio 0]"] 
Level 0 ["[id: 406, obj_id: 406 0KB stale_ratio 0]", "[id: 411, obj_id: 411 0KB stale_ratio 0]"] 

2024-08-15T09:59:33.564638115Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Ready to handle task: 44481 compact_task_statistics CompactTaskStatistics { total_file_count: 44, total_key_count: 47, total_file_size: 17277, total_uncompressed_file_size: 16881 } compression_algorithm 0  parallelism 1 task_memory_capacity_with_parallelism 95312397, enable fast runner: false, Compaction task id: 44481, group-id: 2, type: Dynamic, target level: 0, target sub level: 6973684350451712 watermark: 6977970254643200, target_file_size: 33554432, splits: 1
Input: 
Level 0 ["[id: 2895, obj_id: 2895 0KB stale_ratio 0]", "[id: 639, obj_id: 639 0KB stale_ratio 0]"] 
Level 0 ["[id: 638, obj_id: 638 0KB stale_ratio 0]", "[id: 2894, obj_id: 2894 0KB stale_ratio 0]"] 
Level 0 ["[id: 2893, obj_id: 2893 0KB stale_ratio 0]", "[id: 637, obj_id: 637 0KB stale_ratio 0]"] 
Level 0 ["[id: 2892, obj_id: 2892 0KB stale_ratio 0]"] 
Level 0 ["[id: 636, obj_id: 636 0KB stale_ratio 0]", "[id: 2891, obj_id: 2891 0KB stale_ratio 0]"] 
Level 0 ["[id: 578, obj_id: 578 0KB stale_ratio 0]", "[id: 652, obj_id: 652 0KB stale_ratio 0]", "[id: 647, obj_id: 647 0KB stale_ratio 0]"] 
Level 0 ["[id: 646, obj_id: 646 0KB stale_ratio 0]"] 
Level 0 ["[id: 645, obj_id: 645 0KB stale_ratio 0]", "[id: 635, obj_id: 635 0KB stale_ratio 0]"] 
Level 0 ["[id: 644, obj_id: 644 0KB stale_ratio 0]", "[id: 634, obj_id: 634 0KB stale_ratio 0]"] 
Level 0 ["[id: 643, obj_id: 643 0KB stale_ratio 0]", "[id: 577, obj_id: 577 0KB stale_ratio 0]", "[id: 651, obj_id: 651 0KB stale_ratio 0]"] 
Level 0 ["[id: 440, obj_id: 440 0KB stale_ratio 0]", "[id: 642, obj_id: 642 0KB stale_ratio 0]", "[id: 576, obj_id: 576 0KB stale_ratio 0]"] 
Level 0 ["[id: 633, obj_id: 633 0KB stale_ratio 0]", "[id: 641, obj_id: 641 0KB stale_ratio 0]"] 
Level 0 ["[id: 575, obj_id: 575 0KB stale_ratio 0]", "[id: 439, obj_id: 439 0KB stale_ratio 0]", "[id: 590, obj_id: 590 0KB stale_ratio 0]"] 
Level 0 ["[id: 438, obj_id: 438 0KB stale_ratio 0]", "[id: 574, obj_id: 574 0KB stale_ratio 0]", "[id: 589, obj_id: 589 0KB stale_ratio 0]"] 
Level 0 ["[id: 588, obj_id: 588 0KB stale_ratio 0]"] 
Level 0 ["[id: 632, obj_id: 632 0KB stale_ratio 0]", "[id: 587, obj_id: 587 0KB stale_ratio 0]"] 
Level 0 ["[id: 586, obj_id: 586 0KB stale_ratio 0]", "[id: 437, obj_id: 437 0KB stale_ratio 0]", "[id: 573, obj_id: 573 0KB stale_ratio 0]"] 
Level 0 ["[id: 585, obj_id: 585 0KB stale_ratio 0]"] 
Level 0 ["[id: 584, obj_id: 584 0KB stale_ratio 0]", "[id: 631, obj_id: 631 0KB stale_ratio 0]"] 
Level 0 ["[id: 436, obj_id: 436 0KB stale_ratio 0]", "[id: 572, obj_id: 572 0KB stale_ratio 0]", "[id: 583, obj_id: 583 0KB stale_ratio 0]"] 
Level 0 ["[id: 582, obj_id: 582 0KB stale_ratio 0]"] 
Dropped table_ids: {18, 21} 

2024-08-15T09:59:33.589717291Z  INFO lazy_load_identity: aws_smithy_runtime::client::identity::cache::lazy: identity cache miss occurred; added new identity (took Ok(191.411µs))
2024-08-15T09:59:33.662504113Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Ready to handle task: 44482 compact_task_statistics CompactTaskStatistics { total_file_count: 46, total_key_count: 119, total_file_size: 23587, total_uncompressed_file_size: 23074 } compression_algorithm 0  parallelism 1 task_memory_capacity_with_parallelism 99642554, enable fast runner: false, Compaction task id: 44482, group-id: 2, type: Dynamic, target level: 0, target sub level: 6977866216112128 watermark: 6977970254643200, target_file_size: 33554432, splits: 1
Input: 
Level 0 ["[id: 3340, obj_id: 3340 2KB stale_ratio 0]", "[id: 3365, obj_id: 3365 4KB stale_ratio 0]", "[id: 3357, obj_id: 3357 0KB stale_ratio 0]", "[id: 2985, obj_id: 2985 0KB stale_ratio 0]", "[id: 3220, obj_id: 3220 4KB stale_ratio 0]"] 
Level 0 ["[id: 3218, obj_id: 3218 0KB stale_ratio 0]", "[id: 3225, obj_id: 3225 0KB stale_ratio 0]", "[id: 2974, obj_id: 2974 0KB stale_ratio 0]"] 
Level 0 ["[id: 3223, obj_id: 3223 0KB stale_ratio 0]", "[id: 3217, obj_id: 3217 0KB stale_ratio 0]"] 
Level 0 ["[id: 3216, obj_id: 3216 0KB stale_ratio 0]", "[id: 3222, obj_id: 3222 0KB stale_ratio 0]"] 
Level 0 ["[id: 3215, obj_id: 3215 0KB stale_ratio 0]", "[id: 3221, obj_id: 3221 0KB stale_ratio 0]"] 
Level 0 ["[id: 3214, obj_id: 3214 0KB stale_ratio 0]", "[id: 3210, obj_id: 3210 0KB stale_ratio 0]"] 
Level 0 ["[id: 3209, obj_id: 3209 0KB stale_ratio 0]", "[id: 3213, obj_id: 3213 0KB stale_ratio 0]"] 
Level 0 ["[id: 3212, obj_id: 3212 0KB stale_ratio 0]", "[id: 3208, obj_id: 3208 0KB stale_ratio 0]"] 
Level 0 ["[id: 3207, obj_id: 3207 0KB stale_ratio 0]", "[id: 3211, obj_id: 3211 0KB stale_ratio 0]"] 
Level 0 ["[id: 3200, obj_id: 3200 0KB stale_ratio 0]", "[id: 3206, obj_id: 3206 0KB stale_ratio 0]"] 
Level 0 ["[id: 3199, obj_id: 3199 0KB stale_ratio 0]", "[id: 3205, obj_id: 3205 0KB stale_ratio 0]"] 
Level 0 ["[id: 3198, obj_id: 3198 0KB stale_ratio 0]", "[id: 3204, obj_id: 3204 0KB stale_ratio 0]"] 
Level 0 ["[id: 3203, obj_id: 3203 0KB stale_ratio 0]", "[id: 3197, obj_id: 3197 0KB stale_ratio 0]"] 
Level 0 ["[id: 3202, obj_id: 3202 0KB stale_ratio 0]", "[id: 3196, obj_id: 3196 0KB stale_ratio 0]"] 
Level 0 ["[id: 3195, obj_id: 3195 0KB stale_ratio 0]", "[id: 3201, obj_id: 3201 0KB stale_ratio 0]"] 
Level 0 ["[id: 3190, obj_id: 3190 0KB stale_ratio 0]", "[id: 3194, obj_id: 3194 0KB stale_ratio 0]"] 
Level 0 ["[id: 3193, obj_id: 3193 0KB stale_ratio 0]", "[id: 3189, obj_id: 3189 0KB stale_ratio 0]"] 
Level 0 ["[id: 3188, obj_id: 3188 0KB stale_ratio 0]", "[id: 3192, obj_id: 3192 0KB stale_ratio 0]"] 
Level 0 ["[id: 3187, obj_id: 3187 0KB stale_ratio 0]", "[id: 3191, obj_id: 3191 0KB stale_ratio 0]"] 
Level 0 ["[id: 3186, obj_id: 3186 0KB stale_ratio 0]", "[id: 3180, obj_id: 3180 0KB stale_ratio 0]"] 
Level 0 ["[id: 3185, obj_id: 3185 0KB stale_ratio 0]", "[id: 3179, obj_id: 3179 0KB stale_ratio 0]"] 
Dropped table_ids: {20, 21, 17, 19, 18} 

2024-08-15T09:59:33.668219282Z  INFO risingwave_storage::hummock::compactor::compactor_runner: Ready to handle task: 44483 compact_task_statistics CompactTaskStatistics { total_file_count: 43, total_key_count: 48, total_file_size: 11933, total_uncompressed_file_size: 11501 } compression_algorithm 0  parallelism 1 task_memory_capacity_with_parallelism 93144262, enable fast runner: false, Compaction task id: 44483, group-id: 2, type: Dynamic, target level: 0, target sub level: 6977885806723072 watermark: 6977970254643200, target_file_size: 33554432, splits: 1
Input: 
Level 0 ["[id: 3409, obj_id: 3409 0KB stale_ratio 0]", "[id: 3413, obj_id: 3413 0KB stale_ratio 0]"] 
Level 0 ["[id: 3412, obj_id: 3412 0KB stale_ratio 0]", "[id: 3408, obj_id: 3408 0KB stale_ratio 0]"] 
Level 0 ["[id: 3407, obj_id: 3407 0KB stale_ratio 0]", "[id: 3411, obj_id: 3411 0KB stale_ratio 0]"] 
Level 0 ["[id: 3400, obj_id: 3400 0KB stale_ratio 0]", "[id: 3406, obj_id: 3406 0KB stale_ratio 0]"] 
Level 0 ["[id: 3399, obj_id: 3399 0KB stale_ratio 0]", "[id: 3405, obj_id: 3405 0KB stale_ratio 0]"] 
Level 0 ["[id: 3398, obj_id: 3398 0KB stale_ratio 0]", "[id: 3404, obj_id: 3404 0KB stale_ratio 0]"] 
Level 0 ["[id: 3403, obj_id: 3403 0KB stale_ratio 0]", "[id: 3397, obj_id: 3397 0KB stale_ratio 0]"] 
Level 0 ["[id: 3396, obj_id: 3396 0KB stale_ratio 0]", "[id: 3402, obj_id: 3402 0KB stale_ratio 0]"] 
Level 0 ["[id: 3395, obj_id: 3395 0KB stale_ratio 0]", "[id: 3401, obj_id: 3401 0KB stale_ratio 0]"] 
Level 0 ["[id: 3394, obj_id: 3394 0KB stale_ratio 0]", "[id: 3390, obj_id: 3390 0KB stale_ratio 0]"] 
Level 0 ["[id: 3389, obj_id: 3389 0KB stale_ratio 0]", "[id: 3393, obj_id: 3393 0KB stale_ratio 0]"] 
Level 0 ["[id: 3392, obj_id: 3392 0KB stale_ratio 0]", "[id: 3388, obj_id: 3388 0KB stale_ratio 0]"] 
Level 0 ["[id: 3387, obj_id: 3387 0KB stale_ratio 0]", "[id: 3391, obj_id: 3391 0KB stale_ratio 0]"] 
Level 0 ["[id: 3386, obj_id: 3386 0KB stale_ratio 0]"] 
Level 0 ["[id: 3379, obj_id: 3379 0KB stale_ratio 0]", "[id: 3385, obj_id: 3385 0KB stale_ratio 0]"] 
Level 0 ["[id: 3384, obj_id: 3384 0KB stale_ratio 0]", "[id: 3377, obj_id: 3377 1KB stale_ratio 0]"] 
Level 0 ["[id: 3376, obj_id: 3376 0KB stale_ratio 0]", "[id: 3383, obj_id: 3383 0KB stale_ratio 0]"] 
Level 0 ["[id: 3382, obj_id: 3382 0KB stale_ratio 0]", "[id: 3375, obj_id: 3375 0KB stale_ratio 0]"] 
Level 0 ["[id: 3374, obj_id: 3374 0KB stale_ratio 0]", "[id: 3381, obj_id: 3381 0KB stale_ratio 0]"] 
Level 0 ["[id: 3373, obj_id: 3373 0KB stale_ratio 0]", "[id: 3360, obj_id: 3360 0KB stale_ratio 0]"] 
Level 0 ["[id: 3372, obj_id: 3372 0KB stale_ratio 0]", "[id: 3359, obj_id: 3359 0KB stale_ratio 0]"] 
Level 0 ["[id: 3358, obj_id: 3358 0KB stale_ratio 0]", "[id: 3371, obj_id: 3371 0KB stale_ratio 0]"] 
Dropped table_ids: {19, 18, 21, 20, 17} 

thread 'rw-compaction' panicked at /risingwave/src/storage/hummock_sdk/src/key.rs:1070:21:
key UserKey { 14, TableKey { 00000001300000000000000001 } } epoch EpochWithGap(6977885756850176) >= prev epoch EpochWithGap(6977885756850176)
stack backtrace:
thread 'rw-compaction' panicked at /risingwave/src/storage/hummock_sdk/src/key.rs:1070:21:
key UserKey { 14, TableKey { 00000001300000000000000001 } } epoch EpochWithGap(6973604389847040) >= prev epoch EpochWithGap(6973604389847040)
thread 'rw-compaction' panicked at /risingwave/src/storage/hummock_sdk/src/key.rs:1070:21:
key UserKey { 14, TableKey { 00000001300000000000000001 } } epoch EpochWithGap(6973686644277248) >= prev epoch EpochWithGap(6973686644277248)
2024-08-15T09:59:33.87164462Z  INFO risingwave_common::telemetry::report: Telemetry post success, id d9041958-eb38-4f02-a547-f54f973cb288
   0:     0x55eca0564dc1 - std::backtrace_rs::backtrace::libunwind::trace::h15cd4b3a49672107
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1:     0x55eca0564dc1 - std::backtrace_rs::backtrace::trace_unsynchronized::habf3179f943732d3
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55eca0564dc1 - std::sys_common::backtrace::_print_fmt::h9fc8383c08468e91
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:68:5
   3:     0x55eca0564dc1 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::ha34cc821de3866cf
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x55ec94d61fb0 - core::fmt::rt::Argument::fmt::h9851a115188427ff
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/fmt/rt.rs:142:9
   5:     0x55ec94d61fb0 - core::fmt::write::h4e1f28798e26ba6f
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/fmt/mod.rs:1120:17
   6:     0x55eca055ccff - std::io::Write::write_fmt::h6af0ee6dc9ad1b7c
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/io/mod.rs:1810:15
   7:     0x55eca0564b94 - std::sys_common::backtrace::_print::h04f9ec982f9279ed
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x55eca0564b94 - std::sys_common::backtrace::print::hf424c58db2e21d74
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x55eca0566257 - std::panicking::default_hook::{{closure}}::h0b39732fd4f8aec5
  10:     0x55eca0565fbf - std::panicking::default_hook::hef43c7fbe94d1e2d
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:292:9
  11:     0x55ec9dd03cc7 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h8828dc5b0a87a2aa
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/alloc/src/boxed.rs:2029:9
  12:     0x55ec9dd03cc7 - risingwave_rt::panic_hook::set_panic_hook::{{closure}}::ha17f3f053ae20f00
                               at /risingwave/src/utils/runtime/src/panic_hook.rs:25:9
  13:     0x55ec9dd03cc7 - std::panicking::update_hook::{{closure}}::h9afc0d183380dd79
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:234:47
  14:     0x55eca05668ba - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hf509c472fd2756b5
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/alloc/src/boxed.rs:2029:9
  15:     0x55eca05668ba - std::panicking::rust_panic_with_hook::h0d0bb132a2a5244d
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:783:13
  16:     0x55eca0566632 - std::panicking::begin_panic_handler::{{closure}}::h84ac5cb90d28af24
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:657:13
  17:     0x55eca05652b6 - std::sys_common::backtrace::__rust_end_short_backtrace::hc04745579118525a
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:171:18
  18:     0x55eca05663a0 - rust_begin_unwind
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:645:5
  19:     0x55ec94d5e695 - core::panicking::panic_fmt::h1863fa40d79fa655
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/panicking.rs:72:14
  20:     0x55ec9e1787c2 - risingwave_hummock_sdk::key::FullKeyTracker<T,_>::observe_multi_version::h4fb28d7ad6c04794
                               at /risingwave/src/storage/hummock_sdk/src/key.rs:1070:21
  21:     0x55ec9e1787c2 - risingwave_hummock_sdk::key::FullKeyTracker<T,_>::observe::h702b9de8343b5229
                               at /risingwave/src/storage/hummock_sdk/src/key.rs:1019:9
  22:     0x55ec9e0232a4 - risingwave_storage::hummock::compactor::compactor_runner::compact_and_build_sst::{{closure}}::hf250ad3c9d2b3590
                               at /risingwave/src/storage/src/hummock/compactor/compactor_runner.rs:661:31
  23:     0x55ec9e030c13 - <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll::hab2b07835863922c
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/future.rs:119:15
  24:     0x55ec9e030c13 - risingwave_storage::hummock::compactor::Compactor::compact_key_range_impl::{{closure}}::hb9cd77f75de76b5f
                               at /risingwave/src/storage/src/hummock/compactor/mod.rs:278:10
  25:     0x55ec9dffe571 - <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll::h9a97c353f9f889b9
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/future.rs:119:15
  26:     0x55ec9e0c16d0 - risingwave_storage::hummock::compactor::Compactor::compact_key_range::{{closure}}::hf44e2df70c1dd492
                               at /risingwave/src/storage/src/hummock/compactor/mod.rs:172:18
  27:     0x55ec9e0c16d0 - risingwave_storage::hummock::compactor::compactor_runner::CompactorRunner::run::{{closure}}::hd6d407141bbaef18
                               at /risingwave/src/storage/src/hummock/compactor/compactor_runner.rs:152:14
  28:     0x55ec9e0c16d0 - risingwave_storage::hummock::compactor::compactor_runner::compact::{{closure}}::{{closure}}::he5848e5f2457646a
                               at /risingwave/src/storage/src/hummock/compactor/compactor_runner.rs:488:18
  29:     0x55ec9e0c16d0 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}::h3214b8827b7a66e1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:391:31
  30:     0x55ec9e0c16d0 - tokio::task::task_local::LocalKey<T>::scope_inner::h158ea600428b15ad
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:217:19
  31:     0x55ec9e0c16d0 - <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::h9cbcd010eb37708f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/task/task_local.rs:387:19
  32:     0x55ec9dfc7a6c - await_tree::root::TreeRoot::instrument::{{closure}}::h697e32eeaaf70b43
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/root.rs:43:34
  33:     0x55ec9dfc7a6c - <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll::h16ebe3371a2b272e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/either.rs:109:32
  34:     0x55ec9df372c2 - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h734a0508169a8b2b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  35:     0x55ec9e297229 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::ha5d96e01993c5ca0
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  36:     0x55ec9e297229 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h9f46e9c79dcada4b
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  37:     0x55ec9e297229 - tokio::runtime::task::core::Core<T,S>::poll::h1254b7e27ed8183e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
  38:     0x55ec9e297229 - tokio::runtime::task::harness::poll_future::{{closure}}::h04347a9b5c30c739
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  39:     0x55ec9e297229 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hdbfe2c685772c279
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/panic/unwind_safe.rs:272:9
  40:     0x55ec9e297229 - std::panicking::try::do_call::hfb2d685bab69e816
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:552:40
  41:     0x55ec9e297229 - std::panicking::try::h351be2435fa78179
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:516:19
  42:     0x55ec9e06447f - std::panic::catch_unwind::h4b1cb8e98070defb
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panic.rs:142:14
  43:     0x55ec9e06447f - tokio::runtime::task::harness::poll_future::he63a4cf1d9fbd86a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  44:     0x55ec9e06447f - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h1c1f2891f30dce53
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  45:     0x55ec9e06447f - tokio::runtime::task::harness::Harness<T,S>::poll::ha55e684712d55d6a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  46:     0x55eca05daa2b - tokio::runtime::task::raw::RawTask::poll::hfb7a062970e27ae3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  47:     0x55eca05daa2b - tokio::runtime::task::LocalNotified<S>::run::h53dcd6c3ef3dab2f
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9
  48:     0x55eca05daa2b - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}::h12af89e24c45974d
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:639:22
  49:     0x55eca05daa2b - tokio::runtime::coop::with_budget::h10e8d61d9d0f2201
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5
  50:     0x55eca05daa2b - tokio::runtime::coop::budget::hbbe53e7575507e74
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5
  51:     0x55eca05daa2b - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h7ea88addd9b9a29c
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
2024-08-15T09:59:34.449430099Z  INFO risingwave_storage::hummock::compactor: running_parallelism_count=4 pull_task_ack=false pending_pull_task_count=4
  52:     0x55eca05d6186 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h0826c61578df103a
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
  53:     0x55eca05d6186 - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}::h48b9e58dd82578e1
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
  54:     0x55eca05d6186 - tokio::runtime::context::scoped::Scoped<T>::set::h4ba6c1c713606dc4
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9
  55:     0x55eca0618a5a - tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::h0a9594d5749b3e00
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
  56:     0x55eca0618a5a - tokio::runtime::context::runtime::enter_runtime::hafc5f0a3a949624e
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16
  57:     0x55eca05da70c - tokio::runtime::scheduler::multi_thread::worker::run::hd96a208dbc33ea55
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
  58:     0x55eca05f2923 - tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}::h764ef05798ac0f87
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
  59:     0x55eca05f2923 - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::hbc4ed9764a6d00d6
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21
  60:     0x55eca061044b - <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h4bdcdc33cc4fde28
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  61:     0x55eca05e3642 - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::h4b6a9daf9dcc7eb2
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  62:     0x55eca05e3642 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h8951cfb8e4f59ca6
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  63:     0x55eca05e3642 - tokio::runtime::task::core::Core<T,S>::poll::h8ae04c6610623127
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
  64:     0x55eca0601e8a - tokio::runtime::task::harness::poll_future::{{closure}}::hddc2fe7b711bba62
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  65:     0x55eca0601e8a - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hff473cd04962ee51
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/panic/unwind_safe.rs:272:9
  66:     0x55eca0601e8a - std::panicking::try::do_call::h1b78d6a6fa31c65a
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:552:40
  67:     0x55eca0601e8a - std::panicking::try::hf39932f850f35d42
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:516:19
  68:     0x55eca0601e8a - std::panic::catch_unwind::h57b42ee16aa4beb4
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panic.rs:142:14
  69:     0x55eca0601e8a - tokio::runtime::task::harness::poll_future::h6b90b6d144ef7122
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  70:     0x55eca0601e8a - tokio::runtime::task::harness::Harness<T,S>::poll_inner::h7e02a514be9cb697
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  71:     0x55eca0601e8a - tokio::runtime::task::harness::Harness<T,S>::poll::h2ebd645d0362b4c2
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  72:     0x55eca05e932e - tokio::runtime::task::raw::RawTask::poll::hfb7a062970e27ae3
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  73:     0x55eca05e932e - tokio::runtime::task::UnownedTask<S>::run::hb601ab1a8078afdf
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9
  74:     0x55eca05e932e - tokio::runtime::blocking::pool::Task::run::h79fb6432f633d889
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9
  75:     0x55eca05e932e - tokio::runtime::blocking::pool::Inner::run::hd57a9e2eb9988367
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17
  76:     0x55eca05f0c70 - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::h39928988f3181359
                               at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13
  77:     0x55eca05f0c70 - std::sys_common::backtrace::__rust_begin_short_backtrace::hb0737151ecb489b7
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:155:18
  78:     0x55eca05ebaed - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::h4f787999cb4b4735
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/thread/mod.rs:529:17
  79:     0x55eca05ebaed - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h3a6e7af7bbfeb491
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/panic/unwind_safe.rs:272:9
  80:     0x55eca05ebaed - std::panicking::try::do_call::he1ac4f4f47ae453f
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:552:40
  81:     0x55eca05ebaed - std::panicking::try::h3afe60011210868a
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panicking.rs:516:19
  82:     0x55eca05ebaed - std::panic::catch_unwind::h895340cbf7b1cbc0
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/panic.rs:142:14
  83:     0x55eca05ebaed - std::thread::Builder::spawn_unchecked_::{{closure}}::h1ba0006455dbee8c
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/thread/mod.rs:528:30
  84:     0x55eca05ebaed - core::ops::function::FnOnce::call_once{{vtable.shim}}::h64d9d397094814ba
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/ops/function.rs:250:5
  85:     0x55eca056ca65 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h6f4c302ba28cbed5
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/alloc/src/boxed.rs:2015:9
  86:     0x55eca056ca65 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::ha560e496ef3d1d36
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/alloc/src/boxed.rs:2015:9
  87:     0x55eca056ca65 - std::sys::unix::thread::Thread::new::thread_start::hd14be232f43d096b
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys/unix/thread.rs:108:17
  88:     0x7f36f1eebac3 - start_thread
                               at ./nptl/pthread_create.c:442:8
  89:     0x7f36f1f7d850 - __GI___clone3
                               at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
  90:                0x0 - <unknown>
stack backtrace:
   0:     0x55eca0564dc1 - std::backtrace_rs::backtrace::libunwind::trace::h15cd4b3a49672107
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1:     0x55eca0564dc1 - std::backtrace_rs::backtrace::trace_unsynchronized::habf3179f943732d3
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55eca0564dc1 - std::sys_common::backtrace::_print_fmt::h9fc8383c08468e91
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:68:5
   3:     0x55eca0564dc1 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::ha34cc821de3866cf
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x55ec94d61fb0 - core::fmt::rt::Argument::fmt::h9851a115188427ff
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/fmt/rt.rs:142:9
   5:     0x55ec94d61fb0 - core::fmt::write::h4e1f28798e26ba6f
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/core/src/fmt/mod.rs:1120:17
   6:     0x55eca055ccff - std::io::Write::write_fmt::h6af0ee6dc9ad1b7c
                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/io/mod.rs:1810:15
   7:     0x55eca0564b94 - std::sys_common::backtrace::_print::h04f9ec982f9279ed


*** await tree context of current task ***

Compaction Task 44482 Split 0  [956.012ms]
  compact [956.012ms]
    compact_and_build_sst [956.012ms]  <== current


                               at /rustc/e4c626dd9a17a23270bf8e7158e59cf2b9c04840/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x55eca0564b94 - std::sys_common::backtrace::print::hf424c58db2e21d74

To Reproduce

No response

Expected behavior

No panic anymore

How did you deploy RisingWave?

via Helm Chart, here is the values
Command to install:

helm upgrade --install risingwave-distributed risingwavelabs/risingwave -f values-distributed-stg.yaml -n risingwave-distributed

values-custom.yml -> Here is the content of file: https://ctxt.io/2/AADY0PElFg

The version of RisingWave

dev=> select version();
                                    version
--------------------------------------------------------------------------------
 PostgreSQL 13.14.0-RisingWave-1.9.1 (4fa6c8b9d3afdb76ca9267725626f032aff02369)

Additional context

No response
@PhanSon95 PhanSon95 added the type/bug Something isn't working label Aug 15, 2024
@github-actions github-actions bot added this to the release-2.0 milestone Aug 15, 2024
@PhanSon95 PhanSon95 mentioned this issue Aug 15, 2024
Merged
9 tasks
@fuyufjh
Copy link
Member

fuyufjh commented Aug 20, 2024

Another user from the community reports the same issue today. Let me set it to high-prority.

@fuyufjh
Copy link
Member

fuyufjh commented Aug 27, 2024

Same issue as #16235. Close for duplication.

@fuyufjh fuyufjh closed this as not planned Won't fix, can't repro, duplicate, stale Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Li0k Li0k

Labels
type/bug Something isn't working
Projects
None yet
Milestone
release-2.0
Development

No branches or pull requests
3 participants
@fuyufjh @Li0k @PhanSon95