Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug(sink into table): error during backfilling will cause inconsistency graph #17944

Closed
fuyufjh opened this issue Aug 6, 2024 · 0 comments · Fixed by #17961 or #17960
Closed

bug(sink into table): error during backfilling will cause inconsistency graph #17944

fuyufjh opened this issue Aug 6, 2024 · 0 comments · Fixed by #17961 or #17960
Assignees
Labels
type/bug Something isn't working
Milestone

Comments

@fuyufjh
Copy link
Member

fuyufjh commented Aug 6, 2024

Describe the bug

When using etcd as Meta backend, the bug results in inconsistency graph and the log was pasted in the following section. When using PG as backend, it becomes another error.

Error message/log

ETCD as Meta backend:

2024-08-06T16:36:56.888399+08:00  WARN      rw-standalone-meta bootstrap_recovery{prev_epoch=6927017962635264}:recovery_attempt: risingwave_meta::barrier::recovery: build_actors failed error=merged RPC Error, in worker node 1001: gRPC request to stream service failed: Internal error: actor 1025 not found in info table

PG as Meta backend:

thread 'rw-main' panicked at src/frontend/src/catalog/schema_catalog.rs:163:54:
called `Option::unwrap()` on a `None` value
stack backtrace:
   0: rust_begin_unwind
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:652:5
   1: core::panicking::panic_fmt
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/panicking.rs:72:14
   2: core::panicking::panic
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/panicking.rs:146:5
   3: core::option::unwrap_failed
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/option.rs:1987:5
   4: core::option::Option<T>::unwrap
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/option.rs:935:21
   5: risingwave_frontend::catalog::schema_catalog::SchemaCatalog::drop_table
             at ./src/frontend/src/catalog/schema_catalog.rs:163:25
   6: risingwave_frontend::catalog::root_catalog::Catalog::drop_table
             at ./src/frontend/src/catalog/root_catalog.rs:304:9
   7: risingwave_frontend::observer::observer_manager::FrontendObserverNode::handle_catalog_notification
             at ./src/frontend/src/observer/observer_manager.rs:272:50
   8: <risingwave_frontend::observer::observer_manager::FrontendObserverNode as risingwave_common_service::observer_manager::ObserverState>::handle_notification
             at ./src/frontend/src/observer/observer_manager.rs:70:17
   9: risingwave_common_service::observer_manager::ObserverManager<T,S>::start::{{closure}}::{{closure}}
             at ./src/common/common_service/src/observer_manager.rs:164:25
  10: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/future/future.rs:123:9
  11: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  12: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  13: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  14: tokio::runtime::task::core::Core<T,S>::poll
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
  15: tokio::runtime::task::harness::poll_future::{{closure}}
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  16: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/panic/unwind_safe.rs:272:9
  17: std::panicking::try::do_call
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:559:40
  18: ___rust_try
  19: std::panicking::try
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:523:19
  20: std::panic::catch_unwind
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panic.rs:149:14
  21: tokio::runtime::task::harness::poll_future
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  22: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  23: tokio::runtime::task::harness::Harness<T,S>::poll
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  24: tokio::runtime::task::raw::poll
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
  25: tokio::runtime::task::raw::RawTask::poll
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  26: tokio::runtime::task::LocalNotified<S>::run
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:427:9
  27: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:639:17
  28: tokio::runtime::coop::with_budget
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:107:5
  29: tokio::runtime::coop::budget
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/coop.rs:73:5
  30: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:575:9
  31: tokio::runtime::scheduler::multi_thread::worker::Context::run
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:526:24
  32: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:491:21
  33: tokio::runtime::context::scoped::Scoped<T>::set
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/scoped.rs:40:9
  34: tokio::runtime::context::set_scheduler::{{closure}}
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:26
  35: std::thread::local::LocalKey<T>::try_with
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/thread/local.rs:283:12
  36: std::thread::local::LocalKey<T>::with
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/thread/local.rs:260:9
  37: tokio::runtime::context::set_scheduler
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context.rs:176:9
  38: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:486:9
  39: tokio::runtime::context::runtime::enter_runtime
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/context/runtime.rs:65:16
  40: tokio::runtime::scheduler::multi_thread::worker::run
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:478:5
  41: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/scheduler/multi_thread/worker.rs:447:45
  42: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/task.rs:42:21
  43: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
  44: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:328:17
  45: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/loom/std/unsafe_cell.rs:16:9
  46: tokio::runtime::task::core::Core<T,S>::poll
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/core.rs:317:13
  47: tokio::runtime::task::harness::poll_future::{{closure}}
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:485:19
  48: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/core/src/panic/unwind_safe.rs:272:9
  49: std::panicking::try::do_call
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:559:40
  50: ___rust_try
  51: std::panicking::try
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panicking.rs:523:19
  52: std::panic::catch_unwind
             at /rustc/72fdf913c53dd0e75313ba83e4aa80df3f6e2871/library/std/src/panic.rs:149:14
  53: tokio::runtime::task::harness::poll_future
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:473:18
  54: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:208:27
  55: tokio::runtime::task::harness::Harness<T,S>::poll
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/harness.rs:153:15
  56: tokio::runtime::task::raw::poll
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:271:5
  57: tokio::runtime::task::raw::RawTask::poll
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/raw.rs:201:18
  58: tokio::runtime::task::UnownedTask<S>::run
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/task/mod.rs:464:9
  59: tokio::runtime::blocking::pool::Task::run
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:159:9
  60: tokio::runtime::blocking::pool::Inner::run
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:513:17
  61: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /Users/eric/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.37.0/src/runtime/blocking/pool.rs:471:13

To Reproduce

  1. Create a streaming sink that is possible to fail. Here, we use a PG sink with a non-null constraint.
create table t1 (k int, v varchar, primary key(k));

CREATE SINK pgsink FROM t1 WITH (
    connector = 'jdbc',
    jdbc.url = 'jdbc:postgresql://127.0.0.1:5432/test?user=eric&password=',
    table.name = 't_nonnull',
    type = 'upsert',
    primary_key = 'k'
);

On the target PG

create table t_nonnull (k int, v varchar not null, primary key(k));
  1. Create another table and prepare some data that may cause a failure. Here we use a null-row, which violates the non-null constraint in PG sink
create table t2 (k int, v varchar, primary key(k));

insert into t2 values (2, null);
  1. Finally, create the table sink
create sink t2_to_t1 into t1 from t2;

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment