Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: panic when update passing a force append only sink into the table #14157

Closed
Tracked by #14063
st1page opened this issue Dec 22, 2023 · 1 comment · Fixed by #14237
Closed
Tracked by #14063

bug: panic when update passing a force append only sink into the table #14157

st1page opened this issue Dec 22, 2023 · 1 comment · Fixed by #14237
Assignees
@st1page
Milestone
release-1.6

Comments

@st1page
Copy link
Contributor

st1page commented Dec 22, 2023

reproduce

create table s(k int, v int);
create table t(k int, v int);
SET RW_IMPLICIT_FLUSH TO true;
create sink sk into t from s FORMAT PLAIN ENCODE NATIVE(
    force_append_only='true'
);
insert into s values (1,1);
update s set v = 2 where k = 1;

dev=> create table s(k int, v int);
CREATE_TABLE
dev=> create table t(k int, v int) APPEND ONLY;
NOTICE:  APPEND ONLY TABLE is currently an experimental feature.
CREATE_TABLE
dev=> SET RW_IMPLICIT_FLUSH TO true;
SET_VARIABLE
dev=> create sink sk into t from s FORMAT PLAIN ENCODE NATIVE(
    force_append_only='true'
);
CREATE_SINK
dev=> insert into s values (1,1);
INSERT 0 1
dev=> update s set v = 2 where k = 1;
ERROR:  Failed to run the query

Caused by these errors (recent errors listed first):
  1: gRPC request to meta service failed: Internal error
  2: gRPC request failed: Unknown error
  3: transport error
  4: connection error
  5: connection reset

CN log

thread 'risingwave-streaming-actor' panicked at src/stream/src/executor/row_id_gen.rs:76:65:
called `Option::unwrap()` on a `None` value
stack backtrace:
2023-12-22T17:50:37.319102269+08:00  INFO risingwave_storage::hummock::compactor: running_task_count=0 pull_task_ack=false pending_pull_task_count=2
   0: rust_begin_unwind
             at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/std/src/panicking.rs:597:5
   1: core::panicking::panic_fmt
             at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/panicking.rs:72:14
   2: core::panicking::panic
             at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/panicking.rs:127:5
   3: core::option::Option<T>::unwrap
             at /rustc/249624b5043013d18c00f0401ca431c1a6baa8cd/library/core/src/option.rs:925:21
   4: risingwave_stream::executor::row_id_gen::RowIdGenExecutor::gen_row_id_column_by_op
             at ./src/stream/src/executor/row_id_gen.rs:76:59
   5: risingwave_stream::executor::row_id_gen::RowIdGenExecutor::execute_inner::{{closure}}
             at ./src/stream/src/executor/row_id_gen.rs:100:25
   6: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
   7: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
   8: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
   9: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  10: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  11: <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/await-tree-0.1.1/src/future.rs:124:23
  12: risingwave_stream::executor::wrapper::trace::instrument_await_tree::{{closure}}
             at ./src/stream/src/executor/wrapper/trace.rs:124:10
  13: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
  14: risingwave_stream::executor::wrapper::schema_check::schema_check::{{closure}}
             at ./src/stream/src/executor/wrapper/schema_check.rs:24:1
  15: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
  16: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
  17: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  18: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  19: risingwave_stream::executor::wrapper::epoch_check::epoch_check::{{closure}}
             at ./src/stream/src/executor/wrapper/epoch_check.rs:31:44
  20: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
  21: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
  22: <S as futures_core::stream::TryStream>::try_poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:196:9
  23: futures_util::stream::try_stream::TryStreamExt::try_poll_next_unpin
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/try_stream/mod.rs:1068:9
  24: <futures_util::stream::try_stream::try_next::TryNext<St> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/try_stream/try_next.rs:32:9
  25: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/tokio-1.32.0/src/task/task_local.rs:347:31
  26: tokio::task::task_local::LocalKey<T>::scope_inner
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/tokio-1.32.0/src/task/task_local.rs:217:19
  27: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/tokio-1.32.0/src/task/task_local.rs:343:19
  28: risingwave_common::util::epoch::task_local::scope::{{closure}}
             at ./src/common/src/util/epoch.rs:210:47
  29: risingwave_stream::executor::wrapper::epoch_provide::epoch_provide::{{closure}}
             at ./src/stream/src/executor/wrapper/epoch_provide.rs:31:59
  30: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
  31: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
  32: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  33: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  34: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/tracing-0.1.37/src/instrument.rs:272:9
  35: risingwave_stream::executor::wrapper::trace::trace::{{closure}}
             at ./src/stream/src/executor/wrapper/trace.rs:50:69
  36: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
  37: risingwave_stream::executor::wrapper::update_check::update_check::{{closure}}
             at ./src/stream/src/executor/wrapper/update_check.rs:27:1
  38: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
  39: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
  40: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
  41: risingwave_stream::executor::mview::materialize::MaterializeExecutor<S,SD>::execute_inner::{{closure}}
             at ./src/stream/src/executor/mview/materialize.rs:113:5
  42: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
  43: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
  44: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
  45: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  46: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  47: <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/await-tree-0.1.1/src/future.rs:124:23
  48: risingwave_stream::executor::wrapper::trace::instrument_await_tree::{{closure}}
             at ./src/stream/src/executor/wrapper/trace.rs:124:10
  49: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
  50: risingwave_stream::executor::wrapper::schema_check::schema_check::{{closure}}
             at ./src/stream/src/executor/wrapper/schema_check.rs:24:1
  51: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
  52: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
  53: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  54: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  55: risingwave_stream::executor::wrapper::epoch_check::epoch_check::{{closure}}
             at ./src/stream/src/executor/wrapper/epoch_check.rs:31:44
  56: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
  57: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
  58: <S as futures_core::stream::TryStream>::try_poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:196:9
  59: futures_util::stream::try_stream::TryStreamExt::try_poll_next_unpin
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/try_stream/mod.rs:1068:9
  60: <futures_util::stream::try_stream::try_next::TryNext<St> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/try_stream/try_next.rs:32:9
  61: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/tokio-1.32.0/src/task/task_local.rs:347:31
  62: tokio::task::task_local::LocalKey<T>::scope_inner
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/tokio-1.32.0/src/task/task_local.rs:217:19
  63: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/tokio-1.32.0/src/task/task_local.rs:343:19
  64: risingwave_common::util::epoch::task_local::scope::{{closure}}
             at ./src/common/src/util/epoch.rs:210:47
  65: risingwave_stream::executor::wrapper::epoch_provide::epoch_provide::{{closure}}
             at ./src/stream/src/executor/wrapper/epoch_provide.rs:31:59
  66: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
  67: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
  68: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  69: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-util-0.3.28/src/stream/stream/next.rs:32:9
  70: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/tracing-0.1.37/src/instrument.rs:272:9
  71: risingwave_stream::executor::wrapper::trace::trace::{{closure}}
             at ./src/stream/src/executor/wrapper/trace.rs:50:69
  72: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
  73: risingwave_stream::executor::wrapper::update_check::update_check::{{closure}}
             at ./src/stream/src/executor/wrapper/update_check.rs:27:1
  74: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
  75: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
  76: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
  77: <risingwave_stream::executor::dispatch::DispatchExecutor as risingwave_stream::executor::StreamConsumer>::execute::{{closure}}
             at ./src/stream/src/executor/dispatch.rs:382:9
  78: <futures_async_stream::try_stream::GenTryStream<G> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-async-stream-0.2.9/src/lib.rs:506:33
  79: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:120:9
  80: <&mut S as futures_core::stream::Stream>::poll_next
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/futures-core-0.3.28/src/stream.rs:104:9
  81: <tokio_stream::stream_ext::next::Next<St> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/git/checkouts/tokio-968c02b7a1a41bea/fe39bb8/tokio-stream/src/stream_ext/next.rs:42:9
  82: <tokio_stream::stream_ext::try_next::TryNext<St> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/git/checkouts/tokio-968c02b7a1a41bea/fe39bb8/tokio-stream/src/stream_ext/try_next.rs:43:9
  83: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/tracing-0.1.37/src/instrument.rs:272:9
  84: <await_tree::future::Instrumented<F,_> as core::future::future::Future>::poll
             at /home/stonepage/.cargo/registry/src/mirrors.ustc.edu.cn-61ef6e0cd06fb9b8/await-tree-0.1.1/src/future.rs:124:23
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.


*** await tree context of current task ***

Actor 24: `CREATE TABLE t (k INT, v INT)` [946.667ms]
  Epoch 5636321223376896 [903.333ms]
    Materialize 1800000005 (actor 24) [903.333ms]
      RowIdGen 1800000004 (actor 24) [903.333ms]  <== current


Fri Dec 22 09:50:39 AM UTC 2023 [risedev]: Program exited with 134
@st1page st1page self-assigned this Dec 22, 2023
@github-actions github-actions bot added this to the release-1.6 milestone Dec 22, 2023
@st1page st1page mentioned this issue Dec 22, 2023
Open
13 tasks
@shanicky
Copy link
Contributor

The reason here is that during the update, the StreamChunk arriving at the row id gen executor carries a hidden delete, and compact this chunk will no longer cause a panic

@st1page st1page mentioned this issue Dec 27, 2023
Merged
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@st1page st1page

Labels
None yet
Projects
None yet
Milestone
release-1.6
Development

Successfully merging a pull request may close this issue.

fix(stream): rowIdGen executor handle invisible delete record risingwavelabs/risingwave
2 participants
@shanicky @st1page