bug(CREATE SINK INTO): set "STREAMING_PARALLELISM" cause error.

[MrTaozui]

Describe the bug

setting "STREAMING_PARALLELISM" cause error.
simple case:
CREATE TABLE product(
id int,
product_name VARCHAR,
product_color VARCHAR,
primary key(id)
);
insert into product values(1,'p1','blue');
insert into product values(2,'p2','blue');
insert into product values(3,'p3','blue');

CREATE TABLE product_sink_target(
id int,
product_name VARCHAR,
product_color VARCHAR,
primary key(id)
);

set STREAMING_PARALLELISM=2;

CREATE sink product_sink into product_sink_target as select * from product;

error:
ERROR: Failed to run the query

Caused by these errors (recent errors listed first):
1: gRPC request to meta service failed: Internal error
2: merged RPC Error, in worker node 2
3: gRPC request to stream service failed: Internal error
4: actor 13 not found in info table

Then I restart meta nodes and retry with out setting STREAMING_PARALLELISM. Success finally!!!

Error message/log

To Reproduce

apiVersion: risingwave.risingwavelabs.com/v1alpha1
kind: RisingWave
metadata:
name: risingwave-datalab-cluster
namespace: risingwave-operator-system
spec:
metaStore:
postgresql:
enabled: true
host: risingwave-mysql-oss-postgresql
port: 5432
database: risingwave
credentials:
secretName: risingwave-mysql-oss-postgresql
usernameKeyRef: postgres-username
passwordKeyRef: postgres-password

stateStore:
dataDirectory: datalab-hummock02
minio:
endpoint: risingwave-mysql-oss-minio:9000
bucket: risingwave
credentials:
secretName: risingwave-mysql-oss-minio
usernameKeyRef: root-user
passwordKeyRef: root-password
image: xxxxx/risingwave:v2.0.0
components:
meta:
nodeGroups:
- replicas: 2
name: ""
template:
spec:
volumes:
- ephemeral:
volumeClaimTemplate:
metadata:
labels:
type: scratch-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: datalab-alicloud-local-quota
volumeMode: Filesystem
name: heap
volumeMounts:
- mountPath: /heap
name: heap
subPathExpr: $(SERVER_POD_IP)/$(SERVER_POD_NAME)
env:
- name: MALLOC_CONF
value: prof:true,lg_prof_interval:-1,lg_prof_sample:20,prof_prefix:/heap/
- name: RW_HEAP_PROFILING_DIR
value: /heap
- name: SERVER_POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: SERVER_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
resources:
limits:
cpu: 1
memory: 4Gi
requests:
cpu: 1
memory: 4Gi
nodeSelector:
batch-node-group: batch
frontend:
nodeGroups:
- replicas: 2
name: ""
template:
spec:
resources:
limits:
cpu: 1
memory: 8Gi
requests:
cpu: 1
memory: 8Gi
#nodeSelector:
#batch-node-group: batch
compute:
nodeGroups:
- replicas: 3
name: "streaming"
template:
spec:
volumes:
- ephemeral:
volumeClaimTemplate:
metadata:
labels:
type: scratch-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: datalab-alicloud-local-quota
volumeMode: Filesystem
name: heap
volumeMounts:
- mountPath: /heap
name: heap
subPathExpr: $(SERVER_POD_IP)/$(SERVER_POD_NAME)
env:
- name: MALLOC_CONF
value: prof:true,lg_prof_interval:-1,lg_prof_sample:20,prof_prefix:/heap/
- name: RW_HEAP_PROFILING_DIR
value: /heap
- name: RW_COMPUTE_NODE_ROLE
value: streaming
- name: SERVER_POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: SERVER_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
resources:
limits:
cpu: 1
memory: 8Gi # Memory limit will be set to RW_TOTAL_MEMORY_BYTES
requests:
cpu: 1
memory: 8Gi
#nodeSelector:
#batch-node-group: batch
- replicas: 3
name: "serving"
template:
spec:
volumes:
- ephemeral:
volumeClaimTemplate:
metadata:
labels:
type: scratch-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: datalab-alicloud-local-quota
volumeMode: Filesystem
name: heap
volumeMounts:
- mountPath: /heap
name: heap
subPathExpr: $(SERVER_POD_IP)/$(SERVER_POD_NAME)
env:
- name: MALLOC_CONF
value: prof:true,lg_prof_interval:-1,lg_prof_sample:20,prof_prefix:/heap/
- name: RW_HEAP_PROFILING_DIR
value: /heap
- name: RW_COMPUTE_NODE_ROLE
value: serving
- name: SERVER_POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: SERVER_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
resources:
limits:
cpu: 1
memory: 8Gi # Memory limit will be set to RW_TOTAL_MEMORY_BYTES
requests:
cpu: 1
memory: 8Gi
#nodeSelector:
#batch-node-group: batch
compactor:
nodeGroups:
- replicas: 3
name: ""
template:
spec:
volumes:
- ephemeral:
volumeClaimTemplate:
metadata:
labels:
type: scratch-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: datalab-alicloud-local-quota
volumeMode: Filesystem
name: heap
volumeMounts:
- mountPath: /heap
name: heap
subPathExpr: $(SERVER_POD_IP)/$(SERVER_POD_NAME)
env:
- name: MALLOC_CONF
value: prof:true,lg_prof_interval:-1,lg_prof_sample:20,prof_prefix:/heap/
- name: RW_HEAP_PROFILING_DIR
value: /heap
- name: SERVER_POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: SERVER_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
resources:
limits:
cpu: 1
memory: 8Gi
requests:
cpu: 1
memory: 8Gi
#nodeSelector:
#batch-node-group: batch

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

[github-actions]

This issue has been open for 60 days with no activity.

If you think it is still relevant today, and needs to be done in the near future, you can comment to update the status, or just manually remove the no-issue-activity label.

You can also confidently close this issue as not planned to keep our backlog clean.
Don't worry if you think the issue is still valuable to continue in the future.
It's searchable and can be reopened when it's time. 😄