bug(PostgreSQL CDC): use PostgreSQL multiple CDC , Only partial data was Ingested

[ediconss]

Describe the bug

CREATE SOURCE pg_mydb  WITH (
    connector = 'postgres-cdc',
    hostname = '192.168.10.34',
    port = '5434',
    username = 'rep',
    password = 'xxxxxx',
    database.name = 'data_warehouse',
    slot.name = 'mydb_slot_dim'
);

CREATE TABLE ods.ods_hl_stock_record
(record_id int8  
, request_id varchar
, record_type int2
, before_change_unit_quantity numeric
, before_change_total_quantity numeric
, unit_quantity numeric
, total_quantity numeric
, batch_id int8
, create_time timestamp with time zone
, create_by int8
, update_time timestamp with time zone
, deleted int2
, note varchar
, is_batch int4
, business_id int8 
, PRIMARY KEY (record_id)
)  FROM pg_mydb TABLE 'ods.ods_hl_stock_record';

in postgresql have 7 million data, but rw only have 4 ten thousand。

bu use single cdc it's ok

CREATE TABLE ods.ods_hl_stock_record
(record_id int8  
, request_id varchar
, record_type int2
, before_change_unit_quantity numeric
, before_change_total_quantity numeric
, unit_quantity numeric
, total_quantity numeric
, batch_id int8
, create_time timestamp with time zone
, create_by int8
, update_time timestamp with time zone
, deleted int2
, note varchar
, is_batch int4
, business_id int8 
, PRIMARY KEY (record_id)
)
WITH (
    connector = 'postgres-cdc',
    hostname = '192.168.10.34',
    port = '5434',
    username = 'rep',
    password = 'xxxxxx',
    database.name = 'data_warehouse',
    schema.name = 'ods',
    table.name = 'ods_hl_stock_record'
);

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

docker
PostgreSQL 9.5-RisingWave-1.6.0

Additional context

No response

[StrikeW]

Hi @ediconss, could you take a look of the log of compute node? There could be some errors that cause data cannot inject correctly.

[ediconss]

not find error

024-01-26 09:08:54 2024-01-26T01:08:54.662348597Z INFO rw-main risingwave_connector_node: The publication covers the table 'ods.ods_hl_stock_record'. thread="Thread-12" class="com.risingwave.connector.source.common.PostgresValidator"
2024-01-26 09:08:54 2024-01-26T01:08:54.747240171Z WARN rw-main risingwave_common::util::resource_util::runtime: failed to get cpu quota in container, use system value instead error=not a number cgroup_version=V1
2024-01-26 09:08:54 2024-01-26T01:08:54.750451078Z WARN rw-main risingwave_common::util::resource_util::runtime: failed to get cpu quota in container, use system value instead error=not a number cgroup_version=V1
2024-01-26 09:08:54 2024-01-26T01:08:54.808612781Z INFO rw-streaming actor{otel.name="Actor 14" actor_id=14 prev_epoch=5832450621440000 curr_epoch=5832450686779392}:executor{otel.name="StreamCdcScan E00002715 (actor 14)" actor_id=14}: risingwave_stream::executor::backfill::cdc::cdc_backfill: start cdc backfill upstream_table_id=4294967294 upstream_table_name="ods.ods_hl_stock_record" initial_binlog_offset=Some(Postgres(PostgresOffset { txid: 960112360, lsn: 10728577877288 })) current_pk_pos=None is_finished=false snapshot_row_count=0 chunk_size=256
2024-01-26 09:08:55 2024-01-26T01:08:55.74768517Z WARN rw-main risingwave_common::util::resource_util::runtime: failed to get cpu quota in container, use system value instead error=not a number cgroup_version=V1
2024-01-26 09:08:55 2024-01-26T01:08:55.748579017Z INFO rw-streaming actor{otel.name="Actor 14" actor_id=14 prev_epoch=5832450686779392 curr_epoch=5832450752315392}:executor{otel.name="StreamCdcScan E00002715 (actor 14)" actor_id=14}: risingwave_stream::executor::backfill::cdc::cdc_backfill: start cdc backfill loop upstream_table_id=4294967294 initial_binlog_offset=Some(Postgres(PostgresOffset { txid: 960112360, lsn: 10728577877288 })) current_pk_pos=None
2024-01-26 09:08:56 2024-01-26T01:08:56.747663429Z WARN rw-main risingwave_common::util::resource_util::runtime: failed to get cpu quota in container, use system value instead error=not a number cgroup_version=V1
2024-01-26 09:08:56 2024-01-26T01:08:56.748545653Z WARN rw-streaming actor{otel.name="Actor 14" actor_id=14 prev_epoch=5832450752315392 curr_epoch=5832450817851392}:executor{otel.name="StreamCdcScan E00002715 (actor 14)" actor_id=14}: risingwave_common::util::chunk_coalesce: dropping non-empty data chunk builder remaining=98

[StrikeW]

Hi @ediconss, there is an internal table to track the progress of snapshot loading, could you also take a look of it? The instruction is here:

To check the progress of backfilling historical data, find the corresponding internal table using the SHOW INTERNAL TABLES command and query from it. For instance, the following SQL query shows the progress of a CDC table named orders_rw.

SELECT * FROM __internal_orders_rw_4002_streamcdcscan_5002;

-[ RECORD 1 ]-----+---------------------------------------------------------------
split_id | 5001
o_orderkey | 4024320
backfill_finished | f
row_count | 1006080
cdc_offset | {"MySql": {"filename": "binlog.000005", "position": 60946679}}

[ediconss]

{
"select * from __internal_ods_hl_stock_record_3_streamcdcscan_1004\r\n": [
	{
		"split_id" : "4294967294",
		"record_id" : 32848,
		"backfill_finished" : false,
		"row_count" : 32768,
		"cdc_offset" : "{\"Postgres\": {\"lsn\": 10732191096576, \"txid\": 960144396}}"
	}
]}

Will we pull the full of data when creating this table?

[StrikeW]

Will we pull the full of data when creating this table?

Yes.

Thank you. Based on this state, the backfilling is not finished (false in the output) and it should continue to load the historical data. And I notice that you are using the v1.6.0 version which actually has some bugs, I am not quite sure whether the bug will cause your problem. But I suggest you can upgrade to the risingwave:git-55ebf241c3639f05b240695b37a60c0bc73870c2 image to have a try, which contains bug fixes.
Btw there are some issues to be addressed, which blocking the release of v1.6.1 version. cc @huangjw806 @lmatz

[ediconss]

pull images access denied for risingwave

[StrikeW]

pull images access denied for risingwave

Try this?
docker pull ghcr.io/risingwavelabs/risingwave:git-55ebf241c3639f05b240695b37a60c0bc73870c2

[ediconss]

okey, i pulled , but still no full data

[StrikeW]

okey, i pulled , but still no full data

Is there any MV built on top of the table? And you can take a look of the grafana dashboard on localhost:3001, and check if the Barrier Latency is normal.

[ediconss]

I found the reason, it's due to network issues. I used an SSH tunnel，

[ediconss]

Thank you for your help！