Skip to content

Commit

Permalink
Clickhouse raw table: merge tree, different order by (#2104)
Browse files Browse the repository at this point in the history
Raw table DDL is now featuring merge tree with order by of batch ID and
destination table name:

```sql

CREATE TABLE IF NOT EXISTS raw (
		_peerdb_uid String NOT NULL,
		_peerdb_timestamp Int64 NOT NULL,
		_peerdb_destination_table_name String NOT NULL,
		_peerdb_data String NOT NULL,
		_peerdb_record_type Int NOT NULL,
		_peerdb_match_data String,
		_peerdb_batch_id Int,
		_peerdb_unchanged_toast_columns String
) ENGINE = MergeTree() ORDER BY (_peerdb_batch_id, _peerdb_destination_table_name);
```

Functionally tested and normalize IIS seems to be slightly faster than
what it is currently
  • Loading branch information
Amogh-Bharadwaj authored Sep 26, 2024
1 parent f2c99f9 commit ebda937
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion flow/connectors/clickhouse/cdc.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ func (c *ClickHouseConnector) CreateRawTable(ctx context.Context, req *protos.Cr
_peerdb_match_data String,
_peerdb_batch_id Int,
_peerdb_unchanged_toast_columns String
) ENGINE = ReplacingMergeTree ORDER BY _peerdb_uid;`
) ENGINE = MergeTree() ORDER BY (_peerdb_batch_id, _peerdb_destination_table_name);`

err := c.execWithLogging(ctx,
fmt.Sprintf(createRawTableSQL, rawTableName))
Expand Down

0 comments on commit ebda937

Please sign in to comment.