Support rw_int256 type in Postgres CDC source

[stdrc]

Is your feature request related to a problem? Please describe.

We need to support convert Postgres numeric which actually stores an int256 value to our rw_int256. Current implementation only allow decimal type through rust_decimal crate, and the accuracy is not enough for some user cases.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[lmatz]

what would be the solution?

1. let users use varchar to ingest numeric first and then cast to int256
2. or allow users to map from numeric to int256 directly

[bestmike007]

Thanks @stdrc

This issue was initially discussed on Slack: https://risingwave-community.slack.com/archives/C03BW71523T/p1711419543170449

It'll be even better if it can be configurable to cast numeric fields into int256/uin256(if you have it in the future)/double/varchar.

[bestmike007]

what would be the solution?

@lmatz I vote for 2, which requires less storage

[fuyufjh]

int256/uin256(if you have it in the future)/double/varchar.

Agree. We may also support varchar and double for different use cases.

[bestmike007]

And I haven't tested yet, can you sink rw_int256 fields into postgres numeric fields?

[KeXiangWang]

Hi @bestmike007. According to our discussion today, I've created a hotfix for your case. The code is in branch wkx/hotfix-cdc-source-int256, which is based on the most recently released v1.8. You can get the image with docker pull ghcr.io/risingwavelabs/risingwave:git-cb2561e7d4ab1cffa8e8f06b510df4e404f4b603

Here are some notes worth mentioning:

• The implemented solution adapts the second approach discussed earlier. It requires a simple change when creating a table: modifying the field type from numeric to rw_int256. For example, if the table in pg is create table local_numeric(id int PRIMARY KEY, num numeric), then you should create table in Risingwave with CREATE TABLE local_numeric (id int,num rw_int256,PRIMARY KEY (id)) WITH ( ....
• Currently, this solution is specific to PostgreSQL CDC, resembling a workaround rather than a robust feature, which is why it's not yet PR towards main branch.
• The fix contains a convertion for decimal of rust-postgres to RW's rw_int256. In order to achieve this, I utilized another repo to first convert pg-decimal to str and then convert str to rw_int256. The repo has not been actively maintained for some time, so this fix may be buggy. I'll continue to look for better solution.

Please let us know, if there's any further problem meet.

The issue should be open until we find a good general solution and merged it into main branch.

[xiangjinwu]

Additional note: special values nan/inf/-inf may not work properly: #16395