Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Custom Query Type Format #429

Open
JakkuSakura opened this issue Nov 22, 2024 · 4 comments
Open

Support Custom Query Type Format #429

JakkuSakura opened this issue Nov 22, 2024 · 4 comments
Labels
enhancement New feature or request

Comments

@JakkuSakura
Copy link

Is your feature request related to a problem? Please describe.
I'm trying to integrate 3rd party types into clickhouse-connect. For example, whenever.Instant
I found them useful here:
https://clickhouse.com/docs/en/integrations/python#write-formats
https://clickhouse.com/docs/en/integrations/python#read-formats
The write-format could be related, but I found the query string formatting is its own function and rather limited.

Describe the solution you'd like
It would be best to add set_query_format, and refers to it when constructing query SQL.

Describe alternatives you've considered
I tried monkey-patching. I did patching format_bind_value and format_query_value. but the meanings of them are unclear.

@JakkuSakura JakkuSakura added the enhancement New feature or request label Nov 22, 2024
@genzgd
Copy link
Collaborator

genzgd commented Nov 22, 2024

I don't understand what you're trying to accomplish. As you pointed out, clickhouse-connect does very limited query formatting -- it lets you do "client side binding" of format values within your query. Such binding is generally just for convenience. In general this is a low level library and you are responsible for building your own queries.

Do you have an example of what you think "set_query_format" would do?

@JakkuSakura
Copy link
Author

JakkuSakura commented Nov 22, 2024

For example, I want to use datetime in the query, I can do

db.query('SELECT %s', [datetime.datetime.now()])

it will be formatted as
`SELECT 'YYYY-MM-DD HH:MM:SS``
Note that subsecond values are gone. With DT64Param, datetime contains precision up to microseconds(and I loss nanosecond). Neither client-side binding nor server-side binding is sufficient.

Therefore, I decided to use whenever.Instant, the custom type that supports nanosecond precision. I tried set_read_format('DateTime64', 'int'), but it doesn't work for queries.

If custom types were supported via set_query_format(whenever.Instant, formatter), where formatter is

def format_whenever_instant(value: whenever.Instant, server_tz: tzinfo):
    return format_time(value.to_tz(server_tz) # that formats to YYYY-MM-DD HH:MM:SS.nnnnnnnnn with proper timezone

It will be easily supported just as datetime

db.query('SELECT %s', [whenever.Instant.now()])

On the implementation side:

query_formatter_map = {}
def set_query_format(cls: type, formatter):
     query_formatter_map[cls] = formatter
def format_bind_value(value: Any, server_tz: tzinfo = pytz.UTC, top_level: bool = True):
    if type(value) in query_formatter_map:
        formatter = query_formatter_map[type(value)]
        return formatter(value, server_tz)
    pass # the remaining code as usual
def format_query_value(value: Any, server_tz: tzinfo = pytz.UTC):
    if type(value) in query_formatter_map:
        formatter = query_formatter_map[type(value)]
        return formatter(value, server_tz)
    pass # the remaining code as usual

I'd like to file a PR if you are happy with the change

@JakkuSakura
Copy link
Author

In general this is a low level library and you are responsible for building your own queries

The same is technically true for all queries. But binding of some built-in types is already implemented. set_read/write_format also supports different types in a similar fashion. Supporting custom types will improve user ergonomics without resorting a wrapper.

@genzgd
Copy link
Collaborator

genzgd commented Nov 22, 2024

Okay, now that I understand that this particular issue is limited to query binding I think it's fine if you want to submit a PR. To add customized formatters like this for read/write formats is more complicated and definitely has performance implications, so I'm not in a hurry to add that kind of functionality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants