Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(go/adbc/driver/flightsql): Parsing column metadata in FlightSQL driver #2481

Open
wants to merge 18 commits into
base: main
Choose a base branch
from

Conversation

xborder
Copy link

@xborder xborder commented Jan 22, 2025

  • Fixes #2434
  • Moved driver-agnostic metadata parsing out of shared_utils.go
  • Each driver can now implement its own logic for handling XDBC_ fields using fields metadata
    • Snowflake and BigQuery drivers seem unaffected as they use separate implementations
  • Relocated ToXdbcDataType function to shared_utils.go to resolve circular dependency issues
  • Aligned implementation with documentation by:
    • Ensuring all documented XDBC_ fields are properly fetched
    • Removing undocumented fields (COMMENT, CHARACTER_MAXIMUM_LENGTH, XDBC_SCALE, XDBC_PRECISION)
      *COMMENT, CHARACTER_MAXIMUM_LENGTH seem to be snowflake specific even though it seems snowflake adbc driver has a separate implementation.
    • Couldn't find any documentation regarding XDBC_SCALE, XDBC_PRECISION
  • Restored tests that verify proper metadata field population

@xborder xborder marked this pull request as ready for review January 22, 2025 17:19
@xborder xborder requested a review from zeroshade as a code owner January 22, 2025 17:19
@github-actions github-actions bot added this to the ADBC Libraries 17 milestone Jan 22, 2025
@davidhcoe
Copy link
Contributor

I will take a look.

go/adbc/driver/flightsql/flightsql_adbc_server_test.go Outdated Show resolved Hide resolved

suite.Equal(tt.columnName, columnName, "columnName")
//suite.Equal(tt.ordinalPosition, ordinalPosition, "ordinalPosition")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we losing the ordinal positions for comparison?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. The ordinal position is set based on the order in which the columns are processed unless there's a metadata key ORDINAL_POSITION.
While writing the test I realized that even tho the schema is built always in the same order, the deserialization would (sometimes) cause columns to be in different order. I couldn't exactly understand the issue (if it even is an issue).

To work around this, I ignored the ordinal position (flight doesn't even set ORDINAL_POSITION) and appended the name of the columns being processed to every fields

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

most likely the reason for that is because of the usage of a map. Golang maps do not have a guaranteed order and are in fact randomized when iterating. So you can't rely on the order of a map during iteration.

go/adbc/driver/flightsql/flightsql_connection.go Outdated Show resolved Hide resolved
go/adbc/driver/internal/shared_utils.go Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FLIGHT][GO] CommandGetTables metadata being ignored
3 participants