Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Duration type #363

Merged
merged 2 commits into from
Apr 1, 2022

Conversation

Ponewor
Copy link
Contributor

@Ponewor Ponewor commented Dec 5, 2021

Fixes: #320

This works great with Scylla but for some reason it fails with Cassandra (at least with 4.0.1). The column type code that Cassandra is sending for Duration is 0 instead of 0x15 and it is followed with type's name: "org.apache.cassandra.db.marshal.DurationType". How should we deal with it?

Apart from that (and a few missing tests or docstrings), is there anything serious missing here?

Pre-review checklist

  • I have split my patch into logically separate commits.
  • All commit messages clearly explain what they change and why.
  • I added relevant tests for new features and bug fixes.
  • All commits compile, pass static checks and pass test.
  • PR description sums up the changes and reasons why they should be introduced.
  • I added appropriate Fixes: annotations to PR description.

@Ponewor Ponewor force-pushed the 320-support-for-duration-cql-type branch 2 times, most recently from e85dcdc to a08989a Compare December 5, 2021 04:22
@psarna
Copy link
Contributor

psarna commented Dec 5, 2021

@Ponewor I have no idea why Cassandra decided to encode duration as a custom type instead of a native type, but it is legal in CQL v4. Here's an excerpt from https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v4.spec:

The option for <type> is either a native
          type (see below), in which case the option has no value, or a
          'custom' type, in which case the value is a [string] representing
          the fully qualified class name of the type represented. Valid option
          ids are:
            0x0000    Custom: the value is a [string], see above.
            0x0001    Ascii
            0x0002    Bigint
            0x0003    Blob
            (...)

So this is actually a mild compatibility bug - not a serious one, because our driver should be free to not recognize specific custom types, but perhaps we should have a map of handlers for a few known ones - e.g. org.apache.cassandra.db.marshal.DurationType.

I created #364 to track the problem. For now, either make the test optional or add a Cassandra-specific bypass - namely, let the test pass either when it returns a correct duration, or when it fails because it received a custom type. If you decide to go with the latter, please also add a comment about it in the test case.

@piodul
Copy link
Collaborator

piodul commented Dec 5, 2021

@Ponewor The Duration type was introduced in v5 of the CQL protocol. There was a need for this type to be supported in Scylla - however, Scylla only supports v4 so the support for it was backported to the older version of the protocol.

If you look at the protocol specifications as defined in the Cassandra repository, you can see that in v4 there is no concept of Duration (see here), it is only introduced in v5 (here).

In order to be compatible with the protocol v4, I guess that Cassandra decides to encode Durations as a "Custom" type. I'm not aware of how the encoding looks, though. I agree with @psarna on how to handle this for Cassandra.

scylla/src/frame/value.rs Show resolved Hide resolved
scylla/src/frame/value.rs Outdated Show resolved Hide resolved

fn vint_encode(v: i64, buf: &mut Vec<u8>) {
let mut v = zig_zag_encode(v);
let number_of_bytes = 8 - v.leading_zeros() / 8;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other drivers (Gocql and the Datastax Java Driver) seem to calculate this in a clever, branchless way - look here.

scylla/src/frame/response/result.rs Outdated Show resolved Hide resolved
scylla/src/frame/value.rs Outdated Show resolved Hide resolved
scylla/src/frame/value.rs Outdated Show resolved Hide resolved
scylla/src/frame/value_tests.rs Outdated Show resolved Hide resolved
Comment on lines 677 to 678
months: months as i32,
days: days as i32,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to check that the decoded i64's do not overflow.

scylla/src/frame/value.rs Outdated Show resolved Hide resolved
scylla/src/frame/value.rs Outdated Show resolved Hide resolved
@psarna
Copy link
Contributor

psarna commented Dec 5, 2021

@Ponewor The Duration type was introduced in v5 of the CQL protocol. There was a need for this type to be supported in Scylla - however, Scylla only supports v4 so the support for it was backported to the older version of the protocol.

Ah, makes sense. Btw, we could consider allowing the user to declare the version explicitly to allow "version spoofing". That would allow users to set v5 here, e.g. if they know that they'll only use the subset of v5 that's actually implemented. It would act as a trivial workaround for issues like this.

If you look at the protocol specifications as defined in the Cassandra repository, you can see that in v4 there is no concept of Duration (see here), it is only introduced in v5 (here).

In order to be compatible with the protocol v4, I guess that Cassandra decides to encode Durations as a "Custom" type. I'm not aware of how the encoding looks, though. I agree with @psarna on how to handle this for Cassandra.

psarna added a commit that referenced this pull request Dec 24, 2021
CQL protocol allows sending custom types identified by their
string representation. One of the cases in which the trick is used
is when Cassandra returns duration columns to a CQLv4 client, which
is not supposed to know this type.

WIP: the actual implementation will be done once #363 is merged.

Refs #364
@psarna psarna mentioned this pull request Dec 24, 2021
6 tasks
@psarna
Copy link
Contributor

psarna commented Jan 19, 2022

any updates on this one?

psarna added a commit that referenced this pull request Mar 17, 2022
CQL protocol allows sending custom types identified by their
string representation. One of the cases in which the trick is used
is when Cassandra returns duration columns to a CQLv4 client, which
is not supposed to know this type.

WIP: the actual implementation will be done once #363 is merged.

Refs #364
@Ponewor Ponewor force-pushed the 320-support-for-duration-cql-type branch 2 times, most recently from 185d862 to ac262e9 Compare March 30, 2022 12:58
@Ponewor Ponewor requested a review from piodul March 30, 2022 12:59
@Ponewor Ponewor marked this pull request as ready for review March 30, 2022 12:59
scylla/src/frame/types.rs Outdated Show resolved Hide resolved
scylla/src/frame/response/result.rs Show resolved Hide resolved
scylla/src/frame/types.rs Show resolved Hide resolved
scylla/src/frame/value_tests.rs Outdated Show resolved Hide resolved
scylla/src/frame/value_tests.rs Outdated Show resolved Hide resolved
@Ponewor Ponewor force-pushed the 320-support-for-duration-cql-type branch from ac262e9 to 7df1daf Compare March 31, 2022 15:57
@Ponewor Ponewor requested a review from piodul March 31, 2022 15:57
@piodul piodul merged commit 6851f32 into scylladb:main Apr 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support for Duration CQL type
3 participants