feat(expr): add json path functions #13568

wangrunji0408 · 2023-11-21T13:13:44Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

This PR implements all jsonpath operators and functions:

jsonb @? jsonpath → boolean
jsonb @@ jsonpath → boolean
jsonb_path_exists ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → boolean
jsonb_path_match ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → boolean
jsonb_path_query ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → setof jsonb
jsonb_path_query_array ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → jsonb
jsonb_path_query_first ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → jsonb

This PR doesn't introduce the jsonpath data type. However, this does not affect the normal use of the above functions. All jsonpath arguments in the signature are actually varchar. If we have jsonpath type, there will be only one parsing during 'literal' :: jsonpath. Currently, parsing happens every time the eval is called (not every time evaluating a row). I think it's not a big problem.

The jsonpath parsing and query logic is separated into an independent crate: sql-json-path. Please also review it if you are interested. (After investigation, I found no existing crate that can meet our requirements. Most of them follow the IETF standard and only support serde-json.)

Almost all jsonpath syntax is supported, except .datetime() and some escaped string literal. I use Postgres regression test jsonb_jsonpath.sql to ensure correctness. Please check this file to see what is (not) supported yet. Detailed explanation: #13568 (comment)

This PR is based on #13545, which is needed to run regression test. After it merged, I will rebase this PR onto main.

References:

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added test labels as necessary. See details.
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Support jsonpath operators and functions:

jsonb @? jsonpath → boolean
jsonb @@ jsonpath → boolean
jsonb_path_exists ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → boolean
jsonb_path_match ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → boolean
jsonb_path_query ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → setof jsonb
jsonb_path_query_array ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → jsonb
jsonb_path_query_first ( target jsonb, path jsonpath [, vars jsonb [, silent boolean ]] ) → jsonb

See postgres docs and search "jsonpath" for more details. Note that we don't support jsonpath data type for now. All jsonpath arguments in the signature are actually varchar.

For the syntax of jsonpath, see 9.16.2. The SQL/JSON Path Language. Almost all features in the docs have been supported, except that:

.datetime() and . datetime(template) method is not supported.
Some implementations for the silent argument may be incorrect.
Some implementations for regexp matching may be incorrect.

BugenZhao · 2023-11-22T05:39:18Z

src/expr/impl/src/scalar/jsonb_path.rs

+
+#[function(
+    "jsonb_path_exists(jsonb, varchar) -> boolean",
+    prebuild = "JsonPath::new($1).map_err(parse_error)?"


wangrunji0408 · 2023-11-23T08:22:24Z

I have fixed regression tests as much as possible.
The remaining failing cases are as follows: (- is expected result, + is actual output)

 # wrong silent handling
 select jsonb_path_query('[1,"2",3]', '+$', '{}', true);
  jsonb_path_query 
 ------------------
- 1
-(1 row)
+(0 rows)

 # wrong silent handling
 select jsonb '["1",2,0,3]' @? '-$[*]';
  ?column? 
 ----------
- t
+ 
 (1 row)

 # double precision problem
 select jsonb_path_query('{"a": 2.5}', '-($.a * $.a).floor() % 4.3');
- jsonb_path_query 
-------------------
- -1.7
+  jsonb_path_query   
+---------------------
+ -1.7000000000000002
 (1 row)

 # I think both results are valid according to the specification. It depends on implementation.
 select jsonb_path_query('[[null, 1, "abc", "abcabc"]]', 'lax $ ? (@[*] starts with "abc")');
-      jsonb_path_query      
-----------------------------
- [null, 1, "abc", "abcabc"]
-(1 row)
+ jsonb_path_query 
+------------------
+(0 rows)

 # wrong regexp matching
 select jsonb_path_query('[null, 1, "a\b", "a\\b", "^a\\b$"]', 'lax $[*] ? (@ like_regex "a\\b" flag "")');
  jsonb_path_query 
 ------------------
+ "^a\\b$"
+ "a\\b"
  "a\b"
-(1 row)
+(3 rows)

 # wrong regexp matching
 select jsonb_path_query('[null, 1, "a\b", "a\\b", "^a\\b$"]', 'lax $[*] ? (@ like_regex "^a\\b$" flag "")');
  jsonb_path_query 
 ------------------
- "a\b"
-(1 row)
+(0 rows)

 # wrong silent handling
 SELECT jsonb_path_query_first('[{"a": 1}, {"a": 2}, {}]', 'strict $[*].a', '{}', true);
  jsonb_path_query_first 
 ------------------------
- 1
+ 
 (1 row)

As these are mostly corner cases, I think we can move on and leave them to be fixed later.

Signed-off-by: Runji Wang <[email protected]>

codecov · 2023-11-28T09:29:53Z

Codecov Report

Attention: 174 lines in your changes are missing coverage. Please review.

Comparison is base (a73d940) 68.19% compared to head (b96fe7e) 68.15%.

Files	Patch %	Lines
src/expr/impl/src/scalar/jsonb_path.rs	23.80%	144 Missing ⚠️
src/frontend/src/binder/expr/binary_op.rs	18.75%	13 Missing ⚠️
src/common/src/types/jsonb.rs	16.66%	5 Missing ⚠️
src/sqlparser/src/tokenizer.rs	0.00%	4 Missing ⚠️
src/frontend/src/expr/mod.rs	0.00%	3 Missing ⚠️
src/sqlparser/src/ast/operator.rs	0.00%	2 Missing ⚠️
src/sqlparser/src/parser.rs	33.33%	2 Missing ⚠️
src/expr/macro/src/gen.rs	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #13568      +/-   ##
==========================================
- Coverage   68.19%   68.15%   -0.04%     
==========================================
  Files        1524     1525       +1     
  Lines      262232   262454     +222     
==========================================
+ Hits       178829   178879      +50     
- Misses      83403    83575     +172

Flag	Coverage Δ
rust	`68.15% <24.67%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/expr/impl/Cargo.toml

Signed-off-by: Runji Wang <[email protected]>

BugenZhao

Cool!

BugenZhao · 2023-12-05T03:47:06Z

src/expr/impl/src/scalar/jsonb_path.rs

+    let matched = path
+        .query::<ValueRef<'_>>(target.into())
+        .map_err(eval_error)?;
+    Ok(matched.into_iter().map(|json| json.into_owned().into()))


Is it possible to return an iterator for query?

It is hard in stable Rust as all query functions need to be written as generator.

What about those 3rd party libraries available under stable 🥵? Like https://github.com/Wolvereness/remit/

Interesting! Will have a try later.

TennyZhuang

The PR and test suite LGTM. Although not all corner cases were covered, I think it’s still good enough to merge.

Signed-off-by: Runji Wang <[email protected]>

Signed-off-by: Runji Wang <[email protected]> Co-authored-by: Runji Wang <[email protected]>

wangrunji0408 requested review from neverchanje, TennyZhuang, xiangjinwu and BugenZhao November 21, 2023 13:13

wangrunji0408 requested a review from a team as a code owner November 21, 2023 13:13

github-actions bot added the type/feature label Nov 21, 2023

wangrunji0408 marked this pull request as draft November 22, 2023 02:21

BugenZhao reviewed Nov 22, 2023

View reviewed changes

wangrunji0408 marked this pull request as ready for review November 23, 2023 08:13

Base automatically changed from wrj/implicit-cast to main November 28, 2023 08:51

wangrunji0408 added 12 commits November 28, 2023 16:58

parser: add @@ and @? operator

d0a3a80

Signed-off-by: Runji Wang <[email protected]>

implement jsonpath functions

60c236d

Signed-off-by: Runji Wang <[email protected]>

support silent argument

64f3226

Signed-off-by: Runji Wang <[email protected]>

update Cargo.lock

24874d3

Signed-off-by: Runji Wang <[email protected]>

fix jsonb_path_match

57ef1f1

Signed-off-by: Runji Wang <[email protected]>

remove named arguments

85017ed

Signed-off-by: Runji Wang <[email protected]>

enable regress test and comment some unsupported cases

15164e3

Signed-off-by: Runji Wang <[email protected]>

switch dependency to github

8665b18

Signed-off-by: Runji Wang <[email protected]>

cargo sort

064a5a8

Signed-off-by: Runji Wang <[email protected]>

fix: @@ and @? should silent error

6031324

Signed-off-by: Runji Wang <[email protected]>

fix more regression tests

fa98f48

Signed-off-by: Runji Wang <[email protected]>

support .**

a9f393b

Signed-off-by: Runji Wang <[email protected]>

wangrunji0408 force-pushed the wrj/jsonb-path branch from f032ed1 to a9f393b Compare November 28, 2023 08:59

TennyZhuang reviewed Nov 29, 2023

View reviewed changes

src/expr/impl/Cargo.toml Outdated Show resolved Hide resolved

wangrunji0408 added 2 commits December 1, 2023 15:04

switch sql-json-path to crates.io version

d6ca13e

Signed-off-by: Runji Wang <[email protected]>

Merge branch 'main' into wrj/jsonb-path

b96fe7e

wangrunji0408 requested review from TennyZhuang and BugenZhao December 4, 2023 04:59

BugenZhao reviewed Dec 5, 2023

View reviewed changes

TennyZhuang approved these changes Dec 5, 2023

View reviewed changes

wangrunji0408 added this pull request to the merge queue Dec 5, 2023

wangrunji0408 added the need-cherry-pick-v1.5.0 label Dec 5, 2023

Merged via the queue into main with commit e0cfb82 Dec 5, 2023
15 of 16 checks passed

wangrunji0408 deleted the wrj/jsonb-path branch December 5, 2023 05:01

github-actions bot mentioned this pull request Dec 5, 2023

feat(expr): add json path functions (#13568) #13798

Merged

github-actions bot pushed a commit that referenced this pull request Dec 5, 2023

feat(expr): add json path functions (#13568)

353ec33

Signed-off-by: Runji Wang <[email protected]>

wangrunji0408 added a commit that referenced this pull request Dec 5, 2023

feat(expr): add json path functions (#13568) (#13798)

4205179

Signed-off-by: Runji Wang <[email protected]> Co-authored-by: Runji Wang <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(expr): add json path functions #13568

feat(expr): add json path functions #13568

wangrunji0408 commented Nov 21, 2023 •

edited

Loading

BugenZhao Nov 22, 2023

wangrunji0408 commented Nov 23, 2023 •

edited

Loading

codecov bot commented Nov 28, 2023 •

edited

Loading

BugenZhao left a comment

BugenZhao Dec 5, 2023

wangrunji0408 Dec 5, 2023

BugenZhao Dec 5, 2023

wangrunji0408 Dec 5, 2023

TennyZhuang left a comment

feat(expr): add json path functions #13568

feat(expr): add json path functions #13568

Conversation

wangrunji0408 commented Nov 21, 2023 • edited Loading

What's changed and what's your intention?

Checklist

Documentation

BugenZhao Nov 22, 2023

Choose a reason for hiding this comment

wangrunji0408 commented Nov 23, 2023 • edited Loading

codecov bot commented Nov 28, 2023 • edited Loading

Codecov Report

BugenZhao left a comment

Choose a reason for hiding this comment

BugenZhao Dec 5, 2023

Choose a reason for hiding this comment

wangrunji0408 Dec 5, 2023

Choose a reason for hiding this comment

BugenZhao Dec 5, 2023

Choose a reason for hiding this comment

wangrunji0408 Dec 5, 2023

Choose a reason for hiding this comment

TennyZhuang left a comment

Choose a reason for hiding this comment

wangrunji0408 commented Nov 21, 2023 •

edited

Loading

wangrunji0408 commented Nov 23, 2023 •

edited

Loading

codecov bot commented Nov 28, 2023 •

edited

Loading