[backport 3.5]feat(router/atc): http segments matching and other improvements #12397

chronolaw · 2024-01-23T07:59:45Z

Summary

git cherry-pick 59670a1 #11721
git cherry-pick 5f5e272 #11904
git cherry-pick 6ce1262 #11905
git cherry-pick 12f45ad #11914
git cherry-pick 12f45ad #11948
git cherry-pick c9fd6c1 #12127
git cherry-pick 8c2b5a4 #12258
git cherry-pick 86f0a6c #12261, fixing *.test
git cherry-pick f49abd6 #12180, fixing *.test (https://github.com/Kong/kong/pull/12268/files#diff-50dcfdce82665f85d2027363af6956fc788e5b8f6c6b8de907e9bd82e8871501)
git cherry-pick 11d7639 #12266
git cherry-pick 560fdfb #11950
git cherry-pick ce12f68 #12307, fixing *.test
git cherry-pick 9305665 #12283
git cherry-pick ef87932 #12378

Checklist

The Pull Request has tests
A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

Fix #[issue number]

nowNick

Looks good! 🚀

I double checked commits & PRs so I don't have any other comments than those described here: #12400 (review)

chobits · 2024-03-01T03:13:59Z

I have double checked PRs that need to be cherry-picked

TODO: I'll assist with running perf test on release/3.5.x after this PR is merged.

…11721) resty.core.utils.str_replace_char() is a better way to replace - to _. In the future string.lua will gather more functions to simplify tools.utils.lua. See: #10443

#11904) This helps with generating easier to read expressions, and the code is more straightforward. However, we must fallback to the old style escaping if the value contains `"#` (very unlikely case). KAG-2952

…pressions` flavor (#11905) `traditional_compatible` flavor remains case insensitive to stay compatible with `traditional` flavor. This change allow `expressions` route authors to pick whether they want case sensitive or insensitive matches. KAG-2905 --------- Co-authored-by: Datong Sun <[email protected]>

…ssions router schema (#11914) KAG-2961 --------- Co-authored-by: Datong Sun <[email protected]>

…outer (#12127) Cache key and context generation are closely related on field present inside configured expressions. It is advantageous to unify the logic for generating them to: 1. Improve cache hit rate, so that only fields referenced inside expressions participates in cache key generation. This is particularly important since we plan on adding more match fields into expressions in the future 2. Improve performance, allows field value to be cached and reused between cache key and context generation 3. Reduced code duplication KAG-3032

… new context (#12258) To avoid frequent memory allocation/deallocations. KAG-3448

…12261) KAG-3032

Fix `03-http-log/01-log_spec.lua` Fix `13-cors/01-access_spec.lua` Fix `spec/03-plugins/03-http-log/01-log_spec.lua`

…expression routes (#11950) These fields are available in stream (L4) routes, making them also available in HTTP to reduce discrepancy between HTTP and stream routes. KAG-2963 KAG-3032

…ase of cache hits (#12307) `upstream_uri` is dependent on request path, and should not be automatically cached inside the route cache due to potential for path matching not being configured inside routes. KAG-3505

…12283) KAG-3351

KAG-3583

@ADD-SP

Full changelog: https://github.com/Kong/atc-router/releases/tag/v1.5.1 This primarily contains the small Bazel change which reduced artifact size by 50%. Thanks @ADD-SP for the work.

This field represents how much segments the request path contains. For example, "/a/b/c/" contains 3 segments, "/a" contains 1 and "/" contains 0 segment. It is useful for implementing segment based routing logic such as these used in OpenAPI spec. KAG-3604

… field (#12406) KAG-3604 --------- Co-authored-by: Datong Sun <[email protected]>

…g segments from it (#12407) KAG-3351

…ot" operator support in ATC (#12405) KAG-3605 --------- Co-authored-by: Datong Sun <[email protected]>

… folder (#12419) KAG-3605

* perf(router): use static functions for callbacks Signed-off-by: Aapo Talvensaari <[email protected]> * tuning some code * style clean * style clean * style clean --------- Signed-off-by: Aapo Talvensaari <[email protected]> Co-authored-by: chronolaw <[email protected]>

StarlightIbuki

Since this is a backport, I'm OK with the change. Just leaving some comments about namings.

StarlightIbuki · 2024-03-06T06:30:05Z

kong/tools/string.lua

The naming is over-generized. The functions in it are just for conversion between naming conventions. Please consider renaming this file and the functions.
We cannot tell what the function does by its name: replace dashes for what?

StarlightIbuki · 2024-03-06T06:32:55Z

kong/router/fields.lua

+local PREFIX_LEN = 13 -- #"http.headers."
+local HTTP_HEADERS_PREFIX = "http.headers."


Suggested change

local PREFIX_LEN = 13 -- #"http.headers."

local HTTP_HEADERS_PREFIX = "http.headers."

local HTTP_HEADERS_PREFIX = "http.headers."

local PREFIX_LEN = #"http.headers."

Thanks. I will do it later.

StarlightIbuki · 2024-03-06T06:34:59Z

kong/router/expressions.lua

@@ -19,8 +24,35 @@ local PROTOCOLS_OVERRIDE = {
 }


-local function get_exp_and_priority(route)
+-- net.port => net.dst.port
+local function transform_expression(route)


How about naming this function "migrate_deprecated_variables"?

It will have more transform actions in the future, not only for migration.

StarlightIbuki · 2024-03-06T06:47:32Z

kong/router/atc.lua

+  HTTP_SCHEMA   = generate_schema(fields.HTTP_FIELDS)
+  STREAM_SCHEMA = generate_schema(fields.STREAM_FIELDS)
+
+  -- used by running router
+  CACHED_SCHEMA = is_http and HTTP_SCHEMA or STREAM_SCHEMA


It seems one of those is not used. Is it?

No, HTTP_SCHEMA and STREAM_SCHEMA will be used by validation.

pull-request-size bot added the size/XXL label Jan 23, 2024

github-actions bot added core/pdk core/db schema-change-noteworthy core/wasm Everything relevant to [proxy-]wasm labels Jan 23, 2024

github-actions bot assigned chronolaw Jan 23, 2024

chronolaw changed the title ~~[cherry]feat(router/atc): http segments matching and other improvements~~ [backport]feat(router/atc): http segments matching and other improvements Jan 23, 2024

chronolaw added the backport label Jan 23, 2024

chronolaw changed the title ~~[backport]feat(router/atc): http segments matching and other improvements~~ [backport 3.5]feat(router/atc): http segments matching and other improvements Jan 24, 2024

chronolaw marked this pull request as ready for review January 27, 2024 01:20

chronolaw force-pushed the cherry/http_segments_matching_35 branch from 5c96d34 to 0b38029 Compare February 26, 2024 02:27

kikito requested review from StarlightIbuki and nowNick February 27, 2024 09:06

nowNick approved these changes Feb 27, 2024

View reviewed changes

chobits approved these changes Mar 1, 2024

View reviewed changes

chronolaw and others added 14 commits March 5, 2024 15:13

perf(router): use resty.core.utils.str_replace_char() for dashes (#…

b7cdced

…11721) resty.core.utils.str_replace_char() is a better way to replace - to _. In the future string.lua will gather more functions to simplify tools.utils.lua. See: #10443

refactor(router): use ATC raw string literal in expressions generation (

7cf1662

#11904) This helps with generating easier to read expressions, and the code is more straightforward. However, we must fallback to the old style escaping if the value contains `"#` (very unlikely case). KAG-2952

fix(router): http and stream subsystems no longer share the expre…

53e96a1

…ssions router schema (#11914) KAG-2961 --------- Co-authored-by: Datong Sun <[email protected]>

refactor(router): simplify the functions of cache calculation (#11948)

7710be7

perf(router): reuse ATC context in router match instead of creating a…

8bd062a

… new context (#12258) To avoid frequent memory allocation/deallocations. KAG-3448

fix(router): add missing preserve_host logic in stream subsystem (#…

1ec0472

…12261) KAG-3032

tests(plugins): fix previous master test failures

8f4fd55

Fix `03-http-log/01-log_spec.lua` Fix `13-cors/01-access_spec.lua` Fix `spec/03-plugins/03-http-log/01-log_spec.lua`

docs(changelog): fix a typo in #11258 (#12266)

b56cc0b

feat(router/atc): support net.src.* and net.dst.* fields in HTTP …

17ce591

…expression routes (#11950) These fields are available in stream (L4) routes, making them also available in HTTP to reduce discrepancy between HTTP and stream routes. KAG-2963 KAG-3032

feat(router): support segment based matching in expressions flavor (#…

9ec9665

…12283) KAG-3351

perf(router/atc): lazy generate and cache field visit functions (#12378)

bcc8703

KAG-3583

dndx and others added 7 commits March 5, 2024 15:13

chore(requirements): bump atc-router to v1.5.1 (#12390)

f0a1497

Full changelog: https://github.com/Kong/atc-router/releases/tag/v1.5.1 This primarily contains the small Bazel change which reduced artifact size by 50%. Thanks @ADD-SP for the work.

docs(changelog): add changelog entry for the http.path.segments.len…

f473d89

… field (#12406) KAG-3604 --------- Co-authored-by: Datong Sun <[email protected]>

tests(router/atc): ensure request path is normalized before extractin…

f2093e1

…g segments from it (#12407) KAG-3351

chore(deps): bump atc-router to v1.6.0 and add changelog entry for "n…

415d115

…ot" operator support in ATC (#12405) KAG-3605 --------- Co-authored-by: Datong Sun <[email protected]>

docs(changelog): move "expressions_not_operator" entry to the correct…

31a9745

… folder (#12419) KAG-3605

locao force-pushed the cherry/http_segments_matching_35 branch from 0b38029 to 24480f4 Compare March 5, 2024 18:13

StarlightIbuki reviewed Mar 6, 2024

View reviewed changes

ADD-SP approved these changes Mar 7, 2024

View reviewed changes

ADD-SP merged commit 25aa629 into release/3.5.x Mar 7, 2024
24 checks passed

ADD-SP deleted the cherry/http_segments_matching_35 branch March 7, 2024 08:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[backport 3.5]feat(router/atc): http segments matching and other improvements #12397

[backport 3.5]feat(router/atc): http segments matching and other improvements #12397

chronolaw commented Jan 23, 2024 •

edited

Loading

nowNick left a comment

chobits commented Mar 1, 2024 •

edited

Loading

StarlightIbuki left a comment

StarlightIbuki Mar 6, 2024

StarlightIbuki Mar 6, 2024

chronolaw Mar 6, 2024

StarlightIbuki Mar 6, 2024

chronolaw Mar 6, 2024

StarlightIbuki Mar 6, 2024

chronolaw Mar 6, 2024

		local PREFIX_LEN = 13 -- #"http.headers."
		local HTTP_HEADERS_PREFIX = "http.headers."

[backport 3.5]feat(router/atc): http segments matching and other improvements #12397

[backport 3.5]feat(router/atc): http segments matching and other improvements #12397

Conversation

chronolaw commented Jan 23, 2024 • edited Loading

Summary

Checklist

Issue reference

nowNick left a comment

Choose a reason for hiding this comment

chobits commented Mar 1, 2024 • edited Loading

StarlightIbuki left a comment

Choose a reason for hiding this comment

StarlightIbuki Mar 6, 2024

Choose a reason for hiding this comment

StarlightIbuki Mar 6, 2024

Choose a reason for hiding this comment

chronolaw Mar 6, 2024

Choose a reason for hiding this comment

StarlightIbuki Mar 6, 2024

Choose a reason for hiding this comment

chronolaw Mar 6, 2024

Choose a reason for hiding this comment

StarlightIbuki Mar 6, 2024

Choose a reason for hiding this comment

chronolaw Mar 6, 2024

Choose a reason for hiding this comment

chronolaw commented Jan 23, 2024 •

edited

Loading

chobits commented Mar 1, 2024 •

edited

Loading