rest_api: simplify dependent resource relationship configuration

[burnash]

Background

The current syntax for defining relationships between resources in the rest_api source is verbose and requires explicit configuration of parameter resolution. This makes the configuration more complex than necessary, especially for common parent-child relationships were only a simple field reference is needed.

Current syntax:

{
    "resources": [
        {
            "name": "issues",
            "endpoint": {
                "path": "issues",
            },
        },
        {
            "name": "issue_comments",
            "endpoint": {
                "path": "issues/{issue_number}/comments",
                "params": {
                    "issue_number": {
                        "type": "resolve",
                        "resource": "issues",
                        "field": "number",
                    }
                },
            },
        },
    ],
}

Proposal

1. Introduce a simplified reference syntax using resources.<resource_name>.<field_name>
2. Allow direct field references in path templates and query parameters
3. Add support incremental context object via exposing incremental object to template strings
4. The existing resolve mechanism would be kept for backward compatibility

New syntax:

For path templates:

source = rest_api_source({
    "client": {
        "base_url": "https://example.com/api",
    },
    "resources": [
        {
            "name": "issues",
            "endpoint": {
                "path": "issues",
            },
        },
        {
            "name": "issue_comments",
            "endpoint": {
                "path": "issues/{resources.issues.number}/comments",  # reference parent field directly
            },
        },
    ],
})

For query parameters:

source = rest_api_source({
    "client": {
        "base_url": "https://example.com/api",
    },
    "resources": [
        "issues", # using the short form here
        {
            "name": "issue_comments",
            "endpoint": {
                "path": "issue/comments",
                "params": {
                    "issue_number": "{resources.issues.number}",  # reference parent field in query params
                },
            },
        },
    ],
}

For query parameters with incremental sync:

source = rest_api_source({
    "client": {
        "base_url": "https://example.com/api",
    },
    "resources": [
        "issues",
        {
            "name": "issue_comments",
            "endpoint": {
                "path": "issue/comments",
                "params": {
                    "issue_number": "{resources.issues.number}",  # reference parent field in query params
                    "since": "{incremental.last_value}", # the incremental config is defined below
                },
                "incremental": {
                     "cursor_path": "created_at",
                     "initial_value": "2025-01-12"
                }
            },
        },
    ],
}

So the HTTP request for issue_comments resource would be:

GET /issues/comments?issue_number=123&since=2025-01-12

(this should fix #1978 but with a different syntax)

Benefits:

1. More intuitive configuration of resource relationships
2. Reduced boilerplate code
3. Consistent syntax for both path and query parameters (also possible to use in headers, see Enhanced parameter resolution for RESTAPIConfig and REST API source #2071)

Additional Examples:

Multiple field references:

{
    "name": "user_details",
    "endpoint": {
        "path": "groups/{resources.users.group_id}/users/{resources.users.id}/details",
    },
}

Nested field access (TBD, how to define JSONPath expressions):

{
    "name": "user_details",
    "endpoint": {
        "path": "groups/{resources.users.group.id}/users/{resources.users.id}/details",
    },
}

[francescomucio]

this is a nice idea, but I wonder if it can be extended to handle also "aggregates".

An example of aggregates is an endpoint which can provide details for multiple resources (within limits) if passed as an array in a json.

One idea could be something like (to get chunks of 100 user ids):

{
    "name": "user_details",
    "parent": "users",
    "endpoint": {
        "path": "users/details",
        "json: {
              "user_ids": ["{list(parent.id, 100)}"]
        }
    },
}

[kiernan]

this is a nice idea, but I wonder if it can be extended to handle also "aggregates".

An example of aggregates is an endpoint which can provide details for multiple resources (within limits) if passed as an array in a json.

One idea could be something like (to get chunks of 100 user ids)

👍 I've needed to fall back to using the REST Client rather than REST API Source due to a 3rd party API only accepting params via request body instead of path or query params.