[CT-1770] [Feature] [Spike] Review programmatic entry point for top-level commands

[jtcohen6]

Status quo

Here's how the "thin layer" for programmatically invoking dbt works today, on the feature/click-cli feature branch:

# initialize
from dbt.cli.main import dbtRunner
dbt = dbtRunner()

# need to pass CLI-style args as an ordered list
cli_args = ['--fail-fast', 'run', '--select', 'tag:my_tag+2', 'another_model', '--exclude', 'not_this_model']

# 'results' is List[RunResult], 'success' is boolean
results, success = dbt.invoke(cli_args)

It's very very important that dbt-core's Python API is able to accept anything that you could also throw at its CLI. We should by no means take away this implementation; it works exactly in the ways it needs to. That said, it might not be in line what what community members are expecting the Python API to look & feel like.

Proposal

Another entry point could enable users to pass in structured data, and call a top-level method matching the desired command:

# initialize
from dbt.main import dbtRunner  # not dbt.cli.main
dbt = dbtRunner()

# https://docs.getdbt.com/reference/global-configs
global_configs = {"fail_fast": true}

# these could still be CLI-style strings
select = "tag:my_tag+ another_model"
exclude = "not_this_model"

# top-level method matching the CLI subcommand
results, success = dbt.run(
    select=select,
    exclude=exclude,
    global_configs=global_configs
)

For selection syntax, we could also mirror the data structure of yaml selectors:

one_off_selector = {
    "union": [
        {"method": "tag", "value": "my_tag", "children": true, "children_depth": 2},
        "another_model",
        "exclude": ["not_this_model"]
    ]
}
results, success = dbt.run(selector=one_off_selector, global_configs=global_configs)

Additional thoughts

• I could also imagine passing global_configs into dbtRunner initialization: dbt = dbtRunner(global_configs = ...), with the ability to optionally override them for specific commands. This starts to feel a lot like setting UserConfig ([CT-1470] Whither UserConfig? #6207), or passing in env vars as "data" ([CT-1765] [Feature] Provide env vars as data during runtime #6545).
• Would we ever want users to import SelectionSpec and SelectorConfig, and construct a selector that way, using the Python objects directly...? We need to be very clear about what's part of the public API, and what are private internals.
• Order matters for CLI parameters today, but IMO it shouldn't: [CT-1737] Support all "global" flags after all subcommands #6497

[nathaniel-may]

A big part of this ticket is to discuss what using dbt as a "library" really looks like. We may need follow up tickets to this one after that discussion.

[ChenyuLInx]

Internal conversation going on about this topic. Closing in favor upcoming issues