[CT-1889] [Epic] API-ification + CLI - Phase 2

[leahwicz]

This epic tracks follow-on work from Phase 1 of API-ification. Our goal at the end of this epic is to have a rock-solid, well-documented top-level API that is powering a compelling CLI and an initial Python API for programmatic invocations.

(40%) UX & documentation

CLI Improvements

Preview Give feedback

1. [CT-1869] Turn --output-keys into MultiOption #6676
   Team:Execution cli python_api +3
   
2. [CT-1981] Issue when passing --select and --model to a command together #6800
   Team:Execution cli tech_debt +3
   

Documentation

Preview Give feedback

1. [CT-1369] Human-written docs for top level API via Sphinx #6099
   Team:Execution cli +2
2. [CT-1548] Example script to demo click API usage #6310
   Team:Execution python_api +2
   
3. [CT-916] High Level API - Documentation #5540
   Team:Execution stale +2
4. [CT-1766] [DX] Review command + param help text in CLI / top-level API #6546
   Refinement cli enhancement paper_cut python_api +5
   
5. [CT-1768] [Docs] Top-level API docs glow-up #6548
   4 of 4
   Team:Execution enhancement python_api user docs +4
6. [CT-2114] In-repo documentation (readmes): architecture/sequence diagram of core execution flow #6969
   python_api spike +2
7. [CT-2115] In-repo documentation: glossary of terms #6970
   Team:Adapters enhancement python_api +3
8. [CT-2116] In-repo documentation: update ARCHITECTURE.md #6971
   stale +1

(30%) Enable smooth upgrades

Restore previous behavior (bugs & telemetry), and provide clear warnings for deprecated/renamed functionality.

Bugs

Preview Give feedback

1. [CT-2078] Logging issues since feature branch merge #6933
   Team:Execution logging +2
   
2. [CT-2108] [Regression] log path does not respect --project-dir #6957
   Team:Execution bug python_api regression +4
   
3. [CT-2142] [Regression] dbt 1.5.0 return invalid exit codes. #7010
   Team:Execution bug regression +3
   
4. [CT-2267] [Bug] Unexpected stacktrace for dbt --fail-fast run #7118
   Team:Execution bug cli +3
   
5. [CT-2069] Unexpected behavior with global/subcommand flags #6913
   Team:Execution bug cli regression +4
   
6. [CT-2127] [Bug] Running dbt commands before deps results in dbt_packages folder creation at invocation location #6985
   bug file_system help_wanted +3

Telemetry

Preview Give feedback

1. [CT-1367] New tracking event: Adapter info #6097
   Team:Execution cli +2
   
2. [CT-1368] New tracking event: Project ID #6098
   Team:Execution cli +2
   
3. [CT-1898] Turn on event tracking integration testing #3631
   jira tech_debt +2

Deprecations

Preview Give feedback

1. [CT-2109] Re-name print flag to meet naming convention #6960
   Team:Execution cli +2
   
2. [CT-2066] Handle old param/flag names in new CLI #6903
   Team:Execution cli +2
   
3. [CT-2054] Add deprecation warnings for LOG_PATH and TARGET_PATH in dbt_project.yml #6882
   Team:Execution tech_debt +2
   

(30%) Tech debt

Flags

Preview Give feedback

1. [CT-1615] [improvement] Use click option defaults outside of click #6397
   python_api stale +2
   
2. [CT-1511] A general solution for accessing attributes in the flag object safely #6259
   tech_debt +1
3. [CT-2028] A general solution for setting attributes in flag objects to consistent types #6857
   spike tech_debt +2
   
4. [CT-1470] Whither UserConfig? #6207
   Refinement enhancement +2
   
5. [CT-2169] [Bug] All global configs should also be settable in ProjectFlags #7036
   bug user docs +2

Misc

Preview Give feedback

1. [CT-1746] Flags.MP_CONTEXT is not JSON serializable #6509
   python_api +1
   
2. [CT-1900] Test Click CliRunner #6722
   Team:Execution +1
   
3. [CT-1213] [Improvement] Update profiler to remove "enabled" arg #5886
   enhancement good_first_issue +2
4. [CT-1510] Decouple Target from Project #6257
   Team:Execution cli python_api +3
5. [CT-1586] [Feature] Refactor tasks to be initialized using Profile, Project, and Flags instead of RuntimeConfig #6360
   Refinement enhancement python_api tech_debt +4
6. [Spike] [CT-1575] Improve Console Log Capture Mechanism #6350
   logging python_api spike tech_debt +4
   
7. [CT-1846] Refactor dbt init to use click more directly. #6646
   cli enhancement python_api tech_debt +4

[leahwicz]

Phase 1 work can be found here: #5527

[jtcohen6]

Closing in favor of the more narrowly scoped epic for v1.5: #6706