refactor: architecture documentation (#1732)

* refactor: architecture reference

* update release version

* updates

* uses markdownlint-cli2

* updates

* fix

* update references to conduit site docs

.github/pull_request_template.md

@@ -6,9 +6,9 @@ List any dependencies that are required for this change.
 
 Fixes # (issue)
 
-### Quick checks:
+### Quick checks
 
 - [ ] I have followed the [Code Guidelines](https://github.com/ConduitIO/conduit/blob/main/docs/code_guidelines.md).
 - [ ] There is no other [pull request](https://github.com/ConduitIO/conduit/pulls) for the same update/change.
 - [ ] I have written unit tests.
-- [ ] I have made sure that the PR is of reasonable size and can be easily reviewed.
+- [ ] I have made sure that the PR is of reasonable size and can be easily reviewed.

README.md

@@ -115,7 +115,7 @@ Download the right `.deb` file for your machine architecture from the
 [latest release](https://github.com/conduitio/conduit/releases/latest), then run:
 
 ```sh
-dpkg -i conduit_0.8.0_Linux_x86_64.deb
+dpkg -i conduit_0.10.0_Linux_x86_64.deb
 ```
 
 ### RPM
@@ -124,7 +124,7 @@ Download the right `.rpm` file for your machine architecture from the
 [latest release](https://github.com/conduitio/conduit/releases/latest), then run:
 
 ```sh
-rpm -i conduit_0.8.0_Linux_x86_64.rpm
+rpm -i conduit_0.10.0_Linux_x86_64.rpm
 ```
 
 ### Build from source
@@ -310,21 +310,23 @@ For more information about the UI refer to the [Readme](ui/README.md) in `/ui`.
 ## Documentation
 
 To learn more about how to use Conduit
-visit [docs.Conduit.io](https://docs.conduit.io).
+visit [Conduit.io/docs](https://conduit.io/docs).
 
 If you are interested in internals of Conduit we have prepared some technical
 documentation:
 
-- [Pipeline Semantics](docs/pipeline_semantics.md) explains the internals of how
+- [Pipeline Semantics](https://conduit.io/docs/features/pipeline-semantics) explains the internals of how
   a Conduit pipeline works.
-- [Pipeline Configuration Files](docs/pipeline_configuration_files.md)
+- [Pipeline Configuration Files](https://conduit.io/docs/pipeline-configuration-files)
   explains how you can define pipelines using YAML files.
 - [Processors](https://conduit.io/docs/processors/getting-started) contains examples and more information about
   Conduit processors.
-- [Conduit Architecture](docs/architecture.md)
+- [Conduit Architecture](https://conduit.io/docs/getting-started/architecture)
   will give you a high-level overview of Conduit.
 - [Conduit Metrics](docs/metrics.md)
   provides more information about how Conduit exposes metrics.
+- [Conduit Package structure](docs/package_structure.md)
+  provides more information about the different packages in Conduit.
 
 ## Contributing
 
@@ -345,8 +347,7 @@ We also value contributions in form of pull requests. When opening a PR please
 ensure:
 
 - You have followed
-  the [Code Guidelines](https://github.com/ConduitIO/conduit/blob/main/docs/code_guidelines.md)
-  .
+  the [Code Guidelines](https://github.com/ConduitIO/conduit/blob/main/docs/code_guidelines.md).
 - There is no other [pull request](https://github.com/ConduitIO/conduit/pulls)
   for the same update/change.
 - You have written unit tests.

docs/architecture-decision-records/20230123-pipeline-configuration-files.md

@@ -65,4 +65,4 @@ can be ( only incompatible changes cause the state to be deleted).
 
 ## Related
 
-[Pipeline configuration files documentation](../pipeline_configuration_files.md)
+[Pipeline configuration files documentation](https://conduit.io/docs/pipeline-configuration-files)

docs/package_structure.md

@@ -0,0 +1,38 @@
+# Package structure
+
+- `cmd` - Contains main applications. The directory name for each application should match the name of the executable
+  (e.g. `cmd/conduit` produces an executable called `conduit`). It is the responsibility of main applications to do 3
+  things, it should not include anything else:
+  1. Read the configuration (from a file, the environment or arguments).
+  2. Instantiate, wire up and run internal services.
+  3. Listen for signals (i.e. SIGTERM, SIGINT) and forward them to internal services to ensure a graceful shutdown
+(e.g. via a closed context).
+  - `conduit` - The entrypoint for the main Conduit executable.
+- `pkg` - The internal libraries and services that Conduit runs.
+  - `conduit` - Defines the main runtime that ties all Conduit layers together.
+  - `connector` - Code regarding connectors, including connector store, connector service, connector configurations
+      and running instances.
+  - `foundation` - Foundation contains reusable code. Should not contain any business logic. A few honorable mentions:
+    - `cerrors` - Exposes error creation and wrapping functionality. This is the only package for errors used in Conduit.
+    - `ctxutil` - Utilities to operate with context.Context.
+    - `grpcutil` - Utilities to operate with GRPC requests.
+    - `log` - Exposes a logger. This is the logger used throughout Conduit.
+    - `metrics` - Exposes functionality for gathering and exposing metrics.
+  - `inspector` - Exposes a service that can be used to inspect and operate with the state of Conduit.
+  - `orchestrator` - Code regarding the orchestration layer.
+  - `pipeline` - Code regarding pipelines, including pipeline store, pipeline service, running pipeline instances.
+  - `plugin` - Currently contains all logic related to plugins as well as the plugins themselves. In the future a lot of
+      this code will be extracted into separate repositories, what will be left is a plugin service that manages built-in
+      and external plugins.
+  - `processor` - Code regarding processors, including processor store, processor service, running processor instances.
+  - `provisioning` - Exposes a provisioning service that can be used to provision Conduit resources.
+  - `schemaregistry` - Code regarding the schema registry.
+  - `web` - Everything related to Conduit APIs or hosted pages like the UI or Swagger.
+
+Other folders that don't contain Go code:
+
+- `docs` - Documentation regarding Conduit that's specific to this repository (more documentation can be found at [Conduit.io/docs](https://conduit.io/docs)).
+- `proto` - Protobuf files (e.g. gRPC API definition).
+- `scripts` - Contains scripts that are useful for development.
+- `test` - Contains configurations needed for integration tests.
+- `ui` - A subproject containing the web UI for Conduit.