From e7c66fdd7b28e087e300e590eb6324fe9029396a Mon Sep 17 00:00:00 2001 From: Stefan Majer Date: Fri, 29 Nov 2024 09:25:09 +0100 Subject: [PATCH 1/7] Hello metal stack chain --- .../src/development/proposals/MEP99/README.md | 156 +++++++ .../MEP99/metal-stack-chain.drawio.svg | 404 ++++++++++++++++++ .../MEP99/needle-control-plane-ha.drawio.svg | 229 ++++++++++ .../MEP99/needle-control-plane.drawio.svg | 217 ++++++++++ .../MEP99/needle-partition.drawio.svg | 81 ++++ 5 files changed, 1087 insertions(+) create mode 100644 docs/src/development/proposals/MEP99/README.md create mode 100644 docs/src/development/proposals/MEP99/metal-stack-chain.drawio.svg create mode 100644 docs/src/development/proposals/MEP99/needle-control-plane-ha.drawio.svg create mode 100644 docs/src/development/proposals/MEP99/needle-control-plane.drawio.svg create mode 100644 docs/src/development/proposals/MEP99/needle-partition.drawio.svg diff --git a/docs/src/development/proposals/MEP99/README.md b/docs/src/development/proposals/MEP99/README.md new file mode 100644 index 0000000000..e5e1cbd526 --- /dev/null +++ b/docs/src/development/proposals/MEP99/README.md @@ -0,0 +1,156 @@ +# Solve the bootstrap problem + +Setting up a metal-stack.io environment in your own datacenter requires a control plane to be present which hosts the metal-stack api. +If you plan to spin up kubernetes clusters, either with gardener.cloud or cluster api, the requirement for this control plane raises. +The control plane must be running in a kubernetes cluster, which offers at least the following features: + +- Loadbalancing +- Persistent Storage +- Access to a object storage for automatic backups of the stateful sets +- Access to a DNS provider which is supported by one of the dns extensions in use. + +This cluster must also be highly available to prevent complete loss of control over the managed resources in the datacenter. +Regular kubernetes updates to apply security fixes and feature updates must be possible in a automated manner. + +The most obvious and simple solution is to use one of the managed kubernetes offerings from another cloud provider. + +But there are use cases, where it is not possible because of network restrictions, or because the company compliances does forbidd the usage of external datacenter products. +For such cases a solution must be found which produces the control plane inside the own datacenter but with reasonable day two operational effort. + +## Possible Solutions + +No complete list. + +- vmware and rancher +- talos +- 3 physical machines with kubespray +... + +All of these solutions add another stack which is probably new to the team which already operates the metal-stack environment. + +TODO: can we provide a list which of the requirements can be solved with all of the alternatives. + +## Use your own dogfood + +With metal-stack.io we already have the possibility to create an manage kubernetes cluster with the help of gardener.cloud. +Use this stack to create a the control plane clusters only. Do not try to create more clusters for other purposes than metal-stack control planes. +If this restriction applies, the requirement for a control plane for this metal-stack setup can be minimal. + +This metal-stack setup also requires a control plane to host metal-api and gardener, but this control plane does not have huge resource requirements in terms of cpu, memore and storage. +For this initial control plane cluster we could use [kind](https://kind.sigs.k8s.io/) running on a single server which manages the initial metal-stack partitin to host the control plane for the real setup. + +This is a chain of two metal-stack environments. + +### Architekture + +A high-level architecture consists of two metal-stack.io environments, one for the control plane, the second one for the production or real environment. It might also be possible to call the initial metal-stack.io environment the metal-stack seed, and the actual production environment the metal-stack seed. + +We could even use some names for this environments which match better to metal, like `needle` and nail. So, a `needle` metal-stack is used to create a `nail` metal-stack environment. + +![metal-stack-chain](metal-stack-chain.drawio.svg) + +The `needle` and the `nail` metal-stack have both a control plane and a set of physical bare metal machines they manage and operate on. + +#### Needle + +In case of the `needle` the control plane is small and running inside a kind cluster, the physical bare metal machines, can be any machines and switches which are supported by metal stack, but can be smaller in terms of cpu, memory and network speed, because these machines must only be capable of running the `nail` metal stack control plane. + +1. Control Plane + +In the most simple case the `needle` control plane is based on kind which is running on a machine which was setup manually/partly automated with a debian:12 operating system. +This machine provides a decent amount of cpu, memory and storage locally to store all persistent data locally. The amount of cpus and memory depends on the required size of the expected `nail` control plane. A typical single socket server with 8-16 cores and 64GB of RAM and two NVMe drives of 1TB would be a good starting point. + +In a typical kind setup, stateful set would not survive their data once the kind cluster was terminated and started again. But there is a possibility to define parts of the local storage of the server to be provided to the kind cluster for the PVCs. With that, kind could be terminated and started again, for example to update and reboot the host os, or update kind itself, the data will still be there. + +Example kind configuration for persistent storage on the hosts os: + +```yaml +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +name: needle-control-plane +nodes: +- role: control-plane + # add a mount from /path/to/my/files on the host to /files on the node + extraMounts: + - hostPath: /path/to/my/files + containerPath: /files + +``` + +As mentioned before, kind is used to host the `needle` control plane. For a gardener managed kubernetes setup, metal-stack and gardener will be deployed into this cluster. This deployment can be done by a gitlab runner which is running on this machine. +The mini-lab will be used as a base for this deployment. The current development of gardener-in-minilab must be extended to host all required extensions to make this a working metal stack control plane which can manage the machines in the attached bare metal setup. + +A second kind cluster is started on this machine to host services which are required to complete the service. A non-complete list would be: + +- PowerDNS to server as a DNS Server for all dns entries which needs to be created in the needle, like api.needle.metal-stack.local, gardener-api.needle.metal-stack.local and the dns entries for the api servers of the create kubernetes clusters. +- NTP +- Monitoring for the `needle` partition ? +- Optional: Container Registry to host all metal-stack and gardener containers +- Optional: Letsencrypt [boulder](https://github.com/letsencrypt/boulder) as a certificate authority +- ... + +![needle-control-plane](needle-control-plane.drawio.svg) + +1.1. Control Plane High Availability + +Running the `needle` control plane on a single physical server is not as available as it should be in such a use case. It should be possible to survive a loss of this server, because the server could be lost by many events, such as hardware failure, disk corruption or even failure of the datacenter location where this server is deployed. + +Setting up a second server with the same software components is an option, but the problem of data redundancy must be solved, because neither the gardener control plane, nor the metal-stack control plane can be instantiated twice. + +Given that we provide part of the local storage of the server as backing storage for the stateful sets in the kind cluster, the data stored on the server itself must be synced to a second server in some way. + +Her comes [DRBD](https://github.com/LINBIT/drbd) into play, this is a linux kernel module which can be configured to mirror one or more local block devices to another server connected over tcp. With the help of [pacemaker](https://www.clusterlabs.org/pacemaker/) a coordinated failover of resources running on top of filesystems created on such replicated drbd devices, a high available statefule server pair is possible. It is also possible to prevent split brain if both servers have a out-of-band management build in with power off capability. +DRBD can also be configured to sync storage between WAN links with a higher latency by using a async mechanism. + +Sample drbd configuration: + +```conf +resource needle-control-plane { + meta-disk internal; + device /dev/drbd0; + syncer { + verify-alg sha1; + } + net { + allow-two-primaries; + } + on needle1 { + disk /dev/nvme0n1; + address 192.168.1.101:7789; + } + on needle2 { + disk /dev/nvme0n1; + address 192.168.1.102:7789; + } +} +``` + +TODO: Pacemaker or Heartbeat ? +TODO: LVM Volumes + +![needle-control-plane-ha](needle-control-plane-ha.drawio.svg) + +1. Partition + +The partition which is managed by the metal-stack `needle` can be a simple and small hardware setup but yet capable enough to host the metal-stack `nail` control plane. +It can follow the metal-stack minimal setup which provides about 8-16 small servers connected to a 1G/s or 10G/s network dataplane. Central storage is optional as the persistence of the services running in these clusters is always backed up to a central object storage. Operations would be much easier if a central storage is provided. + +TODO: howto provide storage +TODO: Maybe the `needle` control plane server provides storage from the drbd volume ? Bad Idea ? +TODO: Small Lightbits, other appliance like Synology at.al. + +A seed must be created which is responsible for hosting the control planes of the shoots in this partition. The amount of shoots should be minimal, most of the time, two shoots, one for hosting gardener and one for metal-stack. + +![needle-partition](needle-partition.drawio.svg) + +1. Network Diagram + +TODO: Where to connect the `needle` servers + +## Open Topics + +- Naming of the metal-stack chain elements, is `needle` and `nail` appropriate ? +- Storage in the `needle` partition +- S3 Object storage is considered as provided +- AirGapped is out of scope for now +- IP address ranges and families diff --git a/docs/src/development/proposals/MEP99/metal-stack-chain.drawio.svg b/docs/src/development/proposals/MEP99/metal-stack-chain.drawio.svg new file mode 100644 index 0000000000..cd5cf00705 --- /dev/null +++ b/docs/src/development/proposals/MEP99/metal-stack-chain.drawio.svg @@ -0,0 +1,404 @@ + + + + + + + + + + +
+
+
+ Partition 1 +
+
+
+ + + Partition 1 + + + + + + + +
+
+
+ seeds +
+
+
+ + + seeds + + + + + + + +
+
+
+ shoots +
+
+
+ + + shoots + + + + + + + + + +
+
+
+ Partition 2 +
+
+
+ + + Partition 2 + + + + + + + +
+
+
+ seeds +
+
+
+ + + seeds + + + + + + + +
+
+
+ shoots +
+
+
+ + + shoots + + + + + + + + + +
+
+
+ Partition 3 +
+
+
+ + + Partition 3 + + + + + + + +
+
+
+ seeds +
+
+
+ + + seeds + + + + + + + +
+
+
+ shoots +
+
+
+ + + shoots + + + + + + + + + +
+
+
+ Production Control Plane +
+
+
+ + + Production Control Plane + + + + + + + +
+
+
+ metal-stack +
+ kubernetes cluster +
+
+
+ + + metal-stack... + + + + + + + +
+
+
+ gardener +
+ kubernetes cluster +
+
+
+ + + gardener... + + + + + + + +
+
+
+ + Manages + +
+
+
+ + + Manages + + + + + + + + + + + +
+
+
+ Control Plane Partition +
+
+
+ + + Control Plane Partition + + + + + + + + +
+
+
+ backup of stateful sets +
+
+
+ + + backup of stateful sets + + + + + + + + + +
+
+
+ bare metal machine +
+
+
+ + + bare metal machine + + + + + + + +
+
+
+ metal-stack +
+ and +
+ gardener +
+ kubernetes cluster +
+ running in kind +
+
+
+ + + metal-stack... + + + + + + + +
+
+
+ + Manages + +
+
+
+ + + Manages + + + + + + + + +
+
+
+ S3 +
+
+
+ + + S3 + + + + + + + +
+
+
+ Needle +
+
+
+ + + Needle + + + + + + +
+
+
+ + Nail + +
+
+
+ + + Nail + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/src/development/proposals/MEP99/needle-control-plane-ha.drawio.svg b/docs/src/development/proposals/MEP99/needle-control-plane-ha.drawio.svg new file mode 100644 index 0000000000..51564fa19c --- /dev/null +++ b/docs/src/development/proposals/MEP99/needle-control-plane-ha.drawio.svg @@ -0,0 +1,229 @@ + + + + + + + + + +
+
+
+ Cluster interconnect +
+
+
+ + + Cluster interconnect + + + + + + + + + +
+
+
+ OutOfBand Connection +
+
+
+ + + OutOfBand Connection + + + + + + + + + + +
+
+
+ NVME +
+
+
+ + + NVME + + + + + + + + + + +
+
+
+ NVME +
+
+
+ + + NVME + + + + + + + +
+
+
+ Active Kind Cluster +
+ /dev/drbd0 -> /mnt/kind +
+
+
+ + + Active Kind Cluster... + + + + + + + + +
+
+
+ Sync +
+
+
+ + + Sync + + + + + + + +
+
+
+ drbd +
+ /dev/drbd0 +
+ from +
+ /dev/nvme0n1 +
+
+
+ + + drbd... + + + + + + + +
+
+
+ Stopped Kind Cluster +
+ /dev/drbd0 -> /mnt/kind +
+
+
+ + + Stopped Kind Cluster... + + + + + + + +
+
+
+ drbd +
+ /dev/drbd0 +
+ from +
+ /dev/nvme0n1 +
+
+
+ + + drbd... + + + + + + + +
+
+
+ Pacemaker +
+
+
+
+ + + Pacemaker + + + + + + + +
+
+
+ Pacemaker +
+
+
+
+ + + Pacemaker + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/src/development/proposals/MEP99/needle-control-plane.drawio.svg b/docs/src/development/proposals/MEP99/needle-control-plane.drawio.svg new file mode 100644 index 0000000000..cb5f464b9c --- /dev/null +++ b/docs/src/development/proposals/MEP99/needle-control-plane.drawio.svg @@ -0,0 +1,217 @@ + + + + + + + +
+
+
+ Needle Server +
+
+
+ + + Needle Server + + + + + + + +
+
+
+ kind: needle-control-plane +
+
+
+ + + kind: needle-control-plane + + + + + + + +
+
+
+ kind: services +
+
+
+ + + kind: services + + + + + + + +
+
+
+ metal-api +
+
+
+ + + metal-api + + + + + + + +
+
+
+ gardener +
+
+
+ + + gardener + + + + + + + +
+
+
+ powerdns +
+
+
+ + + powerdns + + + + + + + +
+
+
+ boulder +
+
+
+ + + boulder + + + + + + + +
+
+
+ gardener-extension-provider-metal +
+
+
+ + + gardener-extension-provider-metal + + + + + + + +
+
+
+ gardener-extension-dns-powerdns +
+
+
+ + + gardener-extension-dns-powerdns + + + + + + + +
+
+
+ gardener-extension-s3 +
+
+
+ + + gardener-extension-s3 + + + + + + + +
+
+
+ more ... +
+
+
+ + + more ... + + + + + + + +
+
+
+ gitlab/github runner container +
+
+
+ + + gitlab/github runner contain... + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file diff --git a/docs/src/development/proposals/MEP99/needle-partition.drawio.svg b/docs/src/development/proposals/MEP99/needle-partition.drawio.svg new file mode 100644 index 0000000000..483484f4b6 --- /dev/null +++ b/docs/src/development/proposals/MEP99/needle-partition.drawio.svg @@ -0,0 +1,81 @@ + + + + + + + +
+
+
+ Needle Partition +
+
+
+ + + Needle Partition + + + + + + + +
+
+
+ Needle Seed +
+
+
+ + + Needle Seed + + + + + + + +
+
+
+ GardenCluster Shoot +
+
+
+ + + GardenCluster Shoot + + + + + + + +
+
+
+ MetalCluster Shoot +
+
+
+ + + MetalCluster Shoot + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file From e444e6d2cfd949e4a0715587731824313a34bd69 Mon Sep 17 00:00:00 2001 From: Stefan Majer Date: Fri, 29 Nov 2024 10:16:26 +0100 Subject: [PATCH 2/7] Physical view --- .../src/development/proposals/MEP99/README.md | 6 + .../proposals/MEP99/needle-rack.drawio.svg | 228 ++++++++++++++++++ 2 files changed, 234 insertions(+) create mode 100644 docs/src/development/proposals/MEP99/needle-rack.drawio.svg diff --git a/docs/src/development/proposals/MEP99/README.md b/docs/src/development/proposals/MEP99/README.md index e5e1cbd526..a5a1efa464 100644 --- a/docs/src/development/proposals/MEP99/README.md +++ b/docs/src/development/proposals/MEP99/README.md @@ -128,8 +128,14 @@ resource needle-control-plane { TODO: Pacemaker or Heartbeat ? TODO: LVM Volumes +Logical View + ![needle-control-plane-ha](needle-control-plane-ha.drawio.svg) +Physical View + +![needle-rack](needle-rack.drawio.svg) + 1. Partition The partition which is managed by the metal-stack `needle` can be a simple and small hardware setup but yet capable enough to host the metal-stack `nail` control plane. diff --git a/docs/src/development/proposals/MEP99/needle-rack.drawio.svg b/docs/src/development/proposals/MEP99/needle-rack.drawio.svg new file mode 100644 index 0000000000..e85e56e3e6 --- /dev/null +++ b/docs/src/development/proposals/MEP99/needle-rack.drawio.svg @@ -0,0 +1,228 @@ + + + + + + + + + + + + + + + 1 + + + + + 2 + + + + + 3 + + + + + 4 + + + + + 5 + + + + + 6 + + + + + 7 + + + + + 8 + + + + + 9 + + + + + 10 + + + + + 11 + + + + + 12 + + + + + + + + + +
+
+
+ internet-router-management +
+
+
+ + + internet-router-management + + + + + + + + + +
+
+
+ management-switch-and-server +
+
+
+ + + management-switch-and-server + + + + + + + + + +
+
+
+ leaf01 +
+
+
+ + + leaf01 + + + + + + + + + +
+
+
+ leaf02 +
+
+
+ + + leaf02 + + + + + + + + + + + + + +
+
+
+ needle01 +
+
+
+ + + needle01 + + + + + + + + + + + + + +
+
+
+ needle02 +
+
+
+ + + needle02 + + + + + + + + + + + + + +
+
+
+ mirocloud +
+
+
+ + + mirocloud + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file From 046f4d5866c9ca0a0bbd875fec963d9334df922c Mon Sep 17 00:00:00 2001 From: Stefan Majer Date: Fri, 29 Nov 2024 10:49:15 +0100 Subject: [PATCH 3/7] Physical view --- docs/src/development/proposals/MEP99/README.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/src/development/proposals/MEP99/README.md b/docs/src/development/proposals/MEP99/README.md index a5a1efa464..2bc1643379 100644 --- a/docs/src/development/proposals/MEP99/README.md +++ b/docs/src/development/proposals/MEP99/README.md @@ -132,10 +132,15 @@ Logical View ![needle-control-plane-ha](needle-control-plane-ha.drawio.svg) -Physical View +Physical View, minimal ha setup which is only suitable for 1 Seed and 1 Shoot ![needle-rack](needle-rack.drawio.svg) +Physical View, bigger ha setup which is spread to two datacenters, capable to create 1 Seed with 3 nodes and 2 Shoots with 3 nodes each and still 2 waiting machines. + +![needle-rack-big](needle-rack-big.drawio.svg) + + 1. Partition The partition which is managed by the metal-stack `needle` can be a simple and small hardware setup but yet capable enough to host the metal-stack `nail` control plane. From aeca0eecad3ae0cb34516ef8f275fbe39f652a20 Mon Sep 17 00:00:00 2001 From: Stefan Majer Date: Fri, 29 Nov 2024 10:55:43 +0100 Subject: [PATCH 4/7] Physical view --- .../src/development/proposals/MEP99/README.md | 2 + .../MEP99/needle-rack-big.drawio.svg | 435 ++++++++++++++++++ 2 files changed, 437 insertions(+) create mode 100644 docs/src/development/proposals/MEP99/needle-rack-big.drawio.svg diff --git a/docs/src/development/proposals/MEP99/README.md b/docs/src/development/proposals/MEP99/README.md index 2bc1643379..c468f19b7d 100644 --- a/docs/src/development/proposals/MEP99/README.md +++ b/docs/src/development/proposals/MEP99/README.md @@ -1,5 +1,7 @@ # Solve the bootstrap problem +TODO: this is not a MEP, better put it into installation + Setting up a metal-stack.io environment in your own datacenter requires a control plane to be present which hosts the metal-stack api. If you plan to spin up kubernetes clusters, either with gardener.cloud or cluster api, the requirement for this control plane raises. The control plane must be running in a kubernetes cluster, which offers at least the following features: diff --git a/docs/src/development/proposals/MEP99/needle-rack-big.drawio.svg b/docs/src/development/proposals/MEP99/needle-rack-big.drawio.svg new file mode 100644 index 0000000000..01907e6d44 --- /dev/null +++ b/docs/src/development/proposals/MEP99/needle-rack-big.drawio.svg @@ -0,0 +1,435 @@ + + + + + + + + + + + + + + + 1 + + + + + 2 + + + + + 3 + + + + + 4 + + + + + 5 + + + + + 6 + + + + + 7 + + + + + 8 + + + + + 9 + + + + + 10 + + + + + 11 + + + + + 12 + + + + + + + + + +
+
+
+ internet-router-management +
+
+
+ + + internet-router-management + + + + + + + + + +
+
+
+ management-switch-and-server +
+
+
+ + + management-switch-and-server + + + + + + + + + +
+
+
+ spine01 +
+
+
+ + + spine01 + + + + + + + + + +
+
+
+ leaf01 +
+
+
+ + + leaf01 + + + + + + + + + +
+
+
+ leaf02 +
+
+
+ + + leaf02 + + + + + + + + + + + + + +
+
+
+ mirocloud +
+
+
+ + + mirocloud + + + + + + + + + + + + + +
+
+
+ needle01 +
+
+
+ + + needle01 + + + + + + + + + + + + + + + 1 + + + + + 2 + + + + + 3 + + + + + 4 + + + + + 5 + + + + + 6 + + + + + 7 + + + + + 8 + + + + + 9 + + + + + 10 + + + + + 11 + + + + + 12 + + + + + + + + + +
+
+
+ internet-router-management +
+
+
+ + + internet-router-management + + + + + + + + + +
+
+
+ management-switch-and-server +
+
+
+ + + management-switch-and-server + + + + + + + + + +
+
+
+ spine02 +
+
+
+ + + spine02 + + + + + + + + + +
+
+
+ leaf01 +
+
+
+ + + leaf01 + + + + + + + + + +
+
+
+ leaf02 +
+
+
+ + + leaf02 + + + + + + + + + + + + + +
+
+
+ needle02 +
+
+
+ + + needle02 + + + + + + + + + + + + + +
+
+
+ mirocloud +
+
+
+ + + mirocloud + + + + + + + + + Text is not SVG - cannot display + + + + \ No newline at end of file From 2467515007737d378f41ddf9555277a678e679a9 Mon Sep 17 00:00:00 2001 From: Stefan Majer Date: Tue, 3 Dec 2024 10:55:01 +0100 Subject: [PATCH 5/7] More references --- docs/src/development/proposals/MEP99/README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/src/development/proposals/MEP99/README.md b/docs/src/development/proposals/MEP99/README.md index c468f19b7d..6c5683a693 100644 --- a/docs/src/development/proposals/MEP99/README.md +++ b/docs/src/development/proposals/MEP99/README.md @@ -164,6 +164,12 @@ TODO: Where to connect the `needle` servers - Naming of the metal-stack chain elements, is `needle` and `nail` appropriate ? - Storage in the `needle` partition + - https://min.io/docs/directpv + - lightOS + - https://github.com/poettering/diskomator --> Crazy + - https://jing.rocks/2023/06/13/Experimenting-with-NVMe-over-TCP.html from the needle, also replicated with drbd ? - S3 Object storage is considered as provided - AirGapped is out of scope for now - IP address ranges and families +- Consider https://github.com/gardener/gardener/blob/master/docs/proposals/28-autonomous-shoot-clusters.md for the `needle` `seed` +- Take a look at: https://github.com/robgil/microdatacenter \ No newline at end of file From 1c51d79c533967d804c8c071b73c9a21dcc2656c Mon Sep 17 00:00:00 2001 From: Stefan Majer Date: Tue, 3 Dec 2024 11:30:44 +0100 Subject: [PATCH 6/7] More storage work --- docs/src/development/proposals/MEP99/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/src/development/proposals/MEP99/README.md b/docs/src/development/proposals/MEP99/README.md index 6c5683a693..40629d6f5a 100644 --- a/docs/src/development/proposals/MEP99/README.md +++ b/docs/src/development/proposals/MEP99/README.md @@ -164,10 +164,12 @@ TODO: Where to connect the `needle` servers - Naming of the metal-stack chain elements, is `needle` and `nail` appropriate ? - Storage in the `needle` partition - - https://min.io/docs/directpv + - https://min.io/docs/directpv --> new to me, dont know exactly how this works, looks interesting - lightOS - https://github.com/poettering/diskomator --> Crazy - - https://jing.rocks/2023/06/13/Experimenting-with-NVMe-over-TCP.html from the needle, also replicated with drbd ? + - the needle server as initiator, maybe also replicated with drbd ? + - https://ssdcentral.net/getting-started-with-nvme-over-fabrics-with-tcp/ + - https://jing.rocks/2023/06/13/Experimenting-with-NVMe-over-TCP.html - S3 Object storage is considered as provided - AirGapped is out of scope for now - IP address ranges and families From 6bef3a6fb54efe47938228d30a67311f08ee4643 Mon Sep 17 00:00:00 2001 From: Stefan Majer Date: Wed, 4 Dec 2024 15:19:22 +0100 Subject: [PATCH 7/7] move to installation --- .../metal-stack-chain.drawio.svg | 0 .../needle-control-plane-ha.drawio.svg | 0 .../needle-control-plane.drawio.svg | 0 .../needle-partition.drawio.svg | 0 .../needle-rack-big.drawio.svg | 0 .../needle-rack.drawio.svg | 0 .../autonomous-control-plane.md} | 13 ++++++------- 7 files changed, 6 insertions(+), 7 deletions(-) rename docs/src/{development/proposals/MEP99 => installation/autonomous-control-plane-images}/metal-stack-chain.drawio.svg (100%) rename docs/src/{development/proposals/MEP99 => installation/autonomous-control-plane-images}/needle-control-plane-ha.drawio.svg (100%) rename docs/src/{development/proposals/MEP99 => installation/autonomous-control-plane-images}/needle-control-plane.drawio.svg (100%) rename docs/src/{development/proposals/MEP99 => installation/autonomous-control-plane-images}/needle-partition.drawio.svg (100%) rename docs/src/{development/proposals/MEP99 => installation/autonomous-control-plane-images}/needle-rack-big.drawio.svg (100%) rename docs/src/{development/proposals/MEP99 => installation/autonomous-control-plane-images}/needle-rack.drawio.svg (100%) rename docs/src/{development/proposals/MEP99/README.md => installation/autonomous-control-plane.md} (95%) diff --git a/docs/src/development/proposals/MEP99/metal-stack-chain.drawio.svg b/docs/src/installation/autonomous-control-plane-images/metal-stack-chain.drawio.svg similarity index 100% rename from docs/src/development/proposals/MEP99/metal-stack-chain.drawio.svg rename to docs/src/installation/autonomous-control-plane-images/metal-stack-chain.drawio.svg diff --git a/docs/src/development/proposals/MEP99/needle-control-plane-ha.drawio.svg b/docs/src/installation/autonomous-control-plane-images/needle-control-plane-ha.drawio.svg similarity index 100% rename from docs/src/development/proposals/MEP99/needle-control-plane-ha.drawio.svg rename to docs/src/installation/autonomous-control-plane-images/needle-control-plane-ha.drawio.svg diff --git a/docs/src/development/proposals/MEP99/needle-control-plane.drawio.svg b/docs/src/installation/autonomous-control-plane-images/needle-control-plane.drawio.svg similarity index 100% rename from docs/src/development/proposals/MEP99/needle-control-plane.drawio.svg rename to docs/src/installation/autonomous-control-plane-images/needle-control-plane.drawio.svg diff --git a/docs/src/development/proposals/MEP99/needle-partition.drawio.svg b/docs/src/installation/autonomous-control-plane-images/needle-partition.drawio.svg similarity index 100% rename from docs/src/development/proposals/MEP99/needle-partition.drawio.svg rename to docs/src/installation/autonomous-control-plane-images/needle-partition.drawio.svg diff --git a/docs/src/development/proposals/MEP99/needle-rack-big.drawio.svg b/docs/src/installation/autonomous-control-plane-images/needle-rack-big.drawio.svg similarity index 100% rename from docs/src/development/proposals/MEP99/needle-rack-big.drawio.svg rename to docs/src/installation/autonomous-control-plane-images/needle-rack-big.drawio.svg diff --git a/docs/src/development/proposals/MEP99/needle-rack.drawio.svg b/docs/src/installation/autonomous-control-plane-images/needle-rack.drawio.svg similarity index 100% rename from docs/src/development/proposals/MEP99/needle-rack.drawio.svg rename to docs/src/installation/autonomous-control-plane-images/needle-rack.drawio.svg diff --git a/docs/src/development/proposals/MEP99/README.md b/docs/src/installation/autonomous-control-plane.md similarity index 95% rename from docs/src/development/proposals/MEP99/README.md rename to docs/src/installation/autonomous-control-plane.md index 40629d6f5a..523b866f80 100644 --- a/docs/src/development/proposals/MEP99/README.md +++ b/docs/src/installation/autonomous-control-plane.md @@ -49,7 +49,7 @@ A high-level architecture consists of two metal-stack.io environments, one for t We could even use some names for this environments which match better to metal, like `needle` and nail. So, a `needle` metal-stack is used to create a `nail` metal-stack environment. -![metal-stack-chain](metal-stack-chain.drawio.svg) +![metal-stack-chain](autonomous-control-plane-images/metal-stack-chain.drawio.svg) The `needle` and the `nail` metal-stack have both a control plane and a set of physical bare metal machines they manage and operate on. @@ -91,7 +91,7 @@ A second kind cluster is started on this machine to host services which are requ - Optional: Letsencrypt [boulder](https://github.com/letsencrypt/boulder) as a certificate authority - ... -![needle-control-plane](needle-control-plane.drawio.svg) +![needle-control-plane](autonomous-control-plane-images/needle-control-plane.drawio.svg) 1.1. Control Plane High Availability @@ -132,16 +132,15 @@ TODO: LVM Volumes Logical View -![needle-control-plane-ha](needle-control-plane-ha.drawio.svg) +![needle-control-plane-ha](autonomous-control-plane-images/needle-control-plane-ha.drawio.svg) Physical View, minimal ha setup which is only suitable for 1 Seed and 1 Shoot -![needle-rack](needle-rack.drawio.svg) +![needle-rack](autonomous-control-plane-images/needle-rack.drawio.svg) Physical View, bigger ha setup which is spread to two datacenters, capable to create 1 Seed with 3 nodes and 2 Shoots with 3 nodes each and still 2 waiting machines. -![needle-rack-big](needle-rack-big.drawio.svg) - +![needle-rack-big](autonomous-control-plane-images/needle-rack-big.drawio.svg) 1. Partition @@ -154,7 +153,7 @@ TODO: Small Lightbits, other appliance like Synology at.al. A seed must be created which is responsible for hosting the control planes of the shoots in this partition. The amount of shoots should be minimal, most of the time, two shoots, one for hosting gardener and one for metal-stack. -![needle-partition](needle-partition.drawio.svg) +![needle-partition](autonomous-control-plane-images/needle-partition.drawio.svg) 1. Network Diagram