diff --git a/dev/apidocs/apidocs/index.html b/dev/apidocs/apidocs/index.html index dbb4224512..f211cb0070 100644 --- a/dev/apidocs/apidocs/index.html +++ b/dev/apidocs/apidocs/index.html @@ -1,2 +1,2 @@ -API Documentation · metal-stack
Edit on GitHub
+API Documentation · metal-stack
Edit on GitHub
diff --git a/dev/development/client_libraries/index.html b/dev/development/client_libraries/index.html index 10a1489bae..636bb76613 100644 --- a/dev/development/client_libraries/index.html +++ b/dev/development/client_libraries/index.html @@ -1,2 +1,2 @@ -Client Libraries · metal-stack
Edit on GitHub
+Client Libraries · metal-stack
Edit on GitHub
diff --git a/dev/development/contributing/index.html b/dev/development/contributing/index.html index f8fc0169be..be303532b0 100644 --- a/dev/development/contributing/index.html +++ b/dev/development/contributing/index.html @@ -1,2 +1,2 @@ -Contributing · metal-stack
Edit on GitHub

Contributing

This document describes the way we want to contribute code to the projects of metal-stack, which are hosted on github.com/metal-stack.

The document is meant to be understood as a general guideline for contributions, but not as burden to be placed on a developer. Use your best judgment when contributing code. Try to be as clean and precise as possible when writing code and try to make your code as maintainable and understandable as possible for other people.

Even if it should go without saying, we live an open culture of discussion, in which everybody is welcome to participate. We treat every contribution with respect and objectiveness with the general aim to write software of quality.

If you want, feel free to propose changes to this document in a pull request.

How Can I Contribute?

Open a Github issue in the project you would like to contribute. Within the issue, your idea can be discussed. It is also possible to directly create a pull request when the set of changes is relatively small.

Pull Requests

The process described here has several goals:

  • Maintain quality
  • Enable a sustainable system to review contributions
  • Enable documented and reproducible addition of contributions
  1. Create a meaningful issue describing the WHY? of your contribution
  2. Create a repository fork within the context of that issue.
  3. Create a Draft Pull Request to the master branch of the target repository.
  4. Develop, document and test your contribution (try not to solve more than one issue in a single pull request)
  5. Ask for merging your contribution by removing the draft marker
  6. If code owners are defined, try to assign the request to a code owner

General Objectives

This section contains language-agnostic topics that all metal-stack projects are trying to follow.

Code Ownership

The code base is owned by the entire team and every member is allowed to contribute changes to any of the projects. This is considered as collective code ownership[1].

As a matter of fact, there are persons in a project, which already have experience with the sources. These are defined directly in the repository's CODEOWNERS file. If you want to merge changes into the master branch, it is advisable to include code owners into the proecess of discussion and merging.

Microservices

One major ambition of metal-stack is to follow the idea of microservices. This way, we want to achieve that we can

  • adapt to changes faster than with monolithic architectures,
  • be free of restrictions due to certain choices of technology,
  • leverage powerful traits of cloud infrastructures (e.g. high-scalability, high-availability, ...).

Programming Languages

We are generally open to write code in any language that fits best to the function of the software. However, we encourage golang to be the main language of metal-stack as we think that it makes development faster when not establishing too many different languages in our architecture. Reason for this is that we are striving for consistent behavior of the microservices, similar to what has been described for the Twelve-Factor App (see 12 Factor). We help enforcing unified behavior by allowing a small layer of shared code for every programming language. We will refer to this shared code as "libraries" for the rest of this document.

Artifacts

Artifacts are always produced by a CI process (Github Actions).

Docker images are published on the Github Container Registry of the metal-stack organization.

Binary artifacts or OS images can be uploaded to images.metal-stack.io if necessary.

When building Docker images, please consider our build tool docker-make or the specific docker-make action respectively.

APIs

We are currently making use of Swagger when we exposing traditional REST APIs for end-users. This helps us with being technology-agnostic as we can generate clients in almost any language using go-swagger. Swagger additionally simplifies the documentation of our APIs.

Most APIs though are not required to be user-facing but are of technical nature. These are preferred to be implemented using grpc.

Versioning

Artifacts are versioned by tagging the respective repository with a tag starting with the letter v. After the letter, there stands a valid semantic version.

Documentation

In order to make it easier for others to understand a project, we document general information and usage instructions in a README.md in any project.

In addition to that, we document a microservice in the docs repository. The documentation should contain the reasoning why this service exists and why it was being implemented the way it was being implemented. The aim of this procedure is to reduce the time for contributors to comprehend architectural decisions that were made during the process of writing the software and to clarify the general purpose of this service in the entire context of the software.

Guidelines

This chapter describes general guidelines on how to develop and contribute code for a certain programming language.

Golang

Development follows the official guide to:

Development Decisions

  • Dependency Management by using Go modules
  • Build and Test Automation by using GNU Make.
  • End-user APIs should consider using go-swagger and Go-Restful Technical APIs should consider using grpc

Libraries

metal-stack maintains several libraries that you should utilize in your project in order unify common behavior. Some of these projects are:

Error Handling with Generated Swagger Clients

From the server-side you should ensure that you are returning the common error json struct in case of an error as defined in the metal-lib/httperrors. Ensure you are using go-restful >= v2.9.1 and go-restful-openapi >= v0.13.1 (allows default responses with error codes other than 200).

Documentation

We want to share knowledge and keep things simple. If things cannot kept simple we want enable everybody to understand them by:

  • Document in short sentences[4].
  • Do not explain the HOW (this is already documented by your code and documenting the obvious is considered a defect).
  • Explain the WHY. Add a "to" in your documentation line to force yourself to explain the reasonning (e.g. "<THE WHAT> to <THE TO>").

Python

Development follows the official guide to:

  • Style Guide for Python Code (PEP 8)[5]
    • The use of an IDE like PyCharm helps to write compliant code easily
  • Consider setuptools for packaging
  • If you want to add a Python microservice to the mix, consider pyinstaller on Alpine to achieve small image sizes
+Contributing · metal-stack
Edit on GitHub

Contributing

This document describes the way we want to contribute code to the projects of metal-stack, which are hosted on github.com/metal-stack.

The document is meant to be understood as a general guideline for contributions, but not as burden to be placed on a developer. Use your best judgment when contributing code. Try to be as clean and precise as possible when writing code and try to make your code as maintainable and understandable as possible for other people.

Even if it should go without saying, we live an open culture of discussion, in which everybody is welcome to participate. We treat every contribution with respect and objectiveness with the general aim to write software of quality.

If you want, feel free to propose changes to this document in a pull request.

How Can I Contribute?

Open a Github issue in the project you would like to contribute. Within the issue, your idea can be discussed. It is also possible to directly create a pull request when the set of changes is relatively small.

Pull Requests

The process described here has several goals:

  • Maintain quality
  • Enable a sustainable system to review contributions
  • Enable documented and reproducible addition of contributions
  1. Create a meaningful issue describing the WHY? of your contribution
  2. Create a repository fork within the context of that issue.
  3. Create a Draft Pull Request to the master branch of the target repository.
  4. Develop, document and test your contribution (try not to solve more than one issue in a single pull request)
  5. Ask for merging your contribution by removing the draft marker
  6. If code owners are defined, try to assign the request to a code owner

General Objectives

This section contains language-agnostic topics that all metal-stack projects are trying to follow.

Code Ownership

The code base is owned by the entire team and every member is allowed to contribute changes to any of the projects. This is considered as collective code ownership[1].

As a matter of fact, there are persons in a project, which already have experience with the sources. These are defined directly in the repository's CODEOWNERS file. If you want to merge changes into the master branch, it is advisable to include code owners into the proecess of discussion and merging.

Microservices

One major ambition of metal-stack is to follow the idea of microservices. This way, we want to achieve that we can

  • adapt to changes faster than with monolithic architectures,
  • be free of restrictions due to certain choices of technology,
  • leverage powerful traits of cloud infrastructures (e.g. high-scalability, high-availability, ...).

Programming Languages

We are generally open to write code in any language that fits best to the function of the software. However, we encourage golang to be the main language of metal-stack as we think that it makes development faster when not establishing too many different languages in our architecture. Reason for this is that we are striving for consistent behavior of the microservices, similar to what has been described for the Twelve-Factor App (see 12 Factor). We help enforcing unified behavior by allowing a small layer of shared code for every programming language. We will refer to this shared code as "libraries" for the rest of this document.

Artifacts

Artifacts are always produced by a CI process (Github Actions).

Docker images are published on the Github Container Registry of the metal-stack organization.

Binary artifacts or OS images can be uploaded to images.metal-stack.io if necessary.

When building Docker images, please consider our build tool docker-make or the specific docker-make action respectively.

APIs

We are currently making use of Swagger when we exposing traditional REST APIs for end-users. This helps us with being technology-agnostic as we can generate clients in almost any language using go-swagger. Swagger additionally simplifies the documentation of our APIs.

Most APIs though are not required to be user-facing but are of technical nature. These are preferred to be implemented using grpc.

Versioning

Artifacts are versioned by tagging the respective repository with a tag starting with the letter v. After the letter, there stands a valid semantic version.

Documentation

In order to make it easier for others to understand a project, we document general information and usage instructions in a README.md in any project.

In addition to that, we document a microservice in the docs repository. The documentation should contain the reasoning why this service exists and why it was being implemented the way it was being implemented. The aim of this procedure is to reduce the time for contributors to comprehend architectural decisions that were made during the process of writing the software and to clarify the general purpose of this service in the entire context of the software.

Guidelines

This chapter describes general guidelines on how to develop and contribute code for a certain programming language.

Golang

Development follows the official guide to:

Development Decisions

  • Dependency Management by using Go modules
  • Build and Test Automation by using GNU Make.
  • End-user APIs should consider using go-swagger and Go-Restful Technical APIs should consider using grpc

Libraries

metal-stack maintains several libraries that you should utilize in your project in order unify common behavior. Some of these projects are:

Error Handling with Generated Swagger Clients

From the server-side you should ensure that you are returning the common error json struct in case of an error as defined in the metal-lib/httperrors. Ensure you are using go-restful >= v2.9.1 and go-restful-openapi >= v0.13.1 (allows default responses with error codes other than 200).

Documentation

We want to share knowledge and keep things simple. If things cannot kept simple we want enable everybody to understand them by:

  • Document in short sentences[4].
  • Do not explain the HOW (this is already documented by your code and documenting the obvious is considered a defect).
  • Explain the WHY. Add a "to" in your documentation line to force yourself to explain the reasonning (e.g. "<THE WHAT> to <THE TO>").

Python

Development follows the official guide to:

  • Style Guide for Python Code (PEP 8)[5]
    • The use of an IDE like PyCharm helps to write compliant code easily
  • Consider setuptools for packaging
  • If you want to add a Python microservice to the mix, consider pyinstaller on Alpine to achieve small image sizes
diff --git a/dev/development/proposals/MEP1/README/index.html b/dev/development/proposals/MEP1/README/index.html index 6a83d5bf9b..d2e57f6621 100644 --- a/dev/development/proposals/MEP1/README/index.html +++ b/dev/development/proposals/MEP1/README/index.html @@ -1,2 +1,2 @@ -Distributed Metal Control Plane · metal-stack
Edit on GitHub

Distributed Metal Control Plane

Problem Statement

We face the situation that we argue for running bare metal on premise because this way the customers can control where and how their software and data are processed and stored. On the other hand, we have currently decided that our metal-api control plane components run on a kubernetes cluster (in our case on a cluster provided by one of the available hyperscalers).

Running the control plane on Kubernetes has the following benefits:

  • Ease of deployment
  • Get most, if not all, of the required infrastructure services like (probably incomplete):
    • IPs
    • DNS
    • L7-Loadbalancing
    • Storage
    • S3 Backup
    • High Availability

Using a kubernetes as a service offering from one of the hyperscalers, enables us to focus on using kubernetes instead of maintaining it as well.

Goal

It would be much saner if metal-stack has no, or only minimal dependencies to external services. Imagine a metal-stack deployment in a plant, it would be optimal if we only have to deliver a single rack with servers and networking gear installed and wired, plug that rack to the power supply and a internet uplink and its ready to go.

Have a second plant which you want to be part of all your plants? Just tell both that they are part of something bigger and metal-api knows of two partitions.

Possible Solutions

We can think of two different solutions to this vision:

  1. Keep the central control plane approach and require some sort of kubernetes deployment accessible from the internet. This has the downside that the user must, provide a managed kubernetes deployment in his own datacenter or uses a hyperscaler. Still not optimal.
  2. Install the metal-api and all its dependencies in every partition, replicate or shard the databases to every connected partition, make them know each other. Connect the partitions over the internet with some sort of vpn to make the services visible to each other.

As we can see, the first approach does not really address the problem, therefore i will describe solution #2 in more details.

Central/Current setup

Stateful services

Every distributed system suffer from handling state in a scalable, fast and correct way. To start how to cope with the state, we first must identify which state can be seen as partition local only and which state must be synchronous for read, and synchronous for writes across partitions.

Affected states:

  • masterdata: e.g. tenant and project must be present in every partition, but these are entities which are read often but updates are rare. A write can therefore be visible with a decent delay in a distinct partition with no consequences.
  • ipam: the prefixes and ip´s allocated from machines. These entities are also read often and rare updates. But we must differentiate between dirty reads for different types. A machine network is partition local, ips acquired from such a network must by synchronous in the same partition. Ips acquired from global networks such as internet must by synchronous for all partitions, as otherwise a internet ip could be acquired twice.
  • vrf ids: they must only be unique in one partition
  • image and size configurations: read often, written seldom, so no high requirements on the storage of these entities.
  • images: os images are already replicated from a central s3 storage to a per partition s3 service. metal-hammer kernel and initrd are small and pull always from the central s3, can be done similar to os images.
  • machine and machine allocation: must be only synchronous in the partition
  • switch: must be only synchronous in the partition
  • nsq messages: do not need to cross partition boundaries. No need to keep the messages persistent, even the opposite is true, we don't want to have the messages persist for a longer period.

Now we can see that the most critical state to held and synchronize are the IPAM data, because these entities must be guaranteed to be synchronously updated, while being updated frequently.

Datastores:

We use three different types of datastores to persist the states of the metal application.

  • rethinkdb is the main datastore for almost all entities managed by metal-api
  • postgresql is used for masterdata and ipam data.
  • nsq uses disk and memory tho store the messages.

Stateless services

These are the easy part, all of our services which are stateless can be scaled up and down without any impact on functionality. Even the stateful services like masterdata and metal-api rely fully on the underlying datastore and can therefore also be scaled up and down to meet scalability requirements.

Albeit, most of these services need to be placed behind a loadbalancer which does the L4/L7 balancing across the started/available replicas of the service for the clients talking to it. This is actually provided by kubernetes with either service type loadbalancer or type clusterip.

One exception is the metal-console service which must have the partition in it´s dns name now, because there is no direct network connectivity between the management networks of the partitions. See "Network Setup)

Distributed setup

State

In order to replicate certain data which must be available across all partitions we can use on of the existing open source databases which enable such kind of setup. There are a few avaible out there, the following uncomplete list will highlight the pro´s and cons of each.

  • RethinkDB

    We already store most of our data in RethinkDB and it gives already the ability to synchronize the data in a distributed manner with different guarantees for consistency and latency. This is described here: Scaling, Sharding and replication. But because rethinkdb has a rough history and unsure future with the last release took more than a year, we in the team already thought that we eventually must move away from rethinkdb in the future.

  • Postgresql

    Postgres does not have a multi datacenter with replication in both directions, it just can make the remote instance store the same data.

  • CockroachDB

    Is a Postgresql compatible database enginge on the wire. CockroachDB gives you both, ACID and geo replication with writes allowed from all connected members. It is even possible to configure Follow the Workload and Geo Partitioning and Replication.

If we migrate all metal-api entities to be stored the same way we store masterdata, we could use cockroachdb to store all metal entities in one ore more databases spread across all partitions and still ensure consistency and high availability.

A simple setup how this would look like is shown here.

Simple CockroachDB setup

go-ipam was modified in a example PR here: PR 17

API Access

In order to make the metal-api accessible for api users like cloud-api or metalctl as easy at it is today, some effort has to be taken. One possible approach would be to use a external loadbalancer which spread the requests evenly to all metal-api endpoints in all partitions. Because all data are accessible from all partitions, a api request going to partition A with a request to create a machine in partition B, will still work. If on the other hand partition B is not in a connected state because the interconnection between both partitions is broken, then of course the request will fail.

IMPORTANT The NSQ Message to inform metal-core must end in the correct partition

To provide such a external loadbalancer we have several opportunities:

  • Cloudflare or comparable CDN service.
  • BGP Anycast from every partition

Another setup would place a small gateway behind the metal-api address, which forwards to the metal-api in the partition where the request must be executed. This gateway, metal-api-router must inspect the payload, extract the desired partition, and forward the request without any modifications to the metal-api endpoint in this partition. This can be done for all requests, or if we want to optimize, only for write accesses.

Network setup

In order to have the impact to the overall security concept as minimal as possible i would not modify the current network setup. The only modifications which has to be made are:

  • Allow https ingress traffic to all metal-api instances.
  • Allow ssh ingress traffic to all metal-console instances.
  • Allow CockroachDB Replication between all partitions.
  • No NSQ traffic from outside required anymore, except we cant solve the topic above.

A simple setup how this would look like is shown here, this does not work though because of the forementioned NSQ issue.

API and Console Access

Therefore we need the metal-api-router:

Working API and Console Access

Deployment

The deployment of our components will substantially differ in a partition compared to a the deployment we have actually. Deploying it in kubernetes in the partition would be very difficult to achieve because we have no sane way to deploy kubernetes on physical machines without a underlying API. I would therefore suggest to deploy our components in the same way we do that for the services running on the management server. Use systemd to start docker containers.

Deployment

+Distributed Metal Control Plane · metal-stack
Edit on GitHub

Distributed Metal Control Plane

Problem Statement

We face the situation that we argue for running bare metal on premise because this way the customers can control where and how their software and data are processed and stored. On the other hand, we have currently decided that our metal-api control plane components run on a kubernetes cluster (in our case on a cluster provided by one of the available hyperscalers).

Running the control plane on Kubernetes has the following benefits:

  • Ease of deployment
  • Get most, if not all, of the required infrastructure services like (probably incomplete):
    • IPs
    • DNS
    • L7-Loadbalancing
    • Storage
    • S3 Backup
    • High Availability

Using a kubernetes as a service offering from one of the hyperscalers, enables us to focus on using kubernetes instead of maintaining it as well.

Goal

It would be much saner if metal-stack has no, or only minimal dependencies to external services. Imagine a metal-stack deployment in a plant, it would be optimal if we only have to deliver a single rack with servers and networking gear installed and wired, plug that rack to the power supply and a internet uplink and its ready to go.

Have a second plant which you want to be part of all your plants? Just tell both that they are part of something bigger and metal-api knows of two partitions.

Possible Solutions

We can think of two different solutions to this vision:

  1. Keep the central control plane approach and require some sort of kubernetes deployment accessible from the internet. This has the downside that the user must, provide a managed kubernetes deployment in his own datacenter or uses a hyperscaler. Still not optimal.
  2. Install the metal-api and all its dependencies in every partition, replicate or shard the databases to every connected partition, make them know each other. Connect the partitions over the internet with some sort of vpn to make the services visible to each other.

As we can see, the first approach does not really address the problem, therefore i will describe solution #2 in more details.

Central/Current setup

Stateful services

Every distributed system suffer from handling state in a scalable, fast and correct way. To start how to cope with the state, we first must identify which state can be seen as partition local only and which state must be synchronous for read, and synchronous for writes across partitions.

Affected states:

  • masterdata: e.g. tenant and project must be present in every partition, but these are entities which are read often but updates are rare. A write can therefore be visible with a decent delay in a distinct partition with no consequences.
  • ipam: the prefixes and ip´s allocated from machines. These entities are also read often and rare updates. But we must differentiate between dirty reads for different types. A machine network is partition local, ips acquired from such a network must by synchronous in the same partition. Ips acquired from global networks such as internet must by synchronous for all partitions, as otherwise a internet ip could be acquired twice.
  • vrf ids: they must only be unique in one partition
  • image and size configurations: read often, written seldom, so no high requirements on the storage of these entities.
  • images: os images are already replicated from a central s3 storage to a per partition s3 service. metal-hammer kernel and initrd are small and pull always from the central s3, can be done similar to os images.
  • machine and machine allocation: must be only synchronous in the partition
  • switch: must be only synchronous in the partition
  • nsq messages: do not need to cross partition boundaries. No need to keep the messages persistent, even the opposite is true, we don't want to have the messages persist for a longer period.

Now we can see that the most critical state to held and synchronize are the IPAM data, because these entities must be guaranteed to be synchronously updated, while being updated frequently.

Datastores:

We use three different types of datastores to persist the states of the metal application.

  • rethinkdb is the main datastore for almost all entities managed by metal-api
  • postgresql is used for masterdata and ipam data.
  • nsq uses disk and memory tho store the messages.

Stateless services

These are the easy part, all of our services which are stateless can be scaled up and down without any impact on functionality. Even the stateful services like masterdata and metal-api rely fully on the underlying datastore and can therefore also be scaled up and down to meet scalability requirements.

Albeit, most of these services need to be placed behind a loadbalancer which does the L4/L7 balancing across the started/available replicas of the service for the clients talking to it. This is actually provided by kubernetes with either service type loadbalancer or type clusterip.

One exception is the metal-console service which must have the partition in it´s dns name now, because there is no direct network connectivity between the management networks of the partitions. See "Network Setup)

Distributed setup

State

In order to replicate certain data which must be available across all partitions we can use on of the existing open source databases which enable such kind of setup. There are a few avaible out there, the following uncomplete list will highlight the pro´s and cons of each.

  • RethinkDB

    We already store most of our data in RethinkDB and it gives already the ability to synchronize the data in a distributed manner with different guarantees for consistency and latency. This is described here: Scaling, Sharding and replication. But because rethinkdb has a rough history and unsure future with the last release took more than a year, we in the team already thought that we eventually must move away from rethinkdb in the future.

  • Postgresql

    Postgres does not have a multi datacenter with replication in both directions, it just can make the remote instance store the same data.

  • CockroachDB

    Is a Postgresql compatible database enginge on the wire. CockroachDB gives you both, ACID and geo replication with writes allowed from all connected members. It is even possible to configure Follow the Workload and Geo Partitioning and Replication.

If we migrate all metal-api entities to be stored the same way we store masterdata, we could use cockroachdb to store all metal entities in one ore more databases spread across all partitions and still ensure consistency and high availability.

A simple setup how this would look like is shown here.

Simple CockroachDB setup

go-ipam was modified in a example PR here: PR 17

API Access

In order to make the metal-api accessible for api users like cloud-api or metalctl as easy at it is today, some effort has to be taken. One possible approach would be to use a external loadbalancer which spread the requests evenly to all metal-api endpoints in all partitions. Because all data are accessible from all partitions, a api request going to partition A with a request to create a machine in partition B, will still work. If on the other hand partition B is not in a connected state because the interconnection between both partitions is broken, then of course the request will fail.

IMPORTANT The NSQ Message to inform metal-core must end in the correct partition

To provide such a external loadbalancer we have several opportunities:

  • Cloudflare or comparable CDN service.
  • BGP Anycast from every partition

Another setup would place a small gateway behind the metal-api address, which forwards to the metal-api in the partition where the request must be executed. This gateway, metal-api-router must inspect the payload, extract the desired partition, and forward the request without any modifications to the metal-api endpoint in this partition. This can be done for all requests, or if we want to optimize, only for write accesses.

Network setup

In order to have the impact to the overall security concept as minimal as possible i would not modify the current network setup. The only modifications which has to be made are:

  • Allow https ingress traffic to all metal-api instances.
  • Allow ssh ingress traffic to all metal-console instances.
  • Allow CockroachDB Replication between all partitions.
  • No NSQ traffic from outside required anymore, except we cant solve the topic above.

A simple setup how this would look like is shown here, this does not work though because of the forementioned NSQ issue.

API and Console Access

Therefore we need the metal-api-router:

Working API and Console Access

Deployment

The deployment of our components will substantially differ in a partition compared to a the deployment we have actually. Deploying it in kubernetes in the partition would be very difficult to achieve because we have no sane way to deploy kubernetes on physical machines without a underlying API. I would therefore suggest to deploy our components in the same way we do that for the services running on the management server. Use systemd to start docker containers.

Deployment

diff --git a/dev/development/proposals/MEP10/README/index.html b/dev/development/proposals/MEP10/README/index.html index e586b9c7f7..0ea193e84e 100644 --- a/dev/development/proposals/MEP10/README/index.html +++ b/dev/development/proposals/MEP10/README/index.html @@ -84,4 +84,4 @@ "mgmtVrfEnabled": "true" } } -}

IP forwarding is deactivated on eth0, and no IP Masquerade is configured.

+}

IP forwarding is deactivated on eth0, and no IP Masquerade is configured.

diff --git a/dev/development/proposals/MEP11/README/index.html b/dev/development/proposals/MEP11/README/index.html index b8aab21a01..cd934fc084 100644 --- a/dev/development/proposals/MEP11/README/index.html +++ b/dev/development/proposals/MEP11/README/index.html @@ -1,2 +1,2 @@ -Auditing of metal-stack resources · metal-stack
Edit on GitHub

Auditing of metal-stack resources

Currently no logs of the ownership of resources like machines, networks, ips and volumes are generated or kept. Though due to legal requirements data centers are required to keep track of this ownership over time to prevent liability issues when opening the platform for external users.

In this proposal we want to introduce a flexible and low-maintenance approach for auditing on top of Meilisearch.

Overview

In general our auditing logs will be collected by a request interceptor or middleware. Every request and response will be processed and eventually logged to Meilisearch. Meilisearch will be configured to regularly create chunks of the auditing logs. These finished chunks will be backed up to a S3 compatible storage with a read-only option enabled.

Of course sensitve data like session keys or passwords will be redacted before logging. We want to track relevant requests and responses. If auditing the request fails, the request itself will be aborted and will not be processed further. The requests and responses that will be audited will be annotated with a correlation id.

Transferring the meilisearch auditing data chunks to the S3 compatible storage will be done by a sidecar cronjob that is executed periodically. To avoid data manipulation the S3 compatible storage will be configured to be read-only.

Whitelisting

To reduce the amount of unnecessary logs we want to introduce a whitelist of resources and operations on those that should be logged. Other requests will be passed directly to the next middleware or web service without any further processing.

As we are only interested in mutating endpoints, we ignore all GET requests. The whitelist includes all POST, PUT, PATCH and DELETE endpoints of the HTTP middleware except for the following (non-manipulating) route suffixes:

  • /find
  • /notify
  • /try and /match
  • /capacity
  • /from-hardware

Regarding GRPC audit trails, they are not so interesting because only internal clients are using this API. However, we can log the trails of the Boot service, which can be interesting to revise the machine lifecycle.

Chunking in Meilisearch

We want our data to be chunked in Meilisearch. To accomplish this, we rotate the index identifier on a scheduled basis. The index identifiers will be derived from the current date and time.

To keep things simple, we only support hourly, daily and monthly rotation. The eventually prefixed index names will only include relevant parts of date and time like 2021-01, 2021-01-01 or 2021-01-01_13.

The metal-api will only write to the current index and switches to the new index on rotation. The metal-api will never read or update data in any indices.

Moving chunks to S3 compatible storage

As Meilisearch will be filled with data over time, we want to move completed chunks to a S3 compatible storage. This will be done by a sidecar cronjob that is executed periodically. Note that the periods of the index rotation and the cronjob execution don't have to match.

When the backup process gets started, it initiates a Meilisearch dump of the whole database across all indices. Once the returned task is finished, the dump must be copied from a Meilisearch volume to the S3 compatible storage. After a successful copy, the dump can be deleted.

Now we want to remove all indices from Meilisearch, except the most recent one. For this, we get all indices, sort them and delete each index except the most recent one to avoid data loss.

For the actual implementation, we can build upon backup-restore-sidecar. But due to the index rotation and the fact, that older indices need to be deleted, this probably does not fit into the mentioned sidecar.

S3 compatible storage

The dumps of chunks should automatically deleted after a certain amount of time, once we are either no longer allowed or required to keep them. The default retention time will be 6 months. Ideally already uploaded chunks should be read-only to prevent data manipulation.

A candidate for the S3 compatible storage is Google Cloud Storage, which allows to configure automatic expiration of objects through a lifecycle rule.

Affected components

  • metal-api grpc server needs an auditing interceptor
  • metal-api web server needs an auditing filter chain / middleware
  • metal-api needs new command line arguments to configure the auditing
  • mini-lab needs a Meilisearch instance
  • mini-lab may need a local S3 compatible storage
  • we need a sidecar to implement the backup to S3 compatible storage
  • Consider auditing of volume allocations and freeings outside of metal-stack

Alternatives considered

Instead of using Meilisearch we investigated using an immutable database like immudb. But immudb does not support chunking of data and due to its immutable nature, we will never be able to free up space of expired data. Even if we are legally allowed or required to delete data, we will not be able to do so with immudb.

In another variant of the Meilisearch approach the metal-api would also be responsible for copying chunks to the S3 compatible storage and deleting old indices. But separating the concerns allows completely different implementations for every deployment stage.

+Auditing of metal-stack resources · metal-stack
Edit on GitHub

Auditing of metal-stack resources

Currently no logs of the ownership of resources like machines, networks, ips and volumes are generated or kept. Though due to legal requirements data centers are required to keep track of this ownership over time to prevent liability issues when opening the platform for external users.

In this proposal we want to introduce a flexible and low-maintenance approach for auditing on top of Meilisearch.

Overview

In general our auditing logs will be collected by a request interceptor or middleware. Every request and response will be processed and eventually logged to Meilisearch. Meilisearch will be configured to regularly create chunks of the auditing logs. These finished chunks will be backed up to a S3 compatible storage with a read-only option enabled.

Of course sensitve data like session keys or passwords will be redacted before logging. We want to track relevant requests and responses. If auditing the request fails, the request itself will be aborted and will not be processed further. The requests and responses that will be audited will be annotated with a correlation id.

Transferring the meilisearch auditing data chunks to the S3 compatible storage will be done by a sidecar cronjob that is executed periodically. To avoid data manipulation the S3 compatible storage will be configured to be read-only.

Whitelisting

To reduce the amount of unnecessary logs we want to introduce a whitelist of resources and operations on those that should be logged. Other requests will be passed directly to the next middleware or web service without any further processing.

As we are only interested in mutating endpoints, we ignore all GET requests. The whitelist includes all POST, PUT, PATCH and DELETE endpoints of the HTTP middleware except for the following (non-manipulating) route suffixes:

  • /find
  • /notify
  • /try and /match
  • /capacity
  • /from-hardware

Regarding GRPC audit trails, they are not so interesting because only internal clients are using this API. However, we can log the trails of the Boot service, which can be interesting to revise the machine lifecycle.

Chunking in Meilisearch

We want our data to be chunked in Meilisearch. To accomplish this, we rotate the index identifier on a scheduled basis. The index identifiers will be derived from the current date and time.

To keep things simple, we only support hourly, daily and monthly rotation. The eventually prefixed index names will only include relevant parts of date and time like 2021-01, 2021-01-01 or 2021-01-01_13.

The metal-api will only write to the current index and switches to the new index on rotation. The metal-api will never read or update data in any indices.

Moving chunks to S3 compatible storage

As Meilisearch will be filled with data over time, we want to move completed chunks to a S3 compatible storage. This will be done by a sidecar cronjob that is executed periodically. Note that the periods of the index rotation and the cronjob execution don't have to match.

When the backup process gets started, it initiates a Meilisearch dump of the whole database across all indices. Once the returned task is finished, the dump must be copied from a Meilisearch volume to the S3 compatible storage. After a successful copy, the dump can be deleted.

Now we want to remove all indices from Meilisearch, except the most recent one. For this, we get all indices, sort them and delete each index except the most recent one to avoid data loss.

For the actual implementation, we can build upon backup-restore-sidecar. But due to the index rotation and the fact, that older indices need to be deleted, this probably does not fit into the mentioned sidecar.

S3 compatible storage

The dumps of chunks should automatically deleted after a certain amount of time, once we are either no longer allowed or required to keep them. The default retention time will be 6 months. Ideally already uploaded chunks should be read-only to prevent data manipulation.

A candidate for the S3 compatible storage is Google Cloud Storage, which allows to configure automatic expiration of objects through a lifecycle rule.

Affected components

  • metal-api grpc server needs an auditing interceptor
  • metal-api web server needs an auditing filter chain / middleware
  • metal-api needs new command line arguments to configure the auditing
  • mini-lab needs a Meilisearch instance
  • mini-lab may need a local S3 compatible storage
  • we need a sidecar to implement the backup to S3 compatible storage
  • Consider auditing of volume allocations and freeings outside of metal-stack

Alternatives considered

Instead of using Meilisearch we investigated using an immutable database like immudb. But immudb does not support chunking of data and due to its immutable nature, we will never be able to free up space of expired data. Even if we are legally allowed or required to delete data, we will not be able to do so with immudb.

In another variant of the Meilisearch approach the metal-api would also be responsible for copying chunks to the S3 compatible storage and deleting old indices. But separating the concerns allows completely different implementations for every deployment stage.

diff --git a/dev/development/proposals/MEP12/README/index.html b/dev/development/proposals/MEP12/README/index.html index e83f449cb5..1ba119a0d9 100644 --- a/dev/development/proposals/MEP12/README/index.html +++ b/dev/development/proposals/MEP12/README/index.html @@ -4,4 +4,4 @@ type MachineAllocation struct { // existing fields are omitted for readability PlacementTags []string `json:"placement_tags" description:"by default machines are spread across the racks inside a partition for every project. if placement tags are provided, the machine candidate has an additional anti-affinity to other machines having the same tags"` -} +} diff --git a/dev/development/proposals/MEP12/partitioning/index.html b/dev/development/proposals/MEP12/partitioning/index.html index 2c301aaa50..a28ac65427 100644 --- a/dev/development/proposals/MEP12/partitioning/index.html +++ b/dev/development/proposals/MEP12/partitioning/index.html @@ -1,2 +1,2 @@ -Multi-Partition-Layout · metal-stack
Edit on GitHub

marp: true theme: metal-stack paginate: true footer: Gerrit Schwerthelm – x-cellent technologies GmbH — metal-stack Training backgroundImage: url("https://metal-stack.io/images/shape/banner.png") –- <!– _class: cover lead –>

h:200px

<!– _class: cover lead –>

Multi-Partition-Layout

<!– _class: lead _backgroundColor: #1f1f1f _backgroundImage: _footer: "" –> bg contain

<!– _class: lead _backgroundColor: #1f1f1f _backgroundImage: _footer: "" –> bg contain

<style>section { font-size: 30px; }</style>

Multi-Partition-Layout Properties

  • Fully independent locations with own storage and own node networks
  • Clusters can only be created independent in every location
    • Failover mechanism for deployed applications requires duplicated deployments, which can serve indepedently
    • Failover through BGP
  • If cluster nodes are spread across partitions (not implemented yet), nodes will not be able to reach each other
    • Would require an overlay network for inter-node-communication

<!– _class: cover lead –>

Single-Partition-Layout

<!– _class: lead _backgroundColor: #1f1f1f _backgroundImage: _footer: "" –> bg contain

<style>section { font-size: 30px; }</style>

Single-Partition-Layout Properties

  • Multiple groups of racks at multiple locations but connected to same CLOS topology
  • All racks can connect to the same storage network
  • Nodes in private networks can communicate
  • When creating a cluster, nodes will be randomly spread across the racks
    • Possible improvement of this situation, see MEP-12: Rack Spreading

MEP-12: Rack Spreading

  • Instead of selecting a machine from a machine pool randomly
  • Get all existing machines in the same project and count to which rack they belong
  • Place machine on the rack with the least amount of machines already allocated
  • Best effort only
+Multi-Partition-Layout · metal-stack
Edit on GitHub

marp: true theme: metal-stack paginate: true footer: Gerrit Schwerthelm – x-cellent technologies GmbH — metal-stack Training backgroundImage: url("https://metal-stack.io/images/shape/banner.png") –- <!– _class: cover lead –>

h:200px

<!– _class: cover lead –>

Multi-Partition-Layout

<!– _class: lead _backgroundColor: #1f1f1f _backgroundImage: _footer: "" –> bg contain

<!– _class: lead _backgroundColor: #1f1f1f _backgroundImage: _footer: "" –> bg contain

<style>section { font-size: 30px; }</style>

Multi-Partition-Layout Properties

  • Fully independent locations with own storage and own node networks
  • Clusters can only be created independent in every location
    • Failover mechanism for deployed applications requires duplicated deployments, which can serve indepedently
    • Failover through BGP
  • If cluster nodes are spread across partitions (not implemented yet), nodes will not be able to reach each other
    • Would require an overlay network for inter-node-communication

<!– _class: cover lead –>

Single-Partition-Layout

<!– _class: lead _backgroundColor: #1f1f1f _backgroundImage: _footer: "" –> bg contain

<style>section { font-size: 30px; }</style>

Single-Partition-Layout Properties

  • Multiple groups of racks at multiple locations but connected to same CLOS topology
  • All racks can connect to the same storage network
  • Nodes in private networks can communicate
  • When creating a cluster, nodes will be randomly spread across the racks
    • Possible improvement of this situation, see MEP-12: Rack Spreading

MEP-12: Rack Spreading

  • Instead of selecting a machine from a machine pool randomly
  • Get all existing machines in the same project and count to which rack they belong
  • Place machine on the rack with the least amount of machines already allocated
  • Best effort only
diff --git a/dev/development/proposals/MEP2/README/index.html b/dev/development/proposals/MEP2/README/index.html index b47f570568..ea3fc77dbd 100644 --- a/dev/development/proposals/MEP2/README/index.html +++ b/dev/development/proposals/MEP2/README/index.html @@ -1,2 +1,2 @@ -Two Factor Authentication · metal-stack
Edit on GitHub
+Two Factor Authentication · metal-stack
Edit on GitHub
diff --git a/dev/development/proposals/MEP3/README/index.html b/dev/development/proposals/MEP3/README/index.html index 9fde8ba875..407389993e 100644 --- a/dev/development/proposals/MEP3/README/index.html +++ b/dev/development/proposals/MEP3/README/index.html @@ -1,2 +1,2 @@ -Machine Re-Installation · metal-stack
Edit on GitHub

Machine Re-Installation

In the current metal-api only machine installations are possible, performing a machine upgrade is only possible by creating a new machine and delete the old one. This has the drawback that in case a lot of data is stored on the local disks, a full restore of the original data must be performed.

To prevent this, we will introduce a new metal-api endpoint to reinstall the machine with a new image, without actually deleting the data stored on the additional hard disks.

Storage is a difficult task to get right and reliable. A short analysis of our different storage requirements lead to 3 different scenarios.

  • Storage for the etcd pvs in the seed cluster of every partition. This is the most important storage in our setup because these etcd pods serve as configuration backend for all customer kubernetes clusters. If they fail, the cluster is down. However gardener deploys a backup and restore sidecar into the etcd pod of every customer kubernetes control plane, and if this sidecar detects a corrupt or missing etcd database file(s) it starts automatic restore from the configured backup location. This will take some minutes. If for example a node dies, and gardener creates a new node instead, the csi-lvm created pv is not present on that node. Kubernetes will not schedule the missing etcd pod on this node because it has a local PV configured and is therefore tainted to run only on that node. To let kubernetes create that pod anyhow, someone has to either remove the taint, or delete the pod. If this is done, the pod starts and the restore of the etcd data can start as well. You can see this is a bit too complicated and will take the customer cluster down for a while (not measured yet but in the range of 5-10 minutes).
  • Storage in customer clusters. This was not promised in 2020. We have a intermediate solution with the provisioning of csi-lvm by default into all customer clusters. Albeit this is only local storage and will get deleted if a node dies.
  • S3 Storage. We have two possibilities to cope with storage:
    • In place update of the OS with a daemonset This will be fast and simple, but might fail because the packages being installed are broken right now, or a filesystem gets full, or any other failure you can think of during a os update. Another drawback is that metal-api does not reflect the updated os image.
    • metal-api get a machine reinstall endpoint With this approach we leverage from existing and already proven mechanisms. Reinstall must keep all data except the sata-dom. Gardener currently is not able to do an update with this approach because it can only do rolling updates. Therefore a additional osupdatestrategy has to be implemented for metal and other providers in gardener to be able to leverage the metal reinstall on the same machineID approach.

If reinstall is implemented, we should focus on the same technology for all scenarios and put ceph via rook.io into the kubernetes clusters as additional StorageClass. It has to be checked whether to use the raw disk or a PV as the underlay block device where ceph stores its data.

API and behavior

The API will get an new endpoint "reinstall" this endpoint takes two arguments:

  • machineID
  • image

No other aspects of the machine can be modified during the re-installation. All data stored in the existing allocation will be preserved, only the image will be modified. Once this endpoint was called, the machine will get a reboot signal with the boot order set to PXE instead of HDD and the network interfaces on the leaf are set to PXE as well. Then the normal installation process starts:

  • unchanged: PXE boot with metal-hammer
  • changed: metal-hammer first checks with the machineID in the metal-api (through metal-core) if there is already a allocation present
  • changed: if a allocation is present and the allocation has set reinstall: true, wipe disk is only executed for the root disk, all other disks are untouched.
  • unchanged: the specified image is downloaded and burned, /install.sh is executed
  • unchanged: successful installation is reported back, network is set the the vrf, boot order is set to HDD.
  • unchanged: distribution kernel is booted via kexec

We can see that the allocation requires one additional parameter: reinstall and metal-hammer must check for already existing allocation at an earlier stage.

Components which requires modifications (first guess):

  • metal-hammer:
    • check for allocation present earlier
    • evaluation of reinstall flag set
    • wipe of disks depends on that flag
    • Bonus: move configuration of disk layout and primary disk detection algorithm (PDDA) from metal-hammer into metal-api. metal-api MUST reject reinstallation if the disk found by PDDA does not have the /etc/metal directory!
  • metal-core:
    • probably nothing
  • metal-api:
    • new endpoint /machine/reinstall
    • add Reinstall bool to data model of allocation
    • make sure to reset Reinstall after reinstallation to prevent endless reinstallation loop
  • metalctl:
    • implement reinstall
  • metal-go:
    • implement reinstall
  • gardener (longterm):
    • add the OSUpgradeStrategy reinstall
+Machine Re-Installation · metal-stack
Edit on GitHub

Machine Re-Installation

In the current metal-api only machine installations are possible, performing a machine upgrade is only possible by creating a new machine and delete the old one. This has the drawback that in case a lot of data is stored on the local disks, a full restore of the original data must be performed.

To prevent this, we will introduce a new metal-api endpoint to reinstall the machine with a new image, without actually deleting the data stored on the additional hard disks.

Storage is a difficult task to get right and reliable. A short analysis of our different storage requirements lead to 3 different scenarios.

  • Storage for the etcd pvs in the seed cluster of every partition. This is the most important storage in our setup because these etcd pods serve as configuration backend for all customer kubernetes clusters. If they fail, the cluster is down. However gardener deploys a backup and restore sidecar into the etcd pod of every customer kubernetes control plane, and if this sidecar detects a corrupt or missing etcd database file(s) it starts automatic restore from the configured backup location. This will take some minutes. If for example a node dies, and gardener creates a new node instead, the csi-lvm created pv is not present on that node. Kubernetes will not schedule the missing etcd pod on this node because it has a local PV configured and is therefore tainted to run only on that node. To let kubernetes create that pod anyhow, someone has to either remove the taint, or delete the pod. If this is done, the pod starts and the restore of the etcd data can start as well. You can see this is a bit too complicated and will take the customer cluster down for a while (not measured yet but in the range of 5-10 minutes).
  • Storage in customer clusters. This was not promised in 2020. We have a intermediate solution with the provisioning of csi-lvm by default into all customer clusters. Albeit this is only local storage and will get deleted if a node dies.
  • S3 Storage. We have two possibilities to cope with storage:
    • In place update of the OS with a daemonset This will be fast and simple, but might fail because the packages being installed are broken right now, or a filesystem gets full, or any other failure you can think of during a os update. Another drawback is that metal-api does not reflect the updated os image.
    • metal-api get a machine reinstall endpoint With this approach we leverage from existing and already proven mechanisms. Reinstall must keep all data except the sata-dom. Gardener currently is not able to do an update with this approach because it can only do rolling updates. Therefore a additional osupdatestrategy has to be implemented for metal and other providers in gardener to be able to leverage the metal reinstall on the same machineID approach.

If reinstall is implemented, we should focus on the same technology for all scenarios and put ceph via rook.io into the kubernetes clusters as additional StorageClass. It has to be checked whether to use the raw disk or a PV as the underlay block device where ceph stores its data.

API and behavior

The API will get an new endpoint "reinstall" this endpoint takes two arguments:

  • machineID
  • image

No other aspects of the machine can be modified during the re-installation. All data stored in the existing allocation will be preserved, only the image will be modified. Once this endpoint was called, the machine will get a reboot signal with the boot order set to PXE instead of HDD and the network interfaces on the leaf are set to PXE as well. Then the normal installation process starts:

  • unchanged: PXE boot with metal-hammer
  • changed: metal-hammer first checks with the machineID in the metal-api (through metal-core) if there is already a allocation present
  • changed: if a allocation is present and the allocation has set reinstall: true, wipe disk is only executed for the root disk, all other disks are untouched.
  • unchanged: the specified image is downloaded and burned, /install.sh is executed
  • unchanged: successful installation is reported back, network is set the the vrf, boot order is set to HDD.
  • unchanged: distribution kernel is booted via kexec

We can see that the allocation requires one additional parameter: reinstall and metal-hammer must check for already existing allocation at an earlier stage.

Components which requires modifications (first guess):

  • metal-hammer:
    • check for allocation present earlier
    • evaluation of reinstall flag set
    • wipe of disks depends on that flag
    • Bonus: move configuration of disk layout and primary disk detection algorithm (PDDA) from metal-hammer into metal-api. metal-api MUST reject reinstallation if the disk found by PDDA does not have the /etc/metal directory!
  • metal-core:
    • probably nothing
  • metal-api:
    • new endpoint /machine/reinstall
    • add Reinstall bool to data model of allocation
    • make sure to reset Reinstall after reinstallation to prevent endless reinstallation loop
  • metalctl:
    • implement reinstall
  • metal-go:
    • implement reinstall
  • gardener (longterm):
    • add the OSUpgradeStrategy reinstall
diff --git a/dev/development/proposals/MEP4/README/index.html b/dev/development/proposals/MEP4/README/index.html index c6701b8f0f..6a0841d3ce 100644 --- a/dev/development/proposals/MEP4/README/index.html +++ b/dev/development/proposals/MEP4/README/index.html @@ -1,2 +1,2 @@ -Multi-tenancy for the metal-api · metal-stack
Edit on GitHub

Multi-tenancy for the metal-api

In the past we decided to treat the metal-api as a "low-level API", i.e. the API does not know anything about projects and tenants. A user with editor access can for example assign machines to every project he desires, he can see all the machines available and control them. Even though we always wanted to keep open the possibility to just offer bare metal machines to the end-user, the ultimate objective has always been to create an API for Kubernetes clusters. Hence, we tried to keep the metal-api code base as small as possible and we added resource scoping to a "higher-level API", the cloud-api, a component that is not open-source. From there, a user would be able to only see his own clusters and IP addresses.

The implication is that the metal-api has no multi-tenancy without another layer on top of it that implements resource scoping. One can say that we treat clusters as first-class citizens. In regard of clusters we fulfill the objective that we had from the very beginning: provide a multi-tenant API for Kubernetes clusters to the end-users.

However, as time passed by, things changed: The metal-stack is becoming an open-source product and we already have promising adopters of our product, who are willing to contribute to metal-stack. This is a serious chance of making our product better and more successful. It turns out that the decision we made was sufficient for us, but for others it is not.

Why adopters need multi-tenancy in the metal-api

Not every adopter will be interested in the cloud-api

For example, users who want to combine the Metal Stack with Gardener, may not want to hide all of the Gardener's functionality behind the cloud-api in the way we do. They want to use the much more powerful Gardener Dashboard instead. The Gardener itself does not need the cloud-api either. It is a cluster-api by itself. It only needs to utilize our "low-level API" and actually expects this API to have multi-tenancy as otherwise every logged in user can create / destroy clusters in every existing project from the Gardener dashboard.

This makes obvious that, with our decision, we placed an unnecessary obstacle in the way of our adopters: They now need to implement an own layer between the Gardener and the metal-api to provide multi-tenancy. From the Gardener-perspective we strongly differ from other cloud providers in this aspect and it is a matter of time when this will become an issue. When we encourage adopters to implement such interfaces on their own we also partly lose control of our product, we increase divergence.

We cannot claim that Metal Stack is a multi-tenant solution on our website

As the cloud-api is not part of the Metal Stack, the promise of multi-tenancy is only true for our network layer. Without the cloud-api to enable multi-tenancy, the network isolation is currently useless for end-users. Users of the Metal Stack can not self-manage machines, networks and ips without compromising the environment and thus, there is no self-service. We lose a valuable selling point when adopters can not immediately make use of our leading edge network isolation where we put so much effort to.

Open partitions for third-party usage

If a third-party uses Gardener and our metal-api had multi-tenancy, we would be able to allow a third-party to create clusters with workers in our own partitions. At the moment, this is not possible because the Gardener needs to know the HMAC secrets to create worker nodes, which would compromise our environment. If a thirdy-party knows our HMAC we lose control over the machines of our own tenants.

We do not actually want to open-source the cloud-api

One could think about solving the multi-tenancy issue by adding machine endpoints to the cloud-api. Gardener would then not consume the metal-api anymore but only the cloud-api.

This approach would not be ideal. We only want to offer a minimum viable product to adopters. The Gardener does not need a cluster-api as provided by the cloud-api. We want to treat additions on top of the basic stack as enterprise products.

The cloud-api contains billing endpoints, which are a perfect example for an optional addition of the Metal Stack. For basic usage of the Metal Stack a user does not need billing. Still, billing functionality can be interesting for some enterprises, who are like us, selling the infrastructure to third-parties.

Increased security for provider admins

Multi-tenancy in the metal-api also has the potential to limit the damage that a provider administrator can cause by mistake. If an administrator has to acquire project permissions on machine-level we can effectively reduce the damage he can make to this single project.

Another example would be the automatic provisioning of a Gitlab CI runner used for integration testing (a use case that we have where we do not require the cloud-api). This can easily be done in automated manner with Ansible and the Metal dynamic inventory + modules. However, with Ansible, mistakes in the automation can be made very quickly and if Ansible would only see machines of a dedicated project, this would also reduce damage it can make.

It is likely that there are more similar use-cases like that to come (maybe even for the storage solution?).

Also the surface for our Gardener components (metal-ccm, gardener-extension-provider-metal, machine-controller-manager) would be reduced to project scopes.

Conclusion

For these reasons the decision we made is very likely to have a negative impact on the adoption-rate of the Metal Stack and we should think about treating machines, networks and ips as first-class citizens as well. This makes us closer to the offer of hyperscalers. As mentioned in the beginning, all the time we tried to keep the possibility open to just offer bare metal machines. Let's continue with decision by adding multi-tenancy to the metal-api.

Required actions

Resource scoping

Just as implemented by the cloud-api, resource scoping needs to be added to almost every endpoint of the metal-api:

  • Machines / Firewalls
    • A user should only be able to view machines / firewalls of the projects he has at least view access to
    • A user should only be able to create and destroy machines / firewalls for projects he has at least editor access to Provider-tenants with at least view access can additionally view machines which have no project assignments Provider-tenants with at least editor access can additionally allocate / reserve machines which have no project assignments
  • Networks
    • A user should only be able to view networks of the projects he has at least view access to
    • A user should only be able to allocate networks of projects he has at least editor access to
    • A user should only be able to free networks assigned to projects he has at least editor access to Provider-tenants with at least view access can additionally view networks which have no project assignments Provider-tenants with at least editor access can additionally edit networks which have no project assignments Provider-tenants with at least admin access can additionally create or remove networks which have no project assignments
  • IPs
    • A user should only be able to view ips of the projects he has at least view access to
    • A user should only be able to allocate ips in networks of projects he has at least editor access to
    • A user should only be able to free ips assigned to projects he has at least editor access to
  • Projects
    • A logged in user is able to create projects when he has the permission to create projects
    • A user should only be able to view projects where he has at least view access to
    • A user should only be able to delete projects where he has admin access to
  • Partitions / Images
    • Only provider-admin users can add, delete, update
    • All logged in users can view
  • IPMI
    • Only provider-tenants can view machine IPMI data
  • Endpoints for internal use
    • Should only be accessible with HMAC auth and the HMAC secrets are only known by components of the Metal Stack (mainly for communication between partition and control plane), never for third-party usage

For all of this we need enhance the database queries with a filter for projects that a user has access to. As we already use a client to the masterdata-api in the metal-api, we can extract project memberships of a logged in user from there.

More permissions

We do not only need kaas-... permissions in the LDAP but also maas-. This way we can differentiate between permissions for the cloud-api and permissions for the metal-api.

Service account tokens / technical users

We need to provide the possibility for users to obtain access tokens to use for technical purposes (CI, third-party tooling like Gardener, ...).

We do not have this functionality yet, but it would also become a necessity for the cloud-api at some point in the future.

Cloud API

  • Project creation and deletion again have to be moved back into the metal-api, this also frees adopters from the need to write an own API in order to manage projects- The cloud-api will (again) only proxy project endpoints through to the metal-api
  • Do not point the secret bindings to a the shared provider secret in a partition. Create an individual provider-secret for the logged in tenant. The Gardener needs to use this tenant-specific provider secret to talk to the metal-api, do not give the Gardener HMAC access anymore.
  • The provider secret partition mapping can be removed from the cloud-api config and from the deployment
+Multi-tenancy for the metal-api · metal-stack
Edit on GitHub

Multi-tenancy for the metal-api

In the past we decided to treat the metal-api as a "low-level API", i.e. the API does not know anything about projects and tenants. A user with editor access can for example assign machines to every project he desires, he can see all the machines available and control them. Even though we always wanted to keep open the possibility to just offer bare metal machines to the end-user, the ultimate objective has always been to create an API for Kubernetes clusters. Hence, we tried to keep the metal-api code base as small as possible and we added resource scoping to a "higher-level API", the cloud-api, a component that is not open-source. From there, a user would be able to only see his own clusters and IP addresses.

The implication is that the metal-api has no multi-tenancy without another layer on top of it that implements resource scoping. One can say that we treat clusters as first-class citizens. In regard of clusters we fulfill the objective that we had from the very beginning: provide a multi-tenant API for Kubernetes clusters to the end-users.

However, as time passed by, things changed: The metal-stack is becoming an open-source product and we already have promising adopters of our product, who are willing to contribute to metal-stack. This is a serious chance of making our product better and more successful. It turns out that the decision we made was sufficient for us, but for others it is not.

Why adopters need multi-tenancy in the metal-api

Not every adopter will be interested in the cloud-api

For example, users who want to combine the Metal Stack with Gardener, may not want to hide all of the Gardener's functionality behind the cloud-api in the way we do. They want to use the much more powerful Gardener Dashboard instead. The Gardener itself does not need the cloud-api either. It is a cluster-api by itself. It only needs to utilize our "low-level API" and actually expects this API to have multi-tenancy as otherwise every logged in user can create / destroy clusters in every existing project from the Gardener dashboard.

This makes obvious that, with our decision, we placed an unnecessary obstacle in the way of our adopters: They now need to implement an own layer between the Gardener and the metal-api to provide multi-tenancy. From the Gardener-perspective we strongly differ from other cloud providers in this aspect and it is a matter of time when this will become an issue. When we encourage adopters to implement such interfaces on their own we also partly lose control of our product, we increase divergence.

We cannot claim that Metal Stack is a multi-tenant solution on our website

As the cloud-api is not part of the Metal Stack, the promise of multi-tenancy is only true for our network layer. Without the cloud-api to enable multi-tenancy, the network isolation is currently useless for end-users. Users of the Metal Stack can not self-manage machines, networks and ips without compromising the environment and thus, there is no self-service. We lose a valuable selling point when adopters can not immediately make use of our leading edge network isolation where we put so much effort to.

Open partitions for third-party usage

If a third-party uses Gardener and our metal-api had multi-tenancy, we would be able to allow a third-party to create clusters with workers in our own partitions. At the moment, this is not possible because the Gardener needs to know the HMAC secrets to create worker nodes, which would compromise our environment. If a thirdy-party knows our HMAC we lose control over the machines of our own tenants.

We do not actually want to open-source the cloud-api

One could think about solving the multi-tenancy issue by adding machine endpoints to the cloud-api. Gardener would then not consume the metal-api anymore but only the cloud-api.

This approach would not be ideal. We only want to offer a minimum viable product to adopters. The Gardener does not need a cluster-api as provided by the cloud-api. We want to treat additions on top of the basic stack as enterprise products.

The cloud-api contains billing endpoints, which are a perfect example for an optional addition of the Metal Stack. For basic usage of the Metal Stack a user does not need billing. Still, billing functionality can be interesting for some enterprises, who are like us, selling the infrastructure to third-parties.

Increased security for provider admins

Multi-tenancy in the metal-api also has the potential to limit the damage that a provider administrator can cause by mistake. If an administrator has to acquire project permissions on machine-level we can effectively reduce the damage he can make to this single project.

Another example would be the automatic provisioning of a Gitlab CI runner used for integration testing (a use case that we have where we do not require the cloud-api). This can easily be done in automated manner with Ansible and the Metal dynamic inventory + modules. However, with Ansible, mistakes in the automation can be made very quickly and if Ansible would only see machines of a dedicated project, this would also reduce damage it can make.

It is likely that there are more similar use-cases like that to come (maybe even for the storage solution?).

Also the surface for our Gardener components (metal-ccm, gardener-extension-provider-metal, machine-controller-manager) would be reduced to project scopes.

Conclusion

For these reasons the decision we made is very likely to have a negative impact on the adoption-rate of the Metal Stack and we should think about treating machines, networks and ips as first-class citizens as well. This makes us closer to the offer of hyperscalers. As mentioned in the beginning, all the time we tried to keep the possibility open to just offer bare metal machines. Let's continue with decision by adding multi-tenancy to the metal-api.

Required actions

Resource scoping

Just as implemented by the cloud-api, resource scoping needs to be added to almost every endpoint of the metal-api:

  • Machines / Firewalls
    • A user should only be able to view machines / firewalls of the projects he has at least view access to
    • A user should only be able to create and destroy machines / firewalls for projects he has at least editor access to Provider-tenants with at least view access can additionally view machines which have no project assignments Provider-tenants with at least editor access can additionally allocate / reserve machines which have no project assignments
  • Networks
    • A user should only be able to view networks of the projects he has at least view access to
    • A user should only be able to allocate networks of projects he has at least editor access to
    • A user should only be able to free networks assigned to projects he has at least editor access to Provider-tenants with at least view access can additionally view networks which have no project assignments Provider-tenants with at least editor access can additionally edit networks which have no project assignments Provider-tenants with at least admin access can additionally create or remove networks which have no project assignments
  • IPs
    • A user should only be able to view ips of the projects he has at least view access to
    • A user should only be able to allocate ips in networks of projects he has at least editor access to
    • A user should only be able to free ips assigned to projects he has at least editor access to
  • Projects
    • A logged in user is able to create projects when he has the permission to create projects
    • A user should only be able to view projects where he has at least view access to
    • A user should only be able to delete projects where he has admin access to
  • Partitions / Images
    • Only provider-admin users can add, delete, update
    • All logged in users can view
  • IPMI
    • Only provider-tenants can view machine IPMI data
  • Endpoints for internal use
    • Should only be accessible with HMAC auth and the HMAC secrets are only known by components of the Metal Stack (mainly for communication between partition and control plane), never for third-party usage

For all of this we need enhance the database queries with a filter for projects that a user has access to. As we already use a client to the masterdata-api in the metal-api, we can extract project memberships of a logged in user from there.

More permissions

We do not only need kaas-... permissions in the LDAP but also maas-. This way we can differentiate between permissions for the cloud-api and permissions for the metal-api.

Service account tokens / technical users

We need to provide the possibility for users to obtain access tokens to use for technical purposes (CI, third-party tooling like Gardener, ...).

We do not have this functionality yet, but it would also become a necessity for the cloud-api at some point in the future.

Cloud API

  • Project creation and deletion again have to be moved back into the metal-api, this also frees adopters from the need to write an own API in order to manage projects- The cloud-api will (again) only proxy project endpoints through to the metal-api
  • Do not point the secret bindings to a the shared provider secret in a partition. Create an individual provider-secret for the logged in tenant. The Gardener needs to use this tenant-specific provider secret to talk to the metal-api, do not give the Gardener HMAC access anymore.
  • The provider secret partition mapping can be removed from the cloud-api config and from the deployment
diff --git a/dev/development/proposals/MEP5/README/index.html b/dev/development/proposals/MEP5/README/index.html index 3d9b677824..899cacd06e 100644 --- a/dev/development/proposals/MEP5/README/index.html +++ b/dev/development/proposals/MEP5/README/index.html @@ -1,2 +1,2 @@ -Shared Networks · metal-stack
Edit on GitHub

Shared Networks

Why are shared networks needed

For special purpose machines that serve shared services with performance critical workloads to all machines of a partition (like persistent storage) it would be good to have kind of a "shared network" that is easily accessible. They do not necessarily need another firewall. This would avoid having two firewalls in the datapath between a machine in a private network and the machines of a shared service.

Constraints that need to hold

  • a shared network is usable from all machines that have a firewall in front, that uses it
  • a shared network is only usable within a single partition (currently we are constrained in bandwidth and have no routing of 10.0.0.0/8 addresses btw. partitions and failure domain should be the partition but this constraint might get lifted in the future)
  • networks may be marked as shared after network allocation (but there should be no way back from shared to unshared)
  • neither machines nor firewalls may have multiple private, unshared networks configured
  • machines must have a single primary network configured
    • this might be a shared network
    • OR a plain, unshared private network
  • firewalls may participate in multiple shared networks
  • machines can be allocated with a primary network using auto IP allocation or with noauto and a specific IP

Should shared networks be private

Alternative 1: If we implemented shared networks by extending functions around plain, private networks we would not have to manage another CIDR (mini point) and it would be possible to create a k8s cluster with a private network, mark the network as shared and produce shared services from this k8s cluster.

Alternative 2: If shared networks are implemented as first class networks we could customize the VRF and also accomplish an other goal of our roadmap: being able to create machines directly in an external network.

Together with @majst01 and @Gerrit91 we decided to continue to implement Alternative 1.

Firewalls accessing a shared network

Firewalls that access shared networks need to:

  • hide the private network behind an ip address of the shared network if the shared network was configured with nat=true.
  • import the prefixes of the shared VRF to the private VRF and import the prefixes of the private VRF to the shared VRF so that the communication between the two is working in both directions. As long as no nat=true was set on the shared VRF, the original machine ips are visible in both communication directions.

Setup with shared networks and single consumer

Simple Setup

Setup with single shared network and multiple consumers

Advanced Setup

Getting internet acccess

Machines contained in a shared network can access the internet with different scenarios:

  • if they have an own firewall: this is internet accessibility, as common (check whether all traffic gets routed through it!)
  • if they don't have an own firewall, an external HTTP proxy is needed that has an endpoint exposed as Service Type NodePort
+Shared Networks · metal-stack
Edit on GitHub

Shared Networks

Why are shared networks needed

For special purpose machines that serve shared services with performance critical workloads to all machines of a partition (like persistent storage) it would be good to have kind of a "shared network" that is easily accessible. They do not necessarily need another firewall. This would avoid having two firewalls in the datapath between a machine in a private network and the machines of a shared service.

Constraints that need to hold

  • a shared network is usable from all machines that have a firewall in front, that uses it
  • a shared network is only usable within a single partition (currently we are constrained in bandwidth and have no routing of 10.0.0.0/8 addresses btw. partitions and failure domain should be the partition but this constraint might get lifted in the future)
  • networks may be marked as shared after network allocation (but there should be no way back from shared to unshared)
  • neither machines nor firewalls may have multiple private, unshared networks configured
  • machines must have a single primary network configured
    • this might be a shared network
    • OR a plain, unshared private network
  • firewalls may participate in multiple shared networks
  • machines can be allocated with a primary network using auto IP allocation or with noauto and a specific IP

Should shared networks be private

Alternative 1: If we implemented shared networks by extending functions around plain, private networks we would not have to manage another CIDR (mini point) and it would be possible to create a k8s cluster with a private network, mark the network as shared and produce shared services from this k8s cluster.

Alternative 2: If shared networks are implemented as first class networks we could customize the VRF and also accomplish an other goal of our roadmap: being able to create machines directly in an external network.

Together with @majst01 and @Gerrit91 we decided to continue to implement Alternative 1.

Firewalls accessing a shared network

Firewalls that access shared networks need to:

  • hide the private network behind an ip address of the shared network if the shared network was configured with nat=true.
  • import the prefixes of the shared VRF to the private VRF and import the prefixes of the private VRF to the shared VRF so that the communication between the two is working in both directions. As long as no nat=true was set on the shared VRF, the original machine ips are visible in both communication directions.

Setup with shared networks and single consumer

Simple Setup

Setup with single shared network and multiple consumers

Advanced Setup

Getting internet acccess

Machines contained in a shared network can access the internet with different scenarios:

  • if they have an own firewall: this is internet accessibility, as common (check whether all traffic gets routed through it!)
  • if they don't have an own firewall, an external HTTP proxy is needed that has an endpoint exposed as Service Type NodePort
diff --git a/dev/development/proposals/MEP6/README/index.html b/dev/development/proposals/MEP6/README/index.html index a33894595d..19e461adb9 100644 --- a/dev/development/proposals/MEP6/README/index.html +++ b/dev/development/proposals/MEP6/README/index.html @@ -35,4 +35,4 @@ vrfshared: false nat: true shared: true # it's usable from multiple projects -underlay: false

DMZ firewall

The firewall of the DMZ will intersect its private network for attached machines, the DMZ network and the public internet.

Application Firewall

The firewall of application workloads intersects its private network for attached machines and the DMZ network.

Code Changes / Implications

Decision

We decided to follow the second approach with private DMZ networks.

+underlay: false

DMZ firewall

The firewall of the DMZ will intersect its private network for attached machines, the DMZ network and the public internet.

Application Firewall

The firewall of application workloads intersects its private network for attached machines and the DMZ network.

Code Changes / Implications

Decision

We decided to follow the second approach with private DMZ networks.

diff --git a/dev/development/proposals/MEP8/README/index.html b/dev/development/proposals/MEP8/README/index.html index a80260ed4c..26ee233ffa 100644 --- a/dev/development/proposals/MEP8/README/index.html +++ b/dev/development/proposals/MEP8/README/index.html @@ -364,4 +364,4 @@ - device: "/dev/nvmne0n1" wipeonreinstall: false - device: "/dev/nvmne0n2" - wipeonreinstall: false

Components which requires modifications

+ wipeonreinstall: false

Components which requires modifications

diff --git a/dev/development/proposals/MEP9/README/index.html b/dev/development/proposals/MEP9/README/index.html index 764ebdaad0..9addcbf406 100644 --- a/dev/development/proposals/MEP9/README/index.html +++ b/dev/development/proposals/MEP9/README/index.html @@ -10,4 +10,4 @@ Status: Boolean field tailscale: Version: Actual version - ...

bmc-reverse-proxy

TODO

References

  1. WireGuard: Next Generation Secure Network Tunnel
  2. How Tailscale works
  3. Tailscale is officially SOC 2 compliant
  4. Why not Wireguard
  5. Wireguard: Known Limitations
  6. Wireguard: Things That Might Be Accomplished
  7. Headscale: Tailscale control protocol v2
+ ...

bmc-reverse-proxy

TODO

References

  1. WireGuard: Next Generation Secure Network Tunnel
  2. How Tailscale works
  3. Tailscale is officially SOC 2 compliant
  4. Why not Wireguard
  5. Wireguard: Known Limitations
  6. Wireguard: Things That Might Be Accomplished
  7. Headscale: Tailscale control protocol v2
diff --git a/dev/development/proposals/index.html b/dev/development/proposals/index.html index 4f91031a04..0cdb103760 100644 --- a/dev/development/proposals/index.html +++ b/dev/development/proposals/index.html @@ -1,2 +1,2 @@ -Enhancement Proposals · metal-stack
Edit on GitHub

Metal Stack Enhancement Proposals (MEPs)

This section contains proposals which address substantial modifications to metal-stack.

Every proposal has a short name which starts with MEP followed by an incremental, unique number. Proposals should be raised as pull requests in the docs repository and can be discussed in Github issues.

The list of proposal and their current state is listed in the table below.

Possible states are:

  • In Discussion
  • Accepted
  • Declined
  • In Progress
  • Completed
  • Aborted

Once a proposal was accepted, an issue should be raised and the implementation should be done in a separate PR.

NameDescriptionState
MEP-1Distributed Control Plane DeploymentIn Discussion
MEP-2Two Factor AuthenticationAborted
MEP-3Machine Re-Installation to preserve local dataCompleted
MEP-4Multi-tenancy for the metal-apiIn Discussion
MEP-5Shared NetworksCompleted
MEP-6DMZ NetworksCompleted
MEP-8Configurable FilesystemlayoutCompleted
MEP-9No Open Ports To the Data CenterCompleted
MEP-10SONiC SupportCompleted
MEP-11Auditing of metal-stack resourcesCompleted
MEP-12Rack SpreadingCompleted
+Enhancement Proposals · metal-stack
Edit on GitHub

Metal Stack Enhancement Proposals (MEPs)

This section contains proposals which address substantial modifications to metal-stack.

Every proposal has a short name which starts with MEP followed by an incremental, unique number. Proposals should be raised as pull requests in the docs repository and can be discussed in Github issues.

The list of proposal and their current state is listed in the table below.

Possible states are:

  • In Discussion
  • Accepted
  • Declined
  • In Progress
  • Completed
  • Aborted

Once a proposal was accepted, an issue should be raised and the implementation should be done in a separate PR.

NameDescriptionState
MEP-1Distributed Control Plane DeploymentIn Discussion
MEP-2Two Factor AuthenticationAborted
MEP-3Machine Re-Installation to preserve local dataCompleted
MEP-4Multi-tenancy for the metal-apiIn Discussion
MEP-5Shared NetworksCompleted
MEP-6DMZ NetworksCompleted
MEP-8Configurable FilesystemlayoutCompleted
MEP-9No Open Ports To the Data CenterCompleted
MEP-10SONiC SupportCompleted
MEP-11Auditing of metal-stack resourcesCompleted
MEP-12Rack SpreadingCompleted
diff --git a/dev/development/roadmap/index.html b/dev/development/roadmap/index.html index f662505492..ae53970689 100644 --- a/dev/development/roadmap/index.html +++ b/dev/development/roadmap/index.html @@ -1,2 +1,2 @@ -Roadmap · metal-stack
Edit on GitHub

Roadmap

A roadmap with short-, mid- and long-term planning will be available soon. For now, there is only a backlog.

Short-term

Available soon.

Mid-term

Available soon.

Long-term

Available soon.

Backlog

The backlog contains ideas of what could become part of the roadmap in the future. The list is ordered alphabetically. Therefore, the order does not express the importance or weight of a backlog item.

We incorporate community feedback into the roadmap. If you think that important points are missing in the backlog, please share your ideas with us. We have a Slack channel. Please check out metal-stack.io for contact information.

Danger

By no means this list is a promise of what is being worked on in the near future. It is just a summary of ideas that was agreed on to be "nice to have". It is up to the investors, maintainers and the community to choose topics from this list and to implement them or to remove them from the list.

  • Add metal-stack to Gardener conformance test grid
  • Autoscaler for metal control plane components
  • CI dashboard and public integration testing
  • Cilium as the default CNI for metal-stack on Gardener K8s clusters
  • Improved release and deploy processes (GitOps, Spinnaker, Flux)
  • Machine internet without firewalls
  • metal-stack dashboard (UI)
  • Offer our metal-stack extensions as enterprise products (accounting, cluster-api, S3) (neither of them will ever be required for running metal-stack, they just add extra value for certain enterprises)
  • Partition managed by Kubernetes (with Kubelets joining the control plane cluster)
  • Public offering / demo playground
  • Resource scoping in the metal-api (MEP-4)
  • Service / API tokens (for scoped technical user access)
+Roadmap · metal-stack
Edit on GitHub

Roadmap

A roadmap with short-, mid- and long-term planning will be available soon. For now, there is only a backlog.

Short-term

Available soon.

Mid-term

Available soon.

Long-term

Available soon.

Backlog

The backlog contains ideas of what could become part of the roadmap in the future. The list is ordered alphabetically. Therefore, the order does not express the importance or weight of a backlog item.

We incorporate community feedback into the roadmap. If you think that important points are missing in the backlog, please share your ideas with us. We have a Slack channel. Please check out metal-stack.io for contact information.

Danger

By no means this list is a promise of what is being worked on in the near future. It is just a summary of ideas that was agreed on to be "nice to have". It is up to the investors, maintainers and the community to choose topics from this list and to implement them or to remove them from the list.

  • Add metal-stack to Gardener conformance test grid
  • Autoscaler for metal control plane components
  • CI dashboard and public integration testing
  • Cilium as the default CNI for metal-stack on Gardener K8s clusters
  • Improved release and deploy processes (GitOps, Spinnaker, Flux)
  • Machine internet without firewalls
  • metal-stack dashboard (UI)
  • Offer our metal-stack extensions as enterprise products (accounting, cluster-api, S3) (neither of them will ever be required for running metal-stack, they just add extra value for certain enterprises)
  • Partition managed by Kubernetes (with Kubelets joining the control plane cluster)
  • Public offering / demo playground
  • Resource scoping in the metal-api (MEP-4)
  • Service / API tokens (for scoped technical user access)
diff --git a/dev/external/csi-driver-lvm/CONTRIBUTING/index.html b/dev/external/csi-driver-lvm/CONTRIBUTING/index.html index b2eac710cd..b675c7d1cf 100644 --- a/dev/external/csi-driver-lvm/CONTRIBUTING/index.html +++ b/dev/external/csi-driver-lvm/CONTRIBUTING/index.html @@ -1,2 +1,2 @@ -Contributing · metal-stack
Edit on GitHub
+Contributing · metal-stack
Edit on GitHub
diff --git a/dev/external/csi-driver-lvm/README/index.html b/dev/external/csi-driver-lvm/README/index.html index 33cc025b02..ab7dabc4af 100644 --- a/dev/external/csi-driver-lvm/README/index.html +++ b/dev/external/csi-driver-lvm/README/index.html @@ -13,4 +13,4 @@ kubectl delete -f examples/csi-pvc.yaml

Development

In order to run the integration tests locally, you need to create to loop devices on your host machine. Make sure the loop device mount paths are not used on your system (default path is /dev/loop10{0,1}).

You can create these loop devices like this:

for i in 100 101; do fallocate -l 1G loop${i}.img ; sudo losetup /dev/loop${i} loop${i}.img; done
 sudo losetup -a
 # use this for recreation or cleanup
-# for i in 100 101; do sudo losetup -d /dev/loop${i}; rm -f loop${i}.img; done

You can then run the tests against a kind cluster, running:

make test

To recreate or cleanup the kind cluster:

make test-cleanup

Page Tree

+# for i in 100 101; do sudo losetup -d /dev/loop${i}; rm -f loop${i}.img; done

You can then run the tests against a kind cluster, running:

make test

To recreate or cleanup the kind cluster:

make test-cleanup

Page Tree

diff --git a/dev/external/firewall-controller/CONTRIBUTING/index.html b/dev/external/firewall-controller/CONTRIBUTING/index.html index 92ea6232b7..b73d8e78a1 100644 --- a/dev/external/firewall-controller/CONTRIBUTING/index.html +++ b/dev/external/firewall-controller/CONTRIBUTING/index.html @@ -1,2 +1,2 @@ -Contributing · metal-stack
Edit on GitHub
+Contributing · metal-stack
Edit on GitHub
diff --git a/dev/external/firewall-controller/DEVELOP/index.html b/dev/external/firewall-controller/DEVELOP/index.html index 4f3c835fd6..e67b4e2e6a 100644 --- a/dev/external/firewall-controller/DEVELOP/index.html +++ b/dev/external/firewall-controller/DEVELOP/index.html @@ -18,4 +18,4 @@ # watch results k describe -n firewall firewall cat nftables.v4 -cat hosts +cat hosts diff --git a/dev/external/firewall-controller/README/index.html b/dev/external/firewall-controller/README/index.html index 503d00298a..b9c38f1266 100644 --- a/dev/external/firewall-controller/README/index.html +++ b/dev/external/firewall-controller/README/index.html @@ -122,4 +122,4 @@ droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:30 +0000 UTC {"DPT":"650","DST":"1.2.3.4","ID":"12399","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"vlan179","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"40194","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"241","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:30 +0000 UTC"} droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:34 +0000 UTC {"DPT":"2362","DST":"1.2.3.4","ID":"44545","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"40194","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"242","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:34 +0000 UTC"} droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:10 +0000 UTC {"DPT":"63351","DST":"1.2.3.4","ID":"11855","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"vlan179","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"54589","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"245","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:10 +0000 UTC"} -droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:51 +0000 UTC {"DPT":"8002","DST":"1.2.3.4","ID":"17539","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"47615","SRC":"2.3.4.5","SYN":"","TOS":"0x08","TTL":"239","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:51 +0000 UTC"}

You can forward the droptailer logs to any log aggregation infrastructure you have in place.

Page Tree

+droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:51 +0000 UTC {"DPT":"8002","DST":"1.2.3.4","ID":"17539","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"47615","SRC":"2.3.4.5","SYN":"","TOS":"0x08","TTL":"239","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:51 +0000 UTC"}

You can forward the droptailer logs to any log aggregation infrastructure you have in place.

Page Tree

diff --git a/dev/external/metalctl/CONTRIBUTING/index.html b/dev/external/metalctl/CONTRIBUTING/index.html index d5fac26c65..9625ae9c8e 100644 --- a/dev/external/metalctl/CONTRIBUTING/index.html +++ b/dev/external/metalctl/CONTRIBUTING/index.html @@ -1,2 +1,2 @@ -Contributing · metal-stack
Edit on GitHub
+Contributing · metal-stack
Edit on GitHub
diff --git a/dev/external/metalctl/README/index.html b/dev/external/metalctl/README/index.html index e9cb2737ae..f83279c383 100644 --- a/dev/external/metalctl/README/index.html +++ b/dev/external/metalctl/README/index.html @@ -35,4 +35,4 @@ issuer_url: https://keycloak.somedomain.io custom_scopes: roles,openid,profile,email client_id: my-client-id - client_secret: my-secret

Available commands

Full documentation is generated out of the cobra command implementation with:

metalctl markdown

generated markdown is here and here

Development

For MacOS users, running the tests might throw an error because tests are utilizing go-mpatch in order to manipulate the time.Now function. The patch allows testing with fixed timestamps.

Instead, MacOS users can utilize the make test-in-docker target to execute the tests.

Page Tree

+ client_secret: my-secret

Available commands

Full documentation is generated out of the cobra command implementation with:

metalctl markdown

generated markdown is here and here

Development

For MacOS users, running the tests might throw an error because tests are utilizing go-mpatch in order to manipulate the time.Now function. The patch allows testing with fixed timestamps.

Instead, MacOS users can utilize the make test-in-docker target to execute the tests.

Page Tree

diff --git a/dev/external/metalctl/docs/metalctl/index.html b/dev/external/metalctl/docs/metalctl/index.html index faf57f4fed..26127318fe 100644 --- a/dev/external/metalctl/docs/metalctl/index.html +++ b/dev/external/metalctl/docs/metalctl/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_audit/index.html b/dev/external/metalctl/docs/metalctl_audit/index.html index add4d72b25..7a4c2a6ffd 100644 --- a/dev/external/metalctl/docs/metalctl_audit/index.html +++ b/dev/external/metalctl/docs/metalctl_audit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_audit_describe/index.html b/dev/external/metalctl/docs/metalctl_audit_describe/index.html index 87fe28fc10..7ad83dfdb2 100644 --- a/dev/external/metalctl/docs/metalctl_audit_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_audit_describe/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_audit_list/index.html b/dev/external/metalctl/docs/metalctl_audit_list/index.html index fdae3927b2..9f4f8eae3a 100644 --- a/dev/external/metalctl/docs/metalctl_audit_list/index.html +++ b/dev/external/metalctl/docs/metalctl_audit_list/index.html @@ -38,4 +38,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_completion/index.html b/dev/external/metalctl/docs/metalctl_completion/index.html index f70d37d8fa..6bd3e9ee25 100644 --- a/dev/external/metalctl/docs/metalctl_completion/index.html +++ b/dev/external/metalctl/docs/metalctl_completion/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_completion_bash/index.html b/dev/external/metalctl/docs/metalctl_completion_bash/index.html index cce21a7028..1f3d138809 100644 --- a/dev/external/metalctl/docs/metalctl_completion_bash/index.html +++ b/dev/external/metalctl/docs/metalctl_completion_bash/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_completion_fish/index.html b/dev/external/metalctl/docs/metalctl_completion_fish/index.html index 37f362d863..db7c92126c 100644 --- a/dev/external/metalctl/docs/metalctl_completion_fish/index.html +++ b/dev/external/metalctl/docs/metalctl_completion_fish/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_completion_powershell/index.html b/dev/external/metalctl/docs/metalctl_completion_powershell/index.html index 0714506b0f..18f864577e 100644 --- a/dev/external/metalctl/docs/metalctl_completion_powershell/index.html +++ b/dev/external/metalctl/docs/metalctl_completion_powershell/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_completion_zsh/index.html b/dev/external/metalctl/docs/metalctl_completion_zsh/index.html index 29dde6b3f4..edb0cd1e3e 100644 --- a/dev/external/metalctl/docs/metalctl_completion_zsh/index.html +++ b/dev/external/metalctl/docs/metalctl_completion_zsh/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_context/index.html b/dev/external/metalctl/docs/metalctl_context/index.html index 48b6a4d9dd..328886c1de 100644 --- a/dev/external/metalctl/docs/metalctl_context/index.html +++ b/dev/external/metalctl/docs/metalctl_context/index.html @@ -37,4 +37,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_context_short/index.html b/dev/external/metalctl/docs/metalctl_context_short/index.html index cb4f5eaa91..a9b007314d 100644 --- a/dev/external/metalctl/docs/metalctl_context_short/index.html +++ b/dev/external/metalctl/docs/metalctl_context_short/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout/index.html index 5e12b9c12e..3ba0ed06d2 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_apply/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_apply/index.html index 5d3976c8b2..451cb78641 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_apply/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_create/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_create/index.html index 1c137b4498..133db43f4f 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_create/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_create/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_delete/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_delete/index.html index 09339706ff..441bc79e72 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_delete/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_describe/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_describe/index.html index e73bbe321b..452b18189d 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_edit/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_edit/index.html index 69f5fbe92b..9833296469 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_list/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_list/index.html index bce0452604..39817b63b6 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_list/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_list/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_match/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_match/index.html index 7824e09165..9b789c1336 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_match/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_match/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_try/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_try/index.html index 11d2c0e752..e4ee16b220 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_try/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_try/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_filesystemlayout_update/index.html b/dev/external/metalctl/docs/metalctl_filesystemlayout_update/index.html index c002eed374..d67a39d063 100644 --- a/dev/external/metalctl/docs/metalctl_filesystemlayout_update/index.html +++ b/dev/external/metalctl/docs/metalctl_filesystemlayout_update/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firewall/index.html b/dev/external/metalctl/docs/metalctl_firewall/index.html index 8af3835331..fbf215b8c5 100644 --- a/dev/external/metalctl/docs/metalctl_firewall/index.html +++ b/dev/external/metalctl/docs/metalctl_firewall/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firewall_create/index.html b/dev/external/metalctl/docs/metalctl_firewall_create/index.html index 8e556cf7ab..7d7e9a9982 100644 --- a/dev/external/metalctl/docs/metalctl_firewall_create/index.html +++ b/dev/external/metalctl/docs/metalctl_firewall_create/index.html @@ -60,4 +60,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firewall_describe/index.html b/dev/external/metalctl/docs/metalctl_firewall_describe/index.html index 913463448d..52fe686310 100644 --- a/dev/external/metalctl/docs/metalctl_firewall_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_firewall_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firewall_list/index.html b/dev/external/metalctl/docs/metalctl_firewall_list/index.html index 0bdf71b9db..7a6e62763b 100644 --- a/dev/external/metalctl/docs/metalctl_firewall_list/index.html +++ b/dev/external/metalctl/docs/metalctl_firewall_list/index.html @@ -31,4 +31,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firewall_ssh/index.html b/dev/external/metalctl/docs/metalctl_firewall_ssh/index.html index 49a08a56f0..8e093b8a8a 100644 --- a/dev/external/metalctl/docs/metalctl_firewall_ssh/index.html +++ b/dev/external/metalctl/docs/metalctl_firewall_ssh/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firmware/index.html b/dev/external/metalctl/docs/metalctl_firmware/index.html index 999359cd16..5e3ae5c6c3 100644 --- a/dev/external/metalctl/docs/metalctl_firmware/index.html +++ b/dev/external/metalctl/docs/metalctl_firmware/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firmware_delete/index.html b/dev/external/metalctl/docs/metalctl_firmware_delete/index.html index aa46daedf2..f22a5c6ac4 100644 --- a/dev/external/metalctl/docs/metalctl_firmware_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_firmware_delete/index.html @@ -25,4 +25,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firmware_list/index.html b/dev/external/metalctl/docs/metalctl_firmware_list/index.html index 218399a555..6709edc88a 100644 --- a/dev/external/metalctl/docs/metalctl_firmware_list/index.html +++ b/dev/external/metalctl/docs/metalctl_firmware_list/index.html @@ -25,4 +25,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firmware_upload/index.html b/dev/external/metalctl/docs/metalctl_firmware_upload/index.html index 269e94d99a..6d45dcb940 100644 --- a/dev/external/metalctl/docs/metalctl_firmware_upload/index.html +++ b/dev/external/metalctl/docs/metalctl_firmware_upload/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firmware_upload_bios/index.html b/dev/external/metalctl/docs/metalctl_firmware_upload_bios/index.html index 03b23234ec..b007e0a9c4 100644 --- a/dev/external/metalctl/docs/metalctl_firmware_upload_bios/index.html +++ b/dev/external/metalctl/docs/metalctl_firmware_upload_bios/index.html @@ -24,4 +24,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_firmware_upload_bmc/index.html b/dev/external/metalctl/docs/metalctl_firmware_upload_bmc/index.html index 2f399597cd..e23f66fcaa 100644 --- a/dev/external/metalctl/docs/metalctl_firmware_upload_bmc/index.html +++ b/dev/external/metalctl/docs/metalctl_firmware_upload_bmc/index.html @@ -24,4 +24,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_health/index.html b/dev/external/metalctl/docs/metalctl_health/index.html index 93d4b7e2b0..2fa71a925d 100644 --- a/dev/external/metalctl/docs/metalctl_health/index.html +++ b/dev/external/metalctl/docs/metalctl_health/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image/index.html b/dev/external/metalctl/docs/metalctl_image/index.html index c9ba205015..10a99958e7 100644 --- a/dev/external/metalctl/docs/metalctl_image/index.html +++ b/dev/external/metalctl/docs/metalctl_image/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image_apply/index.html b/dev/external/metalctl/docs/metalctl_image_apply/index.html index a168a3d5f3..820a20275b 100644 --- a/dev/external/metalctl/docs/metalctl_image_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_image_apply/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image_create/index.html b/dev/external/metalctl/docs/metalctl_image_create/index.html index f6c8f15521..33629793ca 100644 --- a/dev/external/metalctl/docs/metalctl_image_create/index.html +++ b/dev/external/metalctl/docs/metalctl_image_create/index.html @@ -41,4 +41,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image_delete/index.html b/dev/external/metalctl/docs/metalctl_image_delete/index.html index 05dbc933ba..8690afa524 100644 --- a/dev/external/metalctl/docs/metalctl_image_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_image_delete/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image_describe/index.html b/dev/external/metalctl/docs/metalctl_image_describe/index.html index f11cd02498..48e6bf1913 100644 --- a/dev/external/metalctl/docs/metalctl_image_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_image_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image_edit/index.html b/dev/external/metalctl/docs/metalctl_image_edit/index.html index d7c0bca486..cebdd11f2c 100644 --- a/dev/external/metalctl/docs/metalctl_image_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_image_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image_list/index.html b/dev/external/metalctl/docs/metalctl_image_list/index.html index f9d31d6ee5..e7b90e30e3 100644 --- a/dev/external/metalctl/docs/metalctl_image_list/index.html +++ b/dev/external/metalctl/docs/metalctl_image_list/index.html @@ -29,4 +29,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_image_update/index.html b/dev/external/metalctl/docs/metalctl_image_update/index.html index cabcb8683f..81ff84637e 100644 --- a/dev/external/metalctl/docs/metalctl_image_update/index.html +++ b/dev/external/metalctl/docs/metalctl_image_update/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_login/index.html b/dev/external/metalctl/docs/metalctl_login/index.html index 84013d9740..ae4d3cf780 100644 --- a/dev/external/metalctl/docs/metalctl_login/index.html +++ b/dev/external/metalctl/docs/metalctl_login/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_logout/index.html b/dev/external/metalctl/docs/metalctl_logout/index.html index ec6163e99a..100c4495e9 100644 --- a/dev/external/metalctl/docs/metalctl_logout/index.html +++ b/dev/external/metalctl/docs/metalctl_logout/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine/index.html b/dev/external/metalctl/docs/metalctl_machine/index.html index 21fccb927b..d6b3fa3285 100644 --- a/dev/external/metalctl/docs/metalctl_machine/index.html +++ b/dev/external/metalctl/docs/metalctl_machine/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_apply/index.html b/dev/external/metalctl/docs/metalctl_machine_apply/index.html index f402c8fc92..278ba62b83 100644 --- a/dev/external/metalctl/docs/metalctl_machine_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_apply/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_console/index.html b/dev/external/metalctl/docs/metalctl_machine_console/index.html index 0cadf3a476..e6e4cc543c 100644 --- a/dev/external/metalctl/docs/metalctl_machine_console/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_console/index.html @@ -25,4 +25,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_consolepassword/index.html b/dev/external/metalctl/docs/metalctl_machine_consolepassword/index.html index 10521bde44..ebf0ffcb8a 100644 --- a/dev/external/metalctl/docs/metalctl_machine_consolepassword/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_consolepassword/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_create/index.html b/dev/external/metalctl/docs/metalctl_machine_create/index.html index f78bc82363..1769f3e315 100644 --- a/dev/external/metalctl/docs/metalctl_machine_create/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_create/index.html @@ -93,4 +93,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_delete/index.html b/dev/external/metalctl/docs/metalctl_machine_delete/index.html index 733ae88bb0..4dc6f07ba5 100644 --- a/dev/external/metalctl/docs/metalctl_machine_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_delete/index.html @@ -37,4 +37,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_describe/index.html b/dev/external/metalctl/docs/metalctl_machine_describe/index.html index dc99adc304..e2f06b9e15 100644 --- a/dev/external/metalctl/docs/metalctl_machine_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_edit/index.html b/dev/external/metalctl/docs/metalctl_machine_edit/index.html index 8f39857d53..cad12ef89a 100644 --- a/dev/external/metalctl/docs/metalctl_machine_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_identify/index.html b/dev/external/metalctl/docs/metalctl_machine_identify/index.html index db7bf0aa77..ebfb063250 100644 --- a/dev/external/metalctl/docs/metalctl_machine_identify/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_identify/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_identify_off/index.html b/dev/external/metalctl/docs/metalctl_machine_identify_off/index.html index 6709162109..be5f84ad12 100644 --- a/dev/external/metalctl/docs/metalctl_machine_identify_off/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_identify_off/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_identify_on/index.html b/dev/external/metalctl/docs/metalctl_machine_identify_on/index.html index 46f4239d67..c0db2f802f 100644 --- a/dev/external/metalctl/docs/metalctl_machine_identify_on/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_identify_on/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_ipmi/index.html b/dev/external/metalctl/docs/metalctl_machine_ipmi/index.html index 1026e90810..106c9cedf6 100644 --- a/dev/external/metalctl/docs/metalctl_machine_ipmi/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_ipmi/index.html @@ -44,4 +44,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_ipmi_events/index.html b/dev/external/metalctl/docs/metalctl_machine_ipmi_events/index.html index de2d5aa1eb..8403107d70 100644 --- a/dev/external/metalctl/docs/metalctl_machine_ipmi_events/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_ipmi_events/index.html @@ -24,4 +24,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_issues/index.html b/dev/external/metalctl/docs/metalctl_machine_issues/index.html index 92694952bc..7183d95088 100644 --- a/dev/external/metalctl/docs/metalctl_machine_issues/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_issues/index.html @@ -47,4 +47,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_issues_list/index.html b/dev/external/metalctl/docs/metalctl_machine_issues_list/index.html index b42abc7340..da47cebf68 100644 --- a/dev/external/metalctl/docs/metalctl_machine_issues_list/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_issues_list/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_list/index.html b/dev/external/metalctl/docs/metalctl_machine_list/index.html index 28505791de..9d2ba12b49 100644 --- a/dev/external/metalctl/docs/metalctl_machine_list/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_list/index.html @@ -44,4 +44,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_lock/index.html b/dev/external/metalctl/docs/metalctl_machine_lock/index.html index af77e69305..d8ba47ab1c 100644 --- a/dev/external/metalctl/docs/metalctl_machine_lock/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_lock/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_logs/index.html b/dev/external/metalctl/docs/metalctl_machine_logs/index.html index 6727401803..7f1b769d95 100644 --- a/dev/external/metalctl/docs/metalctl_machine_logs/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_logs/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power/index.html b/dev/external/metalctl/docs/metalctl_machine_power/index.html index 2437405ae8..c418145500 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power_bios/index.html b/dev/external/metalctl/docs/metalctl_machine_power_bios/index.html index aa9527e6c9..dddaa92f00 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power_bios/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power_bios/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power_cycle/index.html b/dev/external/metalctl/docs/metalctl_machine_power_cycle/index.html index c20780218d..718b6791da 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power_cycle/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power_cycle/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power_disk/index.html b/dev/external/metalctl/docs/metalctl_machine_power_disk/index.html index b658eac50e..403359d416 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power_disk/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power_disk/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power_off/index.html b/dev/external/metalctl/docs/metalctl_machine_power_off/index.html index 7c2f67fd37..b3636fbeb1 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power_off/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power_off/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power_on/index.html b/dev/external/metalctl/docs/metalctl_machine_power_on/index.html index c1580af019..d4e610dd33 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power_on/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power_on/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power_pxe/index.html b/dev/external/metalctl/docs/metalctl_machine_power_pxe/index.html index 5a6f058666..0bdcf056f1 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power_pxe/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power_pxe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_power_reset/index.html b/dev/external/metalctl/docs/metalctl_machine_power_reset/index.html index 717a4a8c71..e6b14e4cf0 100644 --- a/dev/external/metalctl/docs/metalctl_machine_power_reset/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_power_reset/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_reinstall/index.html b/dev/external/metalctl/docs/metalctl_machine_reinstall/index.html index f5ea5d10d6..16498af82a 100644 --- a/dev/external/metalctl/docs/metalctl_machine_reinstall/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_reinstall/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_reserve/index.html b/dev/external/metalctl/docs/metalctl_machine_reserve/index.html index 70732e72e7..bfa09b6777 100644 --- a/dev/external/metalctl/docs/metalctl_machine_reserve/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_reserve/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_update-firmware/index.html b/dev/external/metalctl/docs/metalctl_machine_update-firmware/index.html index 43f123a947..3fbea806de 100644 --- a/dev/external/metalctl/docs/metalctl_machine_update-firmware/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_update-firmware/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_update-firmware_bios/index.html b/dev/external/metalctl/docs/metalctl_machine_update-firmware_bios/index.html index 6827a42fef..efb789a51e 100644 --- a/dev/external/metalctl/docs/metalctl_machine_update-firmware_bios/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_update-firmware_bios/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_update-firmware_bmc/index.html b/dev/external/metalctl/docs/metalctl_machine_update-firmware_bmc/index.html index cacebf5856..e44e865fb8 100644 --- a/dev/external/metalctl/docs/metalctl_machine_update-firmware_bmc/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_update-firmware_bmc/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_machine_update/index.html b/dev/external/metalctl/docs/metalctl_machine_update/index.html index 64765d4bd9..090ab3cf64 100644 --- a/dev/external/metalctl/docs/metalctl_machine_update/index.html +++ b/dev/external/metalctl/docs/metalctl_machine_update/index.html @@ -39,4 +39,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_markdown/index.html b/dev/external/metalctl/docs/metalctl_markdown/index.html index e55fc8c332..0c148103db 100644 --- a/dev/external/metalctl/docs/metalctl_markdown/index.html +++ b/dev/external/metalctl/docs/metalctl_markdown/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network/index.html b/dev/external/metalctl/docs/metalctl_network/index.html index 83917ea935..6cc4c1d085 100644 --- a/dev/external/metalctl/docs/metalctl_network/index.html +++ b/dev/external/metalctl/docs/metalctl_network/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_allocate/index.html b/dev/external/metalctl/docs/metalctl_network_allocate/index.html index 6942b444ff..2e501495b1 100644 --- a/dev/external/metalctl/docs/metalctl_network_allocate/index.html +++ b/dev/external/metalctl/docs/metalctl_network_allocate/index.html @@ -28,4 +28,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_apply/index.html b/dev/external/metalctl/docs/metalctl_network_apply/index.html index ea14868893..6f857e2f0a 100644 --- a/dev/external/metalctl/docs/metalctl_network_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_network_apply/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_create/index.html b/dev/external/metalctl/docs/metalctl_network_create/index.html index 3720a80a2c..d52dec6ac0 100644 --- a/dev/external/metalctl/docs/metalctl_network_create/index.html +++ b/dev/external/metalctl/docs/metalctl_network_create/index.html @@ -49,4 +49,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_delete/index.html b/dev/external/metalctl/docs/metalctl_network_delete/index.html index ebe1cf76b1..0c5bb6a09b 100644 --- a/dev/external/metalctl/docs/metalctl_network_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_network_delete/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_describe/index.html b/dev/external/metalctl/docs/metalctl_network_describe/index.html index e887f9ee46..1cca134bcd 100644 --- a/dev/external/metalctl/docs/metalctl_network_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_network_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_edit/index.html b/dev/external/metalctl/docs/metalctl_network_edit/index.html index 99feeea96e..c53fd1e3c1 100644 --- a/dev/external/metalctl/docs/metalctl_network_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_network_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_free/index.html b/dev/external/metalctl/docs/metalctl_network_free/index.html index edeba5105d..c7ed0dc7a3 100644 --- a/dev/external/metalctl/docs/metalctl_network_free/index.html +++ b/dev/external/metalctl/docs/metalctl_network_free/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip/index.html b/dev/external/metalctl/docs/metalctl_network_ip/index.html index 63093a7969..1760fc59fd 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_apply/index.html b/dev/external/metalctl/docs/metalctl_network_ip_apply/index.html index e06e569904..780343e6a6 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_apply/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_create/index.html b/dev/external/metalctl/docs/metalctl_network_ip_create/index.html index ff40fd2ed1..34282b6e0f 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_create/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_create/index.html @@ -43,4 +43,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_delete/index.html b/dev/external/metalctl/docs/metalctl_network_ip_delete/index.html index 9abfa945da..4e882fbee2 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_delete/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_describe/index.html b/dev/external/metalctl/docs/metalctl_network_ip_describe/index.html index 4457729ee3..edbab89f2f 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_edit/index.html b/dev/external/metalctl/docs/metalctl_network_ip_edit/index.html index c87bbc6fc7..89d91a0327 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_issues/index.html b/dev/external/metalctl/docs/metalctl_network_ip_issues/index.html index f8c8d133cf..9352df0cc0 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_issues/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_issues/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_list/index.html b/dev/external/metalctl/docs/metalctl_network_ip_list/index.html index 69065e6df3..77cb92b307 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_list/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_list/index.html @@ -30,4 +30,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_ip_update/index.html b/dev/external/metalctl/docs/metalctl_network_ip_update/index.html index 3144b218ab..276be3e9a1 100644 --- a/dev/external/metalctl/docs/metalctl_network_ip_update/index.html +++ b/dev/external/metalctl/docs/metalctl_network_ip_update/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_list/index.html b/dev/external/metalctl/docs/metalctl_network_list/index.html index ac43d2c705..88639957bb 100644 --- a/dev/external/metalctl/docs/metalctl_network_list/index.html +++ b/dev/external/metalctl/docs/metalctl_network_list/index.html @@ -33,4 +33,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_network_update/index.html b/dev/external/metalctl/docs/metalctl_network_update/index.html index d29892c1fa..93632e3d89 100644 --- a/dev/external/metalctl/docs/metalctl_network_update/index.html +++ b/dev/external/metalctl/docs/metalctl_network_update/index.html @@ -44,4 +44,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition/index.html b/dev/external/metalctl/docs/metalctl_partition/index.html index 83f6c60c57..b6c99a6199 100644 --- a/dev/external/metalctl/docs/metalctl_partition/index.html +++ b/dev/external/metalctl/docs/metalctl_partition/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_apply/index.html b/dev/external/metalctl/docs/metalctl_partition_apply/index.html index 540cce7ae4..6362b1007d 100644 --- a/dev/external/metalctl/docs/metalctl_partition_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_apply/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_capacity/index.html b/dev/external/metalctl/docs/metalctl_partition_capacity/index.html index 1b751a7f54..1d294974d4 100644 --- a/dev/external/metalctl/docs/metalctl_partition_capacity/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_capacity/index.html @@ -24,4 +24,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_create/index.html b/dev/external/metalctl/docs/metalctl_partition_create/index.html index 8d2a0d747e..08fa53b2b4 100644 --- a/dev/external/metalctl/docs/metalctl_partition_create/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_create/index.html @@ -43,4 +43,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_delete/index.html b/dev/external/metalctl/docs/metalctl_partition_delete/index.html index 479bd20b2b..1790cd93dc 100644 --- a/dev/external/metalctl/docs/metalctl_partition_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_delete/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_describe/index.html b/dev/external/metalctl/docs/metalctl_partition_describe/index.html index 5e450ae82c..809903233a 100644 --- a/dev/external/metalctl/docs/metalctl_partition_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_edit/index.html b/dev/external/metalctl/docs/metalctl_partition_edit/index.html index 61bdeff3ad..60432984ec 100644 --- a/dev/external/metalctl/docs/metalctl_partition_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_list/index.html b/dev/external/metalctl/docs/metalctl_partition_list/index.html index adb5f9f257..d688451b86 100644 --- a/dev/external/metalctl/docs/metalctl_partition_list/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_list/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_partition_update/index.html b/dev/external/metalctl/docs/metalctl_partition_update/index.html index 6e14ef7f47..e453f96103 100644 --- a/dev/external/metalctl/docs/metalctl_partition_update/index.html +++ b/dev/external/metalctl/docs/metalctl_partition_update/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project/index.html b/dev/external/metalctl/docs/metalctl_project/index.html index 0b3a7d99fb..9d83c70a3b 100644 --- a/dev/external/metalctl/docs/metalctl_project/index.html +++ b/dev/external/metalctl/docs/metalctl_project/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project_apply/index.html b/dev/external/metalctl/docs/metalctl_project_apply/index.html index a64173e2c9..b55d73ad8e 100644 --- a/dev/external/metalctl/docs/metalctl_project_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_project_apply/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project_create/index.html b/dev/external/metalctl/docs/metalctl_project_create/index.html index 1d527c534a..a4d00f1ed1 100644 --- a/dev/external/metalctl/docs/metalctl_project_create/index.html +++ b/dev/external/metalctl/docs/metalctl_project_create/index.html @@ -44,4 +44,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project_delete/index.html b/dev/external/metalctl/docs/metalctl_project_delete/index.html index f0f276d5bf..e0b0cb7db9 100644 --- a/dev/external/metalctl/docs/metalctl_project_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_project_delete/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project_describe/index.html b/dev/external/metalctl/docs/metalctl_project_describe/index.html index cfea99675d..7a0893225d 100644 --- a/dev/external/metalctl/docs/metalctl_project_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_project_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project_edit/index.html b/dev/external/metalctl/docs/metalctl_project_edit/index.html index 96d2a4a09a..f08591df62 100644 --- a/dev/external/metalctl/docs/metalctl_project_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_project_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project_list/index.html b/dev/external/metalctl/docs/metalctl_project_list/index.html index a7e99fee91..cb53bee57e 100644 --- a/dev/external/metalctl/docs/metalctl_project_list/index.html +++ b/dev/external/metalctl/docs/metalctl_project_list/index.html @@ -25,4 +25,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_project_update/index.html b/dev/external/metalctl/docs/metalctl_project_update/index.html index da557b80d2..8cbcb14b35 100644 --- a/dev/external/metalctl/docs/metalctl_project_update/index.html +++ b/dev/external/metalctl/docs/metalctl_project_update/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size/index.html b/dev/external/metalctl/docs/metalctl_size/index.html index bb7d281728..85427bfc29 100644 --- a/dev/external/metalctl/docs/metalctl_size/index.html +++ b/dev/external/metalctl/docs/metalctl_size/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_apply/index.html b/dev/external/metalctl/docs/metalctl_size_apply/index.html index 53af741c8a..267a63abfe 100644 --- a/dev/external/metalctl/docs/metalctl_size_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_size_apply/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_create/index.html b/dev/external/metalctl/docs/metalctl_size_create/index.html index fdcf0c4c3d..1bb84ca28c 100644 --- a/dev/external/metalctl/docs/metalctl_size_create/index.html +++ b/dev/external/metalctl/docs/metalctl_size_create/index.html @@ -42,4 +42,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_delete/index.html b/dev/external/metalctl/docs/metalctl_size_delete/index.html index fb9a2a281f..618e41fcd1 100644 --- a/dev/external/metalctl/docs/metalctl_size_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_size_delete/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_describe/index.html b/dev/external/metalctl/docs/metalctl_size_describe/index.html index 3a405e0b7c..1984857faa 100644 --- a/dev/external/metalctl/docs/metalctl_size_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_size_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_edit/index.html b/dev/external/metalctl/docs/metalctl_size_edit/index.html index a13bbd2116..8550fbd214 100644 --- a/dev/external/metalctl/docs/metalctl_size_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_size_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint/index.html index 59eab06ed6..0e16136e91 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_apply/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_apply/index.html index cc79dfb562..3b4290f732 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_apply/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_create/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_create/index.html index 95853115b4..5edea5c110 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_create/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_create/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_delete/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_delete/index.html index 26c2f15b1d..89c41432f4 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_delete/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_describe/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_describe/index.html index 1bf35ff665..f58b4f2376 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_edit/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_edit/index.html index 6e16cf1a4a..0376a99760 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_list/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_list/index.html index 61d24f6ba2..1b4450cdfc 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_list/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_list/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_try/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_try/index.html index 1397f34f7b..7259a29e0b 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_try/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_try/index.html @@ -23,4 +23,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_imageconstraint_update/index.html b/dev/external/metalctl/docs/metalctl_size_imageconstraint_update/index.html index 14b0a975a0..664a116bc4 100644 --- a/dev/external/metalctl/docs/metalctl_size_imageconstraint_update/index.html +++ b/dev/external/metalctl/docs/metalctl_size_imageconstraint_update/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_list/index.html b/dev/external/metalctl/docs/metalctl_size_list/index.html index fdd0aca667..663bf298e7 100644 --- a/dev/external/metalctl/docs/metalctl_size_list/index.html +++ b/dev/external/metalctl/docs/metalctl_size_list/index.html @@ -22,4 +22,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_try/index.html b/dev/external/metalctl/docs/metalctl_size_try/index.html index 900f0bed4c..190399d206 100644 --- a/dev/external/metalctl/docs/metalctl_size_try/index.html +++ b/dev/external/metalctl/docs/metalctl_size_try/index.html @@ -24,4 +24,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_size_update/index.html b/dev/external/metalctl/docs/metalctl_size_update/index.html index 3a459a3833..0aa01a9c97 100644 --- a/dev/external/metalctl/docs/metalctl_size_update/index.html +++ b/dev/external/metalctl/docs/metalctl_size_update/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch/index.html b/dev/external/metalctl/docs/metalctl_switch/index.html index 468d2b3ad0..6a5a569ad1 100644 --- a/dev/external/metalctl/docs/metalctl_switch/index.html +++ b/dev/external/metalctl/docs/metalctl_switch/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_connected-machines/index.html b/dev/external/metalctl/docs/metalctl_switch_connected-machines/index.html index b61f17ecb5..5f8532dabc 100644 --- a/dev/external/metalctl/docs/metalctl_switch_connected-machines/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_connected-machines/index.html @@ -37,4 +37,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_console/index.html b/dev/external/metalctl/docs/metalctl_switch_console/index.html index 02afc78f9e..0892a85ee4 100644 --- a/dev/external/metalctl/docs/metalctl_switch_console/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_console/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_delete/index.html b/dev/external/metalctl/docs/metalctl_switch_delete/index.html index aead8d3aa0..20c2536634 100644 --- a/dev/external/metalctl/docs/metalctl_switch_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_delete/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_describe/index.html b/dev/external/metalctl/docs/metalctl_switch_describe/index.html index e069a87737..069ab7ffe0 100644 --- a/dev/external/metalctl/docs/metalctl_switch_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_detail/index.html b/dev/external/metalctl/docs/metalctl_switch_detail/index.html index 1e91e60d8a..14ac71ade1 100644 --- a/dev/external/metalctl/docs/metalctl_switch_detail/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_detail/index.html @@ -27,4 +27,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_edit/index.html b/dev/external/metalctl/docs/metalctl_switch_edit/index.html index a82006aee6..a8799467e8 100644 --- a/dev/external/metalctl/docs/metalctl_switch_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_list/index.html b/dev/external/metalctl/docs/metalctl_switch_list/index.html index 7dcf2a870f..89b08e0837 100644 --- a/dev/external/metalctl/docs/metalctl_switch_list/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_list/index.html @@ -28,4 +28,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_replace/index.html b/dev/external/metalctl/docs/metalctl_switch_replace/index.html index 8f193bad39..868084d87d 100644 --- a/dev/external/metalctl/docs/metalctl_switch_replace/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_replace/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_ssh/index.html b/dev/external/metalctl/docs/metalctl_switch_ssh/index.html index f28efc82ab..fd1190cbb5 100644 --- a/dev/external/metalctl/docs/metalctl_switch_ssh/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_ssh/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_switch_update/index.html b/dev/external/metalctl/docs/metalctl_switch_update/index.html index d81d1906e1..0b1288a8b8 100644 --- a/dev/external/metalctl/docs/metalctl_switch_update/index.html +++ b/dev/external/metalctl/docs/metalctl_switch_update/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_tenant/index.html b/dev/external/metalctl/docs/metalctl_tenant/index.html index e68984d138..65ff6f7032 100644 --- a/dev/external/metalctl/docs/metalctl_tenant/index.html +++ b/dev/external/metalctl/docs/metalctl_tenant/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_tenant_apply/index.html b/dev/external/metalctl/docs/metalctl_tenant_apply/index.html index 63faed321f..fdbc6ca187 100644 --- a/dev/external/metalctl/docs/metalctl_tenant_apply/index.html +++ b/dev/external/metalctl/docs/metalctl_tenant_apply/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_tenant_create/index.html b/dev/external/metalctl/docs/metalctl_tenant_create/index.html index 3f61391a08..10767c8570 100644 --- a/dev/external/metalctl/docs/metalctl_tenant_create/index.html +++ b/dev/external/metalctl/docs/metalctl_tenant_create/index.html @@ -44,4 +44,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_tenant_delete/index.html b/dev/external/metalctl/docs/metalctl_tenant_delete/index.html index 8aabd13364..c6a7aa49d2 100644 --- a/dev/external/metalctl/docs/metalctl_tenant_delete/index.html +++ b/dev/external/metalctl/docs/metalctl_tenant_delete/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_tenant_describe/index.html b/dev/external/metalctl/docs/metalctl_tenant_describe/index.html index 2bcf5fd864..fbd284e677 100644 --- a/dev/external/metalctl/docs/metalctl_tenant_describe/index.html +++ b/dev/external/metalctl/docs/metalctl_tenant_describe/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_tenant_edit/index.html b/dev/external/metalctl/docs/metalctl_tenant_edit/index.html index 1caa76ef0a..4cc5454d99 100644 --- a/dev/external/metalctl/docs/metalctl_tenant_edit/index.html +++ b/dev/external/metalctl/docs/metalctl_tenant_edit/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_tenant_list/index.html b/dev/external/metalctl/docs/metalctl_tenant_list/index.html index 5eaaee0404..32ce8a76f6 100644 --- a/dev/external/metalctl/docs/metalctl_tenant_list/index.html +++ b/dev/external/metalctl/docs/metalctl_tenant_list/index.html @@ -25,4 +25,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_tenant_update/index.html b/dev/external/metalctl/docs/metalctl_tenant_update/index.html index d3a94a1c4f..a7e51eaebe 100644 --- a/dev/external/metalctl/docs/metalctl_tenant_update/index.html +++ b/dev/external/metalctl/docs/metalctl_tenant_update/index.html @@ -36,4 +36,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_update/index.html b/dev/external/metalctl/docs/metalctl_update/index.html index e6efb1fee0..2ab2f6ee42 100644 --- a/dev/external/metalctl/docs/metalctl_update/index.html +++ b/dev/external/metalctl/docs/metalctl_update/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_update_check/index.html b/dev/external/metalctl/docs/metalctl_update_check/index.html index 97983b933b..2150451b3a 100644 --- a/dev/external/metalctl/docs/metalctl_update_check/index.html +++ b/dev/external/metalctl/docs/metalctl_update_check/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_update_do/index.html b/dev/external/metalctl/docs/metalctl_update_do/index.html index f2fe0a9603..ca1c95bbd3 100644 --- a/dev/external/metalctl/docs/metalctl_update_do/index.html +++ b/dev/external/metalctl/docs/metalctl_update_do/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_version/index.html b/dev/external/metalctl/docs/metalctl_version/index.html index 2ab075e0e4..b013289e07 100644 --- a/dev/external/metalctl/docs/metalctl_version/index.html +++ b/dev/external/metalctl/docs/metalctl_version/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_vpn/index.html b/dev/external/metalctl/docs/metalctl_vpn/index.html index 11829b9d27..4186f6960c 100644 --- a/dev/external/metalctl/docs/metalctl_vpn/index.html +++ b/dev/external/metalctl/docs/metalctl_vpn/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_vpn_key/index.html b/dev/external/metalctl/docs/metalctl_vpn_key/index.html index fb5770eb64..5aedc679d0 100644 --- a/dev/external/metalctl/docs/metalctl_vpn_key/index.html +++ b/dev/external/metalctl/docs/metalctl_vpn_key/index.html @@ -26,4 +26,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/metalctl/docs/metalctl_whoami/index.html b/dev/external/metalctl/docs/metalctl_whoami/index.html index 8ea5a56389..65935294d7 100644 --- a/dev/external/metalctl/docs/metalctl_whoami/index.html +++ b/dev/external/metalctl/docs/metalctl_whoami/index.html @@ -21,4 +21,4 @@ metalctl machine list -o template --template "{{ .id }}:{{ .size.id }}" - --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

+ --yes-i-really-mean-it skips security prompts (which can be dangerous to set blindly because actions can lead to data loss or additional costs)

SEE ALSO

diff --git a/dev/external/mini-lab/CONTRIBUTING/index.html b/dev/external/mini-lab/CONTRIBUTING/index.html index d198528c4d..797fca6903 100644 --- a/dev/external/mini-lab/CONTRIBUTING/index.html +++ b/dev/external/mini-lab/CONTRIBUTING/index.html @@ -1,2 +1,2 @@ -Contributing · metal-stack
Edit on GitHub
+Contributing · metal-stack
Edit on GitHub
diff --git a/dev/external/mini-lab/README/index.html b/dev/external/mini-lab/README/index.html index 70d38e9018..dbd2cd399f 100644 --- a/dev/external/mini-lab/README/index.html +++ b/dev/external/mini-lab/README/index.html @@ -1,5 +1,5 @@ -mini-lab · metal-stack
Edit on GitHub

mini-lab

The mini-lab is a small, virtual setup to locally run the metal-stack. It deploys the metal control plane and a metal-stack partition with two simulated leaf switches. The lab can be used for trying out metal-stack, demonstration purposes or development.

overview components

ℹ This project can also be used as a template for writing your own metal-stack deployments.

<!– TOC depthfrom:2 depthto:6 withlinks:true updateonsave:false orderedlist:false –>

<!– /TOC –>

Requirements

  • Linux machine with hardware virtualization support
  • kvm as hypervisor for the VMs (you can check through the kvm-ok command)
  • docker >= 18.09 (for using kind and our deployment base image)
  • docker-compose >= 2.0 (for ease of use and for parallelizing control plane and partition deployment)
  • kind == v0.15.0 (for hosting the metal control plane on a kubernetes cluster v1.25)
  • containerlab == v0.25.1
  • the lab creates a docker network on your host machine (172.17.0.1), this hopefully does not overlap with other networks you have
  • (recommended) haveged to have enough random entropy (only needed if the PXE process does not work)

Here is some code that should help you to set up most of the requirements:

# If UFW enabled.
+mini-lab · metal-stack
Edit on GitHub
mini-lab
The mini-lab is a small, virtual setup to locally run the metal-stack. It deploys the metal control plane and a metal-stack partition with two simulated leaf switches. The lab can be used for trying out metal-stack, demonstration purposes or development.
overview components
ℹ This project can also be used as a template for writing your own metal-stack deployments.
<!– TOC depthfrom:2 depthto:6 withlinks:true updateonsave:false orderedlist:false –>
<!– /TOC –>
Requirements
  • Linux machine with hardware virtualization support
  • kvm as hypervisor for the VMs (you can check through the kvm-ok command)
  • docker >= 18.09 (for using kind and our deployment base image)
  • docker-compose >= 2.0 (for ease of use and for parallelizing control plane and partition deployment)
  • kind == v0.15.0 (for hosting the metal control plane on a kubernetes cluster v1.25)
  • containerlab == v0.25.1
  • the lab creates a docker network on your host machine (172.17.0.1), this hopefully does not overlap with other networks you have
  • (recommended) haveged to have enough random entropy (only needed if the PXE process does not work)
Here is some code that should help you to set up most of the requirements:
# If UFW enabled.
 # Disable the firewall or allow traffic through Docker network IP range.
 sudo ufw status
 sudo ufw allow from 172.17.0.0/16
@@ -75,4 +75,4 @@
 ssh metal@100.255.254.1
To remove the kind cluster, the switches and machines, run:
make cleanup
Reinstall machine
Reinstall a machine with
docker-compose run --rm metalctl machine reinstall \
         --image ubuntu-20.04 \
         e0ab02d2-27cd-5a5e-8efc-080ba80cf258
Free machine
Free a machine with make free-machine01 or
docker-compose run --rm metalctl machine rm e0ab02d2-27cd-5a5e-8efc-080ba80cf258
Flavors
There's few versions of mini-lab environment that you can run. We call them flavors. There's 2 flavors at the moment:
  • default – runs 2 machines.
  • cluster-api – runs 3 machines. Useful for testing Control plane and worker node deployment with Cluster API provider.
  • sonic – use SONiC as network operating system for the leaves
In order to start specific flavor, you can define the flavor as follows:
export MINI_LAB_FLAVOR=cluster-api
-make
Page Tree

+make

Page Tree

diff --git a/dev/index.html b/dev/index.html index 6f9d7aaf3c..10ab1839f7 100644 --- a/dev/index.html +++ b/dev/index.html @@ -1,2 +1,2 @@ -Introduction · metal-stack
Edit on GitHub

Welcome to the metal-stack docs!

metal-stack is an open source software that provides an API for provisioning and managing physical servers in the data center. To categorize this product, we use the terms Metal-as-a-Service (MaaS) or bare metal cloud.

From the perspective of a user, the metal-stack does not feel any different from working with a conventional cloud provider. Users manage their resources (machines, networks and ip addresses, etc.) by themselves, which effectively turns your data center into an elastic cloud infrastructure.

The major difference to other cloud providers is that compute power and data reside in your own data center.

Releases

Your are currently reading the documentation for the metal-stack master release.

Releases and integration tests are published through our release repository. You can also find the release notes for this metal-stack version in there. The release notes contain information about new features, upgrade paths and bug fixes.

If you want, you can sign up at our Slack channel where we are announcing every new release. Often, we provide additional information for metal-stack administrators and adopters at this place, too.

Why metal-stack?

Before we started with our mission to implement the metal-stack, we decided on a couple of key characteristics and constraints that we think are unique in the domain (otherwise we would definitely have chosen an existing solution).

We hope that the following properties appeal to you as well.

On-Premise

Running on-premise gives you data sovereignty and usually a better price / performance ratio than with hyperscalers — especially the larger you grow your environment. Another benefit of running on-premise is an easier connectivity to existing company networks.

Fast Provisioning

Provisioning bare metal machines should not feel much different from virtual machines. metal-stack is capable of provisioning servers in less than a minute. The underlying network topology is based on BGP and allows announcing new routes to your host machines in a matter of seconds.

No-Ops

Part of the metal-stack runs on dedicated switches in your data center. This way, it is possible to automate server inventorization, permanently reconcile network configuration and automatically manage machine lifecycles. Manual configuration is neither required nor wanted.

Security

Our networking approach was designed for highest standards on security. Also, we enforce firewalling on dedicated tenant firewalls before users can establish connections to other networks than their private tenant network. API authentication and authorization is done with the help of OIDC.

API driven

The development of metal-stack is strictly API driven and offers self-service to end-users. This approach delivers the highest possible degree of automation, maintainability and performance.

Ready for Kubernetes

Not only does the metal-stack run smoothly on Kubernetes (K8s). The major intent of metal-stack has always been to build a scalable machine infrastructure for Kubernetes as a Service (KaaS). In partnership with the open-source project Gardener, we can provision Kubernetes clusters on metal-stack at scale.

From the perspective of the Gardener, the metal-stack is just another cloud provider. The time savings compared to providing machines and Kubernetes by hand are significant. We actually want to be able to compete with offers of public cloud providers, especially regarding speed and usability.

Of course, you can use metal-stack only for machine provisioning as well and just put something else on top of your metal infrastructure.

Open Source

The metal-stack is open source and free of constraints regarding vendors and third-party products. The stack is completely built on open source products. We have a community actively working on the metal-stack, which can assist you delivering all reasonable features you are gonna need.

Why Bare Metal?

Bare metal has several advantages over virtual environments and overcomes several drawbacks of virtual machines. We also listed drawbacks of the bare metal approach. Bare in mind though that it is still possible to virtualize on bare metal environments when you have your stack up and running.

Virtual Environment Drawbacks

  • Spectre and Meltdown can only be mitigated with a "cluster per tenant" approach
  • Missing isolation of multi-tenant change impacts
  • Licensing restrictions
  • Noisy-neighbors

Bare Metal Advantages

  • Guaranteed and fastest possible performance (especially disk i/o)
  • Reduced stack depth (Host / VM / Application vs. Host / Container)
    • Reduced attack surface
    • Lower costs, higher performance
    • No VM live-migrations
  • Bigger hardware configurations possible (hypervisors have restrictions, e.g. it is not possible to assign all CPUs to a single VM)

Bare Metal Drawbacks

  • Hardware defects have direct impact (should be considered by design) and can not be mitigated by live-migration as in virtual environments
  • Capacity planning is more difficult (no resource overbooking possible)
+Introduction · metal-stack
Edit on GitHub

Welcome to the metal-stack docs!

metal-stack is an open source software that provides an API for provisioning and managing physical servers in the data center. To categorize this product, we use the terms Metal-as-a-Service (MaaS) or bare metal cloud.

From the perspective of a user, the metal-stack does not feel any different from working with a conventional cloud provider. Users manage their resources (machines, networks and ip addresses, etc.) by themselves, which effectively turns your data center into an elastic cloud infrastructure.

The major difference to other cloud providers is that compute power and data reside in your own data center.

Releases

Your are currently reading the documentation for the metal-stack master release.

Releases and integration tests are published through our release repository. You can also find the release notes for this metal-stack version in there. The release notes contain information about new features, upgrade paths and bug fixes.

If you want, you can sign up at our Slack channel where we are announcing every new release. Often, we provide additional information for metal-stack administrators and adopters at this place, too.

Why metal-stack?

Before we started with our mission to implement the metal-stack, we decided on a couple of key characteristics and constraints that we think are unique in the domain (otherwise we would definitely have chosen an existing solution).

We hope that the following properties appeal to you as well.

On-Premise

Running on-premise gives you data sovereignty and usually a better price / performance ratio than with hyperscalers — especially the larger you grow your environment. Another benefit of running on-premise is an easier connectivity to existing company networks.

Fast Provisioning

Provisioning bare metal machines should not feel much different from virtual machines. metal-stack is capable of provisioning servers in less than a minute. The underlying network topology is based on BGP and allows announcing new routes to your host machines in a matter of seconds.

No-Ops

Part of the metal-stack runs on dedicated switches in your data center. This way, it is possible to automate server inventorization, permanently reconcile network configuration and automatically manage machine lifecycles. Manual configuration is neither required nor wanted.

Security

Our networking approach was designed for highest standards on security. Also, we enforce firewalling on dedicated tenant firewalls before users can establish connections to other networks than their private tenant network. API authentication and authorization is done with the help of OIDC.

API driven

The development of metal-stack is strictly API driven and offers self-service to end-users. This approach delivers the highest possible degree of automation, maintainability and performance.

Ready for Kubernetes

Not only does the metal-stack run smoothly on Kubernetes (K8s). The major intent of metal-stack has always been to build a scalable machine infrastructure for Kubernetes as a Service (KaaS). In partnership with the open-source project Gardener, we can provision Kubernetes clusters on metal-stack at scale.

From the perspective of the Gardener, the metal-stack is just another cloud provider. The time savings compared to providing machines and Kubernetes by hand are significant. We actually want to be able to compete with offers of public cloud providers, especially regarding speed and usability.

Of course, you can use metal-stack only for machine provisioning as well and just put something else on top of your metal infrastructure.

Open Source

The metal-stack is open source and free of constraints regarding vendors and third-party products. The stack is completely built on open source products. We have a community actively working on the metal-stack, which can assist you delivering all reasonable features you are gonna need.

Why Bare Metal?

Bare metal has several advantages over virtual environments and overcomes several drawbacks of virtual machines. We also listed drawbacks of the bare metal approach. Bare in mind though that it is still possible to virtualize on bare metal environments when you have your stack up and running.

Virtual Environment Drawbacks

  • Spectre and Meltdown can only be mitigated with a "cluster per tenant" approach
  • Missing isolation of multi-tenant change impacts
  • Licensing restrictions
  • Noisy-neighbors

Bare Metal Advantages

  • Guaranteed and fastest possible performance (especially disk i/o)
  • Reduced stack depth (Host / VM / Application vs. Host / Container)
    • Reduced attack surface
    • Lower costs, higher performance
    • No VM live-migrations
  • Bigger hardware configurations possible (hypervisors have restrictions, e.g. it is not possible to assign all CPUs to a single VM)

Bare Metal Drawbacks

  • Hardware defects have direct impact (should be considered by design) and can not be mitigated by live-migration as in virtual environments
  • Capacity planning is more difficult (no resource overbooking possible)
diff --git a/dev/installation/deployment/index.html b/dev/installation/deployment/index.html index 827d7a6943..d0ea3d5302 100644 --- a/dev/installation/deployment/index.html +++ b/dev/installation/deployment/index.html @@ -324,4 +324,4 @@ mode: Always autoDetectionMethod: interface=lo typha: - enabled: false
  • For your seed cluster you will need to provide the provider secret for metal-stack containing the key metalAPIHMac, which is the API HMAC to grant editor access to the metal-api
  • Checkout our current provider configuration for infrastructure and control-plane before deploying your shoot
    • Tip

    We are officially supported by Gardener dashboard. The dashboard can also help you setting up some of the resources mentioned above.

    + enabled: false
  • For your seed cluster you will need to provide the provider secret for metal-stack containing the key metalAPIHMac, which is the API HMAC to grant editor access to the metal-api
  • Checkout our current provider configuration for infrastructure and control-plane before deploying your shoot
    • Tip

    We are officially supported by Gardener dashboard. The dashboard can also help you setting up some of the resources mentioned above.

    diff --git a/dev/installation/monitoring/index.html b/dev/installation/monitoring/index.html index ac324f6fca..95eac6790a 100644 --- a/dev/installation/monitoring/index.html +++ b/dev/installation/monitoring/index.html @@ -1,2 +1,2 @@ -Monitoring · metal-stack
    Edit on GitHub

    Monitoring the metal-stack

    We are currently working on providing the sources of our monitoring deployment for public usage. Please come back later.

    +Monitoring · metal-stack
    Edit on GitHub

    Monitoring the metal-stack

    We are currently working on providing the sources of our monitoring deployment for public usage. Please come back later.

    diff --git a/dev/installation/troubleshoot/index.html b/dev/installation/troubleshoot/index.html index 462cf5ef9c..16289a9791 100644 --- a/dev/installation/troubleshoot/index.html +++ b/dev/installation/troubleshoot/index.html @@ -32,4 +32,4 @@ deploy-control-plane | PLAY RECAP ********************************************************************* deploy-control-plane | localhost : ok=29 changed=4 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0 deploy-control-plane | -deploy-control-plane exited with code 2

    Some home routers have a security feature that prevents DNS Servers to resolve anything in the router's local IP range (DNS-Rebind-Protection).

    You need to add an exception for nip.io in your router configuration or add 127.0.0.1 api.172.17.0.1.nip.io to your /etc/hosts.

    FritzBox

    Home Network -> Network -> Network Settings -> Additional Settings -> DNS Rebind Protection -> Host name exceptions -> nip.io

    Operations

    Fixing Machine Issues

    The metalctl machine issues command gives you an overview over machines in your metal-stack environment that are in an unusual state.

    Tip

    Machines that are known not to function properly, should be locked through metalctl machine lock and annotated with a description of the problem. This way, you can mark machine for replacement without being in danger of having a user allocating the faulty machine.

    In the following sections, you can look up the machine issues that are returned by metalctl and find out how to deal with them properly.

    no-event-container

    Every machine in the metal-stack database usually has a corresponding event container where provisioning events are stored. This database entity gets created lazily as soon as a machine is registered by the metal-hammer or a provisioning event for the machine arrives at the metal-api.

    When there is no event container, this means that the machine has never registered nor received a provisioning event. As an operator you should evaluate why this machine is not booting into the metal-hammer.

    This issue is special in a way that it prevents other issues from being evaluated for this machine because the issue calculation usually requires information from the machine event container.

    no-partition

    When a machine has no partition, the metal-hammer has not yet registered the machine at the metal-api. Instead, the machine was created through metal-stack's event machinery, which does not have a lot of information about a machine (e.g. a PXE boot event was reported from the pixiecore), or just by the metal-bmc which discovered the machine through DHCP.

    This can usually happen on the very first boot of a machine and the machine's hardware is not supported by metal-stack, leading to the metal-bmc being unable to report BMC details to the metal-api (a metal-bmc report sets the partition id of a machine) and the metal-hammer not finishing the machine registration phase.

    To resolve this issue, you need to identify the machine in your metal-stack partition that emits PXE boot events and find the reason why it is not properly booting into the metal-hammer. The console logs of this machine should enable you to find out the root cause.

    liveliness-dead

    For machines without an allocation, the metal-hammer consistently reports whether a machine is still being responsive or not. When the liveliness is Dead, there were no events received from this machine for longer than ~5 minutes.

    Reasons for this can be:

    Info

    In order to minimize maintenance overhead, a machine which is dead for longer than an hour will be rebooted through the metal-api.

    In case you want to prevent this action from happening for a machine, you can lock the machine through metalctl machine lock.

    If the machine is dead for a long time and you are sure that it will never come back, you can clean up the machine through metalctl machine rm --remove-from-database.

    liveliness-unknown

    For machines that are allocated by a user, the ownership has gone over to this user and as an operator you cannot access the machine anymore. This makes it harder to detect whether a machine is in a healthy state or not. Typically, all official metal-stack OS images deploy an LLDP daemon, that consistently emits alive messages. These messages are caught by the metal-core and turned into a Phoned Home event. Internally, the metal-api uses these events as an indicator to decide whether the machine is still responsive or not.

    When the LLDP daemon stopped sending packages, the reasons are identical to those of dead machines. However, it's not possible anymore to decide whether the user is responsible for reaching this state or not.

    In most of the cases, there is not much that can be done from the operator's perspective. You will need to wait for the user to report an issue with the machine. When you do support, you can use this issue type to quickly identify this machine.

    liveliness-not-available

    This is more of a theoretical issue. When the machine liveliness is not available check that the Kubernetes CronJob in the metal-stack control plane for evaluating the machine liveliness is running regularly and not containing error logs. Make the machine boot into the metal-hammer and this issue should not appear.

    failed-machine-reclaim

    If a machine remains in the Phoned Home state without having an allocation, this indicates that the metal-bmc was not able to put the machine back into PXE boot mode after metalctl machine rm. The machine is still running the operating system and it does not return back into the allocatable machine pool. Effectively, you lost a machine in your environment and no-one pays for it. Therefore, you should resolve this issue as soon as possible.

    In bad scenarios, when the machine was a firewall, the machine can still reach the internet through the PXE boot network and also attract traffic, which it cannot route anymore inside the tenant VRF. This can cause traffic loss inside a tenant network.

    In most of the cases, it should be sufficient to run another metalctl machine rm on this machine in order to retry booting into PXE mode. If this still does not succeed, you can boot the machine into the BIOS and manually and change the boot order to PXE boot. This should force booting the metal-hammer again and add the machine back into your pool of allocatable machines.

    For further reference, see metal-api#145.

    crashloop

    Under bad circumstances, a machine diverges from its typical machine lifecycle. When this happens, the internal state-machine of the metal-api detects that the machine reboots unexpectedly during the provisioning phase. It is likely that the machine has entered a crash loop where it PXE boots again and again without the machine ever becoming usable.

    Reasons for this can be:

    Please also consider console logs of the machine for investigating the issue.

    The incomplete cycle count is reset as soon as the machine reaches Phoned Home state or there is a Planned Reboot of the machine (planned reboot is also done by the metal-hammer once a day in order to reboot with the latest version).

    last-event-error

    The machine had an error during the provisioning lifecycle recently or events are arriving out of order at the metal-api. This can be an interesting hint for the operator that something during machine provisioning went wrong. You can look at the error through metalctl machine describe or metalctl machine logs.

    This error will disappear after a certain time period from machine issues. You can still look up the error as described above.

    asn-not-unique

    This issue was introduced by a bug in earlier versions of metal-stack and was fixed in PR105

    To resolve the issue, you need to recreate the firewalls that use the same ASN.

    bmc-without-mac

    The metal-bmc is responsible to report connection data for the machine's BMC.

    If it's uncapable of discovering this information, your hardware might not be supported. Please investigate the logs of the metal-bmc to find out what's going wrong with this machine.

    bmc-without-ip

    The metal-bmc is responsible to report connection data for the machine's BMC.

    If it's uncapable of discovering this information, your hardware might not be supported. Please investigate the logs of the metal-bmc to find out what's going wrong with this machine.

    bmc-no-distinct-ip

    The metal-bmc is responsible to report connection data for the machine's BMC.

    When there is no distinct IP address for the BMC, it can be that an orphaned machine used this IP in the past. In this case, you need to clean up the orphaned machine through metalctl machine rm --remove-from-database.

    bmc-info-outdated

    The metal-bmc is responsible to report bmc details for the machine's BMC.

    When the metal-bmc was not able to fetch the bmc info for longer than 20 minutes, something is wrong with the BMC configuration of the machine. This can be caused by one of the following reasons:

    In either case, please check the logs for the given machine UUID on the metal-bmc for further details. Also check that the metal-bmc is configured to only consider BMC IPs in the range they are configured from the DHCP server in the partition. This prevents grabbing unrelated BMCs.

    A machine has registered with a different UUID after reboot

    metal-stack heavily relies on steady machine UUIDs as the UUID is the primary key of the machine entity in the metal-api.

    For further reference also see metal-stack/metal-hammer#52.

    Reasons

    There are some scenarios (can be vendor-specific), which can cause a machine UUID to change over time, e.g.:

    Solution

    1. After five minutes, the orphaned machine UUID will be marked dead (💀) because machine events will be sent only to the most recent UUID
    2. Identify the dead machine through metalctl machine ls
    3. Remove the dead machine forcefully with metalctl machine rm --remove-from-database --yes-i-really-mean-it <uuid>
    +deploy-control-plane exited with code 2

    Some home routers have a security feature that prevents DNS Servers to resolve anything in the router's local IP range (DNS-Rebind-Protection).

    You need to add an exception for nip.io in your router configuration or add 127.0.0.1 api.172.17.0.1.nip.io to your /etc/hosts.

    FritzBox

    Home Network -> Network -> Network Settings -> Additional Settings -> DNS Rebind Protection -> Host name exceptions -> nip.io

    Operations

    Fixing Machine Issues

    The metalctl machine issues command gives you an overview over machines in your metal-stack environment that are in an unusual state.

    Tip

    Machines that are known not to function properly, should be locked through metalctl machine lock and annotated with a description of the problem. This way, you can mark machine for replacement without being in danger of having a user allocating the faulty machine.

    In the following sections, you can look up the machine issues that are returned by metalctl and find out how to deal with them properly.

    no-event-container

    Every machine in the metal-stack database usually has a corresponding event container where provisioning events are stored. This database entity gets created lazily as soon as a machine is registered by the metal-hammer or a provisioning event for the machine arrives at the metal-api.

    When there is no event container, this means that the machine has never registered nor received a provisioning event. As an operator you should evaluate why this machine is not booting into the metal-hammer.

    This issue is special in a way that it prevents other issues from being evaluated for this machine because the issue calculation usually requires information from the machine event container.

    no-partition

    When a machine has no partition, the metal-hammer has not yet registered the machine at the metal-api. Instead, the machine was created through metal-stack's event machinery, which does not have a lot of information about a machine (e.g. a PXE boot event was reported from the pixiecore), or just by the metal-bmc which discovered the machine through DHCP.

    This can usually happen on the very first boot of a machine and the machine's hardware is not supported by metal-stack, leading to the metal-bmc being unable to report BMC details to the metal-api (a metal-bmc report sets the partition id of a machine) and the metal-hammer not finishing the machine registration phase.

    To resolve this issue, you need to identify the machine in your metal-stack partition that emits PXE boot events and find the reason why it is not properly booting into the metal-hammer. The console logs of this machine should enable you to find out the root cause.

    liveliness-dead

    For machines without an allocation, the metal-hammer consistently reports whether a machine is still being responsive or not. When the liveliness is Dead, there were no events received from this machine for longer than ~5 minutes.

    Reasons for this can be:

    Info

    In order to minimize maintenance overhead, a machine which is dead for longer than an hour will be rebooted through the metal-api.

    In case you want to prevent this action from happening for a machine, you can lock the machine through metalctl machine lock.

    If the machine is dead for a long time and you are sure that it will never come back, you can clean up the machine through metalctl machine rm --remove-from-database.

    liveliness-unknown

    For machines that are allocated by a user, the ownership has gone over to this user and as an operator you cannot access the machine anymore. This makes it harder to detect whether a machine is in a healthy state or not. Typically, all official metal-stack OS images deploy an LLDP daemon, that consistently emits alive messages. These messages are caught by the metal-core and turned into a Phoned Home event. Internally, the metal-api uses these events as an indicator to decide whether the machine is still responsive or not.

    When the LLDP daemon stopped sending packages, the reasons are identical to those of dead machines. However, it's not possible anymore to decide whether the user is responsible for reaching this state or not.

    In most of the cases, there is not much that can be done from the operator's perspective. You will need to wait for the user to report an issue with the machine. When you do support, you can use this issue type to quickly identify this machine.

    liveliness-not-available

    This is more of a theoretical issue. When the machine liveliness is not available check that the Kubernetes CronJob in the metal-stack control plane for evaluating the machine liveliness is running regularly and not containing error logs. Make the machine boot into the metal-hammer and this issue should not appear.

    failed-machine-reclaim

    If a machine remains in the Phoned Home state without having an allocation, this indicates that the metal-bmc was not able to put the machine back into PXE boot mode after metalctl machine rm. The machine is still running the operating system and it does not return back into the allocatable machine pool. Effectively, you lost a machine in your environment and no-one pays for it. Therefore, you should resolve this issue as soon as possible.

    In bad scenarios, when the machine was a firewall, the machine can still reach the internet through the PXE boot network and also attract traffic, which it cannot route anymore inside the tenant VRF. This can cause traffic loss inside a tenant network.

    In most of the cases, it should be sufficient to run another metalctl machine rm on this machine in order to retry booting into PXE mode. If this still does not succeed, you can boot the machine into the BIOS and manually and change the boot order to PXE boot. This should force booting the metal-hammer again and add the machine back into your pool of allocatable machines.

    For further reference, see metal-api#145.

    crashloop

    Under bad circumstances, a machine diverges from its typical machine lifecycle. When this happens, the internal state-machine of the metal-api detects that the machine reboots unexpectedly during the provisioning phase. It is likely that the machine has entered a crash loop where it PXE boots again and again without the machine ever becoming usable.

    Reasons for this can be:

    Please also consider console logs of the machine for investigating the issue.

    The incomplete cycle count is reset as soon as the machine reaches Phoned Home state or there is a Planned Reboot of the machine (planned reboot is also done by the metal-hammer once a day in order to reboot with the latest version).

    last-event-error

    The machine had an error during the provisioning lifecycle recently or events are arriving out of order at the metal-api. This can be an interesting hint for the operator that something during machine provisioning went wrong. You can look at the error through metalctl machine describe or metalctl machine logs.

    This error will disappear after a certain time period from machine issues. You can still look up the error as described above.

    asn-not-unique

    This issue was introduced by a bug in earlier versions of metal-stack and was fixed in PR105

    To resolve the issue, you need to recreate the firewalls that use the same ASN.

    bmc-without-mac

    The metal-bmc is responsible to report connection data for the machine's BMC.

    If it's uncapable of discovering this information, your hardware might not be supported. Please investigate the logs of the metal-bmc to find out what's going wrong with this machine.

    bmc-without-ip

    The metal-bmc is responsible to report connection data for the machine's BMC.

    If it's uncapable of discovering this information, your hardware might not be supported. Please investigate the logs of the metal-bmc to find out what's going wrong with this machine.

    bmc-no-distinct-ip

    The metal-bmc is responsible to report connection data for the machine's BMC.

    When there is no distinct IP address for the BMC, it can be that an orphaned machine used this IP in the past. In this case, you need to clean up the orphaned machine through metalctl machine rm --remove-from-database.

    bmc-info-outdated

    The metal-bmc is responsible to report bmc details for the machine's BMC.

    When the metal-bmc was not able to fetch the bmc info for longer than 20 minutes, something is wrong with the BMC configuration of the machine. This can be caused by one of the following reasons:

    In either case, please check the logs for the given machine UUID on the metal-bmc for further details. Also check that the metal-bmc is configured to only consider BMC IPs in the range they are configured from the DHCP server in the partition. This prevents grabbing unrelated BMCs.

    A machine has registered with a different UUID after reboot

    metal-stack heavily relies on steady machine UUIDs as the UUID is the primary key of the machine entity in the metal-api.

    For further reference also see metal-stack/metal-hammer#52.

    Reasons

    There are some scenarios (can be vendor-specific), which can cause a machine UUID to change over time, e.g.:

    Solution

    1. After five minutes, the orphaned machine UUID will be marked dead (💀) because machine events will be sent only to the most recent UUID
    2. Identify the dead machine through metalctl machine ls
    3. Remove the dead machine forcefully with metalctl machine rm --remove-from-database --yes-i-really-mean-it <uuid>
    diff --git a/dev/overview/architecture/index.html b/dev/overview/architecture/index.html index abadd61457..e18d61aa6e 100644 --- a/dev/overview/architecture/index.html +++ b/dev/overview/architecture/index.html @@ -1,4 +1,4 @@ Architecture · metal-stack
    Edit on GitHub

    Architecture

    The metal-stack is a compound of microservices predominantly written in Golang.

    This page gives you an overview over which microservices exist, how they communicate with each other and where they are deployed.

    Target Deployment Platforms

    For our environments, we chose to deploy the metal-stack into a Kubernetes cluster. This means that also our entire installation was developed for metal-stack being run on Kubernetes. Running applications on Kubernetes gives you a lot of benefits regarding ease-of-deployment, scalability, reliability and so on.

    However, very early we decided that we do not want to depend on technical Kubernetes functionality with our software (i.e. we did not implement the stack "kube-native" by using controllers and Kubernetes CRDs and things like that). With the following paragraph we want to point out the reasoning behind this "philosophical" decision that may sound conservative at first glance. But not relying on Kubernetes technology:

    • Makes deployments of the stack without Kubernetes theoretically possible.
      • We believe that cloud providers should be able to act beneath Kubernetes
      • This way it is possible to use metal-stack for providing your own Kubernetes offering without relying on Kubernetes yourself (breaks the chicken-egg problem)
    • Follows an important claim in microservice development: "Be agnostic to your choice of technology"
      • For applications that are purely made for being run on Kubernetes, it does not matter to rely on this technology (we even do the same a lot with our applications that integrate the metal-stack with Gardener) but as soon as you start using things like the underlying reconciliation abilities (which admittedly are fanstatic) you are locking your code into a certain technology
      • We don't know what comes after Kubernetes but we believe that a cloud offering should have the potential to survive a choice of technology
      • By this decision we ensured that we can migrate the stack to another future technology and survive the change

    One more word towards determining the location for your metal control plane: It is not strictly required to run the control plane inside the same data center as your servers. It even makes sense not to do so because this way you can place your control plane and your servers into a different failure domains, which makes your installation more robust to data center meltdown. Externally hosting the control plane brings you up and running quickly plus having the advantage of higher security through geo-distribution.

    Metal Control Plane

    The foundation of the metal-stack is what we call the metal control plane.

    The control plane contains of a couple of essential microservices for the metal-stack including:

    • metal-api The API to manage and control plane resources like machines, switches, operating system images, machine sizes, networks, IP addresses and more. The exposed API is an old-fashioned REST API with different authentication methods. The metal-api stores the state of these entities in a RethinkDB database. The metal-api also has its own IP address management (go-ipam), which writes IP address and network allocations into a PostgreSQL backend.
    • masterdata-api Manages tenant and project entities, which can be described as entities used for company-specific resource separation and grouping. Having these "higher level entities" managed by a separate microservice was a design choice that allows to re-use the information by other microservices without having them to know the metal-api at all. The masterdata gets persisted in a dedicated PostgreSQL database.
    • metal-console Provides access for users to a machine's serial console via SSH. It can be seen as an optional component.
    • nsq A message queuing system (not developed by the metal-stack) used for decoupling microservices and distributing tasks.

    The following figure shows the relationships between these microservices:

    Metal Control Plane

    Figure 1: The metal control plane deployed in a Kubernetes environment with an ingress-controller exposing additional services via service exposal.

    Some notes on this picture:

    • Users can access the metal-api with the CLI client called metalctl.
    • You can programmatically access the metal-api with client libraries (e.g. metal-go).
    • Our databases are wrapped in a specially built backup-restore-sidecar, which is consistently backing up the databases in external blob storage.
    • The metal-api can be scaled out using replicas when being deployed in Kubernetes.

    Partitions

    A partition is our term for describing hardware in the data center controlled by the metal-stack with all the hardware participating in the same network topology. Being in the same network topology causes the hardware inside a partition to build a failure domain. Even though the network topology for running the metal-stack is required to be redundant by design, you should consider setting up multiple partitions. With multiple partitions it is possible for users to maintain availability of their applications by spreading them across the partitions. Installing partitions in multiple data centers would be even better in regards of fail-safe application performance, which would even tolerate the meltdown of a data center.

    Tip

    In our setups, we encode the name of a region and a zone name into our partition names. However, we do not have dedicated entities for regions and zones in our APIs.

    A region is a geographic area in which data centers are located.

    Zones are geographic locations in a region usually in different fire compartments. Regions can consist of several zones.

    A zone can consist of several partitions. Usually, a partition spans a rack or a group of racks.

    We strongly advise to group your hardware into racks that are specifically assembled for running metal-stack. When using modular rack design, the amount of compute resources of a partition can easily be extended by adding more racks to your partition.

    Info

    The hardware that we currently support to be placed inside a partition is described in the hardware document.

    Info

    How large you can grow your partitions and how the network topology inside a partition looks like is described in the networking document.

    The metal-stack has microservices running on the leaf switches in a partition. For this reason, your leaf switches are required to run a Linux distribution that you have full access to. Additionally, there are a servers not added to the pool of user-allocatable machines, which are instead required for running metal-stack and we call them management servers. We also call the entirety of switches inside a partition the switch plane.

    The microservices running inside a partition are:

    • metal-hammer (runs on a server when not allocated by user, often referred to as discovery image) An initrd, which is booted up in PXE mode, preparing and registering a machine. When a user allocates a machine, the metal-hammer will install the target operating system on this machine and kexec into the new operating system kernel.
    • metal-core (runs on leaf switches) Dynamically configures the leaf switch from information provided by the metal-api. It also proxies requests from the metal-hammer to the metal-api including publishment of machine lifecycle events and machine registration requests.
    • pixiecore (preferably runs on management servers, forked by metal-stack) Provides the capability of PXE booting servers in the PXE boot network.
    • metal-bmc (runs on management servers) Reports the ip addresses that are leased to ipmi devices together with their machine uuids to the metal-api. This provides machine discovery in the partition machines and keeps all IPMI interface access data up-to-date. Also forwards metal-console requests to the actual machine, allowing user access to the machine's serial console. Furthermore it processes firmware updates and power on/off, led on/off, boot order changes.

    Partition

    Figure 2: Simplified illustration of services running inside a partition.

    Some notes on this picture:

    • This figure is slightly simplified. The switch plane consists of spine switches, exit routers, management firewalls and a bastion router with more software components deployed on these entities. Please refer to the networking document to see the full overview over the switch plane.
    • The image-cache is an optional component consisting of multiple services to allow caching images from the public image store inside a partition. This brings increased download performance on machine allocation and increases independence of a partition on the internet connection.

    Complete View

    The following figure shows several partitions connected to a single metal control plane. Of course, it is also possible to have multiple metal control planes, which can be useful for staging.

    metal-stack

    Figure 3: Reduced view on the communication between the metal control plane and multiple partitions.

    Some notes on this picture:

    • By design, a partition only has very few ports open for incoming-connections from the internet. This contributes to a smaller attack surface and higher security of your infrastructure.
    • With the help of NSQ, it is not required to have connections from the metal control plane to the metal-core. The metal-core instances register at the message bus and can then consume partition-specfic topics, e.g. when a machine deletion gets issued by a user.

    Machine Provisioning Sequence

    The following sequence diagram illustrates some of the main principles of the machine provisioning lifecycle.

    provisioning sequence

    Figure 4: Sequence diagram of the machine provisioning sequence.

    Here is a video showing a screen capture of a machine's serial console while running the metal-hammer in "wait mode". Then, a user allocates the machine and the metal-hammer installs the target operating system and the machine boots into the new operating system kernel via the kexec system call.

    -
    + diff --git a/dev/overview/comparison/index.html b/dev/overview/comparison/index.html index ea2bcaf44e..3de5a89ca9 100644 --- a/dev/overview/comparison/index.html +++ b/dev/overview/comparison/index.html @@ -1,2 +1,2 @@ -Comparison · metal-stack
    Edit on GitHub

    Comparison with Commercial Solutions

    As metal-stack is the foundation to build Kubernetes clusters on premise on bare metal, there are several commercial solutions available which offer management of Kubernetes. In this document we describe the differences between some of the most popular solutions. It´s is not a complete list.

    Comparison between Gardener on Metal Stack and Openshift running on VMWare.

    Gardener

    Gardener is a Kubernetes cluster manager to organize a fleet of Kubernetes clusters at scale. It is designed to scale to thousands of clusters at a variety of IaaS Providers regardless where - in the cloud or on premise, virtualized or bare metal. It not only manages the creation and deletion of Kubernetes clusters, it also takes care of updating or upgrading Kubernetes and the operating system of the involved worker nodes in a automatic manner. Gardener is designed cloud-native and as such, it defines clusters, workers and all other components as Kubernetes resources (like pods and deployments) and reconciles these resources to the desired state.

    Kubernetes

    Kubernetes is the de facto open-source standard for container scheduling and orchestration in the data center.

    Openshift

    A fork of Kubernetes with proprietary addons, created by RedHat. For all details see: https://en.wikipedia.org/wiki/OpenShift.

    metal-stack

    Is an IaaS provider for bare metal focused to create Kubernetes cluster on premise. Gardener support is built in.

    VMWare

    The most used virtualization technology in the enterprise data centers.

    Comparison of Gardener on Metal Stack vs. Openshift on VMWare

    FeatureGardener on Metal StackOpenshift on VMWare
    Container Runtimedocker, containerd, gvisorcri-o
    Host Operating SystemUbuntu, Debian , also see OSRHEL, Fedora-Core
    Network PluginsCalico, Cilium(soon)Openshift SDN
    StorageLocal NVME, Lightbits NVMEoTCP, all CSI compatible Solutions, also see StorageCSI compatible
    LoadbalancingBGP built inrequires extra HW like F5, VMWare NSX
    IO at Native SpeedPods run on bare metalall IO must go through the Hypervisor
    Hard MultitenancyWorkers, firewall and load balancers are dedicated for every cluster on bare metalShared virtualization hosts, shared load balancers
    UIGardener DashboardOpenshift Console
    Multi-cluster managementYes (through Gardener)Requires extra licences SW: Redhat Advanced Cluster Manager
    Automatic Kubernetes UpdatesYesYes
    Automatic Worker Nodes UpdatesYesYes
    Supported IaaS ProvidersGCP, AWS, Azure, Alibaba, Openstack, VMWare, metal-stack and moreGCP, AWS, Azure Openstack, VMWare
    Monitoring / Logging StackGrafana/Loki, Kibana/ElasticKibana/Elastic
    GitOPSTool of choice via Helm InstallOpenshift GitOPS
    Container Registryall public accessible registries, private deployed registry of choiceall public accessible registries, in cluster registry
    CI/CDTool of choice via Helm InstallJenkins
    SecurityK8s control plane isolated from tenant, PSP enabled by defaultStrong cluster defaults
    CNCF Kubernetes certifiedYes (Gardener)Yes
    Local developmentminikube, kindminishift
    Proprietary extensionsNoDeploymentConfig and others
    kubectl accessYesYes
    +Comparison · metal-stack
    Edit on GitHub

    Comparison with Commercial Solutions

    As metal-stack is the foundation to build Kubernetes clusters on premise on bare metal, there are several commercial solutions available which offer management of Kubernetes. In this document we describe the differences between some of the most popular solutions. It´s is not a complete list.

    Comparison between Gardener on Metal Stack and Openshift running on VMWare.

    Gardener

    Gardener is a Kubernetes cluster manager to organize a fleet of Kubernetes clusters at scale. It is designed to scale to thousands of clusters at a variety of IaaS Providers regardless where - in the cloud or on premise, virtualized or bare metal. It not only manages the creation and deletion of Kubernetes clusters, it also takes care of updating or upgrading Kubernetes and the operating system of the involved worker nodes in a automatic manner. Gardener is designed cloud-native and as such, it defines clusters, workers and all other components as Kubernetes resources (like pods and deployments) and reconciles these resources to the desired state.

    Kubernetes

    Kubernetes is the de facto open-source standard for container scheduling and orchestration in the data center.

    Openshift

    A fork of Kubernetes with proprietary addons, created by RedHat. For all details see: https://en.wikipedia.org/wiki/OpenShift.

    metal-stack

    Is an IaaS provider for bare metal focused to create Kubernetes cluster on premise. Gardener support is built in.

    VMWare

    The most used virtualization technology in the enterprise data centers.

    Comparison of Gardener on Metal Stack vs. Openshift on VMWare

    FeatureGardener on Metal StackOpenshift on VMWare
    Container Runtimedocker, containerd, gvisorcri-o
    Host Operating SystemUbuntu, Debian , also see OSRHEL, Fedora-Core
    Network PluginsCalico, Cilium(soon)Openshift SDN
    StorageLocal NVME, Lightbits NVMEoTCP, all CSI compatible Solutions, also see StorageCSI compatible
    LoadbalancingBGP built inrequires extra HW like F5, VMWare NSX
    IO at Native SpeedPods run on bare metalall IO must go through the Hypervisor
    Hard MultitenancyWorkers, firewall and load balancers are dedicated for every cluster on bare metalShared virtualization hosts, shared load balancers
    UIGardener DashboardOpenshift Console
    Multi-cluster managementYes (through Gardener)Requires extra licences SW: Redhat Advanced Cluster Manager
    Automatic Kubernetes UpdatesYesYes
    Automatic Worker Nodes UpdatesYesYes
    Supported IaaS ProvidersGCP, AWS, Azure, Alibaba, Openstack, VMWare, metal-stack and moreGCP, AWS, Azure Openstack, VMWare
    Monitoring / Logging StackGrafana/Loki, Kibana/ElasticKibana/Elastic
    GitOPSTool of choice via Helm InstallOpenshift GitOPS
    Container Registryall public accessible registries, private deployed registry of choiceall public accessible registries, in cluster registry
    CI/CDTool of choice via Helm InstallJenkins
    SecurityK8s control plane isolated from tenant, PSP enabled by defaultStrong cluster defaults
    CNCF Kubernetes certifiedYes (Gardener)Yes
    Local developmentminikube, kindminishift
    Proprietary extensionsNoDeploymentConfig and others
    kubectl accessYesYes
    diff --git a/dev/overview/hardware/index.html b/dev/overview/hardware/index.html index 7808f66e3e..85db6dd364 100644 --- a/dev/overview/hardware/index.html +++ b/dev/overview/hardware/index.html @@ -1,2 +1,2 @@ -Hardware Support · metal-stack
    Edit on GitHub

    Hardware Support

    In order to keep the automation and maintenance overhead small, we strongly advise against building highly heterogeneous environments with metal-stack. Having a lot of different vendors and server models in your partitions will heavily increase the time and effort for introducing metal-stack in your infrastructure. From experience we can tell that the interfaces for automating hardware provisioning are usually inconsistent between vendors and even between server models of the same vendor. Therefore, we encourage adopters to start off with only a small amount of machine types. If you want to be on the safe side, you should consider buying the hardware that we officially support.

    We came up with a repository called go-hal, which includes the interface required for metal-stack to support a machine vendor. If you plan to implement support for new vendors, please check out this repository and contribute back your efforts in order to make the community benefit from extended vendor support as well.

    Servers

    At the moment we support the following server types:

    VendorSeriesModelBoard TypeStatus
    SupermicroBig-TwinSYS-2029BT-HNRX11DPT-Bstable
    SupermicroSuperServerSSG-5019D8-TR12PX11SDV-8C-TP8Fstable
    SupermicroSuperServer2029UZ-TN20R25MX11DPUstable
    SupermicroMicrocloud5039MD8-H8TNRX11SDD-8C-Fstable
    LenovoThinkSystemSD530alpha

    Switches

    At the moment we support the following switch types:

    VendorSeriesModelOSStatus
    Edge-CoreAS7700 SeriesAS7712-32XCumulus 3.7.13stable
    Edge-CoreAS7700 SeriesAS7726-32XCumulus 4.1.1stable
    Edge-CoreAS7700 SeriesAS7712-32XEdgecore SONiCstable
    Edge-CoreAS7700 SeriesAS7726-32XEdgecore SONiCstable
    Warning

    On our switches we run SONiC. The metal-core writes network configuration specifically implemented for this operating system. Please also consider running SONiC on your switches if you do not want to run into any issues with networking.

    Our previous support for Cumulus Linux will come to an end.

    Of course, contributions for supporting other switch vendors and operating systems are highly appreciated.

    Portable metal-stack Setup DIY

    A minimal physical hardware setup may contain at least the following components:

    Warning

    This setup should work as the components are very similar to the currently supported ones but it's currently untested.

    #VendorSeriesModelFunction
    2xEdge-CoreAS5500 SeriesAS5512-54x (10G)Leaf / Exit switches
    1xSupermicroMicrocloudSYS-5039MA16-H12RFTUsable machines
    1xUnifiEdgemaxEdgerouter ProFront router for internet and out-of-band access to servers and switches

    Besides that, a 6HE rack with 1000mm depth and a portable LTE modem is needed.

    This MVP will yield in 12 usable machines, one of them will be reserved as management server.

    +Hardware Support · metal-stack
    Edit on GitHub

    Hardware Support

    In order to keep the automation and maintenance overhead small, we strongly advise against building highly heterogeneous environments with metal-stack. Having a lot of different vendors and server models in your partitions will heavily increase the time and effort for introducing metal-stack in your infrastructure. From experience we can tell that the interfaces for automating hardware provisioning are usually inconsistent between vendors and even between server models of the same vendor. Therefore, we encourage adopters to start off with only a small amount of machine types. If you want to be on the safe side, you should consider buying the hardware that we officially support.

    We came up with a repository called go-hal, which includes the interface required for metal-stack to support a machine vendor. If you plan to implement support for new vendors, please check out this repository and contribute back your efforts in order to make the community benefit from extended vendor support as well.

    Servers

    At the moment we support the following server types:

    VendorSeriesModelBoard TypeStatus
    SupermicroBig-TwinSYS-2029BT-HNRX11DPT-Bstable
    SupermicroSuperServerSSG-5019D8-TR12PX11SDV-8C-TP8Fstable
    SupermicroSuperServer2029UZ-TN20R25MX11DPUstable
    SupermicroMicrocloud5039MD8-H8TNRX11SDD-8C-Fstable
    LenovoThinkSystemSD530alpha

    Switches

    At the moment we support the following switch types:

    VendorSeriesModelOSStatus
    Edge-CoreAS7700 SeriesAS7712-32XCumulus 3.7.13stable
    Edge-CoreAS7700 SeriesAS7726-32XCumulus 4.1.1stable
    Edge-CoreAS7700 SeriesAS7712-32XEdgecore SONiCstable
    Edge-CoreAS7700 SeriesAS7726-32XEdgecore SONiCstable
    Warning

    On our switches we run SONiC. The metal-core writes network configuration specifically implemented for this operating system. Please also consider running SONiC on your switches if you do not want to run into any issues with networking.

    Our previous support for Cumulus Linux will come to an end.

    Of course, contributions for supporting other switch vendors and operating systems are highly appreciated.

    Portable metal-stack Setup DIY

    A minimal physical hardware setup may contain at least the following components:

    Warning

    This setup should work as the components are very similar to the currently supported ones but it's currently untested.

    #VendorSeriesModelFunction
    2xEdge-CoreAS5500 SeriesAS5512-54x (10G)Leaf / Exit switches
    1xSupermicroMicrocloudSYS-5039MA16-H12RFTUsable machines
    1xUnifiEdgemaxEdgerouter ProFront router for internet and out-of-band access to servers and switches

    Besides that, a 6HE rack with 1000mm depth and a portable LTE modem is needed.

    This MVP will yield in 12 usable machines, one of them will be reserved as management server.

    diff --git a/dev/overview/kubernetes/index.html b/dev/overview/kubernetes/index.html index 6ed8a3512c..3c30864aab 100644 --- a/dev/overview/kubernetes/index.html +++ b/dev/overview/kubernetes/index.html @@ -1,2 +1,2 @@ -Kubernetes Integration · metal-stack
    Edit on GitHub

    Kubernetes Integration

    With the help of the Gardener project, metal-stack can be used for spinning up Kubernetes clusters quickly and reliably on bare metal machines.

    To make this happen, we implemented a couple of components, which are described here.

    metal-ccm

    CCM stands for cloud-controller-manager and is the bridge between Kubernetes and a cloud-provider.

    We implemented the cloud provider interface in the metal-ccm repository. With the help of the cloud-controller-controller we provide metal-stack-specific properties for Kubernetes clusters, e.g. load balancer configuration through MetalLB or node properties.

    firewall-controller

    To make the firewalls created with metal-stack easily configurable through Kubernetes resources, we add our firewall-controller to the firewall image. The controller watches special CRDs, enabling users to manage:

    • nftables rules
    • Intrusion-detection with suricata
    • network metric collection

    Please check out the guide on how to use it.

    Gardener components

    There are some Gardener resources that need be reconciled when you act as a cloud provider for the Gardener. This section briefly describes the controllers implemented for deploying Kubernetes clusters through Gardener.

    If you want to learn how to deploy metal-stack with Gardener, please check out the installation section.

    gardener-extension-provider-metal

    The gardener-extension-provider-metal contains of a set of webhooks and controllers for reconciling or mutating Gardener-specific resources.

    The project also contains a validator for metal-type Gardener resources, which you should also deploy in case you want to use metal-stack in combination with Gardener.

    os-metal-extension

    Due to the reason we use ignition in our operating system images for userdata, we had to provide an own extension controller for metal-stack, which you can find at Github in the os-metal-extension repository.

    machine-controller-manager-provider-metal

    Worker nodes are managed through Gardener's machine-controller-manager (MCM). The MCM allows out-of-tree provider implementation via sidecar, which is what we implemented in the machine-controller-manager-provider-metal repository.

    +Kubernetes Integration · metal-stack
    Edit on GitHub

    Kubernetes Integration

    With the help of the Gardener project, metal-stack can be used for spinning up Kubernetes clusters quickly and reliably on bare metal machines.

    To make this happen, we implemented a couple of components, which are described here.

    metal-ccm

    CCM stands for cloud-controller-manager and is the bridge between Kubernetes and a cloud-provider.

    We implemented the cloud provider interface in the metal-ccm repository. With the help of the cloud-controller-controller we provide metal-stack-specific properties for Kubernetes clusters, e.g. load balancer configuration through MetalLB or node properties.

    firewall-controller

    To make the firewalls created with metal-stack easily configurable through Kubernetes resources, we add our firewall-controller to the firewall image. The controller watches special CRDs, enabling users to manage:

    • nftables rules
    • Intrusion-detection with suricata
    • network metric collection

    Please check out the guide on how to use it.

    Gardener components

    There are some Gardener resources that need be reconciled when you act as a cloud provider for the Gardener. This section briefly describes the controllers implemented for deploying Kubernetes clusters through Gardener.

    If you want to learn how to deploy metal-stack with Gardener, please check out the installation section.

    gardener-extension-provider-metal

    The gardener-extension-provider-metal contains of a set of webhooks and controllers for reconciling or mutating Gardener-specific resources.

    The project also contains a validator for metal-type Gardener resources, which you should also deploy in case you want to use metal-stack in combination with Gardener.

    os-metal-extension

    Due to the reason we use ignition in our operating system images for userdata, we had to provide an own extension controller for metal-stack, which you can find at Github in the os-metal-extension repository.

    machine-controller-manager-provider-metal

    Worker nodes are managed through Gardener's machine-controller-manager (MCM). The MCM allows out-of-tree provider implementation via sidecar, which is what we implemented in the machine-controller-manager-provider-metal repository.

    diff --git a/dev/overview/networking/index.html b/dev/overview/networking/index.html index cba9c0e25c..b160eda12f 100644 --- a/dev/overview/networking/index.html +++ b/dev/overview/networking/index.html @@ -351,4 +351,4 @@ iface swp1 mtu 9000 bridge-access 4000 -# [...]

    Listing 14: VLAN access setup for bare metal server facing ports on leaves.

    Once a bare metal server is provisioned it is deconfigured from PXE VLAN vlan4000 to avoid accidential or unwanted provisioning.

    During provisioning bare metal servers get internet access via the management network of the exit switches. This is because the exit switches are announced as DHCP gateway to the DHCP clients.

    Management Network

    To manage network switches beside the out-of-band system console access a further management access is required. For this purpose the concept of Management VRF is applied. The Management VRF is a subset of VRF. It provides a separation between out-of-band management network and the in-band data plane network by introducing another routing table mgmt. SONiC supports eth0 to be used as the management interface.

    To enable and use the Management VRF all switches have to be connected via their eth0 interface to a management-switch. The management switch is connected to a management server. All access is established from within the managment server. Logins to the switch are set into the Management VRF context once the Managment VRF is enabled.

    +# [...]

    Listing 14: VLAN access setup for bare metal server facing ports on leaves.

    Once a bare metal server is provisioned it is deconfigured from PXE VLAN vlan4000 to avoid accidential or unwanted provisioning.

    During provisioning bare metal servers get internet access via the management network of the exit switches. This is because the exit switches are announced as DHCP gateway to the DHCP clients.

    Management Network

    To manage network switches beside the out-of-band system console access a further management access is required. For this purpose the concept of Management VRF is applied. The Management VRF is a subset of VRF. It provides a separation between out-of-band management network and the in-band data plane network by introducing another routing table mgmt. SONiC supports eth0 to be used as the management interface.

    To enable and use the Management VRF all switches have to be connected via their eth0 interface to a management-switch. The management switch is connected to a management server. All access is established from within the managment server. Logins to the switch are set into the Management VRF context once the Managment VRF is enabled.

    diff --git a/dev/overview/os/index.html b/dev/overview/os/index.html index 22502f33b4..7208572616 100644 --- a/dev/overview/os/index.html +++ b/dev/overview/os/index.html @@ -1,2 +1,2 @@ -Operating Systems · metal-stack
    Edit on GitHub

    Operating Systems

    Our operating system images are built on regular basis from the metal-images repository.

    All images are hosted on GKE at images.metal-stack.io. Feel free to use this as a mirror for your metal-stack partitions if you want. The metal-stack developers continuously have an eye on the supported images. They are updated regularly and scanned for vulnerabilities.

    Supported OS Images

    The operating system images that we build are trimmed down to their bare essentials for serving as Kubernetes worker nodes. Small image sizes make machine provisioning blazingly fast.

    The supported images for worker nodes currently are:

    PlatformDistributionVersion
    LinuxDebian11
    LinuxUbuntu22.04

    The supported images for firewalls are:

    PlatformDistributionVersionBased On
    LinuxUbuntu322.04

    Building Your Own Images

    It is fully possible to build your own operating system images and provide them through the metal-stack.

    There are some conventions though that you need to follow in order to make your image installable through the metal-hammer. You should understand the machine provisioning sequence before starting to write your own images.

    1. Images need to be compressed to a tarball using the lz4 compression algorithm
    2. An md5 checksum file with the same name as the image archive needs to be provided in the download path along with the actual os image
    3. A packages.txt containing the packages contained in the OS image should be provided in the download path (not strictly required)
    4. Consider semantic image versioning, which we use in our algorithms to select latest images (e.g. os-major.minor.patch ➡️ ubuntu-19.10.20191018)
    5. Consider installing packages used by the metal-stack infrastructure
      • FRR to enable routing-to-the-host in our network topology
      • go-lldpd to enable checking if the machine is still alive after user allocation
      • ignition for enabling users to run user-specific initialization instructions before bootup. It's pretty small in size, which is why we use it. However, you are free to use other cloud instance initialization tools if you want to.
    6. You have to provide an install.sh script, which applies user-specific configuration in the installed image
      • This script should consume parameters from the install.yaml file that the metal-hammer writes to /etc/metal/install.yaml
      • Please check this contract between image and the metal-hammer here
    7. For the time being, your image must be able to support kexec into the new operating system kernel, the kexec command is issued by the metal-hammer after running the install.sh. We do this because kexec is much faster than rebooting a machine.
    8. We recommend building images from Dockerfiles as it is done in metal-images repository.
    Info

    Building own operating system images is an advanced topic. When you have just started with metal-stack, we recommend using the public operating system images first.

    +Operating Systems · metal-stack
    Edit on GitHub

    Operating Systems

    Our operating system images are built on regular basis from the metal-images repository.

    All images are hosted on GKE at images.metal-stack.io. Feel free to use this as a mirror for your metal-stack partitions if you want. The metal-stack developers continuously have an eye on the supported images. They are updated regularly and scanned for vulnerabilities.

    Supported OS Images

    The operating system images that we build are trimmed down to their bare essentials for serving as Kubernetes worker nodes. Small image sizes make machine provisioning blazingly fast.

    The supported images for worker nodes currently are:

    PlatformDistributionVersion
    LinuxDebian11
    LinuxUbuntu22.04

    The supported images for firewalls are:

    PlatformDistributionVersionBased On
    LinuxUbuntu322.04

    Building Your Own Images

    It is fully possible to build your own operating system images and provide them through the metal-stack.

    There are some conventions though that you need to follow in order to make your image installable through the metal-hammer. You should understand the machine provisioning sequence before starting to write your own images.

    1. Images need to be compressed to a tarball using the lz4 compression algorithm
    2. An md5 checksum file with the same name as the image archive needs to be provided in the download path along with the actual os image
    3. A packages.txt containing the packages contained in the OS image should be provided in the download path (not strictly required)
    4. Consider semantic image versioning, which we use in our algorithms to select latest images (e.g. os-major.minor.patch ➡️ ubuntu-19.10.20191018)
    5. Consider installing packages used by the metal-stack infrastructure
      • FRR to enable routing-to-the-host in our network topology
      • go-lldpd to enable checking if the machine is still alive after user allocation
      • ignition for enabling users to run user-specific initialization instructions before bootup. It's pretty small in size, which is why we use it. However, you are free to use other cloud instance initialization tools if you want to.
    6. You have to provide an install.sh script, which applies user-specific configuration in the installed image
      • This script should consume parameters from the install.yaml file that the metal-hammer writes to /etc/metal/install.yaml
      • Please check this contract between image and the metal-hammer here
    7. For the time being, your image must be able to support kexec into the new operating system kernel, the kexec command is issued by the metal-hammer after running the install.sh. We do this because kexec is much faster than rebooting a machine.
    8. We recommend building images from Dockerfiles as it is done in metal-images repository.
    Info

    Building own operating system images is an advanced topic. When you have just started with metal-stack, we recommend using the public operating system images first.

    diff --git a/dev/overview/storage/index.html b/dev/overview/storage/index.html index f15eb93244..b0c9b06c34 100644 --- a/dev/overview/storage/index.html +++ b/dev/overview/storage/index.html @@ -9,4 +9,4 @@ resources: requests: storage: 100Mi - storageClassName: csi-lvm-sc-linear

    The solution does not provide cloud-storage or whatsoever, but it improves the user's accessibility of local storage on bare-metal machines through Kubernetes. Check out the driver's documentation here.

    + storageClassName: csi-lvm-sc-linear

    The solution does not provide cloud-storage or whatsoever, but it improves the user's accessibility of local storage on bare-metal machines through Kubernetes. Check out the driver's documentation here.

    diff --git a/dev/quickstart/index.html b/dev/quickstart/index.html index 766fb7f189..795bcfc8ed 100644 --- a/dev/quickstart/index.html +++ b/dev/quickstart/index.html @@ -1,2 +1,2 @@ -Quickstart · metal-stack
    Edit on GitHub

    Getting Started

    Before starting to buy any hardware, you should try out the metal-stack on your notebook and familiarize with the software.

    For this, we made the mini-lab.

    The mini-lab is a fully virtual setup of metal-stack and is supposed to be run locally on a single machine. For this reason, the setup was slightly simplified in comparison to full-blown setups on real hardware. However, the lab should help to understand all ideas behind the metal-stack.

    Get your hands dirty and follow the guide on how to get on with the mini-lab here.

    +Quickstart · metal-stack
    Edit on GitHub

    Getting Started

    Before starting to buy any hardware, you should try out the metal-stack on your notebook and familiarize with the software.

    For this, we made the mini-lab.

    The mini-lab is a fully virtual setup of metal-stack and is supposed to be run locally on a single machine. For this reason, the setup was slightly simplified in comparison to full-blown setups on real hardware. However, the lab should help to understand all ideas behind the metal-stack.

    Get your hands dirty and follow the guide on how to get on with the mini-lab here.

    diff --git a/dev/search/index.html b/dev/search/index.html index 7857fdd275..0d56c28291 100644 --- a/dev/search/index.html +++ b/dev/search/index.html @@ -1,2 +1,2 @@ -Search · metal-stack

    Loading search...

      +Search · metal-stack

      Loading search...