Decrypting the user data on the cluster in an easier way

[firstdorsal]

Hi, first of all i wanted to say that I love karios and wanted to thank you for the development!

I want to create a small 3 node cluster that has all its data encrypted at rest. I have seen your guide on doing this and the use of the kcrypt-challenger that should by design run on another cluster as I understand it?

This would be a bit much for my use case as I would like to create a setup like follows:

• start the machines over wake on lan
• get prompted to enter the encryption key
• the machines decrypt the luks partitions with etcd and its secrets keys for all other drives (stored by longhorn or similar)
• start the kubernetes system

Could something like this be achieved with kairos?

[jimmykarily]

Hi @firstdorsal , thanks for trying out Kairos.
By "wake on lan" (WoL) you don't mean "net boot" right? Afaik WoL only has to do with how the machines get turned on so it as far as Kairos is concerned, it doesn't matter whether someone pushed a button or send a WoL message over the network.

Having said that, where should the prompt appear? One prompt per machine? This means someone would have to enter the passphrase for decryption on each machine whenever they are rebooted.

The encryption feature on Kairos was designed to work automatically without user intervention, so this scenario is not supported.

I'm thinking that you may be able to achieve what you want by encrypting the partitions manually after installation (or using some stage in the kairos config). Then you would get the regular prompt when the machine boots. It's just a rough idea, I don't have the exact steps. Needs some experimentation.

@kairos-io/maintainers any other ideas?

[firstdorsal]

No with WoL I do mean just turning the machines on after the first installation when they were powered down for some reason. Yes this has nothing to do with the boot process or Kairos.

Yes this should involve user intervention for my use case. One prompt outside of the cluster on my dev machine that then sends the encryption key/keys to all the machines. Of course I would develop this part myself, but how would something like this integrate in the kairos system boot process?

On my current single server docker setup it works similar to what I am trying to achieve here

• server starts, either manually with power button or WoL and starts ssh but not docker
• I SSH into the machine and decrypt the drives, with a script on the server that prompts me for the key
• after all drives have been mounted it starts docker and its containers

[jimmykarily]

Thanks for clarifying this @firstdorsal . Since you are going to implement some part of it, one option would be to write your own version of the challenger: https://github.com/kairos-io/kcrypt-challenger/blob/main/pkg/challenger/challenger.go#L215

Basically you need to implement the 2 endpoints /postPass and getPass which are requested by the client. You may be able to get away with just implementing /getPass endpoint which is the first one the client tries to reach: https://github.com/kairos-io/kcrypt-challenger/blob/c42e66a9de78193479e4c15ab5178cb0a60d357f/cmd/discovery/client/client.go#L97

By doing this, you have full control over what the KMS does, how it stores the passphrase, whether it prompts you or not, etc.

I would only suggest that you go down this route, if being prompted for the passphrase is really important to you. Otherwise you can simply deploy a cluster with k3d or kind, deploy the KMS there and call it a day.

[firstdorsal]

Awesome! Thank you! I will have a look at it.

[firstdorsal]

While trying to send a passphrase I encountered the following:

when sending this:

Passphrase: test

{"Passphrase":"dGVzdA==","GeneratedBy":"kcrypt-challenger"}

Environment:
qemu, alpine 3.18, kairos 2.4.3 (because the latest version fails when adding anything encryption or partition related, I know there is an issue for this), emulated TPM 2.0 device

I assume that I sent some malformed data, but also it shouldn't segfault right?

[jimmykarily]

No I don't think it's malformed data. I think it's the auth part that's missing: https://github.com/kairos-io/kcrypt-challenger/blob/c42e66a9de78193479e4c15ab5178cb0a60d357f/pkg/challenger/challenger.go#L250

what does your server look like? I assume you started from scratch and it just returns a json response right?
You may have better luck if you start from our kcrypt-challenger and strip it down to get rid of all the kubernetes stuff (e.g. things below findVolumeFor).

[firstdorsal]

Yes I created a server from scratch.

I thought about editing the challenger but decided against it as I have absolutely no experience with go.

use axum::extract::connect_info::ConnectInfo;
use axum::{
    extract::ws::{Message, WebSocket, WebSocketUpgrade},
    response::IntoResponse,
    routing::get,
    Router,
};
use axum_extra::TypedHeader;
use data_encoding::BASE64URL;
use serde::{Deserialize, Serialize};
use std::net::SocketAddr;
use tower_http::trace::{DefaultMakeSpan, TraceLayer};
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

#[tokio::main]
async fn main() {
    tracing_subscriber::registry()
        .with(
            tracing_subscriber::EnvFilter::try_from_default_env()
                .unwrap_or_else(|_| "example_websockets=debug,tower_http=debug".into()),
        )
        .with(tracing_subscriber::fmt::layer())
        .init();

    // build our application with some routes
    let app = Router::new()
        .route("/getPass", get(ws_handler))
        // logging so we can see whats going on
        .layer(
            TraceLayer::new_for_http()
                .make_span_with(DefaultMakeSpan::default().include_headers(true)),
        );

    // run it with hyper
    let listener = tokio::net::TcpListener::bind("0.0.0.0:30000")
        .await
        .unwrap();
    tracing::debug!("listening on {}", listener.local_addr().unwrap());
    axum::serve(
        listener,
        app.into_make_service_with_connect_info::<SocketAddr>(),
    )
    .await
    .unwrap();
}

async fn ws_handler(
    ws: WebSocketUpgrade,
    user_agent: Option<TypedHeader<headers::UserAgent>>,
    ConnectInfo(addr): ConnectInfo<SocketAddr>,
) -> impl IntoResponse {
    let user_agent = if let Some(TypedHeader(user_agent)) = user_agent {
        user_agent.to_string()
    } else {
        String::from("Unknown browser")
    };
    println!("`{user_agent}` at {addr} connected.");

    ws.on_upgrade(move |socket| handle_socket(socket, addr))
}

#[derive(Serialize, Deserialize, Debug)]
pub struct ResponseData {
    #[serde(rename = "Passphrase")]
    pub passphrase: String,
    #[serde(rename = "GeneratedBy")]
    pub generated_by: String,
}

async fn handle_socket(mut socket: WebSocket, who: SocketAddr) {
    // https://github.com/kairos-io/kcrypt-challenger/blob/main/cmd/discovery/client/client.go#L62

    let passphrase = "test";

    let response_data = ResponseData {
        passphrase: BASE64URL.encode(passphrase.as_bytes()),
        generated_by: "manual-crypt-challenger".to_string(),
    };

    let payload = serde_json::to_string(&response_data).unwrap();

    dbg!(&payload);

    if socket
        .send(Message::Binary(payload.into_bytes()))
        .await
        .is_ok()
    {
        println!("Sent KEY to {who}...");
    }

    // returning from the handler closes the websocket connection
    println!("Websocket context {who} destroyed");
}

[package]
name = "decryptor"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
axum-extra = { version = "0.9.2", features = ["typed-header"] }
axum = { version = "0.7.4", features = ["ws"] }
futures = "0.3"
futures-util = { version = "0.3", default-features = false, features = [
    "sink",
    "std",
] }
headers = "0.4"
tokio = { version = "1.0", features = ["full"] }
tokio-tungstenite = "0.21"
tower = { version = "0.4", features = ["util"] }
tower-http = { version = "0.5.0", features = ["fs", "trace"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
data-encoding = "2.5.0"