Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define a serverless protocol #1

Open
sameo opened this issue Feb 4, 2024 · 10 comments
Open

Define a serverless protocol #1

sameo opened this issue Feb 4, 2024 · 10 comments
Assignees

Comments

@sameo
Copy link
Contributor

sameo commented Feb 4, 2024

A protocol for the host (Lumper) to send commands to the guest (serverless agent #6) and receive replies from it.

  • Define commands and responses
  • Define packet format
@sameo sameo moved this to Todo in Project 2024 Feb 4, 2024
@sameo sameo changed the title Define a protocol between the host and the agent Define a serverless protocol Feb 4, 2024
@mmoreiradj
Copy link
Contributor

mmoreiradj commented Mar 1, 2024

Commands

Configure

Request

This command would configure the agent with fields such as:

  • language
  • variables/inputs
  • code
  • log level

Note: it could be just one command with all the needed configuration.

Response

On success, return OK (or a similar message).

On error, return NOK (or similar), with a reason.

Run

This command can be called once the configuration was successful.

Request

Starts the code

Response

This command would start a build (if needed) and the run code.

It would return nothing on successful reception of the command, but it would log the steps it takes.

Logs

Request

A simple log from the guest to the host.

They include:

  • Error: runtime errors, build error, configuration errors
  • Info: code result, maybe compiler warnings
  • Debug: steps taken, building time, starting time, etc.

Response

Interruptions / Stop

Request

Just a signal to stop the guest

Response

Nothing.

Subjects to consider:

  • Do we need an interruption command ?
  • Metrics ? Can't the VMM get those without the agent communicating them ?
  • Timeouts ? Retries ?
  • Multiple Files ? Since we've separated the configuration and runtime, this could be a possibility.

@mmoreiradj
Copy link
Contributor

Communication

Slip Based

As for the communication, for now, I thought we could either use a SLIP Based protocol over the serial line. It would look like this:

Information Length (in bits)
Message Type 16
Checksum 16
Data ...

The message type could be log, config, etc.

Protobuf

We could also define .proto file and serialize / deserialize the messages with it.

Note that this could be reused if we manage to set up virtio-net and use gRPC further down the line.

@mfernd
Copy link
Contributor

mfernd commented Mar 1, 2024

@virt-do/vmm

@charley04310
Copy link
Contributor

Hello @mmoreiradj and thanks for your job !

Every thing looks good to me !

Regarding the interruption command, I agree that it would be beneficial to have such a feature. It would allow for the proper termination of code execution, which can be more efficient in terms of resource management compared to letting a code run that will inevitably need to be restarted.

As for metrics, it indeed depends on the available time, but having feedback such as execution time and memory usage could be valuable. These metrics could provide insights into the performance and efficiency of the code, which can be useful for optimization purposes.

Concerning multiple files, it still depends on the available time, one potential use case could be when a user's code is divided into several modules or when the codebase is large ?? But is this really useful for the moment ?

Regarding the communication part, I would suggest using Protobuf for its efficient serialization and deserialization capabilities, as well as its strict and maintainable nature and also as you said, it could be reused if we manage to get virtio up and running-net and use gRPC further.

@charley04310
Copy link
Contributor

Also, Muriel, would suggest that the "run" command should return the container ID, which should be used for all other commands (such as metrics, shutdown, etc.).

@mfernd
Copy link
Contributor

mfernd commented Mar 9, 2024

Also, Muriel, would suggest that the "run" command should return the container ID, which should be used for all other commands (such as metrics, shutdown, etc.).

We (agent) needs to run only one process, so I don't think it's useful to return an id for this.

@mmoreiradj
Copy link
Contributor

mmoreiradj commented Mar 16, 2024

After experimenting with communication over serial ports, here's my two cents:

Reminder:

  • One vmm = One Agent
  • Neither the VMM nor the Agent handle orchestration
  • For now we only manage build and execution of code but we should prepare to run http servers for example in the future

Communication

The VMM and agent will exchange messages over the serial port until Virt-IO is ready to be implemented. To make sure each message is received, it should be request response based.

Messages Types

pub enum MessageType {
    Start = 0,
    Exit = 1,
    Interrupt = 2,
    Ok = 3,
    Log = 4,
}

/// Expects OkMessage
/// Sent when the Agent starts
pub struct StartMessage {
    content: String,
}

/// Expects OkMessage
/// Sent when the agent exits
pub struct ExitMessage {
    code: i32,
    content: String,
}

/// Expects OkMessage
/// Sent when the VMM wants to stop the agent
pub struct InterruptMessage {
    signal: i32,
}

/// Mostly used to answer other messages
pub struct OkMessage {}

/// Expects OkMessage
pub struct LogMessage {
    kind: String,
    content: String,
}

ATM I'm unsure of whether the LogMessage is useful since the program could simply send them to stdout, I'd like to have your input.

Message Format

Information Length (in bits)
Message Type 16
Checksum 16
Data ...

We should go for this format, which would be easier to implement. It's the easiest format to make evolve.

The Message Type would identify what to serialize the data into:

pub enum MessageType {
    A = 0,
    B = 1,
    ...
}

For example, if the receiver gets a message type of 1, we would serialize the message in the struct B {}. It will also be used to determine what to answer with.

The checksum is simply here to make sure we didn't miss any data. We just need to determine what algorithm to use and the size of the checksum.

Please give me your input and whether you think we could implement that.

@virt-do/vmm @virt-do/guest-agent

@mmoreiradj
Copy link
Contributor

mmoreiradj commented Mar 16, 2024

Also, I forgot to mention that we should have a terminator to determine when the message is done.

Here's an idea:

Hex Value Dec Value Oct Value Abbreviation Description
0xC0 192 300 END Frame End
0xDB 219 333 ESC Frame Escape
0xDC 220 334 ESC_END Transposed Frame End
0xDD 221 335 ESC_ESC Transposed Frame Escape

@mfernd
Copy link
Contributor

mfernd commented Mar 17, 2024

I have questions, only for the definition, not code related:

  1. What is the purpose of the content field in the StartMessage and ExitMessage? Is it to send the program content, only in StartMessage, why for ExitMessage?
  2. Does the structure you send refers only to the request messages? Do you have an idea on the format for the responses?
    (If they're only for requests, we should refer to them as StartRequest, ExitRequest, etc)
  3. Do you think we should send these messages over IP? (with the SLIP protocol, for a smooth transition with virtio-net in the future)
  4. What does the OkMessage is for? To respond to requests with an Ok/Err?

I also think we should consider the protobuf de/serialization and/or grpc protocol over the slip, maybe it could be easier, but I wonder if it adds too much overhead (since it's using http) and if it's possible over serial

@mmoreiradj
Copy link
Contributor

mmoreiradj commented Mar 17, 2024

  1. You can leave them empty, it's just if we need to send data, config or anything else
  2. They are for request and responses
  3. If you manage to implement it and provide a small proof of concept, I'd love to.
  4. It's to make sure the request was received.

Again, if you manage to make a proof of concept, I'd love to do it.

For information, here's a POC of what I described: https://github.com/mmoreiradj/serial-port-demo

@mfernd mfernd moved this from Todo to Done in Project 2024 May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

No branches or pull requests

6 participants