Skip to content

Commit

Permalink
initial commit (#184)
Browse files Browse the repository at this point in the history
  • Loading branch information
@dgkanatsios
dgkanatsios authored Mar 18, 2022
1 parent bda3386 commit 75e49e1
Show file tree
Hide file tree
Showing 27 changed files with 434 additions and 278 deletions.
28 changes: 14 additions & 14 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,29 +1,29 @@
![CI workflow](https://github.com/playfab/thundernetes/actions/workflows/main.yml/badge.svg)
[![Software License](https://img.shields.io/badge/license-Apache-brightgreen.svg?style=flat-square)](LICENSE)
[![GitHub release](https://img.shields.io/github/release/playfab/thundernetes.svg)](https://github.com/playfab/thundernetes/releases)
![](https://img.shields.io/badge/status-alpha-red.svg)
![](https://img.shields.io/badge/status-beta-lightgreen.svg)
[![CodeQL](https://github.com/PlayFab/thundernetes/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/PlayFab/thundernetes/actions/workflows/codeql-analysis.yml)

# (code name) thundernetes
# Thundernetes

> Thundernetes is an experimental project and not recommended for production use. However, feel free to try it and let us know of any feedback! Thundernetes is a [code name](https://github.com/PlayFab/thundernetes/issues/177), project will soon have a new name!
Thundernetes makes it easy to run your game servers on Kubernetes.

## Description
> Thundernetes is a project still in development, feel free to try it and let us know of any [feedback](#feedback---community)!
Thundernetes is a project from the [Azure PlayFab Multiplayer Servers (MPS)](https://docs.microsoft.com/gaming/playfab/features/multiplayer/servers/) team that enables you to run Linux game servers on your Kubernetes cluster. Thundernetes can be useful in the following scenarios:
## ℹ️ Description

- host your game servers on a Kubernetes cluster, either on a public cloud provider or on-premises
- do manual allocations of game server sessions
- validate your game server integration with GSDK
- as part of your iterative development process, you can use thundernetes to test your game server code before pushing it to the MPS service
- as part of your CI/CD pipeline, you can publish the game server to a container registry and then have it deploy to a Kubernetes cluster where you can run your tests
Thundernetes is a project originating from the [Azure PlayFab Multiplayer Servers (MPS)](https://docs.microsoft.com/gaming/playfab/features/multiplayer/servers/) team and other teams in Azure/XBOX that enables you to run Linux game servers on your Kubernetes cluster. Thundernetes can be useful in the following scenarios:

## Usage
- host your game servers on a Kubernetes cluster, either on a public cloud provider or on-premise and allow your users to connect from everywhere
- pre-warm game servers so that they are ready to accept players within seconds, when the game is about to start
- as part of your iterative development process, you can use Thundernetes to test your game server code

Check [our website](https://playfab.github.io/thundernetes) for more information.
## 📚 Documentation

Check 🔥[our website](https://playfab.github.io/thundernetes)🔥 for more information.

## Feedback
## 💬❓Feedback - Community

As mentioned, thundernetes is in preview and a work in progress. If you find a bug or have a feature request, please file an issue [here](https://github.com/PlayFab/thundernetes/issues) and we will try to get back to you as soon as possible. You can also reach us directly on [Game Stack server on Discord](https://discord.gg/gamestack).
As mentioned, Thundernetes is in preview and a work in progress. If you find a bug or have a feature request, please file an issue [here](https://github.com/PlayFab/thundernetes/issues) and we will try to get back to you as soon as possible. You can also reach us directly on [Game Dev server on Discord](https://aka.ms/msftgamedevdiscord).

> Project will soon have a new name (https://github.com/PlayFab/thundernetes/issues/177)!
68 changes: 8 additions & 60 deletions docs/FAQ.md
View file
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
---
layout: default
title: Frequently Asked Questions
nav_order: 11
nav_order: 13
---

# Frequently Asked Questions

## Can I run a Unity or Unreal game server on thundernetes?
## Can I run a Unity or Unreal game server on Thundernetes?

You can run any game server that supports the [PlayFab GameServer SDK](https://github.com/PlayFab/gsdk). Check a Unity sample [here](https://github.com/PlayFab/thundernetes/tree/main/samples/unity/README.md). On [this](https://github.com/PlayFab/MpsSamples) repository you can find samples for all programming languages GSDK supports, like C#/Java/C++/Go/Unity/Unreal.
You can run any game server that supports the [PlayFab GameServer SDK](https://github.com/PlayFab/gsdk). Check a Unity sample [here](https://github.com/PlayFab/thundernetes/tree/main/samples/unity/README.md) and documentation for the Unreal plugin [here](https://github.com/PlayFab/gsdk/tree/main/UnrealPlugin). On [this](https://github.com/PlayFab/MpsSamples) repository you can find samples for all programming languages GSDK supports, like C#/Java/C++/Go/Unity/Unreal.

## How can I add custom Annotations and/or Labels to my GameServer Pods?

Expand Down Expand Up @@ -45,62 +45,13 @@ spec:
## Can I run my game server pods in a non-default namespace?
You don't need to do anything special to run your game server Pods in a namespace different than "default". Old versions of thundernetes (up to 0.1) made use of a sidecar to access the Kubernetes API Server, so you needed to create special RoleBinding and ServiceAccount in the non-default namespace. With the transition to DaemonSet NodeAgent in 0.2, this is no longer necessary.
## How do I schedule thundernetes Pods and GameServer Pods into different Nodes?
In production environments, you would like to have system and thundernetes Pods (Pods that are created on the kube-system and thundernetes-system namespaces) scheduled on a different set Nodes other than the GameServer Pods. One reason for this might be that you want special Node types for your GameServers. For example, you might want to have dedicated Nodes with special GPUs for your GameServers. Another reason might be that you don't want any interruption whatsoever to Pods that are critical for the cluster to run properly (system and thundernetes Pods). One approach to achieve this isolation on public cloud providers is by using multiple Node Pools. A Node Pool is essentially a group of Nodes that share the same configuration (CPU type, memory, etc) and can be scaled independently of the others. In production scenarios, it is recommended to use three Node Pools:
- one Node Pool for Kubernetes system resources (everything in kube-system namespace) and thundernetes system resources (everything in thundernetes-system namespace)
- one Node Pool for telemetry related Pods (Prometheus, Grafana, etc)
- one Node Pool to host your GameServer Pods
Let's discuss on how to create and use a Node Pool to host the GameServer Pods.
1. First, you would need to create a separate NodePool for the GameServer Pods. Check [here](https://docs.microsoft.com/azure/aks/use-multiple-node-pools) on how to do it on Azure Kubernetes Service. Create this on "user" mode so that "kube-system" Pods are not scheduled on this NodePool. Most importantly, when creating a NodePool, you can specify custom Labels for the Nodes. Let's assume that you apply the `agentpool=gameserver` Label.
1. Use the `nodeSelector` field on your GameServer Pod spec to request that the GameServer Pod is scheduled on Nodes that have the `agentpool=gameserver` Label. Take a look at this [sample YAML file](https://github.com/PlayFab/thundernetes/tree/main/samples/netcore/sample_second_node_pool.yaml) for an example.
1. When you create your GameServerBuild, the GameServer Pods will be scheduled on the NodePool you created.
1. Moreover, you should modify the `nodeSelector` field on the controller Pod spec to make sure it will be scheduled on the system Node Pool. On AKS, if the system Node Pool is called `nodepool1`, you should add this YAML snippet to the `thundernetes-controller-manager` Deployment on the [YAML install file](https://github.com/PlayFab/thundernetes/tree/main/installfiles/operator.yaml):

```YAML
nodeSelector:
agentpool: nodepool1
```

You should add the above YAML snippet to any workloads you don't want to be scheduled on the GameServer NodePool. Check [here](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) for additional information on assigning pods to Nodes and check [here](https://docs.microsoft.com/azure/aks/use-system-pools#system-and-user-node-pools) for more information on AKS system and user node pools.

### Schedule DaemonSet Pods on GameServer Nodes

> For more information on the NodeAgent process running in the DaemonSet, check the architecture document [here](architecture.md#gsdk-integration).

Now that we've shown how to run multiple Node Pools, let's discuss how to schedule DaemonSet Pods running NodeAgent process to run only on Nodes that run game server Pods. Since NodeAgent's only concern is to work with game server Pods on Node's it's been scheduled, it's unnecessary to run in on Nodes that run system resources and/or telemetry. Since we have already split the cluster into multiple Node Pools, we can use the `nodeSelector` field on the DaemonSet Pod spec to request that the DaemonSet Pod is scheduled on Nodes that have the `agentpool=gameserver` Label (or whatever Label you have added to your game server Node Pool). Take a look at the following example to see how you can modify your DaemonSet YAML for this purpose:

```YAML
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: thundernetes-nodeagent
namespace: thundernetes-system
spec:
selector:
matchLabels:
name: nodeagent
template:
metadata:
labels:
name: nodeagent
spec:
nodeSelector: # add this line
agentpool: gameserver # add this line as well
containers:
...
```
Yes. You don't need to do anything special to run your game server Pods in a namespace different than "default".
## How do I make GameServer Pods start before DaemonSet Pods?
When a new Node is added to the Kubernetes cluster, a NodeAgent Pod (part of DaemonSet) will be created there. However, if there were pending GameServer Pods before the Node's addition to the cluster, they will also be scheduled on the new Node. Consequently, GameServer Pods might start at the same time as the NodeAgent Pod. GameServer Pods are heartbeating to the NodeAgent process so there is a chance that some heartbeats will be lost and, potentially, a state change from "" to "Initializing" will not be tracked (however, the GameServer Pod should have no trouble transitioning to StandingBy when the NodeAgent Pod is up and can process heartbeats).
There will be no impact from these lost heartbeats. However, you can tell Kubernetes to schedule NodeAgent Pods before the GameServer Pods by assigning Pod Priorities to the NodeAgent Pods. You can read more about Pod priority [here](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption) and specifically about the impact of Pod priority on scheduling order [here](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#effect-of-pod-priority-on-scheduling-order).
There will be no impact from these lost heartbeats. However, you can ask Kubernetes to schedule NodeAgent Pods before the GameServer Pods by assigning Pod Priorities to the NodeAgent Pods. You can read more about Pod priority [here](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption) and specifically about the impact of Pod priority on scheduling order [here](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#effect-of-pod-priority-on-scheduling-order).
## How can I add resource constraints to my GameServer Pods?
Expand All @@ -127,13 +78,10 @@ template:
For a full sample, you can check [here](https://github.com/PlayFab/thundernetes/tree/main/samples/netcore/sample-requestslimits.yaml).
## Not supported features (compared to MPS)

There are some features of MPS that are not yet supported on Thundernetes.
## How can I run custom code on my container to respond to a terminated event from Kubernetes?
1. Thundernetes, for the time being, supports only Linux game servers. Work to support Windows is tracked [here](https://github.com/PlayFab/thundernetes/issues/8), please leave a comment if that's important for you. If you want to host Windows game servers, you can always use [MPS](https://docs.microsoft.com/gaming/playfab/features/multiplayer/servers/).
1. On PlayFab MPS, you can upload a zip file that contains parts of your game server (referred to as assets). This is decompressed on the VM that your game server runs and is automatically mounted. You cannot do that on Thundernetes, however you can always mount a storage volume onto your Pod (e.g. check [here](https://kubernetes.io/docs/concepts/storage/volumes/#azuredisk) on how to mount an Azure Disk).
You can use the [PreStop container hook](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks). Effectively, this hook is executed when the container is terminated. You can use this hook to perform custom cleanup operations. Read [here](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination) for more information about Pod termination.
## Where does the name 'thundernetes' come from?
## Where does the name 'Thundernetes' come from?
It's a combination of the words 'thunderhead' and 'kubernetes'. 'Thunderhead' is the internal code name for the Azure PlayFab Multiplayer Servers service. Credits to [Andreas Pohl](https://github.com/Annonator) for the naming idea!
53 changes: 23 additions & 30 deletions docs/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,51 +8,44 @@ permalink: /

# Thundernetes

Welcome to (code name) Thundernetes! Please use the links on the left to navigate the documentation.
Welcome to Thundernetes, an open source project from teams in Azure/XBOX that enables you to run Linux game servers on your Kubernetes cluster!

## Description
:exclamation: Latest release: [![GitHub release](https://img.shields.io/github/release/playfab/thundernetes.svg)](https://github.com/playfab/thundernetes/releases)

Thundernetes is a project from the Azure PlayFab Multiplayer Servers (MPS) team that enables you to run Linux game servers on your Kubernetes cluster. Thundernetes can be useful in the following scenarios:
## Prerequisite knowledge

- host your game servers on a Kubernetes cluster, either on a public cloud provider or on-premises
- do manual allocations of game server sessions
- validate your game server integration with GSDK
- as part of your iterative development process, you can use thundernetes to test your game server code before pushing it to the MPS service
- as part of your CI/CD pipeline, you can publish the game server to a container registry and then have it deploy to a Kubernetes cluster where you can run your tests
- For a game server to be able to run in thundernetes, it must use the PlayFab Game Server SDK (GSDK) either directly on the game server binary or indirectly, via a wrapper.
New to Kubernetes or containers? Check our [prerequisites](prerequisites.md) document that has resources that will fill the knowledge gaps when working with technologies within Thundernetes.

Thundernetes requires a Kubernetes cluster with Public IP per Node. We've tested it extensively on [Azure Kubernetes Service - AKS](https://docs.microsoft.com/azure/aks/intro-kubernetes) as well as in local clusters using [kind](https://kind.sigs.k8s.io/). You also need to have ports 10000-12000 open in your cluster, since these are the ports that Thundernetes by default will set up on your Kubernetes Nodes so they can receive game network traffic and forward to your game server Pod.
## Requirements

> This port range is configurable, check [here](howtos/configureportrange.md) for details.
> You can use a Kubernetes cluster without a Public IP. However, you would need to configure your own network architecture if you want to access your game servers. For example, if you use a cloud provider's Load Balancer, you would need to configure routes from Load Balancer's public endpoints to the internal ones on your Kubernetes cluster.
> You can try Azure Kubernetes Service for free [azure.com/free](https://azure.com/free).
Thundernetes requires:

## Prerequisites
- A Kubernetes cluster, either on-premise or on a public cloud provider. Ideally, the cluster should support having a Public IP per Node to allow external incoming connections
- A game server
- integrated with the open source [Game Server SDK](https://playfab.com/gsdk) (GSDK). GSDK has been battle-tested by multiple AAA titles for years on the [Azure PlayFab Multiplayer Servers service](https://docs.microsoft.com/gaming/playfab/features/multiplayer/servers/) and supports multiple popular programming languages and game engines like Unity, Unreal, C#, C++, Java, Go.
- built as a Linux container image. This image should be deployed to a container registry that your Kubernetes cluster can access.

Check our [prerequisites](docs/prerequisites.md) document that has resources that will fill the knowledge gaps when working with technologies within thundernetes.
> **_NOTE_**: You can avoid having to integrate with GSDK by using the [wrapper sample](usingwrapper.md). This sample is great if you want to experiment with Thundernetes, however proper GSDK integration is highly recommended.
## Quickstart

Check the [quickstart](docs/quickstart.md) document on how to install thundernetes on your cluster and run the sample game server.
Check the [quickstart](quickstart.md) document on how to install Thundernetes on your cluster and run a sample game server to verify that Thundernetes is working properly.

### Installing on Azure Kubernetes Service
Click on the following image to see how easy it is to install and use Thundernetes:

Click on the following image for a quick preview of the quickstart:
[![asciicast](https://asciinema.org/a/438455.svg)](https://asciinema.org/a/438455)

[![asciicast](https://asciinema.org/a/438455.png)](https://asciinema.org/a/438455)
## Recommended links

## Links

- [Prerequisites](prerequisites.md) - resources that will fill the knowledge gaps when working with technologies within thundernetes
- [Quickstart](quickstart.md) - Recommended - how to install thundernetes on your cluster and run the sample game server
- [Defining a GameServerBuild](gameserverbuild.md) - Recommended - how to define a GameServerBuild in YAML
- [Your game server](yourgameserver.md) - how to use thundernetes with your own game server
- [Using a wrapper](usingwrapper.md) - use a wrapper process to launch your game server in Thundernetes, without integrating with GSDK
- [Your game server](yourgameserver.md) - how to use Thundernetes with your own game server
- [Defining a GameServerBuild](gameserverbuild.md) - how to define a GameServerBuild in YAML
- [Game Server lifecycle](gameserverlifecycle.md) - game server process lifecycle
- [Architecture](architecture.md)
- [Frequently Asked Questions](FAQ.md)
- [Troubleshooting Guide](troubleshooting/README.md) - public repository for all of thundernetes Troubleshooting guides
- [Development notes](development.md) - useful if you are working on thundernetes development
- [Architecture](architecture.md) - overview of Thundernetes architecture
- [Troubleshooting Guide](troubleshooting/README.md) - public repository for all of Thundernetes troubleshooting guides
- [Development notes](development.md) - useful development notes if you are plan on contributing to Thundernetes
- [Frequently Asked Questions](FAQ.md) - frequently asked questions

## Contributing

If you are interested in contributing to thundernetes, please read our [Contributing Guide](contributing.md) and open a PR. We'd be more than happy to help you out!
If you are interested in contributing to Thundernetes, please read our [Contributing Guide](contributing.md) and open a PR. We'd be more than happy to help you out!
Loading

0 comments on commit 75e49e1

Please sign in to comment.