From 1b05ab7e294b505fe057a17cd253174083e57d14 Mon Sep 17 00:00:00 2001 From: root Date: Wed, 22 May 2024 09:23:10 +0000 Subject: [PATCH] README.md updated --- README.md | 171 +++++++++--------------------------------------------- 1 file changed, 26 insertions(+), 145 deletions(-) diff --git a/README.md b/README.md index fe6c6e8..1e5eee9 100644 --- a/README.md +++ b/README.md @@ -23,174 +23,55 @@ In case you have any questions, feel free to contact us. ## Prerequisites, when running clients in Docker containers - [Python 3.8, 3.9 or 3.10](https://www.python.org/downloads) -- [Docker](https://docs.docker.com/get-docker) -The example assumes working experience with Docker. -## Attach clients to an existing FEDn Network (for workshop) +Creating the compute package and seed model +------------------------------------------- -> [!IMPORTANT] -> For the workshop, we will use PyTorch based compute-package. The workshop leader will provide the dataset and the client configuration file, client.yaml. +Install fedn: +.. code-block:: -### Using Docker (Linux and MacOS) + pip install fedn -Here we assume that the FEDn network is up and running and you have obtained the connection file (client.yaml). In case you are participating in a Scaleout workshop, you will obtain the file from the workshop organizer. If you are working on the tutorial on your own, complete the instructions below before connecting the client. +Clone this repository, then locate into this directory: -> [!NOTE] -> In case you want to run a client on M1 or M2 macOS using TensorFlow and Keras, make sure you have a compatible framework version. +.. code-block:: -Clone the repository + https://github.com/scaleoutsystems/power-consumption-tutorial.git + cd power-consumption-tutorial/Power-consumption-pytorch -``` -git clone https://github.com/scaleoutsystems/Power-consumption-tutorial.git -cd Power-consumption-tutorial -``` - -The following command will connect your client to the network specified in `client.yaml`. Please fix the path of the `power.npz` and `client.yaml` files according to your local setup. - -> [!NOTE] -> Make sure that the Docker daemon is running on your system. - -```sh - docker run -v $PWD/client.yaml:/app/client.yaml -v $PWD/data:/var/data -e ENTRYPOINT_OPTS=--data_path=/var/data/power.npz ghcr.io/scaleoutsystems/power-consumption:pyt_v0.6.0 fedn run client --secure=True --force-ssl -in client.yaml -``` ----- - -In case you do not want to download an external container from a public repository, run the following two commands. - -```sh -cd Power-consumption-pytorch -docker build --no-cache -t power-consumption:local . -docker run -v $PWD/client.yaml:/app/client.yaml -v $PWD/data:/var/data -e ENTRYPOINT_OPTS=--data_path=/var/data/power.npz power-consumption:local fedn run client --secure=True --force-ssl -in client.yaml -``` - -These two commands build a local container and start a new client based on a locally built client container. - ----- - -### Using Docker on Windows - -Here we assume that the FEDn network is up and running and you have obtained the connection file (client.yaml). In case you are participating in a Scaleout workshop, you will obtain the file from the workshop organizer. If you are working on the tutorial on your own, complete the instructions below in the section (Setting up the federation (model initiator)) before connecting the client. - -Start `PowerShell` and install `virtualenv` if needed, then create a `virtualenv` named `scaleout_test` and activate it (just type deactivate to deactivate the virtualenv)) - -```sh -pip install virtualenv -python -m venv scaleout_test -scaleout_test/Scripts/activate -cd scaleout_test -``` - -Clone this repository - -```sh -git clone https://github.com/scaleoutsystems/Power-consumption-tutorial.git -cd Power-consumption-tutorial -``` - -> [!NOTE] -> The workshop leader will provide the dataset and the client configuration file, `client.yaml`. - -Download the `dataset` and `client.yaml` file and move them to the `Power-consumption-tutorial` directory. The directory structure will look as follows: - -``` - README.md - client.yaml - data - Power-consumption-keras - Power-consumption-pytorch -``` - -where the `data` directory contains the `power.npz` file. - -The following command will connect your client to the network specified in `client.yaml`. Please fix the path of the `power.npz` and `client.yaml` files according to your local setup. - -> [!NOTE] -> Make sure that the Docker deamon is running on your system. +Create the compute package: -```sh -docker run -v $PWD/client.yaml:/app/client.yaml -v $PWD/data:/var/data -e ENTRYPOINT_OPTS=--data_path=/var/data/power.npz ghcr.io/scaleoutsystems/power-consumption:pyt_v0.6.0 fedn run client --secure=True --force-ssl -in client.yaml -in client.yaml -``` ------ +.. code-block:: -### Natively on your host (without docker) + fedn package create --path client -Clone this repository -``` -git clone https://github.com/scaleoutsystems/Power-consumption-tutorial.git -cd Power-consumption-tutorial/Power-consumption-pytorch -``` +This should create a file 'package.tgz' in the project folder. -Create virtual env (from root folder in the repository) +Next, generate a seed model (the first model in a global model trail): -On *Linux*: -``` -bin/init_venv.sh -``` +.. code-block:: -On *MacOS* (tested on M1 macs): -``` -bin/init_venv_macos.sh -``` + fedn run build --path client -Install FEDn into the virtual env -``` -.power-consumption-keras/bin/pip install git+https://github.com/scaleoutsystems/fedn.git@master#egg=fedn\&subdirectory=fedn -``` +This will create a seed model called 'seed.npz' in the root of the project. This step will take a few minutes, depending on hardware and internet connection (builds a virtualenv). -Activate the virtual env -``` -source .power-consumption-keras/bin/activate -``` +Download the dataset: -Specify absolute path to your local data is located (replace `/path/to/`) -``` -export ENTRYPOINT_OPTS=--data_path=/path/to/power.npz -``` +For Linux and MacOS -Start the client (assumes you have the client config file `client.yaml`) -``` -fedn run client --secure=True --force-ssl -in client.yaml -``` - ------ - -## Extra (not for workshop): Setting up the federation (model initiator) - -These instructions are for users that want to learn to deploy and intiatialize the federated network (model initiator). - -### Creating the federated learning network -There are two main options to deploy a FEDn network: - -1. Obtain an account in Scaleout Studio. Apply here: [https://studio.scaleoutsystems.com/signup/](https://studio.scaleoutsystems.com/signup/) and contact Scaleout staff in the [Discord server](https://discord.gg/KMg4VwszAd) for this option. -2. [Deploy a FEDn network from scratch](https://github.com/scaleoutsystems/fedn) on your own machine(s). -3. A partition of the power consumption test dataset can be downloaded using the following command: ```sh -wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=1r_dlOEZAnCLhRjY1qFwlRAkeB4PvhgAU' -O power.npz + mkdir power-consumption-tutorial/Power-consumption-pytorch/data + wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=1r_dlOEZAnCLhRjY1qFwlRAkeB4PvhgAU' -O power.npz ``` -### Preparing the environment, the compute package and the seed model. +For Windows, download the dataset using the link: `https://docs.google.com/uc?export=download&id=1r_dlOEZAnCLhRjY1qFwlRAkeB4PvhgAU` -Clone the above-mentioned FEDn repository in step 2. Locate into the directory, pick one of the available examples, then: - -Initialize a virtual environment with all of the required dependencies. -```sh -bin/init_venv.sh -``` - -Build the compute package and seed model. -```sh -bin/build.sh -``` -You should now have two files, `package.tar.gz` and `seed.npz`. Initialize the FEDn network using these, and then follow instructions above to connect clients. - -Build the environment (Docker image) -```sh -docker build -t scaleoutsystems/power-consumption:main -``` -(If you have not made local changes to the package and/or requirements.txt, you can also use the pre-build package available in this repository, `ghcr.io/scaleoutsystems/power-consumption:pyt_v0.6.0`) +Using FEDn Studio +----------------- -You can now connect the client following the instructions above. Note that depending on how you deployed the network, you might need to modify some of the command line options to fedn. Refer to the [FEDn documentation](https://github.com/scaleoutsystems/fedn). +Follow the guide here to set up your FEDn Studio project and learn how to connect clients (using token authentication): `Studio guide `__. +On the step "Upload Files", upload 'package.tgz' and 'seed.npz' created above.