Merge branch 'master' into refactor/SK-936

.github/workflows/build-containers.yaml

@@ -1,6 +1,7 @@
 name: "build containers"
 
 on:
+  workflow_dispatch:
   push:
     branches:
       - master

.github/workflows/code-checks.yaml

@@ -25,6 +25,7 @@ jobs:
           --exclude-dir='docs'
           --exclude-dir='flower-client'
           --exclude='tests.py'
+          --exclude='controller_cmd.py'
           --exclude='README.rst'
           '^[ \t]+(import|from) ' -I . 
 

.github/workflows/push-to-pypi.yaml

@@ -1,8 +1,9 @@
 name: Publish Python distribution to PyPI
 
 on:
+  workflow_dispatch:
   release:
-    types: [created]
+    types: published
 
 jobs:
   build-and-publish:

docker-compose.yaml

@@ -78,7 +78,7 @@ services:
       - mongo
     entrypoint: [ "sh", "-c" ]
     command:
-      - "/venv/bin/pip install --no-cache-dir -e . && /venv/bin/python fedn/network/api/server.py"
+      - "/venv/bin/pip install --no-cache-dir -e . && /venv/bin/fedn controller start"
     ports:
       - 8092:8092
 

docs/apiclient.rst

@@ -1,11 +1,9 @@
+.. _apiclient-label:
+
 APIClient
 =========
 
-.. note:: 
-
-   For access to FEDn Studio API, please see :ref:`studio-api`.
-
-FEDn comes with an *APIClient* for interacting with the FEDn network. The APIClient is a Python3 library that can be used to interact with the FEDn network programmatically. 
+FEDn comes with an *APIClient* - a Python3 library that can be used to interact with FEDn programmatically. 
 
 **Installation**
 
@@ -17,12 +15,15 @@ The APIClient is available as a Python package on PyPI, and can be installed usi
 
 **Initialize the APIClient**
 
-To initialize the APIClient, you need to provide the hostname and port of the FEDn API server. The default port is 8092. The following code snippet shows how to initialize the APIClient:
+The FEDn REST API is available at <controller-host>/api/v1/. To access this API you need the url to the controller-host, as well as an admin API token. The controller host can be found in the project dashboard (top right corner).
+To obtain an admin API token, navigate to the "Settings" tab in your Studio project and click on the "Generate token" button. Copy the 'access' token and use it to access the API using the instructions below. 
+
 
 .. code-block:: python
-   
-   from fedn import APIClient
-   client = APIClient("localhost", 8092)
+
+   >>> from fedn import APIClient
+   >>> client = APIClient(host="<controller-host>", token="<access-token>", secure=True, verify=True)
+
 
 **Set active package and seed model**
 
@@ -38,9 +39,9 @@ To set the initial seed model, you can use the following code snippet:
    
    client.set_active_model(path="path/to/seed.npz")
 
-**Start training session**
+**Start a training session**
 
-Once the active package and seed model are set, you can connect clients to the network and start training models. The following code snippet initializes a session (training rounds):
+Once the active package and seed model are set, you can connect clients to the network and start training models. The following code snippet starts a traing session:
 
 .. code-block:: python
    

docs/architecture.rst

@@ -1,3 +1,5 @@
+.. _architecture-label:
+
 Architecture overview
 =====================
 

docs/conf.py

@@ -12,7 +12,7 @@
 author = "Scaleout Systems AB"
 
 # The full version, including alpha/beta/rc tags
-release = "0.11.0"
+release = "0.11.1"
 
 # Add any Sphinx extension module names here, as strings
 extensions = [

docs/developer.rst

@@ -0,0 +1,168 @@
+.. _developer-label:
+
+Local development
+=================
+
+.. note::
+   These instructions are for users wanting to set up a local development deployment of FEDn (i.e. without FEDn Studio).
+   This requires practical knowledge of Docker and docker-compose. 
+
+Running the FEDn development sandbox (docker-compose)
+=====================================================
+
+During development on FEDn, and when working on own aggregators/helpers, it is 
+useful to have a local development setup of the core FEDn services (controller, combiner, database, object store). 
+For this, we provide Dockerfiles and docker-compose template. 
+
+To start a development sandbox for FEDn using docker-compose:
+
+.. code-block::
+
+   docker compose \
+    -f ../../docker-compose.yaml \
+    -f docker-compose.override.yaml \
+    up
+
+This starts up local services for MongoDB, Minio, the API Server, one Combiner and two clients. 
+You can verify the deployment using these urls: 
+
+- API Server: http://localhost:8092/get_controller_status
+- Minio: http://localhost:9000
+- Mongo Express: http://localhost:8081
+
+This setup does not include the security features of Studio, and thus will not require authentication of clients. 
+To use the APIClient to test a compute package and seed model against a local FEDn deployment: 
+
+.. code-block::
+
+   from fedn import APIClient
+   client = APIClient(host="localhost", port=8092)
+   client.set_active_package("package.tgz", helper="numpyhelper")
+   client.set_active_model("seed.npz")
+
+
+To connect a native FEDn client, you need to make sure that the combiner service can be resolved using the name "combiner". 
+One way to achieve this is to edit your '/etc/hosts' and add a line '127.0.0.1  	combiner'. 
+
+Access message logs and validation data from MongoDB  
+====================================================
+
+You can access and download event logs and validation data via the API, and you can also as a developer obtain 
+the MongoDB backend data using pymongo or via the MongoExpress interface: 
+
+- http://localhost:8081/db/fedn-network/ 
+
+Username and password are found in 'docker-compose.yaml'. 
+
+Access global models   
+====================
+
+You can obtain global model updates from the 'fedn-models' bucket in Minio: 
+
+- http://localhost:9000
+
+Username and password are found in 'docker-compose.yaml'. 
+
+Reset the FEDn deployment   
+=========================
+
+To purge all data from a deployment incuding all session and round data, access the MongoExpress UI interface and 
+delete the entire ``fedn-network`` collection. Then restart all services. 
+
+Clean up
+========
+You can clean up by running 
+
+.. code-block::
+
+   docker-compose -f ../../docker-compose.yaml -f docker-compose.override.yaml down -v
+
+
+Connecting clients using Docker:
+================================
+
+For convenience, we distribute a Docker image hosted on ghrc.io with FEDn preinstalled. For example, to start a client for the MNIST PyTorch example using Docker
+and FEDN 0.10.0, run this from the example folder:   
+
+.. code-block::
+
+   docker run \
+     -v $PWD/client.yaml:/app/client.yaml \
+     -e FEDN_PACKAGE_EXTRACT_DIR=package \
+     -e FEDN_NUM_DATA_SPLITS=2 \
+     -e FEDN_DATA_PATH=/app/package/data/clients/1/mnist.pt \
+     ghcr.io/scaleoutsystems/fedn/fedn:0.10.0 run client -in client.yaml --force-ssl --secure=True
+
+
+Self-managed distributed deployment
+===================================
+
+You can use different hosts for the various FEDn services. These instructions shows how to set up FEDn on a **local network** using a single workstation or laptop as 
+the host for the servier-side components, and other hosts or devices as clients. 
+
+.. note::
+   For a secure and production-grade deployment solution over **public networks**, explore the FEDn Studio service at 
+   **fedn.scaleoutsystems.com**. 
+
+   Alternatively follow this tutorial substituting the hosts local IP with your public IP, open the neccesary 
+   ports (see which ports are used in docker-compose.yaml), and ensure you have taken additional neccesary security 
+   precautions.
+
+**Prerequisites**
+-  `One host workstation and atleast one client device`
+-  `Python 3.8, 3.9, 3.10 or 3.11 <https://www.python.org/downloads>`__
+-  `Docker <https://docs.docker.com/get-docker>`__
+-  `Docker Compose <https://docs.docker.com/compose/install>`__
+
+Launch a distributed FEDn Network 
+---------------------------------
+
+
+Start by noting your host's local IP address, used within your network. Discover it by running ifconfig on UNIX or 
+ipconfig on Windows, typically listed under inet for Unix and IPv4 for Windows.
+
+Continue by following the standard procedure to initiate a FEDn network, for example using the provided docker-compose template. 
+Once the network is active, upload your compute package and seed (for comprehensive details, see the quickstart tutorials).
+
+.. note::
+   This guide covers general local networks where server and client may be on different hosts but able to communicate on their private IPs. 
+   A common scenario is also to run fedn and the clients on **localhost** on a single machine. In that case, you can replace <host local ip>
+   by "127.0.0.1" below.   
+
+Configuring and Attaching Clients
+---------------------------------
+
+On your client device, continue with initializing your client. To connect to the host machine we need to ensure we are 
+routing the correct DNS to our hosts local IP address. We can do this using the standard FEDn `client.yaml`:
+
+.. code-block::
+
+   network_id: fedn-network
+   discover_host: api-server
+   discover_port: 8092
+
+
+We can then run a client using docker by adding the hostname:ip mapping in the docker run command:
+
+.. code-block::
+
+   docker run \
+   -v $PWD/client.yaml:<client.yaml file location> \
+   <potentiel data pointers>
+   —add-host=api-server:<host local ip> \
+   —add-host=combiner:<host local ip> \
+   <image name> run client -in client.yaml --name client1
+
+
+Alternatively updating the `/etc/hosts` file, appending the following lines for running naitively:
+
+.. code-block::
+
+   <host local ip>      api-server
+   <host local ip>      combiner
+
+
+Start a training session
+------------------------
+
+After connecting with your clients, you are ready to start training sessions from the host machine.

docs/index.rst

@@ -4,19 +4,18 @@
 
    introduction
    quickstart
+   projects
 
 .. toctree::
    :maxdepth: 1
    :caption: Documentation
 
-   studio
-   distributed
    apiclient
-   projects
    architecture
    aggregators
    helpers
    auth
+   developer
 
 .. toctree::
    :maxdepth: 1