Docs/SK-958 | Improve docs (#675)

* Update headings in quickstart to align with the typical user-flow in a FL project. * Improved introduction section * Move APIClient examples from Quickstart to APIClient page * Work in progress on APIClient page * Move auth into developer guide * latest wip * Removed prerequisite in projects page * wip * Rework the project page, add code example * Fixed test compute package section * Removed unused file * Fixed API page and some headings --------- Co-authored-by: Andreas Hellander <[email protected]>
scaleoutsystems · Aug 30, 2024 · 4d00645 · 4d00645
1 parent 622b2ad
commit 4d00645
Show file tree

Hide file tree

Showing 9 changed files with 608 additions and 330 deletions.
diff --git a/docs/apiclient.rst b/docs/apiclient.rst
@@ -1,9 +1,13 @@
 .. _apiclient-label:
 
-APIClient
-=========
+Using the Python API
+====================
 
 FEDn comes with an *APIClient* - a Python3 library that can be used to interact with FEDn programmatically. 
+In this tutorial we show how to use the APIClient to initialize the server-side with the compute package and seed models, 
+run and control training sessions, use different aggregators, and to retrieve models and metrics. 
+
+We assume a basic understanding of the FEDn framework, i.e. that the user have taken the Getting Started tutorial: :ref:`quickstart-label`
 
 **Installation**
 
@@ -13,12 +17,11 @@ The APIClient is available as a Python package on PyPI, and can be installed usi
    
    $ pip install fedn
 
-**Initialize the APIClient**
+**Initialize the APIClient to a FEDn Studio project**
 
 The FEDn REST API is available at <controller-host>/api/v1/. To access this API you need the url to the controller-host, as well as an admin API token. The controller host can be found in the project dashboard (top right corner).
 To obtain an admin API token, navigate to the "Settings" tab in your Studio project and click on the "Generate token" button. Copy the 'access' token and use it to access the API using the instructions below. 
 
-
 .. code-block:: python
 
    >>> from fedn import APIClient
@@ -36,28 +39,65 @@ Then passing a token as an argument is not required.
    >>> from fedn import APIClient
    >>> client = APIClient(host="<controller-host>", secure=True, verify=True)
 
+**Set the active compute package and seed model**
 
-**Set active package and seed model**
+To set the active compute package in the FEDn Studio Project:
 
-The active package can be set using the following code snippet:
+.. code:: python
 
-.. code-block:: python
-   
-   client.set_active_package(path="path/to/package.tgz", helper="numpyhelper")
+   >>> from fedn import APIClient
+   >>> client = APIClient(host="<controller-host>", token="<access-token>", secure=True, verify=True)
+   >>> client.set_active_package("package.tgz", helper="numpyhelper")
+   >>> client.set_active_model("seed.npz")
 
-To set the initial seed model, you can use the following code snippet:
+**Start a training session**
 
-.. code-block:: python
-   
-   client.set_active_model(path="path/to/seed.npz")
+Once the active package and seed model are set, you can connect clients to the network and start training models. To run a training session
+using the default aggregator (FedAvg):
 
-**Start a training session**
+.. code:: python
+
+   >>> ...
+   >>> client.start_session(id="test-session", rounds=3)
+   # Wait for training to complete, when controller is idle:
+   >>> client.get_controller_status()
+   # Show model trail:
+   >>> models = client.get_model_trail()
+   # Show performance of latest global model:
+   >>> model_id = models[-1]['model']
+   >>> validations = client.get_validations(model_id=model_id)
 
-Once the active package and seed model are set, you can connect clients to the network and start training models. The following code snippet starts a traing session:
+To run a session using the FedAdam aggregator using custom hyperparamters: 
 
 .. code-block:: python
-   
-   session = client.start_session(id="session_name")
+
+   >>> session_id = "experiment_fedadam"
+
+   >>> session_config = {
+                     "helper": "numpyhelper",
+                     "id": session_id,
+                     "aggregator": "fedopt",
+                     "aggregator_kwargs": {
+                           "serveropt": "adam",
+                           "learning_rate": 1e-2,
+                           "beta1": 0.9,
+                           "beta2": 0.99,
+                           "tau": 1e-4
+                           },
+                     "model_id": seed_model['model'],
+                     "rounds": 10
+                  }
+
+   >>> result_fedadam = client.start_session(**session_config)
+
+**Download a global model**
+
+To download a global model and write it to file:
+
+.. code:: python
+
+   >>> ...
+   >>> client.download_model("<model-id>", path="model.npz")
 
 **List data**
 
@@ -77,17 +117,17 @@ Entities represented in the APIClient are:
 * statuses
 * validations
 
-The following code snippet shows how to list all sessions:
-
+To list all sessions: 
 .. code-block:: python
    
-   sessions = client.get_sessions()
+   >>> sessions = client.get_sessions()
 
-And the following code snippet shows how to get a specific session:
+To get a specific session:
 
 .. code-block:: python
    
-   session = client.get_session(id="session_name")
-
+   >>> session = client.get_session(id="session_name")
 
-For more information on how to use the APIClient, see the :py:mod:`fedn.network.api.client`, and the example `Notebooks <https://github.com/scaleoutsystems/fedn/tree/master/examples/notebooks>`_. 
+For more information on how to use the APIClient, see the :py:mod:`fedn.network.api.client`, and the collection of example Jupyter Notebooks:
+
+- `API Example <https://github.com/scaleoutsystems/fedn/tree/master/examples/notebooks>`_  . 
diff --git a/docs/conf.py b/docs/conf.py
@@ -25,6 +25,7 @@
     "sphinx.ext.viewcode",
     "sphinx_rtd_theme",
     "sphinx_code_tabs",
+    "sphinx_design",
 ]
 
 # The master toctree document.
@@ -97,7 +98,8 @@
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-    (master_doc, "fedn", "FEDn Documentation", author, "fedn", "One line description of project.", "Miscellaneous"),
+    (master_doc, "fedn", "FEDn Documentation", author, "fedn",
+     "One line description of project.", "Miscellaneous"),
 ]
 
 # Bibliographic Dublin Core info.

diff --git a/docs/developer.rst b/docs/developer.rst
@@ -1,20 +1,21 @@
 .. _developer-label:
 
-Local development and deployment
-================================
+================
+Developer guide
+================
 
-.. note::
-   These instructions are for users wanting to set up a local development deployment of FEDn (i.e. without FEDn Studio).
-   This requires practical knowledge of Docker and docker-compose. 
 
-Running the FEDn development sandbox (docker-compose)
-------------------------------------------------------
+Pseudo-distributed sandbox
+===========================
 
-During development on FEDn, and when working on own aggregators/helpers, it is 
-useful to have a local development setup of the core FEDn services (controller, combiner, database, object store). 
-For this, we provide Dockerfiles and docker-compose template. 
+.. note::
+   These instructions are for users wanting to set up a bare-minimum local deployment of FEDn (without FEDn Studio).
+   We here assume practical knowledge of Docker and docker-compose. We recommend all new users of FEDn to start
+   by taking the Getting Started tutorial: :ref:`quickstart-label`
 
-To start a development sandbox for FEDn using docker-compose:
+During development on FEDn, and when working on own extentions including aggregators and helpers, it is 
+useful to have a local development setup of the core FEDn server-side services (controller, combiner, database, object store). 
+We provide Dockerfiles and docker-compose template for an all-in-one local sandbox: 
 
 .. code-block::
 
@@ -24,14 +25,14 @@ To start a development sandbox for FEDn using docker-compose:
     up
 
 This starts up local services for MongoDB, Minio, the API Server, one Combiner and two clients. 
-You can verify the deployment using these urls: 
+You can verify the deployment on localhost using these urls: 
 
 - API Server: http://localhost:8092/get_controller_status
 - Minio: http://localhost:9000
 - Mongo Express: http://localhost:8081
 
-This setup does not include the security features of Studio, and thus will not require authentication of clients. 
-To use the APIClient to test a compute package and seed model against a local FEDn deployment: 
+This setup does not include any of the security and authentication features available in a Studio Project, 
+so we will not require authentication of clients (insecure mode) when using the APIClient:  
 
 .. code-block::
 
@@ -40,8 +41,7 @@ To use the APIClient to test a compute package and seed model against a local FE
    client.set_active_package("package.tgz", helper="numpyhelper")
    client.set_active_model("seed.npz")
 
-
-To connect a native FEDn client, you need to make sure that the combiner service can be resolved using the name "combiner". 
+To connect a native FEDn client to the sandbox deployment, you need to make sure that the combiner service can be resolved by the client using the name "combiner". 
 One way to achieve this is to edit your '/etc/hosts' and add a line '127.0.0.1  	combiner'. 
 
 Access message logs and validation data from MongoDB  
@@ -76,7 +76,6 @@ You can clean up by running
 
    docker-compose -f ../../docker-compose.yaml -f docker-compose.override.yaml down -v
 
-
 Connecting clients using Docker:
 ------------------------------------------------------
 
@@ -93,8 +92,8 @@ and FEDN 0.10.0, run this from the example folder:
      ghcr.io/scaleoutsystems/fedn/fedn:0.10.0 run client -in client.yaml --force-ssl --secure=True
 
 
-Self-managed distributed deployment
-------------------------------------------------------
+Distributed deployment on a local network
+=========================================
 
 You can use different hosts for the various FEDn services. These instructions shows how to set up FEDn on a **local network** using a single workstation or laptop as 
 the host for the servier-side components, and other hosts or devices as clients. 
@@ -116,7 +115,6 @@ the host for the servier-side components, and other hosts or devices as clients.
 Launch a distributed FEDn Network 
 ---------------------------------
 
-
 Start by noting your host's local IP address, used within your network. Discover it by running ifconfig on UNIX or 
 ipconfig on Windows, typically listed under inet for Unix and IPv4 for Windows.
 
@@ -159,3 +157,110 @@ Alternatively updating the `/etc/hosts` file, appending the following lines for
 
    <host local ip>      api-server
    <host local ip>      combiner
+
+
+.. _auth-label:
+
+Authentication and Authorization (RBAC)
+========================================
+
+.. warning:: The FEDn RBAC system is an experimental feature and may change in the future.
+
+FEDn supports Role-Based Access Control (RBAC) for controlling access to the FEDn API and gRPC endpoints. The RBAC system is based on JSON Web Tokens (JWT) and is implemented using the `jwt` package. The JWT tokens are used to authenticate users and to control access to the FEDn API.
+There are two types of JWT tokens used in the FEDn RBAC system:
+- Access tokens: Used to authenticate users and to control access to the FEDn API.
+- Refresh tokens: Used to obtain new access tokens when the old ones expire.
+
+.. note:: Please note that the FEDn RBAC system is not enabled by default and does not issue JWT tokens. It is used to integrate with external authentication and authorization systems such as FEDn Studio. 
+
+FEDn RBAC system is by default configured with four types of roles:
+- `admin`: Has full access to the FEDn API. This role is used to manage the FEDn network using the API client or the FEDn CLI.
+- `combiner`: Has access to the /add_combiner endpoint in the API.
+- `client`: Has access to the /add_client endpoint in the API and various gRPC endpoint to participate in federated learning sessions.
+
+A full list of the "roles to endpoint" mappings for gRPC can be found in the `fedn/network/grpc/auth.py`. For the API, the mappings are defined using custom decorators defined in `fedn/network/api/auth.py`.
+
+.. note:: The roles are handled by a custom claim in the JWT token called `role`. The claim is used to control access to the FEDn API and gRPC endpoints.
+
+To enable the FEDn RBAC system, you need to set the following environment variables in the controller and combiner:
+
+Authentication Environment Variables
+-------------------------------------
+
+.. line-block::
+
+     **FEDN_JWT_SECRET_KEY**
+      - **Type:** str
+      - **Required:** yes
+      - **Default:** None
+      - **Description:** The secret key used for JWT token encryption.
+
+     **FEDN_JWT_ALGORITHM**
+      - **Type:** str
+      - **Required:** no
+      - **Default:** "HS256"
+      - **Description:** The algorithm used for JWT token encryption.
+
+     **FEDN_AUTH_SCHEME**
+      - **Type:** str
+      - **Required:** no
+      - **Default:** "Token"
+      - **Description:** The authentication scheme used in the FEDn API and gRPC interceptors.
+
+Additional Environment Variables
+--------------------------------
+
+For further flexibility, you can also set the following environment variables:
+
+.. line-block::
+
+     **FEDN_CUSTOM_URL_PREFIX**
+      - **Type:** str
+      - **Required:** no
+      - **Default:** None
+      - **Description:** Add a custom URL prefix used in the FEDn API, such as /internal or /v1.
+
+     **FEDN_AUTH_WHITELIST_URL**
+      - **Type:** str
+      - **Required:** no
+      - **Default:** None
+      - **Description:** A URL pattern to the API that should be excluded from the FEDn RBAC system. For example, /internal (to enable internal API calls).
+
+     **FEDN_JWT_CUSTOM_CLAIM_KEY**
+      - **Type:** str
+      - **Required:** no
+      - **Default:** None
+      - **Description:** The custom claim key used in the JWT token.
+
+     **FEDN_JWT_CUSTOM_CLAIM_VALUE**
+      - **Type:** str
+      - **Required:** no
+      - **Default:** None
+      - **Description:** The custom claim value used in the JWT token.
+
+Client Environment Variables
+-----------------------------
+
+For the client, you need to set the following environment variables:
+
+.. line-block::
+
+     **FEDN_AUTH_REFRESH_TOKEN_URI**
+      - **Type:** str
+      - **Required:** no
+      - **Default:** None
+      - **Description:** The URI used to obtain new access tokens when the old ones expire.
+
+     **FEDN_AUTH_REFRESH_TOKEN**
+      - **Type:** str
+      - **Required:** no
+      - **Default:** None
+      - **Description:** The refresh token used to obtain new access tokens when the old ones expire.
+
+     **FEDN_AUTH_SCHEME**
+      - **Type:** str
+      - **Required:** no
+      - **Default:** "Token"
+      - **Description:** The authentication scheme used in the FEDn API and gRPC interceptors.
+
+You can use `--token` flags in the FEDn CLI to set the access token.
diff --git a/docs/helpers.rst b/docs/helpers.rst
@@ -1,6 +1,6 @@
 .. _helper-label:
 
-Model Serialization/Deserialization
+Model marshalling
 ===================================
 
 In federated learning, model updates need to be serialized and deserialized in order to be 

diff --git a/docs/index.rst b/docs/index.rst
@@ -5,16 +5,15 @@
    introduction
    quickstart
    projects
+   apiclient
 
 .. toctree::
    :maxdepth: 1
    :caption: Documentation
 
-   apiclient
    architecture
    aggregators
    helpers
-   auth
    developer
 
 .. toctree::
@@ -24,7 +23,6 @@
    faq
    modules
 
-
 Indices and tables
 ==================