Feature / Move gateway authentication to a separate service (#477)

* Add a new auth service to Gradle build files * Rename GrpcAuthValidator * Add thread local logging helper * Rename GrpcAuthHelpers * Provide a general HTTP/1 auth validator as part of the common auth (internal) package * Include HTTP codecs (1 & 2) in the common lib package * Use the new auth validator as the auth handler in the gateway. Login processing is removed from the gateway and will now be handled by the auth service. * Remove gateway dependency on -lib-auth. Gateway is just using internal auth validation, the same as a regular service. * Disable auth for protocol negotiation test * Explicit protocol version check in protocol negotiation test * Allow for Web Sockets in the HTTP/1 auth validator * Disable auth in the HTTP/1 proxy test * Do not prepare auth in platform tests if auth is disabled * Disable auth in unit tests where it is not needed * Clean up some code warnings * Disable auth for web API testing workflow * Move HTTP test helpers into common test library * Stub class for the authentication service * Add auth service to the platform test framework * Start skeleton auth service in a test case * Stub test cases for guest login * Factor protocol negotiator into a common base class and move to common lib * Move protocol negotiator test to common lib * Make the gateway use the new common protocol negotiator * Move web server onto the common protocol negotiator * Fix setup of HTTP codec for different scenarios in the protocol negotiator * Let the auth service start up and install the auth router * Rename TRAC auth providers package * Move user DB into impl package for TRAC auth * Rename login provider interface * Rename methods in login interface * Rename auth headers interface * Rename auth login package * Rename simple login package * Rename simple login plugin * Restrict simple login classes to package scope * Remove references to connId in http login handler * New auth provider interfaces (sits above login interface) * Update login plugin for guest protocol to use new IAuthProvider interface * make IUserDatabase a private detail of simple login * Make -svc-auth depend on -lib-auth * Include both auth and logic providers as plugin types * Expose login providers in simple login plugin * Include external systems in auth config * Lookup class to register and give out auth providers * Make auth service use the auth provider * Move login content responses to the public login package * Rename login content dir * Add optional config items to for login and return paths * Make login content use config and query params for return path * Remove unused methods in ILoginProvider * Update main login handler to remove code for request pass-through * Working tests for basic login and redirect * Handle differences between Java 17 / 21 in guest login test * Main functionality of gateway auth handler * Fix for removing disabled auth handler * Updates in gateway HTTP/1 auth handler * Updates in gateway HTTP/1 auth handler * Fix handling query params in gateway HTTP/1 proxy * Serve login content from JAR resources * Move guest login test into -lib-auth * Update namespace for login tests * Allow platform test helper to be used for config only (i.e. without starting services) * Update login test to test the login handler in isolation, but supplying the required TRAC config * Rename cookies / headers / metadata keys to match the normal convention * Common wrapper classes for working with HTTP request / response objects across protocol versions * Remove old auth headers classes (using Netty Headers base class instead) * Consolidate auth helpers for managing auth headers, move to using Netty's generic Headers framework * Use new common HTTP framework for login base classes * Update login providers for new framework * Finalize tests for guest login and login refresh * Working version of HTTP/1 login handler - all tests passing with guest login * Update built in login to work with new framework * Move close wrapper in to general test helpers package * Use new auth helpers in HTTP/1 auth validator / gw-handler classes * Update top level namespace for -lib-auth * Rename some classes in the login package * Update login providers after rename * Remove hard coding in basic auth header * Smoke test to ensure auth service is serving the login provider * Add a login test firing multiple requests on the same connection * Add test cases for dummy provider and provider switching * Remove unneeded comment (doesn't relate to this code) * Platform test code updated after auth lib refactor * Add a dummy auth provider for the auth service tests * Update dummy auth provider and associated tests * Update netty setup for auth service main class * Relay handler to pass messages back from an embedded channel * Logging and connection ID in base protocol negotiator * Update config protos for auth * Finalize auth service provider lookup * Auth service config to work with latest config proto * Add service properties to platform config * Use service properties to set idle timeout in auth service * Trace logging in the auth service * Config helpers and constants for service properties * Use service properties to set idle timeout in the gateway * Trace logging in the base protocol negotiator * Use new service props to set idle timeout for the gateway in dev local config * Remove explicit idle timeout field in gateway config (using service properties instead) * Fix common tests after changes * Update config files after config changes * Updates and fixes in auth headers logic * Rename wrapHttpHeaders to make it clear headers are not copied * Handle reading of auth settings config for the gateway * Update IAuthProvider interface in -lib-auth * Always require cookie direction in auth helpers * Working version of gateway auth handler for HTTP/1, no token refresh support yet * Use auth settings to set up auth handler in the gateway * Update configs to include auth service and new service properties * Implemented token refresh in GW HTTP/1 auth handler * Close connections in the GW auth handler if authentication fails * Logging and idle handling improvements for the gateway * Update log messages for token refresh * Remove unused code * Update expiry header name * Rename common auth package, to reflect namespace change in -lib-auth * More intelligent setup of the gw routing for services * Use auth service info to set up login redirect path in auth handler settings * Fix tests for login expiry token * Use some different ports for end-to-end tests * Use service enabled flag to enable / disable the web server * Update dev local config for web server enabled change * Use some different ports for testing * Update dev local and platform template config files * Fix rewrites of the platform config file for web API test in CI * Use the same ports as dev-local in template platform config * Fix rewrites of the platform config file for web API test in CI * Config updates * Web socket fixes * Fix header prefix for header translation in websockets router * Fix log noise for web sockets in the Gateway * Add log for web sockets translation * Logger types for core router * Fix lib auth tests * Fix lib common tests * Simpler log setup in core router baser class
finos · Dec 8, 2024 · f52ca0b · f52ca0b
1 parent 437e62c
commit f52ca0b
Show file tree

Hide file tree

Showing 128 changed files with 6,684 additions and 2,373 deletions.
diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
@@ -79,6 +79,7 @@ jobs:
           
           echo SECRET_KEY=xHjFeFlo$k= >> etc/env.sh
           sed -i "s#/path/to#${deploy_dir}#g" etc/trac-platform.yaml
+          sed -i "s#jwtIssuer: trac_platform#disableAuth: true#" etc/trac-platform.yaml
           
           bin/secret-tool run --task init_secrets
           bin/secret-tool run --task create_root_auth_key EC 256

diff --git a/dev/config/trac-devlocal.yaml b/dev/config/trac-devlocal.yaml
@@ -87,61 +87,63 @@ jobCache:
 
 webServer:
 
-  # To use the TRAC web server, set the enabled flag and uncomment the configuration
+  # To use the TRAC web server, enable it in the services section then add a content root
 
-  enabled: false
-
-#  contentRoot:
-#    protocol: LOCAL
-#    properties:
-#      rootPath: path/to/web/content/root
-#
-#  rewriteRules:
-#
-#  redirects:
-#    - source: /
-#      target: /index.html
-#      status: 302
+  #  contentRoot:
+  #    protocol: LOCAL
+  #    properties:
+  #      rootPath: path/to/web/content/root
+  #
+  #  rewriteRules:
+  #
+  #  redirects:
+  #    - source: /
+  #      target: /index.html
+  #      status: 302
 
 
 gateway:
 
-  idleTimeout: 10
-
   # Routing for the TRAC services will be configured automatically
   # To add custom routes, uncomment this configuration and modify as required
 
-#  routes:
-#
-#    - routeName: Local Web Server
-#      routeType: HTTP
-#
-#      match:
-#        host: localhost
-#        path: /local/app
-#
-#      target:
-#        scheme: http
-#        host: localhost
-#        port: 9090
-#        path: /
+  #  routes:
+  #
+  #    - routeName: Local Development Server
+  #      routeType: HTTP
+  #
+  #      match:
+  #        host: localhost
+  #        path: /local=app/
+  #
+  #      target:
+  #        scheme: http
+  #        host: localhost
+  #        port: 3000
+  #        path: /
 
 services:
 
   gateway:
     port: 8080
+    properties:
+      network.idleTimeout: 10
 
-  metadata:
+  authentication:
     port: 8081
 
-  data:
+  metadata:
     port: 8082
 
-  orchestrator:
+  data:
     port: 8083
 
+  orchestrator:
+    port: 8084
+
   webServer:
-    port: 8090
+    enabled: false
+    port: 8085
 
 
 deployment:

diff --git a/dist/template/etc/trac-platform.yaml b/dist/template/etc/trac-platform.yaml
@@ -88,61 +88,63 @@ jobCache:
 
 webServer:
 
-  # To use the TRAC web server, set the enabled flag and uncomment the configuration
+  # To use the TRAC web server, enable it in the services section then add a content root
 
-  enabled: false
-
-#  contentRoot:
-#    protocol: LOCAL
-#    properties:
-#      rootPath: path/to/web/content/root
-#
-#  rewriteRules:
-#
-#  redirects:
-#    - source: /
-#      target: /index.html
-#      status: 302
+  #  contentRoot:
+  #    protocol: LOCAL
+  #    properties:
+  #      rootPath: path/to/web/content/root
+  #
+  #  rewriteRules:
+  #
+  #  redirects:
+  #    - source: /
+  #      target: /index.html
+  #      status: 302
 
 
 gateway:
 
-  idleTimeout: 10
-
-# Routing for the TRAC services will be configured automatically
-# To add custom routes, uncomment this configuration and modify as required
-
-#  routes:
-#
-#    - routeName: Local Web Server
-#      routeType: HTTP
-#
-#      match:
-#        host: localhost
-#        path: /local/app
-#
-#      target:
-#        scheme: http
-#        host: localhost
-#        port: 9090
-#        path: /
+  # Routing for the TRAC services will be configured automatically
+  # To add custom routes, uncomment this configuration and modify as required
+
+  #  routes:
+  #
+  #    - routeName: Local Development Server
+  #      routeType: HTTP
+  #
+  #      match:
+  #        host: localhost
+  #        path: /local=app/
+  #
+  #      target:
+  #        scheme: http
+  #        host: localhost
+  #        port: 3000
+  #        path: /
 
 services:
 
   gateway:
     port: 8080
+    properties:
+      network.idleTimeout: 10
 
-  metadata:
+  authentication:
     port: 8081
 
-  data:
+  metadata:
     port: 8082
 
-  orchestrator:
+  data:
     port: 8083
 
+  orchestrator:
+    port: 8084
+
   webServer:
-    port: 8090
+    enabled: false
+    port: 8085
 
 
 deployment:

diff --git a/settings.gradle b/settings.gradle
@@ -61,12 +61,14 @@ project(":tracdap-lib-test").projectDir = file("tracdap-libs/tracdap-lib-test")
 
 // Java services
 
+include 'tracdap-svc-auth'
 include 'tracdap-svc-meta'
 include 'tracdap-svc-data'
 include 'tracdap-svc-orch'
 include 'tracdap-gateway'
 include 'tracdap-webserver'
 
+project(":tracdap-svc-auth").projectDir = file("tracdap-services/tracdap-svc-auth")
 project(":tracdap-svc-meta").projectDir = file("tracdap-services/tracdap-svc-meta")
 project(":tracdap-svc-data").projectDir = file("tracdap-services/tracdap-svc-data")
 project(":tracdap-svc-orch").projectDir = file("tracdap-services/tracdap-svc-orch")

diff --git a/tracdap-api/packages/web/wrapper.js b/tracdap-api/packages/web/wrapper.js
@@ -165,7 +165,7 @@
             this.urlPrefix = options["browser"] ? "" : this.hostAddress;
 
             this.rpcMetadata = {
-                "trac_auth_cookies": "true"  // request the auth response is sent back in cookies
+                "trac-auth-cookies": "true"  // request the auth response is sent back in cookies
             }
 
             this.grpcWeb = new grpc.GrpcWebClientBase({format: 'binary'});
@@ -240,7 +240,7 @@
             "accept": "application/grpc-web+proto",
             "x-grpc-web": 1,
             "x-user-agent": "trac-web-transport",  // TODO: version
-            "trac_auth_cookies": "true"  // request the auth response is sent back in cookies
+            "trac-auth-cookies": "true"  // request the auth response is sent back in cookies
         }
 
         const FILTER_RESPONSE_HEADERS = ["cookie", "set-cookie", "authorization"]

diff --git a/tracdap-api/tracdap-config/src/main/proto/tracdap/config/common.proto b/tracdap-api/tracdap-config/src/main/proto/tracdap/config/common.proto
@@ -53,7 +53,11 @@ message AuthenticationConfig {
   sint32 jwtLimit = 6;
   sint32 jwtRefresh = 7;
 
-  optional PluginConfig provider = 3;
+  PluginConfig provider = 3;
+
+  optional string loginPath = 12;
+  optional string refreshPath = 13;
+  optional string returnPath = 14;
 
   bool disableAuth = 4;
   bool disableSigning = 5;
@@ -62,6 +66,8 @@ message AuthenticationConfig {
   string systemUserName = 9;
   sint32 systemTicketDuration = 10;
   sint32 systemTicketRefresh = 11;
+
+  map<string, PluginConfig> externalSystems = 15;
 }
 
 
@@ -82,4 +88,6 @@ message ServiceConfig {
 
   string alias = 2;
   uint32 port = 3;
+
+  map<string, string> properties = 4;
 }
diff --git a/tracdap-api/tracdap-config/src/main/proto/tracdap/config/platform.proto b/tracdap-api/tracdap-config/src/main/proto/tracdap/config/platform.proto
@@ -44,9 +44,9 @@ message PlatformConfig {
 
   map<string, TenantConfig> tenants = 10;
 
-  optional WebServerConfig webServer = 11;
+  WebServerConfig webServer = 11;
 
-  optional GatewayConfig gateway = 13;
+  GatewayConfig gateway = 13;
 
   map<string, ServiceConfig> services = 4;
 
@@ -71,7 +71,9 @@ message TenantConfig {
 
 message WebServerConfig {
 
-  bool enabled = 1;
+  // Setting removed, use service config instead
+  reserved "enabled";
+  reserved 1;
 
   PluginConfig contentRoot = 3;
   repeated WebServerRewriteRule rewriteRules = 4;
@@ -93,10 +95,11 @@ message WebServerRedirect {
 
 message GatewayConfig {
 
-  uint32 idleTimeout = 1;
+  // Idle timeout moved into common service properties
+  reserved 1;
+  reserved "idleTimeout";
 
   repeated RouteConfig routes = 2;
-
   repeated WebServerRedirect redirects = 3;
 }