[GSoC'23] M1.3.2 - oppia#18255 Initial docker setup for dev server (o…

…ppia#18578)

* initial docker setup

* configured webpack-compiler and angular webserver

* added google-cloud-sdk dockerfile configurations

* initian docker configurations for GAE and cloud-datastore

* firebase emulator setup in docker compose file

* configured redis and elasticsearch as docker-compose service

* added some comments

* Update requirements.txt

* pyarrow uncommented

* configured compose-services + modified build.py

* configured protobuf files

* issues with GAE launch

* installed pip3 within the docker container

* configured google cloud sdk using zip file

* modified python script compile and install protobuf

* installed proto files using bash script in dockerfile

* server launching - dist folder not generating

* server launching - firebase and redis problem

* server launching - redis fixed

* working on firebase + wait-for-it script

* configured wait-for-it bash script to wait for a folder to get generated

* configured firebase

* server up properly -- elasticsearch service left to be configured

* Delete UsersshivkantchauhanDesktopopen-sour.txt

* docker setup ready with all services

* added initial healthcheck configurations + basic Make commands

* added healthchecks for each container

* implemented save_datastore flag

* implemented disable_host_checking flag

* removed disable_host_checking flag

* implementing flags - prod_env is having 1 error related to file hashes

* modified scripts to work under docker setup + implementing flags

* modified scripts to work under docker setup

* fixed lints

* reverted python scripts changes to work in both docker and python setup

* Update docker_setup.yml

* modified docker action

* modified healthchecks + python scripts

* fixed firebase connection error

* fixed firebase connection error + modified scripts

* fixed firebase export error + added new app_dev_dockerfile

* fixed firebase export error + added new app_dev_dockerfile

* fixed healthcheck for testing the folder

* fixed firebase dockerfile + docker action

* uncommented webserver and webpack compiler commands

* added docker directory

* fixing firebase volumes error

* added a log in docker action to see logs in CI

* added a log in docker action to see logs in CI

* tried optimising if-else in bash commands

* fixed firebase within docker compose

* fixed firebase within docker compose

* added no_auto_restart flag

* modified dockerfile

* frontend tests working in docker - not tested check_coverage + working on backend tests - getting error now

* fixed docker compose

* migrated mypy checks in docker

* modified docker compose

* initial docker setup

* fixed backend tests

* fixed lints

* fixed lints

* docker setup in merge actions

* try fixing lint checks for imports

* try fixing lint checks for imports

* modified dockerfile

* added separate script for dependencies.json package + multi-stage dockerfile

* modified redis and elasticsearch docker compose service

* added backend test in shard.json

* moved google cloud sdk in the root

* vol mapping for redis

* modified redis

* fixing lints

* fixing lints

* Update docker_setup.yml

* added test for minifying in prod_env

* deleted mock app.yaml files

* modified auth.service FE test

* Update auth.service.spec.ts

* modified docker compose to bind with local root dir

* modified docker compose to bind with local root dir

* fixing FE tests

* modified firebase service

* modified firebase service

* added dockerhub images versions

* removed env from firebase service

* fixed lints

* FE tests

* fixed digestIDs of dockerhub images

* changed node:16.13.0 digestID in frontend

* modified the start-time for each service

* bulk run for the docker action

* bulk run for docker action

* bulk run for docker action

* modified bulk run

* reverted bulk config for my repo

* modified comment in dockerfile

* improved docker action

* improved docker action

* added build stage in services

* added services named volumes

* removed backend target

* removed frontend build target

* removed digestIDs of dockerhub images

* removed digestID of python image

* added todo for firebase dependency

* changed dumb.db vol

* changed dumb.db vol

* changed dumb.db vol

* Update docker_setup.yml

* fixed redis/elasticsearch env issue

* setup

* added env variable in docker compose

* fixed dockerfile

* added stages in compose services

* changed env variable for docker

* added TODO

* optimising docker setup

* try fixing docker setup

* try fixing docker setup

* try fixing docker setup

* removing deps of build.py in dev_mode

* fixing CI tests

* fixing CI tests

* fixing CI failures

* fixing CI failures

* fixing CI failures

* fixing docker compose

* added build.py in dev mode in dockerfile

* fixing docker action

* removed angular-build healthchecks

* increased start time for angular-build for CI test to pass

* modified vol names

* modified compose file

* removed ports for redis and es

* modified install_third_party

* modified docker setup

* separated dockerfiles

* modified setup

* fixed scripts dependencies

* added multistage dockerfile

* separated dockerfiles

* increased docker setup reachability time on CI

* fixing CI tests

* make install_deps scripts independent

* added env in app_dev_docker for hosts

* fixed redis/es + lints

* fixing build_test

* fixing CI tests

* fixed backend coverage for install_deps script

* fixed backend coverage for common.py

* fixed backend coverage for common.py

* optimised docker setu
p

* optimised docker setup

---------

Co-authored-by: Shivkant Chauhan <[email protected]>

.firebase.json

@@ -1,7 +1,12 @@
 {
   "emulators": {
     "auth": {
+      "host": "0.0.0.0",
       "port": "9099"
+    },
+    "ui": {
+      "host": "0.0.0.0",
+      "port": "4000"
     }
   }
 }

.github/CODEOWNERS

@@ -186,6 +186,10 @@
 /.pylintrc @oppia/dev-workflow-reviewers
 /.stylelintrc @oppia/dev-workflow-reviewers
 /.yarnrc @oppia/dev-workflow-reviewers
+/app_dev_docker.yaml @oppia/dev-workflow-reviewers
+/docker-compose.yml @oppia/dev-workflow-reviewers
+/docker/Dockerfile.frontend @oppia/dev-workflow-reviewers
+/docker/Dockerfile.backend @oppia/dev-workflow-reviewers
 /tox.ini @oppia/dev-workflow-reviewers
 /scripts/*.py @oppia/dev-workflow-reviewers
 /scripts/check_e2e_tests_are_captured_in_ci*.py @oppia/automated-qa-reviewers
@@ -523,6 +527,7 @@
 /core/templates/services/interaction-specs.service*.ts @oppia/lace-frontend-reviewers
 /core/templates/services/translation-file-hash-loader-backend-api.service.ts @oppia/angular-migration-reviewers
 /redis.conf @oppia/data-and-stability-reviewers
+/redis_docker.conf @oppia/data-and-stability-reviewers
 /.firebase.json @oppia/data-and-stability-reviewers
 
 

.github/workflows/docker_setup.yml

@@ -0,0 +1,52 @@
+name: Docker Setup Test
+on:
+  merge_group:
+    types: [checks_requested]
+  push:
+    branches:
+      - develop
+      - release-*
+  pull_request:
+    branches:
+      - develop
+      - release-*
+
+jobs:
+  build:
+    runs-on:  ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-22.04]
+    steps:
+      - name: Checkout to the Oppia repository code
+        uses: actions/checkout@v3
+
+      - uses: ./.github/actions/merge
+
+      - name: Build Images
+        run: |
+          echo ---Building Docker Images---
+          docker compose build
+          
+      - name: Start Containers
+        run: |
+          echo ---Starting Containers---
+          docker compose up angular-build -d
+          docker compose up -d
+
+      - name: Checking server reachability
+        run: |
+          for i in {1..50}; do
+            if curl -s http://localhost:8181/; then
+              echo ---Dev-server is up---
+              exit 0
+            else
+              echo ---Retrying in 30 seconds---
+              sleep 30
+            fi
+          done
+          echo ---Dev-server is not started after all retries---
+          exit 1
+          
+      - name: Stop Containers
+        run: docker compose down

app_dev_docker.yaml

@@ -0,0 +1,149 @@
+# TODO(#18260): This file is used within the Docker setup for local development.
+# Modify the `app_dev.yaml` file when we permanently switch to Docker.
+runtime: custom
+instance_class: F2
+# The "version" line is added here so that MR jobs can run locally (see issue
+# #6534 on oppia/oppia).
+version: default
+# Timeout is set to 60 so that the gunicorn workers don't die while sending
+# the Apache Beam jobs to Google Cloud Dataflow.
+entrypoint: gunicorn --timeout 60 --bind :$PORT main:app
+
+inbound_services:
+- warmup
+- mail
+
+default_expiration: "10m"
+handlers:
+# PRODUCTION STATIC
+- url: /favicon.ico
+  static_files: assets/favicon.ico
+  upload: assets/favicon.ico
+  secure: always
+- url: /robots.txt
+  static_files: assets/robots.txt
+  upload: assets/robots.txt
+  secure: always
+- url: /sitemap.xml
+  static_files: assets/sitemap.xml
+  upload: assets/sitemap.xml
+  secure: always
+- url: /build
+  static_dir: build
+  secure: always
+  expiration: "90d"
+- url: /third_party/static
+  static_dir: third_party/static
+  secure: always
+  expiration: "90d"
+- url: /.well-known/security.txt
+  static_files: assets/.well-known/security.txt
+  upload: assets/.well-known/security.txt
+  secure: always
+
+# DEVELOPMENT STATIC
+- url: /webpack_bundles
+  static_dir: webpack_bundles
+  secure: always
+  expiration: "0"
+# /dist is the build output folder for angular.
+- url: /dist
+  static_dir: dist/
+  secure: always
+  expiration: "0"
+- url: /assets
+  static_dir: assets
+  secure: always
+  http_headers:
+    # This is replaced by a specific origin when doing a deployment.
+    Access-Control-Allow-Origin: "*"
+  expiration: "0"
+# Serve CSS files under core/templates.
+# This regex allows us to recursively serve CSS files.
+# "\1" inserts text captured by the capture group in the URL pattern.
+- url: /templates/(.*\.(css))$
+  static_files: core/templates/\1
+  upload: core/templates/(.*\.(css))$
+  secure: always
+  expiration: "0"
+- url: /templates/(.*\.(html))$
+  static_files: core/templates/\1
+  upload: core/templates/(.*\.(html))$
+  secure: always
+  expiration: "0"
+- url: /third_party/generated
+  static_dir: third_party/generated
+  secure: always
+  http_headers:
+    # This is replaced by a specific origin when doing a deployment.
+    Access-Control-Allow-Origin: "*"
+  expiration: "0"
+- url: /extensions/interactions/(.*)/static/(.*\.(css|png))
+  static_files: extensions/interactions/\1/static/\2
+  upload: extensions/interactions/(.*)/static/(.*\.(css|png))
+  secure: always
+  expiration: "0"
+- url: /extensions/(interactions|rich_text_components)/(.*)/directives/(.*\.(html))
+  static_files: extensions/\1/\2/directives/\3
+  upload: extensions/(interactions|rich_text_components)/(.*)/directives/(.*\.(html))
+  secure: always
+  expiration: "0"
+# Serve PNG images for interactions, rich_text_components, objects, classifiers,
+# and custom ckeditor plugins under extensions in dev mode. This regex allows
+# us to recursively serve PNG images scripts under the five specified
+# directories. "\1" and "\2" insert capture groups from the url pattern.
+- url: /extensions/(interactions|rich_text_components|objects|classifiers|ckeditor_plugins)/(.*\.(png))$
+  static_files: extensions/\1/\2
+  upload: extensions/(interactions|rich_text_components|objects|classifiers|ckeditor_plugins)/(.*\.(png))$
+  secure: always
+  expiration: "0"
+- url: /extensions/ckeditor_plugins/(.*\.(js))$
+  static_files: extensions/ckeditor_plugins/\1
+  upload: extensions/ckeditor_plugins/(.*\.(js))$
+  secure: always
+  expiration: "0"
+- url: /extensions/visualizations/(.*\.html)
+  static_files: extensions/visualizations/\1
+  upload: extensions/visualizations/(.*\.html)
+  secure: always
+  expiration: "0"
+- url: /extensions/objects/templates/(.*\.html)
+  static_files: extensions/objects/templates/\1
+  upload: extensions/objects/templates/(.*\.html)
+  secure: always
+  expiration: "0"
+
+# This is an access connector that connects to the Google Cloud Redis instance
+# used for memory caching. Please replace PROJECT_ID with the correct project id
+# for oppia on the production server.
+vpc_access_connector:
+  name: projects/PROJECT_ID/locations/us-central1/connectors/oppia-redis-connector-1
+
+env_variables:
+# We need these env variables to be set so that the services that are executed
+# by the Google App Engine knows which host they must bind with that they are
+# running in the docker environment.
+  REDIS_HOST: "redis"
+  ES_HOST: "elasticsearch"
+  PYTHONHTTPSVERIFY: 1
+# GAE_USE_SOCKETS_HTTPLIB is needed for the redis connector in the Python 2
+# runtime. More information can be found here:
+# https://cloud.google.com/appengine/docs/standard/python/connecting-vpc#configuring
+# (The second note under 'Configuring your app to use a connector') which
+# redirects to these instructions to disable URL fetch:
+# https://cloud.google.com/appengine/docs/standard/python/sockets#making_httplib_use_sockets
+  GAE_USE_SOCKETS_HTTPLIB : "anyvalue"
+# FIREBASE_AUTH_EMULATOR_HOST is needed to allow the Firebase SDK to connect
+# with the Firebase emulator. THIS MUST NOT BE DEPLOYED TO PRODUCTION. We
+# protect against this in the build script.
+  FIREBASE_AUTH_EMULATOR_HOST: "firebase:9099"
+# These environment variables allows Cloud NDB services to communicate with the
+# Cloud Datastore Emulator. THESE MUST NOT BE DEPLOYED TO PRODUCTION! We strip
+# them from the deployed app.yaml in scripts.build.
+  DATASTORE_DATASET: "dev-project-id"
+  DATASTORE_EMULATOR_HOST: "datastore:8089"
+  DATASTORE_EMULATOR_HOST_PATH: "datastore:8089/datastore"
+  DATASTORE_HOST: "http://datastore:8089"
+  DATASTORE_PROJECT_ID: "dev-project-id"
+  DATASTORE_USE_PROJECT_ID_AS_APP_ID: "true"
+  SECRETS: "{\"ANDROID_BUILD_SECRET\": \"android-build-secret\", \"VM_ID\": \"vm_default\", \"SHARED_SECRET_KEY\": \"1a2b3c4e\"}"

core/feconf.py

@@ -81,6 +81,8 @@ def check_dev_mode_is_true() -> None:
 
 check_dev_mode_is_true()
 
+# TODO(#18260): Remove this when we permanently move to the Dockerized Setup.
+OPPIA_IS_DOCKERIZED = bool(os.environ.get('OPPIA_IS_DOCKERIZED', False))
 CLASSIFIERS_DIR = os.path.join('extensions', 'classifiers')
 TESTS_DATA_DIR = os.path.join('core', 'tests', 'data')
 SAMPLE_EXPLORATIONS_DIR = os.path.join('data', 'explorations')
@@ -531,6 +533,7 @@ def get_empty_ratings() -> Dict[str, int]:
 # Valid Mailchimp tags.
 VALID_MAILCHIMP_TAGS = ['Account', 'Android', 'Web']
 
+ES_HOST = os.environ.get('ES_HOST', 'localhost')
 ES_LOCALHOST_PORT = 9200
 # NOTE TO RELEASE COORDINATORS: Replace this with the correct ElasticSearch
 # auth information during deployment.
@@ -541,7 +544,7 @@ def get_empty_ratings() -> Dict[str, int]:
 # Port when switching to prod server. Keep this in sync with redis.conf in the
 # root folder. Specifically, REDISPORT should always be the same as the port in
 # redis.conf.
-REDISHOST = 'localhost'
+REDISHOST = os.environ.get('REDIS_HOST', 'localhost')
 REDISPORT = 6379
 
 # The DB numbers for various Redis instances that Oppia uses. Do not reuse these
@@ -1359,7 +1362,8 @@ def get_empty_ratings() -> Dict[str, int]:
 # use alpha-numeric characters, hence the tighter restriction.
 FIREBASE_AUTH_ID_REGEX = '^[A-Za-z0-9]{1,128}$'
 
-CLOUD_DATASTORE_EMULATOR_HOST = 'localhost'
+# TODO(#18260): Change this when we permanently move to the Dockerized Setup.
+CLOUD_DATASTORE_EMULATOR_HOST = os.environ.get('DATASTORE_HOST', 'localhost')
 CLOUD_DATASTORE_EMULATOR_PORT = 8089
 
 FIREBASE_EMULATOR_CONFIG_PATH = '.firebase.json'

core/platform/search/elastic_search_services.py

@@ -39,7 +39,7 @@
 # where loading a exploration from local yaml file takes
 # longer than ElasticSearch expects.
 ES = elasticsearch.Elasticsearch(
-    ('localhost:%s' % feconf.ES_LOCALHOST_PORT)
+    ('%s:%s' % (feconf.ES_HOST, feconf.ES_LOCALHOST_PORT))
     if feconf.ES_CLOUD_ID is None else None,
     cloud_id=feconf.ES_CLOUD_ID,
     http_auth=(

core/templates/services/auth.service.spec.ts

@@ -65,11 +65,14 @@ describe('Auth service', function() {
     };
   });
 
-  it('should return emulator config when emulator is enabled', () => {
+  it('should return emulator config when emulator is enabled under ' +
+     'docker environment', () => {
     spyOnProperty(AuthService, 'firebaseEmulatorIsEnabled', 'get')
       .and.returnValue(true);
 
-    expect(AuthService.firebaseEmulatorConfig).toEqual(['localhost', 9099]);
+    // TODO(#18260): Change this when we permanently move to the Docker Setup.
+    process.env.OPPIA_IS_DOCKERIZED = 'true';
+    expect(AuthService.firebaseEmulatorConfig).toEqual(['0.0.0.0', 9099]);
   });
 
   it('should return undefined when emulator is disabled', () => {

core/templates/services/auth.service.ts

@@ -129,8 +129,11 @@ export class AuthService {
   }
 
   static get firebaseEmulatorConfig(): readonly [string, number] | undefined {
+    let firebaseHost = (
+      process.env.OPPIA_IS_DOCKERIZED ? '0.0.0.0' : 'localhost');
+    // TODO(#18260): Change this when we permanently move to the Docker Setup.
     return AuthService.firebaseEmulatorIsEnabled ?
-      ['localhost', 9099] : undefined;
+      [firebaseHost, 9099] : undefined;
   }
 
   async handleRedirectResultAsync(): Promise<boolean> {