Skip to content

Latest commit

 

History

History
276 lines (229 loc) · 17.5 KB

dev-setup.md

File metadata and controls

276 lines (229 loc) · 17.5 KB
Raw

Setting up your Developer Environment (gradle, node, Chrome/Firefox)

This section describes the recommended Developer Environment and how to set it up. You are welcome to use any other tools you prefer.

Download open source software

  1. JDK 15. This can also be either installed, or downloaded as a .zip. If you do not use the installer, be sure to set the JAVA_HOME environment variable to the location of the JDK.
  2. git. While this is not required, it is highly recommended if you will be doing active development on ANET.

Download ANET source code

  1. Checkout the source code from GitHub. 
    git clone [email protected]:NCI-Agency/anet.git
  2. Install the recommended git hooks 
    cd anet
git config core.hooksPath scripts/githooks

Possible Problems

  • You cannot access the source code repo. Solution: Get someone who does have admin access to add you as a collaborator. Ensure that you have the correct public key installed to GitHub. See Connecting to GitHub with SSH for more information on troubleshooting this step.
  • The git clone command takes a long time, then fails. Solution: Some networks block ssh. Try using the https URL from GitHub to download the source code.

Set Up Gradle

The frontend is run with yarn. We recommend running the backend gradle if you are only doing frontend development.

  1. Set up Gradle
    1. This step is not needed unless want to use other settings and passwords than the default ones (see build.gradle for the defaults). You can define custom settings in a local settings file as follows:
    2. Open a command line in the anet directory that was retrieved from GitHub.
    3. Create a new empty file at localSettings.gradle. (touch localSettings.gradle on linux/mac). This will be a file for all of your local settings and passwords that should not be checked into GitHub.
  2. Update the settings in application.yml for your environment. See the ANET Configuration documentation for more details on these configuration options.

Set Up Docker

Several of the components used by the back-end run as Docker containers. Make sure you have at least Docker Engine (a.k.a. Docker CE) installed. Check that you can manage Docker as a non-root user.

You may opt to manage containers using a GUI like Docker Desktop, Rancher Desktop or Podman Desktop. Be aware that several of these use their own Docker context, which you can see with:

docker context ls

Output can be something like:

NAME              DESCRIPTION                               DOCKER ENDPOINT
default *         Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
desktop-linux     Docker Desktop                            unix:///home/developer/.docker/desktop/docker.sock
rancher-desktop   Rancher Desktop moby context              unix:///home/developer/.rd/docker.sock

You can always switch contexts with:

docker context use NAME

e.g.

docker context use default

If your container management GUI uses a different endpoint, make sure that the Gradle build and your GUI agree. You should add a small configuration section to your localSettings.gradle and localTestSettings.gradle:

docker {
  url = 'unix:///home/developer/.docker/desktop/docker.sock' // Use Docker Desktop context
}

Copy the endpoint you want to use.

If you also want to be able to manage the containers from your IDE, make sure to configure the same there too. E.g. for IntelliJ this would be under Services, then Docker, then Connections From Contexts, then double-click on the context you want to connect to.

Java Backend

Development database backend configuration

  1. Only PostgreSQL can be used for your database. You can run it entirely on your local machine and develop offline.
    1. PostgreSQL
      1. This is the default, so you don't need to do anything special
      2. If you want to change any of the default database settings (see build.gradle for the defaults), you can paste them as following in your localSettings.gradle file (do it for the ones you want to change and with the correct values): 
        run.environment("ANET_DB_USERNAME","username")
run.environment("ANET_DB_PASSWORD", "password")
run.environment("ANET_DB_SERVER", "db server hostname")
run.environment("ANET_DB_NAME","database name")

Setup development database

  1. Create the DB Docker container and the initial database: ./gradlew dockerCreateDB
  2. Start the DB Docker container: ./gradlew dockerStartDB
  3. Run ./gradlew run to download all dependencies (including client dependencies like nodejs and yarn), build the project, and start the server. When starting the server, database migrations are run automatically, and the initial data is loaded.
    1. The database schema is stored in src/main/resources/migrations.xml.
    2. The initial data is stored in insertBaseData-psql.sql.
    3. If for some reason you want to re-load the initial data, run: ./gradlew dbLoad.

The Base Data Set

Provided with the ANET source code is the file insertBaseData-psql.sql. This file contains a series of raw SQL commands that insert some sample data into the database that is both required in order to pass all the unit tests, and also helpful for quickly developing and testing new features. The Base Data Set includes a set of fake users, organizations, locations, and reports. Here are some of the accounts that you can use to log in and test with:

User username organization position role
Erin Erinson erin EF2.2 EF2.2 Advisor D Regular person who can also approve their own reports
Rebecca Beccabon rebecca EF2.2 EF2.2 Final Reviewer Superuser
Arthur Dmin arthur ANET Admins ANET Administrator Administrator
Jack Jackson jack EF2.1 EF2.1 Advisor B Regular person
Henry Henderson henry EF2.1 EF2.1 Superuser Superuser
Steve Steveson - MoD Cost Adder Regular person
Ihave Noposition nopos - - Regular person

To log in as one of the base data users, when prompted for a username and password, enter their username as both the username and the password. See Users defined locally in the realm for other possible users.

Developing

  1. When starting the server, database migrations are run automatically. For background info on some of these Liquibase commands, see: https://www.dropwizard.io/en/latest/manual/migrations.html

    1. You may need to occasionally destroy, re-migrate, and re-seed your database if it has fallen too far out of sync; you can do this with env ANET_DB_DROP=true ./gradlew run -- BE CAREFUL, this will drop and re-populate your database unconditionally!

  2. Make sure the Keycloak authentication server is started (in a Docker container) in your local development environment: ./gradlew dockerConfigureKeycloak dockerStartKeycloak

  3. Run ./gradlew run to run the server via Gradle

    1. If you have set smtp: disabled to true in application.yml, you're good to go; otherwise, you can start an SMTP server (in a Docker container) in your local development environment: ./gradlew dockerCreateFakeSmtpServer dockerStartFakeSmtpServer
    2. The following output indicates that the server is ready: 
      INFO  [2017-02-10 16:44:59,902] org.eclipse.jetty.server.Server: Started @4098ms
> Building 75% > :run

  4. Go to http://localhost:8080/ in your browser.

    1. When prompted for credentials:
      • Username: erin
      • Password: same as username
    2. You will get an error about a missing index.ftl file; this is expected and means the backend server is working. The error looks like: 
      ERROR [2017-02-10 16:49:33,967] javax.ws.rs.ext.MessageBodyWriter: Template Error
! freemarker.template.TemplateNotFoundException: Template not found for name "/views/index.ftl".

    The web page will say Template Error

  5. If you want to see the app running, continue to the React Frontend instructions.

Testing

Initial Setup Test Database

First configure the database backend. The following instructions initialize a database for testing purposes, within the database container. Use the -PtestEnv property to access the test environment settings in gradle.

  1. Create the PostgreSQL Docker container and test database ./gradlew -PtestEnv dockerCreateDB
  2. Start the PostgreSQL Docker container: ./gradlew -PtestEnv dockerStartDB
  3. Run ./gradlew -PtestEnv build to download all dependencies and build the project.

Override Default Gradle Settings

Override the default gradle settings if you want to run your tests on a different database:

  1. Open a command line in the anet directory that was retrieved from GitHub.
  2. Create a new empty file at localTestSettings.gradle. (touch localTestSettings.gradle on linux/mac). This will be a file for all of your local test settings and passwords that should not be checked into GitHub.

Server side tests

  1. The tests always start with a clean test-database
  2. Make sure the Keycloak authentication server is started (in a Docker container) in your local development environment: ./gradlew dockerConfigureKeycloak dockerStartKeycloak
  3. Start a test SMTP server (in a Docker container) in your local development environment: ./gradlew -PtestEnv dockerCreateFakeSmtpServer dockerStartFakeSmtpServer
  4. Run the server side tests with a clean build: ./gradlew -PtestEnv cleanTest test

Note that the server-side tests use the GraphQL Java Generator to generate Java test classes from the GraphQL schema. If you have changed the schema, you need to update it before you can run the tests (especially if you want to test your schema changes). Since the schema is derived from the GraphQL service endpoint, you need to take the following steps for updating it:

  1. Start the server: ./gradlew run
  2. Generate the schema: ./gradlew yarn_run_generate-graphql-schema
  3. Check the generated schema, e.g.: git diff src/test/resources/anet.graphql

If you have updated the generated schema and need to update the generated Java classes, you can run:

  1. ./gradlew generateClientCode to re-generate the Java sources
  2. ./gradlew compileTestJava to re-generate the Java sources and compile them into Java class files

Client-side tests

How the client-side tests work

Our tests use selenium to simulate interacting with the app like a user. To do this, we need to connect a browser to the JavaScript tests. We do that via a driver. This driver can either run the tests locally on your system, or remotely via BrowserStack.

The tests are reliant on the data looking pretty similar to what you'd get after a fresh run of insertBaseData-psql.sql. If the tests crash and do not complete, they could leave the data set in a state which would cause future test runs to fail. Make sure you start with a clean test-database.

Prerequisites

  1. The tests always start with a clean test-database
  2. Start a test SMTP server (in a Docker container) in your local development environment: ./gradlew -PtestEnv dockerCreateFakeSmtpServer dockerStartFakeSmtpServer
  3. In order to run the client-side tests you must start a server using the test-database: ./gradlew -PtestEnv run
  4. Optionally, make sure you have the proper Node.js and Yarn in your path (see the React Frontend instructions).

Run ./gradlew yarn_run_lint:fix to automatically fix some kinds of lint errors. Run ./gradlew yarn_run_prettier:format to reformat the JavaScript code according to our style guide.

Client-side testing locally

To run the tests locally, make sure you have the server using the test-database running as above.

  1. Run the client side E2E tests against the test database: ./gradlew yarn_run_test-e2e
  2. Run the client side wdio tests against the test database: ./gradlew yarn_run_test-wdio
  3. Run the client side jest tests against the test database: ./gradlew yarn_run_test-jest
  4. Or run all client side tests against the test database: ./gradlew yarn_run_test-all

To run the tests locally, by having chromedriver as an npm dependency, we automatically have access to run in Chrome. To use Firefox instead, see geckodriver.

When writing browser tests, remember that when you take an action, you need to give the browser time to update in response before you start making assertions. Use the driver.wait method to do this.

If the tests are failing, and you don't know why, run them with env var DEBUG_LOG=true:

DEBUG_LOG=true ./gradlew yarn_run_test-e2e

You can also insert the following into your code to make the browser pause, allowing you to investigate what is currently happening:

await t.context.waitForever()

In rare circumstances, when using Chrome, the tests will hang on the data:, URL. I don't know why this is. If you re-run the test, you should not see the issue a second time.

Client-side testing remotely

To run the tests remotely, you need a username and access key for BrowserStack.

Look up your settings and put them in client/config/default.json:

{
  "browserstack_user": "myusername123",
  "browserstack_key": "mYbRoWsErStAcKkEy"
}

If you want step-by-step screenshots from your tests (Visual Logs on BrowserStack) you can also add:

  "browserstack_debug": "true"

to your default.json.

Note that both the E2E and the wdio tests will automatically start (and stop) BrowserStackLocal during the tests.

When all is set up, run the remote tests:

  1. Configure scripts with TEST_ENV environment variable for remote testing: 
    export TEST_ENV=remote
  2. Run;
    1. the client side E2E tests: 
      ./gradlew yarn_run_test-e2e
    2. the client side wdio tests: 
      ./gradlew yarn_run_test-wdio
    3. the client side wdio-ie tests: 
      ./gradlew yarn_run_test-wdio-ie
      About IE tests: Internet Explorer is not fully supported by ANET and all features are NOT guaranteed to work with IE. For that reason, a warning banner is displayed when IE detected. test-wdio-ie runs tests for this scenario and these tests run only on remote testing. When testing locally, they gracefully abort.
    4. all client side tests: 
      ./gradlew yarn_run_test-all
  3. You can view the progress and results on BrowserStack.

Simulator

ANET has a simulator that can exercise of the functions. It is located in 'client/test/sim'. It works by interfacing with ANET through GraphQL queries. The simulator executes stories which are assigned to different user types and have different probabilities.

The simulator can be started by running './gradlew yarn_run_sim'.

React Frontend

All the frontend code is in the client/ directory.

Initial Setup

  1. You can run all client-side scripts via Yarn through Gradle. Otherwise, make sure you have the proper Node.js and Yarn in your path; example: 
    export YARN_HOME=<anet_root_path>/.gradle/yarn/yarn-v1.22.15
export NODEJS_HOME=<anet_root_path>/.gradle/nodejs/node-v14.18.1-linux-x64
export PATH="$YARN_HOME/bin:$NODEJS_HOME/bin:$PATH"
cd client/
    Note: Node.js and Yarn versions might have changed in the meanwhile, check inside <anet_root_path>/.gradle/nodejs/ and <anet_root_path>/.gradle/yarn/ for which versions are being used and change the path accordingly.
  2. Run the server: ./gradlew yarn_run_start
  3. Go to http://localhost:3000/ in your browser.
    1. When prompted for credentials:
      • Username: erin
      • Password: same as username

NB: You only need node.js and the npm dependencies for developing. When we deploy for production, everything is compiled to static files. No javascript dependencies are necessary on the server.

Development Mode

To simulate a "new user" in development mode, create a new user in the Keycloak realm, then log on to ANET as that user. This will start the new user workflow (onboarding). Note: if you enter an unknown username, ANET will reject you.