Merge pull request #111 from databio/dev

v1.4.0 release
databio · Apr 14, 2022 · 21bd8ff · 21bd8ff
2 parents 2ddd6d6 + e34c480
commit 21bd8ff
Show file tree

Hide file tree

Showing 17 changed files with 240 additions and 253 deletions.
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -0,0 +1,70 @@
+name: Docker Image CI
+
+on:
+  push:
+    branches: [ master, dev ]
+  pull_request:
+    branches: [ master ]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Git checkout
+        uses: actions/checkout@v2
+
+      - name: Build DEV
+        if: github.ref == 'refs/heads/dev'
+        run: |
+          docker build -t ghcr.io/databio/lolaweb:dev .
+
+      - name: Build MASTER
+        if: github.ref == 'refs/heads/master'
+        run: |
+          docker build -t ghcr.io/databio/lolaweb:latest .
+
+      - name: Login to GHCR
+        uses: docker/login-action@v1
+        with:
+          registry: ghcr.io
+          username: ${{ secrets.GHCR_USERNAME }}
+          password: ${{ secrets.GHCR_PAT }}
+
+      - name: Push :latest to GHCR
+        id: push_latest_ghcr
+        if: github.ref == 'refs/heads/master'
+        run: docker push ghcr.io/databio/lolaweb:latest
+
+      - name: Push :dev to GHCR
+        id: push_dev_ghcr
+        if: github.ref == 'refs/heads/dev'
+        run: docker push ghcr.io/databio/lolaweb:dev
+
+      - name: Image digest
+        run: echo ${{ steps.push_ghcr.outputs.digest }}
+
+  deploy:
+    runs-on: ubuntu-latest
+    needs: build
+    steps:
+      - 
+        name: Set up AWS credentials
+        uses: aws-actions/configure-aws-credentials@v1
+        with:
+          aws-access-key-id: ${{ secrets.LOLA_AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.LOLA_AWS_SECRET_ACCESS_KEY }}
+          aws-region: us-east-1
+      - 
+        name: Send SQS message
+        if: github.ref == 'refs/heads/master'
+        uses: isbang/[email protected]
+        with:
+          sqs-url: https://queue.amazonaws.com/474683445819/lola-updates
+          message: 'master'
+      - 
+        name: Send SQS message
+        if: github.ref == 'refs/heads/dev'
+        uses: isbang/[email protected]
+        with:
+          sqs-url: https://queue.amazonaws.com/474683445819/lola-updates
+          message: 'dev'
diff --git a/.travis.yml b/.travis.yml
diff --git a/docker/Dockerfile → Dockerfile b/docker/Dockerfile → Dockerfile
@@ -1,14 +1,14 @@
-FROM databio/shinybase:latest
+FROM ghcr.io/databio/shinybase:latest
 
-MAINTAINER VP Nagraj "[email protected]"
+MAINTAINER VP Nagraj "[email protected]"
 
 # move conf files
-COPY shiny-server.conf  /etc/shiny-server/shiny-server.conf
-COPY shiny-server.sh /usr/bin/shiny-server.sh
+COPY ./shiny-server.conf  /etc/shiny-server/shiny-server.conf
+COPY ./shiny-server.sh /usr/bin/shiny-server.sh
 
 # get the code
 WORKDIR /srv/shiny-server
-RUN git clone https://github.com/databio/LOLAweb.git
+COPY apps/LOLAweb /srv/shiny-server/LOLAweb/apps/LOLAweb
 
 # add dir for cache
 RUN mkdir LOLAweb/cache
@@ -18,5 +18,5 @@ RUN chown -R shiny:shiny LOLAweb/cache
 RUN mkdir LOLAweb/apps/LOLAweb/plots
 RUN chown -R shiny:shiny LOLAweb/apps/LOLAweb/plots
 
-# run the server setup script
+## run the server setup script
 CMD ["/usr/bin/shiny-server.sh"]
diff --git a/README.md b/README.md
@@ -1,12 +1,132 @@
 # LOLAweb
+
+[![Docker Image CI](https://github.com/databio/LOLAweb/actions/workflows/build.yml/badge.svg?branch=dev)](https://github.com/databio/LOLAweb/actions/workflows/build.yml)
+
 LOLAweb is a web server and interactive results viewer for enrichment of overlap between a query region set (a bed file) and a database of region sets. It provides an interactive result explorer to visualize the highest ranked enrichments from the database. You can access the web server at <http://lolaweb.databio.org>.
 
-This repository contains two components: 1) the shiny [app source code](apps/LOLAweb/) and 2) [Docker implementation](docker/) for LOLAweb. 
+This repository contain the shiny [app source code](apps/LOLAweb/) and Docker implementation for LOLAweb. 
 
 ## Shiny app
 
 LOLAweb is implemented as an interactive Shiny app. You can run this app locally by following the [instructions in the appfolder](apps/LOLAweb/).
 
 ## Docker
 
-We have also produced a Dockerfile and published an image on Docker Hub (`databio/lolaweb`). Learn how to run LOLAweb locally in a docker container by reading the [instructions in the docker folder](docker/). 
+The `ghcr.io/databio/lolaweb` container is based on the `ghcr.io/databio/shinybase` container, which you can find in its [GitHub repository](https://github.com/databio/shinyBase) or in the [GitHub Container Registry](https://github.com/databio/shinyBase/pkgs/container/shinybase).
+
+### `build` the container image yourself
+
+1. Clone this repository
+2. Build locally using Docker. Run this command from the same directory as the `Dockerfile`.
+
+```docker build --no-cache -t lolaweb .```
+
+
+### Or `pull` the container image:
+
+```docker pull ghcr.io/databio/lolaweb```
+
+The container image itself is hosted in the GitHub Container Registry: https://github.com/databio/LOLAweb/pkgs/container/lolaweb
+
+### Container volumes and reference data
+
+LOLAweb needs access to a few folders where it can store results or logs, or access necessary files like the database. To handle this, we've set up the app to look for two shell environment variables:
+
+* **$LWREF**, for LOLAweb reference data, which may be read-only
+* **$LWLOCAL**, where local results can be written.
+
+To run the LOLAweb container (locally or on a server), you need to set these environment variables (for example, in your `.bashrc` or `.profile` file. These variables will be injected into the container when it is run.
+
+For example, set these variables to two different paths if you like. Or if you keep all five subfolders together in the same path, set these variables to the same value.
+
+```
+# Example locations. Set to match your environment
+export LWREF='/home/subdir/lola/'
+export LWLOCAL='/var/things/loladata/'
+```
+
+LOLAweb will look at the value in `$LWREF` for the reference data. This folder should have subfolders called `databases`, `universes`, and `examples`. In each of these subfolders are another layer of subfolders for genome assemblies
+
+LOLAweb looks for `$LWLOCAL` to have two subfolders: `cache`, and `shinylogs`. This is where the app will write results and log files. If running LOLAweb on a server, be sure these directories are writeable by the Docker process.
+
+The following instructions demonstrat how to download and configure the LOLAweb data directories for a minimal example using `hg19` reference data:
+
+```
+## assign env vars for data path
+## NOTE: must include trailing /
+LWLOCAL="/path/to/local/data/"
+LWREF="/path/to/reference/data/"
+
+## change reference data dir
+cd $LWREF
+
+## create dir for databases
+mkdir -p databases
+## create examples and universe dir
+## NOTE: these must include subdirs named corresponding to appropriate ref genome
+mkdir -p examples/hg19
+mkdir -p universes/hg19
+
+## download example universe and user set
+curl http://cloud.databio.org.s3.amazonaws.com/vignettes/lola_vignette_data_150505.tgz | tar xvz
+
+## move example universe and user set files to hg19 dir
+mv lola_vignette_data/activeDHS_universe.bed universes/hg19/.
+mv lola_vignette_data/setB_100.bed examples/hg19/.
+
+## clean up
+rm -rf lola_vignette_data
+
+## download databases
+curl http://cloud.databio.org.s3.amazonaws.com/regiondb/LOLACoreCaches_170206.tgz | tar xvz
+curl http://cloud.databio.org.s3.amazonaws.com/regiondb/LOLAExtCaches_170206.tgz | tar xvz
+
+## move databases to appropriate spots
+mv scratch/ns5bc/resources/regions/LOLACore databases/Core
+mv scratch/ns5bc/resources/regions/LOLAExt databases/Extended
+
+## clean up
+rm -rf scratch
+
+## change ot local data dir
+cd $LWLOCAL
+
+## create placeholder dirs for cache and shinylog
+mkdir -p cache
+mkdir -p shinylog
+```
+
+### Run the LOLAweb container locally with reference data:
+
+```
+## run the docker image
+## NOTE: this run command uses image pulled from ghcr.io/databio/lolaweb
+docker run -d \
+  -p 80:80 \
+  -e LWREF=$LWREF \
+  -e LWLOCAL=$LWLOCAL \
+  --volume ${LWLOCAL}:${LWLOCAL} \
+  --volume ${LWREF}:${LWREF} \
+  --volume ${LWLOCAL}/shinylog:/var/log/shiny-server \
+  ghcr.io/databio/lolaweb
+```
+
+Open a browser to:
+```
+http://localhost
+```
+
+### Running a `dev` container
+
+You could also run the `dev` version of the container by pulling `ghcr.io/databio/lolaweb:dev`. This will retrieve the dev tagged image from the GitHub Container Registry. Just add `:dev` to the container name at the end of the `docker run` command above.
+
+### Running multiple LOLAweb containers simultaneously with Docker Swarm
+
+For the typical use case of an individual user, a single running container will suffice. But if you need to set up an enterprise-level LOLAweb server that can handle concurrent users, we've also made that easy by using Docker Swarm. This is how we run the main LOLAweb servers, and you could do the same thing if you want your own local implementation. Docker Swarm is a technique for running multiple instances of the same container. [Read more](swarm/README.md) about how to set up your own swarm.
+
+### Troubleshooting
+
+The LOLAweb Docker implementation includes a mechanism to write Shiny Server logs to `$LWLOCAL/shinylog`. These log files may be useful when troubleshooting problems with running LOLAweb via Docker. They include errors with R processing as well as information as to whether the Shiny Server process was killed due to resource limitations (i.e., not enough RAM allocated to Docker daemon).
+
+For additional support with the LOLAweb Docker implementation, please file a [GitHub issue](https://github.com/databio/LOLAweb/issues).
+
diff --git a/apps/LOLAweb/README.md b/apps/LOLAweb/README.md
@@ -25,39 +25,8 @@ source("https://bioconductor.org/biocLite.R")
 biocLite(c("LOLA", "GenomicRanges"))
 ```
 
-You'll also need underlying data that is not available in this repository in order to establish the universes, example user sets and reference genome directories. Run the following from within the root of this folder to download the data and create the `universes/`, `userSets/`, `reference/` and `cache/` directories respectively:
+You'll also need underlying data that is not available in this repository in order to establish the universes, example user sets and reference genome directories. Detailed guidance for how to download and organize reference data is available in the [LOLAweb project README](https://github.com/databio/lolaweb/#readme).
 
-```
-# create universes dir
-mkdir universes
-mkdir userSets
-
-# example universe and user set
-curl http://cloud.databio.org.s3.amazonaws.com/vignettes/lola_vignette_data_150505.tgz | tar xvz
-
-mv lola_vignette_data/activeDHS_universe.bed universes/.
-mv lola_vignette_data/setB_100.bed userSets/.
-
-rm -rf lola_vignette_data
-```
+Note that one additional step for running the app locally will be to either 1) pass the `$LWREF` and `$LWLOCAL` environment variables to your R session or 2) overwite the definition of `localDir` and `refDir` in `misc.R` with the respective paths on your machine.
 
-```
-# create reference dir
-mkdir reference
-
-# core
-curl http://cloud.databio.org.s3.amazonaws.com/regiondb/LOLACoreCaches_170206.tgz | tar xvz
-mv scratch/ns5bc/resources/regions/LOLACore reference/Core
-rm -rf scratch
-
-# extended
-curl http://cloud.databio.org.s3.amazonaws.com/regiondb/LOLAExtCaches_170206.tgz | tar xvz
-mv scratch/ns5bc/resources/regions/LOLAExt reference/Extended
-rm -rf scratch
-```
-
-```
-# create cache dir
-mkdir cache
-```
-With all of the above installed you can now launch the app with `shiny::runApp()` from within R at the root of this directory.
+With dependencies installed and data configuration complete you can now launch the app with `shiny::runApp()` from within R at the root of this directory.
diff --git a/apps/LOLAweb/about.md b/apps/LOLAweb/about.md
@@ -340,8 +340,8 @@ Here are some links to other resources related to LOLA and LOLAweb:
 
 - [LOLAweb issue tracker](https://github.com/databio/LOLAweb/issues) - please use this for support requests for LOLAweb.
 - [LOLAweb GitHub source code](https://github.com/databio/LOLAweb)
-- [Dockerhub docker image](https://hub.docker.com/r/databio/lolaweb/)
-- Manuscript (pending)
+- [Docker image](https://hub.docker.com/r/databio/lolaweb/)
+- [Manuscript](https://doi.org/10.1093/nar/gky464)
 - [LOLA R package at Bioconductor](http://bioconductor.org/packages/LOLA/)
 - [LOLA R package source code at GitHub](https://github.com/nsheff/LOLA)
 - [LOLA R package documentation](http://code.databio.org/LOLA)