-
Notifications
You must be signed in to change notification settings - Fork 5
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Atlas single cell web cli * Fix cli test and remove more unnecessary things. * Wrong deps * CreateExperiment cli and handle failed accessions * Log success with accessKeyGenerated * Explains that create also updates. * A bit of refactoring * Review suggestions * Remove unneeded tests that were just dragged. * Pjn to web core with atlas-bioentities pointer * Be able to give only private accessions if needed. * Missing deps compared to web-bulk cli PR * Most closely resemble PR ebi-gene-expression-group/atlas-web-bulk#88 * Latest web core and attempt profile !Cli for currently offending part * More profile additions that end up producing more complicated errors. * Remove any Spring dependencies from parent plug-in We want app to include Spring framework and CLI to include Spring Boot * Don’t include web config (i.e. ServletContext-dependent classes) in the CLI module * Avoid null pointer exception by makings sure collections are not null. Co-authored-by: Alfonso Muñoz-Pomer Fuentes <[email protected]>
- Loading branch information
1 parent
b23017d
commit fa7a3a9
Showing
22 changed files
with
423 additions
and
20 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
name: Build atlas-web-single-cell cli | ||
on: | ||
pull_request | ||
|
||
jobs: | ||
compile_cli: | ||
name: compile_cli | ||
strategy: | ||
fail-fast: false # if one of the jobs in the matrix expansion fails, the rest of the jobs will be cancelled | ||
matrix: | ||
os: ["ubuntu-latest"] | ||
jdk: [11] | ||
runs-on: ${{ matrix.os }} | ||
env: | ||
JDK_VERSION: ${{ matrix.jdk }} | ||
steps: | ||
- uses: actions/checkout@v2 | ||
with: | ||
submodules: true | ||
- uses: actions/setup-java@v1 | ||
with: | ||
java-version: ${{ matrix.jdk }} | ||
- name: Compile cli | ||
run: ./gradlew --no-daemon :cli:bootJar | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Submodule atlas-web-core
updated
19 files
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,72 @@ | ||
# Expression Atlas CLI Bulk | ||
A minimal Spring Boot wrapper to run Single Cell Expression Atlas tasks from the command line. | ||
|
||
## Requirements | ||
- Java 11 | ||
- Expression Atlas environment (PostgreSQL server; SolrCloud cluster; bioentity annotation and experiment files) | ||
|
||
## Usage | ||
There are two main ways to run the application: as an executable JAR or via Gradle. The latter is recommended on | ||
development environments and Java is preferred in production environments. Be aware that any changes made to the | ||
properties file won’t take effect unless you rebuild the JAR file. | ||
|
||
### Gradle | ||
```bash | ||
./gradlew :cli:bootRun --args="<task-name> <options>" | ||
``` | ||
|
||
### Executable JAR | ||
Build the JAR file: | ||
```bash | ||
./gradlew :cli:bootJar | ||
``` | ||
|
||
Then run it with Java: | ||
```bash | ||
java -jar ./cli/build/libs/atlas-cli-sc.jar <task-name> <options> | ||
``` | ||
|
||
## Configuration | ||
Configuration variables are set with `-Dproperty=value` if you run the application via `java -jar ...`, or by adding | ||
`-Pproperty=value` to the Gradle task (in the tables below: Java property name, and Gradle propery name, respectively). | ||
|
||
**IMPORTANT**: At the very least you will need to set the environment variables described in the Default value columns | ||
to run/compile the application with Gradle. However, notice that the `-D` arguments will override whatever was set at | ||
compile time, so if you forget or your environment changes, you don’t need to recompile. | ||
|
||
### Expression Atlas file options: `configuration.properties` | ||
| Java property name | Gradle property name | Default value | | ||
|-----------------------------|---------------------------|--------------------------| | ||
| `data.files.location` | `dataFilesLocation` | `${ATLAS_DATA_PATH}` | | ||
| `experiment.files.location` | `experimentFilesLocation` | `${ATLAS_DATA_PATH}/gxa` | | ||
|
||
### Expression Atlas database options: `jdbc.properties` | ||
| Java Property name | Gradle property name | Default value | | ||
|--------------------|----------------------|---------------------------------------------------------------------| | ||
| `jdbc.url` | `jdbcUrl` | `jdbc:postgresql://${ATLAS_POSTGRES_HOST}:5432/${ATLAS_POSTGRES_DB` | | ||
| `jdbc.username` | `jdbcUsername` | `${ATLAS_POSTGRES_USER}` | | ||
| `jdbc.password` | `jdbcPassword` | `${ATLAS_POSTRES_PASSWORD}` | | ||
|
||
### Expression Atlas Solr options: `solr.properties` | ||
| Java property name | Gradle property name | Default value | | ||
|--------------------|----------------------|----------------------| | ||
| `zk.host` | `zkHost` | `${ATLAS_ZK_HOST}` | | ||
| `zk.port` | `zkPort` | `2181` | | ||
| `solr.host` | `solrHost` | `${ATLAS_SOLR_HOST}` | | ||
| `solr.port` | `solrPort` | `8983` | | ||
|
||
## Tasks | ||
Run without any arguments to get a list of available tasks: | ||
``` | ||
Usage: <main class> [COMMAND] | ||
Commands: | ||
update-experiment-design Updates the experiment designs of an accession or a group of accessions. | ||
``` | ||
|
||
Pass the name of a task to obtain a detailed description of available options: | ||
```bash | ||
$ java -jar ./cli/build/libs/atlas-cli-bulk.jar update-experiment-design | ||
... | ||
|
||
``` | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
plugins { | ||
id 'atlas-web-app.java-conventions' | ||
id 'org.springframework.boot' version '2.4.2' | ||
id 'io.spring.dependency-management' version '1.0.11.RELEASE' | ||
} | ||
|
||
version = '1.0.0' | ||
|
||
bootJar { | ||
archiveFileName = 'atlas-cli-sc.jar' | ||
} | ||
|
||
dependencies { | ||
implementation project(':atlas-web-core') | ||
implementation project(':app') | ||
|
||
runtimeOnly 'javax.servlet.jsp:jsp-api:2.2' | ||
|
||
implementation 'info.picocli:picocli:4.6.1' | ||
implementation 'info.picocli:picocli-spring-boot-starter:4.6.1' | ||
|
||
testImplementation 'org.springframework.boot:spring-boot-starter-test' | ||
} | ||
|
||
processResources { | ||
filesMatching('configuration.properties') { | ||
expand( | ||
dataFilesLocation: project.hasProperty('dataFilesLocation') ? dataFilesLocation : "${System.env.ATLAS_DATA_PATH}", | ||
experimentFilesLocation: project.hasProperty('experimentFilesLocation') ? experimentFilesLocation : "${System.env.ATLAS_DATA_PATH}/scxa", | ||
) | ||
} | ||
|
||
filesMatching('jdbc.properties') { | ||
expand( | ||
jdbcUrl: project.hasProperty('jdbcUrl') ? jdbcUrl : "jdbc:postgresql://${System.env.ATLAS_POSTGRES_HOST}:5432/${System.env.ATLAS_POSTGRES_DB}", | ||
jdbcUsername: project.hasProperty('jdbcUsername') ? jdbcUsername : "${System.env.ATLAS_POSTGRES_USER}", | ||
jdbcPassword: project.hasProperty('jdbcPassword') ? jdbcPassword : "${System.env.ATLAS_POSTGRES_PASSWORD}" | ||
) | ||
} | ||
|
||
filesMatching('solr.properties') { | ||
expand( | ||
zkHost: project.hasProperty('zkHost') ? zkHost : "${System.env.ATLAS_ZK_HOST}", | ||
zkPort: project.hasProperty('zkPort') ? zkPort : '2181', | ||
solrHost: project.hasProperty('solrHost') ? solrHost : "${System.env.ATLAS_SOLR_HOST}", | ||
solrPort: project.hasProperty('solrPort') ? solrPort : '8983' | ||
) | ||
} | ||
} |
34 changes: 34 additions & 0 deletions
34
cli/src/main/java/uk/ac/ebi/atlas/cli/AbstractPerAccessionCommand.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
package uk.ac.ebi.atlas.cli; | ||
|
||
import picocli.CommandLine; | ||
import uk.ac.ebi.atlas.cli.utils.AccessionsWriter; | ||
|
||
import java.util.ArrayList; | ||
import java.util.Collection; | ||
import java.util.List; | ||
import java.util.logging.Logger; | ||
|
||
public abstract class AbstractPerAccessionCommand { | ||
|
||
private static final Logger LOGGER = Logger.getLogger(AbstractPerAccessionCommand.class.getName());; | ||
|
||
@CommandLine.Option(names = {"-f", "--failed-accessions-path"}, description = "File to write failed accessions to.") | ||
protected String failedOutputPath; | ||
|
||
@CommandLine.Option(names = {"-e", "--experiment"}, split = ",", description = "one or more experiment accessions") | ||
protected List<String> experimentAccessions = new ArrayList<>(); | ||
|
||
protected int handleFailedAccessions(Collection<String> failedAccessions) { | ||
var status = 0; | ||
if (!failedAccessions.isEmpty()) { | ||
status = 1; | ||
LOGGER.warning(String.format("%s experiments failed", failedAccessions.size())); | ||
LOGGER.info(String.format("Re-run with the following arguments to re-try failed accessions: %s", String.join(",", failedAccessions))); | ||
if (failedOutputPath != null) { | ||
AccessionsWriter writer = new AccessionsWriter(failedOutputPath, failedAccessions); | ||
writer.write(); | ||
} | ||
} | ||
return status; | ||
} | ||
} |
35 changes: 35 additions & 0 deletions
35
cli/src/main/java/uk/ac/ebi/atlas/cli/ExpressionAtlasCliApplication.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
package uk.ac.ebi.atlas.cli; | ||
|
||
import org.springframework.boot.CommandLineRunner; | ||
import org.springframework.boot.ExitCodeGenerator; | ||
import org.springframework.boot.SpringApplication; | ||
import org.springframework.boot.autoconfigure.SpringBootApplication; | ||
import picocli.CommandLine; | ||
|
||
@SpringBootApplication(scanBasePackages = "uk.ac.ebi.atlas") | ||
public class ExpressionAtlasCliApplication implements CommandLineRunner, ExitCodeGenerator { | ||
private final CommandLine.IFactory factory; | ||
private final ExpressionAtlasCliCommand expressionAtlasCliCommand; | ||
|
||
private int exitCode; | ||
|
||
public ExpressionAtlasCliApplication(CommandLine.IFactory factory, | ||
ExpressionAtlasCliCommand expressionAtlasCliCommand) { | ||
this.factory = factory; | ||
this.expressionAtlasCliCommand = expressionAtlasCliCommand; | ||
} | ||
|
||
public static void main(String[] args) { | ||
System.exit(SpringApplication.exit(SpringApplication.run(ExpressionAtlasCliApplication.class, args))); | ||
} | ||
|
||
@Override | ||
public void run(String... args) { | ||
exitCode = new CommandLine(expressionAtlasCliCommand, factory).execute(args); | ||
} | ||
|
||
@Override | ||
public int getExitCode() { | ||
return exitCode; | ||
} | ||
} |
14 changes: 14 additions & 0 deletions
14
cli/src/main/java/uk/ac/ebi/atlas/cli/ExpressionAtlasCliCommand.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
package uk.ac.ebi.atlas.cli; | ||
|
||
import org.springframework.stereotype.Component; | ||
import picocli.CommandLine.Command; | ||
import uk.ac.ebi.atlas.cli.experiment.CreateUpdateExperimentCommand; | ||
import uk.ac.ebi.atlas.cli.experiment.ExperimentDesignCommand; | ||
|
||
@Command(subcommands = { | ||
ExperimentDesignCommand.class, | ||
CreateUpdateExperimentCommand.class | ||
}) | ||
@Component | ||
public class ExpressionAtlasCliCommand { | ||
} |
Oops, something went wrong.