[Bugfix] Update codebase to latest atlas-web-core module (#180)

* Fix test runners

* Upgrade Mockito to 5.2.0

* Add ontology related files to the local Docker env

* Remove large experiments from local env

* Implement ExperimentCellCountDao interface

* Remove experiment design tab from the experiment page

* Remove unused stubbing

* Add null check

* Adjust test to code changes that happened in February 2022

Unfortunately test code was not adjusted when the `download` endpoint has been changed in February 2022 and probably the tests were not executed before the PR as merged. :-(

* Add back E-MTAB-5214 to the experiment's fixtures

* Add E-MTAB-5555 as a private experiment

* Fix experiment fixture for GxaExperimentCrudIT

* Remove E-MTAB-5200 from the small-experiment-fixture.sql as we don't have it local

* Fix DASFeaturesControllerWIT's featuresURLGeneWithoutExpressions method

* Add temporary folders to .gitignore

* Fix port related issue for gradle task

* New section - How to add a private experiment into our local env

* Add known build issue to the README.md

* Improve the README.md

* Fix anchor in the README

* Change property source to git.properties

git.properties is being generated by the plugin

* Define folder name to git.properties

* Remove the definition of the folder for git.properties

* Change the path and name of the generated git props file

* add --recurse-submodules to index-bioentities git repo to pull submodules and add specific branch name to pull only that branch instead of dev

* Remove SOLR_HOSTS and ZK_HOSTS to load jsonl files into a collection

* Fix solr tasks to create bioentities and bulk-analytics collections in 2 steps

* Update the README with the 2 SOLR sub-steps

* Remove `-r` flag from bulk-analytics collection generation

---------

Co-authored-by: Upendra Kumbham <[email protected]>

.gitignore

@@ -3,9 +3,6 @@
 # Compiled class file
 *.class
 
-# Log file
-*.log
-
 # BlueJ files
 *.ctxt
 
@@ -143,3 +140,11 @@ _idea
 .DS_Store
 .module-cache
 
+# Webpack bundles
+app/src/main/webapp/resources/js-bundles
+
+# Temporary folders
+tmp
+temp
+_tmp
+_temp

README.md

@@ -78,6 +78,24 @@ contents of the source directories change. This is especially useful when experi
 or the bioentity properties directory is updated after a release of  Ensembl, WormBase ParaSite, Reactome, Gene
 Ontoloy, Plant Ontology or InterPro.
 
+### How to add a private experiment bundle
+
+Private experiments are not available to download from our FTP site. You can download them from the `codon-cluster` by using the following steps:
+
+1. These steps should be done before the `PostGreSQL` and `Solr` steps. 
+2. After logged in to the `codon-cluster` check if the experiment bundle can be found under this path:
+```/nfs/production/irene/ma/experiments/```.
+2. If it is there, then go to the folder on your local computer where you would like to download the bundle.
+3. Download it by this command:
+```scp -r codon-login:/nfs/production/irene/ma/experiments/<EXPERIMENT_ACCESSION_ID> .```.
+4. Create a temp container with mounting the already existing data volume for our local experiments: 
+````docker container create --name expVol -v gxa_atlas-data-exp:/atlas-data/exp ubuntu:jammy````
+5. Copy the file bundles of the downloaded private experiment into the volume: ```docker cp <EXPERIMENT_ACCESSION_ID> expVol:/atlas-data/exp/magetab/```
+6. Add `<EXPERIMENT_ACCESSION_ID>` into the `PRIVATE_EXP_IDS` variable. It is in to `test-data.env` file under the `docker/prepare-dev-environment` folder. If it is not there, then please create it.
+7. The experiment accession IDs in that variable should be separated by SPACE.
+
+
+
 ### PostGreSQL
 
 To create our PostGreSQL database and run the schema migrations up to the latest version please execute this script:
@@ -86,10 +104,21 @@ To create our PostGreSQL database and run the schema migrations up to the latest
 ```
 
 ### Solr
-To create the collections, their schemas and populate them, please run the following script.
+To create the collections, their schemas and populate them, please run the following scripts.
+
+Currently, this step is separated into 2 sub-steps by Solr collections. 
+There is an inconsistency in our web apps and various shell scripts - that we use together with the Data Prod Team -
+how we use the `SOLR_HOST` and `SOLR_HOSTS` variables. We need to sort this out, 
+but while it is not solved we probably have to keep this 2 sub-steps, unless we find a way to merge them.
 
+To create and populate the `bioentities` collection:
 ```bash
-./docker/prepare-dev-environment/solr/run.sh -r -l solr.log
+./docker/prepare-dev-environment/solr-bioentities/run.sh -r -l solr-bioentities.log
+```
+
+To create and populate the `bulk-analytics` collection:
+```bash
+./docker/prepare-dev-environment/solr-analytics/run.sh -l solr-analytics.log
 ```
 
 Run the script with the `-h` flag for more details.
@@ -239,6 +268,9 @@ The script `debug-single-test.sh` is a shortcut for this task. It takes the same
 ```
 
 ## Run web application
+
+Please check this first in the troubleshooting session: [Known Build Issue](#known-build-issue)
+
 The web application is compiled in two stages:
 1. Front end JavaScript packages are transpiled into “bundles” with [Webpack](https://webpack.js.org/)
 2. Bundles and back end Java code are built as a WAR file
@@ -270,9 +302,31 @@ If you don't give any flags, or you add both then the script is going to build b
 
 ## Troubleshooting
 
+### <a name="known-build-issue"></a>Known Build Issue
+
+This current version of our developer env has a bug when we build and execute the application
+with the `build-and-deploy-webapp.sh` script.
+We have a ticket to fix this in our backlog: [Update bulk with the latest webpack and its dependencies](https://github.com/ebi-gene-expression-group/atlas-web-bulk/issues/176)
+
+You can build a working WAR with the following steps:
+1. Manually build the UI:
+```bash
+./compile-front-end-packages.sh -iu
+```
+
+2. Manually build the backend:
+```bash
+./gradlew :app:war
+```
+
+3. Use this script to start up the web app on your local environment:
+```bash
+./build-and-deploy-webapp.sh -n
+```
+
 ### SolrCloud nodes shut down on macOS
 Docker for macOS sets fairly strict resource limits for all Docker containers. If your containers require e.g. more
-memory you need to increase the available amount in the Docker Dashboard. For bulk Expression Atlas, plase set Memory
+memory you need to increase the available amount in the Docker Dashboard. For bulk Expression Atlas, please set Memory
 to between 8-12 GB and disk image to 100 GB or more. Please see the screenshot below for reference:
 
 ![Screenshot-2021-02-18-at-18-27-40](https://user-images.githubusercontent.com/4425744/109644570-8ccee680-7b4d-11eb-9db0-7a29fb4d9e2b.png)

app/build.gradle

@@ -6,7 +6,7 @@ plugins {
 }
 
 gitProperties {
-    gitPropertiesName = "resources/git.properties"
+    gitPropertiesName = "resources/configuration.properties"
 }
 
 version '36.0.0'

app/src/main/java/uk/ac/ebi/atlas/experimentpage/content/ExperimentPageContentService.java

@@ -10,14 +10,12 @@
 import org.springframework.cache.annotation.Cacheable;
 import org.springframework.stereotype.Component;
 import uk.ac.ebi.atlas.commons.readers.TsvStreamer;
-import uk.ac.ebi.atlas.experimentpage.ExperimentDesignFile;
 import uk.ac.ebi.atlas.experimentpage.ExternallyAvailableContentService;
 import uk.ac.ebi.atlas.experimentpage.json.JsonBaselineExperimentController;
 import uk.ac.ebi.atlas.experimentpage.qc.MicroarrayQcFiles;
 import uk.ac.ebi.atlas.experimentpage.qc.QcReportController;
 import uk.ac.ebi.atlas.model.download.ExternallyAvailableContent;
 import uk.ac.ebi.atlas.model.experiment.Experiment;
-import uk.ac.ebi.atlas.model.experiment.ExperimentDesignTable;
 import uk.ac.ebi.atlas.model.experiment.ExperimentType;
 import uk.ac.ebi.atlas.model.experiment.sample.ReportsGeneExpression;
 import uk.ac.ebi.atlas.resource.DataFileHub;
@@ -85,12 +83,6 @@ private JsonObject experimentPageContentForExperiment(final Experiment<? extends
                                             ExternallyAvailableContent.ContentType.PLOTS))));
         }
 
-        if (dataFileHub.getExperimentFiles(experiment.getAccession()).experimentDesign.exists()) {
-            availableTabs.add(
-                    experimentDesignTab(new ExperimentDesignTable(experiment).asJson(),
-                            ExperimentDesignFile.makeUrl(experiment.getAccession(), accessKey)));
-        }
-
         availableTabs.add(
                 customContentTab(
                         "multipart",
@@ -223,11 +215,4 @@ private JsonObject heatmapTab(JsonArray groups, String geneDistributionUrl, Json
         props.add("availableDataUnits", availableDataUnits);
         return customContentTab("heatmap", "Results", props);
     }
-
-    private JsonObject experimentDesignTab(JsonObject table, String downloadUrl) {
-        JsonObject props = new JsonObject();
-        props.add("table", table);
-        props.addProperty("downloadUrl", downloadUrl);
-        return customContentTab("experiment-design", "Experiment Design", props);
-    }
 }

app/src/main/java/uk/ac/ebi/atlas/experiments/ExperimentCellCountDaoImpl.java

@@ -0,0 +1,20 @@
+package uk.ac.ebi.atlas.experiments;
+
+import org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate;
+import org.springframework.stereotype.Repository;
+import org.springframework.transaction.annotation.Transactional;
+
+@Repository
+@Transactional(transactionManager = "txManager", readOnly = true)
+public class ExperimentCellCountDaoImpl implements ExperimentCellCountDao {
+    private final NamedParameterJdbcTemplate namedParameterJdbcTemplate;
+
+    public ExperimentCellCountDaoImpl(NamedParameterJdbcTemplate namedParameterJdbcTemplate) {
+        this.namedParameterJdbcTemplate = namedParameterJdbcTemplate;
+    }
+
+    @Override
+    public Integer fetchNumberOfCellsByExperimentAccession(String experimentAccession) {
+        return 0;
+    }
+}

app/src/main/java/uk/ac/ebi/atlas/monitoring/JsonBuildInfoController.java

@@ -12,7 +12,7 @@
 import java.util.Map;
 
 @RestController
-@PropertySource("classpath:configuration.properties")
+@PropertySource("classpath:resources/configuration.properties")
 public class JsonBuildInfoController extends JsonExceptionHandlingController {
     private final ImmutableMap<String, String> buildInfo;
 

app/src/test/java/uk/ac/ebi/atlas/acceptance/rest/tests/DAS/DASFeaturesControllerWIT.java

@@ -85,7 +85,7 @@ void featuresURLGeneWithoutExpressions() throws Exception {
                 .andExpect(status().isOk())
                 .andExpect(view().name("das-features"))
                 .andExpect(model().attribute(
-                        MODEL_ATTRIBUTE_NAMES[ThreadLocalRandom.current().nextInt(2, MODEL_ATTRIBUTE_NAMES.length)],
+                        MODEL_ATTRIBUTE_NAMES[ThreadLocalRandom.current().nextInt(4, MODEL_ATTRIBUTE_NAMES.length)],
                         "Not studied or no differential expression found for this gene"));
     }
 

app/src/test/java/uk/ac/ebi/atlas/download/DownloadControllerWIT.java

@@ -46,7 +46,7 @@ void downloadModelHaveFtpFileInfo() throws Exception {
         this.mockMvc.perform(get("/download"))
                 .andExpect(status().isOk())
                 .andExpect(view().name("download"))
-                .andExpect(model().attributeExists("fileName", "fileSize", "fileTimestamp"));
+                .andExpect(model().attributeExists("mainTitle"));
     }
 
     @Test

app/src/test/java/uk/ac/ebi/atlas/experimentimport/GxaExperimentCrudIT.java

@@ -64,9 +64,10 @@ void updatesLastUpdateIfExperimentIsAlreadyPresent() {
         var lastUpdateBeforeUpdate = experimentBeforeUpdate.getLastUpdate();
 
         subject.createExperiment(experimentAccession, RNG.nextBoolean());
-        assertThat(subject.readExperiment(experimentAccession)).get()
+        var updatedExperiment = subject.readExperiment(experimentAccession).orElseThrow();
+        assertThat(updatedExperiment)
                 .isEqualToIgnoringGivenFields(experimentBeforeUpdate, "lastUpdate", "isPrivate");
-        assertThat(subject.readExperiment(experimentAccession).orElseThrow().getLastUpdate())
+        assertThat(updatedExperiment.getLastUpdate())
                 .isAfter(lastUpdateBeforeUpdate);
     }
 

app/src/test/java/uk/ac/ebi/atlas/experimentimport/admin/ExperimentOpsExecutionServiceTest.java

@@ -6,14 +6,11 @@
 import org.mockito.Mock;
 import org.mockito.junit.MockitoJUnitRunner;
 import uk.ac.ebi.atlas.experimentimport.ExperimentCrud;
-import uk.ac.ebi.atlas.experimentimport.ExperimentDto;
 import uk.ac.ebi.atlas.experimentimport.analyticsindex.AnalyticsIndexerManager;
 import uk.ac.ebi.atlas.experimentimport.coexpression.BaselineCoexpressionProfileLoader;
 
-import java.util.Optional;
 
 import static org.mockito.Mockito.verify;
-import static org.mockito.Mockito.when;
 
 @RunWith(MockitoJUnitRunner.class)
 public class ExperimentOpsExecutionServiceTest {
@@ -28,14 +25,10 @@ public class ExperimentOpsExecutionServiceTest {
     @Mock
     private AnalyticsIndexerManager analyticsIndexerManager;
 
-    @Mock
-    private ExperimentDto experimentDto;
-
     private ExperimentOpsExecutionService subject;
 
     @Before
     public void setUp() {
-        when(experimentCrudMock.readExperiment(ACCESSION)).thenReturn(Optional.of(experimentDto));
         subject =
                 new ExpressionAtlasExperimentOpsExecutionService(
                         experimentCrudMock,
@@ -46,7 +39,7 @@ public void setUp() {
     @Test
     public void updateExperimentDesignShouldRemoveExperimentFromCache() throws Exception {
         subject.attemptExecuteStatefulOp(ACCESSION, Op.UPDATE_DESIGN);
-//        verify(experimentTrader).removeExperimentFromCache(ACCESSION);
+        verify(experimentCrudMock).updateExperimentDesign(ACCESSION);
     }
 
     @Test

app/src/test/java/uk/ac/ebi/atlas/search/JsonBioentityInformationControllerWIT.java

@@ -61,7 +61,7 @@ public void payloadIsValidJson() throws Exception {
 
     @Test
     public void geneIdContainingDotIsNotTruncated() throws Exception {
-        String geneId = solrUtils.fetchRandomGeneOfSpecies("Arabidopsis_lyrata"); // has gene IDs containing dots
+        String geneId = solrUtils.fetchRandomGeneOfSpecies("Schizosaccharomyces_pombe"); // has gene IDs containing dots
 
         if(!geneId.isEmpty()) {
             this.mockMvc

app/src/test/java/uk/ac/ebi/atlas/tracks/GenomeBrowserControllerWIT.java

@@ -303,7 +303,7 @@ void differentialVertebrateExperimentEnsembl() throws Exception {
     @Test
     void privateExperimentWithoutAccessKey() throws Exception {
         this.mockMvc.perform(
-                get(MessageFormat.format(URL_TEMPLATE, "ensembl", "E-MTAB-4106", "SPAC869.01", "g5_g1")))
+                get(MessageFormat.format(URL_TEMPLATE, "ensembl", "E-MTAB-5555", "ENSMUSG00000000001", "g4_g2")))
                 .andExpect(status().isNotFound())
                 .andReturn();
     }

app/src/test/java/uk/ac/ebi/atlas/tracks/TracksControllerWIT.java

@@ -64,7 +64,7 @@ void setUp() {
 
     @Test
     void getTrackFile() throws Exception {
-        this.mockMvc.perform(get(BASELINE_URL_TEMPLATE.format(new Object[] {"E-MTAB-5423", "g391"})))
+        this.mockMvc.perform(get(BASELINE_URL_TEMPLATE.format(new Object[] {"E-MTAB-5422", "g2_g1"})))
                 .andExpect(status().isOk());
     }
 
@@ -77,7 +77,7 @@ void getTrackOfUnknownExperiment() throws Exception {
     @Test
     void getUnknownTrackEndsUpIn404NotFound() throws Exception {
         MvcResult result =
-                this.mockMvc.perform(get(BASELINE_URL_TEMPLATE.format(new Object[] {"E-MTAB-5423", "gFooBar"})))
+                this.mockMvc.perform(get(BASELINE_URL_TEMPLATE.format(new Object[] {"E-MTAB-5422", "gFooBar"})))
                         .andExpect(status().isOk())
                         .andReturn();
 
@@ -87,22 +87,23 @@ void getUnknownTrackEndsUpIn404NotFound() throws Exception {
 
     @Test
     void getTrackPrivateExperimentWithoutAcessKey() throws Exception {
-        this.mockMvc.perform(get(DIFFERENTIAL_URL_TEMPLATE.format(new Object[] {"E-MTAB-4106", "g5_g3"})))
+        this.mockMvc.perform(get(DIFFERENTIAL_URL_TEMPLATE.format(new Object[] {"E-MTAB-5555", "g4_g2"})))
                 .andExpect(status().isNotFound());
     }
 
     @Test
     void getTrackPrivateExperimentWitIncorrectAcessKey() throws Exception {
         this.mockMvc.perform(
-                get(DIFFERENTIAL_URL_TEMPLATE.format(new Object[] {"E-MTAB-4106", "g5_g3"}) + "?accessKey=foo-bar"))
+                get(DIFFERENTIAL_URL_TEMPLATE.format(new Object[] {"E-MTAB-5555", "g4_g2"})
+                        + "?accessKey=foo-bar"))
                 .andExpect(status().isNotFound());
     }
 
     @Test
     void getTrackPrivateExperimentWithGoodAcessKey() throws Exception {
         this.mockMvc.perform(
                 get(DIFFERENTIAL_URL_TEMPLATE.format(
-                        new Object[] {"E-MTAB-4106", "g5_g3"}) + "?accessKey=89319682-55b6-4b4b-a5ef-f50e1e0915ea"))
+                        new Object[] {"E-MTAB-5555", "g4_g2"}) + "?accessKey=ef1db551-15ef-4074-b56c-ed9a1b1157d2"))
                 .andExpect(status().isOk());
     }
 }

app/src/test/java/uk/ac/ebi/atlas/widget/BaselineAndDifferentialAnalyticsServiceIT.java

@@ -5,6 +5,7 @@
 import com.google.gson.JsonObject;
 import org.hamcrest.Matchers;
 import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assertions;
 import org.junit.jupiter.api.BeforeAll;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.TestInstance;
@@ -29,7 +30,6 @@
 import static org.hamcrest.Matchers.greaterThan;
 import static org.hamcrest.Matchers.not;
 import static org.hamcrest.core.Is.is;
-import static org.junit.Assert.assertTrue;
 import static uk.ac.ebi.atlas.utils.GsonProvider.GSON;
 
 @ExtendWith(SpringExtension.class)
@@ -72,7 +72,7 @@ void geneExpressedInBaselineAndDifferentialExperiments() {
         JsonObject result = baselineAnalyticsSearchService.findFacetsForTreeSearch(SemanticQuery.create(BASELINE_GENE),
                 SemanticQuery.create(), new Species("Foous baris", SpeciesProperties.UNKNOWN));
         assertThat(result.entrySet(), not(Matchers.empty()));
-        assertTrue("This Ensembl gene has a homo sapiens result", result.has("homo sapiens"));
+        Assertions.assertTrue(result.has("homo sapiens"), "This Ensembl gene has a homo sapiens result");
     }
 
     @Test
@@ -106,11 +106,11 @@ void nonExistentGene() {
     @Test
     void differentialAnalyticsSearchServiceHasTheRightReturnFormat() {
         JsonObject result =
-                differentialAnalyticsSearchService.fetchResults(SemanticQuery.create("GO:0008150"), EMPTY_QUERY);
+                differentialAnalyticsSearchService.fetchResults(SemanticQuery.create("GO:2000651"), EMPTY_QUERY);
         testDifferentialResultsAreInRightFormat(result);
     }
 
-    private ImmutableList<String> fieldsNeededInDifferentialResults = ImmutableList.of(
+    private final ImmutableList<String> fieldsNeededInDifferentialResults = ImmutableList.of(
             "species",
             "kingdom",
             "experimentType",
@@ -127,14 +127,14 @@ void differentialAnalyticsSearchServiceHasTheRightReturnFormat() {
             "colour",
             "id");
 
-    private ImmutableList<String> fieldsNeededInMicroarrayDifferentialResults =
+    private final ImmutableList<String> fieldsNeededInMicroarrayDifferentialResults =
             ImmutableList.<String>builder()
                     .addAll(fieldsNeededInDifferentialResults)
                     .add("tStatistic")
                     .build();
 
     private void testDifferentialResultsAreInRightFormat(JsonObject result) {
-        assertTrue(GSON.toJson(result), result.has("results"));
+        Assertions.assertTrue(result.has("results"), GSON.toJson(result));
         assertThat(result.get("results").getAsJsonArray().size(), greaterThan(0));
 
         for (JsonElement jsonElement: result.get("results").getAsJsonArray()) {
@@ -143,11 +143,11 @@ private void testDifferentialResultsAreInRightFormat(JsonObject result) {
 
             if (experimentType.isMicroarray()) {
                 for (String fieldName: fieldsNeededInMicroarrayDifferentialResults) {
-                    assertTrue("result has " + fieldName, jsonElement.getAsJsonObject().has(fieldName));
+                    Assertions.assertTrue(jsonElement.getAsJsonObject().has(fieldName), "result has " + fieldName);
                 }
             } else {
                 for (String fieldName: fieldsNeededInDifferentialResults) {
-                    assertTrue("result has " + fieldName, jsonElement.getAsJsonObject().has(fieldName));
+                    Assertions.assertTrue(jsonElement.getAsJsonObject().has(fieldName), "result has " + fieldName);
                 }
             }
         }