Ves-hx-77 migration from Hinxton to Harlow(ves-hx-77-x)

[upendrakumbham]

The storage team copied data from Hinxton VM to Harlow. Iris worked on this as well.

I will test it and update the VM status, if required I will update configuration files and deploy a new war..etc

Pivotal story reference for the history :ticket

[upendrakumbham]

I tested our Bulk VM - ves-hx-77-x and fixed the Solr config issue. All looks good to me, but I could see this message - No data match your filtering criteria or your original query. Please, change your query or your filters and try again. in all the experiments.

I can see some documents on Solr collections on the UI side.

I will confirm the location of the datafiles with @iris as she worked on this.

[upendrakumbham]

Solr restart didn't help to fix the above issue.

Verified configuration files for the data:

[fg_atlas@ves-hx-77-x gxa]$ cat /nfs/public/rw/fg/atlas/tc8-fg-gxa/test/configuration.properties 
data.files.location = /srv/gxa/data
experiment.files.location = /srv/gxa/data/gxa
experiment.design.location = /srv/gxa/data/gxa/expdesign

[upendrakumbham]

I had a meeting with Iris for the data prod team. She said that she was not aware of the config file data paths. But mentioned that /nfs/public/ro/gxa/experiments_test/ data set are same for both VMs(Ves-hx-77 and ves-hx--77-x) after rsync by storage team.

[upendrakumbham]

[tc_fg02@ves-hx-77 magetab]$ find . -type d | wc -l
22039

[fg_atlas@ves-hx-77-x magetab]$ find . -type d | wc -l
22039

Same data on both VMs

[upendrakumbham]

After restarting Tomcat manually and through Jenkins: I got the below error:

17:12:27.557 [http-nio-8080-exec-4] ERROR uk.ac.ebi.atlas.controllers.HtmlExceptionHandlingController [HtmlExceptionHandlingController.java:46] - org.apache.commons.configuration2.ex.ConfigurationException: Unable to load the configuration from the URL file:/srv/gxa/data/gxa/magetab/E-PROT-3/E-PROT-3-factors.xml - uk.ac.ebi.atlas.model.resource.XmlFile$ReadOnly.get(XmlFile.java:36)
	uk.ac.ebi.atlas.model.resource.XmlFile$ReadOnly.get(XmlFile.java:19)
	uk.ac.ebi.atlas.trader.ConfigurationTrader.getBaselineFactorsConfiguration(ConfigurationTrader.java:25)
	uk.ac.ebi.atlas.trader.factory.BaselineExperimentFactory.create(BaselineExperimentFactory.java:54)
	uk.ac.ebi.atlas.trader.GxaExperimentRepository.getExperiment(GxaExperimentRepository.java:62)
	jdk.internal.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
	java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.base/java.lang.reflect.Method.invoke(Method.java:566)
	org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
	org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	org.springframework.cache.interceptor.CacheInterceptor.lambda$invoke$0(CacheInterceptor.java:53)
	org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:365)
	org.springframework.cache.interceptor.CacheAspectSupport.lambda$execute$1(CacheAspectSupport.java:381)

[upendrakumbham]

[upendrakumbham]

[upendrakumbham]

After restarting ves-hx-77 and ves-hx-77-x VMs tomcat, I had the same error on both machines while reading E-PROT-3 experiment factors.xml file

[upendrakumbham]

I tried to change permissions, but I can't change anything in that directory.

[upendrakumbham]

[fg_atlas@ves-hx-77-x E-PROT-3]$ chmod u+r E-PROT-3-factors.xml
chmod: changing permissions of `E-PROT-3-factors.xml': Read-only file system
[fg_atlas@ves-hx-77-x E-PROT-3]$ ls -lh
total 4.0M
drwxr-xr-x 2 fg_atlas fg_pub      186 Jan 20  2021 archive
-rw-r--r-- 1 fg_atlas microarray  682 Jan  2  2016 E-PROT-3-analysis-methods.tsv
-rw-r--r-- 1 fg_atlas fg_pub      39K Jan 20  2021 E-PROT-3.condensed-sdrf.tsv
-rw-r--r-- 1 fg_atlas microarray 8.9K Feb  8  2016 E-PROT-3-configuration.xml
-rwxr-x--- 1 fg_atlas microarray 1.3K Jan 25  2016 E-PROT-3-factors.xml
-rw-r--r-- 1 fg_atlas fg_pub     4.0K Jan 18  2021 E-PROT-3.idf.txt
-rw-r--r-- 1 fg_atlas fg_pub     117K Jan 20  2021 E-PROT-3.tsv
-rw-r--r-- 1 fg_atlas fg_pub     881K Jan  5  2016 E-PROT-3.tsv.orig
-rw-r--r-- 1 fg_atlas fg_pub     1.8M Jan 25  2016 E-PROT-3.tsv.undecorated
-rw-r--r-- 1 fg_atlas fg_pub      15K Jan 20  2021 E-PROT-3-zoomifications-log.tsv
[fg_atlas@ves-hx-77-x E-PROT-3]$ 

[upendrakumbham]

[tc_fg02@ves-hx-77 magetab]$ cd E-PROT-3
[tc_fg02@ves-hx-77 E-PROT-3]$ ls -lh
total 4.5M
drwxr-xr-x 2 fg_atlas fg_pub      186 Jan 20  2021 archive
-rw-r--r-- 1 fg_atlas microarray  682 Jan  2  2016 E-PROT-3-analysis-methods.tsv
-rw-r--r-- 1 fg_atlas fg_pub      39K Jan 20  2021 E-PROT-3.condensed-sdrf.tsv
-rw-r--r-- 1 fg_atlas microarray 8.9K Feb  8  2016 E-PROT-3-configuration.xml
-rwxr-x--- 1 fg_atlas microarray 1.3K Jan 25  2016 E-PROT-3-factors.xml
-rw-r--r-- 1 fg_atlas fg_pub     4.0K Jan 18  2021 E-PROT-3.idf.txt
-rw-r--r-- 1 fg_atlas fg_pub     117K Jan 20  2021 E-PROT-3.tsv
-rw-r--r-- 1 fg_atlas fg_pub     881K Jan  5  2016 E-PROT-3.tsv.orig
-rw-r--r-- 1 fg_atlas fg_pub     1.8M Jan 25  2016 E-PROT-3.tsv.undecorated
-rw-r--r-- 1 fg_atlas fg_pub      15K Jan 20  2021 E-PROT-3-zoomifications-log.tsv
[tc_fg02@ves-hx-77 E-PROT-3]$ pwd
/srv/gxa/data/gxa/magetab/E-PROT-3

[upendrakumbham]

I verified on Bulk staging server for the same E-PROT-3 for the file permissions:

[fg_atlas@wp-p1m2-98 magetab]$ cd E-PROT-3
[fg_atlas@wp-p1m2-98 E-PROT-3]$ ls -lh
total 4.9M
drwxr-xr-x 2 fg_atlas fg_pub      237 Nov  9  2021 archive
-rw-r--r-- 1 fg_atlas microarray  682 Jan  2  2016 E-PROT-3-analysis-methods.tsv
-rw-r--r-- 1 fg_atlas fg_pub      39K Nov  9  2021 E-PROT-3.condensed-sdrf.tsv
-rw-r--r-- 1 fg_atlas microarray 8.9K Feb  8  2016 E-PROT-3-configuration.xml
-rwxr-xr-x 1 fg_atlas microarray 1.3K Jan 25  2016 E-PROT-3-factors.xml
-rw-r--r-- 1 fg_atlas fg_pub     4.0K Nov  9  2021 E-PROT-3.idf.txt
-rw-r--r-- 1 fg_atlas fg_pub     574K Nov  9  2021 E-PROT-3.tsv
-rw-r--r-- 1 fg_atlas fg_pub     881K Jan  5  2016 E-PROT-3.tsv.orig
-rw-r--r-- 1 fg_atlas fg_pub     1.8M Jan 25  2016 E-PROT-3.tsv.undecorated
-rw-r--r-- 1 fg_atlas fg_pub      15K Feb 25  2021 E-PROT-3-zoomifications-log.tsv
[fg_atlas@wp-p1m2-98 E-PROT-3]$ 

E-PROT-3-factors.xml file has read permissions along with execution. I believe the above two Bulk test VMs missed read permissions, throwing the above error while building experiments.

[upendrakumbham]

I set up a meeting with Iris and Karoly to discuss the permissions issue.
The outcome of the meeting is that we will wait until Iris moves experiments from the read-only path to the read-write path.
Then, the data prod team will try to give read permissions and the dev team(ukumbham) test the VM again.