Export Geoserver layers from MTG to ATOS

[ghilbrae]

We need to start planning this task and agree on dates, procedures, etc.

@negroscuro whenever you are ready let us know.

[LauraMTG]

Hi Mario

I have access to ATOS' FTP for Clarity. The layers that we have in MTG right now, should go to your directories, I think these:

Europe:
1-should go all the already processed hazard indexes of ZAMG, I suggest making a new directory -TIF- to differentiate them from the original -.nc-
2. The population and landslide layers should also be raised. In the directory you indicate.

Dc1-Naples
1-Availability of population layers in Naples by age range, in total 3 layers (less than 15 years, between 15 and 65 and over 65). I suggest to create a new directory -Population-

DC4-Spain
1-Availability of fuel layers in TIF format. I suggest opening a new directory -FUELS-
2-Pilot tracks are now available in the ATOS ftp
3-The pilot's MDT is now available on the ATOS FTP.

This file transfer is fast. You will be able to have the layers in your FTP as soon as we agree on the directories of the submissions.
However, to make the layers available on the ATOS geoserver, the matter is complicated. Each layer must be created previously:
-a file store
-a style of legend
And check the access and publication permits.

If we shuffle a total of 400 layers, the update process in ATOS geoserver can take several days. I don't know an automatic way to do this...

I look forward to your comments

[negroscuro]

Hi Laura,
it would be possible for you to upload to the ATOS FTP the whole Geoserver folder from your tomcat or whatever server you are using?
That way we can replicate it in our system, and there will be one thing pending, to reproduce the same folder structure for all datasets/layers published in that Geoserver instance.
Then we will also need to know the folder structure where you are storying all the data, that means to know your local file system paths pointed from Geoserver.
Is this possible?
thank you.

[LauraMTG]

I don't really understand what do you mean. I have copied the data with the same folder structure in your server. I am going to generate a list with the directories in order you can replicate it in your Geoserver instance.

Should you have any problem, let me know.

[ghilbrae]

@negroscuro let us know if everything is ok, we hope the styles, stores and workspaces are migrated correctly because that could be rather painful to do manually. We could check if there's some other way of migrating that in case there's some sort of problem.

[ghilbrae]

Hi! We've checked the existing layers on our geoserver and we have come up with a list of changes that are needed. They are covered in the attached document.

Geoserver clean up.xlsx

Let us now if you have any questions.

[negroscuro]

Nice, we will use it to check everything is there after migration.
For now we are still trying to configure the docker containers.

[p-a-s-c-a-l]

Btw, who is going to update the Data Packages and resources, respectively which are currently pointing to https://clarity.meteogrid.com/geoserver/ ?

[ghilbrae]

That's a good question.
I'm afraid to ask but... Do we need to do it manually? 😟

If that is the case we'll have to share the work among several people, there are simply too many layers for just one or two people.

[p-a-s-c-a-l]

Another hint: TLS should be enabled for http://services.clarity-h2020.eu:8080/geoserver.

[p-a-s-c-a-l]

Do we need to do it manually?

Since we have access to the CSIS Drupal database, it should be possible to do it by some simple SQL statements. But I don't know the Drupal data model. WDYT @patrickkaleta and @fgeyer16

[therter]

Since we have access to the CSIS Drupal database, it should be possible to do it by some simple SQL statements. But I don't know the Drupal data model.

The paths to the wms should be saved in the table node__field_reference_path. So it should be possible to update the wms urls with an update statement.

[negroscuro]

It looks like we are gettin errors if we try to deploy the copy of the Meteogrid Geoserver isntance:

/usr/local/tomcat/lib/tomcat-i18n-es.jar jar is broken
/usr/local/tomcat/lib/jsp-api.jar jar is broken
/usr/local/tomcat/lib/tomcat-coyote.jar jar is broken
/usr/local/tomcat/lib/jasper-el.jar jar is broken
/usr/local/tomcat/lib/catalina.jar jar is broken
/usr/local/tomcat/lib/catalina-storeconfig.jar jar is broken
/usr/local/tomcat/lib/tomcat-util-scan.jar jar is broken
/usr/local/tomcat/lib/tomcat-websocket.jar jar is broken
/usr/local/tomcat/lib/tomcat-i18n-fr.jar jar is broken
/usr/local/tomcat/lib/annotations-api.jar jar is broken
/usr/local/tomcat/lib/el-api.jar jar is broken
/usr/local/tomcat/lib/tomcat-dbcp.jar jar is broken
/usr/local/tomcat/lib/servlet-api.jar jar is broken
/usr/local/tomcat/lib/catalina-tribes.jar jar is broken
/usr/local/tomcat/lib/catalina-ha.jar jar is broken
/usr/local/tomcat/lib/tomcat-util.jar jar is broken
/usr/local/tomcat/lib/jasper.jar jar is broken
/usr/local/tomcat/lib/tomcat-jdbc.jar jar is broken
/usr/local/tomcat/lib/websocket-api.jar jar is broken
/usr/local/tomcat/lib/tomcat-api.jar jar is broken
/usr/local/tomcat/lib/tomcat-i18n-ja.jar jar is broken
/usr/local/tomcat/lib/ecj-4.6.3.jar jar is broken
/usr/local/tomcat/lib/tomcat-jni.jar jar is broken
/usr/local/tomcat/lib/catalina-ant.jar jar is broken

But a clean deployment of a Geoserver works.... the problem is the 500 layers already configured in Meteogrid's.

Regarding:

@negroscuro
This is the info on our geoserver installation:
TOMCAT

server.info=Apache Tomcat/8.5.35
server.number=8.5.35.0

GEOSERVER

GeoServer Version
2.14.0
Git Revision
2ecf6413b0d5d304c7dc23d8d7e9f3cfad29f37c
Build Date
21-Sep-2018 20:18
GeoTools Version
20.0 (rev ffc271f317c04e714ea44a4879dd4601bd723d5e)
GeoWebCache Version
1.14.0 (rev 1.14.x/775e94eb2ae59dd9b74aecc709b67d7b968a4317)

Wich JAVA version are you using?

[ghilbrae]

According to our geoserver, java version is: Oracle Corporation: 1.8.0_212 (OpenJDK 64-Bit Server VM).

I think we cannot drag this too long. We are updating and changing layers from time to time and our versions are diverging. I've discussed this with @LauraMTG and we think that if this does not work we should try doing the thing by hand, (yeah, I know...) We can just divide the 500 layers among the four of us (@negroscuro @bullz + @LauraMTG and me) and do it. We could share a short doc with the guidelines for naming, styles and such.

[ghilbrae]

@negroscuro @bullz any inputs on this issue? We shouldn't let this linger for much longer.

[p-a-s-c-a-l]

Any progress to report here @DanielRodera ?

[DanielRodera]

Hi,

yes, we have deployed a Geoserver instance in the new Atos server, we have deployed the 2.16.2 version. We have included the importer module to be able to import all the MTG data and we also take care of the CORS filter.

Today we will have a telco with MTG to restore all the data.

[p-a-s-c-a-l]

Today we will have a telco with MTG to restore all the data.

How is it going? 😃

[DanielRodera]

People from MTG are working restoring the data with the importer module. Once is done the only remaining task will be to assign the layers styles. @ghilbrae is splitting this task between Atos and MTG people to finish it as soon as possible.

[ghilbrae]

Most of the layers have already been uploaded to the new geoserver. I'm going through the remaining layers (around 25) to solve whatever is going on with them. Anyway, all of the layers related to hazards for Europe are working correctly, the only thing that we have to do now is update the styles.

[therter]

I have seen that the DNS entry for the url https://geoserver.myclimateservice.eu still points to the geoserver instance on the AIT server instead of the new ATOS geoserver instance. I think, we should change this now.
What is the current url (or the IP) of the new geoserver? So that I can test whether the response header are set properly for the "Include in Report" operation.

[therter]

@DanielRodera the url https://geoserver.myclimateservice.eu points now to 95.211.163.27, but I get a "503 Service Temporarily Unavailable", when I try to access the url.

[p-a-s-c-a-l]

https://geoserver.myclimateservice.eu/ gives me 'connection refused' but I can access http://95.211.163.27:8080/geoserver/web/

But when looking at the layer previews (16 layers), this doesn't seem to be the correct geoserver instance.

[DanielRodera]

Hi @p-a-s-c-a-l , @therter ,

Let me explain:

http://95.211.163.27:8080/geoserver/web/ --> is another instance for another project. No related with CLARITY.

https://geoserver.myclimateservice.eu --> previously was pointing to an empty GeoServer in another server (I don't know which Geoserver and where). Yesterday our IT team mapped the subdomain to our server, but the Geoserver was exposed in a different port (not the default one), it because you couldn't reach the server. After some work, we manage to set up everything properly.

The Atos Geoserver is http://geoserver.myclimateservice.eu/geoserver. This Geoserver stores all the MTG data and in a few days should have all the data Mario generated until today (we are working on it, we will let you know when is ready).

Next step: get the httpS. Our IT people are working on it.

[DanielRodera]

Hi,

The certificates are now properly set, the Geoserver is running on:
https://geoserver.myclimateservice.eu/geoserver

Once we are sure the migration is successful, we will use the old instance of Geoserver for making tests for the input layers scripts.

@p-a-s-c-a-l, @humerh could you please check if all the layers are served as you needed?

[p-a-s-c-a-l]

ATM layers fail to load. Examples:

• https://geoserver.myclimateservice.eu/geoserver/europe/wms?&service=WMS&request=GetMap&layers=europe%3AWI_98perc_sfcWindmax_historical_19710101-20001231_ensmean&styles=&format=image%2Fpng&transparent=true&version=1.1.1&width=1536&height=1536&srs=EPSG%3A3857&bbox=2622095.8182946863,5772524.376096511,2680799.456017702,5831228.013819526
• https://geoserver.myclimateservice.eu/geoserver/europe/wms?service=WMS&request=GetLegendGraphic&version=1.1.0&format=image%2Fpng&width=20&height=20&layer=europe:WI_98perc_sfcWindmax_historical_19710101-20001231_ensmean&style=&legend_options=forceLabels:on

502 Bad Gateway

[DanielRodera]

Thank you for reporting @p-a-s-c-a-l , we are working on it.

[p-a-s-c-a-l]

It seems that GeoServer is totally overloaded and returns 502 Bad Gateway in case of a timeout. ATM it's not possible to serve a single users, so we cannot go live with this setup.

The Status Page shows physical memory usage at 96.27 % (3.3 GiB) and swap memory usage at 54.06 % (8 GiB). It is possible to increase physical memory? 8 GiB should be the absolute minimum, better 16-32 GiB.

[DanielRodera]

We have to restart the server so you may experience connections problems. Should be ready by today, I will let you know anyway.

Thank you!

[DanielRodera]

We assigned more memory to the ATOS server as well as the Geoserver. Now should work as expected,

Sorry for the inconveniences,

[p-a-s-c-a-l]

We assigned more memory to the ATOS server as well as the Geoserver.

Thanks. :-)

What's the status of the styles? I noticed that there are still some of the missing.

[p-a-s-c-a-l]

It seems that getFeatureInfo is not enabled / working on https://geoserver.myclimateservice.eu. You can test it here

This is how it works on services.clarity-h2020.eu:8080:

@therter Do you know whether this a per-layer-option or can it be controlled globally?

[p-a-s-c-a-l]

It seems that getFeatureInfo is not enabled / working on https://geoserver.myclimateservice.eu. You can test it here

@therter I've set the PROXY base URL, the mixed content warning is gone.

[DanielRodera]

Hi @p-a-s-c-a-l,

I have tried and its working,

Anyone else has this problem??

[p-a-s-c-a-l]

I have tried and its working,

Yes. after setting PROXY base URL it's working now.