Bug with a layer of biology_occurrence_data

[maelle]

@salvafern is this expected? The layer is "abiotic_observations".

library("EMODnetWFS")
service <- "biology_occurrence_data"
layers_info <- emodnet_get_wfs_info(service = service)
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> ✔ WFS client created successfully
#> ℹ Service: "https://geo.vliz.be/geoserver/Dataportal/wfs"
#> ℹ Version: "2.0.0"
layer <- layers_info[layers_info[["format"]] == "sf",][1,][["layer_name"]]
    emodnet_get_layers(service = service, layers = layer)[[1]] |>
        sf::st_crs() |>
        print()
#> ✔ WFS client created successfully
#> ℹ Service: "https://geo.vliz.be/geoserver/Dataportal/wfs"
#> ℹ Version: "2.0.0"
#> Warning: Download of layer "abiotic_observations" failed: Error in gsub("<!--.*?-->",
#> "", text): 'R_Calloc' could not allocate memory (18446744071562067968 of 1
#> bytes)
#> Coordinate Reference System: NA

^{Created on 2024-03-21 with reprex v2.1.0}

Standard output and standard error

-- nothing to show --

[maelle]

oh actually this gsub() happens in ows4r, from what I can tell. @eblondel have you encountered this problem before? Could it be solved with the switch to xml2? I am wondering whether there's another way to remove the XML comments. Now, if the XML is gigantic, probably there could be other issues later down the road. 😅

[maelle]

https://github.com/eblondel/ows4R/blob/553c524766b3945faac16606bb672e641ca5a8b9/R/OWSHttpRequest.R#L134

[eblondel]

@maelle The switch to xml2 from XML will take time for ows4R, and other packages with complex XML models. I've made the transition for some other packages. It's done in geosapi, i've started tackling it in atom4R, but there is not always clear equivalent of XML methods in xml2, and some of questions in xml2 still remain with answer. From the exercices I've run, I can say XML has still more flexibility in writing XML than xml2.

Here the issue you face is the size of the response, beyond the operation gsub. I'm not sure that having an xml2 object will solve the issue.

Let me try to remember why I've introduced this line to remove XML comments, surely it was not dealing with a WFS GetFeatures response for which we never get XML comments (at least in my experience). In worst case we could make this check optional depending on the service we target. This could solve the issue. I'll let you know when I find something

[eblondel]

By the way, you can still bypass this by using another outputFormat to retrieve the features. CSV or JSON for example. Given the amount of the data, I would give a try to CSV.

[eblondel]

Please see small enhancement done to ows4R above.
I can run this example:

requireo(ows4R)
WFS = WFSClient$new(url = "https://geo.vliz.be/geoserver/Dataportal/wfs", serviceVersion = "2.0.0", logger = "DEBUG")
ft = WFS$capabilities$findFeatureTypeByName("abiotic_observations", F)
sf = ft$getFeatures()

This example works, although I get a message Memory allocation failed : growing input buffer
Please note that you can here the maximum nb of features as declared in your Geoserver WFS Settings (ie 1,000,000 features).

Let me know

Cheers
Emmanuel

[maelle]

Thanks so much!!

Please note that you can here the maximum nb of features as declared in your Geoserver WFS Settings (ie 1,000,000 features).

I don't understand, could you please clarify?

[eblondel]

In the Geoserver admin interface, you have a WFS settings to control the maximum nb of features to be returned in a response. If a request exceeds this number, only the max nb of features will be retrieved. Depending on your data, this can be adapted. In this example, you reach the default threshold which is 1 million features
See https://docs.geoserver.org/main/en/user/services/wfs/webadmin.html#features

[maelle]

@eblondel Merci beaucoup !

@bart-v @salvafern see above, should the setting be tweaked for the biology occurrence data service and if so, who would be in charge?

[maelle]

@bart-v @salvafern friendly reminder 🙏

[bart-v]

I don't understand why this is a problem at the WFS server?
You clearly ask for too much data could not allocate memory
Just ask less and paginate...

[maelle]

Ok, thanks, I was asking because of this comment: #164 (comment)

[bart-v]

There is 40M+ records, I'm sure it will even crash worse on that...
The default 1M limit is fine and can stay

[salvafern]

As Bart said I think the 1M limit is reasonable. @eblondel was looking into giving support to pagination to ows4R (see eblondel/ows4R#70) but I can imagine this is not an easy task. So for now we can maybe close this issue?