locale problem in qgis_query_algorithms()

[jannes-m]

Things to check beforehand

• To avoid duplication, I have checked existing issues.
• I use a currently supported QGIS version, i.e. shown green in the QGIS release schedule.

Are QGIS plugins the cause?

The problem remains with all plugins disabled.

Description of the observed problem

Hey @florisvdh,
I finally had some time to address geocompx/docker#52.
When running our image in a bash session (docker run -it --rm ghcr.io/geocompx/docker:qgis R), library(qgisprocess) fails.
I tracked down the error to jsonlite::fromJSON() called in qgisprocess::qgis_query_algorithms().
This is a locale problem and can be solved by setting the locale (see below).
Not setting the encoding argument in result = qgisprocess:::qgis_run(args = c("list", "--json"), encoding = "UTF-8") run in qgis_query_algorithms() would also work but that might cause unwanted side effects, I guess.
Hence, I wonder if we should set the locale in qgisprocess::qgis_query_algorithms() and restore the original values on exit?

@Robinlovelace the qgis container works just fine when run in the browser in an rstudio session because the rstudio session already uses en_US.UTF-8.

Sys.getlocale()
#> [1] "C"
try(qgisprocess:::qgis_query_algorithms())
#> Attempting to load the package cache ...
#> Standard error message from 'qgis_process':
#> QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
#>
#>
#> Success!
#>
#> Standard error message from 'qgis_process':
#> QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
#> Error : parse error: premature EOF
#>                                        {   "gdal_version": "3.8.5",
#>                      (right here) ------^
Sys.setlocale("LC_ALL", "en_US.UTF-8")
#> [1] "LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=en_US.UTF-8;LC_MESSAGES=C;LC_PAPER=C;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=C;LC_IDENTIFICATION=C"
qgisprocess:::qgis_query_algorithms() |>
  head()
#>
#> Standard error message from 'qgis_process':
#> QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
#> # A tibble: 6 �~W 24
#>   provider provider_title algorithm               algorithm_id   algorithm_title
#>   <chr>    <chr>          <chr>                   <chr>          <chr>
#> 1 3d       QGIS (3D)      3d:tessellate           tessellate     Tessellate
#> 2 gdal     GDAL           gdal:aspect             aspect         Aspect
#> 3 gdal     GDAL           gdal:assignprojection   assignproject�~@� Assign project�~@�
#> 4 gdal     GDAL           gdal:buffervectors      buffervectors  Buffer vectors
#> 5 gdal     GDAL           gdal:buildvirtualraster buildvirtualr�~@� Build virtual �~@�
#> 6 gdal     GDAL           gdal:buildvirtualvector buildvirtualv�~@� Build virtual �~@�
#> # �~D� 19 more variables: provider_can_be_activated <lgl>,
#> #   provider_is_active <lgl>, provider_long_name <chr>, provider_version <chr>,
#> #   provider_warning <chr>, can_cancel <lgl>, deprecated <lgl>, group <chr>,
#> #   has_known_issues <lgl>, help_url <chr>, requires_matching_crs <lgl>,
#> #   short_description <chr>, tags <list>, default_raster_file_extension <chr>,
#> #   default_vector_file_extension <chr>,
#> #   supported_output_raster_extensions <list>, �~@�

^{Created on 2024-09-04 with reprex v2.1.0}
~

QGIS version

3.36.2

R session info

R version 4.4.0 (2024-04-24)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 22.04.4 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so; LAPACK version 3.10.0

locale:
[1] C

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] qgisprocess_0.3.0.9000

loaded via a namespace (and not attached):
[1] vctrs_0.6.5 cli_3.6.2 knitr_1.46 rlang_1.1.3
[5] xfun_0.43 stringi_1.8.3 processx_3.8.4 assertthat_0.2.1
[9] jsonlite_1.8.8 glue_1.7.0 clipr_0.8.0 htmltools_0.5.8.1
[13] ps_1.7.6 rappdirs_0.3.3 fansi_1.0.6 rmarkdown_2.26
[17] evaluate_0.23 tibble_3.2.1 fastmap_1.1.1 yaml_2.3.8
[21] lifecycle_1.0.4 stringr_1.5.1 compiler_4.4.0 fs_1.6.4
[25] pkgconfig_2.0.3 rstudioapi_0.16.0 digest_0.6.35 R6_2.5.1
[29] reprex_2.1.0 utf8_1.2.4 pillar_1.9.0 callr_3.7.6
[33] magrittr_2.0.3 tools_4.4.0 withr_3.0.0

Additional context

No response

[Robinlovelace]

Very interesting, thanks for the investigations Jannes. 👍 to the suggested solution.

[florisvdh]

Thanks for bringing this up and for pursuing this @jannes-m!

From #176 (comment), it seems that this might also be tied to the missing XDG_RUNTIME_DIR variable. Can you set this environment variable to a directory with permission 0700 and see if that solves it?

Very interesting that you could get it working by adjusting the locale. However it returns malformed characters in your example.

In the referred issue, it seemed that the JSON string malformation was linked to the environment variable. If that is something that can be linked to qgis_process behaviour (not R), then it will have to be reported to QGIS. Also how this interacts with the locale is useful.

[jannes-m]

Thanks for your thoughts @florisvdh!

Setting the XDG_RUNTIME_DIR variable to a directory with permission 0700 does not resolve the issue.
The issue only occurs in the R session.
Running QT_QPA_PLATFORM="offscreen" qgis_process list --json | jq works fine.

The malformed characters only appear in the reprex but not if you run the example interactively.
You can reproduce the behavior by running the example in our container (docker run -it --rm ghcr.io/geocompx/docker:qgis R).

Overall, jsonlite::fromJSON() does not work when the locale is `"C"`` but works again when setting the locale to UFT-8 (as shown in the reprex above).

[florisvdh]

OK, thanks for clarifying @jannes-m. I believe in most local platforms you won't have that the locale is C, but maybe it's standard in docker images. And then indeed:

if we should set the locale in qgisprocess::qgis_query_algorithms() and restore the original values on exit

I think it would be fine to implement that in a conditional manner: check if the locale is C (or some other case that we know causes the problem) and if so, only then change the locale for the duration of the function call.

What do you think?

[jannes-m]

I absolutely agree, I'll draft a PR and of course I'll let you know -:)

[florisvdh]

BTW in jeroen/jsonlite#338 (comment) I read that JSON is always UTF-8 by definition. {qgisprocess} relies on JSON.

So it might also make sense to adjust the locale of the container instead. Addressing the problem in non UTF-8 locales would potentially apply to any package relying on JSON strings.

But I would definitely welcome your PR!

[jannes-m]

mmh, you are right, I'll think about it

[Robinlovelace]

Looks like this could do it in the Dockerfile?

# Set the locale
RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && \
    locale-gen
ENV LANG en_US.UTF-8  
ENV LANGUAGE en_US:en  
ENV LC_ALL en_US.UTF-8    

Source: https://stackoverflow.com/questions/28405902/how-to-set-the-locale-inside-a-debian-ubuntu-docker-container

[florisvdh]

It seems there has been some discussion in Rocker about this in the past as well (which you rely on). rocker-org/rocker#19

[Robinlovelace]

It seems there has been some discussion in Rocker about this in the past as well (which you rely on). rocker-org/rocker#19

Wow, what a thread! Many thanks for finding and sharing that. A solution for RStudio aligns with my suggestion: https://github.com/rstudio/r-docker/pull/18/files

So let's build on that, do you want to give that a go and see if that fixes it Jannes?

[florisvdh]

Closed by PR #219.

[Robinlovelace]

Great work! New release in order I imagine. Many thanks 🙏

[florisvdh]

@jannes-m @Robinlovelace Do you need to have the fix on CRAN to get this fixed on your side or do you use qgisprocess from github/r-universe ? In the latter case, I'd want to wait with CRAN release in order to investigate processx-related problems with some webservice URLs, specific to Windows, discovered by a colleague (but that might take some time, weeks or so).

[Robinlovelace]

@florisvdh yes please as it's not in remotes: https://github.com/geocompx/geocompkg/blob/e6c5d110632fbdaf150f27ba667ea2e460308409/DESCRIPTION#L81

[florisvdh]

OK, expect a CRAN release with this fix quite soon, then 🙂

[jannes-m]

@Robinlovelace of course you are right that we do not use remotes. However, I would say, we could wait a few weeks for the fix since the problem only ocurrs when running the container in a bash session. And I assume there are not so many people using this option. I might be wrong but at least nobody aside from Robin reported this bug. But of course, I am also happy if you publish a new CRAN release right now.

[Robinlovelace]

Let's go with CRAN now, unless any downside. Will make testing a bit easier and the image has a surprisingly large n. downloads for something that is temporarily broken.

[florisvdh]

FYI CRAN submission is pending since a couple of days, after replying about a supposedly false positive; PR #222.

[Robinlovelace]

Many thanks 🙏

[florisvdh]

Finally the third attempt made it to CRAN (while releasing this feature should have been a no-brainer). See #222 for more on this.

[Robinlovelace]

Epic! Hopefully will be easier next time 🙏

[jannes-m]

Thanks a lot for your efforts 🙏