Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pull] master from OSGeo:master #49

Merged
merged 94 commits into from
Oct 13, 2023
Merged

[pull] master from OSGeo:master #49

merged 94 commits into from
Oct 13, 2023

Conversation

pull[bot]
Copy link

@pull pull bot commented Oct 11, 2023

See Commits and Changes for more details.


Created by pull[bot]

Can you help keep this open source service alive? 💖 Please sponsor : )

…h now forwards to ReadDirEx(); modify all implementations to implement ReadDirEx()
…r any other Arrow supported value as an option of the generic implementation
- Multiple enhancements upstreamed from Planet's version:
  * RPC metadata comparison (can be disabled with SKIP_RPC option)
  * GEOLOCATION metadata comparison (disabled with SKIP_GEOLOCATION)
  * Fix comparing NaN nodata values
  * Dump differences of pixel content (enabled with DUMP_DIFFS option)
  * Improve comparison of float32/float64 data by not just checking
    checksum but also statistics
  * Overview comparison can be skipped with SKIP_OVERVIEWS
  * Add sanity check when comparing pixel-interleave datasets with
    more than 10 bands since that can be slow.

- Binary comparison can be done on /vsi files now

- Command line utility: add ``-dumpdiffs``, ``-skip_binary``,
  ``-skip_overviews``, ``-skip_geolocation``, ``-skip_geotransform``,
  ``-skip_metadata``, ``-skip_rpc``, and ``-skip_srs`` options

- Add a test_gdalcompare.py script
…nstead of a OGRSpatialReference*. Affects out-of-tree drivers (fixes #8493)
…compile-time option on views using Spatialte functions (refs #8514)
…e able to save statistics of datasets returned by GDALMDArray::AsClassicDataset()
…E and CPL_VSIL_CURL_CACHE_SIZE are used (refs #8499)
rouault and others added 29 commits October 11, 2023 13:18
GDALDataset::ICreateLayer(): now takes a const OGRSpatialReference* instead of a OGRSpatialReference*
Make SQLite and GPKG drivers work with a sqlite3 built with -DSQLITE_TRUSTED_SCHEMA=0
Add a OGRLayer::WriteArrowBatch method
actually fix ef90e9f which only worked
on string fields having numeric values...

Thanks to QGIS test suite to catch that...
OGR SQL: fix min/max on string fields (master only)
VSI7Z: Accept ArcGIS Pro Project Packages
That is when:
- the input driver reports OLCFastGetArrowStream
- and the output driver reports OLCFastWriteArrowBatch
- and the input ArrowSchema is accepted by the target layer
- and there are basically no ogr2ogr options (other than -dsco, -lco, -limit, -where and -spat).

Note that the efficiency in case of -where and -spat can be less than
using the classic OGRFeature strategy when the source is Parquet and a
lot of features are discarded.

With that Parquet->Parquet or GPKG->Parquet translation can be up to 3x
times faster.

$ time ogr2ogr out.parquet nz-building-outlines.parquet --config OGR2OGR_USE_ARROW_API NO

real    0m11,246s
user    0m10,934s
sys     0m0,974s

$ time ogr2ogr out.parquet nz-building-outlines.parquet

real    0m4,311s
user    0m3,968s
sys     0m0,889s

$ time ogr2ogr out.parquet nz-building-outlines.gpkg --config OGR2OGR_USE_ARROW_API NO

real    0m12,120s
user    0m11,360s
sys     0m0,764s

$ time ogr2ogr out.parquet nz-building-outlines.gpkg

real    0m3,853s
user    0m5,053s
sys     0m1,028s
ogr2ogr: use Arrow interface in reading and writing when possible
Fix potential memleaks related to sqlite3_open() failure (fixes #8554)
…tr_filter

FlatGeoBuf: make GetNextArrowArray() capable of dealing with attribute filter
GeoJSON & GeoJSONSeq: use faster file write() primitive and detect write() errors
jpeg.rst: Fix typo for bitmask: `valid date` → `valid data` [ci skip]
@pull pull bot merged commit 1ce1fb4 into innFactory:master Oct 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants