Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lib/gis: Update lz4 library to latest upstream version (v1.10.0) #4666

Merged
merged 1 commit into from
Nov 14, 2024

Conversation

ymdatta
Copy link
Contributor

@ymdatta ymdatta commented Nov 7, 2024

Sync lz4 library with the upstream version v 1.10.0.

This has fixes simplifying and making the library robust in multiple areas.

This also fixes the initial null pointer arithmetic error found by cppcheck, which prompted syncing with the upstream library for new fixes and updates.

@github-actions github-actions bot added C Related code is in C libraries labels Nov 7, 2024
lib/gis/lz4.c Outdated Show resolved Hide resolved
@metzm
Copy link
Contributor

metzm commented Nov 7, 2024

Please check first if the issue is resolved in upstream https://github.com/lz4/lz4
The preferred solution would be to update the local copy in lib/gis/ to the latest release of LZ4

@ymdatta
Copy link
Contributor Author

ymdatta commented Nov 8, 2024

@metzm : Thanks for pointing it out. I didn't realize that it was an external library and believed we were maintaining a local version.

I have checked the code in the upstream and indeed they have fixed this issue! I can update this PR to have changes which sync our version to the upstream.

@ymdatta ymdatta changed the title lib/gis: Fix possible null pointer arithmetic scenario in lz4 module lib/gis: Update lz4 library to latest upstream version (v1.10.0) Nov 8, 2024
This has fixes simplifying and making the library robust
in multiple areas. This also fixes the initial null pointer
arithmetic error found by cppcheck, which prompted syncing
with the upstream library for new fixes and updates.

Signed-off-by: Mohan Yelugoti <[email protected]>
@ymdatta
Copy link
Contributor Author

ymdatta commented Nov 8, 2024

Some testing done as part of the library update:

  1. Manually checked function calls to LZ4 library in cmprlz4.c to make sure that there is no mismatch to function signatures in the new version.
  2. Compiled the code to make sure it's not erroring out.

Only changes come in the form of clang format check specific changes to the upstream code.

@metzm : Let me know if I have missed anything or can improve patch set in some way. Thanks!

@nilason
Copy link
Contributor

nilason commented Nov 8, 2024

@metzm or any one with knowledge: Do we have test that limits the risk of regression of this?

@neteler
Copy link
Member

neteler commented Nov 9, 2024

How about borrowing a test from https://github.com/lz4/lz4/tree/dev/tests/?

@ymdatta
Copy link
Contributor Author

ymdatta commented Nov 12, 2024

Some things I want to highlight:

  • I am looking at lz4 library history in our codebase and in the two previous instances where the library was upgraded, it was done without any regression tests.
  • I am not sure if we can adopt lz4 tests as regression tests. Because, the APIs we are using: LZ4_compressBound, LZ4_compress_default, and LZ4_decompress_safe are not being directly used anywhere in the lz4 tests.
  • I observed that these APIs had the same interface between older and upgraded version. Release notes indicate that none of these APIs have been deprecated or are backwards incompatible now. So, I think we should be good.

@nilason
Copy link
Contributor

nilason commented Nov 12, 2024

Some things I want to highlight:

  • I am looking at lz4 library history in our codebase and in the two previous instances where the library was upgraded, it was done without any regression tests.

If I'm not mistaken, that was the time before CI run tests and I'm still not sure whether we have tests for the different supported compression methods. It may have been tested manually by committer with the former updates.

  • I am not sure if we can adopt lz4 tests as regression tests. Because, the APIs we are using: LZ4_compressBound, LZ4_compress_default, and LZ4_decompress_safe are not being directly used anywhere in the lz4 tests.

In practice we can, see r.compress for setting and using different compression methods.

  • I observed that these APIs had the same interface between older and upgraded version. Release notes indicate that none of these APIs have been deprecated or are backwards incompatible now. So, I think we should be good.

All being said, my hunch is that it is probably safe too. @metzm what is your opinion on this?

@metzm
Copy link
Contributor

metzm commented Nov 12, 2024

I agree that using r.compress to test different compression methods should be sufficient.

The LZ4 API is very similar, often identical to the APIs of the other compression methods used in GRASS. I am not expecting any changes here in the default functions as used by us because the have been established for many years in different projects.

We are having a local copy because the code base is so small and consists of only two files. Therefore I decided back then to simply add these two files to lib/gis/ instead of adding another external dependency to the Make system.

@nilason nilason added this to the 8.5.0 milestone Nov 14, 2024
@nilason nilason merged commit fa990b2 into OSGeo:main Nov 14, 2024
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C Related code is in C libraries
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants