v0.3.0

These are the pre-built PackSquash executables for x64 Linux, macOS and Windows, and AArch64/ARM64 Linux.

For information about how to install and use PackSquash, please see the installation guide and the options files documentation.

Options files for v0.2.1 or v0.3.0-rc.1 are incompatible with v0.3.0.

📌 Changelog from v0.2.1

Additions

• Text files used by Minecraft that contain the End Poem, credits and main screen splashes are now supported.
• Initial support for data packs was added, by handling .nbt and .mcfunction files. Currently, these data pack specific files are just copied to the generated ZIP file, but improved file-specific optimization techniques will be introduced in the future.
• JSON files (.json, .mcmeta and, if OptiFine is allowed, .jem and .jpm) can now contain Unix shell style line comments (# line comment), C style block comments (/* block comment */), and C style line comments (// line comment). Comments are ignored and removed from the optimized JSON that is stored in the ZIP file. JSON with comments files (.jsonc) are now supported, and their extension is canonicalized to .json in the generated ZIP file, as Minecraft expects.
• Pack file names can now at least contain every character in the Unicode Basic Multilingual Plane. This is a substantial improvement from the previous limitation of only ASCII characters.
• The pack.mcmeta file is now validated for egregious errors by default, like missing pack_format or description keys. This validation can be disabled if necessary.
• An APT repository was created, to make it easier to install PackSquash and keep it updated on Debian-like Linux distros on the officially supported architectures. This is now the recommended method to install PackSquash on distros that can use this repository, like Debian and Ubuntu.

Compression improvements

• The PackSquash ZIP compressor was greatly revamped to allow for identical file deduplication and a greater range of ZIP specification conformance levels, so you can now choose the right balance between optimization and interoperability for your needs in a more precise way.
• Key-value pairs in JSON files that are known to be ignored by Minecraft, but usually present anyway, are now removed. This behavior can be disabled.
• The number of Zopfli compression iterations applied to pack files to store them in the generated ZIP file is now variable and configurable. This is useful to customize the desired tradeoff between compression and performance. The precise number of iterations done to a file depends on its size, with smaller files being compressed more, and larger files being compressed less, so performance is more predictable and consistent. As packs used with PackSquash tend to contain lots of small files but only a handful of big ones that are not already compressed, this new behavior can yield more space savings.
• PNG files can now be quantized to 2, 4 and 16 colors, in addition to the previously hardcoded, but still default, 256 colors.
• Improved shader minification. The exact savings depend on the input file, with bigger and more intrincate files being subject to more savings, but some test cases have shown a more than 10% optimized size decrease, compared to the previous shader minification technique.
• The generated ZIP files can now make use of ZIP64 extensions when needed to overcome certain limits of the original ZIP file format, like a maximum of 65535 files, or a total of less than 4 GiB of compressed data. Note that this ZIP64 support does not improve the per-file 4 GiB limit, because such big files do not make sense to use with Minecraft.

Performance improvements

• ZIP files that PackSquash has generated in previous runs can now be automatically used to vastly speed up next runs, as unchanged files can just be copied from that ZIP file instead of being processed again. PackSquash relies on the file modification timestamps provided by the filesystem to quickly deduce if a file may have been changed.
• Gigantic images (> 8192x8192) are now rejected by default, because they usually make little sense to use with Minecraft, unless they are atlases or animated textures. Limiting this ensures that PackSquash does not take too long processing them and helps authoring resource packs that work well accross a wide range of maximum GPU texture sizes.
• Images that have a lot of pixels are now optimized much faster, thanks to an adaptative compression strategy that calculates an appropriate number of Zopfli compression iterations beforehand to meet acceptable performance targets. This behavior can be tuned to be even faster in exchange for worse compression, or prioritize compression at the expense or performance.
• Buffering is now used by default in a way that is better adapted to the device PackSquash runs on and can be configured by the user. This substantially improves performance when dealing with big (> 64 MiB) packs if there is plenty of available memory, while also improving stability in environments with low available memory. When a buffer gets rolled over to disk because the available memory is exhausted, now it will be replaced by a much smaller buffer, so performance does not drop as much as it did before.
• PackSquash is now designed around a multithreaded asynchronous programming model, as opposed to the previously used multithreaded synchronous programming model. This new model leverages cooperative task scheduling to improve thread utilization. Some internal tests showed that the total execution time was reduced by ≈5%, but this figure may change depending on the contents of your pack and the device you run PackSquash on.
• Lots of micro-optimizations in regards to both memory and CPU usage.

Extraction protection improvements

• Now PackSquash uses much more sophisticated extraction protection techniques, with additional configuration knobs that allow tuning the generated ZIP file for increased compressibility or obfuscation. These extraction protection techniques were possible to implement thanks to original research on Minecraft internals, in addition to suggestions and ideas from some Discord users.

Fixes

• Allowing the OptiFine mod now properly deals with the Custom Entity Models feature, which adds .jem and .jpm files to resource packs. Previously these files were treated as unknown junk files and skipped.
• Fixed issue #12, which made grayscale-looking color images look wrong on older versions of Minecraft, by working around the underlying Minecraft quirk.
• Input PNG files with embedded gamma information are now corrected to a display gamma value of 2.2, which closely matches that of the sRGB color space, because this is the gamma that Minecraft and most other end-user applications and devices assume and use. Previously, input images with an unconventional gamma would have that gamma information stripped without their colors being corrected, which led to wrong-looking results down the line.
• Fixed issue #13 as much as possible.
• JSON, shader and OptiFine-added text files with a UTF-8 BOM now work with PackSquash. The BOM is routinely added to UTF-8 text files by some editors, like Windows Notepad, but it serves no purpose within a pack, so it is now ignored and removed.
• Audio files are no longer downmixed to mono by default, because the different 3D effects that downmixing may imply can be inappropriate, and doing so did not yield very significant space savings anyway, due to Ogg Vorbis joint encoding.
• Shader files with Windows (CR + LF) line endings now work fine with PackSquash.
• Shaders that contain the #elif preprocessor directive can now be parsed and optimized by PackSquash. The non-standard #elseif preprocessor directive is no longer accepted.
• Fixed issue #34 about the #moj_import preprocessor directive added in Minecraft 1.17 not being supported in shaders.
• Some GLSL statements are no longer broken by PackSquash optimizations.
• The OxiPNG library is now told to stop trying further optimizations on images if the process has been running for more than ten minutes. This deadline is just a best-effort.

Stability improvements

• PackSquash' software architecture was reworked to support dealing with files larger than the available memory. Nevertheless, the usefulness of this change is limited in practice by libraries that require the entire file to be addressable in main memory to be processed. Currently, only audio and copied files can be processed without storing their entire contents in memory.
• Now PackSquash aborts its execution if an error occurs while processing the pack, instead of continuing and risking generating corrupt or incomplete ZIP files. This behavior also allows using PackSquash as a pack validator more efficiently.
• A configurable limit of simultaneously open files was introduced. This avoids errors caused by the open file limit imposed by the operating system being reached, which can happen in big packs if lots of threads are used.

Codebase changes

• The entire codebase was refactored to be much more modular, testable, and maintainable. In particular, the application and logic code were separated in two binary and library crates, respectively, promoting separation of concerns and paving the way for a future API.
• Automated unit tests were added for the most critical components of PackSquash. This increases the confidence that those components work okay, and continue to work fine after changes. These tests are run automatically for each build.
• Improved build reproducibility by checking in to the repository the Cargo.lock file.
• Improved the GitHub Actions workflow, fixing semantic version calculations, build artifacts caching, and other changes.
• Visual Studio Code configuration files were checked in to the repository. This should help new contributors configure their development environment more easily.
• An .editorconfig file was added to improve code readability in the GitHub website.
• Updated libraries and toolchain components for the latest fixes and improvements.

Miscellaneous

• JSON files can now be prettified instead of minified before storing them in the generated ZIP file in a per-file basis. This may be useful for packs under a permissive license that wants to encourage contributions from its users. Similarly, minification of shader and .properties files can now be disabled.
• Opus audio files are now supported as input files in resource packs. They are transcoded to Ogg so that Minecraft can read them.
• Several tidy-ups and improvements to the text that PackSquash outputs. The error message shown when the output path file was a folder was improved to be much more user-friendly.
• Detailed version, author and license information is now shown on startup. More detailed information can be seen with the -v command line switch.
• The hardcoded list of system files that are ignored when ignore_system_and_hidden_files is set to true was expanded and improved.
• Progress messages are now printed to the standard error stream, following the trend and conventions established by other command line applications.
• Official builds for the Linux AArch64/ARM64 platform are now available. Even though these builds should work okay, support for this platform is limited to a best-effort, because PackSquash has not actually been run on this platform on the device of a repository maintainer.
• Revamped the panic handling code to output prettier and more user-friendly error and backtrace information, without compromising its technical usefulness.
• Now PackSquash shows, under certain circumstances, URLs pointing to contextually relevant documentation.
• PackSquash now exits with the status code 128 if any error happens while processing a pack. If the error was caused by a ZIP file that PackSquash attempts to reuse but it is not able to, the returned status code is 129.
• Information for sponsoring PackSquash developers was added to the README.md file. A FUNDING.yml file was also added so that GitHub displays a relevant "Sponsor" button in its web interface.

Options files changes

• Settings files have been renamed to options files.
• The following options are new (i.e. they were introduced in v0.3.0 and do not have an exact equivalent in v0.2.1):
  • zip_compression_iterations
  • work_around_minecraft_quirks
  • size_increasing_zip_obfuscation
  • percentage_of_zip_structures_tuned_for_obfuscation_discretion
  • never_store_squash_times
  • spooling_buffers_size
  • open_files_limit
  • zip_spec_conformance_level
  • validate_pack_metadata_file
  • automatic_minecraft_quirks_detection
  • color_quantization_target
  • image_data_compression_iterations
  • skip_alpha_optimizations
  • minify_json
  • delete_bloat_keys
  • maximum_width_and_height
  • minify_shader
  • minify_properties
• The following options were superseded:
  • strict_zip_spec_compliance, by zip_spec_conformance_level
  • quantize_image, by color_quantization_target
• The following options were renamed, with no change in functionality or values they accept:
  • resource_pack_directory → pack_directory
  • compress_already_compressed_files → recompress_compressed_files
  • allowed_mods → allow_mods

📌 Changelog from v0.3.0-rc.1

Additions

• Text files used by Minecraft that contain the End Poem, credits and main screen splashes are now supported.
• The set of Minecraft quirks that PackSquash needs to work around is now automatically detected by default. It is still possible to manually specify the set of quirks via the options file, though.
• The pack.mcmeta file is now validated for egregious errors by default, like missing pack_format or description keys. This validation can be disabled if necessary.
• An APT repository was created, to make it easier to install PackSquash and keep it updated on Debian-like Linux distros on the officially supported architectures. This is now the recommended method to install PackSquash on distros that can use this repository, like Debian and Ubuntu.
• Introduced a stick parity bit in the Squash Time storage format, to probabilistically detect whether a previously generated ZIP file can indeed be reused. For realistic packs, the detection works reliably and avoids most pitfalls related to using this feature effectively.

Compression improvements

• Improved shader minification. The exact savings depend on the input file, with bigger and more intrincate files being subject to more savings, but some test cases have shown a more than 10% optimized size decrease, compared to the previous shader minification technique.

Performance improvements

• Images that have a lot of pixels are now optimized much faster, thanks to an adaptative compression strategy that calculates an appropriate number of Zopfli compression iterations beforehand to meet acceptable performance targets. This behavior can be tuned to be even faster in exchange for worse compression, or prioritize compression at the expense or performance.
• Minor micro-optimizations in regards to memory usage.

Fixes

• ZIP files generated when the zip_spec_conformance_level option is set to disregard no longer fail to load sometimes with older Minecraft versions, if PackSquash is instructed to work around a newly-added quirk. (Thanks Dvalin#5304 for reporting this over Discord)
• Shaders that contain the #elif preprocessor directive can now be parsed and optimized by PackSquash. The non-standard #elseif preprocessor directive is no longer accepted.
• Fixed issue #34 about the #moj_import preprocessor directive added in Minecraft 1.17 not being supported in shaders.
• Some GLSL statements are no longer broken by PackSquash optimizations.
• PNG file optimization errors caused by exceeding the configured image width and height limits now show a proper error message, instead of a totally unrelated one like "invalid bit depth".
• Pack directory relative paths that started with . and .. are now properly dealt with, like they were on v0.2.1.
• Fixed the last Windows service pack install date not being considered a volatile system identifier as it was intended.
• PackSquash for Linux now works on distros that ship an slightly older version of the GNU C library, like Debian Buster, as v0.2.1 did.
• File sizes are no longer used in some cases to decide execution flow, which prevents a class of race conditions from happening in rare circumstances.
• Each PackSquash build no longer has a different application salt for no reason, making the ZIP file reuse feature more effective across builds, as originally intended. Nevertheless, the application salt can still change any time without notice.

Stability improvements

• Pack processing is now interrupted properly even if a panic happens.

Miscellaneous

• Revamped the panic handling code to output prettier and more user-friendly error and backtrace information, without compromising its technical usefulness.
• Several minor dependency updates.
• Removed some development dependencies that were no longer necessary.
• Some strings that PackSquash outputs were tweaked with the intent of making their meaning clearer for all users.
• Now PackSquash shows, under certain circumstances, URLs pointing to contextually relevant PackSquash documentation.
• The .tmp extension was added to the hardcoded list of system files extensions that ignore_system_and_hidden_files uses when set to true.
• PackSquash now exits with the status code 128 if any error happens while processing a pack. If the error was caused by a ZIP file that PackSquash attempts to reuse but it is not able to, the returned status code is 129.

Options files changes

• The default value for maximum_width_and_height was changed to 8192, as a compromise between rejecting too big single textures and allowing PackSquash to work with most animated textures and atlases by default.
• The grayscale_textures_gamma_miscorrection quirk was renamed to grayscale_images_gamma_miscorrection to be more consistent with the names PNG files receive in the documentation.
• Added the java8_zip_parsing quirk.
• The following options were introduced:
  • validate_pack_metadata_file
  • automatic_minecraft_quirks_detection
  • image_data_compression_iterations
  • skip_alpha_optimizations

Thanks to sponsors like Silmarost, who did a fantastic job at motivating PackSquash development; sya-ri, who shared candid feedback and did notable code contributions on the sister GitHub action project; pau101, who submitted a PR in the glsl fork repository that PackSquash uses implementing some shader optimization fixes and improvements, and everyone else who reached out in one way or another to make PackSquash better!