Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cross platform support (e.g. with CMake) #77

Open
wants to merge 36 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
875edb4
Initial commit
neurolabusc Dec 11, 2019
e45a874
Typo
neurolabusc Dec 11, 2019
788bce3
CMake (not yet working)
neurolabusc Dec 14, 2019
dd48942
Reorganize project structure.
ningfei Dec 17, 2019
691089a
Update .gitignore.
ningfei Dec 17, 2019
2bc38ca
Fix Superbuild.
ningfei Dec 17, 2019
05e63d1
Fix warning due to signed integer in malloc size.
ningfei Dec 17, 2019
c1f8a97
Merge pull request #1 from ningfei/master
neurolabusc Dec 17, 2019
28dd6a9
Updated notes
neurolabusc Dec 18, 2019
93ef899
Option for zlib-ng build
neurolabusc Jan 29, 2020
b11156c
Support for zlib-ng
neurolabusc Jan 31, 2020
02ef1d7
Initial Windows support (https://github.com/madler/pigz/issues/62), s…
neurolabusc Feb 1, 2020
5a9bc91
Clean up Windows build
neurolabusc Feb 5, 2020
5373336
Add support for Intel GZip
neurolabusc Feb 6, 2020
f1e4cf9
Experimental Intel zlib support
neurolabusc Feb 13, 2020
cd70fd6
Merge pull request #2 from neurolabusc/windows
neurolabusc Feb 13, 2020
8d9e823
MSVC WCHAR -> utf8 (https://github.com/madler/pigz/issues/62)
neurolabusc Feb 15, 2020
0d0abad
Merge pull request #3 from neurolabusc/windows
neurolabusc Feb 15, 2020
02b01f0
MSVC modifications
neurolabusc Feb 17, 2020
0b7b407
Chromium zlib
neurolabusc Feb 18, 2020
f60c05b
Download zlib and pthreads on demand for MSVC
neurolabusc Feb 19, 2020
ce36332
Merge pull request #4 from neurolabusc/windows
neurolabusc Feb 19, 2020
baf468a
MinGW64 support for non-Latin characters (https://github.com/madler/p…
neurolabusc Feb 22, 2020
add838c
Change project name in top CMakelists.
ningfei Feb 23, 2020
dbf845e
Add pthreads4w support for MSVC.
ningfei Feb 23, 2020
54c7a2a
Cosmetic, fix indent.
ningfei Feb 23, 2020
bd71d41
Update pthreads4w branch name.
ningfei Feb 23, 2020
c030953
Use https://github.com/ningfei/zlib.git fior now.
ningfei Feb 23, 2020
7665a2c
Fix LDFLAG for MinGW.
ningfei Feb 23, 2020
84b918e
Statically link to libwinpthread when using MinGW.
ningfei Feb 23, 2020
d6bd6f4
Fix file permission.
ningfei Feb 23, 2020
024ebb6
Merge pull request #5 from ningfei/master
neurolabusc Feb 25, 2020
b4b2cc0
Improve documentation
neurolabusc Feb 25, 2020
f172675
Remove Intel zlib support due to broken CMake (https://github.com/jtk…
neurolabusc Feb 27, 2020
d9ae7e3
Remove option for unsupported zlib variants
neurolabusc Feb 28, 2020
3b592d7
Revert makefile. Use cmake for Windows build
neurolabusc Mar 10, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
*.o
/pigz
/unpigz
/pigzj
/pigzt
/pigzn
/build/
/bin/
/src/pigz.*
/src/unpigz.*
/src/pigzj.*
/src/pigzt.*
/src/pigzn.*
.DS_Store
9 changes: 9 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
cmake_minimum_required(VERSION 3.1.0)

if(COMMAND CMAKE_POLICY)
CMAKE_POLICY(SET CMP0003 NEW)
endif()

project(pigz)

include(${CMAKE_SOURCE_DIR}/SuperBuild/SuperBuild.cmake)
47 changes: 0 additions & 47 deletions README

This file was deleted.

92 changes: 92 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
## About


[pigz](https://zlib.net/pigz/), which stands for parallel implementation of gzip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data. pigz was written by [Mark Adler](https://en.wikipedia.org/wiki/Mark_Adler), and uses the [zlib](https://zlib.net) and [pthread](https://en.wikipedia.org/wiki/POSIX_Threads) libraries. To compile and use pigz, please read the README file in the source code distribution. You can read the pigz manual page [here](https://zlib.net/pigz/pigz.pdf).

Most users can install pigz with a simple command (Debian: `sudo yum install pigz`, RedHat: `apt-get install pigz`, MacOS: `brew install pigz`).

This repository provides the source code, allowing users to build their own executable. One reason to build your own copy is to improve performance (see CMake notes below). An additional reason is to ensure pigz is build with a current version of glibc. Specifically, pigz can generate an `internal threads error` on [Linux servers with old versions of glibc](https://github.com/madler/pigz/issues/68). If you see these errors you should upgrade your glibc library and recompile pigz.

## Compiling with Make

This is the simplest way to compile pigz. Type "make" in the source directory ("pigz/src") to build the "pigz" executable. You can then install the executable wherever you like in your path (e.g. /usr/local/bin/). Type "pigz" to see the command help and all of the command options.

## Compiling with CMake

Compiling with CMake is more complicated than using Make. However, it does allow you to use different variants of the zlib compression library that can [improve](https://github.com/neurolabusc/pigz-bench) performance. Compiling with CMake requires the computer to have CMake and git installed. By default, CMake will use the [CloudFlare zlib](https://github.com/cloudflare/zlib) which provides outstanding compression performance:

```
git clone https://github.com/madler/pigz
cd pigz
mkdir build && cd build
cmake ..
make
```

Alternatively, you can build for your system zlib, which will likely provide the poorest performance (but is the most popular so least likely to have any issues):

```
git clone https://github.com/madler/pigz
cd pigz
mkdir build && cd build
cmake -DZLIB_IMPLEMENTATION=System ..
make
```

Finally, you can build for [zlib-ng](https://github.com/zlib-ng/zlib-ng). At the moment this provides compression performance between CloudFlare and System, but provides exceptionally fast decompression:

```
git clone https://github.com/madler/pigz
cd pigz
mkdir build && cd build
cmake -DZLIB_IMPLEMENTATION=ng ..
make
```

Note that the process is a little different if you are using the Windows operating system. Windows users can compile using the [Microsoft C Compiler](https://visualstudio.microsoft.com/downloads/) or [MinGW]( http://mingw-w64.org/doku.php). Be aware there are several variations of the MinGW compiler, and the CMake script expects a version that supports the [-municode linker flag]( https://sourceforge.net/p/mingw-w64/wiki2/Unicode%20apps/). This flag is required to handle non-Latin letters in filenames. Here is an example of compiling on Windows targeting the Cloudflare zlib:

```
git clone https://github.com/madler/pigz
cd pigz
mkdir build
cd build
cmake -DZLIB_IMPLEMENTATION=Cloudflare ..
cmake --build . --config Release
```

## Details

pigz 2.4 (26 Dec 2017) by Mark Adler

pigz, which stands for Parallel Implementation of GZip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data.

pigz was written by Mark Adler and does not include third-party code. I am making my contributions to and distributions of this project solely in my personal capacity, and am not conveying any rights to any intellectual property of any third parties.

This version of pigz is written to be portable across Unix-style operating systems that provide the zlib and pthread libraries.

Type "make" in the source directory ("pigz/src") to build the "pigz" executable. You can then install the executable wherever you like in your path (e.g. /usr/local/bin/). Type "pigz" to see the command help and all of the command options.

The latest version of pigz can be found at http://zlib.net/pigz/ . You need zlib version 1.2.3 or later to compile pigz. zlib version 1.2.6 or later is recommended, which reduces the overhead between blocks. You can find the latest version of zlib at http://zlib.net/. You can look in pigz.c for the change history.

Questions, comments, bug reports, fixes, etc. can be emailed to Mark at his address in the license below.

The license from pigz.c is copied here:

This software is provided 'as-is', without any express or implied
warranty. In no event will the author be held liable for any damages
arising from the use of this software.

Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:

1. The origin of this software must not be misrepresented; you must not
claim that you wrote the original software. If you use this software
in a product, an acknowledgment in the product documentation would be
appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be
misrepresented as being the original software.
3. This notice may not be removed or altered from any source distribution.

Mark Adler
[email protected]
14 changes: 14 additions & 0 deletions SuperBuild/External-CLOUDFLARE-ZLIB.cmake
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
set(CLOUDFLARE_BRANCH gcc.amd64) # Cloudflare zlib branch

ExternalProject_Add(zlib
GIT_REPOSITORY "${git_protocol}://github.com/cloudflare/zlib.git"
GIT_TAG "${CLOUDFLARE_BRANCH}"
SOURCE_DIR cloudflare-zlib
BINARY_DIR cloudflare-zlib-build
CMAKE_ARGS
-Wno-dev
-DCMAKE_BUILD_TYPE:STRING=${CMAKE_BUILD_TYPE}
-DCMAKE_INSTALL_PREFIX:PATH=${DEP_INSTALL_DIR}
)

set(ZLIB_ROOT ${DEP_INSTALL_DIR})
19 changes: 19 additions & 0 deletions SuperBuild/External-PTHREADS4W.cmake
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
set(PTHREADS4W_TAG cmakebuild)

set(CLEANUP_STYLE VC)
set(PTW32_VER 3)

ExternalProject_Add(pthreads4w
GIT_REPOSITORY "${git_protocol}://github.com/ningfei/pthreads4w.git"
GIT_TAG "${PTHREADS4W_TAG}"
SOURCE_DIR pthreads4w
BINARY_DIR pthreads4w-build
CMAKE_ARGS
-Wno-dev
-DCMAKE_BUILD_TYPE:STRING=${CMAKE_BUILD_TYPE}
-DCMAKE_INSTALL_PREFIX:PATH=${DEP_INSTALL_DIR}
-DCLEANUP_STYLE:STRING=${CLEANUP_STYLE}
)

set(PTHREADS4W_INCLUDE_DIRS ${DEP_INSTALL_DIR}/include)
set(PTHREADS4W_LIBRARIES ${DEP_INSTALL_DIR}/lib/libpthread${CLEANUP_STYLE}${PTW32_VER}.lib)
18 changes: 18 additions & 0 deletions SuperBuild/External-ZLIBng.cmake
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
set(NG_BRANCH develop) # zlib-ng branch

ExternalProject_Add(zlib
GIT_REPOSITORY "${git_protocol}://github.com/zlib-ng/zlib-ng.git"
GIT_TAG "${NG_BRANCH}"
SOURCE_DIR ng-zlib
BINARY_DIR ng-zlib-build
BUILD_BYPRODUCTS ${ZLIB_STATIC_LIBRARIES}
CMAKE_ARGS
-Wno-dev
-DCMAKE_BUILD_TYPE:STRING=${CMAKE_BUILD_TYPE}
-DCMAKE_INSTALL_PREFIX:PATH=${DEP_INSTALL_DIR}
-DZLIB_COMPAT:STRING=ON
-DBUILD_SHARED_LIBS:STRING=OFF
)

set(CMAKE_FIND_LIBRARY_SUFFIXES .a)
set(ZLIB_ROOT ${DEP_INSTALL_DIR})
106 changes: 106 additions & 0 deletions SuperBuild/SuperBuild.cmake
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Check if git exists
find_package(Git)
if(NOT GIT_FOUND)
message(FATAL_ERROR "Cannot find Git. Git is required for Superbuild")
endif()

# Use git protocol or not
option(USE_GIT_PROTOCOL "If behind a firewall turn this off to use http instead." OFF)
if(USE_GIT_PROTOCOL)
set(git_protocol "git")
else()
set(git_protocol "https")
endif()

# Basic CMake build settings
if(NOT CMAKE_BUILD_TYPE)
set(CMAKE_BUILD_TYPE "Release" CACHE STRING
"Choose the type of build, options are: Debug Release RelWithDebInfo MinSizeRel." FORCE)
set_property(CACHE CMAKE_BUILD_TYPE PROPERTY STRINGS "Debug;Release;RelWithDebInfo;MinSizeRel")
endif()
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)

option(USE_STATIC_RUNTIME "Use static runtime" ON)

if(USE_STATIC_RUNTIME)
if(${CMAKE_CXX_COMPILER_ID} STREQUAL "GNU")
find_file(STATIC_LIBCXX "libstdc++.a" ${CMAKE_CXX_IMPLICIT_LINK_DIRECTORIES})
mark_as_advanced(STATIC_LIBCXX)
if(NOT STATIC_LIBCXX)
unset(STATIC_LIBCXX CACHE)
# Only on some Centos/Redhat systems
message(FATAL_ERROR
"\"USE_STATIC_RUNTIME\" set to ON but \"libstdcxx.a\" not found! \
\"yum install libstdc++-static\" to resolve the error.")
endif()
endif()
endif()

include(ExternalProject)

set(DEPENDENCIES)

option(INSTALL_DEPENDENCIES "Optionally install built dependent libraries for future use." OFF)

if(INSTALL_DEPENDENCIES)
set(DEP_INSTALL_DIR ${CMAKE_INSTALL_PREFIX})
else()
set(DEP_INSTALL_DIR ${CMAKE_BINARY_DIR})
endif()

if(MSVC)
message("-- Use pthreads4w for MSVC")
message("-- Will build pthreads4w from github")
include(${CMAKE_SOURCE_DIR}/SuperBuild/External-PTHREADS4W.cmake)
list(APPEND DEPENDENCIES pthreads4w)
endif()

set(ZLIB_IMPLEMENTATION "Cloudflare" CACHE STRING "Choose zlib implementation.")
set_property(CACHE ZLIB_IMPLEMENTATION PROPERTY STRINGS "Cloudflare;System;ng;Custom")
if(${ZLIB_IMPLEMENTATION} STREQUAL "Cloudflare")
message("-- Build with Cloudflare zlib: ON")
include(${CMAKE_SOURCE_DIR}/SuperBuild/External-CLOUDFLARE-ZLIB.cmake)
list(APPEND DEPENDENCIES zlib)
set(BUILD_CLOUDFLARE-ZLIB TRUE)
message("-- Will build Cloudflare zlib from github")
elseif(${ZLIB_IMPLEMENTATION} STREQUAL "ng")
message("-- Build with zlib-ng: ON")
include(${CMAKE_SOURCE_DIR}/SuperBuild/External-ZLIBng.cmake)
list(APPEND DEPENDENCIES zlib)
set(BUILD_NG-ZLIB TRUE)
message("-- Will build zlib-ng from github")
elseif(${ZLIB_IMPLEMENTATION} STREQUAL "Custom")
set(ZLIB_ROOT ${ZLIB_ROOT} CACHE PATH "Specify custom zlib root directory.")
if(NOT ZLIB_ROOT)
message(FATAL_ERROR "ZLIB_ROOT needs to be set to locate custom zlib!!!")
endif()
endif()

set(CMAKE_ARGS
-Wno-dev
--no-warn-unused-cli
-DCMAKE_BUILD_TYPE:STRING=${CMAKE_BUILD_TYPE}
-DCMAKE_INSTALL_PREFIX:PATH=${CMAKE_BINARY_DIR}
-DCMAKE_C_FLAGS:STRING=${CMAKE_C_FLAGS}
-DCMAKE_CXX_FLAGS:STRING=${CMAKE_CXX_FLAGS}
-DCMAKE_VERBOSE_MAKEFILE:BOOL=${CMAKE_VERBOSE_MAKEFILE}
-DUSE_STATIC_RUNTIME:BOOL=${USE_STATIC_RUNTIME}
-DZLIB_IMPLEMENTATION:STRING=${ZLIB_IMPLEMENTATION}
-DZLIB_ROOT:PATH=${ZLIB_ROOT})

if(MSVC)
list(APPEND CMAKE_ARGS
-DPTHREADS4W_INCLUDE_DIRS:PATH=${PTHREADS4W_INCLUDE_DIRS}
-DPTHREADS4W_LIBRARIES:STRING=${PTHREADS4W_LIBRARIES})
endif()

ExternalProject_Add(pigz
DEPENDS ${DEPENDENCIES}
DOWNLOAD_COMMAND ""
SOURCE_DIR ${CMAKE_SOURCE_DIR}/src
BINARY_DIR pigz-build
CMAKE_ARGS ${CMAKE_ARGS}
)

install(DIRECTORY ${CMAKE_BINARY_DIR}/bin/ DESTINATION bin
USE_SOURCE_PERMISSIONS)
Loading