Build executor with caching and distribution without relying on compiler support thanks to IO detouring
PPB is a parallelised build system written in Go. It supports caching of compiled artifacts, as well as tasks distribution in a cluster of workers over network. Caching and distribution both rely on IO detouring using DLL injection on Windows.
This is an initial release forked from PoPploPopPo Engine, and code still needs to be cleaned out of some old assumptions specific to PPE. Sources for IO detouring DLL library are still hosted on the original repository and you will only find prebuilt binaries for Windows here.
The current version of the toolchain is stronlgy assuming C++ is the main language, but the goal is to provide a language/tool agnostic toolchain supporting painless caching and distribution (could run a DCC to generate assets for instance).
- Data-driven C++ module declaration using JSON
- Modules can have private, public or runtime dependencies
- Generate compile_commands.json and Visual Studio Code workspace
- Build execution is parallelized and dependencies are tracked to guarantee mininal rebuilds
- Supports compilation and link caching with deterministic builds
- Projects can use precompiled headers (PCH) or convert them to C++20 header units, to allow caching and distribution
- Cached compilation results are compressed using facebook zstd or lz4
- Actions can be distributed over a cluster of workers using google QUIC peer-to-peer protocol
- Distribution is decentralized and workload is balanced according to resources available on peers
- Remote workers can directly access local filesystem using a webdav server and IO detouring
- Resources allocated to job distribution can be customized (work idle, N threads, max memory usage)
- Add sources for IOWrapper & IODetouring helpers (only prebuilt binaries)
- Find a better setup for testing distribution (current DockerFile on Windows is very limiting)
- Provide a way to import actions without describing modules with data-driver layer
- Implement Visual Studio solution and project generator
- Test distribution and caching with UnrealBuildTool
- github.com/DataDog/zstd
- github.com/Showmax/go-fqdn
- github.com/djherbis/times
- github.com/goccy/go-json
- github.com/klauspost/compress
- github.com/mholt/archiver/v3
- github.com/minio/sha256-simd
- github.com/pierrec/lz4/v4
- github.com/pkg/profile
- github.com/quic-go/quic-go
- github.com/shirou/gopsutil
- golang.org/x/exp
- golang.org/x/net
- golang.org/x/sys