Managing packages which are not part of you own source tree is a troubling task when it needs to work on all platforms and without hassle. For many programming languages there exist tools which handle the search, retrieval and management of packages. For CMake there exists something rudimentary like Modules, Config and ExternalProject. However these are do not really manage a dependency graph and or not formulated all too concisely.
I was inspired by great dependency managers like npm and nuget. (Also: maven, pip, apt-get, ...) They really make it easy to use and share packages. Also the ideas behind ryppl were very interesting to me.
I however wanted a dependency manager which works with out-of-the-box cmake and does not have any initial dependencies.
The reason to use cmake as the basis is because of its platform independence and because it can be viewed as the root dependency of everything (you can use cmake to build everything on almost every platform). I coerced cmake into a usable form for a complex dependency manager by providing a lot of missing functionality.
I require decentralized package sources - git, svn, hg,svn, github, bitbucket, archives, remote archives, local folders, .... I do not want a central service as it does not reflect how c++ projects are organized at the moment. Also I want to have the possibility of adding new package sources as the need for them arises (apt-get, brew, convention based (like cmake's module files),... ). These package sources only perform two things: search and retrieval. They take a normalized input in uri form which can identify any package in the world and return metadata and content in a consistent interface. This part of the dependency management is implemented in my package source functions.
The dependency management part is required to be completely separated from the <package source>s. It shall use the metadata provided by the package sources to calculate a dependency configuration (with complex constraints like versioning, incompatibilities, optional dependencies, OS and location based constraints ...) and manages the materialization and dematerialization of these packages while still being easily extensible.
The project functions need to be usable from within any cmake script so that build automization becomes possible while also providing an easy to use and intuitive command line client that the dev can use to control his or her project.
The project functions are based on a project handle which is also a package handle as use by the package sources. The project handle contains all information about a project and needs to be serializable and portable. Also it may be extended by custom data.
To keep the project functions open for extension but closed for modification I chose to use an event system to emit events to which extensions can react and modify the project and package handles according to their requirements. The project lifecycle is defined by these events. The state of the project always needs to be correct which is why I also use a state machine to manage it.
The project lifecycle is at its base very simple with just five states:
When a new project handle is created it is in the unknown state and will first be set to closed before entering the lifecycle. The project handle can only be persisted and read when it is in the closed state otherwise it is considered to be in an inconsistent state.
<project handle> ::= <package handle> v { # see package sources for information
uri: "project:root" # your project is always addressable as "project:root"
content_dir: <absolute path> also called the `project_dir`
project_descriptor: <project descriptor>
package_descriptor: <package descriptor> # see package sources for information
materialization_descriptor: <materialization descriptor> # stores information on were and how data of the package is stored
...
}
The project descriptor contains data which describes the state of the project.
##all <relative path>s are relative to the `project_dir` unless stated otherwise
<project descriptor> ::= {
package_cache: { <admissable uri> : <package handle> } # contains all packages known to project
package_source: <package source> a package source object used to retrieve package metadata and files.
package_materializations: { <package uri> : <package handle> } # contains all materialized packages.
dependency_configuration: { <package_uri> : <bool> } # the currently configured dependencies
dependencies: { <package uri>:<package handle> } # all dependencies that this project has including transient dependencies.
dependency_dir: <relative path = "packages"> # path relative to project root which is used as a the default dependency locations
config_dir: <relative path = ".cps"> # the locations of the configuration folder
project_file: <relative path = ".cps/project.scmake"> # the location of the project's config file
package_descriptor_file: <relative path>? # if specified the path of the package descriptor. This will be read or written when project is opened or closes
...
}
The command line interface wraps these functions and provides an alias which you can use from your console of choice. Its usage is described here:
When a project is opened the following will happen (the ovals are states and the boxes are events that are emitted):
Closing happens in a inverse fashion but will never result in an unknown state:
Two wrappers for open and close exist: project_read and project_write which makes it easier to open a project file and save. It is also noteworthy to say that project close removes any none portable data and project open restores it.
Here is the list of functions
When the project is opened it is possible to modify and work with it. The functions which are available are as follows:
A project can materialize and dematerialize packages which may or may not be dependencies. What this means is that adding or removing a dependency is not the same thing as materializing and dematerializing a dependency. This split between materialization and dependency management may seem strange at first but it allows you more manual control.
The actual dependency manager I implemented is based on every package defining its dependency constraints which are then combined into a large boolean satisfyability problem. The solution to this problem is called the dependency configuration. The big advantage that using a SAT solver over a topological order is that it allows cyclomatic dependendencies and can solve ambiguous dependencies (it actually just has to find one set of dependencies which work). The dependency configuration then a simple map which maps { <package uri> : <bool> }
The following figure illustrates a problem which will be solved by the SAT solver. The packages dependencies specified in the packages all have constraints which can be fulfilled by multiple instances of package B. However there is a consensus candidate which will solve the problem for all packages in the dependency graph:
A package dependency is defined as follows:
<package dependency> ::= { <admissable uri> : <package constraint> }
## example:
{ 'github:toeb/cmakepp':true }
{ 'http://www.cmake.org/files/v3.2/cmake-3.2.1.tar.gz':false }
{ 'bitbucket:toeb/test_repo_hg':null}
{ 'github:toeb/cppdynamic':{ ... } }
These package dependencies can be combined:
<package dependencies> ::= <package dependency> v <package dependency>
A package constraint is defined as follows:
<package constraint> ::= <true> | <false> | <null> | <complex package constraint>
<complex package constraint> ::= {
... values that configure or constrain the packges ...
}
The simple constraints are easy to explain:
truethe dependency is necessary. anypackage uriwhich is identified by theadmissable uriwill fullfill the dependency for the current package.falseanypackage_uriidentified by theadmissable uriis incompatible with the current package.nullanypackage uriidentified byadmissable urican be optionally installed. (same as complex contstraint{is_optional:true}){ ... }complex constraints are not completely implemented yet (onlyis_optional) but I plan to add version constraints, installation location constraints, os dependencies, dependency descriptor modifications. Also may contain any other proeprties which can be used by extensions to configure that particular project/package/package dependency. See for example thepackage symlinkerextension.
In the metadata for every package (ie the package_descriptor)
You
cmakepp listens for the project events and uses them to to provide extra functionality which is described here
package_descriptor.cmakepp.create_files : { <filename> : <filecontent> }all keys specified here will be created in thepackage'scontent_dirwith the specified content. This is useful if you want to define a package completely in apackage descriptorpackage_descriptor.cmakepp.export : <glob ignore expression>includes all the files specified by the glob ignore expression in cmake allowing your package to provide cmake macros and functions to other packages. WARNING cmake only has one function scope so you need to be careful that you do not overwrite any functions which are needed elsewhere. The best practice would be for you to add a namespace string before each function name e.g.mypkg_myfunction.
Hooks are invoked for every package which allows it to react to the project lifecycle more easily. These hooks are called using package_handle_invoke_hook. You can use any function that you defined in your cmakepp.exports (except if stated otherwise) and also specify a file relative to the package's root direcotry.
package_descriptor.cmakepp.hooks.on_loadedcalled after a package and all its dependencies are loaded. You can also load custom data here or setup the project / package.package_descriptor.cmakepp.hooks.on_unloadingcalled when the package is unloaded. You can store all information that you want to keep in thepackage_handle. Or you could use this hook to persist custom datapackage_descriptor.cmakepp.hooks.on_materializedcalled after the package content is available but before the package is loaded. Here you can only specifiy a script file because the exports might not be available (but you can include them yourself)package_descriptor.cmakepp.hooks.on_dematerializingcalled before the package dematerializes. this allows you to perform cleanup before the package content is destroyedpackage_descriptor.cmakepp.hooks.on_runcalled on project package if when command line client is invoked (seecmakepp_project_cli)package_descriptor.cmakepp.hooks.on_readyis invoked when all become ready and the package itself is materializedpackage_descriptor.cmakepp.hooks.on_unreadyis invoked if any dependency becomes unready