[RFC] DDlog Packages and Rust integration

[ryzhyk]

[RFC] DDlog Packages and Rust integration

This RFC proposes a systematic way to organize DDlog code into packages and map
these packages into Rust crates. As part of this, we propose a design for
integrating native Rust code into a DDlog project.

Motivation

We address the following limitations of DDlog-1:

• DDlog-1 does not have the notion of a project or package. Starting with the
  user-specified top module, the compiler finds all its transitive dependencies and
  generates a Rust project by automatically partitioning these dependencies into
  crates based on heuristics. This approach has proved flawed in multiple ways.

  • Suboptimal crate structure (with too many of too few crates).
  • Configuration options, e.g., library paths, must be specified as CLI arguments
    to the DDlog compiler.
  • The lack of a project metadata file complicates IDE integration.
  • It is near-impossible to write native Rust code that depends on definitions
    from other modules, as these modules may end up in arbitrary crates.
  • It is hard to distribute DDlog libraries
  • Managing dependencies on extern Rust crates is tricky. One must make sure
    that at most one DDlog module imports each Rust crate in its module.toml
    file.

• Native Rust code is not self-contained. A .dl module can be accompanied by
  a .rs file that implements extern functions and types declared in this module.
  The contents of the .rs file is concatenated with DDlog-generated Rust code and
  copied to the generated Rust project. This leads to poor ergonomics, as
  developers cannot use standard Rust development tools to write these files.
  Moreover, Rust compiler errors point to locations in the generated file that must be
  manually mapped back to the original Rust code.

Packages

DDlog-2 code is organized in packages. Like a Rust crate, a package
consists of a tree of modules and a metadata file specifying package
dependencies. There are two types of modules: DDlog modules and native
Rust modules. The DDlog-2 compiler converts the package into a Rust crate
by generating Cargo.toml for the package and a Rust module module.rs for
each DDlog module module.dl. Native Rust modules are included in the Rust
project as is, and in place, so that Rust compiler messages point to actual
source code locations.

Package structure

A DDlog package looks a lot like a Rust crate. The package.toml file in the
root directory contains package metadata: name, version, description, etc., path
to the main module (e.g., lib.dl), and dependencies. A package can have two
kinds of dependencies: Rust crates and other DDlog packages. The former can
point to crates.io, git repository, or local folder, the latter can initially only point
to a local folder or a git repo, but it should be possible to implement support
for both git repositories and for crates.io in the future (see below).

my_package/
├── package.toml
└── src/
    ├── lib.dl
    ├── mod1.dl
    ├── mod2.dl
    ├── mod2.rs
    └── mod3/
        └── mod.dl

A module can consist of a single file or a file tree. In the above example,
mod1.dl is a single-file DDlog module, and mod2.rs is a single-file Rust
module. mod2.dl contains DDlog bindings for Rust definitions
in mod2.rs. This file can only contain function prototypes without
implementation and type definitions (see discussion of extern types below).
Ideally, this file should be generated automatically from mod2.rs, but we may
want to leave this for future work.

Similar to lib and bin crates in Rust, we may want to distinguish library
packages and executable packages, where only the latter can be used to
instantiate a dataflow.

Generated code structure

In contrast to DDlog-1, we place the generated Rust code under the package directory.
We generate Cargo.toml in the top-level folder. Each .dl module is compiled to a
Rust module and stored in the src_rs directory that mirrors the module structure
of the DDlog package. Native Rust modules remain unmodified at their original location.
Rust's #[path] attribute is used to link the native modules to the generated
Rust project:

my_package/
├── Cargo.toml
├── package.toml
├── src/
│   ├── lib.dl
│   ├── mod1.dl
│   ├── mod2.dl
│   ├── mod2.rs
│   └── mod3/
│       └── mod.dl
└── src_rs/
    ├── lib.rs
    ├── mod1.rs
    └── mod3/
        └── mod.rs

Native types and functions

As discussed above, a DDlog package can contain native modules implemented in Rust.
A native module is accompanied by a .dl file that declares DDlog bindings for
types, functions, and trait implementations exported by the Rust module, e.g.,

/// Function signature (no implementation).
pub fn f<T: Ord>(arg: T) -> bool;

/// `impl` block consisting of function signatures only.
impl MyStruct {
    fn f1(self) -> bool;
}

/// Trait `impl` without a body.
impl Ord for MyStruct;

/// OpaqueType can be declared in Rust as a struct, enum, or alias.
type OpaqueType;

We sometimes want to expose Rust types like Option and Result to DDlog not as
opaque types but as structs or enums with constructors. In DDlog-1 we did so by
re-declaring the types in DDlog and implementing conversion functions to/from
std and ddlog_std versions of the type. This did not exactly improve Rust
developers' experience.

We therefore propose that in DDlog-2 one can use Rust structs and enums directly
by describing the structure of the struct or enum to the DDlog compiler. Consider
the following native Rust module and accompanying DDlog binding that expose the
Rust Option type to DDlog:

/// option.rs

// Re-export Rust `Option` type to DDlog
pub use std::option::Option;

/// option.dl

// Instead of declaring `Option` as an opaque type, tell DDlog about its constructors.
// DDlog knows that `module.dl` contains bindings for native Rust code in `module.rs` and
// will not generate a duplicate Rust definition for this type.
pub enum Option<T> {
    None,
    Some(T)
}

Distributing DDlog package via crates.io

One advantage of the proposed design is that DDlog packages map directly to Rust crates and
can be distributed as such. This has dual benefits: DDlog developers can use the crates
ecosystem for software distribution and dependency management; conversely, Rust developers
can easily incorporate DDlog libraries in their programs. Since crates.io doesn't support DDlog
package file format, one must include the generated Cargo.toml file in the distribution.
The next question is whether we need to distribute generated Rust sources along with .dl
files. This is probably undesirable and can be avoided by using a build.rs that invokes
the DDlog compiler to convert DDlog sources to Rust at build time.

[mihaibudiu]

Frankly this proposal is great. If you can achieve all this, it's fabulous.
But are there some obstacles which could make this difficult? You mentioned circular dependencies between modules.
Do we have workarounds for such obstacles?
Do we need circular module dependencies?

[mihaibudiu]

Adding traits to existing types is another one that comes to mind.

[ryzhyk]

You mentioned circular dependencies between modules.

Circular module dependencies are not a problem, as they are allowed by Rust.
Circular crate dependencies on the other hand are illegal. Currently the compiler automatically splits the module dependency graph into SCCs to form crates. In DDlog-2 the programmer will be responsible for this, which I think is better in practice.

Adding traits to existing types is another one that comes to mind.

That's a good point. This will still require wrapper types, unless you control either the crate that declares the type of the crate that declares the trait.