`dds` Alpha Release 4 is Available!

Along with the brand new website, the fourth dds alpha version has been posted.

What’s Happened?

Quite a lot has been going on since my last dds alpha release… You’ll have to excuse my tardiness in posting updates.

Besides the new stressors we all find ourselves living with, I’ve been spending a lot of time yak-shaving on dds tasks. (For those that are unaware, yak shaving refers to “any apparently useless activity which, by allowing you to overcome intermediate difficulties, allows you to solve a larger problem.” It is a delightfully useful bit of jargon.)

So what, pray tell, have I been yak-shaving on? dds, of course! The very first important feature that I want to get working is a better package distribution story. At the moment, dds obtains packages by shallow-cloning a Git repository, and the package listing is only stored in a local catalog database: There are no remote catalog/repository capabilities yet.

Unfortunately, I have placed some restrictions on myself that have prevented me from pulling libraries that would otherwise be very useful in achieving this goal (e.g. Boost.Beast for HTTP, libsodium for file signing). I set out on a path to bootstrap a lot of these components myself. A herculean task, to be sure. Am I up to the challenge?

Kind of?

I made a lot of good progress, and I have a lot of useful code to show for my yak-shaving efforts, but I haven’t made enough progress yet that would directly affect dds itself.

This is what one would call “Letting perfect become the enemy of good.”

To this end, I have postponed my yak-shaving efforts in lieu of getting a “good enough” package distribution story in place, where “good enough” means “good enough to be a useful tool.”

What’s New?

The most significant change to dds with alpha 4 is updates to the package catalog.

A Domain

I felt it prudent to give dds a dedicated website and domain. While browsing the options available, one particular name caught my eye, and I simply couldn’t pass up the opportunity:

It is still very early, but documentation and information about dds is now available online at dds.pizza

Catalog Changes

While we don’t yet have a central (or private) software repository solution, dds still allows you to download and build project dependencies through a local catalog database.

The catalog database is a file that enumerates packages (and versions thereof) with information about how they may be obtained. The catalog has been included since Alpha 2, but it has always started out empty. You download dds, but it has no support for any existing packages! Pretty useless, right?

When dds builds a project, it will first solve the dependencies of that project using the local repository and the local catalog. The local repository contains sources for packages that have been downloaded previously, and dds will use them to build from. If a dependency solution identifies a package that is not yet in the local repository, dds will use the information stored in the catalog to obtain a copy of the sources of that package and place it in the local repository, ready to be built. This all happens automatically as part of the build process.

To add entries to the catalog, one must write a JSON5 file that describes the packages and how they are to be obtained. This file is then imported from the command line:

$ dds catalog import --json catalog-data.json5

The available packages can be enumerated with a command:

$ dds catalog list
[...]
spdlog@1.6.0
spdlog@1.6.1
spdlog@1.7.0
tomlpp@1.0.0
tomlpp@1.1.0
tomlpp@1.2.0
[...]

More information can be obtained about any package using catalog show with the package-id:

$ dds catalog show spdlog@1.7.0
Name:     spdlog
Version:  1.7.0
Depends:  fmt@[>=6.0.0]
Git URL:  https://github.com/gabime/spdlog.git
Git Ref:  v1.7.0
Description:
    Fast C++ logging library

There are several other catalog subcommands, but they are not relevant.

Everything mentioned above has been around since Alpha 2. So what’s new?

A Shipping Catalog!

dds uses itself to build itself, and it manages its own dependencies in a catalog as outlined above. The dds repository contains a catalog.json file that is used to populate its own catalog with its dependencies. Until now, that catalog file hasn’t been anything more than part of dds’s CI process.

With dds Alpha 4, upon the initial migrations to the catalog database, dds will seed the catalog with the contents of the catalog.json that is contained in its own repository!

This means that dds will finally be able to download several common packages straight out-of-the-box.

In addition to dds’s own dependencies, several common and useful C and C++ libraries have been added to the catalog.json, even if they aren’t yet used by dds itself (This also means that they are less thoroughly tested). Upon initial start-up, dds will be able to pull the following packages as dependencies:

Abseil
Asio
Boost.LEAF
Boost.mp11
Boost.PFR (AKA magic_get)
Catch2
Cereal
CTRE *
fmt *
Inja
libsodium
lua
magic_enum
ms-wil * (Windows Implementation Library**)
nameof
neo-concepts *
neo-fun *
neo-sqlite3 *
nlohmann-json **
pcg-cpp (PCG random)
pegtl
pubgrub *
pybind11
range-v3 *
semver *
sol2
spdlog *
tomlpp
vob-json5 *
vob-semester *
zlib

Libraries marked * are used directly by dds and are known to integrate correctly. Libraries marked ** are only available in outdated pinned versions. neo-, vob-, and semver libraries are my own.

It’s a modest list so far, but hopefully enough to get something useful started.

These initial entries are only created the first time the local catalog database is created, so users with existing catalog files won’t see them appear.

In such case, adding/reloading these initial entries can be done with the --initial flag:

$ dds catalog import --initial

Automated Transforms

Of the above listed dependencies, many are ready to be built and imported by dds simply by their directory layout matching what dds expects. A few, however, have their own unique layouts that need to be “patched up” so that dds can build and import them.

As part of a transitional period to allow dds to consume libraries that do not yet meet its layout requirements, catalog listings can now contain “filesystem transformations,” that instruct dds how to rearrange the result of git clone such that dds can build and import the libraries therein.

For example, the Lua source tree can be built by dds by simply moving the source files from the top-level directory into a src subdirectory. This is encoded in the catalog.json as:

{
  move: {
    from: ".",
    to: "src/",
    include: ["*.c", "*.h"]
  }
}

More details are included in the updated documentation.

Updating the Catalog

While dds doesn’t have a central package catalog that it knows how to download to update its local copy, the catalog.json that is used to seed the database is available to download. Combined with catalog import, one can import that catalog JSON file in a single line:

$ curl -Ls dds.pizza/ctlg | dds catalog import --stdin

(Windows users can use Invoke-WebRequest in PowerShell for the same effect.)

This “poor-man’s apt-get” isn’t perfect, but it’s good enough for experimenting with an alpha release. A future release will have “for real” remote catalog and package distribution capabilities.

Other Changes

Dependency “Statements”

The depends key was previously a JSON object with keys being package names and values being version range strings, much like they are in NPM’s package.json. This has been changed to depends being an array of strings, with an abbreviated syntax of “dependency statements.” The transform is simple:

{
  depends: {
    foo: "^1.2.3"
  }
}

becomes:

{
  depends: [
    "foo^1.2.3"
  ]
}

Using a JSON object for depends is deprecated and will be removed in the next release.

Similarly, the arguments to build-deps required a space separating the package name and the range string:

$ dds build-deps "foo ^1.2.3"

which was reflected in the CMake PMM module. This is unnecessarily awkward and strange looking. To change this, one must simply remove the whitespace. This will also reduce need for seemingly-arbitrary argument quoting:

$ dds build-deps foo^1.2.3

Test Timeouts

Each test is now permitted ten seconds to execute before it is considered to be stuck, after which it will be terminated.

This time will probably be lowered in a future release.

Parallel Downloads

When dds git-clones dependencies, it will now do so in parallel rather than serially. This can help reduce CI times for projects which would have a large number of dependencies.

Removals

Prior dds alphas used a primitive key/value pair file format. While it worked okay, it was needlessly esoteric. With Alpha 3, these files were replaced with JSON5 configuration files. The old .dds-file support was left in place, but deprecated with a warning. Alpha 4 removes this completely, and dds no longer accepts package.dds/library.dds files. This can be a breaking change if your local repository contains any packages of this shape, and may require that the offending package directories be deleted.

Other Tweaks

The .inc and .ipp file extensions are now recognized as header files.
Compile and link commands are generated using relative paths. This fixes issues with absurdly long command-line strings producing errors from Windows’ CreateProcess.

What’s Next?

There has been a lot of work towards having a real package repository as opposed to simply cloning Git repositories. While the git-clone method can go far, it will become hard to manage, as it requires manually keeping track of metadata in the catalog. Despite this work, it has not yet made its way into a public release.

Implementing remote catalog updates and package repositories is currently at the top of my priorities, so it will receive the most attention.

Once we have a real package catalog, there are several more features that I want to address:

Conditional dependencies (e.g. “I depend on foo only on Windows”)
Dependency toggles/knobs for dependees. (e.g. “Enable encrypted streams in networking library XYZ.”).
Conditional dependencies based on feature toggles (e.g. “I, XYZ, require libsodium if the dependee wants me to provide encryption.”)
Better test framework integrations
Command-line APIs for IDEs and editors

dds Alpha Release 4 is Available!