`dds` Alpha Release 6 is Available!

This is the sixth alpha release of dds, and the final “alpha” release before it enters beta.

Changes in Alpha 6

It’s been a few months, but there’s been some significant improvements.

Feature: Remote Package Repositories!

The largest, most significant, and most fundamental feature of any package managing tool has arrived. This release finally introduces package repositories to dds. The prior methods of package procurement were clunky, only designed to allow dds development to be dogfed, and slow to update and deploy. This package repositories feature required a huge amount of yak-shaving and took the majority of development time for the past year.

The new package repositories work similar to how you would see them in other package managing software: A remote server maintains a listing of packages that it provides, and services clients wishing to download packages. Here’s a rundown of how they work in dds (as of alpha.6):

Package repositories are static content. Any HTTP server that can serve from a filesystem can serve a repository. There is no dynamic data or API on dds repositories. This makes dds repositories trivial to manage as a system administrator.
dds downloads the package listings from a repository as a database of all listings in a single file. dds imports the listings and associates them with the remote repository. It can then be locally browsed, searched, and resolved without communicating with the remote.
dds ships with a new subcommand, repoman, designed to manage repository directories that are then served on HTTP.
Right now there is only one repository by default, at repo-1.dds.pizza. (The name is qualified with a -1 suffix to allow room for the stable dds repository at repo.dds.pizza in the future.) This repository is registered when dds performs its local database migration.
New repositories can be added or removed as needed with the dds pkg repo subcommand. Package listings are updated with dds pkg repo update.

At the moment, repo-1.dds.pizza does not have a huge catalog, but over the coming weeks you can anticipate it growing as time is allocated to fleshing it out.

The packages available for use can be obtained using the dds pkg search subcommand.

Note that although dds can communicate via HTTPS, it does not yet perform TLS certificate validation!

Refer to the documentation on remote packages for more information.

Feature: Cached Compiler Output

When a compiler emits output to stdout/stderr, dds will display that output as a compiler warning. If dds build is executed again without any changes to the source files, compilation of those files is skipped. This is referred to as incremental compilation, and is essential for a rapid build development cycle. Any major build tool today features incremental compilation, and dds has had it since the beginning.

But what happens the output of the compiler when we skip compiling a file due to incremental compilation? For most tools, and in dds alpha.5, the answer was simple: We didn’t run the compiler, so there was no output to display.

dds alpha.6 changes that, and now, when dds sees a compiler emit some output after a compilation, that output is stored for later display. On subsequent builds, if the compilation of a source file is skipped but the previous compilation emitted compiler warnings, those compiler warnings are displayed again. This prevents warnings from being forgotten because an incremental rebuild skipped over the compilation of a file that was up-to-date. Compiler output that appears from a prior compilation is annotated as “cached compiler output” to distinguish it.

Feature: New CLI

The command-line interface of dds has been completely rewritten from scratch. This improves what was previously a confusing set of subcommands and reduces them by reorganizing them into a more useful hierarchy. Additionally, dds will now issue “did-you-mean” messages for every command-line argument, improving discoverability.

Feature: Tweak-Headers

Tweak-Headers are a new approach to compile-time library configuration customization. dds supports these natively via the --tweaks-dir command-line argument to dds build, dds build-deps, and dds compile-file.

--tweaks-dir should be provided as an argument to the build and point to the directory from which tweak-header #include directives should resolve. The recursive file listing of a --tweaks-dir is used as a cache key during incremental compilation and dds will insert a cache-busting compiler option to prevent tools like ccache from missing tweak-header additions (a known issue when the result of __has_include() can change).

Removal: Catalog-everything

Before remote repositories, dds used a “calalog” JSON format. This has been entirely removed, as well as “fs-transforms.” If a package needs to be rearranged or patched before import, it should be patched separately, archived as a source distribution, and imported into an HTTP repository for dds to download.

Changed Terminology

dds had a featured called the “local repository”, which was just a directory containing source distributions of external packages. This terminology has been dropped in favor of the “local package cache.” Also, the “catalog database” is now just the “local database.” These changes are simply to relieve undue confusion, especially now that dds has remote repositories which have remote catalog databases.

Other Changes

dds output is now much more colorful!
More documentation on installation, and docs on basic dependency management.
dds can “install itself” with the dds install-yourself subcommand (refer to the documentation on download and installation).
Updated the embedded Catch2 header.
Fix garbled output on build cancellation.
Reduced static binary size on Linux.
Fix behavior on Windows with non-ASCII paths.

A Year’s Retrospective

It’s been nearly a year since the initial dds announcement. It’s been a loooong time, and we’re still in alpha??

Yep, we’re still only in testing phase. What’s taking so long?

If you read the prior release post, you’ll probably remember that there’s been a lot of yak-shaving, and that’s the main reason that it has taken so long. Almost every component in dds’s stack is “home-grown,” including “fundamentals” such as HTTP, networking, gzip, and command-line parsing (of course).

I’ve heard often complaints that “software development anymore is just gluing libraries and frameworks together and no one ever makes anything,” and I can now say with some confidence that I’m glad that I don’t have to constantly re-invent everything when I want to just download a file. Over this last year I’ve experienced it. It took me months just to get dds to download a single webpage over HTTP. It’s been a great learning experience, but its taken quite a bit of time, and only now do I have something to show for it.

Going forward, iteration between dependencies and components will be significantly faster now that there is a separate package repository that can be updated when I need to post a bugfix to an external component.

A Year of Eating Dog Food

dds and every library that I’ve built along the way are built and tested by dds itself. Times that I have used CMake or Visual Studio in the past year could be counted on one hand, and I can say that I’m very confident and comfortable with using dds as my main driver, even if it is still in an early stage and has not had a package repository all this time.

Of course I will be biased, but I offer this perspective in hopes that others may take the time to try it out with their own personal projects. I have been building dds with intent that it can be anyone’s main development iteration tool, from simple prototyping, to library development, to tooling, and eventually to full desktop applications.

The Final Stretch: What’s Next?

As noted, this is the last alpha-release of dds, after which it will be released as beta, and the focus will be on bugfixes, performance, documentation, and code cleanup. Following beta will be the first stable release 1.0.0, of course!

Until the beta, though, there are still some features to add:

TLS certificate validation: dds does not yet authenticate and validate TLS certificates when it connects to a TLS server! Certificate validation will need to be handled properly, and will also entail new dds configuration options on certificate authority trust and validation bypass options.
Package signing: The previous post mentioned that packages are not yet signed. While usage of TLS and certificate trust somewhat alleviates some concerns, we will want package authors to be able to sign packages themselves before those packages are posted to an HTTP repository.
A new command-line UX: The dds command-line is fairly rudimentary and consists of basic logging, but a more concise and informative UI is preferrable.
An interprocess API for external tooling and IDE integration: Currently, dds’s only integration is by emitting a compile_commands.json file. While this offers some amount of functionality to some tools, it is insufficient for many.

Try It Out!

I’m very happy with how dds is going, and I am excited to continue work. In the coming weeks I’ll be updating the alpha package repository with more (and updated) packages, and I will be listening for feedback.

If your brand new, or just curious: Check the Getting/Installing dds page of the documentation.

Send Feedback, Ask Questions, Post Ideas, and More

GitHub has released a beta feature of the platform, known as Discussions. I have opted to enable them on the dds repository to answer user questions and solicit feedback rather than use GitHub issues for that purpose.

If you have any questions or want to provide feedback, please feel free to open a new discussion on the Discussions section. I’ll especially be looking out for “How do I…?” questions, since those may warrant documentation changes.

Link to the dds Discussions page

dds Alpha Release 6 is Available!