bpt - The Build and Package Tool
Downloads Docs GitHub News
The First bpt Beta Release
Posted on 2022-06-28

A Long Time Coming

It’s been over two years since the initial announcement of dds in early 2020, and to call those two years “rough” would be a gross understatement.

The latter of those years was spent all during the availability of 0.1.0-alpha.6, which was announced to be the “final alpha” release of the product. Much of that year was spent assessing, refining, and reconsidering dds and its architecture.

Well, after a year of slow progress and slow changes, the first 1.0.0 beta release is ready, and quite a bit has changed!

A New Name and A New Home

Firstly and most obviously, dds has had a name change. The tool is now known as bpt. With this comes a new domain name. The prior domain, dds.pizza, will remain available for a while as any users migrate away, but will eventually be shut down. The old repository will no longer receive any new package updates.

A New Project Format

In the alpha versions, projects represented themselves using separate files: one package.json5 and one or more library.json5 files. While bpt retains the distinction between packages and their libraries, a new single project file is used to represent them both.

Dropping JSON5

JSON5 is a nice enough file format, but it remains somewhat esoteric. For this reason, bpt is moving away from JSON5.

Simplifying the Manifest

Previously, a package.json5 project file would declare its package dependencies, and then each library would have to separately declare in a different file that it was “using” libraries from these packages. This has been simplified. Instead, each project dependency declaration itself contains the “using” statements for the project’s libraries.

Test Dependencies

The alpha versions of dds used a test_driver project-level option to specify a built-in testing library to use for a project. This had to be specially supported in dds and wasn’t scalable: You couldn’t use anything other than what was already built-in to dds! This has been changed to test-dependencies, which are much more common and familiar in the world of software project management.

A Single File

All of this appears in the new project file, bpt.yaml. Such a new project manifest may look something like this:

## bpt.yml
name: my-project
version: 1.0.5
dependencies:
  - fmt@8.0.3 using fmt
test-dependencies:
  - catch2@2.13.7 using main

For those averse to YAML’s whitespace sensitivity, you can always just use it as a JSON with more features:

## bpt.yml
{
  name: 'my-project',
  version: '1.0.5',
  dependencies: [
    'fmt@8.0.3 using fmt',
  ],
  test-dependencies: [
    'catch2@2.13.7 using main',
  ],
}

Other Changes

There are several other things that can be used in the bpt.yaml file, but I won’t go into detail here. Instead, refer to the documentation for more information.

A New Package Format

Previously, dds used a somewhat ad-hoc package format that was designed to closely replicate the shape of a project as it would appear on a developer’s own system. The new bpt now comes with a package format known as compile-ready source, or CRS.

The defining feature of CRS is in the name: A CRS package contains source code of the software in a form that is ready to be compiled and linked into an application or library without any further processing. A CRS package need not necessarily be created by bpt, and the sources therein need not be the same source that would be editted by a developer directly.

In CRS, the package metadata is a simple JSON document that describes the package, the libraries it contains, how those libraries link together, and the external dependencies of those libraries. From this information, a tool can construct a dependency solution and set of compile and link rules.

In a bpt project, the content of bpt.yaml will be used to generate the metadata that is embedded within a CRS package.

bpt is able to generate a CRS package from any bpt project. Outside of bpt, for most libraries and projects, the generation of a CRS package will often be a very simple transformation of the source tree and the creation of a pkg.json file.

Stateless Builds

The most significant change with bpt beta is that of being stateless. That is: Every operation depends only on its immediate inputs, regardless of prior operations. Executing bpt build with a given set of command line options will (should) always produce the same result.

Importantly, this includes the set of packages available during dependency resolution. Each bpt operation that needs to resolve dependencies takes a list of repositories to use for the operation. There is no ambient set of “enabled” or “disabled” repositories: bpt build will use only the package repositories that were expressly requested for that invocation of bpt build, and only the packages available in such repositories will be considered during dependency resolution.

(This differs from the dds alpha releases, which used a dds repo subcommand to add/remove repositories from dependency resolution in subsequent dds invocations. The bpt repo subcommand now serves a different purpose.)

With bpt beta, the --use-repo command-line argument is used to control the repositories available for dependency resolution:

# Packages from "https://repo.example/my-custom-repo" will be available.
$ bpt build --use-repo "https://repo.example/my-custom-repo"

By default, bpt will behave as if you specified --use-repo=<default-repo-url> on each invocation (the actual default repo URL is liable to change in the final release). To disable the default repository, use --no-default-repo:

$ bpt build --no-default-repo --use-repo "$my_repo_url"
# Only packages available from the URL in $my_repo_url will be available for
# dependency resolution for this build.

Toolchain Statelessness

This isn’t a change since the dds alpha, but is a good reminder: bpt considers toolchain and build options to be part of the inputs as well, and these are likewise “stateless”. e.g. unlike other build tools, bpt does not record information about the toolchain on “first run”, because every execution of bpt is (should be) equivalent to the “first run”.

Caching

Of course, if bpt started from scratch every single time it wanted to do anything, one might expect that it will be slow. This is not the case: bpt employs caching and incremental compilation to keep itself fast and keep iteration times low. Unlike other tools, though, the caches bpt keeps are meant only for its own benefit of accelerating repeated operations: The presence or absence of any cache data will not (should not) change the outcome of any build. (If you find a case where bpt’s caching is affecting build results, this would be a bug!)

bpt does not (yet) include CCache-style caching of compiler outputs, but use of ccache-style tools is supported and recommended. These can be controlled with the compiler_launcher setting in a toolchain file.

What’s Next?

With the release of the 1.0.0-beta.1 version, bpt is feature-stable. That is: All major existing features are here to stay. However: We’re still in a pre-release state, and some of those features may be tweaked before the final stable version. In particular, file and repository formats are still somewhat experimental. CRS will need some polishing, and there are plenty of performance improvements and optimizations to be made in the current bpt design.

For the upcoming next phase of the release, bpt development will focus on:

  • Addressing critical user-facing bugs and usability issues. (Please try bpt out and report those issues!)
  • Polishing and tweaking the CRS metadata and package format.
  • Expanding the catalogue of ready-to-use libraries available in the default repository.
  • Writing extensive documentation, especially introductory materials and troubleshooting guides.
  • Cleaning up bpt’s error-handling and diagnostic systems.
  • In general: ensuring bpt is ready to be used in production environments!

Help Wanted!

At this time, the most effective way most people can contribute to bpt is to simply try it out. Try to break it! Explore the dark edge cases that might have been missed!

It’s been a tumultuous two years, and it has taken a lot just to get this far. Despite all that, we’re really only just beginning.

This site is built without cookies, trackers, or analytics.