Post History

Makefile for auto-generating dependencies in module-based C++ projects

C++20 modules are awesome, and the `std` module introduced in C++23 makes them even more awesome.

Unfortunately, tooling has been slow to catch up to the promise. Heavy duty build systems like [build2](https://build2.org/) and [CMake](https://cmake.org/) do support building module-based projects, but for simple experimentation, they can be a bit much. It would be nice if we could use good old `make`.

There are a ton of blog posts and such out there demonstrating how to use GCC or Clang to build module-based projects, and many of them include a passing mention of “blah-blah you can also put this in a `Makefile`”, but none of them really dig into *how*. Certainly none of them explain how to *automatically* generate the dependency graph in a module-based project; most of them just imply you manually write out all the dependencies, and leave it at that.

This `Makefile` is designed to help build simple projects, for experimentation with module-based C++ project design. Obviously for serious, large-scale projects, you should use a real build system like build2 or CMake (and along with CMake you’ll probably need something like conan or vcpkg or whatever). But for simple trials and experiments, a simple `Makefile` should suffice. A simple project with a simple `Makefile` is a lot easier to understand and debug than one using a more complex build system, which is better for learning and experimentation.

## Limitations

**This `Makefile` requires GNU `make`.** I have tried to use as little that is non-POSIX as possible, but… frankly, given what I’m trying to do, that was never going to be feasible.

**This `Makefile` currently only works with GCC.** It should work with GCC 15+, though I’ve only tested it with GCC 16.1. I do intend to add Clang support, eventually.

You *must* name your module interface units according to some simple rules:

*   All **module interface units** *must* have a special extension. This is `.mxx` by default, but you can change it to `.cppm` or whatever else you please. This extension *cannot* be the same as your “regular” C++ extension (like `.cxx` or `.cpp` or whatever).
*   A **primary module interface unit** *must* be given the same base name as the module. So the interface unit for module `foo` *must* be `foo.mxx` (or whatever extension you choose). It does not matter what directory the file is in.
*   A **module partition interface unit** *must* be given a base name based on the module and partition names, separated by a hyphen-minus character (`-`). So the interface unit for module partition `foo:part` *must* be `foo-part.mxx` (or whatever extension you choose). It does not matter what directory the file is in.

All other C++ source files, including module implementation units, must use the same extension (which must be different from the module interface extension).

Note that while module *interface* units must have specific names, module *implementation* units are not constrained. (Except that they must have the configured C++ source file extension.)

By default, the module interface extension is `.mxx` and the source/implementation unit extension is `.cxx`.

If you want to use third-party module-based libraries, you should probably define them in their own `Makefile`, and then either include them (risky!) or use a recursive `make` operation. The latter probably makes more sense, because then you can keep third-party libraries self-contained in their own directories. It would probably make the most sense to build third-party libraries as a module BMI plus a static (`.a`) or shared (`.so`) library; GCC can automatically find the BMI, and you just need to remember to link with the static/shared library.

## How it works

With traditional header-based builds, you don’t need any information before you can start compiling each unit. Each unit lists its `#include` requirements, and each `#include` requirement is literally just a path (that is based on your configured include search directories). So the compiler can open a unit, then just open each `#include` as it discovers it in the source code.

With module-based builds, you can’t just jump in and compile a unit. Modules are imported by name (`import foo;`), which has no connection to the path the actual module BMI or interface can be found at. So all modules need to be built *before* any unit that uses them, so the compiler already has the BMIs cached in a known location. So before we actually start compiling anything, we need to know all the modules, and where their module interface units are.

That means we can’t use the classic trick for traditional header-based builds, where we just build everything immediately and generate the dependency information as a side effect, for making subsequent builds faster. We have to do a first pass to generate all the dependencies, then we will know what the modules are, and how to build them, so `make` can figure out the correct order to build everything in.

So when the `Makefile` is first run on a project, it tries to include all the dependency files… and fails because they don’t exist yet… so it generates them, then aborts, then retries. On the next run, the dependency files are all there and up-to-date, so it can determine the build graph, and you’re off to the races.

The one snag is that the dependency information generated by GCC is hopelessly broken.

GCC generates pointless `.PHONY` targets for every module, which force every module to be completely rebuilt every run. It also names those `.PHONY` targets badly; module partitions get named like `foo:part.c++-module`… yes, with the colon unescaped… which breaks `make`.

So *before* we can generate the dependency information, we first generate an `awk` script to *fix* the dependency information. That `awk` script looks like this:

```awk
#!/bin/awk -f

################################################################################
#                                                                              #
# This program is free software: you can redistribute it and/or modify it      #
# under the terms of the GNU General Public License as published by the        #
# Free Software Foundation, either version 3 of the License, or (at your       #
# option) any later version.                                                   #
#                                                                              #
# This program is distributed in the hope that it will be useful, but WITHOUT  #
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or        #
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for     #
# more details.                                                                #
#                                                                              #
# You should have received a copy of the GNU General Public License along with #
# this program. If not, see <https://www.gnu.org/licenses/>.                   #
#                                                                              #
################################################################################

{
	# Join continued lines together.
	while ($NF == "\\") {
		if ((getline l) < 0)
			break
		$0 = substr($0, 0, length($0) - 1) " " l
	}

	# If the line is defining a phony module target, get rid of it.
	if (/^[[:blank:]]*[[:alpha:]][[:alnum:]_]*(:[[:alpha:]][[:alnum:]_]*)?\.c\+\+-module[[:blank:]]*:/)
		next
	if (/^[[:blank:]]*\.PHONY[[:blank:]]*:[[:blank:]]*[[:alpha:]][[:alnum:]_]*(:[[:alpha:]][[:alnum:]_]*)?\.c\+\+-module/)
		next

	# We have no need for the "CXX_IMPORTS" macro. Dump it.
	if (/^[[:blank:]]*CXX_IMPORTS[[:blank:]]*[=+:]/)
		next

	# Change all "???.c++-module" dependencies to "gcm.cache/???.gcm".
	while (match($0, /[[:alpha:]][[:alnum:]_]*(:[[:alpha:]][[:alnum:]_]*)?\.c\+\+-module/)) {
		# Extract the "???" part of the "???.c++-module" string.
		p = substr($0, RSTART, RLENGTH - 11)

		# Replace (unescaped!!!) colons in module partition names with dashes.
		sub(":", "-", p)

		# Replace the dependency with the corrected string.
		$0 = substr($0, 0, RSTART - 1) "gcm.cache/" p ".gcm" substr($0, RSTART + RLENGTH)
	}

	print
}
```

After passing the generated dependency information through that, we get *proper* `Makefile` dependency lists that use the compiled BMI rather than `.PHONY` targets.

Now all we need to do is include those, and define recipes for the compiled BMI based on the module interface unit. We also include a special recipe for the `std` module, to make sure that gets built if needed.

## The `Makefile`

Here is the `Makefile`:

```make
################################################################################
#                                                                              #
# This program is free software: you can redistribute it and/or modify it      #
# under the terms of the GNU General Public License as published by the        #
# Free Software Foundation, either version 3 of the License, or (at your       #
# option) any later version.                                                   #
#                                                                              #
# This program is distributed in the hope that it will be useful, but WITHOUT  #
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or        #
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for     #
# more details.                                                                #
#                                                                              #
# You should have received a copy of the GNU General Public License along with #
# this program. If not, see <https://www.gnu.org/licenses/>.                   #
#                                                                              #
################################################################################

# Executable
exe ::= ...

# Sources
src ::= ...

# Configuration ################################################################

ext_mxx ?= .mxx
ext_cxx ?= .cxx
ext_obj ?= .o
ext_dep ?= .d

depsdir ?= .deps

# Setup ########################################################################

SHELL ::= /bin/sh

.SUFFIXES :
.SUFFIXES : ${ext_mxx} ${ext_cxx} ${ext_obj} ${ext_dep}

# Default target ###############################################################

.PHONY : all
all : ${exe}

# Executable ###################################################################

mxx ::= $(filter %${ext_mxx},${src})
cxx ::= $(filter %${ext_cxx},${src})

obj_mxx ::= $(patsubst %,%${ext_obj},${mxx})
obj_cxx ::= $(patsubst %,%${ext_obj},${cxx})

${exe} : ${obj_cxx}
	${CXX} ${CXXFLAGS} ${CPPFLAGS} ${LDFLAGS} ${^} -o ${@} ${LDLIBS}

# Compile ######################################################################

${obj_cxx} : %${ext_cxx}${ext_obj} : %${ext_cxx}
	${CXX} ${CXXFLAGS} ${CPPFLAGS} -c -o ${@} ${<}

# Modules ######################################################################

define module_bmi_recipe
gcm.cache/$$(basename $$(notdir ${1})).gcm : ${1}
	$${CXX} $${CXXFLAGS} $${CPPFLAGS} -x c++ -c -o $${<}$${ext_obj} $${<}
endef

$(foreach module,${mxx},$(eval $(call module_bmi_recipe,${module})))

gcm.cache/std.gcm : 
	@${CXX} ${CXXFLAGS} ${CPPFLAGS} -fsearch-include-path -c bits/std.cc -o ${depsdir}/std${ext_obj}

# Dependencies #################################################################

dep_mxx ::= $(patsubst %,${depsdir}/%${ext_dep},${mxx})
dep_cxx ::= $(patsubst %,${depsdir}/%${ext_dep},${cxx})

${dep_mxx} : ${depsdir}/%${ext_dep} : % | ${depsdir}/module-deps.gcc.awk
	@mkdir -p -- ${@D}
	@${CXX} ${CXXFLAGS} ${CPPFLAGS} -x c++ -M -MP -MT ${*}${ext_obj} ${<} | awk -f $(firstword ${|}) >${@}

${dep_cxx} : ${depsdir}/%${ext_dep} : % | ${depsdir}/module-deps.gcc.awk
	@mkdir -p -- ${@D}
	@${CXX} ${CXXFLAGS} ${CPPFLAGS} -M -MP -MT ${*}${ext_obj} ${<} | awk -f $(firstword ${|}) >${@}

${depsdir}/module-deps.gcc.awk :
	@mkdir -p -- ${@D}
	@{ \
		printf '%s\n' '{' ; \
		printf '%s\n' '  while ($$NF == "\\") {' ; \
		printf '%s\n' '    if ((getline l) < 0)' ; \
		printf '%s\n' '      break' ; \
		printf '%s\n' '    $$0 = substr($$0, 0, length($$0) - 1) " " l' ; \
		printf '%s\n' '  }' ; \
		printf '%s\n' '  if (/^[[:blank:]]*[[:alpha:]][[:alnum:]_]*(:[[:alpha:]][[:alnum:]_]*)?\.c\+\+-module[[:blank:]]*:/)' ; \
		printf '%s\n' '    next' ; \
		printf '%s\n' '  if (/^[[:blank:]]*\.PHONY[[:blank:]]*:[[:blank:]]*[[:alpha:]][[:alnum:]_]*(:[[:alpha:]][[:alnum:]_]*)?\.c\+\+-module/)' ; \
		printf '%s\n' '    next' ; \
		printf '%s\n' '  if (/^[[:blank:]]*CXX_IMPORTS[[:blank:]]*[=+:]/)' ; \
		printf '%s\n' '    next' ; \
		printf '%s\n' '  while (match($$0, /[[:alpha:]][[:alnum:]_]*(:[[:alpha:]][[:alnum:]_]*)?\.c\+\+-module/)) {' ; \
		printf '%s\n' '    # Extract the "???" part of the "???.c++-module" string.' ; \
		printf '%s\n' '    p = substr($$0, RSTART, RLENGTH - 11)' ; \
		printf '%s\n' '    sub(":", "-", p)' ; \
		printf '%s\n' '    $$0 = substr($$0, 0, RSTART - 1) "gcm.cache/" p ".gcm" substr($$0, RSTART + RLENGTH)' ; \
		printf '%s\n' '  }' ; \
		printf '%s\n' '  print' ; \
		printf '%s\n' '}' ; \
	} >${@}

include ${dep_mxx}
include ${dep_cxx}

# Clean target #################################################################

.PHONY : clean
clean :
	-@rm -rf -- ${depsdir} gcm.cache/
	-@rm -f -- ${exe} ${obj_mxx} ${obj_cxx}
```

To use it, simply set `exe` to the desired name of the executable to be generated, and list your source, interface, and implementation files in `src` (or use a shell-find to do it automatically).

You obviously need to make sure that you’re using the necessary command-line options to GCC; you probably want at least `-fmodules`.

And then, just `make` to build, and `make clean` to clean up, as per usual. (No, there is no `make install`, because this is just intended for simple experimentation.)

By my experiments, it *seems* to do minimal rebuilds when source files change, and it works correctly with parallel `make` runs.

## Example

Given the following project source tree (starting in the project root):

*   `Makefile`
*   `src`
    *   `indi`
        *   `indi.cxx`
        *   `indi.mxx`
        *   `indi-part.cxx`
        *   `indi-part.mxx`
    *   `main.cxx`

Where:

`src/main.cxx` is:

```c++
import indi;

auto main() -> int
{
    hello();
}
```

`src/indi/indi.mxx` is:

```c++
export module indi;

export auto hello() -> void;
```

`src/indi/indi.cxx` is:

```c++
module indi;

import std;

import :part;

auto hello() -> void
{
    std::println("Hello from module indi!");
    hello_part();
}
```

`src/indi/indi-part.mxx` is:

```c++
export module indi:part;

auto hello_part() -> void;
```

`src/indi/indi-part.cxx` is:

```c++
module indi;

import std;

auto hello_part() -> void
{
    std::println("Hello from module indi (internal implementation)!");
}
```

In `Makefile` we set the project configuration like so:

```make
exe ::= hello

src ::= src/main.cxx \
        src/indi/indi.mxx \
        src/indi/indi.cxx \
        src/indi/indi-part.mxx \
        src/indi/indi-part.cxx
```

(You could also do `src ::= $(shell find . -type f -name '*.mxx' -o -name '*.cxx')` to automatically find the sources, if you please.)

And then you build like this:

```text
$ export CXXFLAGS='-std=c++26 -pedantic -Wall -Wextra -fmodules -fcontracts'
$ make -j 8
g++ -std=c++26 -pedantic -Wall -Wextra -fmodules -fcontracts  -x c++ -c -o src/indi/indi-part.mxx.o src/indi/indi-part.mxx
g++ -std=c++26 -pedantic -Wall -Wextra -fmodules -fcontracts  -x c++ -c -o src/indi/indi.mxx.o src/indi/indi.mxx
g++ -std=c++26 -pedantic -Wall -Wextra -fmodules -fcontracts  -c -o src/main.cxx.o src/main.cxx
g++ -std=c++26 -pedantic -Wall -Wextra -fmodules -fcontracts  -c -o src/indi/indi.cxx.o src/indi/indi.cxx
g++ -std=c++26 -pedantic -Wall -Wextra -fmodules -fcontracts  -c -o src/indi/indi-part.cxx.o src/indi/indi-part.cxx
g++ -std=c++26 -pedantic -Wall -Wextra -fmodules -fcontracts   src/indi/indi.mxx.o src/indi/indi-part.mxx.o src/main.cxx.o src/indi/indi.cxx.o src/indi/indi-part.cxx.o -o hello 
$ ./hello
Hello from module indi!
Hello from module indi (internal implementation)!
$ 
```

And to demonstrate that `clean` works:

```text
$ tree -a .
tree -a .
.
├── .deps
│   ├── module-deps.gcc.awk
│   ├── src
│   │   ├── indi
│   │   │   ├── indi.cxx.d
│   │   │   ├── indi.mxx.d
│   │   │   ├── indi-part.cxx.d
│   │   │   └── indi-part.mxx.d
│   │   └── main.cxx.d
│   └── std.o
├── gcm.cache
│   ├── indi.gcm
│   ├── indi-part.gcm
│   └── std.gcm
├── hello
├── Makefile
└── src
    ├── indi
    │   ├── indi.cxx
    │   ├── indi.cxx.o
    │   ├── indi.mxx
    │   ├── indi.mxx.o
    │   ├── indi-part.cxx
    │   ├── indi-part.cxx.o
    │   ├── indi-part.mxx
    │   └── indi-part.mxx.o
    ├── main.cxx
    └── main.cxx.o

7 directories, 22 files
$ make clean
$ tree -a .
.
├── Makefile
└── src
    ├── indi
    │   ├── indi.cxx
    │   ├── indi.mxx
    │   ├── indi-part.cxx
    │   └── indi-part.mxx
    └── main.cxx

3 directories, 6 files
$ 
```

## Questions for reviewers

I would like to request a general review of the idea, and the actual `Makefile` code.

Remember, the intended goal of this make file is just for experimentation with modules, and particularly how to structure module-based projects. It is not intended to be used as the primary build system in a serious library/program released for public consumption.

Things I am particularly interested in:

*   **Portability:** Right now, a lot of GNU `make` features are used. Can they be replaced by more portable alternatives? If I can achieve POSIX-compliance, that would be sweet. (That even includes the `awk` script.) But would it be worth it?
*   **Edge cases:** For example, should I force the `C` locale for `awk`? If so, how to do so portably?
*   **Extensibility:** I want to add Clang support, but Clang works *very* differently, and it seems like I’ll have to parse/generate JSON files. 😬 If anyone is familiar with auto-generating module-based dependencies for Clang, could you chime and suggest whether it can be worked in easily, or whether an entirely different structure will be necessary?