Makefile for auto-generating dependencies in module-based C++ projects

C++20 modules are awesome, and the std module introduced in C++23 makes them even more awesome.

Unfortunately, tooling has been slow to catch up to the promise. Heavy duty build systems like build2 and CMake do support building module-based projects, but for simple experimentation, they can be a bit much. It would be nice if we could use good old make.

There are a ton of blog posts and such out there demonstrating how to use GCC or Clang to build module-based projects, and many of them include a passing mention of “blah-blah you can also put this in a Makefile”, but none of them really dig into how. Certainly none of them explain how to automatically generate the dependency graph in a module-based project; most of them just imply you manually write out all the dependencies, and leave it at that.

This Makefile is designed to help build simple projects, for experimentation with module-based C++ project design. Obviously for serious, large-scale projects, you should use a real build system like build2 or CMake (and along with CMake you’ll probably need something like conan or vcpkg or whatever). But for simple trials and experiments, a simple Makefile should suffice. A simple project with a simple Makefile is a lot easier to understand and debug than one using a more complex build system, which is better for learning and experimentation.

Limitations

This Makefile requires GNU make. I have tried to use as little that is non-POSIX as possible, but… frankly, given what I’m trying to do, that was never going to be feasible.

This Makefile currently only works with GCC. It should work with GCC 15+, though I’ve only tested it with GCC 16.1. I do intend to add Clang support, eventually.

You must name your module interface units according to some simple rules:

• All module interface units must have a special extension. This is .mxx by default, but you can change it to .cppm or whatever else you please. This extension cannot be the same as your “regular” C++ extension (like .cxx or .cpp or whatever).
• A primary module interface unit must be given the same base name as the module. So the interface unit for module foo must be foo.mxx (or whatever extension you choose). It does not matter what directory the file is in.
• A module partition interface unit must be given a base name based on the module and partition names, separated by a hyphen-minus character (-). So the interface unit for module partition foo:part must be foo-part.mxx (or whatever extension you choose). It does not matter what directory the file is in.

All other C++ source files, including module implementation units, must use the same extension (which must be different from the module interface extension).

Note that while module interface units must have specific names, module implementation units are not constrained. (Except that they must have the configured C++ source file extension.)

By default, the module interface extension is .mxx and the source/implementation unit extension is .cxx.

If you want to use third-party module-based libraries, you should probably define them in their own Makefile, and then either include them (risky!) or use a recursive make operation. The latter probably makes more sense, because then you can keep third-party libraries self-contained in their own directories. It would probably make the most sense to build third-party libraries as a module BMI plus a static (.a) or shared (.so) library; GCC can automatically find the BMI, and you just need to remember to link with the static/shared library.

How it works

With traditional header-based builds, you don’t need any information before you can start compiling each unit. Each unit lists its #include requirements, and each #include requirement is literally just a path (that is based on your configured include search directories). So the compiler can open a unit, then just open each #include as it discovers it in the source code.

With module-based builds, you can’t just jump in and compile a unit. Modules are imported by name (import foo;), which has no connection to the path the actual module BMI or interface can be found at. So all modules need to be built before any unit that uses them, so the compiler already has the BMIs cached in a known location. So before we actually start compiling anything, we need to know all the modules, and where their module interface units are.

That means we can’t use the classic trick for traditional header-based builds, where we just build everything immediately and generate the dependency information as a side effect, for making subsequent builds faster. We have to do a first pass to generate all the dependencies, then we will know what the modules are, and how to build them, so make can figure out the correct order to build everything in.

So when the Makefile is first run on a project, it tries to include all the dependency files… and fails because they don’t exist yet… so it generates them, then aborts, then retries. On the next run, the dependency files are all there and up-to-date, so it can determine the build graph, and you’re off to the races.

The one snag is that the dependency information generated by GCC is hopelessly broken.

GCC generates pointless .PHONY targets for every module, which force every module to be completely rebuilt every run. It also names those .PHONY targets badly; module partitions get named like foo:part.c++-module… yes, with the colon unescaped… which breaks make.

So before we can generate the dependency information, we first generate an awk script to fix the dependency information. That awk script looks like this:

#!/bin/awk -f

################################################################################
#                                                                              #
# This program is free software: you can redistribute it and/or modify it      #
# under the terms of the GNU General Public License as published by the        #
# Free Software Foundation, either version 3 of the License, or (at your       #
# option) any later version.                                                   #
#                                                                              #
# This program is distributed in the hope that it will be useful, but WITHOUT  #
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or        #
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for     #
# more details.                                                                #
#                                                                              #
# You should have received a copy of the GNU General Public License along with #
# this program. If not, see <https://www.gnu.org/licenses/>.                   #
#                                                                              #
################################################################################

{
	# Join continued lines together.
	while ($NF == "\\") {
		if ((getline l) < 0)
			break
		$0 = substr($0, 0, length($0) - 1) " " l
	}

	# If the line is defining a phony module target, get rid of it.
	if (/^[[:blank:]]*[[:alpha:]][[:alnum:]_]*(:[[:alpha:]][[:alnum:]_]*)?\.c\+\+-module[[:blank:]]*:/)
		next
	if (/^[[:blank:]]*\.PHONY[[:blank:]]*:[[:blank:]]*[[:alpha:]][[:alnum:]_]*(:[[:alpha:]][[:alnum:]_]*)?\.c\+\+-module/)
		next

	# We have no need for the "CXX_IMPORTS" macro. Dump it.
	if (/^[[:blank:]]*CXX_IMPORTS[[:blank:]]*[=+:]/)
		next

	# Change all "???.c++-module" dependencies to "gcm.cache/???.gcm".
	while (match($0, /[[:alpha:]][[:alnum:]_]*(:[[:alpha:]][[:alnum:]_]*)?\.c\+\+-module/)) {
		# Extract the "???" part of the "???.c++-module" string.
		p = substr($0, RSTART, RLENGTH - 11)

		# Replace (unescaped!!!) colons in module partition names with dashes.
		sub(":", "-", p)

		# Replace the dependency with the corrected string.
		$0 = substr($0, 0, RSTART - 1) "gcm.cache/" p ".gcm" substr($0, RSTART + RLENGTH)
	}

	print
}

After passing the generated dependency information through that, we get proper Makefile dependency lists that use the compiled BMI rather than .PHONY targets.

Now all we need to do is include those, and define recipes for the compiled BMI based on the module interface unit. We also include a special recipe for the std module, to make sure that gets built if needed.

The Makefile

Here is the Makefile:

################################################################################
#                                                                              #
# This program is free software: you can redistribute it and/or modify it      #
# under the terms of the GNU General Public License as published by the        #
# Free Software Foundation, either version 3 of the License, or (at your       #
# option) any later version.                                                   #
#                                                                              #
# This program is distributed in the hope that it will be useful, but WITHOUT  #
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or        #
# FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for     #
# more details.                                                                #
#                                                                              #
# You should have received a copy of the GNU General Public License along with #
# this program. If not, see <https://www.gnu.org/licenses/>.                   #
#                                                                              #
################################################################################

# Executable
exe ::= ...

# Sources
src ::= ...

# Configuration ################################################################

ext_mxx ?= .mxx
ext_cxx ?= .cxx
ext_obj ?= .o
ext_dep ?= .d

depsdir ?= .deps

# Setup ########################################################################

SHELL ::= /bin/sh

.SUFFIXES :
.SUFFIXES : ${ext_mxx} ${ext_cxx} ${ext_obj} ${ext_dep}

# Default target ###############################################################

.PHONY : all
all : ${exe}

# Executable ###################################################################

mxx ::= $(filter %${ext_mxx},${src})
cxx ::= $(filter %${ext_cxx},${src})

obj_mxx ::= $(patsubst %,%${ext_obj},${mxx})
obj_cxx ::= $(patsubst %,%${ext_obj},${cxx})

${exe} : ${obj_cxx}
	${CXX} ${CXXFLAGS} ${CPPFLAGS} ${LDFLAGS} ${^} -o ${@} ${LDLIBS}

# Compile ######################################################################

${obj_cxx} : %${ext_cxx}${ext_obj} : %${ext_cxx}
	${CXX} ${CXXFLAGS} ${CPPFLAGS} -c -o ${@} ${<}

# Modules ######################################################################

define module_bmi_recipe
gcm.cache/$$(basename $$(notdir ${1})).gcm : ${1}
	$${CXX} $${CXXFLAGS} $${CPPFLAGS} -x c++ -c -o $${<}$${ext_obj} $${<}
endef

$(foreach module,${mxx},$(eval $(call module_bmi_recipe,${module})))

gcm.cache/std.gcm : 
	@${CXX} ${CXXFLAGS} ${CPPFLAGS} -fsearch-include-path -c bits/std.cc -o ${depsdir}/std${ext_obj}

# Dependencies #################################################################

dep_mxx ::= $(patsubst %,${depsdir}/%${ext_dep},${mxx})
dep_cxx ::= $(patsubst %,${depsdir}/%${ext_dep},${cxx})

${dep_mxx} : ${depsdir}/%${ext_dep} : % | ${depsdir}/module-deps.gcc.awk
	@mkdir -p -- ${@D}
	@${CXX} ${CXXFLAGS} ${CPPFLAGS} -x c++ -M -MP -MT ${*}${ext_obj} ${<} | awk -f $(firstword ${|}) >${@}

${dep_cxx} : ${depsdir}/%${ext_dep} : % | ${depsdir}/module-deps.gcc.awk
	@mkdir -p -- ${@D}
	@${CXX} ${CXXFLAGS} ${CPPFLAGS} -M -MP -MT ${*}${ext_obj} ${<} | awk -f $(firstword ${|}) >${@}

${depsdir}/module-deps.gcc.awk :
	@mkdir -p -- ${@D}
	@{ \
		printf '%s\n' '{' ; \
		printf '%s\n' '  while ($$NF == "\\") {' ; \
		printf '%s\n' '    if ((getline l) < 0)' ; \
		printf '%s\n' '      break' ; \
		printf '%s\n' '    $$0 = substr($$0, 0, length($$0) - 1) " " l' ; \
		printf '%s\n' '  }' ; \
		printf '%s\n' '  if (/^[[:blank:]]*[[:alpha:]][[:alnum:]_]*(:[[:alpha:]][[:alnum:]_]*)?\.c\+\+-module[[:blank:]]*:/)' ; \
		printf '%s\n' '    next' ; \
		printf '%s\n' '  if (/^[[:blank:]]*\.PHONY[[:blank:]]*:[[:blank:]]*[[:alpha:]][[:alnum:]_]*(:[[:alpha:]][[:alnum:]_]*)?\.c\+\+-module/)' ; \
		printf '%s\n' '    next' ; \
		printf '%s\n' '  if (/^[[:blank:]]*CXX_IMPORTS[[:blank:]]*[=+:]/)' ; \
		printf '%s\n' '    next' ; \
		printf '%s\n' '  while (match($$0, /[[:alpha:]][[:alnum:]_]*(:[[:alpha:]][[:alnum:]_]*)?\.c\+\+-module/)) {' ; \
		printf '%s\n' '    # Extract the "???" part of the "???.c++-module" string.' ; \
		printf '%s\n' '    p = substr($$0, RSTART, RLENGTH - 11)' ; \
		printf '%s\n' '    sub(":", "-", p)' ; \
		printf '%s\n' '    $$0 = substr($$0, 0, RSTART - 1) "gcm.cache/" p ".gcm" substr($$0, RSTART + RLENGTH)' ; \
		printf '%s\n' '  }' ; \
		printf '%s\n' '  print' ; \
		printf '%s\n' '}' ; \
	} >${@}

include ${dep_mxx}
include ${dep_cxx}

# Clean target #################################################################

.PHONY : clean
clean :
	-@rm -rf -- ${depsdir} gcm.cache/
	-@rm -f -- ${exe} ${obj_mxx} ${obj_cxx}

To use it, simply set exe to the desired name of the executable to be generated, and list your source, interface, and implementation files in src (or use a shell-find to do it automatically).

You obviously need to make sure that you’re using the necessary command-line options to GCC; you probably want at least -fmodules.

And then, just make to build, and make clean to clean up, as per usual. (No, there is no make install, because this is just intended for simple experimentation.)

By my experiments, it seems to do minimal rebuilds when source files change, and it works correctly with parallel make runs.

Example

Given the following project source tree (starting in the project root):

• Makefile
• src
  • indi
    • indi.cxx
    • indi.mxx
    • indi-part.cxx
    • indi-part.mxx
  • main.cxx

Where:

src/main.cxx is:

import indi;

auto main() -> int
{
    hello();
}

src/indi/indi.mxx is:

export module indi;

export auto hello() -> void;

src/indi/indi.cxx is:

module indi;

import std;

import :part;

auto hello() -> void
{
    std::println("Hello from module indi!");
    hello_part();
}

src/indi/indi-part.mxx is:

export module indi:part;

auto hello_part() -> void;

src/indi/indi-part.cxx is:

module indi;

import std;

auto hello_part() -> void
{
    std::println("Hello from module indi (internal implementation)!");
}

In Makefile we set the project configuration like so:

exe ::= hello

src ::= src/main.cxx \
        src/indi/indi.mxx \
        src/indi/indi.cxx \
        src/indi/indi-part.mxx \
        src/indi/indi-part.cxx

(You could also do src ::= $(shell find . -type f -name '*.mxx' -o -name '*.cxx') to automatically find the sources, if you please.)

And then you build like this:

$ export CXXFLAGS='-std=c++26 -pedantic -Wall -Wextra -fmodules -fcontracts'
$ make -j 8
g++ -std=c++26 -pedantic -Wall -Wextra -fmodules -fcontracts  -x c++ -c -o src/indi/indi-part.mxx.o src/indi/indi-part.mxx
g++ -std=c++26 -pedantic -Wall -Wextra -fmodules -fcontracts  -x c++ -c -o src/indi/indi.mxx.o src/indi/indi.mxx
g++ -std=c++26 -pedantic -Wall -Wextra -fmodules -fcontracts  -c -o src/main.cxx.o src/main.cxx
g++ -std=c++26 -pedantic -Wall -Wextra -fmodules -fcontracts  -c -o src/indi/indi.cxx.o src/indi/indi.cxx
g++ -std=c++26 -pedantic -Wall -Wextra -fmodules -fcontracts  -c -o src/indi/indi-part.cxx.o src/indi/indi-part.cxx
g++ -std=c++26 -pedantic -Wall -Wextra -fmodules -fcontracts   src/indi/indi.mxx.o src/indi/indi-part.mxx.o src/main.cxx.o src/indi/indi.cxx.o src/indi/indi-part.cxx.o -o hello 
$ ./hello
Hello from module indi!
Hello from module indi (internal implementation)!
$ 

And to demonstrate that clean works:

$ tree -a .
tree -a .
.
├── .deps
│   ├── module-deps.gcc.awk
│   ├── src
│   │   ├── indi
│   │   │   ├── indi.cxx.d
│   │   │   ├── indi.mxx.d
│   │   │   ├── indi-part.cxx.d
│   │   │   └── indi-part.mxx.d
│   │   └── main.cxx.d
│   └── std.o
├── gcm.cache
│   ├── indi.gcm
│   ├── indi-part.gcm
│   └── std.gcm
├── hello
├── Makefile
└── src
    ├── indi
    │   ├── indi.cxx
    │   ├── indi.cxx.o
    │   ├── indi.mxx
    │   ├── indi.mxx.o
    │   ├── indi-part.cxx
    │   ├── indi-part.cxx.o
    │   ├── indi-part.mxx
    │   └── indi-part.mxx.o
    ├── main.cxx
    └── main.cxx.o

7 directories, 22 files
$ make clean
$ tree -a .
.
├── Makefile
└── src
    ├── indi
    │   ├── indi.cxx
    │   ├── indi.mxx
    │   ├── indi-part.cxx
    │   └── indi-part.mxx
    └── main.cxx

3 directories, 6 files
$ 

Questions for reviewers

I would like to request a general review of the idea, and the actual Makefile code.

Remember, the intended goal of this make file is just for experimentation with modules, and particularly how to structure module-based projects. It is not intended to be used as the primary build system in a serious library/program released for public consumption.

Things I am particularly interested in:

• Portability: Right now, a lot of GNU make features are used. Can they be replaced by more portable alternatives? If I can achieve POSIX-compliance, that would be sweet. (That even includes the awk script.) But would it be worth it?
• Edge cases: For example, should I force the C locale for awk? If so, how to do so portably?
• Extensibility: I want to add Clang support, but Clang works very differently, and it seems like I’ll have to parse/generate JSON files. 😬 If anyone is familiar with auto-generating module-based dependencies for Clang, could you chime and suggest whether it can be worked in easily, or whether an entirely different structure will be necessary?