Self contained task runners in Python

Every coding project has recurring tasks, like test execution or code generation. Over the years I been exploring a variety of ways to include these tasks in an executable form as part of the repository. They range from make to invoke, task, or just. As of late, I started to rely more and more on plain self-contained Python scripts for my own projects. In this post, I would like to describe the technical details and why I like this setup.

The idea is to collect the recurring tasks of a project in an x.py file at the root level. The script itself contains a number of task functions configured via decorators. For example:

# [...]

@cmd()
def format():
    run("python", "-m", "black", ".")


@cmd()
@arg("--full", action="store_true", help="if given, run also slow tests")
def test(full=False):
    run("python", "-m", "pytest", ".", *(() if not full else ("-m", "slow")))

# [...]

The individual tasks are available as commands when executing the script. For example to format the source, I run the script as

$ python x.py format
:: python.exe -m black .
All done! ✨ 🍰 ✨
1 file left unchanged.

The two decorators @cmd and @arg form a declarative command line parser based on argparse. For example the definition of the test function above corresponds roughly to the following argparse calls:

subparser = parser.add_subparser("test")
subparser.set_defaults(main_func=test)
subparser.add_argument(
    "--full", action="store_true", help="if given, run also slow tests",
)

The implementation is designed for succinctness as I copy and paste it between projects. An additional requirement is that it passes through black without being reformatted. The cmd and arg decorators store their arguments as attributes of the decorated functions. They are defined as

__effect = lambda effect: lambda func: [func, effect(func.__dict__)][0]
cmd = lambda **kw: __effect(lambda d: d.setdefault("@cmd", {}).update(kw))
arg = lambda *a, **kw: __effect(lambda d: d.setdefault("@arg", []).append((a, kw)))

The the actual argument parsing is performed in the "__main__" guard of the script. In essence it searches all global variables with @cmd attributes and builds the argument parser using the information collected by the @cmd, @arg decorators. It reads

if __name__ == "__main__":
    _sps = (_p := __import__("argparse").ArgumentParser()).add_subparsers()
    for _f in (f for f in list(globals().values()) if hasattr(f, "@cmd")):
        (_sp := _sps.add_parser(_f.__name__, **getattr(_f, "@cmd"))).set_defaults(_=_f)
        [_sp.add_argument(*a, **kw) for a, kw in reversed(getattr(_f, "@arg", []))]
    (_a := vars(_p.parse_args())).pop("_", _p.print_help)(**_a)

And that's it. I copy these roughly 10 lines of Python with slight variations into each project and I have way to execute tasks without external tools apart from a Python interpreter. You can find a full example here.

So how does it stack up? Overall I'm quite happy with this setup. As Python is a full programming language it can for example include loops to avoid duplication or conditions for platform specific behavior. With pure configuration languages, often such logic often cannot be expressed as part of the task file itself.

Further, as these scripts are self-contained there is no need to activate a virtual environment to execute the script. Python's standard library covers most uses cases out of the box. In rase cases, in which I do need extra packages, I import them lazily to allow running basic tasks without activating the virtual environment.

Also because everything is contained in a single file and the implementation is not hidden, it's easy to inspect everything and tune the behavior as required.

So what are the downsides of this approach? Without a Python interpreter at hand, a task runner written in the main programming language for the project may work better, such as task for Go projects or just for Rust projects.

Also argparse can be a bit idiosyncratic at times. In most cases, I find it easy to configure, but in particular boolean flags require me to lookup the details quite often.

While I assume this is setup does not work for everybody, I have been quite happy with it over the last couple of months.

In addition to the script described above, I also use a small Powershell function to simplify its usage. It searches for the nearest x.py script in the parents of the current directory and executes it. With this function, I can simply type xx <task> in any directory of a project to execute the corresponding task. The function reads

function XX() {
    $dir = $PWD.ProviderPath
    $xscript = $null
    do {
        if (Test-Path -Type Leaf -LiteralPath "$dir/x.py") {
            $xscript = "$dir/x.py"
            break
        }
    } while ($dir = Split-Path -LiteralPath $dir)

    if ($null -eq $xscript) {
        throw "Could not find the x.py script"
    }

    python $xscript @args
}

Skeleton of a task file:

# ruff: noqa: E401, E731
__effect = lambda effect: lambda func: [func, effect(func.__dict__)][0]
cmd = lambda **kw: __effect(lambda d: d.setdefault("@cmd", {}).update(kw))
arg = lambda *a, **kw: __effect(lambda d: d.setdefault("@arg", []).append((a, kw)))

# ...

if __name__ == "__main__":
    _sps = (_p := __import__("argparse").ArgumentParser()).add_subparsers()
    for _f in (f for f in list(globals().values()) if hasattr(f, "@cmd")):
        (_sp := _sps.add_parser(_f.__name__, **getattr(_f, "@cmd"))).set_defaults(_=_f)
        [_sp.add_argument(*a, **kw) for a, kw in reversed(getattr(_f, "@arg", []))]
    (_a := vars(_p.parse_args())).pop("_", _p.print_help)(**_a)