Christopher Prohm

Understanding which parts of your code is exercised by tests via code coverage can be essential. Therefore, I am happy that ipytest supports coverage inside notebooks starting with release 0.14.1. This release includes ipytest.cov, a Coverage.py plugin, that also allows to use pytest-cov in notebooks. In this blog post I would like to describe the underlying issue and how ipytest solves it.

Coverage.py works in three phases: Execution, Analysis, Reporting.

Phase 1 executes as-is in notebook environments and does not need any specialized support. It uses the builtin tracing mechanism of Python to record information about the code executed. Phase 2 and 3 however require access to a program's source code.

These phases fail in notebooks without extra configuration due to the underlying execution model of Jupyter notebooks. Jupyter notebooks execute code in a separate process called the kernel. The kernel does not see the complete notebook, but rather each cell as it is executed, without the larger context.

Luckily, IPython stores the executed code to show it in tracebacks. I first dug into the details when I implemented assertion rewriting. IPython's traceback mechanism uses the linecache module. This module includes a dict that maps filename to source code and other meta data. IPython inserts the cell's source code into this cache with a "random" filename¹.

ipytest in turn uses this cache to get the source code of the individual cells required by phases 2 and 3. The included Coverage.py plugin detects whenever a Python file refers to a notebook cell and makes sure these files are correctly handled.

To use the plugin, it has to be included in the Coverage.py config, e.g., by creating a file .coveragerc next to the notebook with the content

[run]
plugins =
    ipytest.cov

When using ipytest, it can be configured via ipytest.autoconfig(coverage=True). This command enables pytest-cov with a coveragerc file included in ipytest itself. In the simplest case:

# In[1]
import ipytest
ipytest.autoconfig(coverage=True)

# In[2]
%%ipytest

def test():
    ...

If you are using ipytest, I hope you find the new coverage support helpful and would love feedback on Mastodon

---