Christopher Prohm

Reducing boilerplate in Serde serializers with nested Rust macros

Sun, 05 Oct 2025 00:00:00 +0000
The implementation of serde-arrow</code></a> features many serializers, one for each Arrow data type and some additional serializers to connect everything. Each of these serializers requires the implementation of all of the 28 required methods of theserde::Serializer</code> trait</a>. This combination results in lots of repetitive code. In this post, I would like to explain my attempt to simplify this setup using macros and a trick of writing nested macros I encountered along the way.</p>


Building a custom elements Pomodoro timer: from React to Web Components
Sun, 30 Mar 2025 00:00:00 +0000
For some time now, I've been using the Pomodoro Technique</a> to structure myself and maintain
focus when working on complex tasks, such as programming or writing projects. Since 2014, I have
been using my own spin</a> on a Pomodoro timer written in React. Recently, I took some time to
rewrite it as a custom element. In this post, I would like to share my learnings.</p>


Refactoring serde_arrow
Wed, 11 Sep 2024 00:00:00 +0000
Currently, I am  in the midst of a major refactoring effort for serde_arrow</code></a>.
Prompted by this comment</a> by v1gnesh</a> on the related PR, I would like to
summarize the ideas behind the changes and outline future plans.</p>


ipytest.cov - Coverage.py in Jupyter notebooks
Sat, 27 Jan 2024 00:00:00 +0000
Understanding which parts of your code is exercised by tests via code coverage can be essential.
Therefore, I am happy that ipytest</code></a> supports coverage inside notebooks starting with release 0.14.1.
This release includes ipytest.cov</code>, a Coverage.py</a> plugin, that also allows to use pytest-cov</code></a> in notebooks.
In this blog post I would like to describe the underlying issue</a> and how ipytest</code> solves it.</p>


Whisper in ONNX with key value caching pt. 2
Mon, 18 Dec 2023 00:00:00 +0000
As promised in the last post</a>, I will cover the ONNX</a> conversion
of the full Whisper</a> model, including key-value caching, in this post.
The full model resists a straightforward ONNX conversion due to its reliance on
hooks and branching control flow. In this post I will discuss how to patch the
model to make it exportable.</p>


Whisper in ONNX with key value caching pt. 1
Sun, 19 Nov 2023 00:00:00 +0000
This post continues my quest to convert the Whisper</a> model to ONNX.
Last time</a>, I discussed how to convert the model without the optional
performance optimization of key value caching. After some trial and error, I
managed to convert the full model. In this post, I will discuss how the Whisper
model implements key value caching, before I cover the actual conversion in the
next post.</p>


Whisper in ONNX
Sun, 12 Nov 2023 00:00:00 +0000
Recently, I have been looking into using the Whisper</a> text to speech
model from Rust. While packages like burn</a> or candle</a> do look
quite promising and do offer Whisper models, I decided to stick with
ONNX</a> for now, as it served me quite well in the past. In this post, I
would like to discuss the necessary steps to convert the Whisper model to ONNX.</p>


Vega-Lite plots with HTMX using custom elements
Sun, 03 Sep 2023 00:00:00 +0000
In this post I would like to describe how to create Vega-Lite plots that
integrate nicely into an HTMX application. A difficulty in the design is that
Vega requires asynchronous operations and the newly created element must ensure
updates are applied in the correct order. In the following I describe the issue
and a potential solution.</p>


Nested declarative macros for enum dispatch in Rust
Mon, 14 Aug 2023 00:00:00 +0000
serde_arrow</code></a> relies heavily on Rust macros to remove code
duplication. One prominent example is the bytecode introduced in version 0.7</code>.
It compiles the underlying state machines into a list of instructions that are
then interpreted. Each instruction is one of a large number of variants in an
enum. In this post I would like to summarize how serde_arrow</code> uses nested
declarative macross, that is macros defining macros, to avoid code duplication
in the interpreter.</p>


Self contained task runners in Python
Thu, 08 Jun 2023 00:00:00 +0000
Every coding project has recurring tasks, like test execution or code
generation. Over the years I been exploring a variety of ways to include these
tasks in an executable form as part of the repository. They range from
make</code></a> to invoke</code></a>, task</code></a>, or just</code></a>. As of
late, I started to rely more and more on plain self-contained Python scripts for
my own projects. In this post, I would like to describe the technical details
and why I like this setup.</p>


Plotting Polars data frames with Seaborn
Tue, 21 Feb 2023 00:00:00 +0000
Recently, I found myself using Polars</a> more and more as my data frame
library of choice. Compared to Pandas, it has much better support for complex
column types, like list of structs, but also better performance.
Pandas</a> however is  still clearly the most popular data frame library
in the ecosystem. Case in point, Seaborn</a> does not support  Polars
data frames out of the box. In this post, I would like to walk you through a
quick way to combine both libraries.</p>


Mixing React and htmx
Tue, 18 Oct 2022 00:00:00 +0000
A recent project required a complex HTML user interface with multiple
interdependent elements. With the goal of using hmtx</a>, I implemented the
initial prototype using custom elements, but it grew in complexity quite fast.
In the end, I found this to be one occasion in which React</a> provides a
clean and elegant way to structure code and data flow. Not content with giving
up on htmx</a> for the rest of the application, I set out to combine both
libraries. In this blog post I would like to walk you through the resulting
setup.</p>


React & htmx
Tue, 18 Oct 2022 00:00:00 +0000
<</span>script</span> </span></span>
    src</span>=</span>"https://unpkg.com/react@18/umd/react.development.js"</span> </span></span>
    integrity</span>=</span>"sha384-0HL/VWVbwweJfp0saUL50fXRuSABCdVeinTBoJCDXprLkJ49VI0QMWNGMRt8ebnT"</span> </span></span>
    crossorigin</span>=</span>"anonymous"</span>></</span>script</span>></span></span>
<</span>script</span> </span></span>
    src</span>=</span>"https://unpkg.com/react-dom@18/umd/react-dom.development.js"</span> </span></span>
    integrity</span>=</span>"sha384-79Od0yhavbvtuP2nWl+Y6mwgs8AlknSIikYSw0+uOc65GTyH8SW7e2hCyCB303Y2"</span> </span></span>
    crossorigin</span>=</span>"anonymous"</span>></</span>script</span>></span></span>
<</span>script</span> </span></span>
    src</span>=</span>"https://unpkg.com/htmx.org@1.8.2"</span> </span></span>
    integrity</span>=</span>"sha384-+8ISc/waZcRdXCLxVgbsLzay31nCdyZXQxnsUy++HJzJliTzxKWr0m1cIEMyUzQu"</span> </span></span>
    crossorigin</span>=</span>"anonymous"</span>></</span>script</span>></span></span></code></pre>customElements.</span>define</span>(</span></span>
    "react-root"</span>,</span></span>
    class extends</span> HTMLElement</span> {</span></span>
        constructor</span>() {</span></span>
            super</span>();</span></span>
            this</span>.</span>attachShadow</span>({mode:</span> "open"</span>});</span></span>
            this</span>.root</span> =</span> null</span>;</span></span>
</span>
            new</span> MutationObserver</span>(()</span> =></span> this</span>.</span>render</span>()).</span>observe</span>(</span></span>
                this</span>,  {</span></span>
                    subtree:</span> true</span>,</span></span>
                    childList:</span> true</span>, </span></span>
                    attributes:</span> true</span>, </span></span>
                    attributeFilter: [</span>"props"</span>,</span> "type"</span>],</span></span>
                });</span></span>
        }</span></span>
</span>
        render</span>() {</span></span>
            if</span>(</span>this</span>.root</span> ==</span> null</span>) {</span></span>
                this</span>.root</span> =</span> ReactDOM.</span>createRoot</span>(</span>this</span>.shadowRoot);</span></span>
            }</span></span>
                                    </span></span>
            const</span> element</span> =</span> window[</span>this</span>.rootType];</span></span>
            if</span>(element</span> ==</span> null</span>) {</span></span>
                throw new</span> Error</span>(</span>`Could not find component ${</span>this</span>.</span>rootType</span>}`</span>);</span></span>
            }</span></span>
</span>
            this</span>.root.</span>render</span>(</span></span>
                React.</span>createElement</span>(</span></span>
                    element, </span></span>
                    {root:</span> this</span>,</span> ...</span>this</span>.rootProps},</span></span>
                ),</span></span>
            );</span></span>
        }</span></span>
</span>
        get</span> rootType</span>() {</span></span>
            const</span> el</span> =</span> this</span>.</span>querySelector</span>(</span>"react-element"</span>);</span></span>
            return</span> (el</span> !=</span> null</span>)</span> ?</span> el.</span>getAttribute</span>(</span>"type"</span>)</span> :</span> null</span>;</span></span>
        }</span></span>
</span>
        get</span> rootProps</span>() {</span></span>
            const</span> el</span> =</span> this</span>.</span>querySelector</span>(</span>"react-element"</span>);</span></span>
            return</span> (el</span> !=</span> null</span>)</span> ?</span> JSON</span>.</span>parse</span>(el.</span>getAttribute</span>(</span>"props"</span>))</span> :</span> null</span>;</span></span>
        }</span></span>
</span>
        set</span> rootType</span>(</span>value</span>) {</span></span>
            const</span> el</span> =</span> this</span>.</span>querySelector</span>(</span>"react-element"</span>);</span></span>
            if</span>(el</span> ==</span> null</span>)</span> throw new</span> Error</span>(</span>"Not template element found"</span>);</span></span>
            el.</span>setAttribute</span>(</span>"type"</span>, value);</span></span>
        }</span></span>
</span>
        set</span> rootProps</span>(</span>value</span>) {</span></span>
            const</span> el</span> =</span> this</span>.</span>querySelector</span>(</span>"react-element"</span>);</span></span>
            if</span>(el</span> ==</span> null</span>)</span> throw new</span> Error</span>(</span>"Not template element found"</span>);</span></span>
            el.</span>setAttribute</span>(</span>"props"</span>,</span> JSON</span>.</span>stringify</span>(value));</span></span>
        }</span></span>
</span>
        disconnectedCallback</span>() {</span></span>
            if</span>(</span>this</span>.root</span> !=</span> null</span>) {</span></span>
                this</span>.root.</span>unmount</span>();</span></span>
                this</span>.root</span> =</span> null</span>;</span></span>
            }</span></span>
        }</span></span>
    }</span></span>
);</span></span></code></pre>


Reusable htmx components with custom elements
Mon, 16 May 2022 00:00:00 +0000
While working on my chat bot project</a>, I was recently looking for a
way to test locally on my laptop before pushing to my phone. To make inroads
quickly, I settled on a local web app. But neither manual dom manipulation nor a
full scale single page app felt particularly appealing. Luckily, htmx</a>
offers a middle ground - as it turns out a quite productive one.</p>
In this post I would like to showcase how custom elements allow to build
reusable components that encapsulate the htmx configuration and the handling of
htmx events.</p>


cargo without project
Wed, 11 Aug 2021 00:00:00 +0000
Trying out ideas quickly and iterating on the details have become central to my
style of working ever since I started using Python. Exactly this kind of
interactivity I miss, when I use Rust. In the absence of a proper repl, I found
small scripts to be the next best thing. To support this setup, I wrote a crate
called cargo-wop</a> that I found quite helpful and would like to
showcase in this blog post in the hope that it may also be helpful to others.</p>


Rust: the emergent complexity of a simple language
Fri, 19 Mar 2021 00:00:00 +0000
As promised last time</a>, I would like to reflect a bit about Rust
after using it for the last year. This post is not meant to be an introduction
to Rust, rather I would like to give you subjective view of how it feels to use
Rust1</a>.</p>


The project that got me started with Rust
Mon, 15 Feb 2021 00:00:00 +0000
When all these articles featuring Rust</a> as one of the most beloved
programming languages came out, I was rather sceptic. Coming from C++ I expected
a complex and hard to pick up language. At some point though, curiosity got the
better of me and after a bit over a year I have to admit: I am sold. In
the near future, I plan to write a bit more about my experience with Rust.
To set the stage, I would like to sketch how I ended up using Rust in this post
by tracing the evolution of a long-running personal project.</p>


Linearizing graphs
Sun, 03 Jan 2021 00:00:00 +0000
From my time to time I find myself in the situation of having to linearize
graphs</a>. Typically this issue comes up in the context of task
scheduling. When tasks have dependencies among themselves, they form a directed
acyclical graph. Finding an order of the tasks that respects their dependencies
is called a topological sorting or linearization. In the following, I will use
the term linearization, as it seems more self-explanatory. In this post I would
like to discuss  a simple algorithm for graph linearization and compare how this
problem is solved in the context of multiple inheritance in Python.</p>


Hacking python's import system for single file packages
Wed, 27 May 2020 00:00:00 +0000
The last couple of days I have been playing around with the Kaggle Halite
environment</a>. Quickly, I ran into issues with their requirement
that any agent has to be a single python file. Initially, my code was short and
easy to keep self-contained. However, after adding more and more functionality,
it soon became unwieldy and unintelligible. At this point, I decided to look
for alternatives. In this blog post, I describe how to hack Python's import
system to embed full packages into a single python module.</p>


asyncio testing inside notebooks
Sun, 28 Jul 2019 00:00:00 +0000
Recently I started to work on a project mixing lots of network IO and concurrent
tasks. Due to these requirements
asyncio</a> seemed like a prefect
fit. However this project was also my first big foray into the async world of
python and I needed a bit of time to learn the ropes. As mentioned in a
previous post</a>, I do love
to familiarize myself with new technology by experimenting in a notebook and
only then to refactor the result into python packages.</p>
In this blog post, I would like to show how to combine asyncio and pytest inside
notebooks. IPython has had an excellent async
support</a> for quite
some while now. However, running async tests inside IPython requires a bit of
care. Below I describe what changes in
ipytest</code></a> were necessary.</p>


Prototyping to tested code, or developing across notebooks and modules
Thu, 08 Nov 2018 00:00:00 +0000
Two weeks ago, I was lucky enough to attend the PyCon.DE 2018 in Karsruhe. I
also got the opportunity to present my ideas on the role of notebooks and
modules in Python development, with a particular focus on testing. Since the
slides</a> and video</a> are available online, I do not want to rehash the talk
here, but rather spend the bulk of this post to explain my setup in more detail.</p>


Causality and function approximations
Sun, 26 Aug 2018 00:00:00 +0000
Motivated by the recent release of Judea Pearl's Book of Why, I started to sort
my notes on causality and write some parts down in a more structured way. In
doing so, I stumbled over a question I bumped into repeatedly in past projects:
The simplest causal system is one in which features determine actions and the
combination of features and actions determine the outcome. One popular approach
for this system is to apply inverse probability of treatment weighting. My
intuition always was that this step should not be necessary. After some
research, the answer is a resounding "Yes, but ...".</p>
In short: no causal treatment is necessary for this system according to theory.
However, causal treatment becomes important when applying function
approximations with limited capacity (or AI).</p>
The options sketched above are often discussed under the names "direct method"
(no reweighting) and "inverse propensity score method" (reweighting). Both
methods can be used to estimate causal effects, but typically the latter is
preferred over the former [e.g., 6].  To investigate this issue, I turned
multiple times to simulated data in the past, but more often than not, failed to
observe shortcomings of the direct method. The goal of this post is to discuss
one particular way the direct method can fail, specifically in the case of model
regularization.</p>
The rest of the blog post is structured as follows: After a brief reminder on
causality, I will apply Judea Pearl's causal calculus and derive that
reweighting is not necessary.  I will demonstrate numerically that reweighting
becomes important however when applying function approximations with limited
capacity (here, linear models and gradient boosted trees).  Finally, I show how
the effects can be explained analytically for a linear model fitted with maximum
likelihood and \(L_2\)  regularization.</p>


Better test output with AST rewriting and a patched standard library
Wed, 20 Jun 2018 00:00:00 +0000
Quite often, I prototype code in notebooks. To make it easier to transfer the
code outside, I also write tests using ipytest</a>, a small package I
wrote a while back</a>. However, I never could get pytest</a>'s
nice assertions to work properly. Prompted by this GH issue</a>, I decided
to have another look.</p>
To cut a long story short: the current version of ipytest</a> offers the
full pytest</a> experience from inside the notebook. Just run pip install ipytest</code> and give it a try.</p>
In the following paragraphs I would like to describe how I implemented the
assertion support. But first: did you ever wonder how Jupyter shows stack
traces? Bear with me. It is less obvious than it sounds. As a motivating
example, consider running the following code inside a notebook</p>
def</span> foo</span>():</span></span>
    raise</span> ValueError</span>()</span></span>
</span>
foo()</span></span></code></pre>
If you do, you get a nice stack trace similar to</p>
-------------------------------------------------------------</span></span>
ValueError                  Traceback (most recent call last)</span></span>
~/Code/misc-exp/BuildingBlocks/TimeSeries.ipynb in <module>()</span></span>
      2     raise ValueError()</span></span>
      3</span></span>
----> 4 foo()</span></span>
</span>
~/Code/misc-exp/BuildingBlocks/TimeSeries.ipynb in foo()</span></span>
      1 def foo():</span></span>
----> 2     raise ValueError()</span></span>
      3</span></span>
      4 foo()</span></span>
</span>
ValueError:</span></span></code></pre>
So far so good. But when you now try to use the builtin exec</code> function, things
start to break down. Just execute</p>
exec</span>(</span>'''</span></span>
def foo():</span></span>
    raise ValueError()</span></span>
</span>
foo()</span></span>
'''</span>)</span></span></code></pre>
and the stack trace becomes a disappointing</p>
-------------------------------------------------------------</span></span>
ValueError                  Traceback (most recent call last)</span></span>
~/Code/misc-exp/BuildingBlocks/TimeSeries.ipynb in <module>()</span></span>
      4</span></span>
      5 foo()</span></span>
----> 6 ''')</span></span>
</span>
~/Code/misc-exp/BuildingBlocks/TimeSeries.ipynb in <module>()</span></span>
</span>
~/Code/misc-exp/BuildingBlocks/TimeSeries.ipynb in foo()</span></span>
</span>
ValueError:</span></span></code></pre>
It does neither show source code nor line numbers making debugging quite painful
(in the python interpreter you get line numbers, but still no source). However,
surely there has to be a way to fix this issue: in Jupyter most code is run via
exec</code>, but exceptions reliably give nice stack traces.</p>
Naively, I hoped to find a documented API somewhere in the python standard
library. After an hour of fruitless search, I decided to have a look into how
IPython is doing it (that you can is one of the reasons I love open source). As
it turns out there is no API after all and IPython relies on Python internals to
show stack traces.</p>
The relevant code starts in InteractiveShell.run_cell</code></a> with the
skeleton</p>
cell_name</span> =</span> self</span>.compile.cache(cell,</span> self</span>.execution_count)</span></span>
code_ast</span> =</span> compiler.ast_parse(cell,</span> filename</span>=</span>cell_name)</span></span>
code_ast</span> =</span> self</span>.transform_ast(code_ast)</span></span>
self</span>.run_ast_nodes(code_ast.body, cell_name,</span> ...</span>)</span></span></code></pre>
For the most part, the codes reads how I would have written it. Only the line
where the cell code is cached sticks out. When you dig deeper, you will find in
essence the following code</a>:</p>
# Stdlib imports</span></span>
...</span></span>
import</span> linecache</span></span>
...</span></span>
# Now, we must monkeypatch the linecache directly so that parts of the</span></span>
# stdlib that call it outside our control go through our codepath</span></span>
# (otherwise we'd lose our tracebacks).</span></span>
linecache.checkcache</span> =</span> check_linecache_ipython</span></span>
...</span></span>
</span>
def</span> cache</span>(self, code, number</span>=</span>0</span>):</span></span>
    """...</span></span>
    Returns</span></span>
    -------</span></span>
    The name of the cached code (as a string). Pass this as the filename</span></span>
    argument to compilation, so that tracebacks are correctly hooked up.</span></span>
    """</span></span>
    ...</span></span></code></pre>
To summarize: IPython gets nice stack traces by monkeypatching the stdlib
linecache</code> module and registering fictitious files. Once a traceback is
constructed, the patched cache is queried and the content of this fictitious
file returned. I have to admit this process seems a bit brittle in my eyes.</p>
But back to the issue at hand: how to get nicely formatted asserts when running
pytest</code> inside a notebook? When run on the commandline, pytest</code></a> will
modify your code to execute additional steps that will show the inputs of the
assert. All the code modification magic lives inside the internal</em> package
_pytest.assertion.rewrite</code></a>. This module includes a function
rewrite_asserts</code> that modified the AST passed to it.</p>
So how did I end up implementing the assertion rewriting in ipytest</code>? As you
do, I ignored package boundaries and simply hooked into the pytest</code> internals
(the hypocrisy is not lost on me). Ignoring issues of encapsulation, you will
run into the problem of missing stack traces with exec</code>, once you try to
execute the modified AST. Thankfully, we can just piggy back on IPython's
solution to this problem.</p>
After putting everything together, this code</p>
# cell 1</span></span>
import</span> ipytest.magics</span></span>
__file__</span> =</span> 'Untitled.ipynb'</span></span>
</span>
# cell 2</span></span>
%%</span>run_pytest</span> -</span>v</span></span>
</span>
def</span> test_foo</span>():</span></span>
    actual</span> =</span> [</span>1</span>,</span> 2</span>,</span> 3</span>]</span></span>
    assert</span> actual</span> ==</span> [</span>2</span>,</span> 3</span>,</span> 4</span>]</span></span></code></pre>
gives nicely formatted error messages (including stack traces):</p>
[...]</span></span>
============================ FAILURES ===========================</span></span>
---------------------------- test_foo ---------------------------</span></span>
    def test_foo():</span></span>
        actual = [1, 2, 3]</span></span>
>       assert actual == [2, 3, 4]</span></span>
E       assert [1, 2, 3] == [2, 3, 4]</span></span>
E         At index 0 diff: 1 != 2</span></span>
E         Full diff:</span></span>
E         - [1, 2, 3]</span></span>
E         + [2, 3, 4]</span></span>
</span>
<ipython-input-3-8b817ee3c044>:4: AssertionError</span></span>
==================== 1 failed in 0.56 seconds ====================</span></span></code></pre>
If you feel ipytest</a> could be useful to you, I would love for you to
give it a try. Just install it via pip install ipytest</code>. If you have any issues
or feedback, please feel free to contact me via @c_prohm</em> on twitter or
github</a>.</p>


Efficient Labeling in IPython
Mon, 13 Nov 2017 00:00:00 +0000
In my last post</a> I described how to
detect speech commands in continuous audio signals.  One important ingredient
was a good amount of labeled data. Quite early on, it became clear to me that an
efficient labeling interface would be extremely helpful.  The first iteration of
that interface was a simple command-line app, quickly to be replaced by an
ipywidgets</a>-based interface.</p>


Keyword detection from scratch
Sun, 13 Aug 2017 00:00:00 +0000
Anybody speaking even with the slightest of accents knows the
agony</a> that is talking to voice recognition systems.  I ran
head-long into this issue, when I recently tried to use voice commands with my
computer.  Both systems I tested, the NSSPeechRecognizer API</a> of OSX
and CMU Sphinx</a>, failed miserably when confronted with my German
accent.  Slightly annoyed by this experience, I wondered, would it be really
that hard to create a customized solution from scratch? As it turns out, not
really, as I hope to convince you in the following paragraphs.</p>


Minimalist personal cross device notes
Sun, 12 Mar 2017 00:00:00 +0000

For quite some time now, I have been trying to optimize my note taking setup. In
the end it took me longer than I expected, since I was looking for</p>

plain text format, preferably markdown</li>
robust sync between Android and OSX</li>
no cloud of any form</li>
</ul>
Finding a solution that fit all my needs was surprisingly hard.  In particular,
the no-cloud requirement seems to be somewhat exotic in the age of connected
scales and connected toothbrushes.</p>
Alas, the solution itself is quite simple: use a proven editor on either
platform (vim on OSX, jotterpad</a> on Android) and sync files via git.
Getting this setup to work robustly however took some time.  In this post, I
want to sketch my solution.</p>

On Android, the main complication was finding a markdown editor that could read
from and write to directly the device memory.  After some research I settled on
jotterpad</a>, a clean, minimalist writing app.  Jotterpad has both a
base and premium version, the latter being fully worth its price: you get
markdown preview, syntax highlighting, and the extended keyboard, that makes
typing pure text bearable on android.</p>

For OSX the choice was much easier, since I have been using vim as my editor of
choice for quite some time now.  When looking into possible plugin support, I
stumbled upon the Swiss army knife of vim-note-taking plugins,
vimwiki</a>.  Fully feature packed, it supports for example table
editing, navigation, and rendering.  In the end, I shied away from it, since
vim's markdown support is pretty great already and I prefer unix-style tools,
that do one job and do it well.</p>
The only feature I truly missed was the ability to follow links.  After
discovering follow-markdown-links</a> lacked some features, most notably
support for reference-style links, i.e., [...][...], I created
mdnav</a>.  mdnav is is designed to work nice with local markdown files
(opened in vim), external URLs (opened in the OS browser), and non- markdown
files, such as PDFs (opened via open</code> on OSX).  For a full list of what is
supported, please look at the Readme</a>.</p>

With editors on both platforms in place, only sync is missing.  Initially, I
tried resilio-sync</a>, hopeful its drop box like interface would
simplify operations.  After some research, however, I was completely
underwhelmed by its conflict management.  Last write wins</a> is
simply not good enough for files such as to-do lists, that often change on both
devices at the same time.  In general, a tool that silently overwrites my
changes is not an option.</p>
The decidedly less user friendly, but much more reliable option is git.  On OSX,
the command line was the obvious choice.  On android, I went with mgit</a>.
While its user interface takes some time getting used to, it does offer reliable
git support on a mobile phone - what more can I ask for? Since git is
decentralized, one question is always which system is the main copy.  Here, I
use my laptop as the copy to which I push my changes from mobile.  While using
this system, I found two details quite helpful: My home router is mapping MAC
addresses to fixed IPs, so I do not have to change the repository URL repeatedly
on my phone.  Also, nowadays, you can push into a checked out branch, after
adapting the repository configuration:</p>
$</span> git config receive.denyCurrentBranch updateInstead</span></span></code></pre>
This way I can push and pull my notes from my phone to my laptop at will, when
inside my flat.</p>

I had this setup now running for about a month and am quite happy with the
results.  While there are some moving parts, each part individually is simple
enough to understand and tweak if necessary.  Currently, I started experimenting
with other tools built on top of my notes, e.g., a simple tag cloud created by a
python script or reading out tables for tracking habits. Since the content is a
collection of plain text files on disk, writing tools around them becomes quite
easy.</p>
All in all, I can whole-heartedly recommend this setup: it is simple, robust,
and extremely flexible, not to mention there is no lock-in whatsoever.</p>


ipytest: running pytest in IPython notebooks
Sat, 31 Dec 2016 00:00:00 +0000
With pytest</a> becoming the de facto standard for writing tests in
python, I started to miss its functionality when prototyping code in
jupyter</a> notebooks. Thanks to some free time between the years, I
decided to add support for running pytest tests inside notebooks to the most
recent release of ipytest</a> (see this example</a>).</p>
As described in the original post</a>, my current workflow often
involves prototyping the implementation and its tests inside notebooks. They
allow for quick iteration cycles and interactive tests of ideas. Overall, I find
that notebooks offer a great developer experience, in particular for small
components.</p>
On the test side, pytest</a> greatly increases productivity, since it
allows for writing clear and succinct test code that is often much cleaner than
corresponding unittest</a> code. In my tests, I now routinely rely on
fixtures</a> for modularized tests, on xfails</a> for
testing not yet implemented features, and parametrize</a> for
testing different variants of a single test.</p>
With its recent release, ipytest</a> supports the combination of both
jupyter notebooks</a> and pytest</a>. If you'd like to give ipytest
a spin, follow theses steps:</p>

Install ipytest and pytest via pip (pip install ipytest pytest</code>)</li>
Set __file__</code> in your notebook to the notebook name</li>
Import the ipytest magics (import ipytest.magics</code>)</li>
Add %%run_pytest</code> on top of any cell containing tests (
%%run_pytest[clean]</code> to delete any previously defined tests).</li>
</ol>
A full example can be found below. If you find ipytest useful or have ideas for
improvements, I would love some feedback (issue tracker</a>).</p>
The following, somewhat silly, example (full notebook</a>)
demonstrates how to use ipytest</a>:</p>
# In[1]:</span></span>
</span>
# set the file name (required)</span></span>
__file__</span> =</span> 'Magics.ipynb'</span></span>
</span>
# add ipython magics</span></span>
import</span> ipytest.magics</span></span>
</span>
import</span> pytest</span></span>
</span>
# In[2]:</span></span>
</span>
%%</span>run_pytest[clean]</span> -</span>qq</span></span>
</span>
def</span> test_sorted</span>():</span></span>
    assert</span> sorted</span>([</span>5</span>,</span> 1</span>,</span> 4</span>,</span> 2</span>,</span> 3</span>])</span> ==</span> [</span>1</span>,</span> 2</span>,</span> 3</span>,</span> 4</span>,</span> 5</span>]</span></span>
</span>
# In[3]:</span></span>
</span>
%%</span>run_pytest[clean]</span> -</span>qq</span></span>
</span>
# fixtures work, too :)</span></span>
@pytest.fixture</span></span>
def</span> dict_list</span>():</span></span>
    return</span> [</span></span>
        dict</span>(</span>a</span>=</span>'a'</span>,</span> b</span>=</span>3</span>),</span></span>
        dict</span>(</span>a</span>=</span>'c'</span>,</span> b</span>=</span>1</span>),</span></span>
        dict</span>(</span>a</span>=</span>'b'</span>,</span> b</span>=</span>2</span>),</span></span>
    ]</span></span>
</span>
</span>
def</span> test_sorted__key_example_1</span>(dict_list):</span></span>
    assert</span> sorted</span>(dict_list,</span> key</span>=lambda</span> d: d[</span>'a'</span>])</span> ==</span> [</span></span>
        dict</span>(</span>a</span>=</span>'a'</span>,</span> b</span>=</span>3</span>),</span></span>
        dict</span>(</span>a</span>=</span>'b'</span>,</span> b</span>=</span>2</span>),</span></span>
        dict</span>(</span>a</span>=</span>'c'</span>,</span> b</span>=</span>1</span>),</span></span>
    ]</span></span>
</span>
</span>
def</span> test_sorted__key_example_2</span>(dict_list):</span></span>
    assert</span> sorted</span>(dict_list,</span> key</span>=lambda</span> d: d[</span>'b'</span>])</span> ==</span> [</span></span>
        dict</span>(</span>a</span>=</span>'c'</span>,</span> b</span>=</span>1</span>),</span></span>
        dict</span>(</span>a</span>=</span>'b'</span>,</span> b</span>=</span>2</span>),</span></span>
        dict</span>(</span>a</span>=</span>'a'</span>,</span> b</span>=</span>3</span>),</span></span>
    ]</span></span>
</span>
# In[4]:</span></span>
</span>
%%</span>run_pytest[clean]</span> -</span>qq</span></span>
</span>
# as does parametrize</span></span>
@pytest.mark.parametrize</span>(</span>'input,expected'</span>, [</span></span>
    ([</span>2</span>,</span> 1</span>], [</span>1</span>,</span> 2</span>]),</span></span>
    (</span>'zasdqw'</span>,</span> list</span>(</span>'adqswz'</span>)),</span></span>
])</span></span>
def</span> test_examples</span>(input, expected):</span></span>
    actual</span> =</span> sorted</span>(</span>input</span>)</span></span>
    assert</span> actual</span> ==</span> expected</span></span></code></pre>


framequery - SQL on dataframes
Wed, 21 Sep 2016 00:00:00 +0000
Pandas</a> is a superb tool for data handling and the de facto standard
for data science tasks in Python. Great as it is already, I always was a bit
surprised by its lack of an SQL interface. In Spark, for example, you can easily
reuse existing SQL statements on top of dataframes without having to resort to a
database.</p>
A couple of days ago, I sat down and started to implement an SQL interpreter for
Pandas, called framequery</a>. It relies heavily on the many
SQL-like operations implemented by Pandas already, but adds a SQL parser and
interpreter on top. While this package is similar in spirit to
pandasql</a> by yhat, its implementation is quite different. With
framequery</a>, data is never transferred between Pandas and a
database, but always maintained as a dataframe. Most directly, this choice
avoids conversions. In a more indirect way, it also offers the possibility of
using framequery with alternative implementations of the dataframe API, such as
dask</a>.</p>
To test the current state, install the package via pip install framequery</code> and
use the select</code> function as in</p>
import</span> pandas</span> as</span> pd</span></span>
import</span> framequery</span> as</span> fq</span></span>
</span>
sales</span> =</span> pd.read_csv(</span>"sales.csv"</span>)</span></span>
products</span> =</span> pd.read_csv(</span>"products.csv"</span>)</span></span>
</span>
yearly_sales</span> =</span> fq.select(</span>"""</span></span>
    SELECT year, sum(sales) as sales</span></span>
</span>
    FROM sales</span></span>
</span>
    JOIN products</span></span>
    ON sales.product_id = products.id</span></span>
</span>
    GROUP BY year</span></span>
"""</span>)</span></span></code></pre>
Currently, the implementation has a number of shortcomings and quite some
features are still missing, but the core functionality is already usable. If you
find this package useful, I would love your feedback. Similarly, contributions
are also very welcome.</p>
For details, please check the GitHub page</a>.</p>


Sharing data and control
Sun, 03 Apr 2016 00:00:00 +0000
The more time I spent building products that use machine learning at their core
the more it becomes clear to me that the ideal of a single data science language
and framework will never be fully realized. There will always be legacy code
that was written before the framework du jour arrived. Even without legacy code,
the reality looks already rather polyglot. Consider Spark. When using their
Python bindings one will almost surely end up with a mix of Scala and Python.</p>
To support these polyglot architectures more and more effort is being spent to
ensure that the underlying algorithms are the same regardless of the programming
language or programming framework chosen. A good example of such an algorithm is
xgboost</a>. The core algorithm is implemented in C++, but it also
supports current data processing frameworks, like Spark or Flink, and data
science languages, such as Python or R. As one central feature of its design,
xgboost uses its own framework for parallelization: even if you use Spark, the
parallelization will be fully handled by xgboost once the training starts.
Another example of this design is CaffeOnSpark</a>.</p>
However, I am wondering if this really the last step in the design of these
libraries. At its heart there is no fundamental reason why only data and not
control is flowing between parallelization framework and machine learning
library. For me, flow of control in this discussion means that both sides are
taking turns with executing code and notify each other when they are finished.
Think of JavaScript's asynchronous HTTP API. Whenever the user code requests a
HTTP page, the browser takes over and performs the request. Once the request is
complete the browser hands both the result and the control back to the
JavaScript side for further actions.</p>
If similarly control flow would be possible between machine learning library and
parallelization framework, the algorithm could use the native parallelization
mechanisms supplied by the framework. Think of an algorithm that is implemented
in terms of primitives such as allreduce and broadcast - as is
xgboost</a>. Whenever the algorithm requests an allreduce, the
parallelization framework takes over, but hands back control once the reduction
has been performed. This way each part of the system could handle what it does
best: the algorithm library the actual algorithm, the parallelization framework
the parallelization.</p>


ipytest - a IPython test runner
Sun, 07 Feb 2016 00:00:00 +0000
From time to time, I find myself wanting to run unit-tests in IPython notebooks.
After writing the same code over and over again, I decided to publish it as a
pip-installable package</a>. This post aims to be a quick introduction
how I came to create this package and how it can be used.</p>
Nowadays, the first thing I do when starting a new project is firing up an
IPython notebook</a> to prototype an initial implementation. Since the
notebook interface offers instant feedback, it allows for rapid experimentation.
When dealing with large-ish datasets this feature becomes even more powerful.
Parsing and transforming data in python can take some time. Using the notebook
interface, one can perform this initial work once and then rapidly experiment
with code using this data.</p>
However, for all the advantages of IPython, testing code placed in notebooks can
be hard. For this reason, I try move code into external python modules as soon
as possible to make it easy to import and test. Having said this, sometimes,
there is this twilight zone, where code  it not yet quite stable, but still
complex enough to warrant tests. In particular with pure algorithms test-driven
development can save a lot of time in the long run.</p>
For this reason, I wrote the ipytest</a>, which allows to run tests in
IPython notebooks. As a silly example, consider the Fibonacci sequence. With
ipytest</code>, one could have written the implementation and its tests as</p>
# In[1]:</span></span>
def</span> fibonacci</span>(i):</span></span>
    """Compute the fibonacci sequence.</span></span>
</span>
    >>> </span>[fibonacci(n) for n in range(7)]</span></span>
    [1, 1, 2, 3, 5, 8, 13]</span></span>
</span>
    """</span></span>
    if</span> i</span> ==</span> 0</span> or</span> i</span> ==</span> 1</span>:</span></span>
        return</span> 1</span></span>
    return</span> fibonacci(i</span> -</span> 1</span>)</span> +</span> fibonacci(i</span> -</span> 2</span>)</span></span>
</span>
# In[2]:</span></span>
import</span> ipytest</span></span>
ipytest.clean_tests(</span>"test_fibonacci*"</span>)</span></span>
</span>
class</span> test_fibonacci_example</span>(</span>unittest</span>.</span>TestCase</span>):</span></span>
    def</span> test_example</span>():</span></span>
        self</span>.assertEqual(fibonacci(</span>6</span>),</span> 13</span>)</span></span>
</span>
def</span> test_fibonacci_0</span>():</span></span>
    assert</span> fibonacci(</span>0</span>)</span> ==</span> 1</span></span>
</span>
def</span> test_fibonacci_1</span>():</span></span>
    assert</span> fibonacci(</span>1</span>)</span> ==</span> 1</span></span>
</span>
def</span> test_fibonacci_2</span>():</span></span>
    assert</span> fibonacci(</span>2</span>)</span> ==</span> 2</span></span>
</span>
def</span> test_fibonacci_3</span>():</span></span>
    assert</span> fibonacci(</span>3</span>)</span> ==</span> 3</span></span>
</span>
def</span> test_fibonacci_4</span>():</span></span>
    assert</span> fibonacci(</span>4</span>)</span> ==</span> 5</span></span>
</span>
def</span> test_fibonacci_5</span>():</span></span>
    assert</span> fibonacci(</span>5</span>)</span> ==</span> 8</span></span>
</span>
ipytest.run_tests(</span>doctest</span>=</span>True</span>)</span></span></code></pre>
Here, both the function computing the Fibonacci sequence and the tests are
written in the same notebook. After executing it, the notebook looks like
this</a>. If you want to give it a try, just install it with
pip install ipytest</code>. A detailed reference is can be found on the project's
github page</a>.</p>


Debugging with q
Mon, 18 Jan 2016 00:00:00 +0000
Currently a lot of my time is spent on writing services controlling models or
generating reports. In these systems, the control flow moves from the user
facing HTTP server to a task queue (think of celery</a>) that handles the
actual task. Depending on the setup the log output will be scattered over
multiple files and has to be painstakingly pieced together - one reason, I found
that debugging these kinds of applications becomes tiring quite fast.</p>
From his talk</a>, I gather Ka-Ping Yee had a similar problem when he
created q</a>. This small python package is now my first import for a quick and
dirty debugging session. To use q, import it and call it like function:</p>
import</span> q</span> as</span> qq</span> # qq is easier to search for</span></span>
qq(</span>"foo"</span>,</span> "bar"</span>)</span></span></code></pre>
When you use q, you may wonder where the output went? The beauty of q is that it
writes its output to the file $TMPDIR/q</code>. Thereby the debugging output is
collected in a single file and is not mixed with other output.</p>
q can also easily be embedded into expressions, removing the need to rewrite the
code when debugging. Depending on the chosen operator q prints the full
expression or the next expression only:</p>
foo(qq</span>|</span>1</span> +</span> 2</span> +</span> 3</span>)</span> # prints 5</span></span>
foo(qq</span>/</span>1</span> +</span> 2</span> +</span> 3</span>)</span> # prints 1</span></span></code></pre>
Another feature of q is its ability to  trace function calls. Used as a
decorator, it prints out all arguments and the return value of function. With
nested function calls it is particular helpful because it indents the output
nicely. Unfortunately, the way q is implemented the renamed import breaks the
@q</code> decorator. However, using the trace function explicitly works just fine:</p>
@qq.trace</span></span>
def say_hello(name):</span></span>
    print("hello {}".format(name))</span></span></code></pre>
To sum up, q</a> is an incredibly useful package when debugging and I highly
recommend to check it out.</p>


Spring cleaning
Sun, 03 Jan 2016 00:00:00 +0000
Motivated by the unusually warm weather outside and finding myself with some
free time at my hand over the holidays, I decided to push up the spring cleaning
of this page.</p>
Now half a year ago, I left academia and started to work as a Data Scientist at
Blue Yonder</a>. With my attention directed towards this
interesting and challenging new role, I unfortunately neglected updating this
page.</p>
So with renewed enthusiasm, I changed the content of these pages to reflect my
current situation and vowed to myself to adhere to a more regular writing
schedule.</p>


void for traits
Mon, 06 Apr 2015 00:00:00 +0000
One of the great things about high level languages, be it Java or Python, is
reflection and the ability to get information about objects and their type at
runtime. When you start to think about similar concepts in C++, you'll quickly
find out that it's tricky: I guess for performance reasons, the options for
reflection are severely limited in C++. One choice is template based reflection
using SFINAE techniques (substitution failure is not an error)</a>. The
concept can be summarized as: when looking up names the C++ compiler will ignore
all templates which would result in an invalid type.</p>


End-to-end training of deep autoencoders in IPython
Mon, 02 Feb 2015 00:00:00 +0000
While working on my deep learning project, I hacked together a couple of simple
support methods for IPython that, at least for me, greatly increased its
usefulness for iterative optimization. In this post I will concentrate on
IPython, while I defer the discussion about the model and its design to a future
post. At the end of this post you'll find a teaser[1]</a></sup> and I also uploaded the HTML
file</a> and the notebook</a> of the current state.</p>


argmap available on PyPI
Sun, 11 Jan 2015 00:00:00 +0000
I just published my first package, argmap</a>, on the python package
index</a>. Exciting - and much less complicated than I initially thought.
Using pip</a>, you can now install argmap by running</p>
pip</span> install argmap</span></span></code></pre>
In the process I also improved the documentation and made it available online
here</a>.</p>


Argmap: declarative argument parsing with Python
Mon, 08 Dec 2014 00:00:00 +0000
Python is a great tool for scientific computing and I use it on a regular basis
for a wide array of tasks, ranging from simulation setup, data analysis, to the
simulations themselves in some cases. For many of these tasks support for
command line arguments greatly increases the productivity:  for example, a
single script can analyze different simulations. The Python standard library
comes with an argument parser (argparse</a>) out of the box. However, it
results in a lot if boilerplate code which is especially a problem when checking
some idea with only limited lifetime.</p>


tomato timer & the pomodoro technique
Sat, 26 Apr 2014 00:00:00 +0000
For a while now, I'v been using the pomodoro technique</a>
(Wikipedia</a>) to be more efficient when working. I found it to be
quite the efficient procrastination buster. Developed by Francesco Cirillo, the
pomodoro technique is based on the idea of structuring your work into fixed
intervals of roughly 25 minutes. While working avoid any distraction and keep
focussed on the current task. After each work interval take a short break (~5
minutes) and then continue with the next work interval. Every four work
intervals take a longer breaks (15 - 30 minutes).</p>


jsonbind - guice configured with json
Sun, 12 Jan 2014 00:00:00 +0000
A big part of my PhD project are simulations of fluid flow under various
conditions. Often I am actually not sure what the correct setup is until the
paper is written. You try one hypothesis, then another - only to modify it yet
again. In this situation it comes in handy if your program allows you to swap
certain parts of the program, for example the geometry, without influencing the
other. If you can do this at runtime without recompiling the program - even
better.</p>


Yet another short update
Mon, 30 Dec 2013 00:00:00 +0000
Finally I finished up the RBM implementation and squashed the remaining bugs.
Running the RBM gives reasonable samples and the learned filters seem ok. So
everything is working just fine, albeit a bit slow due to the use of pure java
linear algebra. Luckily most of the algorithms are implemented using the
BLAS</a> interface so it should be rather simple to swap out those parts for
a faster implementation.</p>
I really would like to have the option of switching the underlying
implementation when starting the program, which UJMP</a> seems to support.
However it does not support the full BLAS API, so I cannot multiply with the
transpose or do a rank-one update. All other libraries I found seem to focus on
running with a fixed BLAS implementation mostly on the CPU. However long-term I
will try to switch running in OpenCL to get the most performance even when
running in Java. Well, I would really hate to roll my own matrix library,
especially considering there are quite so many out there, but this seems to be
the best option at the moment.</p>
On a more happy note, I got Guice</a> working just the way I wanted it to.
I rolled my own little JSON configuration library which allows me to swap every
little piece of my program using configuration files. After cleaning up a bit
more, I will upload it to github and write a short bit about it.</p>


Short update
Sat, 14 Dec 2013 00:00:00 +0000
Ok, as it turns out using javascript was not one of by best ideas. I learned the
hard way that I have to decide between speed and stability with the two browsers
I tested. With Firefox everything was stable, but quite slow. Using Chrome on
the other hand sped things up considerably, but it never survived running for
more than a couple of hours - without warning the tab would just crash. In the
end I figured it is time to use a language which is more stable - this time I
opted for Java - which is a nice fit due to its large ecosystem. Another
advantage compared to Javascript is the access to the file system, so I can
easily use the standard datasets such as MINST - to compare my implementation
against the literature.</p>
So the course is plotted, but I'm currently somewhat short on time: in between
the end of the year stress - got papers to finish - I seem to be unable to find
the proper time to finish this . Well the holidays are up in a week and then
I'll have time to work on this - further updates are incoming.</p>


Restricted Boltzmann Machines - Theory II
Fri, 25 Oct 2013 00:00:00 +0000
In the last post</a> we have parametrized the model and are now left
with the task of learning it. Learning in the context of machine-learning always
implies maximizing a predefined objective. Generative models aim to maximize the
probability they assign to the input data. We will present the model a number
images where the pixels are stored in the vectors v^{(i)</sup>. We search for
the parameters of the model (here: visible biases b, hidden biases c and weight
matrix W) such that the probability the model assigns to the input images is
maximized. This probability, also called}likelihood</a>, is given as
the product</p>
$$ P(\{v\}) = \prod_i P(v^{(i)}). $$</p>
To maximize this probability we will use gradient ascent. However when we try to
differentiate this product we end up with a sum of products, which we will not
be able to compute. Luckily there is an easy way around this problem: by taking
the logarithm of this product we turn the likelihood into the
log-likelihood</a> and the products into the sum</p>
$$ \log P(\{v\}) = \sum_i \log P(v^{(i)}). $$</p>
As the logarithm is a monotonically increasing function, maximizing the
logarithm also maximizes its argument.</p>
To simplify the next steps let us denote the log-likelihood by J. We can
maximize it by taking small steps along its gradients -  a procedure called
gradient ascent</a>. So we can increase J by updating the weights
by</p>
$$ W_{ij} \rightarrow W_{ij} + \epsilon \frac{\partial J}{\partial W_{ij}}. $$</p>
Gradient ascent can be easily understood if we consider how J changes if we
perturb the weights by a small step ε ΔW_{ij</sub>. If ε is small enough we
can Taylor-expand and find that J changes as</p>}
$$
J(W + \epsilon \Delta W_{ij}) \approx
J(W) + \epsilon \sum_{ij} \frac{\partial J}{\partial W_{ij}} \Delta W_{ij}
$$</p>
By choosing ε ΔW_{ij</sub> equal to ∂J / ∂W this increase is always positive.
However this strictly only holds for infinitesimal small ε.</p>}
Note that you can treat the biases by including units that are always one, so
there is no need to include them in the discussion explicitly.</p>
Inserting the joint probability</a> of the RBM into the log-likelihood
we find its derivative as the difference of the derivative of the energy
function averaged over the data and averaged over the model expectations</p>
$$
\frac{\partial J}{\partial W_{ij}} \propto
-\langle \frac{\partial E}{\partial W_{ij}} \rangle_\mathrm{data}
+\langle \frac{\partial E}{\partial W_{ij}} \rangle_\mathrm{model}.
$$</p>
We can understand this as follows: By following the gradient, we decrease the
energy of the data samples and increase the energy of the model expectations. As
lower energy states are more probable, this increases the probability of the
data and decreases the probability of what the model would come up with on its
own.</p>
We can derive the gradient of the log-likelihood J by the following steps:</p>
$$
\begin{aligned}
\frac{\partial J}{\partial W_{ij}}
&= \sum_k \frac{\partial}{\partial W_{ij}} \log P(v^{(k)}) \\
&= \sum_k \frac{\partial}{\partial W_{ij}}
\left[
\log \sum_{h} \exp(-E(v^{(k)},h)) - \log Z
\right]
\end{aligned}
$$</p>
The first term will give the expectation over the data, while the second term
will give the expectation over the model.</p>
Concentrating on the first term:</p>
$$
\begin{aligned}
&\sum_k \frac{\partial}{\partial W_{ij}} \log \sum_{ { h } } \exp(-E(v^{(k)},h)) \\
&= \sum_k \frac{1}{\sum_{ { h } } \exp(-E(v,h))} \sum_{ { h } } \exp(-E(v^{(k)},h))
(-\frac{\partial}{\partial W_{ij}} E(v^{(k)}, h)) \\
&= -N_v \frac{1}{N_v} \sum_k \sum_{ { h } } P(h | v^{(k)}) \frac{\partial}{\partial W_{ij}} E(v^{(k)}, h)).
\end{aligned}
$$</p>
Here, N_{v</sub> is the number of visible samples. Furthermore the sum over
all data vectors can be considered the sum over all possible visible vectors
multiplied by the probability of that vector being part of the input data. If we
define the probability density of the input data as</p>}
$$
P_\mathrm{data}(v, h) = \frac{1}{N_v} \sum_k \delta(v - v^{(k)}) P(h | v^{(k)}),
$$</p>
we can rewrite the first part of the derivative as</p>
$$
\begin{aligned}
&\sum_k \frac{\partial}{\partial W_{ij}} \log \sum_{ { h } } \exp(-E(v^{(k)},h)) \\
&= -N_v \sum_{ {v, h } } P_\mathrm{data}(v, h) \frac{\partial}{\partial W_{ij}} E(v^{(k)}, h)) \\
&= -N_v \langle \frac{\partial E}{\partial W_{ij}} \rangle_\mathrm{data}.
\end{aligned}
$$</p>
The derivative of the partition functions results in</p>
$$
\begin{aligned}
&-\sum_k \frac{\partial}{\partial W_{ij}} \log Z \\
&= -N_v \frac{\partial}{\partial W_{ij}} \log \sum_{ { v, h } } exp[ -E(v, h) ] \\
&= N_v \log \frac{1}{\sum_{ { v, h } } exp[ -E(v, h) ]}
\sum_{ { v, h } } exp[ -E(v, h) ] \frac{\partial}{\partial W_{ij}} E(v, h) \\
&= N_v  \sum_{ { v, h } } P_\mathrm{model}(v, h) \frac{\partial}{\partial W_{ij}} E(v, h) \\
&= N_v \langle \frac{\partial E}{\partial W_{ij}} \rangle_\mathrm{model}.
\end{aligned}
$$</p>
Combining both terms we find:</p>
$$
\sum_k \frac{\partial}{\partial W_{ij}} \log P(v^{(k)})
= -N_v \langle \frac{\partial E}{\partial W_{ij}} \rangle_\mathrm{data} +
N_v \langle \frac{\partial E}{\partial W_{ij}} \rangle_\mathrm{model}.
$$</p>
All that remains is plugging in the gradient with respect to the weights of the
energy defined in the previous post</a> and we end up with</p>
$$
\frac{\partial J}{\partial W_{ij}} \propto
-\langle v h^T \rangle_\mathrm{data}
+\langle v h^T \rangle_\mathrm{model}.
$$</p>
Still we cannot calculate this gradient, as it involves expectations over the
model, which we cannot perform due to the infeasibility of the partition
function. Here we will try to approximate this expectation value by a finite
number of samples and often resort to using merely a single sample. To generate
the samples we will perform Gibbs sampling</a>.</p>
Remember</a> that we can evaluate the conditional probability of the
hidden vector given the visible vector and the converse. Gibbs sampling exploits
this fact by alternating between sampling the hidden and the visible vector. If
we repeat this process until infinity we will generate unbiased samples from the
model. In practice, as popularized by Hinton, a good approximation is start at
the data vector and perform a small number of Gibbs sampling steps. This
learning rule is called contrastive divergence.</p>
So to summarize the learning algorithm for the Restricted Boltzmann machine can
be written (in python):</p>
for</span> visible</span> in</span> input_data:</span></span>
	hidden</span> =</span> sample_hidden_given_visible(visible)</span></span>
	</span></span>
	gradient</span> +=</span> hidden</span> *</span> outer</span></span>
	</span></span>
	for</span> _</span> in</span> range</span>(k):</span></span>
		visible</span> =</span> sample_visible_given_hidden(hidden)</span></span>
		hidden</span> =</span> sample_hidden_given_visible(visible)</span></span>
		</span></span>
	gradient</span> -=</span> hidden</span> *</span> outer</span></span>
	</span></span>
	weights</span> +=</span> gradient</span></span></code></pre>
An overview of these concepts and a number of practical considerations can be
found in Geoffrey Hinton's guide</a>, which you can find as a pdf on
his home page</a>.</p>


Restricted Boltzmann Machines - Theory I
Mon, 30 Sep 2013 00:00:00 +0000
Before actually building a Restricted Boltzmann machine (RBM), let us first
discuss the math behind it.</p>
First, a quick recap of our goal</a>: We try to map the pixels of
an image (input data) to the content of this image (label). Specifically, we try
to recognize images of numbers. We tackle this problem by using a two step
approach of first building a generative model of the input data and then use the
output of this model for classification.</p>
Like most machine learning algorithms RBMs are closely tied to basic concepts to
probability theory. Specifically RBMs are used to model an a-priori unknown
probability distribution of your input data. If we denote a sample of your input
data by the vector v, RBMS try to learn the probability P(v) of that vector.</p>
Let us consider a visible vectors with two binary elements. For this example
there are four possible values v can take, (0,0), (1,0), (0,1), (1,1), and the
probability distribution P(v) consists four entries. Generally if the visible
vector has d elements the probability distribution has 2^{d</sup> entries.
Even for quite small inputs, as 8x8 pixel image, this is an astronomically large
number with 20 digits.</p>}
As we cannot reason directly about P(v) with its enormous number of entries, we
have to find a suitable parametrization, where the number of parameters is much
smaller. Like many probabilistic models, RBMs are based on the Gibbs
distribution</a>, which assigns each visible vector v a free
energy F(v). The free energy then determines the probability by</p>
$$P(v) = Z^{-1} \exp[-F(v)]$$</p>
Here Z is the so called partition function, which ensure that the probability
sums to one if we sum over all possible visible vectors. So for our example of a
two element binary vector the partition sum is equal to</p>
$$Z = \exp[-F((0,0))] + \exp[-F((1,0))] + \exp[-F((0,1))] + \exp[-F((1,1))]$$</p>
An RBM adds additional unknowns, so called hidden units h, which are coupled to
the visible state. This coupling is again described by an energy function</p>
$$E(v,h) = -v^T W h -v^T b - h^T c$$</p>
Here, W is the weight matrix, b the bias of the visible units and c the bias of
the hidden units. The Gibbs distribution for E(v,h) determines the
joint-probability of a visible vector v and a hidden vector h.</p>
So now that we have a probability P(v,h) assigned to every value of visible and
hidden vector, how do we get the distribution of visible vectors? We can compute
the marginal distribution</a>, by summing over all possible
values of the hidden vectors:</p>
$$P(v) = Z^{-1} \sum_{ \{ h \} } \exp [ -E(v, h) ]$$</p>
As an example consider the case that h is a two dimensional binary vector, then
the set {h} covers the values {(0, 0), (0, 1), (1, 0), (1, 1)} and we sum up the
probabilities by setting the hidden vector to each one of these in turn.</p>
Note, that we never specified the value of the partition function Z. The reason
is that in general we are not able to, as it involves a sum over all possible
values of the visible and hidden vectors. However, as it turns out for many
applications there are good approximations available so we do not need to
determine Z.</p>
One particularity of the restricted Boltzmann machine is the fact that the
conditional distributions can be determined explicitly. The conditional
probability P(v|h) is the probability to find the visible vector v if the hidden
vector is equal to h. The conditional distribution P(h|v) has an equivalent
meaning.</p>
The reason we can determine these probability distribution is that they
factorize in their argument, i.e., we can be write P(v|h) as</p>
$$P(v|h) = \prod_i P(v_i|h).$$</p>
The probability that the i-th visible state is equal to one is given by</p>
$$P(v_i = 1 | h) = \frac{1}{1 + \exp[ - \sum_j (W_{ij} h_j - b_j)]}$$</p>
So all that remains now is to optimize the parameters, i.e., the matrix W and
the vectors b and c, to maximize the probability of the input data. How this is
done will be the topic of the next post.</p>


Machine Learning with Generative Models
Sun, 25 Aug 2013 00:00:00 +0000
After a short hiatus - holidays and finishing up a paper - I am back writing and
decided to write about my current after-work project, namely generative models
for machine learning. I've been interested in machine learning for a quite some
time now. However always found myself in the predicament that finding or
generating high quality training data can be quite the challenge. This applies
in particular, if you try to learn agent behavior in a complex setting.</p>


Recipes for jmpress animations
Sun, 16 Jun 2013 00:00:00 +0000

In the last post</a>, I discussed how to use jmpress's</a>
animations. And while animations are quite flexible, there is no denying, that
this flexibility makes things a bit harder for run-off-the-mill animations. For
most of the time, I rely on a set of limited css classes for my talks, which I
want to discuss in this post.</p>


Wrapping C++ code using boost python
Sat, 18 May 2013 00:00:00 +0000
The saying "The right tool, for the right job." is one of my most beloved rule
of thumb. This especially holds true for programming languages, certain
languages are a natural for fit for certain tasks. The fact that the simulations
I am running take forever already gives me strong motive to squeeze out the last
bit of performance. Hence the choice of C++ as the programming language.</p>


Animations in jmpress
Wed, 15 May 2013 00:00:00 +0000
In the first part</a> of this tutorial, I
talked about how to use jmpress</a> to create simple talks using html5.
In this part I want to show you how to include animations in a slide. Animations
in talks can be very useful to guide the audience through your talk, by
highlighting the current information. Unfortunately adding animations in
jmpress</a> can be hard at times.</p>
As a reference keep the official documentation</a> at hand.</p>
Animations are implement by CSS classes in jmpress and usually have three steps:</p>

The animation will occur, but has not yet</li>
It is currently running</li>
And finally, it has already run</li>
</ol>
Each of these steps is implemented by adding a CSS class to the element being
animated. All animations have a name, which can be chosen arbitrarily, as long
as it is a valid CSS identifier.</p>
In the following I will walk you trough the steps of how to implement a custom
fade animations, which is also implemented in the jmpress
example</a>.</p>
Any element can be animated by adding the data-jmpress attribute. To fade out a
part of text just wrap it in a span, as in</p>
Hello <</span>span</span> data-jmpress</span>=</span>"fade"</span>>world</</span>span</span>></span></span></code></pre>
While presenting, the following css classes will be added to the element</p>

will-fade: from the start</li>
do-fade: once, the animation is running</li>
has-fade: after the animation has run</li>
</ul>
So to implement a fade effect just define the correct css classes (you may need
to add the correct browser prefixes)</p>
.will-fade</span> {</span></span>
    transition-property</span>: opacity;</span></span>
    transition-duration</span>:</span> 0.5</span>s</span>;</span></span>
    opacity</span>:</span> 1.0</span>;</span></span>
}</span></span>
</span>
.do-fade</span> {</span></span>
    opacity</span>:</span> 0.0</span>;</span></span>
}</span></span></code></pre>
The will-fade class informs the browser that any change of the opacity property
should be animated and that this change should take half a second. The do-fade
class makes sure that the opacity of the fading element is reduced to zero,
while the animation is running. Note, that while the documentation says that the
class is only set while the animation is running, it is not removed after the
animation.</p>
If there are multiple animations on each slide, the animations are run in the
order they appear in the document. For example:</p>
<</span>span</span> data-jmpress</span>=</span>"fade"</span>>first</</span>span</span>></span></span>
<</span>span</span> data-jmpress</span>=</span>"fade"</span>>second</</span>span</span>></span></span></code></pre>
To customize this, additional properties can be added. I found, the most
important options are:</p>

(name) after (selector): run the animation after the animation of the element
with the given selector has run</li>
(name) after (time): run the animation after the given time</li>
(name) after (time) (selector): run the animation with a given delay delay
after the animation of the element given by selector has run.</li>
</ul>
There are two special selectors prev</em> and step</em>, which denote the previous
element and the slide beginning respectively. Per default always the previous
animated element is used.</p>
For example:</p>
<</span>span</span> data-jmpress</span>=</span>"fade after #first"</span>>Fade after</</span>span</span>></span></span>
<</span>span</span> id</span>=</span>"first"</span> data-jmpress</span>=</span>"fade after step"</span>>this element</</span>span</span>></span></span>
<</span>span</span> data-jmpress</span>=</span>"fade after 1s"</span>>delayed</</span>span</span>></span></span></code></pre>
You can find a working html version of this example here. For more
information especially about the available options for animations look at the
documentation</a>. In the next part I will discuss how to setup
more complex animations using jmpress.</p>


Exporting D3 graphs
Mon, 08 Apr 2013 00:00:00 +0000
While playing around with D3</a>, I was always disappointed, that the
resulting images are stuck in the browser afterwards. However, since you use
merely use D3 to generate an embedded SVG</a> image, it should not be
that hard to get it out, should it?</p>


Simple html5 talks
Mon, 25 Mar 2013 00:00:00 +0000
In this short tutorial I want to show you how to create simple html5 talks using
jmpress</a>. One particularity about jmpress is, that relies exclusively
on CSS to layout your slides and some working knowledege of CSS is highly
desirable.</p>


HTML5 Presentations
Fri, 15 Mar 2013 00:00:00 +0000
After just getting back from the DPG spring meeting, I thought I'd give a
overview of how to create (scientific) presentations using HTML5. The talk I
gave was setup using jmpress</a>, and it
worked out quite beautifully (at least from my perspective).</p>


Multi-particle collision dynamics
Mon, 28 Jan 2013 00:00:00 +0000
So after spending the last two weeks parallelizing my program, I rediscovered
the ingenuity that are mesoscopic simulation techniques. Instead of simulating
the actual system at hand, you construct a simplified system, which reproduces
only a minimal set of properties of the original system.</p>


Welcome
Sat, 19 Jan 2013 00:00:00 +0000
Welcome to my blog.
Here, I will discuss general programming topics, which I'm currently interested
in, and also projects, which I'm currently pursuing.
All of this will focus on three programming languages, python, javascript and
C++.
While my phd work</a>,
is mainly done in python and C++, I've recently
taking a liking to javascript for my hobbyist projects.
But more on this in later posts.</p>