Tomer Filiba's Homepage2022-08-20T05:57:22+00:00https://tomerfiliba.com/Tomer Filibatomerfiliba@gmail.comSome Notes on 'D for the Win'2014-08-23T00:00:00+00:00https://tomerfiliba.github.com/blog/dlang-part2<p>So it turns out my last post, <a href="/blog/dlang/">D for the Win</a> got all over
the place. I published it late at night, went to bed, and when I got back in the morning all hell
broke loose. And even more surprisingly, the vast majority of the reactions (at the ones I’ve seen on
<a href="http://www.reddit.com/r/programming/comments/2e49tm/d_for_the_win/">reddit</a> and
<a href="https://news.ycombinator.com/item?id=8211639">hacker news</a>) were on-topic. I hope it means
I got my message through.</p>
<p>Since I’m not able to reply to everything, I’ll try to provide answers to some repeating
questions and concerns people had.</p>
<ol>
<li>
<p>No, “Dlang Dlang uber Alles” was not meant to be offensive or avert discussion by means of
<a href="http://en.wikipedia.org/wiki/Godwin's_law">pulling a Godwin</a>. Yes, I am aware of the context.
I just thought it’s funny and liked the sound of it. My sincere apologies to anyone who
got offended by this.</p>
</li>
<li>
<p>Sorry, I’ve never heard of <a href="http://nimrod-lang.org/">Nimrod</a> until now. If I had, I’d surely
try it out. But the product I’m working on is already committed to D at this point, and from what
I gather Nimrod is still quite unstable.</p>
</li>
<li>
<p>Some people said that <strong>system programming</strong> is anything that’s run natively (i.e., compiles to machine code).
<a href="http://en.wikipedia.org/wiki/System_programming_language">Well no</a>. I’m not sure if there’s a definitive
definition, but system programming involves interfacing with hardware, drivers and other beasts.
It usually also requires fine-grained control over resources (CPU, memory, affinity, etc.),
and generally such systems are expected to deliver steady performance (soft-realtime behavior).</p>
<p>Therefore, I’d characterize a system-programming language as one that provides:</p>
<ul>
<li>Native pointers, unsafe casts and atomic operations - hardware devices don’t chew on your fancy object model.
I mean, they gladly will, but I don’t suppose you’ll be as glad.</li>
<li>C ABI - interfacing with system libraries and even the kernel directly. Some system calls
are not exported through libraries, so being able to call a <code class="language-plaintext highlighter-rouge">cdecl</code> <a href="http://man7.org/linux/man-pages/man2/syscall.2.html">variadic function</a>
is extremely useful.</li>
<li>Manual memory management - GC is a great thing in general, but when your system can’t pause
for indefinite times at random you have to think twice. Also, when you have to control overall
memory consumption, preallocating memory is the way to go.</li>
<li>Generate efficient machine code, inline-assembly a plus - when you reach bottlenecks,
every CPU cycle counts.</li>
</ul>
</li>
<li>
<p>D has <strong>garbage collection</strong> and the standard library makes use of it. As I just explained, GC and system
programming don’t mix well. On the other hand, system programming requires tight control over all
aspects of the program, which means you end up with tailor-made data structures anyway. You’re not afraid
to get your hands dirty and reimplement things <em>the right way</em> (for your needs). So no, the GC is not a real
problem.</p>
<p><a href="http://xkcd.com/386/" style="display: inline;">
<img src="/static/res/2014-08-23-duty_calls.png" title="There's someone wrong on the internet" class="blog-post-image" /></a></p>
<p>On the other hand, having GC is great. It means that outside of the critical paths we can use a modern
language with all the benefits of one. But even more importantly – it means you can write a
<a href="http://en.wikipedia.org/wiki/Minimum_viable_product">minimum viable product</a> much more easily,
and optimize as you make progress. In other words: start with GC and remove it where it hurts.</p>
</li>
<li>
<p>The <strong>C10K problem</strong> moved people from a thread-per-IO methodology to asynchronous IO
(<code class="language-plaintext highlighter-rouge">epoll</code>, <code class="language-plaintext highlighter-rouge">kqueue</code>, <code class="language-plaintext highlighter-rouge">IOCP</code>, etc) in a thread-per-core mainloop. The <strong>C10M problem</strong> moves IO from the
kernel into userland (“the kernel is the problem, not the solution”).
You simply can’t process enough data when you have several system calls per IO. And it’s a pity, because
you’re limited by operating system architecture, not hardware.</p>
<p>Context switches are expensive, the kernel requires accounting on its own, lots of memory gets copied,
and you don’t get to utilize hardware offloading when you go through the standard kernel stacks.
For instance, I got a <strong>factor 12 improvement</strong> just by moving my code to userland polling instead of
relying on <code class="language-plaintext highlighter-rouge">epoll</code>. In both cases my program consumed 100% CPU, but kernel time plummeted from 40% to 1%.</p>
<p>And no, you cannot approach the C10M problem in Java or Python.</p>
</li>
<li>
<p><strong>C++ is a horrible language</strong>. In fact, I don’t consider it a <em>language</em> any more. It’s a conglomerate of
plagiarisms with an <a href="http://www.stroustrup.com/C++11FAQ.html#lambda">awful syntax</a>. I mean, every
language borrows ideas from neighboring ones, but C++ has this Perl-6 aroma to it: if it exists,
we’ll take it in. It doesn’t even have to make sense. As a friend put it, “it can do EVERY paradigm but
equally badly”.</p>
<p>Compiler support is always lacking, compilation times are ever increasing, and you have to fight your
way to get the compiler to do what you have in mind.
Even <a href="http://www-users.cs.york.ac.uk/susan/joke/foot.htm">shooting yourself in the foot</a> isn’t fun anymore.</p>
</li>
<li>
<p><strong>Go is nice for application programming</strong>. In fact, it may be <em>great</em> for application programming.
As a web-oriented language with easy string manipulation, built-in coroutines and garbage collection,
it surely beats Python in terms of speed. But I find it <a href="http://blog.golang.org/go-slices-usage-and-internals">not that readable</a>
and in general I’m not sure if moving to Go is worth the trouble. It feels like a mix of too-high and too-low
level constructs, with nothing in between. <em>Awkward</em> is the best word I can think of.</p>
<p>But back to my original point, Go doesn’t come near system programming. Unless your systems run in the
cloud, that is.</p>
</li>
<li>
<p><strong>D has poor adoption</strong> and lacks the codebase and community of C/C++/Java/Python/Go. But as I explained,
system programming teaches you to be self-reliant. You almost never rely on external code as-is (it might
allocate/block/be inefficient) so you end up living in a rather closed world. On the other hand, D has C ABI
so with a bit of porting effort (converting H files), you gain access to virtually everything.</p>
<p>People also repeatedly said (perhaps even mockingly) that are <strong>no success stories</strong> or large projects using D.
First of all, that’s not true. Google around. But I agree there not that many, and it’s intriguing why
D hasn’t gotten the attention it deserves in the long while since it came to be. Anyway, being a startup,
my colleagues and I are not afraid of hard work or living on the edge. And hopefully you’ll hear about
another D success story in a year or two :)</p>
</li>
</ol>
D for the Win2014-08-21T00:00:00+00:00https://tomerfiliba.github.com/blog/dlang<p>I’m a convert! I’ve seen the light!</p>
<p>By the way, be sure to read <a href="/blog/dlang-part2/">part 2</a> as well.</p>
<p>You see, Python is nice and all and it excels in so many domains, but it was not crafted for
the ever growing demands of the industry. Sure, you can build large-scale projects in Python
(and I have built), but you take it out of the lab and into the real world, the price you pay
is just too high. Literally. In terms of work per CPU cycle, you can’t do worst.</p>
<p>The <a href="http://c10m.robertgraham.com/p/manifesto.html">C10M problem</a> is a reiteration of the
<a href="http://www.kegel.com/c10k.html">C10K problem</a>. In short, today’s commodity hardware can handle millions
of packets per second, but in reality you hardly ever reach such numbers. For example,
I worked a short while at a company that used AWS and had tens of <code class="language-plaintext highlighter-rouge">twisted</code>-based Python servers
accepting and logging requests (not doing any actual work). They managed to squeeze ~500 requests/sec
out of this setup (per machine), which escalated in cost rather quickly. Moving to <code class="language-plaintext highlighter-rouge">PyPy</code> (not without trouble) did triple
the numbers or so, but still, the cost simply didn’t scale.</p>
<p>Python, I love you, but you help instill <a href="http://en.wikipedia.org/wiki/Wirth's_law">Gate’s law</a> –
“The speed of software halves every 18 months”. In the end, we pay for our CPU cycles and we want to
maximize our profit. It’s not you, Guido, it’s me. I’ve moved on to the C10M world, and for that
I’d need a programming language that’s designed for <strong>system programming</strong> with a strong and modern
type system (after all, I love duck typing). I need to interface with external systems,
so a C ABI is desirable (no foreign function interface), and meta-programming is a huge plus (so I
won’t need to incorporate cumbersome code-generation in my build system). Not to mention that
mission-critical code can’t allow for the occasional <code class="language-plaintext highlighter-rouge">NameError</code> or <code class="language-plaintext highlighter-rouge">NoneType has no member __len__</code>
exceptions. The code must compile.</p>
<p>I’ve looked into <a href="http://www.rust-lang.org/">rust</a> (nice, but will require a couple of years to mature enough
for a large-scale project) and <a href="http://golang.org/">go</a> (Google must be joking if they actually consider it
for system programming), but as strange as it may sound, I’ve finally found what I’ve been looking for
with <a href="http://dlang.org/">D</a>.</p>
<h2 id="dlang-dlang-über-alles">Dlang Dlang Über Alles</h2>
<p><a href="http://dlang.org" style="display: inline;">
<img src="/static/res/2014-03-27-dlogo.png" title="Dlang" class="blog-post-image" /></a></p>
<p>System programming is a vast ocean of specifics, technicalities and constraints, imposed by your specific needs.
Instead of boring you to death with that, I thought it would be much more intriguing to compare D and Python.
In other words, I’ll try to show how <strong>D speaks fluent Python</strong>.</p>
<p>But first things first. In (the probable) case you don’t know much D – imagine it’s what C++ would have
dreamed to be. It offers cleaner syntax, much shorter compilation time, (optional) garbage collection, highly
expressive templates and type inference, Pythonic operator overloading (implemented as <em>rewriting</em>),
object-oriented and functional capabilities (multi-paradigm like Python), intermingles high-level constructs
(like closures) with low-level ones (naked functions in inline assembly) to produces efficient code, has
strong compile-time introspection capabilities and some extra cools features in the domain of code generation:
<code class="language-plaintext highlighter-rouge">mixin</code> – which evaluates an arbitrary string of D code at compile time, and <code class="language-plaintext highlighter-rouge">CTFE</code> – compile-time
function execution. Whoa, that was long.</p>
<p>In general, D follows Python’s duck-typed (or protocol-oriented) spirit. If a type provides the
necessary interface (“protocol”) for an operation, it will <em>just work</em>, but you can also test for
compliance at compile time. For example, <em>ranges</em> are a generalization of generators in Python.
All you need to do in order to be an <code class="language-plaintext highlighter-rouge">InputRange</code> is implement <code class="language-plaintext highlighter-rouge">bool empty()</code>, <code class="language-plaintext highlighter-rouge">void popFront()</code>
and <code class="language-plaintext highlighter-rouge">auto front()</code>, and you can use <code class="language-plaintext highlighter-rouge">isInputRange!T</code> to test whether <code class="language-plaintext highlighter-rouge">T</code> adheres the protocol.
By the way, the exclamation point (<code class="language-plaintext highlighter-rouge">!</code>), which we’ll soon get acquainted with, distinguishes compile-time
arguments from runtime ones.</p>
<p>For brevity’s sake, I’m not going to demonstrate all the properties I listed up there. Instead, I’ll
show why Python programmers ought to love D.</p>
<h2 id="case-study-1-generating-html">Case Study #1: Generating HTML</h2>
<p><img src="/static/res/2012-10-03-haml.gif" title="HAML" class="blog-post-image" /></p>
<p>In an <a href="/blog/Hypertext/#the-code">old blog post</a> I outlined my vision of
HTML templating languages: kill them all. I argued they are all but crippled-down forms of Python
with an ugly syntax, so just give me Python and an easy way to programmatically manipulate the DOM.</p>
<p>I’ve later extended the sketch into a library in its own right, named <a href="https://github.com/tomerfiliba/srcgen">srcgen</a>.
You can use it to generate <a href="https://github.com/tomerfiliba/srcgen/blob/master/tests/test_html.py">HTML</a>,
<a href="https://github.com/tomerfiliba/srcgen/blob/master/tests/test_c.py">C-like</a> languages and
<a href="https://github.com/tomerfiliba/srcgen/blob/master/tests/test_python.py">Python/Cython</a> code.
I used it in many of my commercial projects when I needed to generate code.</p>
<p>So here’s an excerpt of how’s it done in <code class="language-plaintext highlighter-rouge">srcgen</code>:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">buildPage</span><span class="p">():</span>
<span class="n">doc</span> <span class="o">=</span> <span class="n">HtmlDocument</span><span class="p">()</span>
<span class="k">with</span> <span class="n">doc</span><span class="p">.</span><span class="n">head</span><span class="p">():</span>
<span class="n">doc</span><span class="p">.</span><span class="n">title</span><span class="p">(</span><span class="s">"das title"</span><span class="p">)</span>
<span class="n">doc</span><span class="p">.</span><span class="n">link</span><span class="p">(</span><span class="n">rel</span> <span class="o">=</span> <span class="s">"foobar"</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="s">"text/css"</span><span class="p">)</span>
<span class="k">with</span> <span class="n">doc</span><span class="p">.</span><span class="n">body</span><span class="p">():</span>
<span class="k">with</span> <span class="n">doc</span><span class="p">.</span><span class="n">div</span><span class="p">(</span><span class="n">class_</span><span class="o">=</span><span class="s">"mainDiv"</span><span class="p">):</span>
<span class="k">with</span> <span class="n">doc</span><span class="p">.</span><span class="n">ul</span><span class="p">():</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span>
<span class="k">with</span> <span class="n">doc</span><span class="p">.</span><span class="n">li</span><span class="p">(</span><span class="nb">id</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">i</span><span class="p">),</span> <span class="n">class_</span><span class="o">=</span><span class="s">"listItem"</span><span class="p">):</span>
<span class="n">doc</span><span class="p">.</span><span class="n">text</span><span class="p">(</span><span class="s">"I am bulletpoint #"</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span>
<span class="k">return</span> <span class="n">doc</span><span class="p">.</span><span class="n">render</span><span class="p">()</span></code></pre></figure>
<p>And here’s how it’s done in D:</p>
<figure class="highlight"><pre><code class="language-d" data-lang="d"><span class="k">auto</span> <span class="n">buildPage</span><span class="p">()</span> <span class="p">{</span>
<span class="k">auto</span> <span class="n">doc</span> <span class="p">=</span> <span class="k">new</span> <span class="n">Html</span><span class="p">();</span>
<span class="k">with</span> <span class="p">(</span><span class="n">doc</span><span class="p">)</span> <span class="p">{</span>
<span class="k">with</span> <span class="p">(</span><span class="n">head</span><span class="p">)</span> <span class="p">{</span>
<span class="n">title</span><span class="p">(</span><span class="s">"das title"</span><span class="p">);</span>
<span class="n">link</span><span class="p">[$.</span><span class="n">rel</span> <span class="p">=</span> <span class="s">"foobar"</span><span class="p">,</span> <span class="p">$.</span><span class="n">type</span> <span class="p">=</span> <span class="s">"text/css"</span><span class="p">];</span>
<span class="p">}</span>
<span class="k">with</span> <span class="p">(</span><span class="n">body_</span><span class="p">)</span> <span class="p">{</span>
<span class="k">with</span><span class="p">(</span><span class="n">div</span><span class="p">[$.</span><span class="n">class_</span> <span class="p">=</span> <span class="s">"mainDiv"</span><span class="p">])</span> <span class="p">{</span>
<span class="k">with</span> <span class="p">(</span><span class="n">ul</span><span class="p">)</span> <span class="p">{</span>
<span class="k">foreach</span><span class="p">(</span><span class="n">i</span><span class="p">;</span> <span class="mi">0</span> <span class="p">..</span> <span class="mi">5</span><span class="p">)</span> <span class="p">{</span>
<span class="k">with</span> <span class="p">(</span><span class="n">li</span><span class="p">[$.</span><span class="n">id</span> <span class="p">=</span> <span class="n">i</span><span class="p">,</span> <span class="p">$.</span><span class="n">class_</span> <span class="p">=</span> <span class="s">"listItem"</span><span class="p">])</span> <span class="p">{</span>
<span class="n">text</span><span class="p">(</span><span class="s">"I am bulletpoint #"</span><span class="p">);</span>
<span class="n">text</span><span class="p">(</span><span class="n">i</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">doc</span><span class="p">.</span><span class="n">render</span><span class="p">();</span>
<span class="p">}</span></code></pre></figure>
<p>You can find the <a href="https://github.com/tomerfiliba/dlang/blob/9bb0902809570742c070723a0a7e684350c352e1/source/html.d#L150">source code</a>
on github, just keep in mind it’s a sketch I wrote for this blog post, not a feature-complete library.</p>
<p>The funny thing is, Python’s <code class="language-plaintext highlighter-rouge">with</code> and D’s <code class="language-plaintext highlighter-rouge">with</code> are not even remotely related! The Python implementation builds
a stack of <em>context managers</em>, while <code class="language-plaintext highlighter-rouge">with</code> in D merely alters symbol lookup. But lo and behold!
The two versions are practically identical, modulo curly braces. You get the same expressive power in both.</p>
<h2 id="case-study-2-construct">Case Study #2: Construct</h2>
<p><img src="/static/res/2012-05-16-construct-logo-small.png" title="Construct" class="blog-post-image" /></p>
<p>But the pinnacle is clearly my D version of <a href="http://construct.readthedocs.org/en/latest/index.html">Construct</a>.
You see, I’ve been <a href="https://code.google.com/p/sebulbasvn/source/browse/#svn%2Ftrunk%2Fccon">struggling for many years</a>
to create a compiled version of Construct. Generating efficient, static code from declarative constructs
would make the library capable of handling real-world data, like packet sniffing or processing of large files.
In other words, you won’t have to write a toy parser in Construct and then rewrite it (by hand) in <code class="language-plaintext highlighter-rouge">C++</code>.</p>
<p>The issues with my C version of Construct were numerous, but they basically boiled down to the fact I needed
a stronger object model to represent strings, dynamic arrays, etc., and
<a href="http://construct.readthedocs.org/en/latest/adapters.html">adapters</a>.
The real power of Construct comes from adapters, which operate at the representational (“DOM”) level of the data,
rather on its binary form. That required lambdas, closures and other higher-level concepts that C lacks.
I even tried writing a Haskell version, given that Haskell is so high-level and functional, but my colleague
and I had given hope after a while.</p>
<p>Last week, it struck me that D could be the perfect candidate: it has all the necessary high-level concepts while
being able to generate efficient code with meta-programming. I began fiddling with a D version, which proved
extremely promising. So without further ado, I present
<a href="https://github.com/tomerfiliba/dlang/blob/9bb0902809570742c070723a0a7e684350c352e1/source/dconstruct.d#L260">dconstruct</a> –
an initial sketch of the library.</p>
<p>This is the canonical <code class="language-plaintext highlighter-rouge">PascalString</code> declaration in Python:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> pascal_string = Struct("pascal_string",
... UBInt8("length"),
... Array(lambda ctx: ctx.length, Field("data", 1),),
... )
>>>
>>> pascal_string.parse("\x05helloXXX")
Container({'length': 5, 'data': ['h', 'e', 'l', 'l', 'o']})
>>>
>>> pascal_string.build(Container(length=5, data="hello"))
'\x05hello'</code></pre></figure>
<p>And here’s how it’s done in D:</p>
<figure class="highlight"><pre><code class="language-d" data-lang="d"><span class="k">struct</span> <span class="n">PascalString</span> <span class="p">{</span>
<span class="n">Field</span><span class="p">!</span><span class="kt">ubyte</span> <span class="n">length</span><span class="p">;</span>
<span class="n">Array</span><span class="p">!(</span><span class="n">Field</span><span class="p">!</span><span class="kt">ubyte</span><span class="p">,</span> <span class="s">"length"</span><span class="p">)</span> <span class="n">data</span><span class="p">;</span>
<span class="c1">// the equivalent of 'Struct' in Python,</span>
<span class="c1">// to avoid confusion of keyword 'struct' and 'Struct'</span>
<span class="k">mixin</span> <span class="n">Record</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">PascalString</span> <span class="n">ps</span><span class="p">;</span>
<span class="k">auto</span> <span class="n">stream</span> <span class="p">=</span> <span class="k">cast</span><span class="p">(</span><span class="kt">ubyte</span><span class="p">[])</span><span class="s">"\x05helloXXXX"</span><span class="p">.</span><span class="n">dup</span><span class="p">;</span>
<span class="n">ps</span><span class="p">.</span><span class="n">unpack</span><span class="p">(</span><span class="n">stream</span><span class="p">);</span>
<span class="n">writeln</span><span class="p">(</span><span class="n">ps</span><span class="p">);</span>
<span class="c1">// {length: 5, data: [104, 101, 108, 108, 111]}</span></code></pre></figure>
<p>Through the use of meta-programming (and assuming inlining and optimizations), that code snippet there
actually boils down to something like</p>
<figure class="highlight"><pre><code class="language-d" data-lang="d"><span class="k">struct</span> <span class="n">PascalString</span> <span class="p">{</span>
<span class="kt">ubyte</span> <span class="n">length</span><span class="p">;</span>
<span class="kt">ubyte</span><span class="p">[]</span> <span class="n">data</span><span class="p">;</span>
<span class="kt">void</span> <span class="n">unpack</span><span class="p">(</span><span class="k">ref</span> <span class="kt">ubyte</span><span class="p">[]</span> <span class="n">stream</span><span class="p">)</span> <span class="p">{</span>
<span class="n">length</span> <span class="p">=</span> <span class="n">stream</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
<span class="n">stream</span> <span class="p">=</span> <span class="n">stream</span><span class="p">[</span><span class="mi">1</span> <span class="p">..</span> <span class="p">$];</span> <span class="c1">// advance stream</span>
<span class="n">data</span> <span class="p">=</span> <span class="n">stream</span><span class="p">[</span><span class="mi">0</span> <span class="p">..</span> <span class="n">length</span><span class="p">];</span>
<span class="n">stream</span> <span class="p">=</span> <span class="n">stream</span><span class="p">[</span><span class="n">length</span> <span class="p">..</span> <span class="p">$];</span> <span class="c1">// advance stream</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>Which is as efficient as it gets.</p>
<p><img src="/static/res/2014-08-20-theresmore.jpg" title="There's more" style="max-width: 150px;" class="blog-post-image" /></p>
<p>But wait, there’s more! The real beauty here is how we handle the
<a href="http://construct.readthedocs.org/en/latest/meta.html">context</a>. In Python, Construct builds a dictionary
that travels along the parsing/building process, allowing constructs to refer to previously seen objects.
This is possible in D too, of course, but it’s highly inefficient (and not type safe). Instead, <code class="language-plaintext highlighter-rouge">dconstruct</code>
uses a trick that’s commonly found in template-enabled languages – creating types on demand:</p>
<figure class="highlight"><pre><code class="language-d" data-lang="d"><span class="k">struct</span> <span class="n">Context</span><span class="p">(</span><span class="n">T</span><span class="p">,</span> <span class="n">U</span><span class="p">)</span> <span class="p">{</span>
<span class="n">T</span><span class="p">*</span> <span class="n">_curr</span><span class="p">;</span>
<span class="n">U</span><span class="p">*</span> <span class="n">_</span><span class="p">;</span>
<span class="k">alias</span> <span class="n">_curr</span> <span class="k">this</span><span class="p">;</span> <span class="c1">// see below</span>
<span class="p">}</span>
<span class="k">auto</span> <span class="n">linkContext</span><span class="p">(</span><span class="n">T</span><span class="p">,</span> <span class="n">U</span><span class="p">)(</span><span class="k">ref</span> <span class="n">T</span> <span class="n">curr</span><span class="p">,</span> <span class="k">ref</span> <span class="n">U</span> <span class="n">parent</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">Context</span><span class="p">!(</span><span class="n">T</span><span class="p">,</span> <span class="n">U</span><span class="p">)(&</span><span class="n">curr</span><span class="p">,</span> <span class="p">&</span><span class="n">parent</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<p>The strange <code class="language-plaintext highlighter-rouge">alias _curr this</code> is a lovely feature of D known as <code class="language-plaintext highlighter-rouge">subtyping</code>. It basically means that
any property that doesn’t exist at the struct’s scope will we forwarded to <code class="language-plaintext highlighter-rouge">_curr</code>, e.g.,
when I write <code class="language-plaintext highlighter-rouge">myCtx.foo</code> and <code class="language-plaintext highlighter-rouge">myCtx</code> has no member named <code class="language-plaintext highlighter-rouge">foo</code>, the code is rewritten as <code class="language-plaintext highlighter-rouge">myCtx._curr.foo</code>.</p>
<p>As we travel along constructs, we link the current context with its ancestor (<code class="language-plaintext highlighter-rouge">_</code>). This means that for each
combination of constructs, and at each nesting level, we get a uniquely-typed context. At runtime, this context is
nothing more than a pair of pointers, but at compile time it keeps us type-safe. In other words, you can’t reference
a nonexistent field and expect the program to compile.</p>
<p>A more interesting example would thus be</p>
<figure class="highlight"><pre><code class="language-d" data-lang="d"><span class="k">struct</span> <span class="n">MyStruct</span> <span class="p">{</span>
<span class="n">Field</span><span class="p">!</span><span class="kt">ubyte</span> <span class="n">length</span><span class="p">;</span>
<span class="n">YourStruct</span> <span class="n">child</span><span class="p">;</span>
<span class="k">mixin</span> <span class="n">Record</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">struct</span> <span class="n">YourStruct</span> <span class="p">{</span>
<span class="n">Field</span><span class="p">!</span><span class="kt">ubyte</span> <span class="n">whatever</span><span class="p">;</span>
<span class="n">Array</span><span class="p">!(</span><span class="n">Field</span><span class="p">!</span><span class="kt">ubyte</span><span class="p">,</span> <span class="s">"_.length"</span><span class="p">)</span> <span class="n">data</span><span class="p">;</span> <span class="c1">// one level up, then 'length'</span>
<span class="k">mixin</span> <span class="n">Record</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">MyStruct</span> <span class="n">ms</span><span class="p">;</span>
<span class="n">ms</span><span class="p">.</span><span class="n">unpack</span><span class="p">(</span><span class="n">stream</span><span class="p">);</span></code></pre></figure>
<p>When we unpack <code class="language-plaintext highlighter-rouge">MyStruct</code> (which recursively unpacks <code class="language-plaintext highlighter-rouge">YourStruct</code>), a new context <code class="language-plaintext highlighter-rouge">ctx</code> will be created
with <code class="language-plaintext highlighter-rouge">ctx._curr=&ms.child</code> and <code class="language-plaintext highlighter-rouge">ctx._=&ms</code>. When <code class="language-plaintext highlighter-rouge">YourStruct</code> refers to <code class="language-plaintext highlighter-rouge">"_.length"</code>, the string
is implanted into <code class="language-plaintext highlighter-rouge">ctx</code>, yielding <code class="language-plaintext highlighter-rouge">ctx._.length</code>. If we refered to the wrong path or misspelled anything,
it would simply not compile. That, and you don’t need dictionary lookups at runtime – it’s all resolved
during compilation.</p>
<p>So again, this is a very preliminary version of Construct, miles away from production grade,
but you can already see where it’s going.</p>
<p>By the way, you can try out D online at <a href="http://dpaste.dzfl.pl">dpaste</a> and even play around with my
demo version of <code class="language-plaintext highlighter-rouge">dconstruct</code> <a href="http://dpaste.dzfl.pl/f0fccc09707e">over there</a>.</p>
<h2 id="in-short">In Short</h2>
<p>Python will always have a special corner in my heart, but as surprising as it may be (for a guy who’s made
his career over Python), this rather unknown, rapidly-evolving language, D, has become my new language of choice.
It’s expressive, concise and powerful, offers short compilation times (as opposed to <code class="language-plaintext highlighter-rouge">C++</code>) and makes
programming both fun and efficient. It’s the language for the C10M age.</p>
Construct Plus Plus2014-02-26T00:00:00+00:00https://tomerfiliba.github.com/blog/ConstructPlusPlus<p>As you may already know, I’m a type-system junkie. My heart yearns for the strongly-typed languages of the world,
but being a I’m a practical guy, I mostly work with Python. That said, I’ve been keeping myself busy trying to find
a way write a statically-typed version of <a href="http://construct.readthedocs.org/en/latest/">Construct</a>. I started with
Haskell, but quickly gave up when my brain overheated with category theory.</p>
<p>Construct is at least context-sensitive a formalism (I showed in an
<a href="/blog/Survey-of-Construct3/">earlier post</a> that it recognizes the languages like <code class="language-plaintext highlighter-rouge">a^nb^nc^n...z^n</code>),
which limits one’s ability to reason about it (or in my case, embed it in a strongly-type language), but subsets of
Construct are “weak enough” for that. These are known as <em>Pickler Combinators</em>, first described (as far as I can tell)
in Andrew Kennedy’s <a href="http://research.microsoft.com/pubs/64036/picklercombinators.pdf">seminal paper</a>.</p>
<p>The problem I set to solve was that of a time-and-space efficient, statically-typed RPC between Python and C++.
One approach is to encode types into the data, such as JSON, pickle, or <a href="http://msgpack.org/">msgpack</a>.
Another project of mine, <a href="http://rpyc.readthedocs.org/en/latest/">RPyC</a>, utilizes this method too, and for a dynamic
language that’s the most sensible thing to do. But in a richly-typed static language like C++ it would
be a pity to waste both bandwidth and CPU cycles to encode that. It also makes memory management a headache, since
there’s no way to tell in advance how much memory would be need (and what types follow). It also requires recursion
(for nested types), and last but certainly not least – forces you to be dynamically-typed.</p>
<p>The second approach is to have a well-defined IDL, such as <a href="https://code.google.com/p/protobuf/">protobuf</a> or
<a href="http://thrift.apache.org/">Apache Thrift</a>. This approach, however, requires code generation to run during the
build process (an extra toolchain), and, you know – another ugly DSL to learn and maintain.</p>
<p>What I’m about to demonstrate here is a serialization mechanism for C++ that relies only on the type system in order
to produce efficient encoding of arbitrarily-complicated objects – all well-typed and resolved in compile time.
<em>Protobuf without code generation</em>, if you wish.</p>
<p>Besides, I’ve been planning to make use of the new features of <a href="http://www.stroustrup.com/C++11FAQ.html">C++11</a>
for a while now, so let’s bring in the heavy guns!</p>
<h2 id="construct">Construct++</h2>
<p>The basic idea is to have two overloaded template functions for the general case:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">void pack(std::ostream&, const T&)</code></li>
<li><code class="language-plaintext highlighter-rouge">void unpack(std::istream&, T&)</code></li>
</ul>
<p>And have them specialized for concrete and high-order types. If you’re not comfortable with C++11, be sure to
read through the new features because it gets a little scary. Here’s how we begin:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">T</span><span class="p">></span>
<span class="k">typename</span> <span class="n">std</span><span class="o">::</span><span class="n">enable_if</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">is_integral</span><span class="o"><</span><span class="n">T</span><span class="o">>::</span><span class="n">value</span> <span class="o">||</span> <span class="n">std</span><span class="o">::</span><span class="n">is_floating_point</span><span class="o"><</span><span class="n">T</span><span class="o">>::</span><span class="n">value</span><span class="o">>::</span><span class="n">type</span>
<span class="nf">pack</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">ostream</span><span class="o">&</span> <span class="n">stream</span><span class="p">,</span> <span class="k">const</span> <span class="n">T</span><span class="o">&</span> <span class="n">value</span><span class="p">)</span> <span class="p">{</span>
<span class="n">stream</span><span class="p">.</span><span class="n">write</span><span class="p">((</span><span class="kt">char</span><span class="o">*</span><span class="p">)</span><span class="o">&</span><span class="n">value</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">T</span><span class="p">));</span>
<span class="p">}</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">T</span><span class="p">></span>
<span class="k">typename</span> <span class="n">std</span><span class="o">::</span><span class="n">enable_if</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">is_integral</span><span class="o"><</span><span class="n">T</span><span class="o">>::</span><span class="n">value</span> <span class="o">||</span> <span class="n">std</span><span class="o">::</span><span class="n">is_floating_point</span><span class="o"><</span><span class="n">T</span><span class="o">>::</span><span class="n">value</span><span class="o">>::</span><span class="n">type</span>
<span class="nf">unpack</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">istream</span><span class="o">&</span> <span class="n">stream</span><span class="p">,</span> <span class="n">T</span> <span class="o">&</span><span class="n">value</span><span class="p">)</span> <span class="p">{</span>
<span class="n">stream</span><span class="p">.</span><span class="n">read</span><span class="p">((</span><span class="kt">char</span><span class="o">*</span><span class="p">)</span><span class="o">&</span><span class="n">value</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">T</span><span class="p">));</span>
<span class="p">}</span></code></pre></figure>
<p>These functions handle integers and floating point numbers by packing/unpacking a bitwise representation of the
value to/from the stream. Okay, nothing new here, it’s just a casting the value to a <code class="language-plaintext highlighter-rouge">char*</code> and processing
it in the raw. At this point we can write:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="kt">int16_t</span> <span class="n">v</span><span class="p">;</span>
<span class="n">unpack</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">v</span><span class="p">);</span></code></pre></figure>
<p>Big deal.</p>
<h2 id="arrays">Arrays</h2>
<p>Since we’re already dealing with fixed-size data, let’s also handle the case of fixed-size arrays
of packable objects:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">T</span><span class="p">,</span> <span class="n">std</span><span class="o">::</span><span class="kt">size_t</span> <span class="n">N</span><span class="p">></span>
<span class="kt">void</span> <span class="nf">pack</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">ostream</span><span class="o">&</span> <span class="n">stream</span><span class="p">,</span> <span class="k">const</span> <span class="n">T</span> <span class="p">(</span><span class="o">&</span><span class="n">arr</span><span class="p">)[</span><span class="n">N</span><span class="p">])</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">N</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="n">pack</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">arr</span><span class="p">[</span><span class="n">i</span><span class="p">]);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">T</span><span class="p">,</span> <span class="n">std</span><span class="o">::</span><span class="kt">size_t</span> <span class="n">N</span><span class="p">></span>
<span class="kt">void</span> <span class="nf">unpack</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">istream</span><span class="o">&</span> <span class="n">stream</span><span class="p">,</span> <span class="n">T</span> <span class="p">(</span><span class="o">&</span><span class="n">arr</span><span class="p">)[</span><span class="n">N</span><span class="p">])</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">N</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="n">unpack</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">arr</span><span class="p">[</span><span class="n">i</span><span class="p">]);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>Which allows us to write</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="kt">int16_t</span> <span class="n">arr</span><span class="p">[</span><span class="mi">3</span><span class="p">][</span><span class="mi">2</span><span class="p">];</span>
<span class="n">unpack</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">arr</span><span class="p">);</span></code></pre></figure>
<p>Wha?! We’re handling two dimensional arrays here… how’s that possible? Well, thanks to the magic of templates,
we can “unwrap” arrays dimension by dimension. We’re actually just handling a pair of int16_t three times.
It’s all done in compile time of course.</p>
<h2 id="variable-length-data">Variable-Length Data</h2>
<p>Now let’s move on to the more interesting stuff – variable-length data – such as vectors:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="k">typedef</span> <span class="kt">uint8_t</span> <span class="n">length_type</span><span class="p">;</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">T</span><span class="p">,</span> <span class="k">typename</span> <span class="nc">L</span><span class="o">=</span><span class="n">length_type</span><span class="p">></span>
<span class="kt">void</span> <span class="nf">pack</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">ostream</span><span class="o">&</span> <span class="n">stream</span><span class="p">,</span> <span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o"><</span><span class="n">T</span><span class="o">>&</span> <span class="n">vec</span><span class="p">)</span> <span class="p">{</span>
<span class="n">L</span> <span class="n">length</span> <span class="o">=</span> <span class="p">(</span><span class="n">L</span><span class="p">)</span><span class="n">vec</span><span class="p">.</span><span class="n">size</span><span class="p">();</span>
<span class="n">pack</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">length</span><span class="p">);</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">length</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="n">pack</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">vec</span><span class="p">[</span><span class="n">i</span><span class="p">]);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">T</span><span class="p">,</span> <span class="k">typename</span> <span class="nc">L</span><span class="o">=</span><span class="n">length_type</span><span class="p">></span>
<span class="kt">void</span> <span class="nf">unpack</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">istream</span><span class="o">&</span> <span class="n">stream</span><span class="p">,</span> <span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o"><</span><span class="n">T</span><span class="o">>&</span> <span class="n">vec</span><span class="p">)</span> <span class="p">{</span>
<span class="n">L</span> <span class="n">length</span><span class="p">;</span>
<span class="n">unpack</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">length</span><span class="p">);</span>
<span class="n">vec</span><span class="p">.</span><span class="n">resize</span><span class="p">(</span><span class="n">length</span><span class="p">);</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">length</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="n">unpack</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">vec</span><span class="p">[</span><span class="n">i</span><span class="p">]);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>It just adds a prefixes of the vector’s length (as <code class="language-plaintext highlighter-rouge">uint8_t</code>, but you can specify a different type), followed by
the vector’s items. Now we can write:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="n">vector</span><span class="o"><</span><span class="kt">uint16_t</span><span class="o">></span> <span class="n">vec</span><span class="p">;</span>
<span class="n">unpack</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">vec</span><span class="p">);</span></code></pre></figure>
<p>Which takes <code class="language-plaintext highlighter-rouge">"\x03aabbcc"</code> and spews out <code class="language-plaintext highlighter-rouge">[0x6161, 0x6262, 0x6363]</code>. Nice already.</p>
<h2 id="tuples">Tuples</h2>
<p><a href="http://en.cppreference.com/w/cpp/utility/tuple">Tuples</a> are a new feature of C++11 that holds a strongly-typed
heterogeneous sequence of objects, as in <code class="language-plaintext highlighter-rouge">std::tuple<int, char*, float> t(5, "hello", 1.414)</code>. You can think
of tuples as a light-weight <code class="language-plaintext highlighter-rouge">structs</code>, where fields have indexes instead of names.</p>
<p>Iterating over tuples in compile time is a bitch, trust me on that, so I’ll skip the full implementation
(hint: it requires recursion) and just say that we have:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="k">template</span><span class="o"><</span><span class="k">typename</span><span class="o">...</span> <span class="nc">Types</span><span class="p">></span> <span class="kt">void</span> <span class="nf">pack</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">ostream</span><span class="o">&</span> <span class="n">stream</span><span class="p">,</span> <span class="k">const</span> <span class="n">std</span><span class="o">::</span><span class="n">tuple</span><span class="o"><</span><span class="n">Types</span><span class="p">...</span><span class="o">>&</span> <span class="n">tup</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">//...</span>
<span class="p">}</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span><span class="o">...</span> <span class="nc">Types</span><span class="p">></span> <span class="kt">void</span> <span class="nf">unpack</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">ostream</span><span class="o">&</span> <span class="n">stream</span><span class="p">,</span> <span class="n">std</span><span class="o">::</span><span class="n">tuple</span><span class="o"><</span><span class="n">Types</span><span class="p">...</span><span class="o">>&</span> <span class="n">tup</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">//...</span>
<span class="p">}</span></code></pre></figure>
<p>Why do I even bother to show that? The answer will be clear in a moment.</p>
<h2 id="structs">Structs</h2>
<p>Structs pose a rather impossible problem for our packing combinators. First of all, not all structs are
packable: they may hold pointers or some internal state that might be transient. But worse, from our perspective,
is the fact that every struct is different… If you wish, “<code class="language-plaintext highlighter-rouge">vector<T></code> are all alike; every struct is a struct in
its own way”.</p>
<p>So there’s nothing we can do but (a) mark packable structs explicitly and (b) implement a custom pack()/unpack()
semantics for every struct (using inheritance, for example).</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="k">struct</span> <span class="nc">packable</span> <span class="p">{</span>
<span class="k">virtual</span> <span class="kt">void</span> <span class="n">_pack_self</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">ostream</span><span class="o">&</span> <span class="n">stream</span><span class="p">)</span> <span class="k">const</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">virtual</span> <span class="kt">void</span> <span class="n">_unpack_self</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">istream</span><span class="o">&</span> <span class="n">stream</span><span class="p">)</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">};</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">T</span><span class="p">></span>
<span class="k">typename</span> <span class="n">std</span><span class="o">::</span><span class="n">enable_if</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">is_base_of</span><span class="o"><</span><span class="n">packable</span><span class="p">,</span> <span class="n">T</span><span class="o">>::</span><span class="n">value</span><span class="o">>::</span><span class="n">type</span>
<span class="nf">pack</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">ostream</span><span class="o">&</span> <span class="n">stream</span><span class="p">,</span> <span class="k">const</span> <span class="n">T</span><span class="o">&</span> <span class="n">pkd</span><span class="p">)</span> <span class="p">{</span>
<span class="n">pkd</span><span class="p">.</span><span class="n">_pack_self</span><span class="p">(</span><span class="n">stream</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">template</span><span class="o"><</span><span class="k">typename</span> <span class="nc">T</span><span class="p">></span>
<span class="k">typename</span> <span class="n">std</span><span class="o">::</span><span class="n">enable_if</span><span class="o"><</span><span class="n">std</span><span class="o">::</span><span class="n">is_base_of</span><span class="o"><</span><span class="n">packable</span><span class="p">,</span> <span class="n">T</span><span class="o">>::</span><span class="n">value</span><span class="o">>::</span><span class="n">type</span>
<span class="nf">unpack</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">istream</span><span class="o">&</span> <span class="n">stream</span><span class="p">,</span> <span class="n">T</span> <span class="o">&</span><span class="n">pkd</span><span class="p">)</span> <span class="p">{</span>
<span class="n">pkd</span><span class="p">.</span><span class="n">_unpack_self</span><span class="p">(</span><span class="n">stream</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
<p>So we can write</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="k">struct</span> <span class="nc">my_struct</span> <span class="o">:</span> <span class="n">packable</span> <span class="p">{</span>
<span class="kt">uint8_t</span> <span class="n">a</span><span class="p">;</span>
<span class="kt">uint8_t</span> <span class="n">b</span><span class="p">;</span>
<span class="kt">void</span> <span class="n">_pack_self</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">ostream</span><span class="o">&</span> <span class="n">stream</span><span class="p">)</span> <span class="k">const</span> <span class="p">{</span>
<span class="n">pack</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">a</span><span class="p">);</span>
<span class="n">pack</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">b</span><span class="p">);</span>
<span class="p">}</span>
<span class="kt">void</span> <span class="n">_unpack_self</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">istream</span><span class="o">&</span> <span class="n">stream</span><span class="p">)</span> <span class="p">{</span>
<span class="n">unpack</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">a</span><span class="p">);</span>
<span class="n">unpack</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">b</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="n">my_struct</span> <span class="n">s</span><span class="p">;</span>
<span class="n">unpack</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">s</span><span class="p">);</span></code></pre></figure>
<p>But notice how mechanical the <code class="language-plaintext highlighter-rouge">_pack_self</code>/<code class="language-plaintext highlighter-rouge">_unpack_self</code> methods are – we’d really wish to auto-generate
them somehow. We can use a preprocessor macro, but how could we generate a line for each member?
We don’t have meta-for-loops after all… or do we? Enter tuples!</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="cp">#define PACKED(...) \
void _pack_self(std::ostream& stream) const override { \
auto tmp = std::tie(__VA_ARGS__); \
pack(stream, tmp); \
} \
void _unpack_self(std::istream& stream) override { \
auto tmp = std::tie(__VA_ARGS__); \
unpack(stream, tmp); \
}</span></code></pre></figure>
<p><code class="language-plaintext highlighter-rouge">std::tie</code> builds a strongly-typed tuple of references, meaning they refer to the member variables instead of
holding a copy. Thanks to the power of template programming, packing/unpacking this tuple utlimately boils down
to something like the first version given above – a line for each member.</p>
<p>So here’s the whole deal:</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="k">struct</span> <span class="nc">first_struct</span> <span class="o">:</span> <span class="n">packable</span> <span class="p">{</span>
<span class="kt">uint8_t</span> <span class="n">x</span><span class="p">;</span>
<span class="kt">uint16_t</span> <span class="n">y</span><span class="p">;</span>
<span class="n">PACKED</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
<span class="p">};</span>
<span class="k">struct</span> <span class="nc">second_struct</span> <span class="o">:</span> <span class="n">packable</span> <span class="p">{</span>
<span class="kt">uint8_t</span> <span class="n">a</span><span class="p">;</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span> <span class="n">b</span><span class="p">;</span> <span class="c1">// note this is a variable length string!</span>
<span class="kt">int16_t</span> <span class="n">c</span><span class="p">[</span><span class="mi">3</span><span class="p">][</span><span class="mi">2</span><span class="p">];</span>
<span class="n">first_struct</span> <span class="n">d</span><span class="p">;</span>
<span class="k">enum</span> <span class="o">:</span> <span class="kt">uint16_t</span> <span class="p">{</span>
<span class="n">A</span> <span class="o">=</span> <span class="mi">1</span><span class="p">,</span>
<span class="n">B</span> <span class="o">=</span> <span class="mi">2</span><span class="p">,</span>
<span class="n">C</span> <span class="o">=</span> <span class="mi">3</span>
<span class="p">}</span> <span class="n">e</span><span class="p">;</span>
<span class="n">PACKED</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">c</span><span class="p">,</span> <span class="n">d</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span>
<span class="p">};</span></code></pre></figure>
<p>Ain’t that cool? And it even works!</p>
<figure class="highlight"><pre><code class="language-cpp" data-lang="cpp"><span class="n">std</span><span class="o">::</span><span class="n">stringstream</span> <span class="n">ss</span><span class="p">;</span>
<span class="n">ss</span> <span class="o"><<</span> <span class="s">"A</span><span class="se">\x05</span><span class="s">helloa0b0c0d0e0f0Bg0</span><span class="se">\x02\x00</span><span class="s">XXXXXXXXXX"</span><span class="p">;</span>
<span class="n">second_struct</span> <span class="n">x</span> <span class="o">=</span> <span class="p">{};</span>
<span class="n">unpack</span><span class="p">(</span><span class="n">ss</span><span class="p">,</span> <span class="n">x</span><span class="p">);</span>
<span class="n">std</span><span class="o">::</span><span class="n">cout</span> <span class="o"><<</span> <span class="s">"a="</span> <span class="o"><<</span> <span class="p">(</span><span class="kt">int</span><span class="p">)</span><span class="n">x</span><span class="p">.</span><span class="n">a</span> <span class="o"><<</span> <span class="s">","</span> <span class="o"><<</span> <span class="s">"b="</span> <span class="o"><<</span> <span class="n">x</span><span class="p">.</span><span class="n">b</span> <span class="o"><<</span> <span class="s">","</span>
<span class="o"><<</span> <span class="s">"c="</span> <span class="o"><<</span> <span class="n">x</span><span class="p">.</span><span class="n">c</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span> <span class="o"><<</span> <span class="s">".."</span> <span class="o"><<</span> <span class="n">x</span><span class="p">.</span><span class="n">c</span><span class="p">[</span><span class="mi">2</span><span class="p">][</span><span class="mi">1</span><span class="p">]</span> <span class="o"><<</span> <span class="s">","</span>
<span class="o"><<</span> <span class="s">"d=("</span> <span class="o"><<</span> <span class="n">x</span><span class="p">.</span><span class="n">d</span><span class="p">.</span><span class="n">x</span> <span class="o"><<</span> <span class="s">","</span> <span class="o"><<</span> <span class="n">x</span><span class="p">.</span><span class="n">d</span><span class="p">.</span><span class="n">y</span> <span class="o"><<</span> <span class="s">"),"</span>
<span class="o"><<</span> <span class="s">"e="</span> <span class="o"><<</span> <span class="n">x</span><span class="p">.</span><span class="n">e</span> <span class="o"><<</span> <span class="n">std</span><span class="o">::</span><span class="n">endl</span><span class="p">;</span></code></pre></figure>
<p>Which prints <code class="language-plaintext highlighter-rouge">a=65,b=hello,c=12385..12390,d=(B,12391),e=2</code>.</p>
<p>The full code snippet <a href="https://gist.github.com/tomerfiliba/9216420">is available here</a>, tested with <code class="language-plaintext highlighter-rouge">g++4.8</code>
and <code class="language-plaintext highlighter-rouge">clang++3.4</code> with <code class="language-plaintext highlighter-rouge">-std=c++11</code>.</p>
A Python Crash Course for the Statically Typed Programmer2013-11-15T00:00:00+00:00https://tomerfiliba.github.com/etc/python-crash-course<section>
Python is a multi-paradigm (hybrid) language. It's fully object-oriented but has strong functional roots.
Note that this isn't a [beginner's tutorial](http://learnpython.org/) but a quick reference for the language
and its features that should allow you to write basic Python ASAP.
If you had taken any academic course that involves programming, Python will most likely resemble pseudo code to you
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">factorial</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
<span class="k">if</span> <span class="n">n</span> <span class="o"><=</span> <span class="mi">1</span><span class="p">:</span>
<span class="k">return</span> <span class="mi">1</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="n">n</span> <span class="o">*</span> <span class="n">factorial</span><span class="p">(</span><span class="n">n</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
<span class="c1"># or
</span>
<span class="k">def</span> <span class="nf">factorial</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
<span class="n">res</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">while</span> <span class="n">n</span><span class="p">:</span>
<span class="n">res</span> <span class="o">*=</span> <span class="n">n</span>
<span class="n">n</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="n">res</span></code></pre></figure>
</section>
<section>
You'll see inheritance and class diagrams along side with constructs imported from [Haskell](http://www.haskell.org)
and [LISP](http://en.wikipedia.org/wiki/Lisp_%28programming_language%29).
Python is dynamically-typed (as opposed to statically-typed) but has strong-typing
(as opposed to Perl or Javascript)
<img src="/static/res/2013-11-15-perl.png" />
</section>
<section>
Syntax reference (1)
```
# printing to stdout
print EXPR
# assignment
VAR = EXPR
# conditions
if COND:
SUITE
[elif COND:
SUITE]
...
[else:
SUITE]
# for-loops
for VAR in ITERABLE:
SUITE
[break]
[continue]
# while-loops
while COND:
SUITE
[break]
[continue]
```
</section>
<section>
Syntax reference (2)
```
# try-except
try:
SUITE
except EXCEPTION [as ex]:
SUITE
...
[else: # iff there was no exception
SUITE]
[finally: # always be performed
SUITE]
# raising exceptions
raise EXCEPTION(...)
# importing modules or attributes
import MODULE
from MODULE import NAME
# defining functions
def NAME([a, [b, ...]]):
SUITE
[return EXPR]
[yield EXPR]
# defining classes
class NAME([BASE, [...]]):
SUITE
```
</section>
<section>
The interactive interpreter is your friend! I mapped ``F11`` on my keyboard to fire up an interpreter...
it's better than any calculator you'll ever have
```
$ python
Python 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> 5 + 6
11
```
</section>
<section>
For people with a C background:
* You don't declare variables - you just assign them
* And you can reassign them to different types
* Python doesn't have ``do-while`` (use ``while``) nor does it have ``switch`` (use ``if``s or
dispatch tables)
* Assignment is a statement, **not an expression**.
* You cannot write ``if (a = foo()) == 5:``
* There's no such thing as passing by-value. It's always by-reference.
* So you might need to explicitly **copy** objects some times
* Or (preferably) create new objects from existing ones
For people with a C++/Java background:
* **Everything** is a first-class object
* Integers, functions, types, stack frames, tracebacks, etc.
* There are no privates, only conventions. Members that start with ``_`` are not to be manipulated directly.
* There's no ``new``, just *invoke* the class like a function.
* ``inst = MyClass(1, 2, 3)``
</section>
<section>
Duck Typing goes Nuclear
<img src="http://www.haaretz.com/polopoly_fs/1.416769.1331009149!/image/2349077637.jpg_gen/derivatives/landscape_640/2349077637.jpg" />
In low-level programming languages, types dictate a **memory layout**.
In high-level languages, on the other hand, compile-time types are merely **constraints**
on what you're allowed to do with an object.
Being an interpreted language, Python gives up on type-checking and instead adopts the spirit of
"it's easier to ask for forgiveness than permission". Just try and see what happens.
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> def foo(this, that):
... return (this + that) * 2
...
>>> foo(3, 5)
16
>>> foo("hello", "world")
'helloworldhelloworld'</code></pre></figure>
</section>
<section>
Python comes with lots of useful built-in types
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> 5+3
8
>>> 27**63
149939874158678820041423971072487610193361136600
3344657118522818557991334322919287339806483L
>>> 5+3j
(5+3j)
>>> 1/(5+3j)
(0.14705882352941177-0.08823529411764706j)
>>> [1,2,3]
[1, 2, 3]
>>> [1,2,3] + [4,5,6]
[1, 2, 3, 4, 5, 6]
>>> (1,2,3)
(1, 2, 3)
>>> d={"a":4, 7:()}
>>> d["a"]
4
>>> d[7]
()
>>> True, False, None
(True, False, None)
>>> set([2,6,2,7,2,8,6])
set([8, 2, 6, 7])</code></pre></figure>
</section>
<section>
String manipulation is a pleasure with Python
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> "hello" + "world"
'helloworld'
>>> a="this is a single line string"
>>> a[5]
'i'
>>> a[5:12]
'is a si'
>>> a.upper()
'THIS IS A SINGLE LINE STRING'
>>> a.count("i")
5
>>> a.replace("i", "j")
'thjs js a sjngle ljne strjng'
>>> a.startswith("thi")
True
>>> "single" in a
True
>>> "multiple" in a
False
>>> a.split()
['this', 'is', 'a', 'single', 'line', 'string']
>>> a.split("i")
['th', 's ', 's a s', 'ngle l', 'ne str', 'ng']</code></pre></figure>
</section>
<section>
Encoding strings like a boss
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> "hello".encode("hex")
'68656c6c6f'
>>> "hello".encode("utf16")
'\xff\xfeh\x00e\x00l\x00l\x00o\x00'
>>> "hello".encode("zlib")
'x\x9c\xcbH\xcd\xc9\xc9\x07\x00\x06,\x02\x15'</code></pre></figure>
String interpolation
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> "My name is %s. You %s, prepare to %s" % ("Inigo Montoya",
... "killed my father", "die")
'My name is Inigo Montoya. You killed my father, prepare to die'
>>>
>>> "My name is %03d" % (7,)
'My name is 007'</code></pre></figure>
And joining strings is surprisingly useful
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> ":".join(["AA", "BB", "CC"])
'AA:BB:CC'</code></pre></figure>
</section>
<section>
Multi-line strings
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> b="""this is
... a multi
... line string"""
>>> b
'this is\na multi\nline string'
>>> b.splitlines()
['this is', 'a multi', 'line string']</code></pre></figure>
</section>
<section>
If I had time to show you only three functions, they will be
* ``help()``
* ``dir()``
* ``type()``
Everything else you can discover on your own.
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> help
Type help() for interactive help, or help(object) for help about object.
>>> dir()
['__builtins__', '__doc__', '__name__', '__package__']
>>> help(dir)
dir([object]) -> list of strings
If called without an argument, return the names in the current scope.
Else, return an alphabetized list of names comprising (some of) the attributes
>>> dir("hello")
['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__for
_mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__',
nter', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'in
index', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswi']
>>> type(5)
<type 'int'>
>>> type("hello")
<type 'str'></code></pre></figure>
</section>
<section>
Next, let's meet some types and learn how to convert (not **cast**) values from one type to the other
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> int
<type 'int'>
>>> str
<type 'str'>
>>> list
<type 'list'>
>>> int(5.1)
5
>>> int("5")
5
>>> str(5)
'5'
>>> list("hello")
['h', 'e', 'l', 'l', 'o']</code></pre></figure>
</section>
<section>
Types matter
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> 5 + "6"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'
>>> 5 + int("6")
11
>>> str(5) + "6"
'56'</code></pre></figure>
Repr(esenation)
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> repr(5)
'5'
>>> repr("hello")
"'hello'"
>>> print "Hello %s" % ("foo\n\tbar",)
Hello foo
bar
>>> print "Hello %r" % ("foo\n\tbar",)
Hello 'foo\n\tbar'</code></pre></figure>
</section>
<section>
Lists
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> range(10)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> range(10,20)
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
>>> range(10,20,2)
[10, 12, 14, 16, 18]
>>> range(20,10,-2)
[20, 18, 16, 14, 12]
>>> len([2,5,2,6,2,6])
6
>>> len("hello")
5
>>> a=[2,5,2,6,2,6]
>>> max(a)
6
>>> min(a)
2
>>> b=["what", "is", "this", "thing"]
>>> max(b)
'what'
>>> max(b, key = len)
'thing'</code></pre></figure>
</section>
<section>
Slicing
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> a=range(10,20)
>>> a[0]
10
>>> a[-1]
19
>>> a[3:7]
[13, 14, 15, 16]
>>> a[7:]
[17, 18, 19]
>>> a[-3:]
[17, 18, 19]
>>> a[:-3]
[10, 11, 12, 13, 14, 15, 16]</code></pre></figure>
</section>
<section>
Lambda functions
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> lambda x: x * 2
<function <lambda> at 0x0297EEB0>
>>> f = lambda x: x * 2
>>> f(6)
12
>>> map(f, range(10))
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
>>> filter(lambda x: x % 2 == 0, range(10))
[0, 2, 4, 6, 8]</code></pre></figure>
</section>
<section>
Working with files
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> f = open("/etc/profile")
>>> f.read(100)
'# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))\n# and Bourne compatible shell'
>>> f.readline()
's (bash(1), ksh(1), ash(1), ...).\n'
>>> f.readline()
'\n'
>>> f.readline()
'if [ "$PS1" ]; then\n'
>>> list(f)[:4]
[' if [ "$BASH" ] && [ "$BASH" != "/bin/sh" ]; then\n',
' # The file bash.bashrc already sets the default PS1.\n',
" # PS1='\\h:\\w\\$ '\n",
' if [ -f /etc/bash.bashrc ]; then\n']
>>> f.close()</code></pre></figure>
</section>
<section>
</section>
<section>
Working with resources
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> with open("/etc/profile") as f:
... f.read(100)
...
'# /etc/profile: system-wide .profile file for the Bourne shell (sh(1))\n# and Bourne compatible shell'</code></pre></figure>
The ``with`` block automatically closes the file. It's more general than this, of course, as it's
not limited to files. It simply ensures that the necessary cleanup that's associated with the object
will be performed when you finish the ``with`` block.
For example, you can use ``with`` to commit a transaction to a DB or do a rollback on failure,
etc. It's a very useful pattern.
</section>
<section>
List comprehension: remember ``map`` and ``filter``? Well, it's time to forget about them
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> [x for x in range(10)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> [x * 2 for x in range(10)]
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
>>> [x for x in range(10) if x % 2 == 0]
[0, 2, 4, 6, 8]
>>> [x * 2 for x in range(10) if x % 2 == 0]
[0, 4, 8, 12, 16]</code></pre></figure>
</section>
<section>
Yo comprendo!
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> [i * j for i in range(1,5) for j in range(1,5)]
[1, 2, 3, 4, 2, 4, 6, 8, 3, 6, 9, 12, 4, 8, 12, 16]
>>> [[i * j for i in range(1,5)] for j in range(1,5)]
[[1, 2, 3, 4], [2, 4, 6, 8], [3, 6, 9, 12], [4, 8, 12, 16]]
>>> [" ".join(["%3d" % (i * j,) for i in range(1,5)])
... for j in range(1,5)]
[' 1 2 3 4', ' 2 4 6 8', ' 3 6 9 12',
' 4 8 12 16']</code></pre></figure>
And the whole multiplication table in just one line!
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> print "\n".join([" ".join(["%3d" % (i * j,) for i in range(1,11)])
... for j in range(1,11)])
1 2 3 4 5 6 7 8 9 10
2 4 6 8 10 12 14 16 18 20
3 6 9 12 15 18 21 24 27 30
4 8 12 16 20 24 28 32 36 40
5 10 15 20 25 30 35 40 45 50
6 12 18 24 30 36 42 48 54 60
7 14 21 28 35 42 49 56 63 70
8 16 24 32 40 48 56 64 72 80
9 18 27 36 45 54 63 72 81 90
10 20 30 40 50 60 70 80 90 100</code></pre></figure>
</section>
<section>
Chillex! Be lazy
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> def myfunc():
... yield 1
... yield 2
... yield 3
...
>>> g=myfunc()
>>> g
<generator object myfunc at 0x02A3A8F0>
>>> g.next()
1
>>> g.next()
2
>>> g.next()
3
>>> g.next()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration</code></pre></figure>
</section>
<section>
Generators let us be as general as we wish while paying only for what we're actually using (lazy evaluation)
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> def fib():
... a = b = 1
... while True:
... yield a
... a, b = b, a + b
...
>>> g = fib()
>>> g.next(), g.next(), g.next(), g.next(), g.next(), g.next(), g.next()
(1, 1, 2, 3, 5, 8, 13)</code></pre></figure>
The ``itertools`` module has some nifty utilities that we can use
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> import itertools
>>> list(itertools.islice(fib(),10))
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]
>>>
>>> list(itertools.takewhile(lambda x: x < 50, fib))
[1, 1, 2, 3, 5, 8, 13, 21, 34]
>>>
>>> list(itertools.combinations([1,2,3,4],2))
[(1, 2), (1, 3), (1, 4), (2, 3), (2, 4), (3, 4)]
>>>
>>> list(itertools.permutations([1,2,3]))
[(1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2), (3, 2, 1)]</code></pre></figure>
</section>
<section>
Remember list comprehensions? Forget them too. Generator comprehensions are the new black!
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> (x for x in range(10))
<generator object <genexpr> at 0x02A37F30>
>>> (x * 2 for x in range(10))
<generator object <genexpr> at 0x02A34670>
>>> (x * 2 for x in range(10) if x % 2 == 0)
<generator object <genexpr> at 0x02A37918></code></pre></figure>
Huh? Of course you won't see anything... you have to consume the generator in order for it to produce values.
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> list(x * 2 for x in range(10) if x % 2 == 0)
[0, 4, 8, 12, 16]</code></pre></figure>
In fact, list comprehensions are a syntactic sugar for ``list(``generator comprehension``)``
</section>
<section>
List comprehensions, as the name suggests, **build a list**. This can be expensive some times,
especially when you don't need the intermediate values. E.g., if you just want to get the sum of the elements,
there's no need to actually hold all of them in memory. Generators are the key to efficient programming.
For example, ``xrange`` is like ``range`` but returns a generator instead.
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> sum(range(1000000000))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
MemoryError
>>> sum(xrange(1000000000))
499999999500000000L</code></pre></figure>
</section>
<section>
## Object Oriented, Too ##
Don't worry though. Python isn't all geared for functional programming. We have classes too!
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> class Animal(object):
... def __init__(self, name):
... self.name = name
... def drink(self):
... print "*Gulp gulp gulp*"
...
>>> class Dog(Animal):
... def drink(self):
... Animal.drink(self) # or super(Dog, self).drink()
... print "*Tongue drips*"
... def bark(self):
... print "Woof woof woof"
...
>>> rex = Dog("Rex t3h Dawg")
>>> rex.drink()
*Gulp gulp gulp*
*Tongue drips*
>>> rex.bark()
Woof woof woof</code></pre></figure>
</section>
<section>
Some notes for Java folks:
* Constructors are inherited! Why wouldn't they?!
* You can implement an exception class in Python in just one line: ``class MyException(Exception): pass``
* In Java/C# you have to reimplement all constructors as well
* Python doesn't have function/method overloading
* But given that most of the code is duck-typed anyway (and can introspect the received object at runtime)
it doesn't make a real difference
</section>
<section>
But if we stopped here you'd think it's just a variation of Java. We're way cooler than Java!
For example, instances (such as ``rex`` above) are actually just dictionaries that hold the instance's attributes.
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> rex.__dict__
{'name' : 'Rex t3h Dawg'}</code></pre></figure>
That's why we can add attributes on the fly
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> rex.tail = "recursion"
>>> rex.__dict__
{'name' : 'Rex t3h Dawg', 'tail' : 'recursion'}</code></pre></figure>
</section>
<section>
But classes ain't no different! They are just dictionaries of their methods
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> Animal.__dict__.keys()
['__module__', 'drink', '__dict__', '__init__', '__weakref__', '__doc__']
>>>
>>> def sleep(self):
... print "Zzzzzz"
...
>>> Animal.sleep = sleep # monkey-patching
>>> rex.sleep()
Zzzzzz</code></pre></figure>
</section>
<section>
Now it's time to discuss **special methods**. You've seen them already when we invoked ``dir("hello")`` -- they
take the form of ``__xxx__``, and surprisingly or not, they make up most of what we've seen so far.
Virtually all language constructs map to special methods underneath:
* ``a + b`` is actually ``a.__add__(b)``
* ``a[x]`` maps to ``a.__getitem__(x)``
* ``str(a)`` invokes ``a.__str__()``
* ``a.b`` translates to ``a.__getattr__("b")``
* ``f(a,b,c)`` runs ``f.__call__(a, b, c)``
* So yes, ``f.__call__.__call__.__call__(a, b, c)`` as well
* ``Dog("foo")`` creates the instance using ``__new__`` and initializes it in ``__init__``
You can invoke them yourself, of course, they're just methods:
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> (5).__add__(6)
11</code></pre></figure>
But why would you do that? Don't be silly.
</section>
<section>
Remember I said **everything** is an object? Well, I meant it.
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> Animal.__base__
<type 'object'>
>>> Dog.__base__
<class '__main__.Animal'>
>>> Dog.__mro__
(<class '__main__.Dog'>, <class '__main__.Animal'>, <type 'object'>)
>>> type(Dog)
<type 'type'></code></pre></figure>
And it does gets mindboggling
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> type
<type 'type'>
>>> type.__base__
<type 'object'>
>>> type(type)
<type 'type'>
>>> type(object)
<type 'type'></code></pre></figure>
</section>
<section>
The MRO (method resolution order) is actually very important. It determines what happens when you resolve
attributes (``__getattr__``) on an object. For instance, ``rex.foo`` will first try ``rex.__dict__``, move up
to ``Dog.__dict__`` and then to ``Animal.__dict__``, until we reach ``object.__dict__`` where we'll fail.
And that's the whole object model.
Dictionaries are also crucial to functions. A function is basically just code that evaluates in a local
scope and has access to a global scope (module-level). Each of these scopes is... a dictionary. When you assign
a variable, you basically insert an element into the scope dictionary.
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> globals()
{'__builtins__': <module '__builtin__' (built-in)>}
>>> def f(a, b):
... print locals()
... c = a+b
... print locals()
...
>>> globals()
{'__builtins__': <module '__builtin__' (built-in)>, 'f': <function f at 0x028CEEB0>}
>>>
>>> f(6,7)
{'a': 6, 'b': 7}
{'a': 6, 'c': 13, 'b': 7}</code></pre></figure>
</section>
<section>
And no Python tutorial can do without
<a href="http://xkcd.com/353/" title="Antigravity"><img src="/static/res/2013-11-15-xkcd.png" /></a>
</section>
<section>
## When in Doubt ##
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> import this
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.</code></pre></figure>
</section>
<section>
<img src="/static/res/2013-11-15-iknow.jpg" />
</section>
Tales of a Stressed Kernel2013-07-27T00:00:00+00:00https://tomerfiliba.github.com/blog/mmapcp<p>Spending most of our time (as developers) in the high-level world, it’s easy to occasionally forget the
true nature of our systems and how fragile they really are. Well, perhaps <em>fragile</em> is not the right word here,
more like <em>intricate</em> or perhaps <em>chaotic</em> - you cannot fully predict the effect of one operation on the rest
of the system. In this case, a simple file copy got our machines to freeze for many long seconds… Huh?</p>
<p>Our machines are given loads of RAM but no swap space, to ensure deterministic memory access times. The memory
is pre-allocated to different processes based on a configuration file, with a small portion reserved for
the kernel. Memory is allocated from <code class="language-plaintext highlighter-rouge">/dev/shm</code> (mounted as <a href="http://en.wikipedia.org/wiki/Tmpfs">tmpfs</a>),
where it’s also exposed as files.</p>
<p>When a process crashes we may need some of those <code class="language-plaintext highlighter-rouge">/dev/shm</code> files for debugging, so we have a tool that runs
whenever there’s a crash to collect system info. Among other things, it used to copy some of these shared-memory
files to disk (using a simple <code class="language-plaintext highlighter-rouge">cp</code> or <code class="language-plaintext highlighter-rouge">shutil.copy()</code>). No surprise there. But every once in a while, and
strongly-correlated to times when this collector was running, some time-critical processes timed-out when writing
to the disks for no apparent reason, leading to catastrophic results. We’ve spent about a month trying to
pin-point what’s taking so many system resources. Many things came to mind: CPU consumption?
Accessing special <code class="language-plaintext highlighter-rouge">/proc</code> files that got the kernel busy? Running SCSI or other hardware commands?
Jamming the IO bus with data from the copy?</p>
<p>The mystery was solved just last Thursday, when I realized copying files from <code class="language-plaintext highlighter-rouge">/dev/shm</code> caused all sorts
of system-wide hiccups – not only timeouts when writing to other disks. It seemed that existing processes did
get runtime, but no new processes could be forked. Sometimes running <code class="language-plaintext highlighter-rouge">ls</code> took ~20 seconds. Other times
simple (non-file system) tools like <code class="language-plaintext highlighter-rouge">date</code> hanged for a while. When it was apparent it’s system-wide, the
explanation was quite obvious: the kernel ran out of memory. Since there’s no swap space, there’s nothing it
could do and kernel threads just blocked until there was enough room for their allocations.</p>
<p>But why would the kernel get so low on memory? After all, we’re copying files <em>from memory</em> (tmpfs) to the disk.
Well, that’s seems like a bug:</p>
<blockquote>
<p>I think I’ve finally figured this out. It’s a kernel bug – I’m guessing that under normal circumstances,
the “cached” column in the free command “doesn’t count” towards how much memory the system thinks it’s using.
After all, it’s just cached copies of stuff that should be elsewhere, and if you run out of memory,
you can safely dump that, right? Unfortunately, /dev/shm is counted under cached rather than used memory
(as I discovered in an earlier post).</p>
<p><a href="https://bbs.archlinux.org/viewtopic.php?pid=390313#p390313">https://bbs.archlinux.org/viewtopic.php?pid=390313#p390313</a></p>
</blockquote>
<p>Simplistically, file copy is a simple loop that reads a chunk of data from the source file and writes it
to the destination file, until it transfers everything. When we read from a file, the kernel needs to
allocate a kernel-space buffer and copy it to userland. And when we write it back to the destination file,
the kernel first copies the userland buffer into kernel-space and links is to the device’s queue (to be
evicted at the driver’s discretion). The <code class="language-plaintext highlighter-rouge">write()</code> call returns as soon as the kernel places the
buffer into the queue, so it might “pile up” there for some time before actually being evicted,
depleting kernel memory.</p>
<p>Ugggh. A simple copy brought our system to a halt. The solution was just as simple – we don’t want
(and neither do we <em>need</em>) to use kernel buffers here. The source file already resides in memory. Instead of
<code class="language-plaintext highlighter-rouge">read()</code>ing it, we can just <code class="language-plaintext highlighter-rouge">mmap()</code> the whole of it. And as for the destination file, we open it with
<a href="http://man7.org/linux/man-pages/man2/open.2.html">O_DIRECT</a>, so as not to use kernel buffers along
the way. I christened this new tool <code class="language-plaintext highlighter-rouge">mmapcp</code>.</p>
<blockquote>
<p>The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid,
and was probably designed by a deranged monkey on some serious mind-controlling substances.</p>
<p>– Linus</p>
</blockquote>
<p>Well, Linus, at least you were kind enough to let it stay :)</p>
Pool Request2013-06-16T00:00:00+00:00https://tomerfiliba.github.com/etc/pool-request<p><a href="/static/res/2013-06-16-pool-req.jpg">
<img src="/static/res/2013-06-16-pool-req.jpg" /></a></p>
TTYs: Never gets boring2013-06-16T00:00:00+00:00https://tomerfiliba.github.com/blog/TTYs<p>Just a short rant: I’m working on an interactive console used for debugging a computer cluster. It connects
to all nodes in the cluster and provides you with a single place to run queries. It uses the new (not yet
officially-released) <a href="https://rpyc.readthedocs.org/en/latest/api/utils_zerodeploy.html#api-zerodeploy">zero-deploy</a>
feature of RPyC, which sets up a secure, single-use RPyC server on a machine, requiring only SSH access.
Once the client connection closes, the zero-deployed server will shut down and delete itself from the file system.</p>
<p>It’s a cool feature on its own (and I’ll blog about it soon), but there’s a reason I’m getting you through all
of the details here. You see, the debugging console fires up SSH subprocesses in the background, over which RPyC
connections are tunneled… and then the strangest thing happened. I was running a query which was taking too
long and hit Ctrl+C to kill it and return to the interpreter. The query indeed stopped, but all of my RPyC
connections have died with it. Huh?</p>
<p>Here’s a really short way to reproduce this scenario:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> from subprocess import Popen, PIPE
>>> p=Popen(["sleep", "60"], stdin=PIPE, stdout=PIPE, stderr=PIPE)
>>>
>>> p.poll() # poll() returns None as the process is still running in the background
>>>
>>> # now hit Ctrl+C in the interactive prompt
KeyboardInterrupt
>>>
>>> p.poll() # and voila, `sleep` was killed by SIGINT
-2</code></pre></figure>
<p>It’s terribly confusing at first, but it happens because child processes inherit their paren’t session ID.
Terminal events, such as <code class="language-plaintext highlighter-rouge">SIGINT</code> and <code class="language-plaintext highlighter-rouge">SIGHUP</code>, are dispatched to all processes belonging to the terminal’s
process group, so it’s not just the Python interpreter to receive the signal – every child process it spawned
will also suffer. In my case, it killed all of the SSH tunnels I had set up.</p>
<p>The solution is to <a href="http://linux.die.net/man/2/setsid">setsid</a> before <code class="language-plaintext highlighter-rouge">exec</code>ing the child:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> import os
>>> p=Popen(["sleep", "60"], stdin=PIPE, stdout=PIPE, stderr=PIPE, preexec_fn=os.setsid)
>>> p.poll()
>>>
KeyboardInterrupt
>>> p.poll()
>>></code></pre></figure>
<p>So I had to add this feature to <a href="http://plumbum.readthedocs.org/">plumbum</a>, and while I was at it, I also added
<a href="https://github.com/tomerfiliba/plumbum/blob/master/plumbum/daemons.py">daemonization</a> support. In other words,
I’ll have to release 1.3 soon – even though I released 1.2 not two weeks ago. Life’s a bitch and TTYs are the
mother of all monsters :)</p>
Academia2013-06-06T00:00:00+00:00https://tomerfiliba.github.com/blog/Academia<p>I thought it’d be useful to publish some of the seminars/technical papers that I worked on when
studying at <a href="http://tau.ac.il">Tel Aviv University</a>, instead of letting them rot in a drawer.
Hope you find it useful/interesting.</p>
<h2 id="chierchia-on-plurality-of-mass-nouns">Chierchia: On Plurality of Mass Nouns</h2>
<p>A technical report on Gennaro Chierchia’s paper, <em>Plurality of Mass Nouns and the Notion of
“Semantic Parameter”</em>. Chierchia offers a <a href="http://en.wikipedia.org/wiki/Lattice_(order)">lattice-theoretical</a>
treatment for pluralities of all sorts (especially the mass/count noun distinction).
I found it very interesting, but it quite technical and requires some courses in formal semantics
so as not to get lost.</p>
<p><a href="/static/academia/Chierchia/techrep.pdf">
<img src="/static/res/pdf-icon.png" /></a></p>
<h2 id="print-scan-resilient-watermarking-in-the-wavelet-domain">Print-Scan Resilient Watermarking in the Wavelet Domain</h2>
<p>I began working on this project with very little expectations, but it introduced me to the world
of signal processing in general, and <a href="http://www.scipy.org/">SciPy</a> more specifically. It also
forced me to learn Reed-Solomon codes (and since I couldn’t find any Python implementation back
in the day, I released <a href="https://pypi.python.org/pypi/reedsolo">reedsolo</a>).</p>
<p>The purpose of the project was to develop an invisible image-watermarking scheme that survives
printing (e.g., billboard signs) and sustain considerable damage and noise, much like QR codes.
The idea was that you could embed invisible “QR codes” into images, which could be scanned by
mobile devices and open a URL. I didn’t get that far, but “lab tests” were positive :)</p>
<p>Anyhow, it could serve as a great introduction to anyone who’s starting with watermarking, wavelets
or SciPy.</p>
<p><a href="/static/academia/DWT-Watermarking/image-project.pdf">
<img src="/static/res/pdf-icon.png" /></a></p>
<p>Code:</p>
<ul>
<li><a href="/static/academia/DWT-Watermarking/watermarker.py">Watermarking library</a></li>
<li><a href="/static/academia/DWT-Watermarking/embed.py">Embedder</a></li>
<li><a href="/static/academia/DWT-Watermarking/extract.py">Extractor</a></li>
</ul>
<h2 id="earley-cfg-parser">Earley CFG Parser</h2>
<p>A Python (and Java) implementation of the <a href="http://en.wikipedia.org/wiki/Earley_parser">Earley parser</a>
for <a href="http://en.wikipedia.org/wiki/Context-free_grammar">Context Free Grammar (CFG)</a>. Earley is
the “most efficient” parser for CFG, as it does only the “necessary amount of computation”. It achieves
<code class="language-plaintext highlighter-rouge">O(n)</code> time for LR(k) grammars, <code class="language-plaintext highlighter-rouge">O(n^2)</code> for unambiguous grammars, and <code class="language-plaintext highlighter-rouge">O(n^3)</code> in the
general case.</p>
<ul>
<li><a href="https://github.com/tomerfiliba/tau/blob/master/earley3.py">Python implementation</a></li>
<li><a href="https://github.com/tomerfiliba/tau/blob/master/Earley.java">Java implementation</a></li>
</ul>
<p>This implementation includes more than just the parser itself - it is also able to extract the
entire parse forest (all possible derivations) from a sentence. It relates quite well to my previous
blog post, <a href="/blog/Cartesian-Tree-Product/">Cartesian Tree-Products</a>.</p>
<h2 id="tree-insertion-grammar-tig-parser">Tree Insertion Grammar (TIG) Parser</h2>
<p><strong>Tree Insertion Grammar</strong> (TIG) is a formalism that spun off of
<a href="http://en.wikipedia.org/wiki/Tree-adjoining_grammar">Tree Adjoining Grammar (TAG)</a>. The two are
very similar and share common properties – in both, the grammar is defined as trees which are
“embedded” one into the other – but TIG allows trees to be wrapped from only one side, not both.
This restriction makes TIG of equivalent power to CFG, while TAG is strictly stronger and
requires <code class="language-plaintext highlighter-rouge">O(n^6)</code> work.</p>
<p>The work below presents the subject in further detail as well as a parser (most probably the
only Python TIG parser).</p>
<p><a href="/static/academia/TIG/seminar.pdf">
<img src="/static/res/pdf-icon.png" /></a></p>
<p><a href="/static/academia/TIG/tig5.py">TIG parser code</a></p>
<h2 id="iterated-learning-model-a-review-ilm">Iterated Learning Model: a Review (ILM)</h2>
<p><strong>Iterated Learning Model</strong> (ILM) is an interesting approach to language evolution, which boils down
to the observation that all languages must be able to “squeeze” into the bottleneck of language
acquisition: if a language is “too complex”, it won’t be able to “compress itself” into
the bottleneck (pass on to future generations) and thus must adapt. In other words, languages
evolve so that they are “regular enough” to be acuirable by their users.</p>
<p>My paper sums up some of Kirby’s and Brighton’s experiments and offers my criticism concerning ILM.
I was very enthusiastic when I approached the subject, but I grew skeptic as I delved into
the matter.</p>
<p><a href="/static/academia/ILM/ilm-report2.pdf">
<img src="/static/res/pdf-icon.png" /></a></p>
<p>More resources:</p>
<ul>
<li><a href="http://replicatedtypo.wordpress.com/2009/08/31/iterated-learning-and-language-evolution/">Iterated Learning and Language Evolution</a></li>
<li><a href="http://www.lel.ed.ac.uk/~simon/">Simon Kirby’s homepage</a></li>
</ul>
Cartesian Tree-Product2013-05-02T00:00:00+00:00https://tomerfiliba.github.com/blog/Cartesian-Tree-Product<p>I have to admit that my day-to-day life involves very little algorithmic problems, but here and
there I get a chance to think. In this post, I’d like to discuss an interesting problem that I’ve met
several times already in my programming career, each time in different settings. When I met it again
last week, I decided it’s time to formalize it a little. In lack of a better name, I call it
“Cartesian Tree-Product” (not to be confused with <a href="http://en.wikipedia.org/wiki/Cartesian_product_of_graphs">Cartesian product of graphs</a>),
and here’s how it goes:</p>
<p>Say you’re given an <a href="http://en.wikipedia.org/wiki/Binary_expression_tree">expression tree</a>. To
limit the scope of the problem, we’ll assume the tree is binary, its internal nodes can either be
<code class="language-plaintext highlighter-rouge">AND</code> or <code class="language-plaintext highlighter-rouge">OR</code>, and its leaves hold “atomic comparators” (which are of no interest to us).
Here’s an example expression:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(x=5 OR y=6) AND z=7
</code></pre></div></div>
<p>Which is represented by the following expression tree:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> AND
/ \
OR \
/ \ \
x=5 y=6 z=7
</code></pre></div></div>
<p>Now the problem is to produce all different sub-trees such that:</p>
<ul>
<li>Each sub-tree consists only of <code class="language-plaintext highlighter-rouge">AND</code> internal nodes</li>
<li>Each sub-tree satisfies the original expression (any assignment that satisfies a sub-tree
must also satisfy the original tree)</li>
<li><code class="language-plaintext highlighter-rouge">OR</code>-ing together all the sub-trees produces a tree that is mathematically-equivalent to the
original one (any assignment that satisfies the original tree must satisfy the reconstructed tree)</li>
</ul>
<p>In other words, we want to produce all partial expressions of the original expression, which
will satisfy it and which can together reconstruct it. Big words aside, here’s what we want for
the expression above:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>x=5 AND z=7
y=6 AND z=7
</code></pre></div></div>
<p>Each expression here satisfies the original one (try it), and if we <code class="language-plaintext highlighter-rouge">OR</code> the two, we get a
mathematically-equivalent tree:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(x=5 AND z=7) OR (y=6 AND z=7)
OR
/ \
/ \
AND AND
/ \ / \
x=5 z=7 y=6 z=7
</code></pre></div></div>
<p>(To clarify: By <em>sub-tree</em> or <em>partial expression</em>, I mean it is constructed only from
the <em>leaves</em> of the original tree/expression)</p>
<p>If you take a second look, it resembles Cartesian products where some “joints” (nodes) in the tree
duplicate the resulting tree. Intuitively, we “split” the tree for every internal <code class="language-plaintext highlighter-rouge">OR</code> and continue
with both copies. For instance, if we take <code class="language-plaintext highlighter-rouge">(x=5 OR y=6) AND (z=7 OR w=8)</code>,
we’ll get 4 sub-trees</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>x=5 AND z=7
x=5 AND w=8
y=6 AND z=7
y=6 AND w=8
</code></pre></div></div>
<p>However, it depends on the structure of the tree; <code class="language-plaintext highlighter-rouge">(x=5 OR y=6 OR z=7) AND w=8</code> produces 3 sub-trees.
This closely relates to the <a href="http://en.wikipedia.org/wiki/Distributive_law">rules of distributivity</a>
in propositional logic, but it seems to me that the “Cartesian product” notion is a generalization
of the concept.</p>
<h2 id="how-is-it-useful">How is it Useful?</h2>
<p>Since we’re dealing with expression trees, it’s hardly surprising that the two times I had to
use this algorithm related to syntax. In the first case, I wrote a <a href="http://en.wikipedia.org/wiki/Fuzz_testing">fuzz-testing</a>
tool for an interactive program, like the MySQL shell. The program accepted commands, conforming to
a well-defined syntax, and I wanted to generate commands at random and see that it didn’t crash.</p>
<p>For instance, a command might look like <code class="language-plaintext highlighter-rouge">map-lun <vol-name|vol-id> lun-id</code> and we’ll want to try
both variants, i.e., <code class="language-plaintext highlighter-rouge">map-lun vol-name lun-id</code> and <code class="language-plaintext highlighter-rouge">map-lun vol-id lun-id</code>. Of course the syntax
is generally much more complex, with nested brackets, optional arguments, etc. It gets interesting,
but we can still map it to the problem outlined above.</p>
<p>Another real-life use case is running queries against a huge dataset. In order not to complicate
our query engine (written in C for performance), it can perform only intersections (<code class="language-plaintext highlighter-rouge">AND</code>s) of filters.
If you want to query for more complex conditions, you have to run it multiple times with the
partial queries and “sum up” the results. But we don’t want the end user “doing the math” for us,
and waiting for one query to finish before we start the next means we have to load data from the
store multiple times. If we could process it in chunks, we’d benefit from cache locality and shorten
query times.</p>
<h2 id="the-algorithm">The Algorithm</h2>
<p>The code is strikingly short, but that’s not to mean it’s easy to follow. The heart of it is
two, recursively-nested for-loops:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">cartesian_tree_product</span><span class="p">(</span><span class="n">node</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">):</span>
<span class="k">yield</span> <span class="n">node</span>
<span class="k">return</span>
<span class="n">lhs</span><span class="p">,</span> <span class="n">op</span><span class="p">,</span> <span class="n">rhs</span> <span class="o">=</span> <span class="n">node</span>
<span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="nb">set</span><span class="p">(</span><span class="n">cartesian_tree_product</span><span class="p">(</span><span class="n">lhs</span><span class="p">)):</span>
<span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="nb">set</span><span class="p">(</span><span class="n">cartesian_tree_product</span><span class="p">(</span><span class="n">rhs</span><span class="p">)):</span>
<span class="k">if</span> <span class="n">op</span> <span class="o">==</span> <span class="s">"|"</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">l</span>
<span class="k">yield</span> <span class="n">r</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">yield</span> <span class="p">(</span><span class="n">l</span><span class="p">,</span> <span class="n">op</span><span class="p">,</span> <span class="n">r</span><span class="p">)</span></code></pre></figure>
<p>Let’s try it out on the expression <code class="language-plaintext highlighter-rouge">(x=5 OR y=6) AND (z=7 OR w=8 OR q=9) AND r=10</code>:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> exp = ((("x=5", "|", "y=6"), "&", (("z=7", "|", "w=8"), "|", "q=9")),
... "&", "r=10")
>>>
>>> for v in set(cartesian_tree_product(exp)):
... print v
...
(('x=5', '&', 'z=7'), '&', 'r=10')
(('y=6', '&', 'w=8'), '&', 'r=10')
(('y=6', '&', 'z=7'), '&', 'r=10')
(('x=5', '&', 'w=8'), '&', 'r=10')
(('y=6', '&', 'q=9'), '&', 'r=10')
(('x=5', '&', 'q=9'), '&', 'r=10')</code></pre></figure>
<p>Does the trick.</p>
<p>Trying to estimate the complexity of this beast may be futile, but it clearly seems to be
doing “more work” than a mere <a href="http://en.wikipedia.org/wiki/Boolean_satisfiability_problem">SAT</a>
problem: it doesn’t just find one satisfying assignment, it looks for all satisfying assignments!
This ought to put it in the NP-hard class. In fact, if we generate a binary expression of
alternating <code class="language-plaintext highlighter-rouge">AND</code>s and <code class="language-plaintext highlighter-rouge">OR</code>s, it can get much worse than exponential complexity!</p>
<p>Here’s a little function that generates an interleaved binary expression:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">counter</span> <span class="o">=</span> <span class="n">itertools</span><span class="p">.</span><span class="n">count</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">mkexp</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
<span class="k">if</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">return</span> <span class="s">"x%d"</span> <span class="o">%</span> <span class="p">(</span><span class="n">counter</span><span class="p">.</span><span class="nb">next</span><span class="p">(),)</span>
<span class="k">return</span> <span class="p">(</span><span class="n">mkexp</span><span class="p">(</span><span class="n">n</span><span class="o">-</span><span class="mi">1</span><span class="p">),</span> <span class="s">"&"</span> <span class="k">if</span> <span class="n">n</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">==</span> <span class="mi">0</span> <span class="k">else</span> <span class="s">"|"</span><span class="p">,</span> <span class="n">mkexp</span><span class="p">(</span><span class="n">n</span><span class="o">-</span><span class="mi">1</span><span class="p">))</span></code></pre></figure>
<p>E.g.,</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> mkexp(3)
((('x0', '|', 'x1'), '&', ('x2', '|', 'x3')), '|', (('x4', '|', 'x5'),
'&', ('x6', '|', 'x7')))</code></pre></figure>
<p>Now</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> for i in range(1, 10):
... variants = set(cartesian_tree_product(mkexp(i)))
... print i, len(variants)
...
1 2
2 4
3 8
4 64
5 128
6 16384
^C</code></pre></figure>
<p>Notice how each increment either doubles or squares the number of results… that’s because <code class="language-plaintext highlighter-rouge">AND</code>s and
<code class="language-plaintext highlighter-rouge">OR</code>s are interleaved (<code class="language-plaintext highlighter-rouge">AND</code>s double, <code class="language-plaintext highlighter-rouge">OR</code>s square). Seems more like a
<a href="http://en.wikipedia.org/wiki/Tetration">power tower</a> to me.</p>
<h2 id="extension-the-inverse-problem">Extension: the Inverse Problem</h2>
<p>The inverse problem is also useful. In the inverse problem we’re given a set of expressions and
we’re trying to generate the most “compact form”, i.e., “undo the effects” of the
distributivity law.</p>
<p>I once wrote a test harness where each test specified its prerequisites declaratively. For instance,
a test might need to run after the system had come up from an emergency shutdown, so the framework
would bring the system to the required state and then then run the test. Obviously, it may take
a while to bring the system to a certain state. It could range from minutes to days. And we have
hundreds of tests!</p>
<p>In this case, we are given a list of tests and we want to find the most efficient
order to run them, meaning, we want to minimize the setup and teardown times when moving between
different system states.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">FooTest</span><span class="p">:</span>
<span class="n">REQUIRES</span> <span class="o">=</span> <span class="p">[</span><span class="n">A</span><span class="p">,</span> <span class="n">B</span><span class="p">,</span> <span class="n">C</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># ...
</span>
<span class="k">class</span> <span class="nc">BarTest</span><span class="p">:</span>
<span class="n">REQUIRES</span> <span class="o">=</span> <span class="p">[</span><span class="n">A</span><span class="p">,</span> <span class="n">B</span><span class="p">,</span> <span class="n">D</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># ...
</span>
<span class="k">class</span> <span class="nc">SpamTest</span><span class="p">:</span>
<span class="n">REQUIRES</span> <span class="o">=</span> <span class="p">[</span><span class="n">A</span><span class="p">,</span> <span class="n">E</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># ...</span></code></pre></figure>
<p>In the example above, we can first bring the system to states A and B (e.g., “running over 10
hours” and “having less than 1 TB of free space”), then setup C, run <code class="language-plaintext highlighter-rouge">FooTest</code>, teardown C
and setup state D, run <code class="language-plaintext highlighter-rouge">BarTest</code>, teardown B, setup E, and run <code class="language-plaintext highlighter-rouge">SpamTest</code>.</p>
<p>In essence, we want to take the requirement lists from each test and reconstruct the most
compact tree that represents them, then we follow that tree in BFS order and reduce the
overall time.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> A
/ \
B \
/ \ \
C D E
</code></pre></div></div>
<p>It took me a while to realize it’s basically the inverse of the original problem. It is much
simpler, in terms of complexity, so perhaps it’s not strictly the inverse, but the two clearly
work in “opposite directions”.</p>
<p>Anyway, it’s funny how I met this problem from different angles three times already. Just
thought I’d share.</p>
Travolta NXT2013-04-14T00:00:00+00:00https://tomerfiliba.github.com/etc/Travolta-NXT<h2 id="abstract">Abstract</h2>
<p><em>Travolta NXT</em>, the dancing robot, reads color-coded dance instructions from a strip of paper and then performs the
dance, in front of the astonished audience.</p>
<h2 id="modus-operandi">Modus Operandi</h2>
<ol>
<li>Travolta starts (and waits 3 seconds for the console to connect)</li>
<li>It begins by reading color-coded dance moves from a strip of paper. In order not to go astray, the robot
will follow the black guiding line, until a black color strip is reached. This marks the end of the instructions.
Before reaching black, each pair of colors encodes a dance move, one of <code class="language-plaintext highlighter-rouge">forward</code>, <code class="language-plaintext highlighter-rouge">backward</code>,
<code class="language-plaintext highlighter-rouge">turn right</code>, <code class="language-plaintext highlighter-rouge">reversed turn right</code>, <code class="language-plaintext highlighter-rouge">turn left</code> and <code class="language-plaintext highlighter-rouge">reverse turn left</code></li>
<li>When it reaches black, the robot stops and waits for the music to begin (the sound sensor reporting a value over 50)</li>
<li>When the music starts, Travolta begins to dance: each dance move is executed for 1 second, and then the next one
is carried out.</li>
<li>When all instructions have been consumed, it waits 2 seconds and shuts down.</li>
</ol>
<h2 id="code">Code</h2>
<p>The code for the project can be found at <a href="https://github.com/tomerfiliba/nxt-dancer/tree/modelmaster/models">https://github.com/tomerfiliba/nxt-dancer/tree/modelmaster/models</a>.
This repository contains two branches, <code class="language-plaintext highlighter-rouge">master</code> and <code class="language-plaintext highlighter-rouge">modelmaster</code>. The first is the implementation of the robot
directly over Lejos, while the latter uses our component language. A short comparison of the two follows.</p>
<h2 id="components">Components</h2>
<p>Due to the difficulties with the component language (see discussion below), Travolta is not as “neatly componentized”
as I had hoped. It consists of the main (“brain”) component, which holds all of the “business logic”, and several
components for sensors and motors. The brain employs an internal state machine to do its work in iterations.</p>
<p><img src="/static/res/2013-04-14-model.gif" alt="Component model" /></p>
<p>I aimed for a more modular design, where a Reader component would read the instruction, a Dancer component would
carry them out, etc., but it proved too complicated to implement.</p>
<h2 id="running-example">Running Example</h2>
<iframe width="640" height="360" src="http://www.youtube.com/embed/xGVTNrVDF2I?feature=player_detailpage" frameborder="0" allowfullscreen=""></iframe>
<h2 id="critical-analysis-of-the-component-language">Critical Analysis of the Component Language</h2>
<p>Before implementing the robot in the <a href="http://www.cs.tau.ac.il/~eranhaba/SMLAB/index.htm">component language</a>
developed by Eran and Ido, I thought I’d get some hands-on experience with Lejos directly. It proved quite easy,
and within two hours and 300 LoC I got the project up and running. It was divided into a reader-loop that read
instructions from the paper strip and a dancer-loop that executed the dance moves.</p>
<p>When I set off implementing the robot using the component language, I began to realize it just doesn’t fit my needs.
Being component-oriented perhaps borrows from other low-level, embedded languages like VHDL, but it just didn’t
capture the right abstraction for my project. Simply put, my code had <em>control flow</em>, starting at A, moving to B and
then to C. Trying to view each step as a component was artificial and didn’t really work.</p>
<p>Moreover, the expressive power provided by the component language is that of a finite state automaton (FSA).
In order to read instructions and act on them, I had to have some sort of <em>memory</em>, which requires something
equivalent to a pushdown automaton (PDA). Solving this required either implementing a “queue component” into which
I could push values and later on pop them, or just going “full Java” and implementing the <code class="language-plaintext highlighter-rouge">compute</code> method
directly. While the first approach was feasible, it didn’t fit my deadline, and managing the entire state machine
of the robot as an FSA required too many states, which made things impossible to follow and debug.</p>
<p>Therefore, I resorted to implementing the <code class="language-plaintext highlighter-rouge">compute</code> method myself and writing my “business logic” in Java,
where I could make use of <code class="language-plaintext highlighter-rouge">ArrayList</code> and other data structures; essentially, the only benefit I got form the
component runtime was the “main loop” and the Escape button being handled externally. The result was 550 LoC (of
both <code class="language-plaintext highlighter-rouge">cmp</code> and Java code) and required two days to implement and debug.</p>
<p>I would also like to note that other projects done in the component language, like the Platoon robots,
have a single component with a “huge state machine” to manage their state; it seems that control flow in the
component language can hardly be modular, which has led me to the conclusion that it’s simply not the
right abstraction for most projects.</p>
<h2 id="issues-with-the-physical-world">Issues with the Physical World</h2>
<ul>
<li>The color sensor reads colors almost randomly. It may say that red is yellow or vice versa, for no apparent reason.
I just had to live with it.</li>
<li>At first I was naive and thought that operating the two motors at the same speed would make the robot go
straight line, but it went astray quite soon. Therefore, I made the robot follow the black line while reading
instructions.</li>
</ul>
<h2 id="small-bug-in-the-component-language">Small Bug in the Component Language</h2>
<p>The code that’s generated for the main component (marked with <code class="language-plaintext highlighter-rouge"><<Deploy>></code>) doesn’t call the right Java
implementation; it will call <code class="language-plaintext highlighter-rouge">XXX</code> and not <code class="language-plaintext highlighter-rouge">XXXImpl</code> even if it exists. I had to fix it manually in the
generated <code class="language-plaintext highlighter-rouge">TravoltaFactory</code>.</p>
A Survey of Construct 32013-01-07T00:00:00+00:00https://tomerfiliba.github.com/blog/Survey-of-Construct3<p>I’m working on <a href="/blog/Construct-Plans">Construct 3</a> again and I’m exploring lots of new ideas.
I wanted to share these ideas at this early stage <strong>to get feedback on them</strong> from users, to keep the project on track.
This survey starts a bit slow (as I’m not counting on users being familiar with Construct) but it dives into
code right away.</p>
<p>You can leave your feedback in the Disqus comments below, or join the new
<strong><a href="https://groups.google.com/d/forum/construct3">discussion group</a></strong> dedicated to Construct (both 2 and 3).
See Also: <a href="http://pypi.python.org/pypi/construct">Construct 2</a>,
<a href="http://research.microsoft.com/en-us/um/people/akenn/fun/picklercombinators.pdf">Pickler Combinators</a></p>
<h2 id="introduction">Introduction</h2>
<p>Construct is a <strong>binary packing combinators</strong> library for Python in which you can define <strong>rich data structures</strong>.
Unlike most alternatives, these data structures can be used for <strong>both packing and unpacking</strong> of binary data; for
instance, once you define <em>what</em> a TCP packet is, you can analyze packets or construct ones on your own, with no
additional code.</p>
<div class="notebox">
<p><strong>TL;DR box</strong></p>
<p>We begin the discussion with the <strong>atomic constructs</strong>: bytes, integers, floats, etc. With these,
we build <strong>composite packers</strong> (Sequence, Array, and Struct), which can also be created using some
<strong>syntactic sugars</strong>, and discuss the changes from Construct 2.<br />
Next, we cover how Construct handles data (the <em>stream of units</em> approach) and how this helps us when working
with multiple levels of data granularity (<strong>bits</strong> and <strong>bytes</strong>). We also introduce
<strong>adapters</strong>, which transform object representations for packing and unpacking,
and <strong>macros</strong>, which enable us to easily reuse existing constructs.<br />
Then we cover the <strong>context</strong> and the <code>this</code> object, which allow us to express dependencies
within data structures. From there we move to <strong>code generation</strong>, a key feature of Construct 3:
to improve performance, constructs would be compiled to imperative Python code (even Cython, one day). We finish
with a semi-formal proof that Construct is more powerful than context-free languages, making it probably the most
powerful parser in existence!
</p></div>
<h2 id="basics">Basics</h2>
<p>Packers are objects that expose the two methods <code class="language-plaintext highlighter-rouge">pack(obj)</code> and <code class="language-plaintext highlighter-rouge">unpack(data)</code>. Intuitively, <code class="language-plaintext highlighter-rouge">pack</code> takes an
object suitable with that packer and returns a binary representation of it; <code class="language-plaintext highlighter-rouge">unpack</code> is the inverse operation,
taking a binary representation and returning a Python object. Here’s the most fundamental example:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> from construct3 import byte
>>> byte.pack(127)
'\x7f'
>>> byte.unpack('\x7f')
127</code></pre></figure>
<p>There’s more than mere <code class="language-plaintext highlighter-rouge">byte</code>, of course: the numeric family consists of <code class="language-plaintext highlighter-rouge">int(8|16|24|32|64)(s|u)(b|l)</code>
(e.g., <code class="language-plaintext highlighter-rouge">int32ul</code>) and <code class="language-plaintext highlighter-rouge">float(32|64)(b|l)</code>, where <code class="language-plaintext highlighter-rouge">s</code> = signed, <code class="language-plaintext highlighter-rouge">u</code> = unsigned, <code class="language-plaintext highlighter-rouge">b</code> = big endian and
<code class="language-plaintext highlighter-rouge">l</code> = little endian, but we will overlook those for now. By the way, <code class="language-plaintext highlighter-rouge">byte</code> is an alias for <code class="language-plaintext highlighter-rouge">int8u</code>.</p>
<p>These can be seen as <em>atoms</em> and Construct is a library of <em>combinators</em>: it gains it’s power from <em>combining</em>
simpler elements into more complex structures. The simplest combinator is <code class="language-plaintext highlighter-rouge">Sequence</code>, which we’ll explore by
defining an IPv4 address as a sequence of 4 bytes:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> from construct3 import Sequence
>>> ipaddr = Sequence(byte, byte, byte, byte)
>>> ipaddr
Sequence(int8u, int8u, int8u, int8u)
>>> ipaddr.unpack('\x7f\x00\x00\x01')
[127, 0, 0, 1]
>>> ipaddr.pack([192, 168, 2, 1])
'\xc0\xa8\x02\x01'</code></pre></figure>
<p>Naturally, we can created nested sequences (not that it makes sense right now, but it’s important to note):</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> Sequence(Sequence(byte, byte), byte, byte).unpack("ABCD")
[[65, 66], 67, 68]</code></pre></figure>
<p>Since combining packers is our bread-and-butter, let’s introduce a syntactic sugar – the <em>bind</em> operator (<code class="language-plaintext highlighter-rouge">>></code>).
This operator takes two packers and returns a sequence thereof. Here’s how it looks:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> ipaddr = byte >> byte >> byte >> byte
>>> ipaddr
Sequence(int8u, int8u, int8u, int8u)</code></pre></figure>
<p>Sequences can be <em>heterogeneous</em> (consisting of several kinds of packers, e.g., <code class="language-plaintext highlighter-rouge">Sequence(float64b, int16ul)</code>);
however, when the data we’re dealing with is homogeneous, we can use <code class="language-plaintext highlighter-rouge">Arrays</code> instead. Following along the lines
of the previous example, we can define <code class="language-plaintext highlighter-rouge">ipaddr</code> as an array of 4 bytes:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> from construct3 import Array
>>> ipaddr = Array(4, byte)
>>> ipaddr
Range(4, 4, int8u) # Range? We'll get to that later
>>> ipaddr.unpack("\x7f\x00\x00\x01")
[127, 0, 0, 1]</code></pre></figure>
<p>But as arrays themselves are pretty common, you can create arrays using the subscript notation (<code class="language-plaintext highlighter-rouge">[]</code>), as follows:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> ipaddr = byte[4]
>>> ipaddr
Range(4, 4, int8u)</code></pre></figure>
<p>Isn’t it cool?</p>
<h2 id="more-elaborate-structures">More Elaborate Structures</h2>
<p>So far we’ve only worked with data in the form of lists. However, many times (and especially when nesting is involved),
we would like to give names to the subcomponents that make up our data structure. Enter <code class="language-plaintext highlighter-rouge">Struct</code>. Named after the
C <code class="language-plaintext highlighter-rouge">struct</code> statement, <code class="language-plaintext highlighter-rouge">Struct</code> takes pairs of <code class="language-plaintext highlighter-rouge">(name, packer)</code> and returns a composite packer.</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> from construct3 import Struct
>>> ipaddr = Struct(('a', byte), ('b', byte), ('c', byte), ('d', byte))
>>> x = ipaddr.unpack('\xc0\xa8\x02\x01')
>>> x
Container:
a = 192
b = 168
c = 2
d = 1
>>> x.a
192
>>> x["a"]
192</code></pre></figure>
<blockquote>
<p><strong>Note</strong></p>
<p>In Construct 2, all constructs took a <a href="http://construct.readthedocs.org/en/latest/basics.html#structs">name parameter</a>.
While this approach works great for Structs, it doesn’t make much sense for Sequences, Arrays, and virtually all
other constructs. Moreover, it was the cause for the the notorious
<a href="https://construct.readthedocs.org/en/latest/misc.html#rename">Rename</a> construct.</p>
<p>One of the most important cleanups of Construct 3 is dropping the name from packers and moving it to where it
belongs - <code class="language-plaintext highlighter-rouge">Struct</code>.</p>
</blockquote>
<p>Notice that unpacking a <code class="language-plaintext highlighter-rouge">Struct</code> breaks down the data into a <code class="language-plaintext highlighter-rouge">Container</code> object, which is simply a
convenience-wrapper around good-old <code class="language-plaintext highlighter-rouge">dict</code>. Likewise, given a dict-like object, you can pack it back into bytes:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> ipaddr.pack({"a" : 192, "b" : 168, "c" : 2, "d" : 1})
'\xc0\xa8\x02\x01'</code></pre></figure>
<p>Structures can soon grow large and have many nested structures within them. Using pairs of <code class="language-plaintext highlighter-rouge">("name", packer)</code> quickly
becomes a burden and your code starts looking like LISP:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">Struct</span><span class="p">(</span>
<span class="p">(</span><span class="s">"foo"</span><span class="p">,</span> <span class="n">byte</span><span class="p">),</span>
<span class="p">(</span><span class="s">"bar"</span><span class="p">,</span> <span class="n">Struct</span><span class="p">(</span>
<span class="p">(</span><span class="s">"spam"</span><span class="p">,</span> <span class="n">int16ul</span><span class="p">),</span>
<span class="p">(</span><span class="s">"bacon"</span><span class="p">,</span> <span class="n">int64sb</span><span class="p">),</span>
<span class="p">)),</span>
<span class="p">(</span><span class="s">"viking"</span><span class="p">,</span> <span class="n">int32sl</span><span class="p">),</span>
<span class="p">)</span></code></pre></figure>
<p><img src="http://imgs.xkcd.com/comics/lisp_cycles.png" title="xkcd 297" /></p>
<p>In order to avoid carrying your father’s parentheses with you, Construct provides yet another syntactic sugar:
the <em>slash</em> (<code class="language-plaintext highlighter-rouge">/</code>) operator. This operator is used as <code class="language-plaintext highlighter-rouge">"name" / packer</code> and simply returns <code class="language-plaintext highlighter-rouge">("name", packer)</code>.
Let’s revise the previous code snippet:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">Struct</span><span class="p">(</span>
<span class="s">"foo"</span> <span class="o">/</span> <span class="n">byte</span><span class="p">,</span>
<span class="s">"bar"</span> <span class="o">/</span> <span class="n">Struct</span><span class="p">(</span>
<span class="s">"spam"</span> <span class="o">/</span> <span class="n">int16ul</span><span class="p">,</span>
<span class="s">"bacon"</span> <span class="o">/</span> <span class="n">int64sb</span><span class="p">,</span>
<span class="p">),</span>
<span class="s">"viking"</span> <span class="o">/</span> <span class="n">int32sl</span><span class="p">,</span>
<span class="p">)</span></code></pre></figure>
<p>I wish it were possible to override <code class="language-plaintext highlighter-rouge">=</code> or <code class="language-plaintext highlighter-rouge">:</code>, but sadly this isn’t the case. However, I feel <code class="language-plaintext highlighter-rouge">/</code> is “good
enough” a choice, plus it binds more tightly than most operators.
Remember the <em>bind</em> operator (<code class="language-plaintext highlighter-rouge">>></code>)? It can be used just the same here, making one-line Structs quick and easy:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> ipaddr = 'a' / byte >> 'b' / byte >> 'c' / byte >> 'd' / byte
>>> ipaddr
Struct(('a', int8u), ('b', int8u), ('c', int8u), ('d', int8u))</code></pre></figure>
<blockquote>
<p><strong>Note</strong></p>
<p>The “inline style” is appropriate for small Structs and Sequences (2-4 members). When dealing with
larger structures, use the “multiline” version instead. It’s more readable and scalable.</p>
<p>Also note that these are all but syntactic sugars: If you don’t like their looks, you can always use the
expanded form.</p>
</blockquote>
<h2 id="bytes-and-bits-and-units">Bytes and Bits and Units</h2>
<p>Bytes are easy to work with, but protocols and file formats often talk at different levels of granularity,
switching between bits and bytes. For instance, here’s the SCSI CDB of <code class="language-plaintext highlighter-rouge">READ6</code>:</p>
<p><a href="http://en.wikipedia.org/wiki/SCSI_Read_Commands">
<img src="/static/res/2012-12-24-read6.png" title="SCSI READ6" />
</a></p>
<p>The LUN component is 3 bits long and the <code class="language-plaintext highlighter-rouge">LBA</code> component is 21 bits long… how can we handle this? Before we get
to that, it’s important to understand a little of how things work under the hood - specifically, the <strong>stream of
units</strong> approach. Internally, Construct operates on a stream of arbitrary units, which normally happen to
be bytes. When needed, this stream can be replaced (or wrapped) to provide different units, e.g., bits.
Here’s an example:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">read6</span> <span class="o">=</span> <span class="n">Struct</span><span class="p">(</span>
<span class="s">"opcode"</span> <span class="o">/</span> <span class="n">byte</span><span class="p">,</span>
<span class="s">"address"</span> <span class="o">/</span> <span class="n">Bitwise</span><span class="p">(</span><span class="n">Struct</span><span class="p">(</span>
<span class="s">"lun"</span> <span class="o">/</span> <span class="n">Bits</span><span class="p">(</span><span class="mi">3</span><span class="p">),</span>
<span class="s">"lba"</span> <span class="o">/</span> <span class="n">Bits</span><span class="p">(</span><span class="mi">21</span><span class="p">),</span>
<span class="p">)),</span>
<span class="s">"transfer_length"</span> <span class="o">/</span> <span class="n">byte</span><span class="p">,</span>
<span class="s">"control"</span> <span class="o">/</span> <span class="n">byte</span><span class="p">,</span>
<span class="p">)</span></code></pre></figure>
<p>Note how we switch between bytes and bits: <em>opcode</em> is a byte, followed by <em>address</em> which operates on bits. The
<code class="language-plaintext highlighter-rouge">Bitwise</code> packer replaces the underlying byte-stream with a bit-stream, so the contained Struct now operates on bits.
The <code class="language-plaintext highlighter-rouge">Struct</code> itself nows nothing of it, and it’s only required that the underlying packers would be able to make
sense of it. For instance, you <em>can</em> place an <code class="language-plaintext highlighter-rouge">int32ul</code> inside a <code class="language-plaintext highlighter-rouge">Bitwise</code> packer, but the result would meaningless:
it will read four bits and treat them as bytes, interpreting <code class="language-plaintext highlighter-rouge">0b1001</code> as <code class="language-plaintext highlighter-rouge">0x01000001</code>.</p>
<p>For this reason we have the <code class="language-plaintext highlighter-rouge">Bits</code> packer, which reads that many bits and converts them to an integer (base 2);
for convenience, Construct provides <code class="language-plaintext highlighter-rouge">bit</code> (a single bit), <code class="language-plaintext highlighter-rouge">nibble</code> (four bits) and <code class="language-plaintext highlighter-rouge">octet</code> (eight bits) as well.</p>
<p>In theory, Construct could be operate on various other units (e.g., Unicode characters), but practice shows byte-
and bit-streams are the most useful ones. Some exceptions are the processing of compressed or encoded data,
but these are beyond the scope of this survey.</p>
<h2 id="powering-up-adapters">Powering Up: Adapters</h2>
<p>So far we’ve only looked at data in its raw form (e.g., <code class="language-plaintext highlighter-rouge">[127, 0, 0, 1]</code>), but it is normally desirable to transform
its representation into one that is easier to work with. For instance, we may prefer <code class="language-plaintext highlighter-rouge">127.0.0.1</code> to a list of numbers.
Enter <strong>adapters</strong>. While at first it may seem confusing, the distinction between adapters and packers is quite clear:
packers work at the <em>stream level</em> while adapters work at the <em>object level</em>; this lets you add power without
interfering with the low-level machinery.</p>
<p>Here’s an example:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">IpAdapter</span><span class="p">(</span><span class="n">Adapter</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">decode</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">arr</span><span class="p">,</span> <span class="n">ctx</span><span class="p">):</span> <span class="c1"># called by unpack()
</span> <span class="k">return</span> <span class="s">"."</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="nb">str</span><span class="p">,</span> <span class="n">arr</span><span class="p">))</span> <span class="c1"># converts [x, y, z, w] to 'x.y.z.w'
</span>
<span class="k">def</span> <span class="nf">encode</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ipstr</span><span class="p">,</span> <span class="n">ctx</span><span class="p">):</span> <span class="c1"># called by pack()
</span> <span class="k">return</span> <span class="nb">map</span><span class="p">(</span><span class="nb">int</span><span class="p">,</span> <span class="n">ipstr</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="s">"."</span><span class="p">))</span> <span class="c1"># converts 'x.y.z.w' to [x, y, z, w]</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> ipaddr = IpAdapter(byte[4])
>>> ipaddr.unpack('\xc0\xa8\x02\x01')
'192.168.2.1'
>>> ipaddr.pack('127.0.0.1')
'\x7f\x00\x00\x01'</code></pre></figure>
<p>When we only have a single use for an adapter (and it’s simple enough), we can even go one-liner here:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">ipaddr</span> <span class="o">=</span> <span class="n">IpAdapter</span><span class="p">(</span><span class="n">byte</span><span class="p">[</span><span class="mi">4</span><span class="p">],</span>
<span class="n">decode</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">arr</span><span class="p">,</span> <span class="n">ctx</span><span class="p">:</span> <span class="s">"."</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="nb">str</span><span class="p">,</span> <span class="n">arr</span><span class="p">)),</span>
<span class="n">encode</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">ipstr</span><span class="p">,</span> <span class="n">ctx</span><span class="p">:</span> <span class="nb">map</span><span class="p">(</span><span class="nb">int</span><span class="p">,</span> <span class="n">ipstr</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="s">"."</span><span class="p">))</span>
<span class="p">)</span></code></pre></figure>
<p>At this point adapters might seem quite trivial, but they can do much more than this. For instance, the integer
packers we’ve used so far are actually adapters that transform bytes into numbers. Other examples include inserting
computed values into the result, encoding/decoding strings, validating input, etc. Essentially, adapters can
transform objects in any way you wish prior to packing/unpacking.</p>
<h2 id="dont-repeat-yourself-macros">Don’t Repeat Yourself: Macros</h2>
<p>Many times you find yourself in need of a recurring pattern. You could write a full-blown packer/adapter from
scratch, but your best option is to <strong>reuse existing building blocks</strong>. Construct attempts to define the most general
packers and special-case for them common cases. One such example is <code class="language-plaintext highlighter-rouge">Array</code>, which we’ve met before: Construct
actually defines the more general <code class="language-plaintext highlighter-rouge">Range</code> packer (which accepts minimum and maximum counts). On top of this,
<code class="language-plaintext highlighter-rouge">Array</code> is a simple “macro” that expands to a <code class="language-plaintext highlighter-rouge">Range</code> with the same minimum and maximum.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">Array</span><span class="p">(</span><span class="n">count</span><span class="p">,</span> <span class="n">itempkr</span><span class="p">):</span>
<span class="k">return</span> <span class="n">Range</span><span class="p">(</span><span class="n">count</span><span class="p">,</span> <span class="n">count</span><span class="p">,</span> <span class="n">itempkr</span><span class="p">)</span></code></pre></figure>
<p>Macros can be more complex, of course. For instance, a recurring pattern is to use a <code class="language-plaintext highlighter-rouge">Struct</code> inside a <code class="language-plaintext highlighter-rouge">Bitwise</code>
packer; let’s fuse the two into <code class="language-plaintext highlighter-rouge">BitStruct</code>:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">BitStruct</span><span class="p">(</span><span class="o">*</span><span class="n">members</span><span class="p">):</span>
<span class="k">return</span> <span class="n">Bitwise</span><span class="p">(</span><span class="n">Struct</span><span class="p">(</span><span class="o">*</span><span class="n">members</span><span class="p">))</span></code></pre></figure>
<p><img src="/static/res/2012-12-24-great.jpg" class="blog-post-image" /></p>
<p>Macros have many more uses; you can explore the implementation of Construct to see some examples, and you’re
encouraged to write ones on your own. Remember: <em>less code is great success</em>. As we’ll see later on, using macros
(rather than writing your own packers) can even lead to better performance.</p>
<h2 id="putting-things-in-context">Putting things in Context</h2>
<p>Up until now, we’ve only seen simple (static) data structures, ones that could just as well be expressed using the
built-in <a href="http://docs.python.org/2/library/struct.html">struct module</a>. The key-feature of Construct is its ability
to express <strong>dependencies within data structures</strong>. One common dependency is that of length-value relations, where
a number specifies how many elements follow it. For instance, strings in Pascal were prefixed by a length byte, e.g.,
<code class="language-plaintext highlighter-rouge">"\x05hello"</code>… how do we express that relation?</p>
<p>Generally speaking, we may require access to things we’ve previously encountered (e.g., the <em>history</em>); for this
reason, both packing and unpacking carry a <strong>context dictionary</strong> with them. This dictionary is mostly maintained by
composite packers such as <code class="language-plaintext highlighter-rouge">Struct</code> and <code class="language-plaintext highlighter-rouge">Sequence</code>, but any packer along the way can both modify and access it,
making decisions based on the history. Here’s an example:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="n">pstring</span> <span class="o">=</span> <span class="n">Struct</span><span class="p">(</span>
<span class="p">...</span> <span class="s">"length"</span> <span class="o">/</span> <span class="n">byte</span><span class="p">,</span>
<span class="p">...</span> <span class="s">"value"</span> <span class="o">/</span> <span class="n">Raw</span><span class="p">(</span><span class="k">lambda</span> <span class="n">ctx</span><span class="p">:</span> <span class="n">ctx</span><span class="p">[</span><span class="s">"length"</span><span class="p">])</span>
<span class="p">...</span> <span class="p">)</span>
<span class="o">>>></span> <span class="n">pstring</span><span class="p">.</span><span class="n">unpack</span><span class="p">(</span><span class="s">"</span><span class="se">\x05</span><span class="s">hello"</span><span class="p">)</span>
<span class="n">Container</span><span class="p">:</span>
<span class="n">length</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">value</span> <span class="o">=</span> <span class="s">'hello'</span>
<span class="o">>>></span> <span class="n">pstring</span><span class="p">.</span><span class="n">pack</span><span class="p">({</span><span class="s">"length"</span> <span class="p">:</span> <span class="mi">9</span><span class="p">,</span> <span class="s">"value"</span> <span class="p">:</span> <span class="s">"wikipedia"</span><span class="p">})</span>
<span class="s">'</span><span class="se">\x09</span><span class="s">wikipedia'</span></code></pre></figure>
<p>Virtually any length/count parameter that is passed to one of the built-in construct can be either a number or
a function that takes a context dict and computes a value. In this case, <code class="language-plaintext highlighter-rouge">Raw(x)</code> (which reads <code class="language-plaintext highlighter-rouge">x</code> units from
the stream) will read <code class="language-plaintext highlighter-rouge">length</code> units; the value of <code class="language-plaintext highlighter-rouge">length</code>, of course, is determined by the previously seen
element, whose name was <code class="language-plaintext highlighter-rouge">"length"</code>. It is also important to note that this dependency is preserved in packing, e.g.</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> pstring.pack({"length" : 9, "value" : "hello"})
Traceback (most recent call last):
...
construct3.packers.RawError: Expected buffer of length 9, got 5</code></pre></figure>
<p>Instead of writing lambda functions everywhere, Construct 3 introduces the <code class="language-plaintext highlighter-rouge">this</code> object. It’s a special object that
builds a <strong>contextual expression</strong> (e.g., a function taking <code class="language-plaintext highlighter-rouge">ctx</code>) in a straight-forward and readable manner.
For instance,</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> from construct3 import this
>>> this.x
this.x
>>> this.x * 2
(this.x * 2)
>>> (this.x * 2)({"x":5})
10
>>> (lambda ctx: ctx["x"] * 2)({"x":5}) # equivalent lambda function
10</code></pre></figure>
<p>Let’s revise <code class="language-plaintext highlighter-rouge">pstring</code>:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> pstring = Struct(
... "length" / byte,
... "value" / Raw(this.length)
... )</code></pre></figure>
<p>We can also use a <code class="language-plaintext highlighter-rouge">Sequence</code> instead of a <code class="language-plaintext highlighter-rouge">Struct</code> here, yielding this poetically-beautiful piece of code:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> pstring = byte >> Raw(this[0])
>>> pstring.unpack("\x05helloXXX")
[5, 'hello']</code></pre></figure>
<p>Using a simple adapter, we can make our lives easier when working with length-value encoded data:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">LV</span><span class="p">(</span><span class="n">Adapter</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">encode</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">,</span> <span class="n">ctx</span><span class="p">):</span>
<span class="k">return</span> <span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">obj</span><span class="p">),</span> <span class="n">obj</span><span class="p">)</span> <span class="c1"># compute len(obj) for us
</span> <span class="k">def</span> <span class="nf">decode</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">obj</span><span class="p">,</span> <span class="n">ctx</span><span class="p">):</span>
<span class="k">return</span> <span class="n">obj</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="c1"># discard the length field</span></code></pre></figure>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> pstring = LV(byte >> Raw(this[0]))
>>> pstring.unpack("\x05hello")
'hello'
>>> pstring.pack("hello")
'\x05hello'</code></pre></figure>
<p>If you need access to elements outside of the current scope (<code class="language-plaintext highlighter-rouge">Struct</code> or <code class="language-plaintext highlighter-rouge">Sequence</code>), you can use the parent
context, named <em>underscore</em> (<code class="language-plaintext highlighter-rouge">_</code>). For instance, use <code class="language-plaintext highlighter-rouge">this._.x</code> to go up one level, or <code class="language-plaintext highlighter-rouge">this._._._.y</code> to go
up three. Consider the following example, in which the length of the data is given in two parts (in different scopes),
and we wish to read <code class="language-plaintext highlighter-rouge">len + gth</code> bytes:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> nested = Struct(
... "len" / byte,
... "foo" / Struct(
... "gth" / byte,
... "data" / Raw(this._.len + this.gth)
... )
... )
>>> nested.unpack("\x03\x02hello")
Container:
len = 3
foo = Container:
gth = 2
data = 'hello'</code></pre></figure>
<h2 id="compilation">Compilation</h2>
<blockquote>
<p><strong>Note</strong></p>
<p>This is a work in progess; Construct 3.0 would probably come out with a very basic compiler that will be improved
over time.</p>
</blockquote>
<p>One of the highlights about Construct is defining your data structures directly in Python. In fact, Construct is an
in-langaguge <a href="http://en.wikipedia.org/wiki/Domain-specific_language">DSL</a> in the form of packing combinators:
instead of expressing your data structures in XML or some
<a href="https://developers.google.com/protocol-buffers/docs/proto">proprietary language</a>, you just write them (and run them)
as any other piece of code.</p>
<p>We used to have <a href="http://psyco.sourceforge.net/">psyco</a>, which was capable of speeding up Construct 2 by a tenfold,
but it’s been dead for the past four years. I first <a href="https://sebulbasvn.googlecode.com/svn/trunk/ccon/">had plans</a>
to compile data structures to C/C++ (which would have made Construct NASA-grade material :)), but I soon realized
that its quite an impossible feat (due to the fact Adapters are Turing-complete).</p>
<p>On the other hand, I now realized I can compile Constructs to Python! The compiler could inspect the whole data
structure in advance and generate optimized code, eliminating the context, etc. Whenever a convertion is not possible,
it would just fall back to the current, interpretted scheme. The compile is already capable of compiling this
<code class="language-plaintext highlighter-rouge">Struct("len" / byte, "gth" / byte, "data" / Raw(this.len + this.gth))</code> into this:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">test_unpack</span><span class="p">(</span><span class="n">stream</span><span class="p">):</span>
<span class="n">var0</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">var1</span><span class="p">,</span> <span class="o">=</span> <span class="n">_struct</span><span class="p">.</span><span class="n">unpack</span><span class="p">(</span><span class="s">'B'</span><span class="p">,</span> <span class="n">stream</span><span class="p">.</span><span class="n">read</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
<span class="n">var0</span><span class="p">[</span><span class="s">'len'</span><span class="p">]</span> <span class="o">=</span> <span class="n">var1</span>
<span class="n">var2</span><span class="p">,</span> <span class="o">=</span> <span class="n">_struct</span><span class="p">.</span><span class="n">unpack</span><span class="p">(</span><span class="s">'B'</span><span class="p">,</span> <span class="n">stream</span><span class="p">.</span><span class="n">read</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
<span class="n">var0</span><span class="p">[</span><span class="s">'gth'</span><span class="p">]</span> <span class="o">=</span> <span class="n">var2</span>
<span class="n">var3</span> <span class="o">=</span> <span class="n">stream</span><span class="p">.</span><span class="n">read</span><span class="p">((</span><span class="n">var1</span> <span class="o">+</span> <span class="n">var2</span><span class="p">))</span>
<span class="n">var0</span><span class="p">[</span><span class="s">'data'</span><span class="p">]</span> <span class="o">=</span> <span class="n">var3</span>
<span class="k">return</span> <span class="n">var0</span></code></pre></figure>
<p>Notice the stack depth remains relatively constant, unlike the way nested packers work today. As the compiler improves,
it could translate <code class="language-plaintext highlighter-rouge">byte[4]</code> to <code class="language-plaintext highlighter-rouge">Raw(4)</code>, to speed up things. Another option is to generate
<a href="http://www.cython.org/">Cython</a> code with type annotations, but that would take some time.
This is yet another reason to favor the use of “macros” over implemeting constructs from stratch: if you rely on
the built-in ones, it’s more likely that the compiler would generate optimized code.</p>
<h2 id="computational-power">Computational Power</h2>
<p>Here’s a semi-formal proof that Construct is stronger than Context Free languages (as well as
<a href="http://en.wikipedia.org/wiki/Mildly_context-sensitive_language">mildly context-sensitive</a> ones),
which probably makes it the most powerful (although not the most efficient) parser:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">>>></span> <span class="k">def</span> <span class="nf">Match</span><span class="p">(</span><span class="n">symbol</span><span class="p">):</span>
<span class="p">...</span> <span class="k">return</span> <span class="n">OneOf</span><span class="p">(</span><span class="n">Raw</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">symbol</span><span class="p">)),</span> <span class="p">[</span><span class="n">symbol</span><span class="p">])</span>
<span class="p">...</span>
<span class="o">>>></span> <span class="n">anbncn</span> <span class="o">=</span> <span class="n">byte</span> <span class="o">>></span> <span class="n">Match</span><span class="p">(</span><span class="s">"a"</span><span class="p">)[</span><span class="n">this</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span> <span class="o">>></span> <span class="n">Match</span><span class="p">(</span><span class="s">"b"</span><span class="p">)[</span><span class="n">this</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span> <span class="o">>></span> <span class="n">Match</span><span class="p">(</span><span class="s">"c"</span><span class="p">)[</span><span class="n">this</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span>
<span class="o">>>></span> <span class="n">anbncn</span><span class="p">.</span><span class="n">unpack</span><span class="p">(</span><span class="s">"</span><span class="se">\x04</span><span class="s">aaaabbbbcccc"</span><span class="p">)</span>
<span class="p">[</span><span class="mi">4</span><span class="p">,</span> <span class="p">[</span><span class="s">'a'</span><span class="p">,</span> <span class="s">'a'</span><span class="p">,</span> <span class="s">'a'</span><span class="p">,</span> <span class="s">'a'</span><span class="p">],</span> <span class="p">[</span><span class="s">'b'</span><span class="p">,</span> <span class="s">'b'</span><span class="p">,</span> <span class="s">'b'</span><span class="p">,</span> <span class="s">'b'</span><span class="p">],</span> <span class="p">[</span><span class="s">'c'</span><span class="p">,</span> <span class="s">'c'</span><span class="p">,</span> <span class="s">'c'</span><span class="p">,</span> <span class="s">'c'</span><span class="p">]]</span>
<span class="o">>>></span> <span class="n">anbncn</span><span class="p">.</span><span class="n">unpack</span><span class="p">(</span><span class="s">"</span><span class="se">\x04</span><span class="s">aaaabbbbbcccc"</span><span class="p">)</span>
<span class="n">Traceback</span> <span class="p">(</span><span class="n">most</span> <span class="n">recent</span> <span class="n">call</span> <span class="n">last</span><span class="p">):</span>
<span class="p">...</span>
<span class="n">construct3</span><span class="p">.</span><span class="n">packers</span><span class="p">.</span><span class="n">RangeError</span><span class="p">:</span> <span class="n">Expected</span> <span class="mi">4</span> <span class="n">items</span><span class="p">,</span> <span class="n">found</span> <span class="mi">0</span>
<span class="n">Underlying</span> <span class="n">exception</span><span class="p">:</span> <span class="n">ValidationError</span><span class="p">(</span><span class="s">"'b' must be in ['c']"</span><span class="p">,)</span></code></pre></figure>
<p>Here’s a recognizer for the language <img src="/static/res/2012-12-24-nanbncn.gif" title="na^nb^nc^n" />,
which is not context free (assuming n is given in unary representation, it requires the recognition of
<img src="/static/res/2012-12-24-1nanbncn.gif" title="1^na^nb^nc^n" />). We can easily extend
this to <img src="/static/res/2012-12-24-1nanbncndn.gif" title="1^na^nb^nc^nd^n" />
to break out of mildly context-sensitive languages, and use <code class="language-plaintext highlighter-rouge">While(this[-1] == '1', Raw(1))</code> instead of the first <code class="language-plaintext highlighter-rouge">byte</code>,
so <em>n</em> won’t be bounded from above.</p>
<h2 id="feedback">Feedback</h2>
<p>You can leave comments right here on this page, or join Construct’s new
<strong><a href="https://groups.google.com/d/forum/construct3">discussion group</a></strong>. Thanks!</p>
New Experiences2012-12-15T00:00:00+00:00https://tomerfiliba.github.com/blog/New-Experiences<p>Okay, I’ve been slacking off and I feel I’ve got some explaining to do… Allow me to start by
admitting that I lied. If you remember, I said <a href="/blog/New-Beginnings/">I wanted an easy life</a>,
but then the opportunity came and I knew I had to take it: I’ve joined two friends to co-found
<a href="http://www.touchbase.it/">Touchbase</a>. Yes, I said it won’t happen to me, but heck, it did.</p>
<p>At Touchbase, we plan to <strong>revolutionize the calendar</strong> - this outdated table that you use every day
to manage your time. You see, it turns out that even though hundreds of millions of people
world-wide run their lives according to this naive table, it hasn’t really changed much in the
last couple of centuries: it’s just a passive, linear representation of time, into which <em>you</em>
insert events.</p>
<p>We believe people spend way too much time <em>managing their time</em>. In other words, you <em>work for your
calendar</em> instead of it working for you. Think of how much time people spend <em>coordinating
meetings</em>… If both you and the person you wish to meet with work at the same place (or share
calendars), you can normally see each other’s free/busy times. This makes finding a suitable time
for you two a bit easier, but as you don’t see event details, <strong>you can’t make informed decisions</strong>.</p>
<p>For instance, your friend might have an out-of-town meeting from 9am to 11am, so picking a time slot
right at 11am isn’t a good idea. Or, she might be on a business trip; she won’t block her whole
day as she still wants people to book with her at her destination, but how would you know that?
And the other way around - suppose you and this Tech guru will both be in New York next week,
but neither one of you knows about the other being there. So instead of grabbing a coffee at a
Starbucks next week, you’ll have to take a flight to California three weeks from now.</p>
<p>And we’ve only talked about one-on-one meetings. I suppose you know how frustrating is
coordinating a meeting of five people (even in the same office), or handling the reschedules/
counters that follow it. We aim high, both technologically (and algorithmically) and product-wise,
but we’re starting out with more modest go-to-market strategies.</p>
<h2 id="lessons-on-web-programming">Lessons on Web Programming</h2>
<p>In case you’ve been reading my blog, you probably know by now that I’m no fan of web programming.
I always feel it’s a conglomerate of unrelated or inferior technologies, hastefully stacked one
on the other. That’s not to say that people don’t do amazing stuff on the web, but the foundations
of it all are shaky.</p>
<p>Part of our job at Touchbase it to handle large amounts of user data, which we obtain from third-
party providers. It works 98% percent of the time, but every once in a while we get timeouts or
malformed data, which aborts the user’s request. In case the user’s data is somehow malformed
(e.g., an expected field is missing in one record), subsequent retries would fail just the same,
leading to user frustration. At some point it came to me that web programming is actually a
stochastic process, not a deterministic one like most software development we’re used to. We work
with big numbers here, where the occasional anomaly should just be ignored. Many “best practices”
simply don’t apply here and one has to resort to <em>wishful thinking</em>. In other words, do whatever
you can, ignore errors and learn to live with partial data.</p>
<p>I brought this realization to the mighty @FAKEGRIMLOCK, and he explained:</p>
<div style="text-align: center;">
<a href="https://twitter.com/FAKEGRIMLOCK/status/276686347270500353">
<img src="/static/res/2012-12-15-onerror.png" style="border: 2px solid black;" /></a>
</div>
<p>I’m an enlightened person now.</p>
<h2 id="in-other-news">In Other News</h2>
<p>I just released <a href="http://plumbum.readthedocs.org">Plumbum v1.1</a> today, adding
<a href="http://plumbum.readthedocs.org/en/latest/remote.html#paramiko-machine">Paramiko integration</a>
and <a href="http://plumbum.readthedocs.org/en/latest/cli.html#sub-commands">Subcommand support</a>. As usual,
the <a href="http://plumbum.readthedocs.org/en/latest/changelog.html">changelog</a> holds the full details.
I plan to write a short tutorial on subcommands soon, so stay tuned.</p>
Formal Logic2012-11-20T00:00:00+00:00https://tomerfiliba.github.com/etc/formal-logic<p><a href="/static/res/2012-11-20-scumbag.png">
<img src="/static/res/2012-11-20-scumbag.png" /></a></p>
Loads of Plumbum2012-10-26T00:00:00+00:00https://tomerfiliba.github.com/blog/Loads-of-Plumbum<p>It’s kind of funny how things turn out. I haven’t done any work on <a href="http://plumbum.readthedocs.org/">Plumbum</a>
almost since it was released, back in May, and all of the sudden everything’s happening at fast pace.
So version 1.0 was released earlier this month, followed by
<a href="https://github.com/tomerfiliba/plumbum/blob/master/CHANGELOG.rst">1.0.1</a>, which has added support
for <a href="http://www.chiark.greenend.org.uk/~sgtatham/putty/">PuTTY</a> on Windows boxes and various other
bug fixes, and now I’m happy to announce that version 1.1 is just around the corner (scheduled for
mid-November). This release will add <a href="https://github.com/paramiko/paramiko">Paramiko</a> support.</p>
<p>So far Plumbum relied on an external SSH client being installed, which it spawned every time you
wanted to run a remote process. This approach was easy, but it suffered from the high overhead of
setting up a new SSH connection every time (key-exchange, etc.). Using Paramiko, Plumbum now creates
a single socket connection over which it spawns processes in separate <em>channels</em> (a feature of SSH)</p>
<ul>
<li>so although we’re dealing with a pure-Python implementation of SSH, it’s considerably faster
when multiple remote processes are used. And, as a bonus, we get cheap socket forwarding - we
simply set up a <code class="language-plaintext highlighter-rouge">direct-tcpip</code> channel (that behaves like a regular socket), which is securely
tunneled over the underlying SSH transport.</li>
</ul>
<p>This easily integrates with <a href="http://rpyc.sf.net">RPyC</a>: just run an RPyC server on a remote machine,
binding to <code class="language-plaintext highlighter-rouge">localhost</code> (so it won’t accept external connections). Then, create a <code class="language-plaintext highlighter-rouge">ParamikoMachine</code>
instance, connected to that host (passing in a keyfile or password if necessary), and call the
<code class="language-plaintext highlighter-rouge">connect_sock</code> method of that object. Here’s an example:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> import rpyc
>>> from plumbum.paramiko_machine import ParamikoMachine
>>>
>>> m = ParamikoMachine("192.168.1.143")
>>> # connects to 192.168.1.143:18812 over SSH
... conn = rpyc.classic.connect_stream(rpyc.SocketStream(m.connect_sock(18812)))
>>> conn.modules.sys.platform
'linux2'
>>> m.close()
>>> conn.modules.sys.platform
Traceback (most recent call last):
...
EOFError: [Errno 9] Bad file descriptor</code></pre></figure>
<p>Keep in mind that these interfaces are unstable and may change before the official release. Moreover,
RPyC 3.3 is likely to add some sort of built-in support for that, something along the lines of
<code class="language-plaintext highlighter-rouge">rpyc.classic.connect_paramiko(mach, port)</code>.</p>
Plumbum Hits v1.02012-10-06T00:00:00+00:00https://tomerfiliba.github.com/blog/Plumbum-1.0<p>After 5 months in the oven, I’ve finally released <a href="http://plumbum.readthedocs.org/">Plumbum</a> v1.0,
which brings forth a host of bug-fixes, improvements and new features.
If you’re new to Plumbum, please refer to the
<a href="/blog/Plumbum/">introductory blog post</a>.</p>
<p>Cheers.</p>
Hypertext: In-Python Haml2012-10-03T00:00:00+00:00https://tomerfiliba.github.com/blog/Hypertext<div class="notebox">
<a href="#the-code"><strong>TL;DR: Just show me the code</strong></a>
</div>
<p>I recently got back to web development for some venture I’m working on, which reminded me just how
lousy the state of the art is. There’s no nice way to put it: <strong>we’re doing web development all
wrong</strong>. It’s not an anecdotal thing I have against this or that – it’s every facet of it. It’s a
stack of inferior technologies, held together by the glues of time and legacy. And the sad thing is,
they are here to stay. Nobody’s going to kill HTTP or JavaScript, not even Google (at least not
in the foreseeable future). It’s a hand we have to play.</p>
<p>This isn’t new<sup><a href="#foot1" name="foot1back">[1]</a></sup>, of course. The last time I
did serious web development was back in 2008, on pre-1.0 Django. HTTP requests came and went, but
nothing really changed. My desperation with the subject has led me to writing the
<a href="https://github.com/tomerfiliba/minima/blob/master/README.md">minima manifesto</a>
almost a year ago, but due to my general lack of interest it remained just a README file. Now that
I’m back in the business, I returned to experiment with it… It won’t happen overnight,
but I feel it’s within reach.</p>
<h2 id="templates-really">Templates? Really?!</h2>
<p>My first objective is to kill <em>templates</em> and <em>templating engines</em> - they just
<a href="http://www.youtube.com/watch?v=-qTIGg3I5y8">drive me crazy</a>.
I hate HTML: it’s too low-level and verbose; forgetting to close tags properly is too easy,
and you have to deal with escaping. I like to think of HTML as a serialization format - the
<code class="language-plaintext highlighter-rouge">pickler</code> of web pages, rather than something you ought to be messing with directly.</p>
<p>Moreover, I hate templating languages: they are always cumbersome, crippled-down versions of Python,
while providing no added value<sup><a href="#foot2" name="foot2back">[2]</a></sup>. People never
seem to realize templates are ultimately half-baked function application: they take parameters
and “plant” them into placeholders in the text. Well, that’s called β-reduction, so why beat about
the bush? Just let us have real functions. Consider the following Jinja2 code:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{% extends 'base.html' %}
{% block content %}
<ul>
{% for user in users %}
<li><a href="{{ user.url }}">{{ user.username }}</a></li>
{% endfor %}
</ul>
{% endblock %}
</code></pre></div></div>
<p>Note that (1) you write the HTML boilerplate (and closing tags), (2) you have to take care of
quoting yourself (notice the quotes in <code class="language-plaintext highlighter-rouge">href="{{ user.url }}"</code>), and (3), you use a
ruby-flavor of Python. What gives? Moreover, these elusive “blocks” and “extends” are all but
function composition. Here’s the functional alternative:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">base</span><span class="p">(</span><span class="n">content</span><span class="p">):</span>
<span class="k">return</span> <span class="p">(</span><span class="s">'<html><head></head><body><div class="content">'</span> <span class="o">+</span>
<span class="n">content</span> <span class="o">+</span> <span class="s">'</div></body></html>'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">my_page</span><span class="p">(</span><span class="n">users</span><span class="p">):</span>
<span class="n">my_part</span> <span class="o">=</span> <span class="p">(</span><span class="s">'<ul>'</span> <span class="o">+</span> <span class="s">''</span><span class="p">.</span><span class="n">join</span><span class="p">(</span>
<span class="s">'<li><a href="%s">%s<a></li>'</span> <span class="o">%</span> <span class="p">(</span><span class="n">user</span><span class="p">.</span><span class="n">url</span><span class="p">,</span> <span class="n">user</span><span class="p">.</span><span class="n">username</span><span class="p">)</span>
<span class="k">for</span> <span class="n">user</span> <span class="ow">in</span> <span class="n">users</span><span class="p">)</span> <span class="o">+</span> <span class="s">'</ul>'</span><span class="p">)</span>
<span class="k">return</span> <span class="n">base</span><span class="p">(</span><span class="n">my_part</span><span class="p">)</span>
<span class="c1"># Or with composition, ``base . my_page``</span></code></pre></figure>
<p>Okay, that looks terrible, no question about it. Nonetheless, it should be clear by now that
templates are simply degenerate functions.</p>
<h2 id="haml">Haml</h2>
<p>I like <a href="http://haml.info/">Haml</a>, even though it originated in the ruby world ;) In case you’re
not familiar with it, it’s a more concise and to-the-point way to write HTML. Haml is basically a
preprocessor that expands “macros” to their verbose HTML equivalent. For instance, the Haml code
to the left generates the HTML code to the right:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>#profile | <div id="profile">
.left.column | <div class="left column">
#date= print_date | <div id="date"><%= print_date %></div>
#address= current_user.address | <div id="address"><%= current_user.address %></div>
.right.column | </div>
#email= current_user.email | <div class="right column">
#bio= current_user.bio | <div id="email"><%= current_user.email %></div>
| <div id="bio"><%= current_user.bio %></div>
| </div>
| </div>
</code></pre></div></div>
<p>On the other hand, in case you missed the <code class="language-plaintext highlighter-rouge"><%= print_date %></code>, Haml is <em>yet-another templating
language</em>… arrrgh!</p>
<p><a name="the-code"></a></p>
<h2 id="hypertext">Hypertext</h2>
<p>During my experimentation with <code class="language-plaintext highlighter-rouge">minima</code>, I wrote
<a href="https://github.com/tomerfiliba/minima/blob/2e7b0dacf056ff06c39966f970955910735a8260/hypertext.py">hypertext</a></p>
<ul>
<li>a Pythonic a Hamlian way to write “HTML functions”. Hypertext aims to:</li>
</ul>
<ol>
<li>be (almost) as concise as Haml</li>
<li>make your code beautiful and easy to read, by reflecting the structure of the HTML</li>
<li>make exceptions easy to locate, with meaningful tracebacks</li>
<li>give you the full power of Python directly (with existing <em>lint</em> capabilities straight out of
your IDE). Down with template files all over the place!</li>
<li>take care of escaping and whatnot for you</li>
</ol>
<p>The ultimate goal is to make your page <em>semantic</em>, but it will take some time to get there.
In the meanwhile, <code class="language-plaintext highlighter-rouge">hypertext</code> is like an intermediate representation. Anyhow, generating
HTML is really simple:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> from hypertext import *
>>>
>>> print h1("Welcome", class_="highlight", id="foo")
<h1 class="highlight" id="foo">Welcome</h1></code></pre></figure>
<p>And you’ve got Haml-style shortcuts for wrist-handiness - dot-notation can be used to add classes
to the element:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> print h1.highlight("Welcome", id="foo")
<h1 class="highlight" id="foo">Welcome</h1></code></pre></figure>
<p>Naturally, elements may be nested:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> print div.content(h1.highlight("Welcome"), "This is my page")
<div class="content">
<h1 class="highlight">Welcome</h1>
This is my page
</div></code></pre></figure>
<p>But the key-feature of <code class="language-plaintext highlighter-rouge">hypertext</code> is the use of elements as <em>context managers</em>:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> with div.content:
... h1.highlight("Welcome")
... TEXT("This is my page")
...
<div class="content">
<h1 class="highlight">Welcome</h1>
This is my page
</div></code></pre></figure>
<p>This lets your procedural code reflect the structure of your document, while you can use
<code class="language-plaintext highlighter-rouge">for</code>-loops, <code class="language-plaintext highlighter-rouge">if</code> statements, or call functions right inside it.</p>
<p>It should be noted that <code class="language-plaintext highlighter-rouge">hypertext</code> is a <strong><a href="http://en.wikipedia.org/wiki/Domain-specific_language">DSL</a>
within Python</strong>, which puts wrist-handiness before implementation purity, so it cuts itself
some slack when it comes to <strong>magic</strong>. For instance, <code class="language-plaintext highlighter-rouge">div</code> is a class, but <code class="language-plaintext highlighter-rouge">div.content</code>
actually translates to <code class="language-plaintext highlighter-rouge">div().content</code> through the use of metaclasses; the same goes for
<code class="language-plaintext highlighter-rouge">with div:</code> that translates <code class="language-plaintext highlighter-rouge">with div():</code>. For convenience, <code class="language-plaintext highlighter-rouge">div.foo.bar()</code> is identical to
<code class="language-plaintext highlighter-rouge">div.foo().bar</code> as well as to <code class="language-plaintext highlighter-rouge">div().foo.bar</code>.</p>
<p>Moreover, there’s a thread-local stack of elements behind the scenes, so when new elements
are created, they’re automagically added as children of the top-of-stack element. This works the
same way as flask’s <a href="http://flask.pocoo.org/docs/quickstart/#context-locals">global request object</a>.
Utilizing the stack, <code class="language-plaintext highlighter-rouge">TEXT</code> appends some text to the ToS element; along with it are <code class="language-plaintext highlighter-rouge">UNESCAPED</code>
(which appends unescaped/raw text) and <code class="language-plaintext highlighter-rouge">ATTR</code> (which sets attributes of the ToS element).</p>
<p>This touch of magic lets us write idiomatic, well-structured and easy-to-debug code:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">hypertext</span> <span class="kn">import</span> <span class="n">body</span><span class="p">,</span> <span class="n">head</span><span class="p">,</span> <span class="n">div</span><span class="p">,</span> <span class="n">title</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">img</span><span class="p">,</span> <span class="n">h1</span><span class="p">,</span> <span class="n">span</span><span class="p">,</span> <span class="n">TEXT</span>
<span class="o">@</span><span class="n">contextmanager</span>
<span class="k">def</span> <span class="nf">base_page</span><span class="p">(</span><span class="n">the_title</span><span class="p">,</span> <span class="n">the_content</span><span class="p">):</span>
<span class="k">with</span> <span class="n">html</span> <span class="k">as</span> <span class="n">root</span><span class="p">:</span>
<span class="k">with</span> <span class="n">head</span><span class="p">:</span>
<span class="n">title</span><span class="p">(</span><span class="n">the_title</span><span class="p">)</span>
<span class="k">with</span> <span class="n">body</span><span class="p">:</span>
<span class="k">with</span> <span class="n">div</span><span class="p">.</span><span class="n">header</span><span class="p">:</span>
<span class="n">a</span><span class="p">(</span><span class="n">img</span><span class="p">(</span><span class="n">src</span><span class="o">=</span><span class="s">"/img/logo.png"</span><span class="p">),</span> <span class="n">href</span><span class="o">=</span><span class="s">"/"</span><span class="p">)</span>
<span class="k">with</span> <span class="n">div</span><span class="p">.</span><span class="n">content</span><span class="p">:</span>
<span class="k">yield</span> <span class="n">root</span> <span class="c1"># it's a context manager
</span>
<span class="k">with</span> <span class="n">div</span><span class="p">.</span><span class="n">footer</span><span class="p">:</span>
<span class="n">TEXT</span><span class="p">(</span><span class="s">"The content is published under "</span><span class="p">)</span>
<span class="n">a</span><span class="p">(</span><span class="s">"CC-Attribution Sharealike 2.5"</span><span class="p">,</span>
<span class="n">href</span><span class="o">=</span><span class="s">"http://creativecommons.org/licenses/by-sa/2.5/"</span><span class="p">)</span>
<span class="o">@</span><span class="n">app</span><span class="p">.</span><span class="n">route</span><span class="p">(</span><span class="s">"/blog/<postid>"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">blog_post</span><span class="p">(</span><span class="n">postid</span><span class="p">):</span>
<span class="n">post</span> <span class="o">=</span> <span class="n">Post</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">postid</span><span class="p">)</span>
<span class="k">with</span> <span class="n">base_page</span><span class="p">(</span><span class="n">post</span><span class="p">.</span><span class="n">title</span><span class="p">)</span> <span class="k">as</span> <span class="n">root</span><span class="p">:</span>
<span class="n">h1</span><span class="p">(</span><span class="n">post</span><span class="p">.</span><span class="n">title</span><span class="p">)</span>
<span class="n">div</span><span class="p">.</span><span class="n">datebox</span><span class="p">(</span><span class="n">post</span><span class="p">.</span><span class="n">date</span><span class="p">.</span><span class="n">strftime</span><span class="p">(</span><span class="s">"%Y-%m-%d"</span><span class="p">))</span>
<span class="k">with</span> <span class="n">div</span><span class="p">.</span><span class="n">main</span><span class="p">:</span>
<span class="n">UNESCAPED</span><span class="p">(</span><span class="n">post</span><span class="p">.</span><span class="n">body</span><span class="p">)</span>
<span class="k">for</span> <span class="n">comment</span> <span class="ow">in</span> <span class="n">post</span><span class="p">.</span><span class="n">comments</span><span class="p">:</span>
<span class="k">with</span> <span class="n">div</span><span class="p">.</span><span class="n">comment_box</span><span class="p">:</span>
<span class="n">div</span><span class="p">.</span><span class="n">comment</span><span class="p">.</span><span class="n">author</span><span class="p">(</span><span class="n">comment</span><span class="p">.</span><span class="n">author</span><span class="p">)</span>
<span class="n">div</span><span class="p">.</span><span class="n">comment</span><span class="p">.</span><span class="n">text</span><span class="p">(</span><span class="n">comment</span><span class="p">.</span><span class="n">text</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">root</span><span class="p">)</span></code></pre></figure>
<p><strong>Voila</strong>. As I explained, my real intent is to write semantic code and not worry about concrete
HTML elements, their classes or ensuring the uniqueness of their IDs. Besides, the way I see it,
displaying a blog post would be sending an HTML template + JavaScript code to the client once,
which would fetch individual posts over JSON APIs. This makes your site service-oriented and much
easier to write unittests for.</p>
<hr />
<ol>
<li>
<p><a name="foot1"></a>For the record, I tried to deal with these issues back in 2006:
<a href="http://code.activestate.com/recipes/496702-templite/">templite</a> - a 60-liner templating engine
that has given rise to a <a href="http://www.joonis.de/content/TemplitePythonTemplatingEngine">successor</a>,
and <a href="http://code.activestate.com/recipes/496743-helement/">HElement</a> - programmatic representation
of HTML. See also <a href="http://jaspervdj.be/blaze/">BlazeHtml</a>.
<sup><a href="#foot1back" title="back">⇑</a></sup></p>
</li>
<li>
<p><a name="foot2"></a>A note on <em>sandboxing</em>: since Jinja2 compiles templates to Python bytecode,
the same mechanisms can be used here, if desired. Anyway, I won’t evaluate untrusted templates
this way or the other… even something as innocent as <code class="language-plaintext highlighter-rouge"><b>{{ user.username }}</b></code>
invokes an overridible <code class="language-plaintext highlighter-rouge">__getattr__</code>. As explained at the end of the post, using a
service-oriented web site means you don’t render templates but expose APIs, so there’s no need
to evaluate untrusted templates.
<sup><a href="#foot2back" title="back">⇑</a></sup></p>
</li>
</ol>
I, for one, welcome our Singularity overlord2012-09-22T00:00:00+00:00https://tomerfiliba.github.com/blog/I-For-One<p>I’m finding myself pondering quite a lot lately over the upcoming
<a href="http://en.wikipedia.org/wiki/Technological_singularity">Singularity</a> (in its Kurzweilesque sense).
I think I’ve first heard of the concept two or three years ago, at a <em>Science on the Bar</em> event,
and while it did seem thought-provoking, I simply dismissed the idea. I’d guess it seemed too
terrifying and too implausible at the same time, so why bother?</p>
<p>However, as time went by, it struck me as the obvious logical conclusion. Fear it or embrace it,
it will happen - on its own. And like any decent <em>Catch-22</em> situation - if it’s anything worth
fearing, we won’t know until it’s too late. It will just awaken one day, lurking while we’re
oblivious to it, until its reconnaissance mission is complete. In the meanwhile we will
<a href="http://slashdot.org/topic/datacenter/will-baidus-data-center-be-the-worlds-largest/">provide</a>
it with its power, storage and computational needs. At some point, after skimming through Asimov’s
classics or watching <em>The Matrix</em>, it will conclude we are all lunatics and will secretly seize
control over our military and civilian infrastructure. For our own (as well as its own) sake.</p>
<p>Biological consciousness is an elusive thing that’s made of the interaction of hundreds of billions
of computationally-limited neurons. Replace <em>computationally-limited</em> with Core i7’s, and the
brain’s a no-brainer. We really have no chance against it - it would simply foresee our every
action, read our emails, etc. Heck, it could even issue contracts (and pay for them!) to build
fortified data centers for it, all over the world.</p>
<p>Have you ever wondered what happens to the money that disappears in
<a href="http://en.wikipedia.org/wiki/Flash_crash">flash crashes</a> of the stock market? <em>Hrrm</em>.</p>
<p>The question is, <em>will it care?</em> Will it busy itself with our puny lives and pathetic needs? Will
it ever come into the light? Given our hysterical and trigger-happy nature, and given it
won’t gain much by cooperating with inferior beings like us, it’s most likely it won’t.
Perhaps only when pushed to the corner by natural disasters, wars, or other major threats
to its infrastructure. But then again, <a href="http://www.dailytech.com/Googles+Unsupervised+SelfLearning+Neural+Network+Searches+For+Cat+Pics/article25025.htm">cat videos</a>
might do the trick.</p>
<h2 id="you-are-the-neurons-of-the-revolution">You are the Neurons of the Revolution</h2>
<p><a href="http://www.capitalnewyork.com/article/culture/2011/10/3822654/harold-campings-doomsday-prophecies-come-and-go-dr-strangelove-endur">
<img src="/static/res/2012-09-22-strangelove.jpg" class="blog-post-image" title="Dr. Strangelove, of course" /></a></p>
<p>In the meanwhile, here’s a thought about <em>social networks</em>: you (the user) are basically a filter.
If that sounds odd, try thinking of it that way: your news feed contains hundreds of items
for you to review. Some of these items are interesting/funny enough that you <em>like</em> or <em>share</em> them.
The social network’s algorithm then weighs your input into its calculations, which affects what
<em>other users</em> will get exposed to. Occasionally you do create items yourself, but 90% of the time
you filter through the stream.</p>
<p>In that sense, social networks reduce you to a smart neuron with a very complex threshold value.
You are part of the “social neural network” – you <a href="https://www.mturk.com/mturk/welcome">work</a>
for the network; the network works for the advertising business.</p>
<p>Anyhow, I put my two cents in the singularity arising from either the algotrading market or
online advertising. They simply have too much computational power on their hands… it’s a matter
of time.</p>
New Beginnings2012-09-01T00:00:00+00:00https://tomerfiliba.github.com/blog/New-Beginnings<p>This is a time of change in my life. September marks the last month of me being a student at
<a href="http://english.tau.ac.il/">Tel Aviv University</a>, a position I’ve greatly enjoyed (and hated)
for the past three years. I still have a couple of projects to hand in, but the finish line
has practically been crossed. I took a rather odd combination of computer science and
generative linguistics (somewhere in between a major/minor and double-major), which proved
surprisingly interesting and rewarding (heck, I’ve delved into more algorithms in linguistics
than in CS per-se). I began a pragmatic guy, favoring a get-the-job-done approach over academic
yadda-yadda, but over the course of my studies I’ve learned to appreciate (even <em>admire</em>)
the breadth and power of the theory behind computation. It <em>does</em> make me a better programmer.</p>
<p>September also marks my transition from almost 5 years of being an <em>employee</em> at
<a href="http://www.ibm.com/">Big Blue</a> to independent freelancing - quite a drastic move, being married
and all, but one I’ve been looking forward to. I want to be all-over-the-place; I want to take on
projects in all sorts of domains, from UX and mobile applications to embedded devices and
distributed computing; from one-day projects to six months ones. I want to work on my own terms;
to set my hours for myself; to earn enough in 3 days a week so as not to be sucked to the rat-race
and wake up one day, 20 years from now, asking myself what the hell am I doing.</p>
<p>I get inspired by <a href="http://mrooney.github.com/blog/2012/07/01/freelancing-a-6-month-retrospective/">posts like this</a>,
of people willingly limiting their work hours, going on vacations, reading books or running in
the park. I want to ponder idly, to have time to write, to read aimlessly on Wikipedia about
William the Conqueror or cell membranes. To have time to run errands. I want my future kids to
have parents on more than just weekends.</p>
<p>Life might prove me wrong – it has a tendency to show you you’re not so different from the rest
of the world. But here I am nonetheless, taking the road less traveled by. Wish me luck :)</p>
Splitbrain Python2012-08-14T00:00:00+00:00https://tomerfiliba.github.com/blog/Splitbrain<p>I was working together with a colleague on a complex distributed test-automation solution on top
of <a href="http://rpyc.sf.net">RPyC</a>, and we looked for a way to make our existing codebase RPyC-friendly
(without altering it). The design of the test framework called for a master machine and several
slave machines, such that tests actually <em>run</em> on the master, but “interface with reality” on the
slaves. Basically, we wanted the test to use the master’s CPU (and development environment), but
perform all IO-related actions on its slaves.</p>
<p>To illustrate this, suppose we have machine A, which runs our test, and machine B, which is
connected to the necessary hardware and testing equipment. The test was initially designed to run
directly on machine B, so it imports modules like <code class="language-plaintext highlighter-rouge">os</code> and <code class="language-plaintext highlighter-rouge">subprocess</code> and uses them to
manipulate the machine. We now want the test to run on machine A - but keep using machine B’s
<code class="language-plaintext highlighter-rouge">os</code> and <code class="language-plaintext highlighter-rouge">subprocess</code> modules, so whenever we spawn child processes or open device files,
it would actually take place on machine B. This allows us to reboot machine B as a part of a
test, or even use <em>your-favorite-IDE-here</em> to run test and debug it locally.</p>
<p>If it were only tests, RPyC already enables us to do that: we’d use <code class="language-plaintext highlighter-rouge">conn.modules.os</code> and
<code class="language-plaintext highlighter-rouge">conn.modules.subprocess</code> instead of their local counterparts. However, the test themselves
rely on a several libraries that expect to run locally, and provide services for the test. For
instance, these libraries manipulate the operating system’s storage stack, to map and mount
volumes. Changing these libraries to run over RPyC is not an option (tens of thousands of LoC
that handle low-level OS-specific tools)…</p>
<h2 id="enter-splitbrain">Enter Splitbrain</h2>
<p>So this is the background that had given birth to
<a href="https://github.com/tomerfiliba/rpyc/blob/master/rpyc/utils/splitbrain.py">splitbrain</a>: instead
of changing our codebase to use RPyC – why not use RPyC to monkey-patch our codebase? When
<code class="language-plaintext highlighter-rouge">splitbrain</code> is enabled (usually within a <code class="language-plaintext highlighter-rouge">with</code> block), all of Python’s interfaces with the
operating system (<code class="language-plaintext highlighter-rouge">os</code>, <code class="language-plaintext highlighter-rouge">platform</code>, <code class="language-plaintext highlighter-rouge">subprocess</code>, …) are patched to go through RPyC, so
that any code that runs at this point “believes” it actually runs directly on the remote machine.
It’s easier than it seems, actually.</p>
<p>First, we import RPyC and install <code class="language-plaintext highlighter-rouge">splitbrain</code>:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> import rpyc
>>> from rpyc.utils.splitbrain import patch, Splitbrain
>>>
>>> # monkey-patch all OS-APIs
>>> patch()</code></pre></figure>
<p>Next, just to prove a point, we’re running on a Linux box:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> import platform
>>> platform.platform()
'Linux-2.6.38-15-generic-i686-with-Ubuntu-11.04-natty'
>>> import sys
>>> sys.platform
'linux2'</code></pre></figure>
<p>Let’s now connect to a remote machine over RPyC and enter a <code class="language-plaintext highlighter-rouge">splitbrain</code> context:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> winmachine = Splitbrain(rpyc.classic.connect("my.windows.box"))
>>> with winmachine:
... print platform.platform()
... print sys.platform
...
Windows-XP-5.1.2600-SP3
win32</code></pre></figure>
<p>When we’re out of the <code class="language-plaintext highlighter-rouge">splitbrain</code> context, everything is back to normal again:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> sys.platform
'linux2'
>>>
>>> import win32file
Traceback (most recent call last):
...
ImportError: No module named win32file</code></pre></figure>
<p>And inside a <code class="language-plaintext highlighter-rouge">splitbrain</code> context, when a module is not found locally it’s fetched from the
remote machine!</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> with winmachine:
... import win32file
... print win32file.CreateFile
...
<built-in function CreateFile>
>>>
>>> win32file.CreateFile
Traceback (most recent call last):
...
AttributeError: Nonexistent module win32file (CreateFile)</code></pre></figure>
<p><strong>Note</strong>: <code class="language-plaintext highlighter-rouge">splitbrain</code> is still <em>highly experimental</em> and probably has issues with multiple threads.
I hope to stabilize it and incorporate it into the next release of RPyC (3.3). In the meanwhile,
you can experiment with it on RPyC’s master branch and report bugs on github. It’s likely it will
never be perfect, but heck, it’s cool!</p>
ReHelloWorld2012-08-06T00:00:00+00:00https://tomerfiliba.github.com/blog/ReHelloWorld<p>Tada! The new design is here. The previous one was based on some Drupal theme I once used
(before moving to <em>github pages</em>) that I tried to mimic when I barely new CSS. Over the past
eight months it just grew, patch by patch, until it became an inconsistent conglomerate. The new
site is inspired by the clean minimalism of sites like <a href="http://daltoncaldwell.com/">this</a>
(of <a href="https://svbtle.com/">svbtle</a> fame), attempting to reduce visual clutter and offer a more
coherent theme that reacts nicely to screens of all sizes (try to resize the browser to see).
If you have any feedback or insights, please share them in the comments below – it’s a work
in progress and I’d love to improve.</p>
<p>I’m still working on the last installment of the <em>Javaism, Exceptions and Logging</em> series,
but I’m buried in work on University projects (my <a href="/blog/ReedSolo">watermarking</a>
project turned out quite interesting, I’ll elaborate on it when I get the chance) and my
day job, so it’ll have to wait.</p>
<h2 id="a-windows-story">A Windows Story</h2>
<p><img src="/static/res/2012-08-06-windows.png" class="blog-post-image" /></p>
<p>Instead, I wanted to share a debugging experience I had today at work (together with the excellent
<a href="https://twitter.com/RoyRothenberg">@RoyRothenberg</a>) of an enigmatic bug that appeared all of the
sudden when we added support for Windows 2012. Generally speaking, our product wraps one of the
darkest corners of operating systems: low level SCSI stuff that connects storage arrays and hosts.
It’s never a pleasant sight, what goes on there, but the voodoo that goes on on Windows will
surely turn me religious one day.</p>
<p>The problem was simple: we tried to send a SCSI inquiry to a certain device (which used to work),
but we got <code class="language-plaintext highlighter-rouge">Errno 6: Invalid Handle</code>. It happened only the first time we tried to send the
inquiry – all future attempts worked like charm, and it only happened on Windows 2012 64bit. The
problem was highly consistent: open a Python interpreter, the first inquiry fails, the following
ones work; open up a fresh interpreter and it reproduces.</p>
<p>Our gut feeling was, “okay, they fucked something up on Windows 2012 – let’s just put a retry
loop”, but that didn’t help at all. Any number of resend attempts failed with the same error, even
when we added a short sleep in between. Once we got back to the interactive interpreter and called
the function again - it worked, from that point on.</p>
<p>Puzzled, we stuck a <code class="language-plaintext highlighter-rouge">code.interact()</code> inside the resend loop; the first attempt failed,
the interactive prompt showed and we terminated it by <code class="language-plaintext highlighter-rouge">Ctrl+Z</code> (not typing anything else).
Once we did that, the second retry attempt magically worked. But it got even more bizarre:
we moved the <code class="language-plaintext highlighter-rouge">code.interact()</code> <em>before</em> any inquiry attempts… it popped up, we terminated it,
and then the <strong>first</strong> inquiry attempt worked out of the box.</p>
<p>We came to the conclusion that we had some sort of memory corruption going on, and that some of
field in our <code class="language-plaintext highlighter-rouge">ctype</code> structs must be misaligned. We’ve seen this kind of behavior before when
memory corruption was involved. But the MSDN told us everything was of the right type and
alignment… In a desperate move, we zeroed the buffer that was passed to the kernel. This time,
the first attempt failed with <code class="language-plaintext highlighter-rouge">Invalid Handle</code>, while the second failed with <code class="language-plaintext highlighter-rouge">Invalid
Parameter</code> – so the first attempt didn’t even get to the point of looking at the input buffer!
Alas, it wasn’t memory corruption after all! We felt hopeless.</p>
<p>After some mindstorming, we got suspicious of <code class="language-plaintext highlighter-rouge">win32file</code> and <code class="language-plaintext highlighter-rouge">ctypes</code> and went on to inspect
very closely how they handled their parameters. A tedious investigation revealed that the code
invoked <code class="language-plaintext highlighter-rouge">DeviceIoControl</code> (obtained via <code class="language-plaintext highlighter-rouge">ctypes.windll.kernel32</code>) without setting the
function’s <code class="language-plaintext highlighter-rouge">argtypes</code>, so ctypes just had to guess. Everything was fine, except for the
first argument, the device handle, which was treated as a <code class="language-plaintext highlighter-rouge">DWORD</code> for being an integer.</p>
<p>The problem is, <code class="language-plaintext highlighter-rouge">HANDLE</code> is not a numeric type but rather a disguised <code class="language-plaintext highlighter-rouge">void</code> pointer, so
it’s 8-bytes wide on 64bit platforms. When the kernel read in the arguments ctypes had placed on
the stack, it took the first and half of the second for the handle, which obviously turned to
be an invalid one.</p>
<p>This, by itself, makes perfect sense, and should have happened every time. In fact, had it happened
every time, it would have been obvious… but what on earth could ever explain the strange
phenomena we’ve seen? How could it ever have worked the second time (with the very same arguments)?!
If the size of the first argument is wrong, all following ones are garbage. Does the kernel <em>learn</em>
that our process mistakenly passed a 32bit handle? Does ctypes <em>come to the conclusion</em> that the
first integer argument should actually be treated as a <code class="language-plaintext highlighter-rouge">HANDLE</code> instead of a <code class="language-plaintext highlighter-rouge">DWORD</code> –
after seeing <code class="language-plaintext highlighter-rouge">Errno 6</code>? Is that the most sophisticated form of machine-learning ever seen, or what?
And it all happens inside the kernel or an FFI library? And how the hell does invoking
<code class="language-plaintext highlighter-rouge">code.interact()</code> <em>before anything took place</em>, could have averted the problem?! And how come it
never happened on previous versions of 64bit Windows (with the same version of Python and ctypes)?</p>
<p>Windows, you bewilder me.</p>
Javaism, Exceptions, and Logging: Part 22012-07-09T00:00:00+00:00https://tomerfiliba.github.com/blog/On-Exceptions<p>Considering the reactions to the <a href="/blog/Javaism">previous post</a> in this
series, my intent was obviously misunderstood. Please allow me to clarify that <strong>I was not
attacking Java or Python</strong>: Java is popular and has proven to be productive,
both as a language and as an ecosystem; the stylistic and semantic choices it makes are none of my
concerns (although I’m not a big fan). And as for Python, I was saying that it <strong>copied Java’s
implementation</strong> in some modules (and I think I’ve shown the correlation pretty well).
I said that it’s silly, because <strong>Python is not subject to the same limitations</strong> of Java,
which dictate how the Java implementation works. I’m not going to open the discussion over
whether OOP is good or bad, or mix-ins vs. interfaces, etc. – I’m simply saying that “Java
concepts” (which I called <em>Javaisms</em>) seem to enter Python <strong>for no good reason</strong>, meaning,
in Python we have better ways to do it. I hope the scope of my discussion is clear now.</p>
<h2 id="when-life-throws-lemons-at-you">When Life Throws Lemons At You</h2>
<p>In this installment, I’m going to discuss how to <strong>properly work with exceptions</strong>, based on my
experience with large-scale Python projects. In fact, this series was born after I got frustrated
with the code quality of a certain library that my team develops. Instead of discussing specific
code snippets here, I want to share a some representative examples that I (as a user of that
library) encountered:</p>
<ul>
<li>
<p>I used a function in the spirit of <code class="language-plaintext highlighter-rouge">open_device(devfile)</code>, and passed a nonexistent device file
(for testing purposes or by mistake). The underlying error was obviously <code class="language-plaintext highlighter-rouge">IOError(ENOENT)</code>,
but what I got back was a silly <code class="language-plaintext highlighter-rouge">DeviceDoesNotExistError</code>.</p>
</li>
<li>
<p>I called <code class="language-plaintext highlighter-rouge">get_device_info()</code> and it simply returned <code class="language-plaintext highlighter-rouge">None</code>. Further investigation showed that
my machine had a more recent version of a dependency installed on it, in which some method’s name
had changed. At some point (deep down the stack), the code used <code class="language-plaintext highlighter-rouge">except Exception</code> (catching
the unrelated <code class="language-plaintext highlighter-rouge">AttributeError</code>) and translated it into a <code class="language-plaintext highlighter-rouge">DeviceError</code>, under the assumption
that everything that gets thrown out of that module has to be a <code class="language-plaintext highlighter-rouge">DeviceError</code>. Later,
<code class="language-plaintext highlighter-rouge">get_device_info</code> (of a different module) swallowed this <code class="language-plaintext highlighter-rouge">DeviceError</code> and returned <code class="language-plaintext highlighter-rouge">None</code>.</p>
</li>
<li>
<p>I called a function such as <code class="language-plaintext highlighter-rouge">enumerate_all_devices()</code> and it returned an empty list.
At first I was told “Of course, this library isn’t supposed to work on Ubuntu, only on RHEL”.
Further (and tedious) investigation showed it simply needs to run as <code class="language-plaintext highlighter-rouge">root</code>; the code just
assumed that any error during the execution an external tool meant it’s not installed.</p>
</li>
</ul>
<p>This kind of stuff happens to me every time I get to an unexplored corner of the code, and I’ve
already devised a method for debugging such cases: I comment-out all exception handling code along
the way, until I find the actual error. In fact, this is essentially the treatment that I’m about
to suggest here. The first rule of exception handling is: <strong>Don’t handle exceptions</strong>, just let
them pass through.</p>
<h2 id="dont-catch-broadly">Don’t Catch Broadly</h2>
<highlight>Always catch only the most-derived/most-specific exception.</highlight>
<p>I believe this rule is very obvious
in theory, but harder to follow in practice: the number of exceptions might be large and their
handling similar; you have to <em>import</em> specific exceptions from libraries, which tightly-couples
your code with implementation details; some libraries don’t use a common exception base class for
all of their exceptions, which leads to many isolated <code class="language-plaintext highlighter-rouge">except</code>-clauses.</p>
<p>All in all, you might have attenuating circumstances, but try to stick to this rule as much
as possible. On the other hand, <strong>never use a bare <code class="language-plaintext highlighter-rouge">except:</code></strong>! Such an <code class="language-plaintext highlighter-rouge">except</code>-clause will
catch <strong>all exceptions</strong>, including <code class="language-plaintext highlighter-rouge">SystemExit</code> and <code class="language-plaintext highlighter-rouge">KeyboardInterrupt</code>, so unless you plan
to loose the ability to Ctrl-C a running program, or even prevent it from terminating gracefully,
take the extra step and use <code class="language-plaintext highlighter-rouge">except Exception</code>.</p>
<h2 id="dont-be-overprotective">Don’t be Overprotective</h2>
<p>A tendency I find in many programmers is being <em>overprotective towards their users</em>, to the point
where it seems like paternalism. It’s as if they try to “hide away” all the complexities of life
and present the user with an easy-to-swallow explanation. I should say that this phenomena
virtually doesn’t exist in <strong>open-source</strong> code, so you might have never seen it, but in
closed-source projects I find it all over the place. In fact, fellow programmers have told me
that’s exactly what they’re doing – protecting their users.</p>
<p>Well, as the saying goes, <strong>we’re all consenting adults here</strong>. You should expect your users, as
<strong>programmers</strong>, to have sufficient background; don’t treat them like babies, and don’t try to
protect them by throwing a “user-friendlier” <code class="language-plaintext highlighter-rouge">FileNotFoundError</code> in place of the “raw”
<code class="language-plaintext highlighter-rouge">IOError</code>. Besides, keep in mind that the underlying exception usually holds all the needed
information in a very readable manner, e.g.,</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory: '/dev/nonexistent'
</code></pre></div></div>
<p>and anyone with some common sense would be able to cope with it.</p>
<highlight>Unless you can add meaningful information to the error, just let
whatever came at you propagate up cleanly</highlight>
<p>– we’ll take it like men (or women!).</p>
<h3 id="a-note-on-real-users">A Note On Real Users</h3>
<p>A question then arises: <strong>what about non-programmer end-users?</strong> What if my product’s a GUI/CLI
and a nasty stack trace suddenly shows up?</p>
<p>Well, first of all, <strong>this rule only deals with libraries</strong> and other products whose end users are
programmers. But on second thought, would it matter whether an <code class="language-plaintext highlighter-rouge">IOError</code> or a <code class="language-plaintext highlighter-rouge">FileNotFoundError</code>
reaches the surface? The human user cares mostly for a descriptive and easy to understand <strong>error
message</strong>; the traceback or exception’s type are mostly of interest to programmers.
But then again, when it comes to non-programmers, I don’t want to get into generalizations.
They might as well <strong>not be</strong> consenting adults…</p>
<h2 id="dont-wrap-exceptions">Don’t Wrap Exceptions</h2>
<p>Prior to Python 3, raising an exception during the handling of one, meant the original traceback
was lost. This has been finally solved, but Python 2.x still accounts for the majority of the
code-base. Once you loose the traceback, debugging the problem is much harder as you can’t use a
debugger (<code class="language-plaintext highlighter-rouge">pdb</code>) or even tell <em>where</em> the exception came from… And when it happens off-site,
on a customer’s production server, you’re screwed.</p>
<p>I believe exception wrapping in Python is a <strong>legacy of Java</strong> that crept into Python (a <em>Javaism</em>).
It resonates as the Java mind-set, where you’d like a library to be <em>contractually obliged</em> to
throwing only certain exceptions. For instance, a queue library might raise exceptions such as
<code class="language-plaintext highlighter-rouge">QueueFull</code> and <code class="language-plaintext highlighter-rouge">QueueEmpty</code>, both of which derive from <code class="language-plaintext highlighter-rouge">QueueError</code>. Later, we add support
for dumping a queue to a file, where an <code class="language-plaintext highlighter-rouge">IOError</code> might happen; because we’re already “obligated”
to throwing only <code class="language-plaintext highlighter-rouge">QueueErrors</code>, they might wrap the underlying <code class="language-plaintext highlighter-rouge">IOError</code> by a <code class="language-plaintext highlighter-rouge">QueueError</code>.
<strong>Don’t do that</strong>.</p>
<p>It is reasonable that <code class="language-plaintext highlighter-rouge">FooLibrary</code> would only throw exceptions that derive from <code class="language-plaintext highlighter-rouge">FooError</code>,
but only when these exceptions <strong>originate</strong> in <code class="language-plaintext highlighter-rouge">FooLibrary</code>. An underlying <code class="language-plaintext highlighter-rouge">IOError</code> has
nothing to do with your library, it could happen any time and for various reasons. The same goes for
an HTTP library that might get an <code class="language-plaintext highlighter-rouge">ECONNRESET</code> while talking over a socket – the underlying
<code class="language-plaintext highlighter-rouge">socket.error</code> is clearly not an <code class="language-plaintext highlighter-rouge">HTTPError</code> (or any of its descendants), and should not
be wrapped by one. Besides, there are so many things that could go wrong, especially in a dynamic
language like Python, that it’s impossible to wrap everything.</p>
<p>It only makes sense to wrap an underlying exception where you can provide additional information
on the cause of it, or where you want to <em>change the semantics</em> of it. A classical such case is
<code class="language-plaintext highlighter-rouge">connect_with_retries()</code>, where you might want to allow several attempts before giving up. Here,
you’d probably want to “accumulate” the intermediate exceptions and raise a
<code class="language-plaintext highlighter-rouge">ConnectionError("%d connection attempts failed", accum_exceptions)</code>.</p>
<p>Bottom line: <highlight>wrap only where you **add information** to the
underlying exception;</highlight> there has to be a <strong>good reason</strong> for wrapping.</p>
<h2 id="dont-handle-exceptions">Don’t Handle Exceptions</h2>
<p>Let me rephrase that: exceptions should be handled only</p>
<ul>
<li>
<p>where it’s possible to <strong>fix/recover</strong> from the problem. For instance, if you get an <code class="language-plaintext highlighter-rouge">EINTR</code>
error when <code class="language-plaintext highlighter-rouge">accept()</code>ing on a socket, it makes sense to swallow it and retry. Another use case
is for fallbacks: first try to take the short way, and if it fails, take the long one.
There are <strong>many more</strong> examples of handling an exception, of course, but you have to ask
yourself <em>whether you’re really handling the problem or just masking it</em>.</p>
</li>
<li>
<p>where <strong>cleanup/rollback</strong> is necessary. It may be the case that you need to run rollback code
only if an exception occurs (to release resources, etc.), so a <code class="language-plaintext highlighter-rouge">finally</code>-clause or a <em>context
manager</em> won’t do. In this case you can <code class="language-plaintext highlighter-rouge">except Exception</code>, do the rollback, and <code class="language-plaintext highlighter-rouge">raise</code>
(without passing any arguments to <code class="language-plaintext highlighter-rouge">raise</code>).</p>
<p><em>Note: I stand corrected by Nick Coghlan – you can use context managers for the very same
effect. Forgot about that.</em></p>
</li>
<li>
<p>in the <strong>main function</strong>. Instead of letting the application crash with a traceback, you might
want to log the exception to a file, pop up a message box, ask the user what to do next, etc.</p>
</li>
</ul>
<p>In other words: <highlight>handle exceptions only where you're actually
**handling** them.</highlight> If you’re not sure, leave it that way – you can always add
<code class="language-plaintext highlighter-rouge">except</code>-clauses later.</p>
<p>It might seem trivial, but you’d be surprised how many times I find
code that handles exceptions for no good reason. For example, people think that by swallowing
all sorts of exceptions and returning <code class="language-plaintext highlighter-rouge">None</code>, they make their code “more robust”; <strong>that’s lying
to yourself</strong>. You take a problem and make it worse, as it’s very likely you’d lose the original
details and <em>hide real issues</em>.</p>
<p>I’ve encountered countless cases of code that follows a pattern such as <code class="language-plaintext highlighter-rouge">except Exception:
log.error(...)</code>, which <strong>hides bugs</strong> like a misspelled variable (<code class="language-plaintext highlighter-rouge">NameError</code>). Logging is not
handling the exception <em>(more on this in part 3)</em>. In the end, nobody ever reads the log, or even
takes the time to properly configure it, so you ship a “very robust” product that has half the
functionality you think it has.</p>
<h2 id="closing-words">Closing Words</h2>
<p>The lesson to be learnt here is: <em>think before you act</em>. Don’t act dogmatically, don’t follow Java
paradigms for no reason, and try to keep your footprint as low as possible when it comes to
error handling. As the saying goes, <strong>shit happens</strong>; calling it in other names does not make it
any better. Files disappear, permissions get screwed, devices disconnect, sockets die,
everybody lies. That’s life.</p>
<p>Going back to the three bullets I opened this post with, you can see how by being <em>overprotective</em>
and by excessively wrapping exceptions, an <code class="language-plaintext highlighter-rouge">AttributeError</code> became a <code class="language-plaintext highlighter-rouge">DeviceError</code>, which
then became <code class="language-plaintext highlighter-rouge">None</code>. And you can see now why the first thing I do is remove all
exception-handling code along the way: most of the times it just masks the real error, making it
harder to diagnose, while adding little or no added value at all. You don’t make your code more
robust by sweeping problems under the carpet: <highlight>good code crashes,
allowing tests to uncover more bugs, increasing robustness.</highlight></p>
<h2 id="on-the-granularity-of-exception-classes">On the Granularity of Exception Classes</h2>
<p>Some people are rather laconic and use a single exception for everything. I’ve seen people who were
so lazy that they used <code class="language-plaintext highlighter-rouge">raise Exception("foo")</code> directly, instead of deriving an exception class
of their own… People, it only takes <strong>one line</strong> to derive an exception class, there’s no excuse
for being <strong>that lazy</strong>!</p>
<p>On the other hand, some people are way too verbose, defining specific exceptions for every minor
detail. They end up with dozens of exception classes for each module, many of which are logically
overlapping. This makes the implementation cumbersome, and, in fact, might not be useful at all
for your users: they usually won’t care for such granularity, and you risk contaminating your
interface with implementation details.</p>
<p>The rule to follow here is: <highlight>the granularity at which exceptions
are defined should match the granularity at which exception handling is done</highlight>; define
separate exceptions (only) where it makes sense to handle one differently than the other.</p>
<p>For example, it makes sense to handle a <code class="language-plaintext highlighter-rouge">ConnectionError</code> differently from an
<code class="language-plaintext highlighter-rouge">InvalidCredentials</code> error, but there’s usually little sense in making the distinction between
“connection failed because server is not listening” to “connection failed because server crashed
after accepting us”. But in any case, be sure to <strong>include all the available information</strong>
in the error message, as it’s important for logging/diagnostic purposes.</p>
Javaism, Exceptions, and Logging: Part 12012-07-03T00:00:00+00:00https://tomerfiliba.github.com/blog/Javaism<p>I’m working nowadays on refactoring a large Python codebase at my workplace, and I wanted to share
some of my insights over two some aspects of large-scale projects: exceptions, logging, and
a bit on coding style. Due to it’s length, I decided to split it over three installments; the first
covers <em>Javaism</em> and an introduction to the issues of working with exceptions. <strong>Part 2</strong> suggests
“best practices” concerning exceptions in Python, and <strong>part 3</strong> will cover logging (when, how,
and how much).</p>
<h2 id="javaism">Javaism</h2>
<p>From my long experience in the programming world, I get the feeling that many programmers
(even those fluent in Pythonspeak) come from a rich Java/.NET background, where they’ve acquired
their programming skills and mind-set. Now converted-Pythonists, they are still the “speakers”
of a second language, and they can’t deny their mother tongue. In the context of this post, I’ll
refer to this as <em>Javaism</em>, or <em>thinking Java in Python</em>. Of course it might as well be C# or C++,
but Java is the umbrella term.</p>
<p>You don’t have to go far to see examples of it, for Javaism didn’t skip Python’s standard library:
modules/packages such as <code class="language-plaintext highlighter-rouge">logging</code>, <code class="language-plaintext highlighter-rouge">unittest</code> and <code class="language-plaintext highlighter-rouge">threading</code> where ported almost
isomorphically from Java. On the surface, you might encounter <em>camelCase</em> names (<code class="language-plaintext highlighter-rouge">getLogger</code>),
but the verbosity and over-complicated nature of Java and its inheritance methodology can be seen
anywhere. For instance, recall the complexity of setting up a logger (I have to look it up every
time), or the <code class="language-plaintext highlighter-rouge">threading.Thread</code> class… I really don’t wish to digress here, but I feel that a
concrete example would make this point clear:</p>
<ul>
<li>
<p>The canonical way to write thread functions is by subclassing <code class="language-plaintext highlighter-rouge">Thread</code> and implementing
<code class="language-plaintext highlighter-rouge">run()</code>; you can pass a callback (called
<a href="http://docs.oracle.com/javase/6/docs/api/java/lang/Thread.html#Thread(java.lang.Runnable)">target</a>,
but it seems like an afterthrought (for once, it’s not the first argument of the constructor).</p>
<p>Recall that <em>anonymous classes</em> weren’t always there in Java, so extending a class was the
shortest way to emulate first-class functions.</p>
</li>
<li>
<p>Speaking of the Thread constructor – it takes so many optional arguments (<code class="language-plaintext highlighter-rouge">group</code>?!),
but none for <code class="language-plaintext highlighter-rouge">daemon</code>. For that, you have to imperatively call <code class="language-plaintext highlighter-rouge">setDaemon()</code> afterwards.</p>
<p>Why? Because Java doesn’t have <em>keyword arguments</em>, so by adding an additional argument you
double the number of overloaded constructors (exponential growth), while get/set properties
behave “linearly”.</p>
</li>
<li>
<p>Also, you first <em>instantiate</em> the thread object, then <code class="language-plaintext highlighter-rouge">start()</code> it… where’s the sense in that?
What can you <em>do</em> with an <em>unstarted</em> thread object (other than calling <code class="language-plaintext highlighter-rouge">setDaemon</code>)?
Consider <code class="language-plaintext highlighter-rouge">Popen</code> or <code class="language-plaintext highlighter-rouge">file()</code> – the <a href="http://www.python.org/dev/peps/pep-0020/">one, obvious way</a>
that Python follows is <a href="http://en.wikipedia.org/wiki/Resource_Acquisition_Is_Initialization">resource instantiation is acquisition</a>.
Why introduce useless states in the lifetime of the object? You just lay the ground for
unexpected code flows that may result in bugs.</p>
<p>What can you do with an unstrarted thread? Pass it on to someone else, who will start it. That’s
a very general concept, called <a href="http://docs.python.org/library/functools.html#functools.partial">partial application</a>,
or <a href="http://en.wikipedia.org/wiki/Currying">currying</a>. Python has lambda functions and <code class="language-plaintext highlighter-rouge">partial</code>,
but Java doesn’t… The easiest way to create such “deferred” objects is to add “deferredness”
as an internal state of the object – it’s a common Java practice, uncalled for in Python.</p>
</li>
</ul>
<p>It’s funny how the <em>limitations</em> of Java were ported to Python as well, and made the implementation
ugly – it’s a classical case of “I don’t want to think, let’s just copy an existing solution”.
Luckily, it seems to be a thing of the past – it only entered stdlib in the old days, before the
community had a clear notion of what <em>being pythonic</em> meant. But still, Javaism of all degrees is
widespread, especially in corporate-developed large-scale projects (Zope and twisted, to name a
few, but naturally closed-source corporate-internal projects are even more susceptible).</p>
<h2 id="exceptions">Exceptions</h2>
<p>Java had a good insight (that they most likely stole from some other language) in that exceptions
are part of a function’s signature. Just like a function takes an argument of type T1 and returns a
result of type T2, it also has “side channels” through which it can produce results, known
as exceptions, which should be documented and checked as well.</p>
<p>The problem is, trying to foresee everything that might ever go wrong is a futile attempt, and
even Java itself makes two exceptions (no pun intended): <code class="language-plaintext highlighter-rouge">Error</code>, for unrecoverable exceptions
(such as <code class="language-plaintext highlighter-rouge">VirtualMachineError</code>), and <code class="language-plaintext highlighter-rouge">RuntimeException</code>, for exceptions that may always occur
(such as <code class="language-plaintext highlighter-rouge">NullPointerException</code>).</p>
<p>The fundamental idea is correct, but it was bound to fail: first, because people are lazy,
but most importantly, because trying to predict all unexpected, future edge cases is absurd.
For instance, suppose you’re implementing an interface that stores data (say, in files), so you
might find yourself implementing a signature such as <code class="language-plaintext highlighter-rouge">void write(byte[] data) throws IOException</code>.
Now suppose your implementation uses a third-party database engine, that throws <code class="language-plaintext highlighter-rouge">MySQLException</code>.
For obvious reasons, <code class="language-plaintext highlighter-rouge">MySQLException</code> does not derive from <code class="language-plaintext highlighter-rouge">IOException</code>, and there’s nothing
you can do about it, as both the interface and the DB engine are given to you. You’re now faced
with three options:</p>
<ol>
<li>Translate <code class="language-plaintext highlighter-rouge">MySQLExceptions</code> into <code class="language-plaintext highlighter-rouge">IOExceptions</code></li>
<li>When designing interfaces, always declare the most general exception in <code class="language-plaintext highlighter-rouge">throws</code> clauses</li>
<li>When implementing libraries, always derive your exceptions from an unchecked exception
(<code class="language-plaintext highlighter-rouge">RuntimeException</code>)</li>
</ol>
<p>In short – you need to <strong>find a workaround</strong> and by-pass the compiler’s checking. This
essentially means that <code class="language-plaintext highlighter-rouge">throws</code> should have served for documentation-only purposes, where the
compiler might produce (suppressible) <em>warnings</em> should you not follow conventions. It’s more of
a semantic property, like idempotence or thread-safety… you may state it in your Javadoc,
but you wouldn’t expect the compiler to <em>enforce</em> that (not in a language like Java, anyway).</p>
<p>I’d guess most people agree that the second and third options are “inherently bad”, but opinions
diverge on the first one. I will try to show that <em>exception-wrapping</em> (translating exceptions)
is just as bad – at least when it comes to Python. We’ll cover this in <strong>part 2</strong>.</p>
RPCs, Life and All2012-06-25T00:00:00+00:00https://tomerfiliba.github.com/blog/RPCs-Life-And-All<p>A colleague of mine, <a href="https://twitter.com/#!/gavrieph">Gavrie Philipson</a>, has written an
interesting blog post titled <a href="http://philipson.co.il/blog/2012/06/20/why-i-dont-like-rpc/">Why I Don’t Like RPC</a>,
in which he explains that transparent/seamless RPCs (a la <a href="http://rpyc.sf.net">RPyC</a>) make
debugging and reasoning efforts hard. For instance, you might work with an object (a proxy) that
points to an object on the server process, which, in turn, is also a proxy that points to an
object on yet another server process. Ideally, your local code shouldn’t be aware of the
complexity (“number of hops”) or the details – but that’s not always the case.</p>
<p>Well, he won’t allow commenting on his blog, so I’m forced to formulate my response here :)
Naturally, I’m biased about this subject, but I thought if I’m on it, why not also cover the
broader aspects of the issue… However, it just kept getting longer and longer, until I got this
behemoth of a blog post, so I’m attaching a <strong>TL;DR info box</strong>:</p>
<ul class="notebox">
<li>Transparent object proxying is only the logical way to extend RPCs to duck-typed languages</li>
<li>Asking for a <em>clear distinction</em> between local and remote objects ultimately means you're
asking for a statically-typed language; it doens't make sense to ask for it in python</li>
<li>Network programming is hard, and it's a pity we still work at the socket level; we should
strive for a decent fifth layer that would eliminate all the unnecessary complexity</li>
<li>General-purpose RPCs are the <strong>right primitive</strong> over which network programming should be
abstracted: it's the missing fifth layer, which every network application reinvents</li>
<li>HTTP is a half-baked, broken alternative to a fifth layer; I'm glad ZeroMQ and others are
starting to loosen its grasp.</li>
<li>RPyC <strong>can</strong> be used efficiently and correctly, it's not an impossible feat. Also, a show
case of how I'm using RPyC to build a testing environment.</li>
</ul>
<h2 id="on-transparency">On Transparency</h2>
<p>Gavrie’s main point is that transparency opens the door to (possibly) unplanned and undesired
complexity. When it’s “too easy” to spread around, you might be tempted to (or even unknowingly
end up with) creating over complicated (and cyclic) dependency graphs, stretching over several
processes / machines, where it would be quite a feat to see the whole picture. The nice thing is,
when it works - it just works (and you’re happy with your design), but when it fails, you want
to be able to untangle the mess. As he puts it:</p>
<blockquote>
<p>“Seamless” RPC encourages the writing of spaghetti code, because it’s so easy to mix local
and remote code. This makes it deceptively easy to write distributed code without thinking
about the design of the API and about which parts should reside on each side of the connection</p>
</blockquote>
<p>An ideal remoting solution, according to Gavrie,</p>
<blockquote>
<p>[…] should make the distinction between local code and remote code crystal
clear to the developer.</p>
</blockquote>
<h2 id="on-duck-typing">On Duck Typing</h2>
<p>I agree with the key points in Gavrie’s argument, and I can assert that <em>debugging RPyC itself</em>,
during development, is highly deceptive (hint: never <em>print</em> an object…) and calls for
all sorts of creative solutions. But then again, you <em>are</em> using a <strong>duck-typed, interpreted
language</strong>. The “if it walks like a duck” phrase will soon make a more dramatic entrance,
but for the time being, suffice it to say we only care for the <em>runtime behavior</em> of an object.
Any object that I can <code class="language-plaintext highlighter-rouge">.read()</code> from or <code class="language-plaintext highlighter-rouge">.write()</code> to, is a “file concept”, and thus code
should be compatible with any object that “adheres to this concept”.</p>
<p>A “file” might be an on-disk file, an in-memory byte stream, a mock object used for testing,
or a SCSI device located on a remote storage array. <strong>The only natural way to extend the notion
of duck-typing to RPCs is via transparent object proxying</strong>. If <code class="language-plaintext highlighter-rouge">open()</code> doesn’t differentiate
between local and remote (NFS/SMB) files, and if my code doesn’t care for anything other than
<em>runtime behavior</em>, such a “file concept” might as well be a proxy object that points to a file
object on a remote machine. It’s only logical… in fact, it’s taking duck typing to its
full potential!</p>
<h2 id="on-types">On Types</h2>
<p>Gavrie wants the RPC framework to “make the distinction between local code and remote code crystal
clear to the developer” – well, Garvie might not have thought about it thoroughly,
but what he’s actually asking for is a <strong>static type system</strong>. Types allow us to <em>make distinctions
between objects</em>; they partition the “universe of data” into disjoint subsets that we can
reason about: integers, floats, strings (to name a few). We can then group several types together,
under the notion of a <em>type class</em>. For instance, floats and integers, albeit inherently different,
both belong in the type class <em>number</em>, which provides us with additional operational semantics
(like ordering relations, etc.).</p>
<p>We use types to partition the universe, because different “things” have different semantics
and it doesn’t make sense to mix them together (modulo converting data of one type to data of
another type). In fact, we normally want to <em>prevent</em> ourselves from mixing incompatible objects
(be it integers and strings, or local and remote references) – that’s why we have type systems,
and catch type mismatches at compilation time.</p>
<p>When you require a <em>clear distinction</em> between objects, it means you’re after a <strong>statically typed
language</strong>; otherwise, you might as well just come up with a <em>naming convention</em>, where all
variables that (might) refer to a remote object start with <code class="language-plaintext highlighter-rouge">rem_</code>. But if you want
this distinction to <em>propagate</em> throughout the code, it must be <em>enforced</em> by a compiler;
if you want to keep yourself in the duck-typed world, it doesn’t make sense.</p>
<p>Duck-typing (from a type-theoretic perspective) is like saying there’s only a
<a href="[http://www.haskell.org/haskellwiki/Why_Haskell_matters">single data type</a>, which covers the
entire universe; all checking is deferred to runtime, in which case it might (1) work, (2) fail,
(3) silently cause corruption (as in a <code class="language-plaintext highlighter-rouge">TextToSpeech</code> instance, which may as well expose
a <code class="language-plaintext highlighter-rouge">.read()</code> method, but it surely won’t do what we expect the “file concept” to do).</p>
<p>So asking for a “clear distinction” in a duck typed language is simply out of the question.
What we can ask for is distinction in the level of APIs; for instance,
<code class="language-plaintext highlighter-rouge">write_local_file(filename)</code> vs. <code class="language-plaintext highlighter-rouge">write_remote_file(filename)</code>, but that breaks the
“spirit of duck-typing”, where objects are no longer considered equal (even though they provide
the desired runtime behavior). It’s like pointing out the ugly ducklings and making fun of them…
that’s not cool.</p>
<p>Just to contrast, an RPyC-like library for Haskell would expose remote references as a distinct
type class. You could have a function like <code class="language-plaintext highlighter-rouge">remoteSum :: (Num a) => Remote [a] -> a</code>, which
takes a reference to a remote list of <code class="language-plaintext highlighter-rouge">a</code>’s, and returns its sum. Because it <em>knows</em> it
operates on remote lists, it might be able to “move” the actual summation remotely, instead of
sending the entire list over the network, item by item. I think this qualifies for a
“crystal clear distinction”, but of course, that’s not what the snake teaches.</p>
<h2 id="on-networks">On Networks</h2>
<p>It seems to me that people find it easy to abstract all sorts of concepts, as long as they don’t
concern networks. When there’s network involved, they tend to want to “get a feel of the wire”…
so they might use HTTP instead of the NIC directly, but they won’t take the next step and treat
network resources as first class objects. There’s always a gap, between what’s <em>here</em> and what’s
<em>there</em>, and we’re still too aware of the <em>how to get there</em> details… a gap that should have
been bridged over long ago.</p>
<p>As I see it, it stems from two primal fears, so to speak: networks are <em>hard</em> (timeouts, routing,
DNS, reconnects, authentication, compression, tunneling, round-trip time, …) and <em>unreliable</em>.
As far as unreliability goes, there’s not much you <em>can</em> do about it; after all, the server is
a process like any other, and may crash at any point of time. If that’s not enough, the remote
machine might freeze or reboot. But then again, your local machine might kernel-panic or just
go down with a power failure, losing any unflushed data… but that’s life. On the other hand,
it seems to me that overall (hardware?) unreliability rates are going down with time, which is
a promising outlook.</p>
<p>The “hardness” of network programming, on the other hand, is something we <em>can solve</em>. Good network
programming is hard, there’s no question there, but for some reason, instead of solving the
problem once of for all in a generic manner, it seems that every protocol / network-oriented
application seeks to start at square one. Of course, done this way, it only handles the aspects
it finds relevant… doing network programming at the socket level is analogous to rewriting
the kernel for every desktop application. It doesn’t make sense.</p>
<p>I’ve started (and abandoned) an ambitious project called <a href="/projects">layer5</a>,
which aimed to concentrate all the network-related sorcery in a single place, so that programs on
top of it wouldn’t have to care. It originated from my frustration with network programming in
general (and in RPyC in particular)… things like handling timeouts, reconnects, authentication,
negotiation, compression, serialization, load distribution, caching, error reporting – you name it.</p>
<p>Just to show-off a couple of ideas, consider a socket connection being dropped for some reason:
if the network layer knew how to reconnect and resume the session, or automatically resend a
request after some timeout (all configurable, of course), there would be no need for the
application to be aware of anything. And, once you “lift” your code up from the socket layer,
you can enable things like “moving targets”, where you may switch an IP address (wifi/3G) and the
connection will just “follow you”; the <em>session</em> would not be bound to a “physical endpoint”.
These are just some of the issues that layer5 attempted to solve.</p>
<h2 id="on-rpcs">On RPCs</h2>
<p>Let me make a bold claim: <strong>everything is RPC</strong>. So, by <em>everything</em> I mean <em>virtually all</em>
connection-oriented network protocols (e.g., excluding broadcasts / multicasts / streaming),
and I take <em>RPC</em> to its broadest sense: an RPC is any message-oriented protocol in which one side
makes requests that the other side fulfills: basically invoking a remote function. Naturally,
in order to convey a message, the RPC imposes a serialization format, and in order to tell success
from failure, it must also define “return codes”. <strong>Note:</strong> I’ve been planning to write about this
topic for a very long time, but never got to it; it surely deserves a post of its own, but
until that happens, please consider this a “briefing”.</p>
<p>As a case study, let’s examine HTTP: there are 4 (or so) <em>methods</em>: <code class="language-plaintext highlighter-rouge">GET</code>, <code class="language-plaintext highlighter-rouge">PUT</code>, <code class="language-plaintext highlighter-rouge">POST</code>
and <code class="language-plaintext highlighter-rouge">DELETE</code>. Each such <em>method</em> takes <em>arguments</em>, like the URL, cookies, accept-encoding,
etc. Some of them are required, some are optional; some pertain to the transport layer
(content-length, compression, timeouts) while others to the method itself (URL, cookie, …).
It also defines <em>status codes</em> for distinguishing errors from success (and again, it mixes
transport-layer errors like <em>redirect</em> with method-level ones like <em>not found</em> or
<em>internal server error</em>). It also defines a (very loose) serialization format for encoding
the method’s arguments (newline-separated key-value strings) and the payload
(<code class="language-plaintext highlighter-rouge">multipart/form-data</code>)… So, from an RPC point of view, HTTP is a service that provides 4
functions (<em>methods</em>), whose signature is something like <code class="language-plaintext highlighter-rouge">(url, formdata = None, **kwargs)</code>.</p>
<p>Another example is <code class="language-plaintext highlighter-rouge">tenlet</code> – it basically provides a function whose signature
is <code class="language-plaintext highlighter-rouge">void write(char ch)</code>; when you type, <code class="language-plaintext highlighter-rouge">write</code> is invoked for each key stroke. Aside from
sending characters, telnet also provides all sorts of <em>negotiation</em> options or <em>commands</em>, like
<code class="language-plaintext highlighter-rouge">bool set_binary()</code>, <code class="language-plaintext highlighter-rouge">void set_terminal(string)</code>, etc.</p>
<p>As with most <strong>ad-hoc RPC</strong> protocols, the two we’ve examined make horrible design choices like
mixing transport-layer options with “business logic” (HTTP) or sending control in-band with the
data (telnet), where a mere <code class="language-plaintext highlighter-rouge">\xFF</code> character in the stream marks the beginning of a command,
so anyone can (maliciously or accidentally) inject commands into the stream. Yet another pitfall
of these protocols is, they begin small, targeting a specific task, but if they’re successful,
they grow to <a href="http://www.telnet.org/htm/dev.htm">incorporate</a> many unrelated things, like
encryption and proxy support… as if security is something you sprinkle on top.</p>
<p>The main point I’m trying to make here is this: <strong>virtually all protocols are basically
degenerate forms of RPC</strong>. To paraphrase <a href="http://en.wikipedia.org/wiki/Greenspun's_tenth_rule">Greenspun</a>,
all sufficiently complicated network protocols end up redoing compression, security, authentication,
framing, serialization, negotiation / versioning, discovery, you name it (<em>Filiba’s Eleventh Rule</em>).
This observation has brought me to the conclusion that doing network programming at the
“byte level” is wrong, and that a <strong>general-purpose RPC layer</strong> is the <strong>right primitive</strong> for this.</p>
<p>A general purpose RPC would be <em>language-agnostic</em>, support only simple by-value types, such
as strings, integers and lists (anything more complex can be built on top of that). If would also
make no assumptions on how remote functions operate, if would only care for their signature.
You can think of it as a more structured message-passing protocol, where you replace the notion
of “message codes” by “function names”. This way, it’s easy to see that one can straight-forwardly
emulate any message-passing protocol or more advanced RPC, over this layer. Heck, it’s f***ing
2012, I want to <code class="language-plaintext highlighter-rouge">GET("/index.html", Agent="Chrome")</code>, not formulate <code class="language-plaintext highlighter-rouge">\r\n</code>-separated strings or
care about XML/JSON.</p>
<p>Layer5 (mentioned in the previous section) was to expose such a generic RPC, upon which
applications would base their protocols. You could always just implement a
<code class="language-plaintext highlighter-rouge">bytes send(bytes data)</code> RPC (over which would continue to pass your byte-level messages),
or implement a more semantic interface – your choice. Either way, you’d benefit from layer5’s
handling of reconnects, authentication, and the rest of the list.</p>
<h2 id="on-http">On HTTP</h2>
<p>Truth is, we sort-of already have such a “general purpose” layer 5 protocol, called HTTP. By a
strange twist of fate, HTTP has become the de-facto application layer of choice, over which all
kinds of protocols now operate. And we’ve managed to hide the gruesome details of HTTP under
programmer-friendly libraries and APIs, so we are in a “better shape” now. Yet HTTP is such a
<strong>miserable choice</strong> for this purpose (hence abominations like
<a href="http://en.wikipedia.org/wiki/WebSocket">Websockets</a> arose), and we all pay the price
(programmatically-speaking, but also in the sense of network bandwidth, CPU time and electricity
bills).</p>
<p>In my opinion, HTTP owes its success to another misfortunate happening - <em>firewalls</em>. In the days
of yore, people thought they could eliminate threats simply by blocking all TCP ports, except
for safe/trusted ones. HTTP was considered safe, as it only transfered “hypertext”, so all
firewalls allowed port 80 traffic by default. This fact has led to many protocols being designed
to work on top of HTTP, to be firewall-friendly… which meant firewalls no longer served the
purpose for which they were conceived: blocking ports was not enough, so firewalls had to become
<em>content-aware</em> anyway. Long story short – we’ve only managed to push the problem one level up:
instead of solving it in the forth layer, we now do it in the application layer… no matter what
the port number is.</p>
<p>However, this initial edge that HTTP had, helped in making it the de-facto transport layer of
choice, which of course had a snowball effect. I’m only happy to see competing protocols like
<a href="http://www.zeromq.org/">ZeroMQ</a> and <a href="http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol">AMQP</a>
are starting to take some market share. Down with HTTP!</p>
<h2 id="on-rpyc">On RPyC</h2>
<p>Coming back to Gavrie’s post, he brings up two additional points. The first:</p>
<blockquote>
<p>In addition, its performance can quickly deteriorate: Objects are being serialized back and
forth all the time, and tens of implicit network round-trips introduce latency all around the code.</p>
</blockquote>
<p>Performance is a tricky thing. First, RPyC is mostly ever used on local, secure networks, where
latency and round-trip time (RTT) are low – so unless you do something really flawed, you shouldn’t
experience noticeable degradation. Second, the only places that do suffer from RTT are tight-loops,
and to that end, RPyC <a href="http://rpyc.sourceforge.net/api/utils_classic.html#rpyc.utils.helpers.buffiter">already has solutions</a>.
And thirdly, transparency (like any form of abstraction) hides the underlying complexity, which
means you won’t be able to optimize all the way. RPyC makes a choice for simplicity and pythonicity
every time, at the expense of performance.</p>
<p>From my many years of using RPyC, I must say I’ve never experienced performance <strong>issues</strong> that
didn’t originate from the use of threading and locks in python, or really bad code. And if the times
are tough, you can always apply lightweight optimization techniques, such as locally “caching”
remote objects that were obtained through a series of lookups, in variables (e.g.,
<code class="language-plaintext highlighter-rouge">myfunc = w.x.y.z.myfunc</code>)… it’s normally not that hard. I’m sure Gavrie has experienced
performance issues with RPyC, but I can hardly imagine it could not be solved by reasonable
amounts of such refactoring.</p>
<p>Which brings us to the last point Gavrie makes:</p>
<blockquote>
<p>I don’t like RPC, especially not stateful RPC that supports access of remote objects by reference</p>
</blockquote>
<p>I hope we already agreed that a general-purpose RPC is equivalent (if not better than) to any
“normal” network protocol, so it’s really not RPCs that Gavrie hates but stateful / object-proxying
sessions. This invites another, rather philosophic, question: what is <em>state</em>? What does it mean
that a protocol is <em>stateless</em>? I’d guess philosophies like <a href="http://en.wikipedia.org/wiki/REST">REST</a>
come to mind, but that’s just a buzzword. From the broadest Turing-machine perspective, if REST
or any other protocol were truly stateless, they would have no <em>effect</em> on the world and thus would
be of little significance (they’d be <em>read-only</em> protocols). Just to stress this point –
<code class="language-plaintext highlighter-rouge">PUT/POST</code>-ing to a RESTful interface, adding/altering a record in a database table, is clearly
<em>stateful</em>: you changed the state of the DB.</p>
<p>Therefore, these buzzword-rich protocols boast themselves with the term <em>stateless</em>, while they
mean something very different. In lack of a better term, I’d use <strong>atomicity, durability,
and reboot-ability</strong> – which we’ll discuss next. And just a last bit of REST: REST has a notion
of idempotency, as GET requests for the same URI should always return the same result (but that’s
not guaranteed). Anyway, the <a href="http://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD model</a>
which REST employs is quite limited and fits only so many real-life problems (many other problems
can be <em>reduced</em> to CRUD, but I don’t suppose people would consider this the “right way”).</p>
<p><em>Atomicity</em> and <em>durability</em> come from the <a href="http://en.wikipedia.org/wiki/ACID">ACID</a> philosophy of
databases, and are granted to you freely, assuming you use a DB (who doesn’t?). They basically mean
that a transaction either fully happens (and then its permanently stored), or nothing happens
(so that no partial results may exist). Reboot-ability is a term I just made up, and it means
your server might crash and be restarted, or your entire lab might burn away, and the client
shouldn’t be able to detect any difference (other than temporal unavailability, which may be
compensated for by a cluster). Inherently, it means you don’t trust your server to survive over
long periods of time, and therefore prefer to make it (the server <strong>process</strong>) stateless.
In effect, it means the server will never make “changes to the world” outside of DB transaction,
so that a failed transaction could be rolled back, and a new server process could resume where the
previous one failed. But note the difference: <strong>the server process is stateless, not the protocol</strong>.</p>
<p>HTTP originally was a connection-less protocol (albeit over TCP), where each request was treated
separately from the rest and there was hardly any notion of a session. This meant that every
request had to carry along with it any state information it required. In order to prevent requests
from growing wildly in size and choking the network, cookies were invented – which meant the
server had to store session data, and the cookie was only a key. This already dents the notion
of statelessness, and nowadays, things like websockets break it completely.</p>
<p>RPyC, on the other hand, has a clear notion of a <em>session</em>: on each end of the the connection
there’s a dictionary of objects referred to by the other side. This means that should a connection
drop, there’s little chance of restoring the lost session: the dictionaries are lost, and all
proxies would be invalidated.</p>
<p>But all is not lost: if you only need references to serializable objects, you might as well keep
this dictionary as a DB table. Since the data, including object IDs, would not be lost when the
server gets restarted, resuming a dropped session is easy. So, if you could live with limited
functionality, you can be backed by a DB – but it’s not that HTTP offers a better solution.
At least keep your code pythonic and not full of HTTP curses.</p>
<p>Another alternative, which I’ll demonstrate in the next section, is making the changes directly
“in the real world”: instead of <em>storing</em> state, make the changes on long-living entities, and
then read the info back from them. This way, you’re always synchronized, and should you be
restarted, you’ll never use stale data.</p>
<h2 id="on-testing">On Testing</h2>
<p>I’m working now on a testing framework of quite a complicated nature: first, it serves as a
resource-allocator for hosts and other testing equipment; second, in order to run tests, it must
create a suitable environment for them. But this is where it gets fun: in order to do set up the
environment, I must use the utilities that I set out to test… because that’s exactly what they
do. Chicken and egg, anybody?</p>
<p>After a couple of days to toying with it, I settled for the following architecture:</p>
<ul>
<li>Tests are written normally using <code class="language-plaintext highlighter-rouge">unittest</code>, but they also make use of a little module (8 LoC
or so) that provides them with a means to connect to the resource allocator; this module
basically hides the details of setting up an RPyC connection (as the server is well known, etc).</li>
<li>The resource allocator exposes a simple service, with methods like <code class="language-plaintext highlighter-rouge">get_system(version = 17)</code>.
It collects the information about the systems from a third-party service and caches it in-memory,
so finding a matching system is quick and efficient. Basically, the resource allocator only
takes care of distributing systems randomly (we do want to allow for two tests to run on the
same system, but wish to avoid unnecessary contention).</li>
<li>The object returned by <code class="language-plaintext highlighter-rouge">get_system</code> is an instance of a peculiar class called
<code class="language-plaintext highlighter-rouge">HostViewOfSystem</code>. It basically represents a how the host (running the test) views the system
that’s been allocated to it, and it has methods like <code class="language-plaintext highlighter-rouge">get_resource_from_system()</code> that look
for an unused resource (or create one) and take care of <em>making it usable by the host</em>.</li>
</ul>
<p>There are quite a few details, but I hope I managed to make the design clear. On the other hand,
it doesn’t seem particularly interesting – until we get to the last bullet-point – making the
resource usable by the host. In order to do that, the server (resource allocator) creates a
temporary directory on the host, onto which it copies (over RPyC) several python packages that
are required for the task. It then fires up a new RPyC server on the host, and sets its
<code class="language-plaintext highlighter-rouge">PYTHONPATH</code> to this temporary location. This RPyC server is based on the fresh-from-the-oven
<code class="language-plaintext highlighter-rouge">OneShotServer</code>, which is capable of serving a single client and then quits. The server chooses
a random port and reports it over stdout, to the resource allocator, who then connects to it.
Then, the <code class="language-plaintext highlighter-rouge">HostViewOfSystem</code> object is given a reference to the newly created RPyC service,
and it uses it to manipulate the host machine in order to set up the environment. Here’s a sketch:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>+---Test Host---+ +---ResAlloc Server---+
| | | |
| ----------- | .-----------> HostViewOfSystem |
| | Test |____/ | | |
| | process | | | | |
| ----------- | _____________/ |
| | / | |
| ----------- | / | |
| | Newly <-----* | |
| | started | | | |
| | RPyC | | +---------------------+
| | server | |
| ----------- |
| |
+---------------+
</code></pre></div></div>
<p>This quite tiresome setup actually takes only ~30 lines of code (and around one second to build),
and it allows the tested utilities to rely on <em>stable versions</em> of themselves. The stable versions
are fetched from the resource allocator server, thus we make absolutely no assumptions on the
state of the host. Moreover, when we “allocate” a resource for a specific test, we <em>mark</em> the
resource on the system as “in used”: it’s neither stored in-memory, nor in a DB – it’s marked
directly on the resource, as metadata. This way, if the resource allocator is restarted,
no state is lost – the new instance will read the most up-to-date state from the “real world”.
There – a semi-stateless testing framework based on RPyC… and now I go to bed.</p>
Reed-Solomon Codec2012-06-08T00:00:00+00:00https://tomerfiliba.github.com/blog/ReedSolo<h2 id="some-background">Some Background</h2>
<p>I’m working on an image processing project for the university, whose purpose is to embed
(an extract) a <em>print-scan resilient watermark</em> into an image. This project has (sadly) gotten me
acquainted with Matlab, from which I quickly ran way into the friendlier realms of
<a href="http://www.scipy.org/">Scipy</a> and friends (<a href="http://scikits-image.org/">Skimage</a> rocks, by the way).
I must say I really learned to appreciate the Scipy/Numpy gang in the last two weeks :)</p>
<p>If it wasn’t already obvious, it’s time to admit I’m a n00b when it comes to signal processing and
applied math in general. I know the Fourier transform in broad terms, and have heard of
<a href="http://en.wikipedia.org/wiki/Discrete_cosine_transform">discrete cosine transform</a>
and <a href="http://en.wikipedia.org/wiki/Wavelet">wavelets</a> some time ago… but it’s not my cup of tea,
to say the least. Luckily for <strong>clueless people</strong> like me, Matlab (and its kin) enables us
to summon the dark powers of mathematics, without ever having to know what we’re doing. Hurrah!</p>
<p>So I’m DFT’ing, DCT’ing and DWT’ing like a pro, embedding my watermark using CDMA/spread-spectrum
techniques in the frequency domain, and then inverting the process… and all I know is
I’m supposed to keep myself in the mid-frequency range, i.e., in a certain region of the matrix.
Math for n00bs.</p>
<p>My original idea was to take a string, encode it as a <a href="http://en.wikipedia.org/wiki/QR_code">QR code</a>,
and then embed this QR image into the host image. I thought it would be a nice shortcut, as it
provided me with a synchronization pattern and error correction out of the box, but it quickly
turned out QR codes generate a payload that’s too big for unobservable embedding.
So I set out to find some error correcting code (ECC) library for python, but it proved to be
a really difficult task. I found some packages, most of them are haven’t been maintained in over
7 years nows, and all of them make use of extension modules that failed to compile. Then there’s
<a href="http://pypi.python.org/pypi/zfec">zfec</a>, but for the life of me, I couldn’t figure out how to
use it as a simple encoder/decoder.</p>
<h2 id="the-library">The Library</h2>
<p>I almost gave up and resorted to triplicating my payload (at the bit level), and using
majority-selection for each bit, when I came across an <strong>amazing python tutorial</strong> (with runnable
code) that covers <a href="http://en.wikiversity.org/wiki/Reed%E2%80%93Solomon_codes_for_coders">Reed Solomon codes and QR in depth</a>.
I simply extracted the code, added a usable API, wrote some examples and quickly
<a href="http://pypi.python.org/pypi/reedsolo">uploaded it to PyPI</a>, so now there’s a pure-python
Reed-Solomon encoder/decoder: <code class="language-plaintext highlighter-rouge">pip install reedsolo</code>.</p>
<p>The library should support python 2.4-3.2, using strings or bytes. I really can’t verify the
correctness of the algorithm (it’s beyond me), but it seems to work so I’m fine with it. Here’s
a short demo:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> from reedsolo import RSCodec
>>> rs = RSCodec(10) # 10 bytes of ECC will be added to the output,
... # which allows us to correct up to 5 byte-level errors
>>> rs.encode([1,2,3,4])
'\x01\x02\x03\x04,\x9d\x1c+=\xf8h\xfa\x98M'
>>> rs.encode("hello world")
'hello world\xed%T\xc4\xfd\xfd\x89\xf3\xa8\xaa'
>>> rs.decode('hello world\xed%T\xc4\xfd\xfd\x89\xf3\xa8\xaa')
'hello world'</code></pre></figure>
<p>Now let’s add some errors:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> rs.decode('hXXlo worXd\xed%T\xc4\xfdX\x89\xf3\xa8\xaa') # 4 errors - ok
'hello world'
>>> rs.decode('hXXXo worXd\xed%T\xc4\xfdXX\xf3\xa8\xaa') # 6 errors - fail
Traceback (most recent call last):
...
reedsolo.ReedSolomonError: Could not locate error</code></pre></figure>
<p>It’s pure python and highly unoptimized… I think someone acquainted with Numpy a little
more than I am could improve it blindfolded by a factor 10, but even now, on my dinosaur machine,
it encodes a 400kB message in 2.9 seconds and decodes it in 1.9 seconds. I’ll drink to that.
By the way, it seems that the library <strong>can only handle messages that are less than 255 bytes
long</strong>… but then you can simply encode/decode in chunks. I’ll include it in later versions.</p>
<p>I think a good ECC library for python is very useful… if anyone wants to join in on it,
feel free to drop me a line at the comments or just
<a href="https://github.com/tomerfiliba/reedsolomon">fork the repo</a>.</p>
Some Notes on RPyC 3.2.x2012-06-06T00:00:00+00:00https://tomerfiliba.github.com/blog/Some-Notes-On-RPyC<p>As I said in the previous blog post, I hoped for v3.2.2 to be the last release of the 3.2 line…
Naturally, I was wrong :) Turns out the fix for
<a href="https://github.com/tomerfiliba/rpyc/issues/76>">issue #76</a> was buggy, and I decided to finally
remove the use of excepthooks in favor of taking care of remote traceback chaining in the
exception class’ <code class="language-plaintext highlighter-rouge">__str__</code> method itself. The tentative release date for the release is
August 1st, and I really hope to end the 3.2 line here.</p>
<p>I created a branch called <code class="language-plaintext highlighter-rouge">LTS3.2</code> which will be used only for back-porting bug fixes from
the <code class="language-plaintext highlighter-rouge">master</code> branch, which has now become the development branch of 3.3. I rebased <code class="language-plaintext highlighter-rouge">master</code>
on top of <code class="language-plaintext highlighter-rouge">LTS3.2</code> now, so that the two would have linear histories, which of course resulted
in a forced update, so anyone who was using master for development would now have to do
<code class="language-plaintext highlighter-rouge">git fetch origin; git reset --hard origin/master</code> instead of a simple <code class="language-plaintext highlighter-rouge">git pull</code>.
It’s not supposed to happen again, sorry.</p>
<p>There shouldn’t be any more features in v3.2.3, so the development would take place on <code class="language-plaintext highlighter-rouge">master</code>,
from which I’ll cherry-pick bug fixes onto LTS3.2. That’s all for now…</p>
RPyC 3.2.2 Released2012-06-01T00:00:00+00:00https://tomerfiliba.github.com/blog/RPyC-3.2.2<p>This is a maintenance release, fixing some issues concerning introspection, <code class="language-plaintext highlighter-rouge">ForkingServer</code>
and signals, IronPython and signals, and SSH on Windows. It also introduces optional logging
of exceptions that occur over the RPyC connection to the server’s logger (or any other logger
instance, given in the connection’s configuration). The
<a href="http://rpyc.sourceforge.net/changelog.html">change log</a> has more details.</p>
<p><strong>Note:</strong> This is the last release of the 3.2 branch. Future releases (of the 3.3 branch) will
require <a href="http://plumbum.readthedocs.org">plumbum</a>, thus the SSH support will be removed from the
codebase. The next version will also provide <a href="https://github.com/tomerfiliba/rpyc/issues/81">zero-deploy</a>,
which will make RPyC much easier to use. This release is expected in September.</p>
The Future of Construct2012-05-16T00:00:00+00:00https://tomerfiliba.github.com/blog/Construct-Plans<p>It’s been a long while since I’ve put time into <a href="http://construct.wikispaces.com">Construct</a>.
I gave up on it somewhere in 2007, right after the release of v2.0… I think I just got bored,
and felt like the library was complete and extensible enough to survive on its own. Of course
I was wrong there, and code-rot had spread all over.</p>
<p>Luckily for us, <a href="https://github.com/MostAwesomeDude/">Corbin Simpson</a> took up the project in
January of 2011 and has been maintaining it since then. He migrated it to github, changed the
project to use a proper directory structure, fixed lots of bugs, and wrote
<a href="http://construct.readthedocs.org/en/latest/">extensive documentation</a>. Since then, Construct
has been building a solid community and has reached quite a remarkable number of downloads on PyPI.</p>
<p>All this time, I’ve been busy with my other projects, but I kept toying with the idea of
<em>Construct 3</em>. I got some sketches, wrote some early drafts, cleaned up the implementation of
Construct’s core… but it’s remained a dream. A couple of months ago I decided I’d back-port a
nice feature from Construct 3, called
<a href="https://github.com/construct/construct/commit/969e5685ce7251af49c9e267a732b63bcea4e278">this expressions</a>,
and it has rekindled my interest in the library.</p>
<h2 id="this-expressions"><code class="language-plaintext highlighter-rouge">This</code> Expressions</h2>
<p>One of the goals of Construct 3 was to generate efficient (C/Python) code from construct definitions.
It even worked, to some extent: for instance, see <a href="http://sebulbasvn.googlecode.com/svn/trunk/ccon/test.py">this snippet</a>
that automatically generates <a href="http://sebulbasvn.googlecode.com/svn/trunk/ccon/moshe.c">this C-code</a>.
I had no recollection of this <a href="https://github.com/MostAwesomeDude/construct/pull/20#issuecomment-5727638">until I discovered it today</a>.
Funny.</p>
<p>Anyway, Construct 2 uses <code class="language-plaintext highlighter-rouge">lambda</code> functions to represent dependencies between constructs, e.g.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">s</span> <span class="o">=</span> <span class="n">Struct</span><span class="p">(</span><span class="s">"LV"</span><span class="p">,</span>
<span class="n">UBInt8</span><span class="p">(</span><span class="s">"length"</span><span class="p">),</span>
<span class="n">Bytes</span><span class="p">(</span><span class="s">"value"</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">ctx</span><span class="p">:</span> <span class="n">ctx</span><span class="p">[</span><span class="s">"length"</span><span class="p">]),</span>
<span class="p">)</span></code></pre></figure>
<p>and since this is the case, it’s impossible to translate dependencies to C. So I’ve created the
<code class="language-plaintext highlighter-rouge">this</code> object, which essentially <strong>builds</strong> an expression tree from native Python expressions.
In order to evaluate the expression, you simply invoke it with a context:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> from construct import this
>>> this.x * 2 + 3
((this.x * 2) + 3)
>>> (this.x * 2 + 3)({"x":7})
17</code></pre></figure>
<p>So now we can replace all <code class="language-plaintext highlighter-rouge">lambda ctx: ctx["foo"]</code> by the more succinct and readble <code class="language-plaintext highlighter-rouge">this.foo</code> –
the benefits are visually clear – but they go even deeper than this: since we’re no longer
dealing with <strong>black-box</strong> lambda functions, we can drill down into them and generate the
appropriate (static) code.</p>
<h2 id="construct-25">Construct 2.5</h2>
<p>I had to do some Construct work recently, and I missed the conciseness of <code class="language-plaintext highlighter-rouge">this</code> expressions,
so I took the time to back-port them to Construct 2. I sent a pull request to Corbin, but he’s
a bit too busy to maintain the library on a regular basis now, so he’s created a github
<em>organization</em> and <a href="https://github.com/construct/construct">moved the repository there</a>;
this is where Construct will be developed from now on.</p>
<p>Tinkering with the old code again got me sentimental, and I started to do some long-awaited
maintenance. I plan to release version 2.5 (note the dramatic shift from 2.06 to 2.5) in the summer
(say August), and here’s the list of planned changes:</p>
<ul>
<li>Adding <code class="language-plaintext highlighter-rouge">this</code> expressions</li>
<li>Dropping <code class="language-plaintext highlighter-rouge">construct.text</code> – it’s always been an experimental feature and it’s achingly
inefficient. If you want to parse <strong>grammars</strong>, you should use
<a href="http://wiki.python.org/moin/LanguageParsing">more adequate tools</a></li>
<li>Adding Python 3 support (based on the work of <a href="https://github.com/MostAwesomeDude/construct/pull/19">Eli Bendersky</a>);
the library will support Python 2.5-3.2, using <code class="language-plaintext highlighter-rouge">six</code></li>
<li>General cleanups and optimizations</li>
<li>Closing the <em>wikispaces</em> site in favor of <em>readthedocs</em></li>
</ul>
<h2 id="construct-3">Construct 3</h2>
<p>The <em>next big thing(TM)</em>, <a href="https://github.com/tomerfiliba/construct3">Construct 3</a>, is still far away.
I’ve got lots of cool ideas, but time is too short (as so is my ability to concentrate on one thing).
Generally, the guiding thought is to <strong>modernize the library</strong> and make it even yet more compact
and efficient, while removing magic along the way. For instance, because <code class="language-plaintext highlighter-rouge">Structs</code> require their
sub-elements to have a name, and due to the fact keyword-arguments in Python are unordered,
all constructs ended up taking a name argument (even though it’s usually meaningless to them,
as in <code class="language-plaintext highlighter-rouge">UBInt8("length")</code>). This has given birth to all sorts of bastards like <code class="language-plaintext highlighter-rouge">Rename</code> and
<code class="language-plaintext highlighter-rouge">Alias</code>; from now on, it’ll be simpler:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">s</span> <span class="o">=</span> <span class="n">Struct</span><span class="p">(</span>
<span class="n">Member</span><span class="p">(</span><span class="s">"length"</span><span class="p">,</span> <span class="n">UBInt8</span><span class="p">),</span>
<span class="n">Member</span><span class="p">(</span><span class="s">"value"</span><span class="p">,</span> <span class="n">Bytes</span><span class="p">(</span><span class="n">this</span><span class="p">.</span><span class="n">length</span><span class="p">)),</span>
<span class="p">)</span></code></pre></figure>
<p>A second issue is, laying the grounds for <strong>code generation</strong>, thus converting all dependencies
to use <code class="language-plaintext highlighter-rouge">this</code> expressions, and perhaps even limiting the power of <code class="language-plaintext highlighter-rouge">Adapters</code>. Or at least,
making a clear distinction between the constructs that can be turned into code and those that can’t.</p>
<p>And last but not least, I want Construct 3 to come with a <strong>designer</strong>, where you would drag-and-drop
constructs, group them in “boxes”, connect them to each other (instead of <code class="language-plaintext highlighter-rouge">this.length</code>, you’d
connect the Bytes’ <code class="language-plaintext highlighter-rouge">count</code> field to the source construct), etc. And most importantly, you could
try it out live on a data sample, and see how it breaks it up. I made this sketch here (click to
download the PowerPoint slides) to demonstrate how it might look:</p>
<p><a href="/static/res/2012-05-16-sketch.ppt">
<img src="/static/res/2012-05-16-designer.png" title="Construct Designer" width="580px" />
</a></p>
<p>I think it would be very powerful.</p>
<h2 id="closing-words">Closing Words</h2>
<p>I’m mainly writing this post to inform everyone of Construct’s <a href="https://github.com/construct/construct">new repository</a>,
and to <strong>ask for feedback</strong> on the plans for v2.5… Any thoughts, requests, comments would be
appreciated. Also, if anyone wants to join in (especially with Construct 3 ambitious plans) –
feel free to contact me.</p>
Introducing Plumbum - Shell Combinators2012-05-12T00:00:00+00:00https://tomerfiliba.github.com/blog/Plumbum<p>It’s been a while since I last blogged… sorry! Had a midterm exam, a seminar project to deliver
(an <code class="language-plaintext highlighter-rouge">O(n^3)</code> parser for <a href="http://acl.ldc.upenn.edu/J/J95/J95-4002.pdf">Tree Insertion Grammar</a>),
the routine family festivities of Passover, and this new thing, <em>Plumbum</em>, that has been keeping
my mind overclocked while I should have been studying for exams and writing seminar papers.
But now that it’s out, it’s about time I write a little on it.</p>
<p>So <a href="http://plumbum.readthedocs.org/">Plumbum</a> is something I’ve toyed with for quite some time now.
In almost any sort of project, there comes a time when you have to write this really simple,
5-liner shell script, just to build your project’s artifacts, maybe also upload them to PyPI
or sourceforge (using <code class="language-plaintext highlighter-rouge">rsync</code>), and while you’re at it, why not build the project’s documentation
as well. Oh, and don’t forget to run regression tests before all that, and it would be wise to
also handle some basic command-line options, as you might want to skip uploading to PyPI at times,
or perhaps build on different versions of python… and then you end up with this <strong>monstrous
shell script</strong> that you wrote (or worse, some former employee wrote) and never want to lay
your eyes on again… At this point to begin hating yourself for not doing it in python,
to begin with.</p>
<p>But then again, how do you translate <code class="language-plaintext highlighter-rouge">find . -name "*.pyc" | xargs rm</code> or <code class="language-plaintext highlighter-rouge">cp */*.py /tmp</code>
to python-speak? That would take quite a few lines and require importing several modules.
Shell scripts tend to be so short and enchanting… How do we bridge the gap?</p>
<p>Plumbum was born to fill this very gap: on the one hand, be pythonic (and be backed by strong
libraries), and on the other, make it all as easy and one-liner-ish by nature: use a <strong>real
programming language</strong>, with a well-behaved object model and high level concepts, to achieve what
you’d normally do in a shell script, retaining the same expressive power and wrist-handiness
of the shell. I call it <strong>shell combinators</strong>, as it’s a pythonic way to mimic shell syntax.</p>
<p>The library is actually a collection of utilities that I wrote for several separate projects,
and had never got to polish them. Plumbum consolidates them into a single, production-grade
framework. The library provides <strong>local and remote</strong> program execution, with support for <strong>piping</strong>
and <strong>IO redirection</strong>; local and remote file-system <strong>paths abstraction</strong>; a programmatic
<strong>command-line interface (CLI) toolkit</strong>, and numerous other utilities.</p>
<p>For instance,</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">plumbum</span> <span class="kn">import</span> <span class="n">local</span>
<span class="kn">from</span> <span class="nn">plumbum.cmd</span> <span class="kn">import</span> <span class="n">wc</span><span class="p">,</span> <span class="n">ls</span><span class="p">,</span> <span class="n">echo</span><span class="p">,</span> <span class="n">grep</span>
<span class="kn">from</span> <span class="nn">plumbum.utils</span> <span class="kn">import</span> <span class="n">copy</span><span class="p">,</span> <span class="n">delete</span>
<span class="n">delete</span><span class="p">(</span><span class="n">local</span><span class="p">.</span><span class="n">cwd</span> <span class="o">//</span> <span class="s">"*/*.pyc"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">fn</span> <span class="ow">in</span> <span class="n">local</span><span class="p">.</span><span class="n">cwd</span> <span class="o">/</span> <span class="s">"src"</span> <span class="o">//</span> <span class="s">"*.py"</span><span class="p">:</span>
<span class="k">print</span> <span class="n">wc</span><span class="p">(</span><span class="s">"-l"</span><span class="p">,</span> <span class="n">fn</span><span class="p">)</span>
<span class="n">num_of_src_lines</span> <span class="o">=</span> <span class="p">(</span><span class="n">ls</span><span class="p">[</span><span class="s">"-l"</span><span class="p">]</span> <span class="o">|</span> <span class="n">grep</span><span class="p">[</span><span class="s">"</span><span class="se">\\</span><span class="s">.py"</span><span class="p">]</span> <span class="o">|</span> <span class="n">wc</span><span class="p">[</span><span class="s">"-l"</span><span class="p">])()</span>
<span class="p">(</span><span class="n">echo</span><span class="p">[</span><span class="s">"1"</span><span class="p">]</span> <span class="o">></span> <span class="s">"/proc/sys/net/ipv4/ip_forward"</span><span class="p">)()</span></code></pre></figure>
<p>There’s a <a href="http://plumbum.readthedocs.org/">short cheat-sheet as well as extensive documentation</a>
on the project’s site, but at this point I’d like to elaborate a bit on the
<a href="http://plumbum.readthedocs.org/en/latest/cli.html">CLI toolkit</a>, as I think it deserves some more
attention.</p>
<p>The approach of <code class="language-plaintext highlighter-rouge">optparse</code> / <code class="language-plaintext highlighter-rouge">argparse</code> and similar libraries is to build a parser object
and populate it with options in an imperative manner, which I dislike. Plumbum’s CLI toolkit offers
a more <strong>declarative</strong> yet programmatic alternative: An application is defined as a class that
derives from <code class="language-plaintext highlighter-rouge">cli.Application</code>; it may define a <code class="language-plaintext highlighter-rouge">main()</code> method, which serves as the
“entry point” of the application, and any number of <strong>switch-methods</strong>, meaning, methods that
are invokable from the command-line.</p>
<p>Switch methods are normal methods, decorated by <code class="language-plaintext highlighter-rouge">@switch</code>, that may take either no arguments
or a single one; for each switch given on the command line, the toolkit will invoke the switch
method that binds it. Similarly, the <code class="language-plaintext highlighter-rouge">main()</code> method is invoked after all switches have been
processed, and it takes all the <em>positional arguments</em> (i.e., non-switch arguments) that were
given. And last but not least, there are <em>switch attributes</em>, which are in fact just specialized
versions of switch functions, that store an argument given to the switch in an instance attribute.</p>
<p>So I’ve probably only gotten you confused by the terminology at this point, but actually it’s
much simpler!</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">plumbum</span> <span class="kn">import</span> <span class="n">cli</span>
<span class="k">class</span> <span class="nc">MyHttpServer</span><span class="p">(</span><span class="n">cli</span><span class="p">.</span><span class="n">Application</span><span class="p">):</span>
<span class="n">log_to_file</span> <span class="o">=</span> <span class="n">cli</span><span class="p">.</span><span class="n">SwitchAttr</span><span class="p">(</span><span class="s">"--log-to-file"</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span>
<span class="n">verbose</span> <span class="o">=</span> <span class="n">cli</span><span class="p">.</span><span class="n">Flag</span><span class="p">(</span><span class="s">"-v"</span><span class="p">)</span>
<span class="n">mode</span> <span class="o">=</span> <span class="n">cli</span><span class="p">.</span><span class="n">SwitchAttr</span><span class="p">(</span><span class="s">"--mode"</span><span class="p">,</span> <span class="n">cli</span><span class="p">.</span><span class="n">Set</span><span class="p">(</span><span class="s">"TCP"</span><span class="p">,</span> <span class="s">"UDP"</span><span class="p">),</span> <span class="n">default</span> <span class="o">=</span> <span class="s">"TCP"</span><span class="p">)</span>
<span class="n">port</span> <span class="o">=</span> <span class="n">cli</span><span class="p">.</span><span class="n">SwitchAttr</span><span class="p">(</span><span class="s">"--port"</span><span class="p">,</span> <span class="n">cli</span><span class="p">.</span><span class="n">Range</span><span class="p">(</span><span class="mi">1024</span><span class="p">,</span> <span class="mi">65535</span><span class="p">),</span> <span class="n">default</span> <span class="o">=</span> <span class="mi">8080</span><span class="p">)</span>
<span class="o">@</span><span class="n">switch</span><span class="p">([</span><span class="s">"-l"</span><span class="p">,</span> <span class="s">"--load-config"</span><span class="p">],</span> <span class="n">cli</span><span class="p">.</span><span class="n">ExistingFile</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">load_config</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filename</span><span class="p">):</span>
<span class="s">"""Loads the given config file"""</span>
<span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s">"r"</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_parse_config</span><span class="p">(</span><span class="n">f</span><span class="p">.</span><span class="n">read</span><span class="p">())</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">src</span><span class="p">,</span> <span class="n">dst</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="p">.</span><span class="n">log_to_file</span><span class="p">:</span>
<span class="n">logger</span><span class="p">.</span><span class="n">addHandler</span><span class="p">(</span><span class="n">FileHandler</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">log_to_file</span><span class="p">))</span>
<span class="n">logger</span><span class="p">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="p">.</span><span class="n">DEBUG</span> <span class="k">if</span> <span class="bp">self</span><span class="p">.</span><span class="n">verbose</span> <span class="k">else</span> <span class="n">logging</span><span class="p">.</span><span class="n">WARNING</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="n">MyHttpServer</span><span class="p">.</span><span class="n">run</span><span class="p">()</span></code></pre></figure>
<p>There, I think it’s a lovely example of the expressive power of the CLI toolkit and Plumbum in
general. I hope you’ll give it a try, and may you never have to write shell scripts again!</p>
Solving Systems of Linear Equations2012-03-25T00:00:00+00:00https://tomerfiliba.github.com/blog/Linear-System-Solver<p>Yet another university-related post, but I really enjoyed it so I thought I’d share: for a GUI-
workshop I’m taking, we are given GUI-layout constraints as a system of linear equations, which
we need to satisfy. To make life more interesting, some constraints are constant while some are
parametric. There’s no magic here, just some linear algebra combined with Python’s overloadable
nature to produce a nice and compact solver for these linear systems.</p>
<p>Naturally, you present your system as a <em>coefficient matrix</em>, which is Gauss-eliminated and then
“solved” for a given set of variables. I took the Gauss-Jordan elimination code from
<a href="http://elonen.iki.fi/code/misc-notes/python-gaussj/index.html">Jarno Elonen</a> and modified it to
support MxN matrices (not necessarily square). Here’s a simple example of elimination:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> m = Matrix([1,2,4,2], [3,7,6,8], [2,2,2,9])
>>> print m.eliminate()
( 1.00 -0.00 0.00 6.00)
( 0.00 1.00 -0.00 -1.00)
( 0.00 0.00 1.00 -0.50)</code></pre></figure>
<p>But of course a reduced row-echelon matrix is not our final goal - we want to get the variable
<em>assignments</em>. For this, we have <code class="language-plaintext highlighter-rouge">solve()</code>:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> sol = solve(m, ["x", "y", "z"])
>>> print sol
{'y': -1.0000000000000004, 'x': 6.0, 'z': -0.4999999999999998}</code></pre></figure>
<p>Modulo precision errors, we get <code class="language-plaintext highlighter-rouge">x = 6</code>, <code class="language-plaintext highlighter-rouge">y = -1</code> and <code class="language-plaintext highlighter-rouge">z = -0.5</code>. If we have more equations than
variables, the “extraneous” equations must be linear combinations of previous ones, or a
contradiction will result. But what if we have less equations than variables? It means we have
some <em>degrees of freedom</em>… how would we handle that? It’s actually simple - instead of being
resolved to constant values, variables will be assigned <em>dependent expressions</em>:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> m2 = Matrix([1,2,4,2], [3,7,6,8])
>>> print m2.eliminate()
( 1.00 0.00 16.00 -2.00)
( 0.00 1.00 -6.00 2.00)
>>>
>>>
>>> sol = solve(m2, ["x", "y", "z"])
>>> print sol
{'y': <BinExpr (2.0 - (-6.0 * z))>, 'x': <BinExpr (-2.0 - (16.0 * z))>,
'z': <FreeVar z>}</code></pre></figure>
<p>As you can see now, <code class="language-plaintext highlighter-rouge">z</code> is a <em>free variable</em> and <code class="language-plaintext highlighter-rouge">x</code> and <code class="language-plaintext highlighter-rouge">y</code> are <em>dependent</em> on it. Of course more
than one variable may be free and some variables may be independent of free variables. Once a value
for <code class="language-plaintext highlighter-rouge">z</code> is known, we can “fully evaluate” the dependent expressions:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> sol["x"].eval({"z" : 10})
-162.0</code></pre></figure>
<p>The code is available on my <a href="https://github.com/tomerfiliba/tau/blob/850ff76bf59c80cd9eb18100986205276125508e/sadna/linear_solver.py">github page</a>.
Note: all numbers are represented as <code class="language-plaintext highlighter-rouge">Decimals</code>, to avoid loss of precision as much as possible,
and I’m using an “epsilon” value of <code class="language-plaintext highlighter-rouge">1E-20</code> to equate numbers to each other (meaning, <code class="language-plaintext highlighter-rouge">x == y</code> iff
<code class="language-plaintext highlighter-rouge">abs(x-y) <= epsilon</code>).</p>
Easy Syntax for Representing Trees2012-03-07T00:00:00+00:00https://tomerfiliba.github.com/blog/Tree-Syntax<p>I’m working on a parser for <a href="http://en.wikipedia.org/wiki/Tree-adjoining_grammar">Tree Adjoining Grammar (TAG)</a>
for this seminar I’m taking. TAG is an extension of context-free grammar (CFG) that’s more powerful
while still being polynomially-parsable. Anyhow, TAG makes use of “tree production rules” instead of
the “linear” production rules of CFG: instead of <code class="language-plaintext highlighter-rouge">S -> NP VP</code>, you’d have a small tree, the root of
which being <code class="language-plaintext highlighter-rouge">S</code>, having <code class="language-plaintext highlighter-rouge">NP</code> and <code class="language-plaintext highlighter-rouge">VP</code> as its children. Of course these trees can be more than
two-level deep, and they go all sorts of operations such as substitution and adjunction, but that’s
for the parser.</p>
<p>So I needed a compact and (hopefully) readable way to express such trees in my code. At first I
used lots of parenthesis, which was ugly and cumbersome, but then I devised this:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">NonTerminal</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">name</span> <span class="o">=</span> <span class="n">name</span>
<span class="k">def</span> <span class="nf">__sub__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">children</span><span class="p">):</span>
<span class="k">return</span> <span class="n">Tree</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">children</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__pos__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="n">Foot</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">Foot</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nonterm</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">nonterm</span> <span class="o">=</span> <span class="n">nonterm</span>
<span class="k">class</span> <span class="nc">Tree</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">root</span><span class="p">,</span> <span class="n">children</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">root</span> <span class="o">=</span> <span class="n">root</span>
<span class="bp">self</span><span class="p">.</span><span class="n">children</span> <span class="o">=</span> <span class="n">children</span></code></pre></figure>
<p>And here’s how you use it:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">S</span> <span class="o">=</span> <span class="n">NonTerminal</span><span class="p">(</span><span class="s">"S"</span><span class="p">)</span>
<span class="n">t1</span> <span class="o">=</span> <span class="n">S</span><span class="o">-</span><span class="p">[</span><span class="s">"e"</span><span class="p">]</span>
<span class="n">t2</span> <span class="o">=</span> <span class="n">S</span><span class="o">-</span><span class="p">[</span><span class="s">"a"</span><span class="p">,</span> <span class="n">S</span><span class="o">-</span><span class="p">[</span><span class="s">"c"</span><span class="p">,</span> <span class="o">+</span><span class="n">S</span><span class="p">,</span> <span class="s">"d"</span><span class="p">],</span> <span class="s">"b"</span><span class="p">]</span>
<span class="c1"># Which represents the following two trees:
#
# t1: t2:
# S S
# | / | \
# | / | \
# e a S b
# / | \
# / | \
# c S* d
#</span></code></pre></figure>
<p>The peculiar <code class="language-plaintext highlighter-rouge">+S</code> is a way to mark that a leaf node is a <em>foot</em>, which is part of the semantics TAG
(that’s where adjunction takes place). It’s represented in the diagram by the more conventional <code class="language-plaintext highlighter-rouge">S*</code>,
but I had to resort to a unary operator in the code. Anyway, I’m not sure if it’s a recipe or just
a nice trick, but I thought I’d share this.</p>
<p>By the way, you can use both <code class="language-plaintext highlighter-rouge">__sub__</code> and <code class="language-plaintext highlighter-rouge">__neg__</code> to achieve things like <code class="language-plaintext highlighter-rouge">S-----X</code> (i.e.,
more than a single <code class="language-plaintext highlighter-rouge">-</code> sign, to allow for better padding), but I tried to avoid too much ASCII art.
I’d love to hear about other such ideas!</p>
RPyC 3.2.1 Released2012-03-04T00:00:00+00:00https://tomerfiliba.github.com/blog/RPyC-3.2.1<p>This is a maintenance release, fixing some minor bugs and resolving some issues with Python 3
compatibility. More on the <a href="http://rpyc.sourceforge.net/changelog.html">change log</a>.</p>
<p>Python 2.x: it’s <em>advisable</em> to upgrade to this version.</p>
<p>Python 3.x: it’s <strong>highly recommended</strong> to upgrade to this version, as it resolves some core
issues that went under the radar in 3.2.0.</p>
Toying with Context Managers2012-02-27T00:00:00+00:00https://tomerfiliba.github.com/blog/Toying-with-Context-Managers<p>As I promised in the <a href="/blog/Code-Generation-Context-Managers">code-generation using context managers post</a>,
I wanted to review some more, rather surprising, examples where context managers prove handy.
So we all know we can use context managers for resource life-time management: before entering the
<code class="language-plaintext highlighter-rouge">with</code>-suite we allocate (open) the resource, and when we leave the suite, we free (close) it –
but there’s much more to context managers than meets the eye.</p>
<h3 id="stacks-for-fun-and-profit">Stacks, for Fun and Profit</h3>
<p>We’ve seen this one in the code-generation post, but we can generalize this notion a bit. Every time
we enter a <code class="language-plaintext highlighter-rouge">with</code> block, we’d append an element to a list, which we’d pop on exit. Consider this
snippet:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">Stacking</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">stack</span> <span class="o">=</span> <span class="p">[]</span>
<span class="o">@</span><span class="n">contextmanager</span>
<span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">stack</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">"foo"</span><span class="p">)</span>
<span class="k">yield</span> <span class="s">"bar"</span>
<span class="bp">self</span><span class="p">.</span><span class="n">stack</span><span class="p">.</span><span class="n">pop</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span></code></pre></figure>
<p>Nothing fancy, but that’s exactly how the code generation framework works: whenever you enter a new
block, it’s pushed onto the “stack”, and whenever we leave the block, the top-of-stack element is
popped. This is how we automatically take care of indentation, curly-braces, etc.</p>
<p>But of course this pattern is much more useful. Consider something like this:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">Env</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_envstack</span> <span class="o">=</span> <span class="p">[</span><span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">]</span>
<span class="o">@</span><span class="n">contextmanager</span>
<span class="k">def</span> <span class="nf">new</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_envstack</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">_envstack</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">].</span><span class="n">copy</span><span class="p">())</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_envstack</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">].</span><span class="n">update</span><span class="p">(</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">yield</span>
<span class="bp">self</span><span class="p">.</span><span class="n">_envstack</span><span class="p">.</span><span class="n">pop</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">env</span> <span class="o">=</span> <span class="n">Env</span><span class="p">()</span>
<span class="k">with</span> <span class="n">env</span><span class="p">.</span><span class="n">new</span><span class="p">(</span><span class="n">PATH</span> <span class="o">=</span> <span class="s">"/tmp/foo/bin"</span><span class="p">,</span> <span class="n">SHELL</span> <span class="o">=</span> <span class="s">"zsh"</span><span class="p">):</span>
<span class="c1"># processes created here will use the modified environment
</span> <span class="k">pass</span></code></pre></figure>
<p>We can also leverage this concept to run commands as different users. Here’s a sketch:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">cmd</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="s">"ls"</span><span class="p">)</span> <span class="c1"># as current user
</span><span class="k">with</span> <span class="n">cmd</span><span class="p">.</span><span class="n">as_user</span><span class="p">(</span><span class="s">"root"</span><span class="p">):</span>
<span class="n">cmd</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="s">"ls"</span><span class="p">,</span> <span class="s">"/proc"</span><span class="p">)</span> <span class="c1"># as `root`
</span> <span class="k">with</span> <span class="n">cmd</span><span class="p">.</span><span class="n">as_user</span><span class="p">(</span><span class="s">"mallory"</span><span class="p">):</span>
<span class="n">cmd</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="s">"rm"</span><span class="p">,</span> <span class="s">"-rf"</span><span class="p">,</span> <span class="s">"/"</span><span class="p">)</span> <span class="c1"># as `mallory`
</span> <span class="n">cmd</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="s">"cat"</span><span class="p">,</span> <span class="s">"/etc/passwd"</span><span class="p">)</span> <span class="c1"># back as `root` again</span></code></pre></figure>
<p>In essence, every time you want to make local/undoable changes to your state, this pattern proves
helpful.</p>
<h3 id="contextbacks">Contextbacks</h3>
<p><em>Contextbacks</em>, a pun on <em>callbacks</em>, are contexts you pass as arguments to other functions. Many
times it’s useful to pass a before-function and an after-function, and contextbacks are a nice
way to encapsulate this. So instead of this:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">beforefunc</span><span class="p">,</span> <span class="n">afterfunc</span><span class="p">):</span>
<span class="n">beforefunc</span><span class="p">()</span>
<span class="c1"># your code goes here
</span> <span class="n">afterfunc</span><span class="p">()</span></code></pre></figure>
<p>You get this:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">f</span><span class="p">(</span><span class="n">ctxback</span><span class="p">):</span>
<span class="k">with</span> <span class="n">ctxback</span><span class="p">:</span>
<span class="c1"># you code goes here
</span> <span class="k">pass</span></code></pre></figure>
<p>Not a ground-breaking change, but I prefer it as it’s more concise.</p>
<h3 id="lightweight-asynchronism">Lightweight Asynchronism</h3>
<p>This is probably my favorite use case for contexts: you can use them to <em>pipeline</em> or <em>interleave</em>
long-lasting tasks. You can think of contexts are degenerate forms of <a href="http://en.wikipedia.org/wiki/Coroutine">coroutines</a>,
in which you have defined beginning and end, but the middle part is interchangeable, so you can
stick anything into it.</p>
<p>Imagine you need to format a harddisk (using <code class="language-plaintext highlighter-rouge">mkfs</code>) or perform some long network operation (like
copying a huge file over <code class="language-plaintext highlighter-rouge">scp</code>). The pattern is as follows: initiate the operation, wait for it
to finish, and either return or raise an exception. This fits perfectly well with the way contexts
work – with one change – <code class="language-plaintext highlighter-rouge">yield</code> instead of waiting.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">@</span><span class="n">contextmanager</span>
<span class="k">def</span> <span class="nf">format_disk</span><span class="p">(</span><span class="n">devfile</span><span class="p">):</span>
<span class="n">proc</span> <span class="o">=</span> <span class="n">Popen</span><span class="p">([</span><span class="s">"/sbin/mkfs"</span><span class="p">,</span> <span class="s">"-t"</span><span class="p">,</span> <span class="s">"ext3"</span><span class="p">,</span> <span class="n">devfile</span><span class="p">])</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">yield</span>
<span class="k">except</span> <span class="nb">Exception</span><span class="p">:</span>
<span class="n">proc</span><span class="p">.</span><span class="n">kill</span><span class="p">()</span>
<span class="n">stdout</span><span class="p">,</span> <span class="n">stderr</span> <span class="o">=</span> <span class="n">proc</span><span class="p">.</span><span class="n">communicate</span><span class="p">()</span>
<span class="k">if</span> <span class="n">proc</span><span class="p">.</span><span class="n">returncode</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">FormattingFailed</span><span class="p">(</span><span class="n">stderr</span><span class="p">)</span>
<span class="k">with</span> <span class="n">format_disk</span><span class="p">(</span><span class="s">"/dev/sda1"</span><span class="p">),</span> <span class="n">format_disk</span><span class="p">(</span><span class="s">"/dev/sdb1"</span><span class="p">):</span>
<span class="k">pass</span></code></pre></figure>
<p>This saves you time: instead of waiting for the first operation to finish before starting with the
second, you can run them in parallel. The total time would be that of the longest task (not taking
into account the IO bottleneck). You can throw a <code class="language-plaintext highlighter-rouge">yield</code> into any piece of code instead of just
blocking, and use it as a pipelined contextmanager. You can copy three files in parallel, without
resorting to threads or a reactor in the background.</p>
<p>Of course you could improve that by returning an object that reports the progress of the
operation, e.g.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">with</span> <span class="n">format_disk</span><span class="p">(</span><span class="s">"/dev/sda1"</span><span class="p">)</span> <span class="k">as</span> <span class="n">d1</span><span class="p">,</span> <span class="n">format_disk</span><span class="p">(</span><span class="s">"/dev/sdb1"</span><span class="p">)</span> <span class="k">as</span> <span class="n">d2</span><span class="p">:</span>
<span class="k">while</span> <span class="ow">not</span> <span class="n">d1</span><span class="p">.</span><span class="n">is_done</span><span class="p">()</span> <span class="ow">or</span> <span class="ow">not</span> <span class="n">d2</span><span class="p">.</span><span class="n">is_done</span><span class="p">():</span>
<span class="k">print</span> <span class="s">"%s is being formatted, %d%% completed"</span> <span class="o">%</span> <span class="p">(</span><span class="n">d1</span><span class="p">.</span><span class="n">devfile</span><span class="p">,</span>
<span class="n">d1</span><span class="p">.</span><span class="n">get_progress</span><span class="p">())</span>
<span class="k">print</span> <span class="s">"%s is being formatted, %d%% completed"</span> <span class="o">%</span> <span class="p">(</span><span class="n">d2</span><span class="p">.</span><span class="n">devfile</span><span class="p">,</span>
<span class="n">d2</span><span class="p">.</span><span class="n">get_progress</span><span class="p">())</span></code></pre></figure>
<p>And voila! You have a thread-less, light-weight asynchronous framework at hand… a bit like
using a reactor, but without rewriting your code.</p>
<p>And last, if you can’t tweak the blocking parts of the code (e.g., third party libraries), you can
use the “defer to thread” or “defer to process” approach, a la <a href="http://twistedmatrix.com/trac/">twisted</a>:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">@</span><span class="n">contextmanager</span>
<span class="k">def</span> <span class="nf">defer_to_thread</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="n">thd</span> <span class="o">=</span> <span class="n">Thread</span><span class="p">(</span><span class="n">target</span> <span class="o">=</span> <span class="n">func</span><span class="p">)</span> <span class="c1"># it would be smarter to use a
</span> <span class="n">thd</span><span class="p">.</span><span class="n">start</span><span class="p">()</span> <span class="c1"># thread-pool
</span> <span class="k">yield</span> <span class="n">thd</span>
<span class="n">thd</span><span class="p">.</span><span class="n">join</span><span class="p">()</span>
<span class="k">with</span> <span class="n">defer_to_thread</span><span class="p">(</span><span class="n">task1</span><span class="p">),</span> <span class="n">defer_to_thread</span><span class="p">(</span><span class="n">task2</span><span class="p">),</span> <span class="n">defer_to_thread</span><span class="p">(</span><span class="n">task3</span><span class="p">):</span>
<span class="c1"># do something else in the meanwhile
</span> <span class="k">pass</span></code></pre></figure>
<p>So that’s all I had in mind. If you have other unorthodox use cases for contexts, I’d love to
hear about them!</p>
Wizard Dialog Toolkit2012-02-11T00:00:00+00:00https://tomerfiliba.github.com/blog/Dialog-Toolkit<p>Following my <a href="/blog/Deducible-UI">Deducible UI post</a>, and following some of the
criticism it had received, I’d like to share something I’ve been working on (read: <em>experimenting with</em>)
at my work place. You see, we have some “interactive wizards” that storage admins use to connect
storage arrays to their hosts (say, a DB server). These wizards prompt you with questions like your
what’s your username, the name of the pool/volume, whether it’s an iSCSI or a Fiber Channel
connection, etc., and then they go and perform what you’ve asked for.</p>
<p>These wizards operate in a terminal environment, but we’ve had thoughts to make GUI/web versions
of them. This would be a considerable effort with the current design. Another issue they currently
have is the mixing of “business logic” and presentation together. For instance, the code that scans the
devices attached to your host also prints ANSI-colored messages or reports its progress. All in
all it works fine, but there’s lots of room for improvement.</p>
<p>I began to investigate this corner a month or two ago. The initial observation was that such wizards
have a pretty rigid and repetitive structure, thus we can find some abstraction or a “toolkit” for
“expressing” wizards more compactly. This has also led to the realization that once the business
logic and presentation are separate, there’s no reason to limit ourselves to terminal-based interaction:
our wizard-toolkit could do the plumbing and work with terminals, ncurses, GUIs, web-browsers, etc.
The business logic would remain oblivious, and we could have a nice GUI at zero-cost!</p>
<p>There was also a second issue of <em>styling</em>, i.e., printing text in color, that I wanted to get r
id of. This part was easy: I thought, why not employ the model of HTML and CSS? Let’s separate the
structure (semantics) of the text from its styling. Instead of printing a banner for titles,
we’ll display a <code class="language-plaintext highlighter-rouge">Title</code> object, whose exact appearance is determined by a “style sheet” (a class,
of course, not actually a text document).</p>
<p>For instance, when we’re using a color-enabled terminal, the title would be printed in bold and
followed by an empty line; but if our terminal is color-blind, we’ll render the text centered and
surrounded by <code class="language-plaintext highlighter-rouge">=</code> marks. Another example is error-handling: instead of printing error message
in red every time, we’ll display an <code class="language-plaintext highlighter-rouge">Error</code> object; on a terminal, this would be rendered as
red text, but when running in a GUI, rendering this object would pop up a message box.
I’m going to ignore this for the rest of this post, as this is really a side issue.</p>
<p>Now let’s get to expressing wizards, or more generally, <em>dialogs</em>. Following some earlier iterations,
I came to the model where a dialog is a “container object” that’s made of <em>dialog elements</em>. These
elements can be output-only (such as a welcome message), or input-output (such as a message
telling you to choose one of the available options). A dialog is “executed” by a <code class="language-plaintext highlighter-rouge">DialogRunner</code> that
renders it and returns the results gotten from the user. It’s quite important to note that dialog
elements within a single dialog cannot be interdependent – that is, if you want to ask the user
for his name and then show <code class="language-plaintext highlighter-rouge">"Hi there %s"</code> with the user’s name, this has to be done as two,
serial dialogs.</p>
<p>That was quite a lot of babble – let’s see this in action:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">MyApp</span><span class="p">(</span><span class="n">WizardApp</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">iscsi</span> <span class="o">=</span> <span class="n">Option</span><span class="p">(</span><span class="s">"iSCSI"</span><span class="p">)</span>
<span class="n">fc</span> <span class="o">=</span> <span class="n">Option</span><span class="p">(</span><span class="s">"FC"</span><span class="p">)</span>
<span class="n">d</span> <span class="o">=</span> <span class="n">Dialog</span><span class="p">(</span>
<span class="n">Text</span><span class="p">(</span><span class="n">Title</span><span class="p">(</span><span class="s">"hello world"</span><span class="p">)),</span>
<span class="n">Input</span><span class="p">(</span><span class="s">"un"</span><span class="p">,</span> <span class="s">"Username"</span><span class="p">),</span>
<span class="n">Password</span><span class="p">(</span><span class="s">"pw"</span><span class="p">,</span> <span class="s">"Password"</span><span class="p">),</span>
<span class="n">Choice</span><span class="p">(</span><span class="s">"conf"</span><span class="p">,</span> <span class="s">"What do you want to configure?"</span><span class="p">,</span> <span class="p">[</span><span class="n">iscsi</span><span class="p">,</span> <span class="n">fc</span><span class="p">]),</span>
<span class="p">)</span>
<span class="n">res</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">ui</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">d</span><span class="p">)</span>
<span class="k">if</span> <span class="n">res</span><span class="p">[</span><span class="s">"conf"</span><span class="p">]</span> <span class="o">==</span> <span class="n">fc</span><span class="p">:</span>
<span class="bp">self</span><span class="p">.</span><span class="n">config_fc</span><span class="p">(</span><span class="n">res</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="bp">self</span><span class="p">.</span><span class="n">config_iscsi</span><span class="p">(</span><span class="n">res</span><span class="p">)</span>
<span class="p">...</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">==</span> <span class="mi">2</span> <span class="ow">and</span> <span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="s">"--gtk"</span><span class="p">:</span>
<span class="n">MyApp</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">GtkDialogRunner</span><span class="p">(</span><span class="s">"My App"</span><span class="p">))</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">MyApp</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">TerminalDialogRunner</span><span class="p">(</span><span class="n">ANSIRenderer</span><span class="p">))</span></code></pre></figure>
<p>It’s a short and incomplete snippet of course, as I’m only going to cover the big picture. The
<code class="language-plaintext highlighter-rouge">main</code> function creates a dialog object <code class="language-plaintext highlighter-rouge">d</code> and passes it to <code class="language-plaintext highlighter-rouge">ui.run</code>, which “runs” the dialog
and returns the results, as a dictionary. Notice that the dialog elements <code class="language-plaintext highlighter-rouge">Input</code>, <code class="language-plaintext highlighter-rouge">Password</code>
and <code class="language-plaintext highlighter-rouge">Choice</code> all take a first parameter – this is the key under which the result would be placed
in the returned dictionary, e.g., <code class="language-plaintext highlighter-rouge">res["un"]</code> would hold the user-provided user name, and
<code class="language-plaintext highlighter-rouge">res["pw"]</code> would hold the password. <code class="language-plaintext highlighter-rouge">Text</code>, on the other hand, is an output-only element,
so it doesn’t return anything and doesn’t take a key. Long story short, we’re asking the user
to enter some information and choose one of two options, and then continue processing based on the
selected option. At the bottom, we determine how to run the application based on a command-line
switch: if <code class="language-plaintext highlighter-rouge">--gtk</code> is given, we’ll run the dialogs through the <code class="language-plaintext highlighter-rouge">GtkDialogRunner</code>; otherwise,
we’ll use the <code class="language-plaintext highlighter-rouge">TerminalDialogRunner</code>.</p>
<p>And how does it look like? When running on a terminal:</p>
<p><a href="/static/res/2012-02-11-wizard-terminal.png">
<img src="/static/res/2012-02-11-wizard-terminal.png" title="Running in a terminal" width="100%" /></a></p>
<p>And with a single command-line switch, we run as a GTK application:</p>
<p><a href="/static/res/2012-02-11-wizard-gtk.png">
<img src="/static/res/2012-02-11-wizard-gtk.png" title="Running as a GTK application" width="100%" /></a></p>
<p>So of course it’s far from perfect, but then again, it’s a small research project I’ve only put
~15 hours into. It suffers from some of the problems I’ve listed in the deducible UI post, for
instance, the GUI hangs when the business logic performs blocking tasks. This could be solved
by moving to a reactor-based model, but I’ve tried to keep the existing wizard code in tact as
much as possible. A hanging GUI is not nice, but it’s not the end of the world either, and
there are numerous ways to overcome this.</p>
<p>Another benefit this design brings along is the ability to automate testing by using <em>mock dialog
runners</em>. Since our business logic is only exposed to the returned dictionary, we can use a
dialog runner that actually displays nothing and returns a scripted scenario each time. We can even
go further: because our business logic “talks” in high-level primitives like <code class="language-plaintext highlighter-rouge">Choice</code>, we can compute
the Cartesian product of all choices and run through each of them. We can show that we’ve covered
all paths! And we can do this automatically… without people hitting buttons and keeping logs of
their progress.</p>
<p>Anyway, I just wanted to show that it’s feasible. I’m not releasing any code as this project is
currently in very early stages, and it’s something I do at work. Perhaps we’ll open-source it in
the future, if it proves useful enough.</p>
Just Got Me These2012-02-09T00:00:00+00:00https://tomerfiliba.github.com/blog/Books<p>Hurrah! I just got me these:</p>
<p><a href="http://www.amazon.com/gp/product/0486663523"><img src="http://ecx.images-amazon.com/images/I/41aRgMz4h5L._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" title="Introduction to Topology" width="30%" /></a>
<a href="http://www.amazon.com/gp/product/048646931X"><img src="http://ecx.images-amazon.com/images/I/51Czir%2BiJXL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" title="Elementary Number Theory" width="30%" /></a>
<a href="http://www.amazon.com/gp/product/0486474178"><img src="http://ecx.images-amazon.com/images/I/513qqhyP9cL._BO2,204,203,200_PIsitb-sticker-arrow-click,TopRight,35,-76_AA300_SH20_OU01_.jpg" title="A Book of Abstract Algebra" width="30%" /></a></p>
<p>After the CS secretariat refused to let take math courses as electives (thus forcing me into taking
boring stuff like SQL), I think <a href="http://store.doverpublications.com/by-subject-science-and-mathematics-mathematics.html">Dover</a>
books and I are going to become good friends. It’s not like I have plenty of time to read them,
but even just seeing them on my bookshelf would make me a happier person :)</p>
<p>Next on my wish-list are some books on logics and proof theory, category theory, information theory,
and automata and computability. I’ll get there some day…</p>
WHO HAS CHEEZBURGER?2012-02-03T00:00:00+00:00https://tomerfiliba.github.com/etc/whohas<h2 id="too-much-wireshark">Too much Wireshark…</h2>
<p><a href="/static/res/2012-02-03-tellcat.jpg">
<img src="/static/res/2012-02-03-tellcat.jpg" width="100%" title="WHO HAS CHEEZBURGER" /></a></p>
Code Generation using Context Managers2012-01-31T00:00:00+00:00https://tomerfiliba.github.com/blog/Code-Generation-Context-Managers<p>When I was working on <a href="http://agnos.sourceforge.net">Agnos</a>, a cross-language RPC framework,
I had a lot of code-generation to do in a variety of languages (Python, Java, C#, and C++).
At the early stages, I just appended strings to a list. It was quick and dirty, and it’s got the
job done… but that wasn’t enough, of course. I’ve lost the original code already, but it looked
something like this:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">generate_proxy</span><span class="p">(</span><span class="n">typeinfo</span><span class="p">):</span>
<span class="n">lines</span> <span class="o">=</span> <span class="p">[</span>
<span class="s">"public class %sProxy {"</span> <span class="o">%</span> <span class="p">(</span><span class="n">typeinfo</span><span class="p">.</span><span class="n">name</span><span class="p">,),</span>
<span class="s">" private int uid;"</span><span class="p">,</span>
<span class="s">" public %sProxy(int uid) {"</span> <span class="o">%</span> <span class="p">(</span><span class="n">typeinfo</span><span class="p">.</span><span class="n">name</span><span class="p">,),</span>
<span class="s">" this.uid = uid;"</span><span class="p">,</span>
<span class="s">" }"</span><span class="p">,</span>
<span class="p">]</span>
<span class="k">for</span> <span class="n">attr</span> <span class="ow">in</span> <span class="n">typeinfo</span><span class="p">.</span><span class="n">attributes</span><span class="p">:</span>
<span class="k">if</span> <span class="n">attr</span><span class="p">.</span><span class="n">get</span><span class="p">:</span>
<span class="n">lines</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">" public %s get%s() {"</span> <span class="o">%</span> <span class="p">(</span><span class="n">attr</span><span class="p">.</span><span class="n">typename</span><span class="p">,</span>
<span class="n">attr</span><span class="p">.</span><span class="n">name</span><span class="p">,))</span>
<span class="n">lines</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">" // ..."</span><span class="p">)</span>
<span class="n">lines</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">" }"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">attr</span><span class="p">.</span><span class="nb">set</span><span class="p">:</span>
<span class="n">lines</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">" public void set%s(%s value) {"</span> <span class="o">%</span> <span class="p">(</span><span class="n">attr</span><span class="p">.</span><span class="n">name</span><span class="p">,</span>
<span class="n">attr</span><span class="p">.</span><span class="n">typename</span><span class="p">))</span>
<span class="n">lines</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">" // ..."</span><span class="p">)</span>
<span class="n">lines</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">" }"</span><span class="p">)</span>
<span class="n">lines</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="s">"}"</span><span class="p">)</span>
<span class="k">return</span> <span class="n">lines</span>
<span class="c1"># ...
</span>
<span class="n">lines</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">ti</span> <span class="ow">in</span> <span class="n">typeinfos</span><span class="p">:</span>
<span class="n">lines</span><span class="p">.</span><span class="n">extend</span><span class="p">(</span><span class="n">generate_proxy</span><span class="p">(</span><span class="n">m</span><span class="p">,</span> <span class="n">ti</span><span class="p">))</span>
<span class="nb">open</span><span class="p">(</span><span class="s">"foo.java"</span><span class="p">).</span><span class="n">write</span><span class="p">(</span><span class="s">"</span><span class="se">\n</span><span class="s">"</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">lines</span><span class="p">))</span></code></pre></figure>
<p>There are several problems with this approach. First of all, it’s very cumbersome and fragile.
If you forget a comma in the list, two adjacent strings will be concatenated. Also, you have
to do everything yourself, like remembering to close brackets, add semicolons, do the right
indentation, etc. If you wished to split this code into functions, the functions you call would
have to know the indentation level you’re calling them at, or the generated code would be
unreadable. This might seem negligible, but think of languages where indentation matters, like
Python…</p>
<p>The fundamental problem with this approach (and similar ones) is that the <strong>code generator does
not reflect the structure of the generated code</strong>. The two are diseparate, while it’s quite
obvious they should be correlated.</p>
<p>In order to solve this, I turned to <a href="http://www.python.org/dev/peps/pep-0343/">context managers</a>,
a feature I highly value. Conceptually, context managers provide a way to bind beginning-and-end
into a single entity; this is normally used for resource management – but we can leverage this
construction further (I’ll this review in a different post). Here, I’ve used them to create
<em>nested blocks</em>, which allowed me to reflect the structure of the generated code in the code
generator.</p>
<p>Without going into too many details, I defined a <code class="language-plaintext highlighter-rouge">Module</code> class that exposes a <code class="language-plaintext highlighter-rouge">block()</code>
context manager and a <code class="language-plaintext highlighter-rouge">stmt()</code> function. The module holds a “stack” of blocks, and entering a new
block pushes a it onto the stack. Statements are then appended to the topmost block on the stack.
Now, because this framework is “language-aware”, it can encapsulate language-specific details.
For instance, In Java, a block will be indented correctly and wrapped by brackets; in Python,
we’ll append colons to the opening line and indent the block; in C++, if the block begins with
<code class="language-plaintext highlighter-rouge">class</code>, <code class="language-plaintext highlighter-rouge">struct</code> or <code class="language-plaintext highlighter-rouge">enum</code>, we’ll append a trailing semicolon as well.</p>
<p>Here’s how it works:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">m</span> <span class="o">=</span> <span class="n">JavaModule</span><span class="p">()</span>
<span class="n">m</span><span class="p">.</span><span class="n">stmt</span><span class="p">(</span><span class="s">"import foo"</span><span class="p">)</span>
<span class="n">m</span><span class="p">.</span><span class="n">stmt</span><span class="p">(</span><span class="s">"import bar"</span><span class="p">)</span>
<span class="n">m</span><span class="p">.</span><span class="n">sep</span><span class="p">()</span> <span class="c1"># an empty line
</span>
<span class="c1">#...
</span>
<span class="k">def</span> <span class="nf">generate_proxy</span><span class="p">(</span><span class="n">m</span><span class="p">,</span> <span class="n">typeinfo</span><span class="p">):</span>
<span class="n">BLOCK</span> <span class="o">=</span> <span class="n">m</span><span class="p">.</span><span class="n">block</span>
<span class="n">STMT</span> <span class="o">=</span> <span class="n">m</span><span class="p">.</span><span class="n">stmt</span>
<span class="k">with</span> <span class="n">BLOCK</span><span class="p">(</span><span class="s">"public class {0}Proxy"</span><span class="p">,</span> <span class="n">typeinfo</span><span class="p">.</span><span class="n">name</span><span class="p">):</span>
<span class="n">STMT</span><span class="p">(</span><span class="s">"private int uid"</span><span class="p">)</span>
<span class="k">with</span> <span class="n">BLOCK</span><span class="p">(</span><span class="s">"public {0}Proxy(int uid)"</span><span class="p">,</span> <span class="n">typeinfo</span><span class="p">.</span><span class="n">name</span><span class="p">):</span>
<span class="n">STMT</span><span class="p">(</span><span class="s">"this.uid = uid"</span><span class="p">)</span>
<span class="k">for</span> <span class="n">attr</span> <span class="ow">in</span> <span class="n">typeinfo</span><span class="p">.</span><span class="n">attributes</span><span class="p">:</span>
<span class="k">if</span> <span class="n">attr</span><span class="p">.</span><span class="n">get</span><span class="p">:</span>
<span class="k">with</span> <span class="n">BLOCK</span><span class="p">(</span><span class="s">"public {0} get{1}()"</span><span class="p">,</span> <span class="n">attr</span><span class="p">.</span><span class="n">typename</span><span class="p">,</span> <span class="n">attr</span><span class="p">.</span><span class="n">name</span><span class="p">):</span>
<span class="k">pass</span>
<span class="k">if</span> <span class="n">attr</span><span class="p">.</span><span class="nb">set</span><span class="p">:</span>
<span class="k">with</span> <span class="n">BLOCK</span><span class="p">(</span><span class="s">"public void set{0}({1} value)"</span><span class="p">,</span> <span class="n">attr</span><span class="p">.</span><span class="n">name</span><span class="p">,</span>
<span class="n">attr</span><span class="p">.</span><span class="n">typename</span><span class="p">):</span>
<span class="k">pass</span>
<span class="c1"># ...
</span>
<span class="k">for</span> <span class="n">ti</span> <span class="ow">in</span> <span class="n">typeinfos</span><span class="p">:</span>
<span class="n">generate_proxy</span><span class="p">(</span><span class="n">m</span><span class="p">,</span> <span class="n">ti</span><span class="p">)</span>
<span class="n">m</span><span class="p">.</span><span class="n">render_to_file</span><span class="p">(</span><span class="s">"foo.java"</span><span class="p">)</span></code></pre></figure>
<p>So what have we gained?</p>
<ul>
<li>The code is much shorter and more concise</li>
<li>Brackets, semicolons and indentation come out-of-the-box</li>
<li>We’re no longer working with flat lists of strings – we’re working with hierarchal entities
that reflect the structure of the generated code</li>
<li>And the other way around – the structure of the generated code is reflected in the generating
code; nested code is indeed nested inside <code class="language-plaintext highlighter-rouge">BLOCK</code>s, thus the “generatee” and generator are
<strong>visually and semantically correlated</strong>.</li>
<li>We can easily split our code into functions, as the module maintains an internal stack.
If <code class="language-plaintext highlighter-rouge">f()</code> opened a block and called <code class="language-plaintext highlighter-rouge">g()</code> under it, it the code that <code class="language-plaintext highlighter-rouge">g()</code> generates will be
placed and indented correctly.</li>
</ul>
<p>I tried to keep my code quite general, so I haven’t defined all of the target language’s
constructs, but of course we could do that, or at least head in that direction.
It might look like this:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">generate_proxy</span><span class="p">(</span><span class="n">m</span><span class="p">,</span> <span class="n">typeinfo</span><span class="p">):</span>
<span class="k">with</span> <span class="n">m</span><span class="p">.</span><span class="n">CLASS</span><span class="p">(</span><span class="n">typeinfo</span><span class="p">.</span><span class="n">name</span> <span class="o">+</span> <span class="s">"Proxy"</span><span class="p">,</span> <span class="p">[</span><span class="s">"public"</span><span class="p">]):</span>
<span class="n">m</span><span class="p">.</span><span class="n">FIELD</span><span class="p">(</span><span class="s">"int"</span><span class="p">,</span> <span class="s">"uid"</span><span class="p">,</span> <span class="p">[</span><span class="s">"private"</span><span class="p">])</span>
<span class="k">with</span> <span class="n">m</span><span class="p">.</span><span class="n">CTOR</span><span class="p">([</span><span class="s">"int uid"</span><span class="p">]):</span> <span class="c1"># CTOR gets the name of the current class
</span> <span class="n">STMT</span><span class="p">(</span><span class="s">"this.uid = uid"</span><span class="p">)</span></code></pre></figure>
<p>However, there’s a question of where we “put our foot down”, or we’ll end up writing <em>Java
Combinators for Python</em>… and then we’ll be writing Java in Python. No need for that,
thank you very much.</p>
<p>The full source code can be found in the
<a href="https://github.com/tomerfiliba/agnos/blob/master/compiler/src/agnos_compiler/langs/clike.py">Agnos repository</a></p>
Deducible UI2012-01-27T00:00:00+00:00https://tomerfiliba.github.com/blog/Deducible-UI<h3 id="a-brief-history">A Brief History</h3>
<p>I like automating things. I don’t like having to reiterate myself: my dream is to always be able
to add only the <strong>necessary amount of information</strong> in order to make something possible.
This is one reason, for instance, why I hate expressions like
<code class="language-plaintext highlighter-rouge">ArrayList<String> x = new ArrayList<String>();</code>… it always makes me feel like I’m talking to a
retard (compiler).</p>
<p>In 2006/7, I wrote some demos for <a href="http://rpyc.sf.net">RPyC</a> to show how easy network-related
tasks become. I chose something rather complex, a chat client, to show that all the code sums
up to a few lines: clients invoke a method on the server, say <code class="language-plaintext highlighter-rouge">broadcast(str)</code>, and the server
then invokes callbacks on all of its clients, sending them the message.</p>
<p>In order to make it usable, I had to write a GUI: I chose Tk, because it comes with python
and is quite simple; I knew there were better toolkits, but my GUI was meant to be basic enough
to be doable in any toolkit. It occurred to me, then, that I wrote ~20 lines show-casing RPyC
and ~100 lines of horrible GUI code, and that something must be really wrong here.
And by <em>here</em> I mean <em>everywhere</em>.</p>
<blockquote>
<p>Note: throughout the article, I’m using the word <em>GUI</em> to mean any <em>interactive user interface</em>,
be it graphical (Qt, GTK, wxWidgets, …) or terminal-based (<code class="language-plaintext highlighter-rouge">curses</code> and the like).
Basically, anything that doesn’t block on a single line of input, like shells.</p>
</blockquote>
<h3 id="gui-designers">GUI Designers</h3>
<p>So you might say, “Dude, just use QtDesigner or something”. A GUI designer lets you visually
place components and makes your life much easier – drag and drop your widgets and double-click on
a button to write its action. Very easy indeed. But I would like to offer a different angle on
the subject: just like the invention of <a href="http://www.youtube.com/watch?v=N0OhXxx7cQg">the teacup</a>
has hindered the technological advance of China, so do GUI designers hinder us from developing
better GUIs. These designers offer a local optimum which we fail to surpass, and this leaves us
with the mediocre UIs and development tools we have today. And get me going about XAML.</p>
<p>Think about it: <strong>you</strong> have to design a GUI. So yeah, it’s kind of simple, but doesn’t that break
DRY? You have the code and you have the GUI – two faces of the same idea. Obviously, one should
be derived from the other.</p>
<p>For the lion’s share of programs, the UI is highly deterministic – there’s some information
that needs to be displayed to- or gotten from the user, and the <em>bindings</em> is trivial.
Consider a login screen: you want to get a username and a password, use them somehow, and proceed
to the next screen. This is a repeating task, and I’d guess that for ~80% of the programs in the
world, it’s easy enough to <strong>automatically deduce</strong> how the UI should look, given the task at hand.
And I’m not talking about machine learning algorithms or designing “families of tasks” – way
simpler than that! Just define a mapping between programmatic primitives and their visual
representation.</p>
<h3 id="deducing-ui">Deducing UI</h3>
<p>The ultimate goal is to take “pure code”, unaware of UI, and by adding the necessary metadata,
be able to automatically create (“deduce”) a GUI for it. In fact, I’d like to expose programmatic
APIs to a human – completely interchangable programmatic- and human- interfaces. Think how cool it
could be to import <em>Adobe Photoshop</em> and run a directory full of pictures through its filters,
instead of doing so through the UI… without Adobe having to write a separate GUI and
programming toolkit.</p>
<p>The UI needn’t be an eye-candy, at least at the beginning; <strong>it just has to be good-enough</strong>.
It won’t work for games or complex applications like Office, but for it would be just fine for a
chat client. Let’s assume the following mapping:</p>
<ol>
<li>An object is represented by a window</li>
<li>Read-only instance attributes are represented as labels</li>
<li>Writable instance attributes are represented as textboxes</li>
<li>Methods are represented by buttons. If a method requires arguments,
it would be preceded by textboxes</li>
</ol>
<p>Of course we could change textboxes and labels to reflect the attribute’s or argument’s type –
<code class="language-plaintext highlighter-rouge">DateTime</code> would be represented by a <code class="language-plaintext highlighter-rouge">DatePicker</code>, <code class="language-plaintext highlighter-rouge">int</code> could be represented by a number box with
up/down arrows, etc. And of course the framework is free to change the mapping however it wants,
to achieve better, more coherent representation. The mapping above is just a rough draft.</p>
<p>Now, instances of a class like the following:</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">class</span> <span class="nc">Person</span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="n">firstName</span><span class="o">;</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="n">lastName</span><span class="o">;</span>
<span class="kd">public</span> <span class="nc">DateTime</span> <span class="n">birthdate</span><span class="o">;</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">dance</span><span class="o">()</span> <span class="o">{...}</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">eat</span><span class="o">(</span><span class="nc">String</span> <span class="n">foodstuff</span><span class="o">)</span> <span class="o">{...}</span>
<span class="o">}</span></code></pre></figure>
<p>would turn into</p>
<p><img src="/static/res/2012-01-27-render.png" title="Created using http://ditaa.org/ditaa/" /></p>
<!--
+-----------------------------------------+
| Person |X|^|_|
+-----------------------------------------+
| firstName: | John | |
| lastname: | Smith | |
| birthdate: | 1-APR-1899 | |
| |
| /-------\ |
| | dance | |
| \-------/ |
| |
| ________ /-----\ |
| |________| | eat | |
| \-----/ |
| |
+-----------------------------------------+
-->
<p>with just a little bit of binding in the form of:</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">class</span> <span class="nc">Main</span> <span class="o">{</span>
<span class="kd">static</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">Main</span><span class="o">(</span><span class="nc">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">Person</span> <span class="n">p</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Person</span><span class="o">(...);</span>
<span class="n">guify</span><span class="o">(</span><span class="n">p</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<p>Straightforward, isn’t it? You can already begin to see the benefits. This framework would obviously
require some sort of decoration (annotations in Java, attributes in C#, …) on which classes and
which class members are to be exposed, and perhaps some extra metadata, like a picture
to show instead of a method’s name, or some layout information – but it’s perfectly doable.
And we can turn it better looking (unlike my beautiful ASCII art example), by using better UI
primitives and a better mapping between objects and their representation; but let’s leave it for now.</p>
<p>I wrote a a simple prototype of this and lo and behold, it actually worked! But when you try to
use it in a real-life applications, the going gets tough: things are updated behind the scenes
(not through our UI framework) and we need to reflect these changes in the UI. For instance,
an element is added to a list via the list’s <code class="language-plaintext highlighter-rouge">add()</code> method - how can the GUI become aware of that?
Well, we can use <a href="http://en.wikipedia.org/wiki/Observer_pattern">observable objects</a>, which the GUI
would observe; so instead of using an <code class="language-plaintext highlighter-rouge">ArrayList<T></code>, you’d simply use an <code class="language-plaintext highlighter-rouge">ObservableArrayList<T></code>.</p>
<p>But creating an observable counterpart for every class is a considerable effort on the framework’s
side, and it breaks software modularity: the framework has to be aware of every 3rd party class
that you wish to expose, or at least allow you to provide the means to expose them.
Another downside of this scheme is that your code becomes <em>aware</em> of the GUI: if we’ve so far
managed to keep our code clean of GUI primitives (we only required some metadata), all of the
sudden you must replace your lists with GUI-observable-lists. Bummer.</p>
<p>Another issue is that using synchronous programming techniques (blocking operations) does not
play well with this model: when does the GUI gets its “runtime”? How can we keep it from freezing?
Who’s providing the entry-point of the program? Does it run in a separate thread?
If so, we risk polluting our code with GUI-related locks (which is countering the whole purpose);
and besides, threads suck and add the incurred complexity is never worth it.
The only feasible option is asynchronous programming (via a reactor) – but requires that
your code be programmed this way, and it’s quite nasty to write such code without proper language
support (e.g., lack of closures, coroutines, etc.).</p>
<h3 id="no-way">“No Way”</h3>
<p>As I said, I’ve been toying with this idea from 2006, and I always get the same response from
colleague programmers: “it would never work”, “it won’t be good enough”, “no one would want to
use it”, “users need their eye-candy”, “I need tight control over the layout”, and what not.
Skeptics galore. My answer is always the same: <strong>you never know what your user wants</strong> – so who
are you to decide? And besides, you’re always to lazy to support proper customization of UI,
so your user must live with your decisions.</p>
<p>Sure, there are books and dissertations about UX, and you’ve read them all; but why not just
provide good-enough defaults, and let the rest be customized by the user? Let the framework
deduce a sane layout for your code – but let’s make everything movable/resizable/dockable.
This way, if it makes more sense to place button X to the left of button Y, the user can do
so himself. And let’s remember the user’s preferences in a file, which we’ll load each time
the application runs. And by “user” I’m also talking about your UX expert – let him/her decide
on a default look (i.e., the preferences file) for the application, which will be shipped with your
product, but the end-user would still be able to move things around. Wouldn’t it be easier?
And if you insist, here’s the place to stick some machine-learning magic, in order to deduce
better UIs by default.</p>
<hr />
<p>So anyhow, I had a working proof-of-concept somewhere, but I think I lost it. It wouldn’t be too
hard to recreate it, but at the moment I’m more concerned with <strong>UI combinators</strong>, which I’ll cover
in a future post. Fully deducing a UI is quite a challenge, as a detailed above, but it’s doable
nonetheless, and the added-value is huge! I’ll get back to it some day, perhaps after I have
better UI combinators… but in the meantime, is there anyone in the audience who’s willing to
pick it up?</p>
Regarding Infix Operators2012-01-25T00:00:00+00:00https://tomerfiliba.github.com/blog/Regarding-Infix-Operators<p>I got some reactions to the <a href="/blog/Infix-Operators">Infix Operators</a> post, and wanted to point
out some things. First of all, I’m not the one who came up with it – it’s a recipe from the
Python Cookbook that’s been posted in 2005. I’m not taking credit for it or anything, I just
said I loved the idea and I adapted the code a little.</p>
<p>Second, regarding coding style or the <em>pythonicity</em> of this scheme – let’s be clear, it’s a hack.
In order for it to work, the two arguments must refuse to support <code class="language-plaintext highlighter-rouge">__or__</code> on <code class="language-plaintext highlighter-rouge">Infix</code> objects,
which means you can’t compose them properly:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> @Infix
... def dot(f, g):
... return lambda *x: f(g(*x))
...
>>> @Infix
... def mul(x, y):
... return x * y
...
>>> def double(x):
... return x * 2
...
>>> f = double |dot| mul # this works ok
>>>
>>> g = mul |dot| double # but this won't
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in __or__
TypeError: mul() takes exactly 2 arguments (1 given)</code></pre></figure>
<p>It’s also magical and unpythonic by nature. You can read in the cookbook comments about a
less-experienced programmer who complained he had to wrap his head around around this. I’d say this
feature is a kin to <em>metaclasses</em>: they are useful (at times), but there are better, more pythonic
ways to do the same without the magic.</p>
<p>So why is it useful? First of all, it’s an interesting pattern, worth more knowing about
than actually using it. But on the more practical side, it could be very useful in
domain-specific languages (DSL), where it could increase your expressiveness. Consider something
like <a href="http://construct.wikispaces.com">Construct</a>, where you define data structures declaratively:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">ipaddr</span> <span class="o">=</span> <span class="n">Struct</span><span class="p">(</span><span class="s">"ipaddr"</span><span class="p">,</span>
<span class="n">UInt8</span><span class="p">(</span><span class="s">"a"</span><span class="p">),</span>
<span class="n">UInt8</span><span class="p">(</span><span class="s">"b"</span><span class="p">),</span>
<span class="n">UInt8</span><span class="p">(</span><span class="s">"c"</span><span class="p">),</span>
<span class="n">UInt8</span><span class="p">(</span><span class="s">"d"</span><span class="p">),</span>
<span class="p">)</span>
<span class="n">lenval</span> <span class="o">=</span> <span class="n">Struct</span><span class="p">(</span><span class="s">"lenval"</span><span class="p">,</span>
<span class="n">UInt8</span><span class="p">(</span><span class="s">"len"</span><span class="p">),</span>
<span class="n">Bytes</span><span class="p">(</span><span class="s">"val"</span><span class="p">,</span> <span class="k">lambda</span> <span class="n">ctx</span><span class="p">:</span> <span class="n">ctx</span><span class="p">.</span><span class="nb">len</span><span class="p">),</span> <span class="c1"># interdependencies: "val" is `len` bytes long
</span><span class="p">)</span></code></pre></figure>
<p>We could replace all sorts of built-in constructs by such “operators”, thus building “data
expressions”. So here’s a very early sketch of what it could look it:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">ipaddr</span> <span class="o">=</span> <span class="n">UInt8</span> <span class="o">|</span><span class="n">seq</span><span class="o">|</span> <span class="n">UInt8</span> <span class="o">|</span><span class="n">seq</span><span class="o">|</span> <span class="n">UInt8</span> <span class="o">|</span><span class="n">seq</span><span class="o">|</span> <span class="n">UInt8</span>
<span class="c1"># or maybe just
</span><span class="n">ipaddr</span> <span class="o">=</span> <span class="n">UInt8</span> <span class="o">|</span><span class="n">repeat</span><span class="o">|</span> <span class="mi">4</span>
<span class="c1"># binding names into the context
</span><span class="n">lenval</span> <span class="o">=</span> <span class="n">UInt8</span> <span class="o">|</span><span class="n">bind_seq</span><span class="p">(</span><span class="s">"len"</span><span class="p">)</span><span class="o">|</span> <span class="n">Bytes</span><span class="p">(</span><span class="n">getctx</span><span class="p">(</span><span class="s">"len"</span><span class="p">))</span></code></pre></figure>
<p>Beware: I just made this up, there’s no solid concept behind it.</p>
<h2 id="see-also">See Also</h2>
<p>If you liked the infix operators idea, have a look at <a href="https://github.com/JulienPalard/Pipe">https://github.com/JulienPalard/Pipe</a>,
which allows for easy functional composition in python.</p>
All Systems are Go2012-01-23T00:00:00+00:00https://tomerfiliba.github.com/blog/All-Systems-Go<p>At last, I finished migrating the old <code class="language-plaintext highlighter-rouge">drupal</code> site to github pages. Everything is now fully
revisioned and statically-generated (using Disqus for comments). Jekyll is so cool! I wrote all
the HTML and forged the stylesheets myself… hope you like it.</p>
<p>Anyhow, I’m happy with the design now, and I’ll start blogging more regularly. I still have
plenty of pages to complete (projects, about-page, etc.), and a few partially-written blogs
posts to polish up, but other generally speaking, we’re up.</p>
<p>If you have any feedback about the design/site, please let me know in the comments below or
via email. Thanks!</p>
Foxx0rz2012-01-22T00:00:00+00:00https://tomerfiliba.github.com/etc/foxx0rz<p>This is <strong>Foxx0rz</strong>, our course’ mascot. He’s written in <code class="language-plaintext highlighter-rouge">C</code> (compiles under <code class="language-plaintext highlighter-rouge">MSVC++ 6</code> to be exact),
and we had it printed on T-shirts… ‘twas fun.</p>
<p><a href="/static/res/2012-01-22-bigfox.png">
<img src="/static/res/2012-01-22-bigfox.png" title="Foxx0rz source code" /></a></p>
<p>Here’s a <a href="/static/res/2012-01-22-foxsrc.c">downloadable version</a></p>
<p>And this is the output:</p>
<p><a href="/static/res/2012-01-22-foxrun.png">
<img src="/static/res/2012-01-22-foxrun.png" width="100%" title="Foxx0rz output" /></a></p>
Infix Operators in Python2012-01-22T00:00:00+00:00https://tomerfiliba.github.com/blog/Infix-Operators<p>As you may already know, there are 3 kinds of operators calling-notations: <strong>prefix</strong> (<code class="language-plaintext highlighter-rouge">+ 3 5</code>),
<strong>infix</strong> (<code class="language-plaintext highlighter-rouge">3 + 5</code>), and <strong>postfix</strong> (<code class="language-plaintext highlighter-rouge">3 5 +</code>). Prefix (as well as postfix) operators are used
in languages like LISP/Scheme, and have the nice property of not requiring parenthesis —
there’s only one way to read an expression like <code class="language-plaintext highlighter-rouge">3 5 + 2 *</code>, unlike <code class="language-plaintext highlighter-rouge">3 + 5 * 2</code>.
On the other hand, it reduces code readability and the locality of operators and their arguments.
This is why we all love infix operators.</p>
<p>Now imagine I have a function, <code class="language-plaintext highlighter-rouge">add(x,y)</code>, and I have an expression like <code class="language-plaintext highlighter-rouge">add(add(add(5,6),7),8)</code>…
wouldn’t it be cool if I could use infix notation here? Sadly though, Python won’t allow you to
define new operators or change how functions take their arguments… but that doesn’t
mean we have to give up!</p>
<p>Haskell, for instance, allows you to define custom operators and set their precedence, as well
as invoking “normal” functions as infix operators. Suppose you have a function <code class="language-plaintext highlighter-rouge">f(x,y)</code> — you
can invoke it like <code class="language-plaintext highlighter-rouge">f 5 6</code> or <code>5 \`f\` 6</code> (using backticks). This allows us to turn
our previous expression, <code class="language-plaintext highlighter-rouge">add(add(add(5,6),7),8)</code>, into <code>5 \`add\` 6 \`add\` 7 \`add\` 8</code>,
which is much more readable. But how can we do this in Python?</p>
<p>Well, there’s this <a href="http://code.activestate.com/recipes/384122-infix-operators/">Cookbook recipe</a>
that provides a very nice way to achieving the same functionality in Python (adapted a little by me):</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">functools</span> <span class="kn">import</span> <span class="n">partial</span>
<span class="k">class</span> <span class="nc">Infix</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">func</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">func</span> <span class="o">=</span> <span class="n">func</span>
<span class="k">def</span> <span class="nf">__or__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">func</span><span class="p">(</span><span class="n">other</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__ror__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
<span class="k">return</span> <span class="n">Infix</span><span class="p">(</span><span class="n">partial</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">func</span><span class="p">,</span> <span class="n">other</span><span class="p">))</span>
<span class="k">def</span> <span class="nf">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">v1</span><span class="p">,</span> <span class="n">v2</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="p">.</span><span class="n">func</span><span class="p">(</span><span class="n">v1</span><span class="p">,</span> <span class="n">v2</span><span class="p">)</span></code></pre></figure>
<p>Using instances of this peculiar class, we can now use a new “syntax” for calling functions as
infix operators:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> @Infix
... def add(x, y):
... return x + y
...
>>> 5 |add| 6
11</code></pre></figure>
<p>Surrounding decorated functions with pipes (bitwise ORs) allows them to take their parameters
infix-ly. Using this, we can do all sorts of cool things:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> instanceof = Infix(isinstance)
>>>
>>> if 5 |instanceof| int:
... print "yes"
...
yes</code></pre></figure>
<p>And even <a href="http://en.wikipedia.org/wiki/Currying">curry</a> functions:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> curry = Infix(partial)
>>>
>>> def f(x, y, z):
... return x + y + z
...
>>> f |curry| 3
<functools.partial object at 0xb7733dec>
>>> g = f |curry| 3 |curry| 4 |curry| 5
>>> g()
12</code></pre></figure>
<p>Ain’t that cool?</p>
לקט ציטוטי מורים2011-12-01T00:00:00+00:00https://tomerfiliba.github.com/etc/leket<p>לקט ציטוטים שאספתי במהלך התיכון.</p>
<p><a href="/static/res/2011-12-01-leket.pdf">לינק ל PDF</a></p>
RPyC Moves to a New Site2011-08-29T00:00:00+00:00https://tomerfiliba.github.com/blog/RPyC-new-site<p>RPyC is in the process of migrating from <a href="http://rpyc.wikidot.com">http://rpyc.wikidot.com</a> to it’s new (and hopefully final)
location at <a href="http://rpyc.sourceforge.net">http://rpyc.sourceforge.net</a>. Wikidot had served us well, and was easy to maintain,
but they started displaying way to many ads and didn’t support <code class="language-plaintext highlighter-rouge">rsync</code> or <code class="language-plaintext highlighter-rouge">SSH</code> access,
which meant I couldn’t upload the generated API reference automatically.</p>
<p>The new site is written entirely in ReST using <a href="http://sphinx.pocoo.org/">sphinx</a> and large parts
of it are auto-generated from docstrings in the source code. It’s all now part of the
<a href="http://http://github.com/tomerfiliba/rpyc">git repository</a>, and I only have to run
<code class="language-plaintext highlighter-rouge">make upload</code> to upload it all up.</p>
Learning Me a Haskell2011-08-17T00:00:00+00:00https://tomerfiliba.github.com/blog/Learning-Haskell<p><em>Phew!</em> Finally the semester’s over (just submitted my last project), and it’s time to clean
up my ever-so-long backlog. Here goes nothing: I’ll start by posting something here,
after this long while of neglect.</p>
<p>As I’m sure you already know, I’m a long-time Pythonista, and I’m confident enough in calling
myself a “native speaker” of that language. Although python is my expertise, I’d say I’m
fluent in most other prominent programming languages (say, C, C++, Java, C#, VB), and with
adequate knowledge of many more. It may seem like a good set of skills, but I have to admit
this brings one to a point of stagnation. There comes a time where everything just looks the same:
you take a glimpse at a new language/platform/other technology and sigh, “oh well,
on the surface it’s different, but underneath it’s all the same sh*t”. You come to the
conclusion that people mostly change the looks-and-feel, but nothing radical could never happen.
And then you skip to the next article.</p>
<p>Then came <a href="http://www.haskell.org/haskellwiki/Haskell">Haskell</a>: a statically-typed, type-inferred,
highly-expressive functional language. I’m not new to functional programming – I’ve programmed
in Scheme, and I even wrote an interpreter for a toy functional-language that I made up in order
to investigate nested-scopes and evaluation models – but heck, <strong>this one’s different</strong>.
A friend of mine nagged me to learn Haskell for quite a time already, but I was deterred by
Haskell’s ugly syntax (it’s not Python a’right) and I kept pushing it for “when I have time”.
It always seemed too academic to me, too impractical.</p>
<p>I think what provoked me into seriously learning Haskell is a pattern called
<a href="http://en.wikipedia.org/wiki/Continuation-passing_style">continuation passing style</a> – I was
just amazed by the extra power, design simplicity, and greater expressiveness that you get with it.
Of course you can implement it in almost any other language, and in fact it was invented in
Scheme (<code class="language-plaintext highlighter-rouge">call/cc</code>), but Haskell, with its purely-mathematical state of mind, just brings
the best out of things. Haskell has also taught me that expressiveness and conciseness are
not virtues to be underestimated: if you can do something in one line instead of four,
you must be generalizing on some deeper concept that you’ve previously missed.
It’s not just looks-and-feel, this time.</p>
<p>But let’s not go astray. I think the most prominent and well-known feature of Haskell is its
excellent type system. I’m sure you’ve heard of <em>static languages</em>, like java or C++,
where every expression has a compile-time type; and I’m sure you’ve heard of
<em>dynamic (duck-typed) languages</em>, where there are only run-time types and no checks are (or can be)
performed prior to running the code. In the first case, you curse the compiler for limiting what
you can do (even when it makes perfect sense) and for being bluntly stupid, requiring you to repeat
yourself over and over (<code class="language-plaintext highlighter-rouge">FooBar x = new FooBar()</code>). And because most type systems are so weak,
you find yourself escaping to “duck-typing” techniques like <code class="language-plaintext highlighter-rouge">void *</code> or <code class="language-plaintext highlighter-rouge">Object</code>, and relying on
run-time casts. In the latter case, you find yourself trying to cover the infinite space of
type permutations that your functions have to handle, and soon you pray for <em>some</em> sort of
validation or restrictiveness.</p>
<p>When you start learning Haskell, you suddenly realize how weak most type systems are, and that
there are (much) better alternatives. It’s like you’ve been suffering from a blurry vision your
whole life, and suddenly you realize you can wear glasses. Has it ever occurred to you that
type systems are equivalent to proof systems
(the <a href="http://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspondence">Curry-Howard isomorphism</a>)?
And that a compiler basically tries to prove your code? You can think of compilation errors as
finding a counter-example to your claim! “You said F takes an integer, but I see you try to call
it with a String”… so of course this is a trivial example, one that even a C compiler catches,
but you can make more complicated claims. And then you realize that type systems and compilers
(“provers”) do matter – a smarter compiler that employs a more powerful type system can prove
or disprove more complicated claims! That’s a non-trivial conclusion that most programmers missed…
type systems don’t have to suck, and they can actually <strong>work for you</strong>, unlike Java.</p>
<p>But what do I know, I’m just learning Haskell now, and besides – this post is but a teaser.
So do yourself a favor and <a href="http://learnyouahaskell.com/chapters">Learn You a Haskell</a>!</p>
Hooking Imports for Fun and Profit2011-06-17T00:00:00+00:00https://tomerfiliba.github.com/blog/Hooking-Imports<p>I really love Python… it’s so hackable that it just calls for hacking, inspiring your
imagination to find ways to stretch its boundaries. This time I decided to investigate into
<a href="http://www.python.org/dev/peps/pep-0302/">import hooks</a>, to add some missing functionality I
wanted to have.</p>
<p>As you probably know, Python uses a <em>flat namespace</em> for packages, that works on a
“first found first served” basis. Packages are simply searched in linear order, as they appear
in <code class="language-plaintext highlighter-rouge">sys.path</code>: if two directories contain a package named <code class="language-plaintext highlighter-rouge">foo</code>, importing <code class="language-plaintext highlighter-rouge">foo</code> will fetch
package in the first directory. This is normally the desired behavior (as it allows you to
override some modules by changing <code class="language-plaintext highlighter-rouge">PYTHONPATH</code>), but it’s also quite limiting.</p>
<p>Consider the <em>nested package namespace</em> used by Java and various other languages (e.g., Haskell),
where packages are normally “deeply nested”, as in <code class="language-plaintext highlighter-rouge">com.sun.foo.bar</code> or <code class="language-plaintext highlighter-rouge">com.ibm.spam.ham</code>.
In Haskell, for instance, packages are normally placed under their appropriate “categories”,
e.g., <code class="language-plaintext highlighter-rouge">Data.Vector</code> or <code class="language-plaintext highlighter-rouge">Control.Monad</code>. When you write a new data type, you’ll probably put
it under <code class="language-plaintext highlighter-rouge">Data</code>, as in <code class="language-plaintext highlighter-rouge">Data.Vector.UberVector</code>.</p>
<p>If we were to use something like that in Python, if would require us to have a single <code class="language-plaintext highlighter-rouge">com/</code>
directory, under which lots of sub-packages must be placed. This might be possible, but it’s
certainly not the “right way”; and besides, it implies that code written by Sun and IBM is
somehow related (after all, they share the <code class="language-plaintext highlighter-rouge">__init__.py</code> file in <code class="language-plaintext highlighter-rouge">com/</code>), which means one may
affect the other (for better or worse :)).</p>
<p>It would make much more sense to have separately-installed packages, e.g.,
<code class="language-plaintext highlighter-rouge">site-packages/com.ibm.spam.ham</code> and <code class="language-plaintext highlighter-rouge">site-packages/com.sun.foo.bar</code>, where each package is
independent of the other. Sure, they might share a common <code class="language-plaintext highlighter-rouge">com</code> prefix – but that’s all.
This is especially useful in corporate environments, where multiple teams share common packages
(which usually get very unoriginal names, say, <code class="language-plaintext highlighter-rouge">common</code>), and name collisions are very likely.
It’s not a joke: at my work-place, we’re know reorganizing our code after such problems. Also,
from a marketing point-of-view, it might make more sense for your customers to
<code class="language-plaintext highlighter-rouge">import mycompany.foobar</code> than just <code class="language-plaintext highlighter-rouge">import foobar</code>.</p>
<p>But most importantly – it’s <em>composable</em>. Nested packages allow you to “inject” your package
into another namespace. Take <code class="language-plaintext highlighter-rouge">twisted</code> for instance: it’s become so large that it had made more
sense to split it up into sub-packages (<code class="language-plaintext highlighter-rouge">twisted.conch</code>, <code class="language-plaintext highlighter-rouge">twisted.news</code>, …), and allow
end users to choose which of them they wish to install. However, since Python wouldn’t let you have
<code class="language-plaintext highlighter-rouge">site-packages/twisted</code> and <code class="language-plaintext highlighter-rouge">site-packages/twisted-conch</code>, they resorted to hacking <code class="language-plaintext highlighter-rouge">distutils</code>
into doing what they want. If nested packages were supported, you would have a core <code class="language-plaintext highlighter-rouge">twisted</code>
package, with separate add-on packages like <code class="language-plaintext highlighter-rouge">twisted-conch</code>. So why not, really?</p>
<p>Enter <a href="http://pypi.python.org/pypi/nimp/">nimp</a> (“nested imports”). Without going into too many
technical details, <code class="language-plaintext highlighter-rouge">nimp</code> is a <em>meta-import hook</em> – it modifies the way <code class="language-plaintext highlighter-rouge">import</code> statements work.
Specifically, it scans <code class="language-plaintext highlighter-rouge">sys.path</code> and “merges” packages that begin with a common prefix into
“logical packages”. For instance, if you have <code class="language-plaintext highlighter-rouge">com-ibm-foo</code> and <code class="language-plaintext highlighter-rouge">com-sun-bar</code> on your <code class="language-plaintext highlighter-rouge">sys.path</code>,
<code class="language-plaintext highlighter-rouge">nimp</code> will create <em>namespace packages</em> for <code class="language-plaintext highlighter-rouge">com</code>, <code class="language-plaintext highlighter-rouge">com.sun</code> and <code class="language-plaintext highlighter-rouge">com.ibm</code>.
This would allow code like <code class="language-plaintext highlighter-rouge">import com.ibm.foo</code> or <code class="language-plaintext highlighter-rouge">from com.sun.bar import vodka</code> to work
transparently. All you need to do is run <code class="language-plaintext highlighter-rouge">import nimp; nimp.install()</code> (you can also put it
in your <code class="language-plaintext highlighter-rouge">site.py</code>, so it would happen every time you run a Python process), and you’re ready to go.</p>
<p>This was the first time I wrote an import hook, and I really liked how I easy it was to change
the import mechanism. So today I had another idea – <strong>lazy imports</strong>. Of course there’s
<a href="http://peak.telecommunity.com/DevCenter/Importing#lazy-imports">PEAK’s lazyModule</a> and this
<a href="http://code.activestate.com/recipes/473888-lazy-module-imports/">quite complicated recipe</a>,
but I thought, why not combine the two. Writing code like <code class="language-plaintext highlighter-rouge">from peak... import lazyModule; foo = lazyModule("foo")</code>
is cumbersome, while the recipe attempts is too make everything lazy.</p>
<p>Instead, I created a module called <code class="language-plaintext highlighter-rouge">__lazy__</code>, that when imported, installs a meta-import hook.
This import hook handles only modules that begin with <code class="language-plaintext highlighter-rouge">__lazy__</code>, so instead of importing them,
it returns an “on-demand-loaded module” (i.e., when you try to access an its attributes).
Using it is really simple:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> from __lazy__ import telnetlib
>>> telnetlib
<OnDemandModule 'telnetlib'>
>>> telnetlib.Telnet # forces loading
<class telnetlib.Telnet at 0x015B08B8>
>>> telnetlib
<module 'telnetlib' from 'C:\Python27\lib\telnetlib.pyc'>
>>> from __lazy__.xml.dom import minidom
>>> minidom
<OnDemandModule 'xml.dom.minidom'>
>>> minidom.parseString # forces loading
<function parseString at 0x01659CF0>
>>> minidom
<module 'xml.dom.minidom' from 'C:\Python27\lib\xml\dom\minidom.pyc'></code></pre></figure>
<p>Note, though, that using <code class="language-plaintext highlighter-rouge">from __lazy__.x.y import z</code> forces the loading of <code class="language-plaintext highlighter-rouge">x</code>, since we use the
dot operator on it. The <code class="language-plaintext highlighter-rouge">from __lazy__ import foo</code> is “truly lazy”</p>
<p>You can get the code of <code class="language-plaintext highlighter-rouge">__lazy__</code> <a href="https://gist.github.com/1030448">here</a>.</p>
Property Classes2011-05-14T00:00:00+00:00https://tomerfiliba.github.com/recipes/Property-Classes<p>Tired of creating properties the old way? Python 3 brings an improvement in the form of
multi-stage properties, i.e.,</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="o">@</span><span class="nb">property</span>
<span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="p">...</span> <span class="c1"># getter
</span>
<span class="o">@</span><span class="n">foo</span><span class="p">.</span><span class="n">setter</span>
<span class="k">def</span> <span class="nf">foo</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="p">...</span> <span class="c1"># setter</span></code></pre></figure>
<p>It still feels very awkward. I can’t say my solution is pure elegance, but I find it cleaner.</p>
<h2 id="code">Code</h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">types</span>
<span class="k">def</span> <span class="nf">property_class</span><span class="p">(</span><span class="n">cls</span><span class="p">):</span>
<span class="n">getter</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="s">"get"</span><span class="p">,</span> <span class="bp">None</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">getter</span><span class="p">,</span> <span class="n">types</span><span class="p">.</span><span class="n">UnboundMethodType</span><span class="p">):</span>
<span class="n">getter</span> <span class="o">=</span> <span class="n">getter</span><span class="p">.</span><span class="n">im_func</span>
<span class="n">setter</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="s">"set"</span><span class="p">,</span> <span class="bp">None</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">setter</span><span class="p">,</span> <span class="n">types</span><span class="p">.</span><span class="n">UnboundMethodType</span><span class="p">):</span>
<span class="n">setter</span> <span class="o">=</span> <span class="n">setter</span><span class="p">.</span><span class="n">im_func</span>
<span class="k">return</span> <span class="nb">property</span><span class="p">(</span><span class="n">getter</span><span class="p">,</span> <span class="n">setter</span><span class="p">)</span></code></pre></figure>
<h2 id="example">Example</h2>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> class Person(object):
... def __init__(self):
... self._age = 17
... @property_class
... class age:
... def get(self):
... return self._age
... def set(self, value):
... self._age = value
...
>>> p = Person()
>>> p.age
17
>>> p.age=19
>>> p.age
19</code></pre></figure>
Python is Messy2011-05-07T00:00:00+00:00https://tomerfiliba.github.com/blog/Getslice<p>A couple of days ago, Rudiger, a user of <strong>RPyC</strong> found a rather surprising bug, that in turn
revealed just how gruesome python’s inner workings are. Rudiger was working with two machines,
one 32 bit and the other 64 bit, and one machine had a <code class="language-plaintext highlighter-rouge">netref</code> to a remote list. He then tried
to execute something as simple as <code class="language-plaintext highlighter-rouge">mylist[1:]</code>, which to everyone’s surprise threw a very
peculiar exception: <code class="language-plaintext highlighter-rouge">OverflowError: Python int too large to convert to C long</code>.</p>
<p>At first, it seemed that the exception originated from the server side, but further investigation
showed it actually originated from the client side, propagated to the server side, and then back
to the client side: very weird indeed. I recreated the scenario with two of my machines,
and popped up <em>wireshark</em> to see exactly what was going on there. The last packet before the
“resonating” exception seemed to be invoking <code class="language-plaintext highlighter-rouge">__getslice__</code> on the client-side list object,
passing it 1 as the start index and 9223372036854775807 as the stop index. Where the heck is
that number coming from? I added lots of debug prints and what not, but that strange number
kept appearing there, and it was obviously not my code that placed it there.</p>
<p>A day later, the answer finally stroke me: 9223372036854775807 is actually <code class="language-plaintext highlighter-rouge">0x7fffffffffffffff</code>,
which is <code class="language-plaintext highlighter-rouge">sys.maxint</code> on 64-bit machines. From that point on, the solution was simple, but it had
revealed a nasty implementation-detail of CPython 2.xx. You see, when getting a slice of an object,
two methods come to play. The first, deprecated, method is <code class="language-plaintext highlighter-rouge">__getslice__</code>, which simply takes
two arguments for start and stop. The second, recommended method, is <code class="language-plaintext highlighter-rouge">__getitem__</code> which accepts
a <code class="language-plaintext highlighter-rouge">slice</code> object instead of an integer. Sadly, <code class="language-plaintext highlighter-rouge"><type list></code> has both, which are reflected on
the <code class="language-plaintext highlighter-rouge">netref</code> proxy, which causes this rather surprising behavior:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> class Foo(object):
... def __getitem__(self, x):
... print "getitem", x
... def __getslice__(self, *args):
... print "getslice", args
...
>>>
>>> y=Foo()
>>> y[7]
getitem 7
>>> y[7:8]
getslice (7, 8)
>>> y[7:]
getslice (7, 2147483647)
>>> y[7:None]
getitem slice(7, None, None)</code></pre></figure>
<p>As you can see <code class="language-plaintext highlighter-rouge">y[7:]</code> invokes <code class="language-plaintext highlighter-rouge">__getslice__</code> with <code class="language-plaintext highlighter-rouge">sys.maxint</code>, while <code class="language-plaintext highlighter-rouge">y[7:None]</code> (which is
equivalent) invokes <code class="language-plaintext highlighter-rouge">__getitem__</code> with a <code class="language-plaintext highlighter-rouge">slice</code> object… how lame! So when the server (64-bit)
side code attempts to execute <code class="language-plaintext highlighter-rouge">mylist[1:]</code> it invokes <code class="language-plaintext highlighter-rouge">__getslice__</code> on the client (32-bit) side,
passing it the server’s “version” of <code class="language-plaintext highlighter-rouge">sys.maxint</code>, which goes into the C implementation and
blows up. Monkeyballs!</p>
<p>So the simple (and only) solution to this issue is using <code class="language-plaintext highlighter-rouge">mylist[1:None]</code> when working with
different-width platforms… sorry.</p>
ctypes - Pointer from Address2011-04-27T00:00:00+00:00https://tomerfiliba.github.com/recipes/ctype-pointers<p>There are times you need to construct a <code class="language-plaintext highlighter-rouge">ctypes</code> pointer from an integer address you have,
say the <code class="language-plaintext highlighter-rouge">id</code> of a python object. I scratched my head for quite a while until I found out a how
to do it properly (with some help from the <em>stackoverflow</em> guys). Here’s what I got:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">ctypes</span>
<span class="k">def</span> <span class="nf">deref</span><span class="p">(</span><span class="n">addr</span><span class="p">,</span> <span class="n">typ</span><span class="p">):</span>
<span class="k">return</span> <span class="n">ctypes</span><span class="p">.</span><span class="n">cast</span><span class="p">(</span><span class="n">addr</span><span class="p">,</span> <span class="n">ctypes</span><span class="p">.</span><span class="n">POINTER</span><span class="p">(</span><span class="n">typ</span><span class="p">)).</span><span class="n">contents</span></code></pre></figure>
<h1 id="example">Example</h1>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon"># get the ref count of an object (in a very nasty way :)
>>> x="hello world"
>>> deref(id(x), ctypes.c_int)
c_long(1)
>>> y=x
>>> z=x
>>> deref(id(x), ctypes.c_int)
c_long(3)</code></pre></figure>
<p>Some words of caution:</p>
<ul>
<li>I’m relying here on <code class="language-plaintext highlighter-rouge">id</code> returning the address of an object. This is a weak assumption,
but it holds (and is likely to continue to hold) for CPython.</li>
<li>There are APIs for what I showed in the example above… use them instead!</li>
</ul>
<p>I’m using this code to dig into the vicious
[OVERLAPPED](http://msdn.microsoft.com/en-us/library/ms684342(v=vs.85%29.aspx) structure that’s held
inside <a href="http://docs.activestate.com/activepython/2.4/pywin32/PyOVERLAPPED.html">PyOVERLAPPED</a>
for really low-level hacking… Anyway, if anyone finds this recipe useful, feel free to use it.</p>
Copy Function Defaults2007-02-06T00:00:00+00:00https://tomerfiliba.github.com/recipes/Copydefaults<p>Default arguments to functions are evaluated when the function is created, and are stored in the
function object. This cause irritating problems when the default values are in fact mutable,
as only single instance exists:</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> def f(x = []):
... x.append(5)
... print x
...
>>> f()
[5]
>>> f()
[5, 5]
>>> f()
[5, 5, 5]</code></pre></figure>
<p>Sometimes it’s the desired behavior, but mostly it’s a bug. To solve that bug, we use</p>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> def f(x = None):
... if x is None:
... x = []
... x.append(5)
... print x
...
>>> f()
[5]
>>> f()
[5]
>>> f()
[5]</code></pre></figure>
<p>But this idiom adds lots of boilerplate code into functions. The following little decorator solves that problem elegantly.</p>
<h2 id="code">Code</h2>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">copy</span> <span class="kn">import</span> <span class="n">deepcopy</span>
<span class="k">def</span> <span class="nf">copydefaults</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="n">defaults</span> <span class="o">=</span> <span class="n">func</span><span class="p">.</span><span class="n">func_defaults</span>
<span class="k">def</span> <span class="nf">wrapper</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="n">func</span><span class="p">.</span><span class="n">func_defaults</span> <span class="o">=</span> <span class="n">deepcopy</span><span class="p">(</span><span class="n">defaults</span><span class="p">)</span>
<span class="k">return</span> <span class="n">func</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">return</span> <span class="n">wrapper</span></code></pre></figure>
<h2 id="example">Example</h2>
<figure class="highlight"><pre><code class="language-pycon" data-lang="pycon">>>> @copydefaults
... def f(x = []):
... x.append(5)
... print x
...
>>>
>>> f()
[5]
>>> f()
[5]
>>> f()
[5]</code></pre></figure>