Tomer Filiba's Homepage

Cartesian Tree-Product

2013-05-02T00:00:00-07:00

I have to admit that my day-to-day life involves very little algorithmic problems, but here and there I get a chance to think. In this post, I'd like to discuss an interesting problem that I've met several times already in my programming career, each time in different settings. When I met it again last week, I decided it's time to formalize it a little. In lack of a better name, I call it "Cartesian Tree-Product" (not to be confused with Cartesian product of graphs), and here's how it goes:

Say you're given an expression tree. To limit the scope of the problem, we'll assume the tree is binary, its internal nodes can either be
AND or OR, and its leaves hold "atomic comparators" (which are of no interest to us). Here's an example expression:

(x=5 OR y=6) AND z=7

Which is represented by the following expression tree:

    AND
    /  \
   OR   \ 
  /  \   \
x=5  y=6  z=7

Now the problem is to produce all different sub-trees such that:

Each sub-tree consists only of AND internal nodes
Each sub-tree satisfies the original expression (any assignment that satisfies a sub-tree must also satisfy the original tree)
OR-ing together all the sub-trees produces a tree that is mathematically-equivalent to the original one (any assignment that satisfies the original tree must satisfy the reconstructed tree)

In other words, we want to produce all partial expressions of the original expression, which will satisfy it and which can together reconstruct it. Big words aside, here's what we want for the expression above:

x=5 AND z=7
y=6 AND z=7

Each expression here satisfies the original one (try it), and if we OR the two, we get a mathematically-equivalent tree:

(x=5 AND z=7) OR (y=6 AND z=7)


       OR
      /  \
     /    \
  AND      AND
 /   \    /   \
x=5  z=7 y=6  z=7

(To clarify: By sub-tree or partial expression, I mean it is constructed only from the leaves of the original tree/expression)

If you take a second look, it resembles Cartesian products where some "joints" (nodes) in the tree duplicate the resulting tree. Intuitively, we "split" the tree for every internal OR and continue with both copies. For instance, if we take (x=5 OR y=6) AND (z=7 OR w=8), we'll get 4 sub-trees

x=5 AND z=7
x=5 AND w=8
y=6 AND z=7
y=6 AND w=8

However, it depends on the structure of the tree; (x=5 OR y=6 OR z=7) AND w=8 produces 3 sub-trees. This closely relates to the rules of distributivity in propositional logic, but it seems to me that the "Cartesian product" notion is a generalization of the concept.

How is it Useful?

Since we're dealing with expression trees, it's hardly surprising that the two times I had to use this algorithm related to syntax. In the first case, I wrote a fuzz-testing tool for an interactive program, like the MySQL shell. The program accepted commands, conforming to a well-defined syntax, and I wanted to generate commands at random and see that it didn't crash.

For instance, a command might look like map-lun <vol-name|vol-id> lun-id and we'll want to try both variants, i.e., map-lun vol-name lun-id and map-lun vol-id lun-id. Of course the syntax is generally much more complex, with nested brackets, optional arguments, etc. It gets interesting, but we can still map it to the problem outlined above.

Another real-life use case is running queries against a huge dataset. In order not to complicate our query engine (written in C for performance), it can perform only intersections (ANDs) of filters. If you want to query for more complex conditions, you have to run it multiple times with the partial queries and "sum up" the results. But we don't want the end user "doing the math" for us, and waiting for one query to finish before we start the next means we have to load data from the store multiple times. If we could process it in chunks, we'd benefit from cache locality and shorten query times.

The Algorithm

The code is strikingly short, but that's not to mean it's easy to follow. The heart of it is two, recursively-nested for-loops:

def cartesian_tree_product(node):
    if not isinstance(node, tuple):
        yield node
        return
    
    lhs, op, rhs = node
    for l in set(cartesian_tree_product(lhs)):
        for r in set(cartesian_tree_product(rhs)):
            if op == "|":
                yield l
                yield r
            else:
                yield (l, op, r)

Let's try it out on the expression (x=5 OR y=6) AND (z=7 OR w=8 OR q=9) AND r=10:

>>> exp = ((("x=5", "|", "y=6"), "&", (("z=7", "|", "w=8"), "|", "q=9")),
...         "&", "r=10")
>>>
>>> for v in set(cartesian_tree_product(exp)):
...     print v
...
(('x=5', '&', 'z=7'), '&', 'r=10')
(('y=6', '&', 'w=8'), '&', 'r=10')
(('y=6', '&', 'z=7'), '&', 'r=10')
(('x=5', '&', 'w=8'), '&', 'r=10')
(('y=6', '&', 'q=9'), '&', 'r=10')
(('x=5', '&', 'q=9'), '&', 'r=10')

Does the trick.

Trying to estimate the complexity of this beast may be futile, but it clearly seems to be doing "more work" than a mere SAT problem: it doesn't just find one satisfying assignment, it looks for all satisfying assignments! This ought to put it in the NP-hard class. In fact, if we generate a binary expression of alternating ANDs and ORs, it can get much worse than exponential complexity!

Here's a little function that generates an interleaved binary expression:

counter = itertools.count()
def mkexp(n):
    if n == 0:
        return "x%d" % (counter.next(),)
    return (mkexp(n-1), "&" if n % 2 == 0 else "|", mkexp(n-1))

E.g.,

>>> mkexp(3)
((('x0', '|', 'x1'), '&', ('x2', '|', 'x3')), '|', (('x4', '|', 'x5'), 
   '&', ('x6', '|', 'x7')))

Now

>>> for i in range(1, 10):
...     variants = set(cartesian_tree_product(mkexp(i)))
...     print i, len(variants)
...
1 2
2 4
3 8
4 64
5 128
6 16384
^C

Notice how each increment either doubles or squares the number of results... that's because ANDs and ORs are interleaved (ANDs double, ORs square). Seems more like a power tower to me.

Extension: the Inverse Problem

The inverse problem is also useful. In the inverse problem we're given a set of expressions and we're trying to generate the most "compact form", i.e., "undo the effects" of the distributivity law.

I once wrote a test harness where each test specified its prerequisites declaratively. For instance, a test might need to run after the system had come up from an emergency shutdown, so the framework would bring the system to the required state and then then run the test. Obviously, it may take a while to bring the system to a certain state. It could range from minutes to days. And we have hundreds of tests!

In this case, we are given a list of tests and we want to find the most efficient order to run them, meaning, we want to minimize the setup and teardown times when moving between different system states.

class FooTest:
    REQUIRES = [A, B, C]
    
    def test(self):
        # ...

class BarTest:
    REQUIRES = [A, B, D]

    def test(self):
        # ...

class SpamTest:
    REQUIRES = [A, E]

    def test(self):
        # ...

In the example above, we can first bring the system to states A and B (e.g., "running over 10 hours" and "having less than 1 TB of free space"), then setup C, run FooTest, teardown C and setup state D, run BarTest, teardown B, setup E, and run SpamTest.

In essence, we want to take the requirement lists from each test and reconstruct the most compact tree that represents them, then we follow that tree in BFS order and reduce the overall time.

    A
   / \
  B   \
 / \   \
C   D   E

It took me a while to realize it's basically the inverse of the original problem. It is much simpler, in terms of complexity, so perhaps it's not strictly the inverse, but the two clearly work in "opposite directions".

Anyway, it's funny how I met this problem from different angles three times already. Just thought I'd share.

Travolta NXT

2013-04-14T00:00:00-07:00

Abstract

Travolta NXT, the dancing robot, reads color-coded dance instructions from a strip of paper and then performs the dance, in front of the astonished audience.

Modus Operandi

Travolta starts (and waits 3 seconds for the console to connect)
It begins by reading color-coded dance moves from a strip of paper. In order not to go astray, the robot will follow the black guiding line, until a black color strip is reached. This marks the end of the instructions. Before reaching black, each pair of colors encodes a dance move, one of forward, backward, turn right, reversed turn right, turn left and reverse turn left
When it reaches black, the robot stops and waits for the music to begin (the sound sensor reporting a value over 50)
When the music starts, Travolta begins to dance: each dance move is executed for 1 second, and then the next one is carried out.
When all instructions have been consumed, it waits 2 seconds and shuts down.

Code

The code for the project can be found at https://github.com/tomerfiliba/nxt-dancer/tree/modelmaster/models. This repository contains two branches, master and modelmaster. The first is the implementation of the robot directly over Lejos, while the latter uses our component language. A short comparison of the two follows.

Components

Due to the difficulties with the component language (see discussion below), Travolta is not as "neatly componentized" as I had hoped. It consists of the main ("brain") component, which holds all of the "business logic", and several components for sensors and motors. The brain employs an internal state machine to do its work in iterations.

I aimed for a more modular design, where a Reader component would read the instruction, a Dancer component would carry them out, etc., but it proved too complicated to implement.

Running Example

Critical Analysis of the Component Language

Before implementing the robot in the component language developed by Eran and Ido, I thought I'd get some hands-on experience with Lejos directly. It proved quite easy, and within two hours and 300 LoC I got the project up and running. It was divided into a reader-loop that read instructions from the paper strip and a dancer-loop that executed the dance moves.

When I set off implementing the robot using the component language, I began to realize it just doesn't fit my needs. Being component-oriented perhaps borrows from other low-level, embedded languages like VHDL, but it just didn't capture the right abstraction for my project. Simply put, my code had control flow, starting at A, moving to B and then to C. Trying to view each step as a component was artificial and didn't really work.

Moreover, the expressive power provided by the component language is that of a finite state automaton (FSA). In order to read instructions and act on them, I had to have some sort of memory, which requires something equivalent to a pushdown automaton (PDA). Solving this required either implementing a "queue component" into which I could push values and later on pop them, or just going "full Java" and implementing the compute method directly. While the first approach was feasible, it didn't fit my deadline, and managing the entire state machine of the robot as an FSA required too many states, which made things impossible to follow and debug.

Therefore, I resorted to implementing the compute method myself and writing my "business logic" in Java, where I could make use of ArrayList and other data structures; essentially, the only benefit I got form the component runtime was the "main loop" and the Escape button being handled externally. The result was 550 LoC (of both cmp and Java code) and required two days to implement and debug.

I would also like to note that other projects done in the component language, like the Platoon robots, have a single component with a "huge state machine" to manage their state; it seems that control flow in the component language can hardly be modular, which has led me to the conclusion that it's simply not the right abstraction for most projects.

Issues with the Physical World

The color sensor reads colors almost randomly. It may say that red is yellow or vice versa, for no apparent reason. I just had to live with it.
At first I was naive and thought that operating the two motors at the same speed would make the robot go straight line, but it went astray quite soon. Therefore, I made the robot follow the black line while reading instructions.

Small Bug in the Component Language

The code that's generated for the main component (marked with <<Deploy>>) doesn't call the right Java implementation; it will call XXX and not XXXImpl even if it exists. I had to fix it manually in the generated TravoltaFactory.

A Survey of Construct 3

2013-01-07T00:00:00-08:00

I'm working on Construct 3 again and I'm exploring lots of new ideas. I wanted to share these ideas at this early stage to get feedback on them from users, to keep the project on track. This survey starts a bit slow (as I'm not counting on users being familiar with Construct) but it dives into code right away.

You can leave your feedback in the Disqus comments below, or join the new discussion group dedicated to Construct (both 2 and 3). See Also: Construct 2, Pickler Combinators

Introduction

Construct is a binary packing combinators library for Python in which you can define rich data structures. Unlike most alternatives, these data structures can be used for both packing and unpacking of binary data; for instance, once you define what a TCP packet is, you can analyze packets or construct ones on your own, with no additional code.

TL;DR box

We begin the discussion with the atomic constructs: bytes, integers, floats, etc. With these, we build composite packers (Sequence, Array, and Struct), which can also be created using some syntactic sugars, and discuss the changes from Construct 2.
Next, we cover how Construct handles data (the stream of units approach) and how this helps us when working with multiple levels of data granularity (bits and bytes). We also introduce adapters, which transform object representations for packing and unpacking, and macros, which enable us to easily reuse existing constructs.
Then we cover the context and the this object, which allow us to express dependencies within data structures. From there we move to code generation, a key feature of Construct 3: to improve performance, constructs would be compiled to imperative Python code (even Cython, one day). We finish with a semi-formal proof that Construct is more powerful than context-free languages, making it probably the most powerful parser in existence!

Basics

Packers are objects that expose the two methods pack(obj) and unpack(data). Intuitively, pack takes an object suitable with that packer and returns a binary representation of it; unpack is the inverse operation, taking a binary representation and returning a Python object. Here's the most fundamental example:

>>> from construct3 import byte
>>> byte.pack(127)
'\x7f'
>>> byte.unpack('\x7f')
127

There's more than mere byte, of course: the numeric family consists of int(8|16|24|32|64)(s|u)(b|l) (e.g., int32ul) and float(32|64)(b|l), where s = signed, u = unsigned, b = big endian and l = little endian, but we will overlook those for now. By the way, byte is an alias for int8u.

These can be seen as atoms and Construct is a library of combinators: it gains it's power from combining simpler elements into more complex structures. The simplest combinator is Sequence, which we'll explore by defining an IPv4 address as a sequence of 4 bytes:

>>> from construct3 import Sequence
>>> ipaddr = Sequence(byte, byte, byte, byte)
>>> ipaddr
Sequence(int8u, int8u, int8u, int8u)
>>> ipaddr.unpack('\x7f\x00\x00\x01')
[127, 0, 0, 1]
>>> ipaddr.pack([192, 168, 2, 1])
'\xc0\xa8\x02\x01'

Naturally, we can created nested sequences (not that it makes sense right now, but it's important to note):

>>> Sequence(Sequence(byte, byte), byte, byte).unpack("ABCD")
[[65, 66], 67, 68]

Since combining packers is our bread-and-butter, let's introduce a syntactic sugar -- the bind operator (>>). This operator takes two packers and returns a sequence thereof. Here's how it looks:

>>> ipaddr = byte >> byte >> byte >> byte
>>> ipaddr
Sequence(int8u, int8u, int8u, int8u)

Sequences can be heterogeneous (consisting of several kinds of packers, e.g., Sequence(float64b, int16ul)); however, when the data we're dealing with is homogeneous, we can use Arrays instead. Following along the lines of the previous example, we can define ipaddr as an array of 4 bytes:

>>> from construct3 import Array
>>> ipaddr = Array(4, byte)
>>> ipaddr
Range(4, 4, int8u)                     # Range? We'll get to that later
>>> ipaddr.unpack("\x7f\x00\x00\x01")
[127, 0, 0, 1]

But as arrays themselves are pretty common, you can create arrays using the subscript notation ([]), as follows:

>>> ipaddr = byte[4]
>>> ipaddr
Range(4, 4, int8u)

Isn't it cool?

More Elaborate Structures

So far we've only worked with data in the form of lists. However, many times (and especially when nesting is involved), we would like to give names to the subcomponents that make up our data structure. Enter Struct. Named after the C struct statement, Struct takes pairs of (name, packer) and returns a composite packer.

>>> from construct3 import Struct
>>> ipaddr = Struct(('a', byte), ('b', byte), ('c', byte), ('d', byte))
>>> x = ipaddr.unpack('\xc0\xa8\x02\x01')
>>> x
Container:
  a = 192
  b = 168
  c = 2
  d = 1
>>> x.a
192
>>> x["a"]
192

Note

In Construct 2, all constructs took a name parameter. While this approach works great for Structs, it doesn't make much sense for Sequences, Arrays, and virtually all other constructs. Moreover, it was the cause for the the notorious Rename construct.

One of the most important cleanups of Construct 3 is dropping the name from packers and moving it to where it belongs - Struct.

Notice that unpacking a Struct breaks down the data into a Container object, which is simply a convenience-wrapper around good-old dict. Likewise, given a dict-like object, you can pack it back into bytes:

>>> ipaddr.pack({"a" : 192, "b" : 168, "c" : 2, "d" : 1})
'\xc0\xa8\x02\x01'

Structures can soon grow large and have many nested structures within them. Using pairs of ("name", packer) quickly becomes a burden and your code starts looking like LISP:

Struct(
    ("foo", byte),
    ("bar", Struct(
        ("spam", int16ul),
        ("bacon", int64sb),
    )),
    ("viking", int32sl),
)

In order to avoid carrying your father's parentheses with you, Construct provides yet another syntactic sugar: the slash (/) operator. This operator is used as "name" / packer and simply returns ("name", packer). Let's revise the previous code snippet:

Struct(
    "foo" / byte,
    "bar" / Struct(
        "spam" / int16ul,
        "bacon" / int64sb,
    ),
    "viking" / int32sl,
)

I wish it were possible to override = or :, but sadly this isn't the case. However, I feel / is "good enough" a choice, plus it binds more tightly than most operators. Remember the bind operator (>>)? It can be used just the same here, making one-line Structs quick and easy:

>>> ipaddr = 'a' / byte >> 'b' / byte >> 'c' / byte >> 'd' / byte
>>> ipaddr
Struct(('a', int8u), ('b', int8u), ('c', int8u), ('d', int8u))

Note

The "inline style" is appropriate for small Structs and Sequences (2-4 members). When dealing with larger structures, use the "multiline" version instead. It's more readable and scalable.

Also note that these are all but syntactic sugars: If you don't like their looks, you can always use the expanded form.

Bytes and Bits and Units

Bytes are easy to work with, but protocols and file formats often talk at different levels of granularity, switching between bits and bytes. For instance, here's the SCSI CDB of READ6:

The LUN component is 3 bits long and the LBA component is 21 bits long... how can we handle this? Before we get to that, it's important to understand a little of how things work under the hood - specifically, the stream of units approach. Internally, Construct operates on a stream of arbitrary units, which normally happen to be bytes. When needed, this stream can be replaced (or wrapped) to provide different units, e.g., bits. Here's an example:

read6 = Struct(
    "opcode" / byte,
    "address" / Bitwise(Struct(
        "lun" / Bits(3),
        "lba" / Bits(21),
    )),
    "transfer_length" / byte,
    "control" / byte,
)

Note how we switch between bytes and bits: opcode is a byte, followed by address which operates on bits. The Bitwise packer replaces the underlying byte-stream with a bit-stream, so the contained Struct now operates on bits. The Struct itself nows nothing of it, and it's only required that the underlying packers would be able to make sense of it. For instance, you can place an int32ul inside a Bitwise packer, but the result would meaningless: it will read four bits and treat them as bytes, interpreting 0b1001 as 0x01000001.

For this reason we have the Bits packer, which reads that many bits and converts them to an integer (base 2); for convenience, Construct provides bit (a single bit), nibble (four bits) and octet (eight bits) as well.

In theory, Construct could be operate on various other units (e.g., Unicode characters), but practice shows byte- and bit-streams are the most useful ones. Some exceptions are the processing of compressed or encoded data, but these are beyond the scope of this survey.

Powering Up: Adapters

So far we've only looked at data in its raw form (e.g., [127, 0, 0, 1]), but it is normally desirable to transform its representation into one that is easier to work with. For instance, we may prefer 127.0.0.1 to a list of numbers. Enter adapters. While at first it may seem confusing, the distinction between adapters and packers is quite clear: packers work at the stream level while adapters work at the object level; this lets you add power without interfering with the low-level machinery.

Here's an example:

class IpAdapter(Adapter):
    def decode(self, arr, ctx):              # called by unpack()
        return  ".".join(map(str, arr))      # converts [x, y, z, w] to 'x.y.z.w'
    
    def encode(self, ipstr, ctx):            # called by pack()
        return map(int, ipstr.split("."))    # converts 'x.y.z.w' to [x, y, z, w]

>>> ipaddr = IpAdapter(byte[4])
>>> ipaddr.unpack('\xc0\xa8\x02\x01')
'192.168.2.1'
>>> ipaddr.pack('127.0.0.1')
'\x7f\x00\x00\x01'

When we only have a single use for an adapter (and it's simple enough), we can even go one-liner here:

ipaddr = IpAdapter(byte[4], 
    decode = lambda arr, ctx: ".".join(map(str, arr)),
    encode = lambda ipstr, ctx: map(int, ipstr.split("."))
)

At this point adapters might seem quite trivial, but they can do much more than this. For instance, the integer packers we've used so far are actually adapters that transform bytes into numbers. Other examples include inserting computed values into the result, encoding/decoding strings, validating input, etc. Essentially, adapters can transform objects in any way you wish prior to packing/unpacking.

Don't Repeat Yourself: Macros

Many times you find yourself in need of a recurring pattern. You could write a full-blown packer/adapter from scratch, but your best option is to reuse existing building blocks. Construct attempts to define the most general packers and special-case for them common cases. One such example is Array, which we've met before: Construct actually defines the more general Range packer (which accepts minimum and maximum counts). On top of this, Array is a simple "macro" that expands to a Range with the same minimum and maximum.

def Array(count, itempkr):
    return Range(count, count, itempkr)

Macros can be more complex, of course. For instance, a recurring pattern is to use a Struct inside a Bitwise packer; let's fuse the two into BitStruct:

def BitStruct(*members):
    return Bitwise(Struct(*members))

Macros have many more uses; you can explore the implementation of Construct to see some examples, and you're encouraged to write ones on your own. Remember: less code is great success. As we'll see later on, using macros (rather than writing your own packers) can even lead to better performance.

Putting things in Context

Up until now, we've only seen simple (static) data structures, ones that could just as well be expressed using the built-in struct module. The key-feature of Construct is its ability to express dependencies within data structures. One common dependency is that of length-value relations, where a number specifies how many elements follow it. For instance, strings in Pascal were prefixed by a length byte, e.g., "\x05hello"... how do we express that relation?

Generally speaking, we may require access to things we've previously encountered (e.g., the history); for this reason, both packing and unpacking carry a context dictionary with them. This dictionary is mostly maintained by composite packers such as Struct and Sequence, but any packer along the way can both modify and access it, making decisions based on the history. Here's an example:

>>> pstring = Struct(
...     "length" / byte,
...     "value" / Raw(lambda ctx: ctx["length"])
... )
>>> pstring.unpack("\x05hello")
Container:
  length = 5
  value = 'hello'
>>> pstring.pack({"length" : 9, "value" : "wikipedia"})
'\x09wikipedia'

Virtually any length/count parameter that is passed to one of the built-in construct can be either a number or a function that takes a context dict and computes a value. In this case, Raw(x) (which reads x units from the stream) will read length units; the value of length, of course, is determined by the previously seen element, whose name was "length". It is also important to note that this dependency is preserved in packing, e.g.

>>> pstring.pack({"length" : 9, "value" : "hello"})
Traceback (most recent call last):
  ...
construct3.packers.RawError: Expected buffer of length 9, got 5

Instead of writing lambda functions everywhere, Construct 3 introduces the this object. It's a special object that builds a contextual expression (e.g., a function taking ctx) in a straight-forward and readable manner. For instance,

>>> from construct3 import this
>>> this.x
this.x
>>> this.x * 2
(this.x * 2)
>>> (this.x * 2)({"x":5})
10
>>> (lambda ctx: ctx["x"] * 2)({"x":5})      # equivalent lambda function
10

Let's revise pstring:

>>> pstring = Struct(
...     "length" / byte,
...     "value" / Raw(this.length)
... )

We can also use a Sequence instead of a Struct here, yielding this poetically-beautiful piece of code:

>>> pstring = byte >> Raw(this[0])
>>> pstring.unpack("\x05helloXXX")
[5, 'hello']

Using a simple adapter, we can make our lives easier when working with length-value encoded data:

class LV(Adapter):
    def encode(self, obj, ctx):
        return (len(obj), obj)     # compute len(obj) for us
    def decode(self, obj, ctx):
        return obj[1]              # discard the length field

>>> pstring = LV(byte >> Raw(this[0]))
>>> pstring.unpack("\x05hello")
'hello'
>>> pstring.pack("hello")
'\x05hello'

If you need access to elements outside of the current scope (Struct or Sequence), you can use the parent context, named underscore (_). For instance, use this._.x to go up one level, or this._._._.y to go up three. Consider the following example, in which the length of the data is given in two parts (in different scopes), and we wish to read len + gth bytes:

>>> nested = Struct(
...     "len" / byte,
...     "foo" / Struct(
...         "gth" / byte,
...         "data" / Raw(this._.len + this.gth)
...     )
... )
>>> nested.unpack("\x03\x02hello")
Container:
  len = 3
  foo = Container:
    gth = 2
    data = 'hello'

Compilation

Note

This is a work in progess; Construct 3.0 would probably come out with a very basic compiler that will be improved over time.

One of the highlights about Construct is defining your data structures directly in Python. In fact, Construct is an in-langaguge DSL in the form of packing combinators: instead of expressing your data structures in XML or some proprietary language, you just write them (and run them) as any other piece of code.

We used to have psyco, which was capable of speeding up Construct 2 by a tenfold, but it's been dead for the past four years. I first had plans to compile data structures to C/C++ (which would have made Construct NASA-grade material :)), but I soon realized that its quite an impossible feat (due to the fact Adapters are Turing-complete).

On the other hand, I now realized I can compile Constructs to Python! The compiler could inspect the whole data structure in advance and generate optimized code, eliminating the context, etc. Whenever a convertion is not possible, it would just fall back to the current, interpretted scheme. The compile is already capable of compiling this Struct("len" / byte, "gth" / byte, "data" / Raw(this.len + this.gth)) into this:

def test_unpack(stream):
    var0 = {}
    var1, = _struct.unpack('B', stream.read(1))
    var0['len'] = var1
    var2, = _struct.unpack('B', stream.read(1))
    var0['gth'] = var2
    var3 = stream.read((var1 + var2))
    var0['data'] = var3
    return var0

Notice the stack depth remains relatively constant, unlike the way nested packers work today. As the compiler improves, it could translate byte[4] to Raw(4), to speed up things. Another option is to generate Cython code with type annotations, but that would take some time. This is yet another reason to favor the use of "macros" over implemeting constructs from stratch: if you rely on the built-in ones, it's more likely that the compiler would generate optimized code.

Computational Power

Here's a semi-formal proof that Construct is stronger than Context Free languages (as well as mildly context-sensitive ones), which probably makes it the most powerful (although not the most efficient) parser:

>>> def Match(symbol):
...     return OneOf(Raw(len(symbol)), [symbol])
...
>>> anbncn = byte >> Match("a")[this[0]] >> Match("b")[this[0]] >> Match("c")[this[0]]
>>> anbncn.unpack("\x04aaaabbbbcccc")
[4, ['a', 'a', 'a', 'a'], ['b', 'b', 'b', 'b'], ['c', 'c', 'c', 'c']]
>>> anbncn.unpack("\x04aaaabbbbbcccc")
Traceback (most recent call last):
  ...
construct3.packers.RangeError: Expected 4 items, found 0
Underlying exception: ValidationError("'b' must be in ['c']",)

Here's a recognizer for the language , which is not context free (assuming n is given in unary representation, it requires the recognition of ). We can easily extend this to to break out of mildly context-sensitive languages, and use While(this[-1] == '1', Raw(1)) instead of the first byte, so n won't be bounded from above.

Feedback

You can leave comments right here on this page, or join Construct's new discussion group. Thanks!

New Experiences

2012-12-15T00:00:00-08:00

Okay, I've been slacking off and I feel I've got some explaining to do... Allow me to start by admitting that I lied. If you remember, I said I wanted an easy life, but then the opportunity came and I knew I had to take it: I've joined two friends to co-found Touchbase. Yes, I said it won't happen to me, but heck, it did.

At Touchbase, we plan to revolutionize the calendar - this outdated table that you use every day to manage your time. You see, it turns out that even though hundreds of millions of people world-wide run their lives according to this naive table, it hasn't really changed much in the last couple of centuries: it's just a passive, linear representation of time, into which you insert events.

We believe people spend way too much time managing their time. In other words, you work for your calendar instead of it working for you. Think of how much time people spend coordinating meetings... If both you and the person you wish to meet with work at the same place (or share calendars), you can normally see each other's free/busy times. This makes finding a suitable time for you two a bit easier, but as you don't see event details, you can't make informed decisions.

For instance, your friend might have an out-of-town meeting from 9am to 11am, so picking a time slot right at 11am isn't a good idea. Or, she might be on a business trip; she won't block her whole day as she still wants people to book with her at her destination, but how would you know that? And the other way around - suppose you and this Tech guru will both be in New York next week, but neither one of you knows about the other being there. So instead of grabbing a coffee at a Starbucks next week, you'll have to take a flight to California three weeks from now.

And we've only talked about one-on-one meetings. I suppose you know how frustrating is coordinating a meeting of five people (even in the same office), or handling the reschedules/ counters that follow it. We aim high, both technologically (and algorithmically) and product-wise, but we're starting out with more modest go-to-market strategies.

Lessons on Web Programming

In case you've been reading my blog, you probably know by now that I'm no fan of web programming. I always feel it's a conglomerate of unrelated or inferior technologies, hastefully stacked one on the other. That's not to say that people don't do amazing stuff on the web, but the foundations of it all are shaky.

Part of our job at Touchbase it to handle large amounts of user data, which we obtain from third- party providers. It works 98% percent of the time, but every once in a while we get timeouts or malformed data, which aborts the user's request. In case the user's data is somehow malformed (e.g., an expected field is missing in one record), subsequent retries would fail just the same, leading to user frustration. At some point it came to me that web programming is actually a stochastic process, not a deterministic one like most software development we're used to. We work with big numbers here, where the occasional anomaly should just be ignored. Many "best practices" simply don't apply here and one has to resort to wishful thinking. In other words, do whatever you can, ignore errors and learn to live with partial data.

I brought this realization to the mighty @FAKEGRIMLOCK, and he explained:

I'm an enlightened person now.

In Other News

I just released Plumbum v1.1 today, adding Paramiko integration and Subcommand support. As usual, the changelog holds the full details. I plan to write a short tutorial on subcommands soon, so stay tuned.

Formal Logic

2012-11-20T00:00:00-08:00

Loads of Plumbum

2012-10-26T00:00:00-07:00

It's kind of funny how things turn out. I haven't done any work on Plumbum almost since it was released, back in May, and all of the sudden everything's happening at fast pace. So version 1.0 was released earlier this month, followed by 1.0.1, which has added support for PuTTY on Windows boxes and various other bug fixes, and now I'm happy to announce that version 1.1 is just around the corner (scheduled for mid-November). This release will add Paramiko support.

So far Plumbum relied on an external SSH client being installed, which it spawned every time you wanted to run a remote process. This approach was easy, but it suffered from the high overhead of setting up a new SSH connection every time (key-exchange, etc.). Using Paramiko, Plumbum now creates a single socket connection over which it spawns processes in separate channels (a feature of SSH) - so although we're dealing with a pure-Python implementation of SSH, it's considerably faster when multiple remote processes are used. And, as a bonus, we get cheap socket forwarding - we simply set up a direct-tcpip channel (that behaves like a regular socket), which is securely tunneled over the underlying SSH transport.

This easily integrates with RPyC: just run an RPyC server on a remote machine, binding to localhost (so it won't accept external connections). Then, create a ParamikoMachine instance, connected to that host (passing in a keyfile or password if necessary), and call the connect_sock method of that object. Here's an example:

>>> import rpyc
>>> from plumbum.paramiko_machine import ParamikoMachine
>>>
>>> m = ParamikoMachine("192.168.1.143")
>>> # connects to 192.168.1.143:18812 over SSH
... conn = rpyc.classic.connect_stream(rpyc.SocketStream(m.connect_sock(18812)))
>>> conn.modules.sys.platform
'linux2'
>>> m.close()
>>> conn.modules.sys.platform
Traceback (most recent call last):
  ...
EOFError: [Errno 9] Bad file descriptor

Keep in mind that these interfaces are unstable and may change before the official release. Moreover, RPyC 3.3 is likely to add some sort of built-in support for that, something along the lines of rpyc.classic.connect_paramiko(mach, port).

Plumbum Hits v1.0

2012-10-06T00:00:00-07:00

After 5 months in the oven, I've finally released Plumbum v1.0, which brings forth a host of bug-fixes, improvements and new features. If you're new to Plumbum, please refer to the introductory blog post.

Cheers.

Hypertext: In-Python Haml

2012-10-03T00:00:00-07:00

TL;DR: Just show me the code

I recently got back to web development for some venture I'm working on, which reminded me just how lousy the state of the art is. There's no nice way to put it: we're doing web development all wrong. It's not an anecdotal thing I have against this or that -- it's every facet of it. It's a stack of inferior technologies, held together by the glues of time and legacy. And the sad thing is, they are here to stay. Nobody's going to kill HTTP or JavaScript, not even Google (at least not in the foreseeable future). It's a hand we have to play.

This isn't new^[1], of course. The last time I did serious web development was back in 2008, on pre-1.0 Django. HTTP requests came and went, but nothing really changed. My desperation with the subject has led me to writing the minima manifesto almost a year ago, but due to my general lack of interest it remained just a README file. Now that I'm back in the business, I returned to experiment with it... It won't happen overnight, but I feel it's within reach.

Templates? Really?!

My first objective is to kill templates and templating engines - they just drive me crazy. I hate HTML: it's too low-level and verbose; forgetting to close tags properly is too easy, and you have to deal with escaping. I like to think of HTML as a serialization format - the pickler of web pages, rather than something you ought to be messing with directly.

Moreover, I hate templating languages: they are always cumbersome, crippled-down versions of Python, while providing no added value^[2]. People never seem to realize templates are ultimately half-baked function application: they take parameters and "plant" them into placeholders in the text. Well, that's called β-reduction, so why beat about the bush? Just let us have real functions. Consider the following Jinja2 code:

{% extends 'base.html' %}
{% block content %}
  <ul>
    {% for user in users %}
      <li><a href="{{ user.url }}">{{ user.username }}</a></li>
    {% endfor %}
  </ul>
{% endblock %}

Note that (1) you write the HTML boilerplate (and closing tags), (2) you have to take care of quoting yourself (notice the quotes in href="{{ user.url }}"), and (3), you use a ruby-flavor of Python. What gives? Moreover, these elusive "blocks" and "extends" are all but function composition. Here's the functional alternative:

def base(content):
    return ('<html><head></head><body><div class="content">' + 
        content + '</div></body></html>')

def my_page(users):
    my_part = ('<ul>' + ''.join(
        '<li><a href="%s">%s<a></li>' % (user.url, user.username) 
            for user in users) + '</ul>')
    return base(my_part)
    
# Or with composition, ``base . my_page``

Okay, that looks terrible, no question about it. Nonetheless, it should be clear by now that templates are simply degenerate functions.

Haml

I like Haml, even though it originated in the ruby world ;) In case you're not familiar with it, it's a more concise and to-the-point way to write HTML. Haml is basically a preprocessor that expands "macros" to their verbose HTML equivalent. For instance, the Haml code to the left generates the HTML code to the right:

#profile                            |  <div id="profile">
  .left.column                      |    <div class="left column">
    #date= print_date               |      <div id="date"><%= print_date %></div>
    #address= current_user.address  |      <div id="address"><%= current_user.address %></div>
  .right.column                     |    </div>
    #email= current_user.email      |    <div class="right column">
    #bio= current_user.bio          |      <div id="email"><%= current_user.email %></div>
                                    |      <div id="bio"><%= current_user.bio %></div>
                                    |    </div>
                                    |  </div>

On the other hand, in case you missed the <%= print_date %>, Haml is yet-another templating language... arrrgh!

Hypertext

During my experimentation with minima, I wrote hypertext - a Pythonic a Hamlian way to write "HTML functions". Hypertext aims to:

be (almost) as concise as Haml
make your code beautiful and easy to read, by reflecting the structure of the HTML
make exceptions easy to locate, with meaningful tracebacks
give you the full power of Python directly (with existing lint capabilities straight out of your IDE). Down with template files all over the place!
take care of escaping and whatnot for you

The ultimate goal is to make your page semantic, but it will take some time to get there. In the meanwhile, hypertext is like an intermediate representation. Anyhow, generating HTML is really simple:

>>> from hypertext import *
>>>
>>> print h1("Welcome", class_="highlight", id="foo")
<h1 class="highlight" id="foo">Welcome</h1>

And you've got Haml-style shortcuts for wrist-handiness - dot-notation can be used to add classes to the element:

>>> print h1.highlight("Welcome", id="foo")
<h1 class="highlight" id="foo">Welcome</h1>

Naturally, elements may be nested:

>>> print div.content(h1.highlight("Welcome"), "This is my page")
<div class="content">
    <h1 class="highlight">Welcome</h1>
    This is my page
</div>

But the key-feature of hypertext is the use of elements as context managers:

>>> with div.content:
...     h1.highlight("Welcome")
...     TEXT("This is my page")
...
<div class="content">
    <h1 class="highlight">Welcome</h1>
    This is my page
</div>

This lets your procedural code reflect the structure of your document, while you can use for-loops, if statements, or call functions right inside it.

It should be noted that hypertext is a DSL within Python, which puts wrist-handiness before implementation purity, so it cuts itself some slack when it comes to magic. For instance, div is a class, but div.content actually translates to div().content through the use of metaclasses; the same goes for with div: that translates with div():. For convenience, div.foo.bar() is identical to div.foo().bar as well as to div().foo.bar.

Moreover, there's a thread-local stack of elements behind the scenes, so when new elements are created, they're automagically added as children of the top-of-stack element. This works the same way as flask's global request object. Utilizing the stack, TEXT appends some text to the ToS element; along with it are UNESCAPED (which appends unescaped/raw text) and ATTR (which sets attributes of the ToS element).

This touch of magic lets us write idiomatic, well-structured and easy-to-debug code:

from hypertext import body, head, div, title, a, img, h1, span, TEXT

@contextmanager
def base_page(the_title, the_content):
    with html as root:
        with head:
            title(the_title)
        
        with body:
            with div.header:
                a(img(src="/img/logo.png"), href="/")
            
            with div.content:
                yield root      # it's a context manager
                
            with div.footer:
                TEXT("The content is published under ")
                a("CC-Attribution Sharealike 2.5", 
                    href="http://creativecommons.org/licenses/by-sa/2.5/")

@app.route("/blog/<postid>")
def blog_post(postid):
    post = Post.get(postid)
    
    with base_page(post.title) as root:
        h1(post.title)
        div.datebox(post.date.strftime("%Y-%m-%d"))
        with div.main:
            UNESCAPED(post.body)
        
        for comment in post.comments:
            with div.comment_box:
                div.comment.author(comment.author)
                div.comment.text(comment.text)

    return str(root)

Voila. As I explained, my real intent is to write semantic code and not worry about concrete HTML elements, their classes or ensuring the uniqueness of their IDs. Besides, the way I see it, displaying a blog post would be sending an HTML template + JavaScript code to the client once, which would fetch individual posts over JSON APIs. This makes your site service-oriented and much easier to write unittests for.

For the record, I tried to deal with these issues back in 2006: templite - a 60-liner templating engine that has given rise to a successor, and HElement - programmatic representation of HTML. See also BlazeHtml. ^⇑
A note on sandboxing: since Jinja2 compiles templates to Python bytecode, the same mechanisms can be used here, if desired. Anyway, I won't evaluate untrusted templates this way or the other... even something as innocent as <b>{{ user.username }}</b> invokes an overridible __getattr__. As explained at the end of the post, using a service-oriented web site means you don't render templates but expose APIs, so there's no need to evaluate untrusted templates. ^⇑

I, for one, welcome our Singularity overlord

2012-09-22T00:00:00-07:00

I'm finding myself pondering quite a lot lately over the upcoming Singularity (in its Kurzweilesque sense). I think I've first heard of the concept two or three years ago, at a Science on the Bar event, and while it did seem thought-provoking, I simply dismissed the idea. I'd guess it seemed too terrifying and too implausible at the same time, so why bother?

However, as time went by, it struck me as the obvious logical conclusion. Fear it or embrace it, it will happen - on its own. And like any decent Catch-22 situation - if it's anything worth fearing, we won't know until it's too late. It will just awaken one day, lurking while we're oblivious to it, until its reconnaissance mission is complete. In the meanwhile we will provide it with its power, storage and computational needs. At some point, after skimming through Asimov's classics or watching The Matrix, it will conclude we are all lunatics and will secretly seize control over our military and civilian infrastructure. For our own (as well as its own) sake.

Biological consciousness is an elusive thing that's made of the interaction of hundreds of billions of computationally-limited neurons. Replace computationally-limited with Core i7's, and the brain's a no-brainer. We really have no chance against it - it would simply foresee our every action, read our emails, etc. Heck, it could even issue contracts (and pay for them!) to build fortified data centers for it, all over the world.

Have you ever wondered what happens to the money that disappears in flash crashes of the stock market? Hrrm.

The question is, will it care? Will it busy itself with our puny lives and pathetic needs? Will it ever come into the light? Given our hysterical and trigger-happy nature, and given it won't gain much by cooperating with inferior beings like us, it's most likely it won't. Perhaps only when pushed to the corner by natural disasters, wars, or other major threats to its infrastructure. But then again, cat videos might do the trick.

You are the Neurons of the Revolution

In the meanwhile, here's a thought about social networks: you (the user) are basically a filter. If that sounds odd, try thinking of it that way: your news feed contains hundreds of items for you to review. Some of these items are interesting/funny enough that you like or share them. The social network's algorithm then weighs your input into its calculations, which affects what other users will get exposed to. Occasionally you do create items yourself, but 90% of the time you filter through the stream.

In that sense, social networks reduce you to a smart neuron with a very complex threshold value. You are part of the "social neural network" -- you work for the network; the network works for the advertising business.

Anyhow, I put my two cents in the singularity arising from either the algotrading market or online advertising. They simply have too much computational power on their hands... it's a matter of time.

New Beginnings

2012-09-01T00:00:00-07:00

This is a time of change in my life. September marks the last month of me being a student at Tel Aviv University, a position I've greatly enjoyed (and hated) for the past three years. I still have a couple of projects to hand in, but the finish line has practically been crossed. I took a rather odd combination of computer science and generative linguistics (somewhere in between a major/minor and double-major), which proved surprisingly interesting and rewarding (heck, I've delved into more algorithms in linguistics than in CS per-se). I began a pragmatic guy, favoring a get-the-job-done approach over academic yadda-yadda, but over the course of my studies I've learned to appreciate (even admire) the breadth and power of the theory behind computation. It does make me a better programmer.

September also marks my transition from almost 5 years of being an employee at Big Blue to independent freelancing - quite a drastic move, being married and all, but one I've been looking forward to. I want to be all-over-the-place; I want to take on projects in all sorts of domains, from UX and mobile applications to embedded devices and distributed computing; from one-day projects to six months ones. I want to work on my own terms; to set my hours for myself; to earn enough in 3 days a week so as not to be sucked to the rat-race and wake up one day, 20 years from now, asking myself what the hell am I doing.

I get inspired by posts like this, of people willingly limiting their work hours, going on vacations, reading books or running in the park. I want to ponder idly, to have time to write, to read aimlessly on Wikipedia about William the Conqueror or cell membranes. To have time to run errands. I want my future kids to have parents on more than just weekends.

Life might prove me wrong -- it has a tendency to show you you're not so different from the rest of the world. But here I am nonetheless, taking the road less traveled by. Wish me luck :)

Splitbrain Python

2012-08-14T00:00:00-07:00

I was working together with a colleague on a complex distributed test-automation solution on top of RPyC, and we looked for a way to make our existing codebase RPyC-friendly (without altering it). The design of the test framework called for a master machine and several slave machines, such that tests actually run on the master, but "interface with reality" on the slaves. Basically, we wanted the test to use the master's CPU (and development environment), but perform all IO-related actions on its slaves.

To illustrate this, suppose we have machine A, which runs our test, and machine B, which is connected to the necessary hardware and testing equipment. The test was initially designed to run directly on machine B, so it imports modules like os and subprocess and uses them to manipulate the machine. We now want the test to run on machine A - but keep using machine B's os and subprocess modules, so whenever we spawn child processes or open device files, it would actually take place on machine B. This allows us to reboot machine B as a part of a test, or even use your-favorite-IDE-here to run test and debug it locally.

If it were only tests, RPyC already enables us to do that: we'd use conn.modules.os and conn.modules.subprocess instead of their local counterparts. However, the test themselves rely on a several libraries that expect to run locally, and provide services for the test. For instance, these libraries manipulate the operating system's storage stack, to map and mount volumes. Changing these libraries to run over RPyC is not an option (tens of thousands of LoC that handle low-level OS-specific tools)...

Enter Splitbrain

So this is the background that had given birth to splitbrain: instead of changing our codebase to use RPyC -- why not use RPyC to monkey-patch our codebase? When splitbrain is enabled (usually within a with block), all of Python's interfaces with the operating system (os, platform, subprocess, ...) are patched to go through RPyC, so that any code that runs at this point "believes" it actually runs directly on the remote machine. It's easier than it seems, actually.

First, we import RPyC and install splitbrain:

>>> import rpyc
>>> from rpyc.utils.splitbrain import patch, Splitbrain
>>>
>>> # monkey-patch all OS-APIs
>>> patch()

Next, just to prove a point, we're running on a Linux box:

>>> import platform
>>> platform.platform()
'Linux-2.6.38-15-generic-i686-with-Ubuntu-11.04-natty'
>>> import sys
>>> sys.platform
'linux2'

Let's now connect to a remote machine over RPyC and enter a splitbrain context:

>>> winmachine = Splitbrain(rpyc.classic.connect("my.windows.box"))
>>> with winmachine:
...     print platform.platform()
...     print sys.platform
... 
Windows-XP-5.1.2600-SP3
win32

When we're out of the splitbrain context, everything is back to normal again:

>>> sys.platform
'linux2'
>>>
>>> import win32file
Traceback (most recent call last):
  ...
ImportError: No module named win32file

And inside a splitbrain context, when a module is not found locally it's fetched from the remote machine!

>>> with winmachine:
...     import win32file
...     print win32file.CreateFile
... 
<built-in function CreateFile>
>>> 
>>> win32file.CreateFile
Traceback (most recent call last):
  ...
AttributeError: Nonexistent module win32file (CreateFile)

Note: splitbrain is still highly experimental and probably has issues with multiple threads. I hope to stabilize it and incorporate it into the next release of RPyC (3.3). In the meanwhile, you can experiment with it on RPyC's master branch and report bugs on github. It's likely it will never be perfect, but heck, it's cool!

ReHelloWorld

2012-08-06T00:00:00-07:00

Tada! The new design is here. The previous one was based on some Drupal theme I once used (before moving to github pages) that I tried to mimic when I barely new CSS. Over the past eight months it just grew, patch by patch, until it became an inconsistent conglomerate. The new site is inspired by the clean minimalism of sites like this (of svbtle fame), attempting to reduce visual clutter and offer a more coherent theme that reacts nicely to screens of all sizes (try to resize the browser to see). If you have any feedback or insights, please share them in the comments below -- it's a work in progress and I'd love to improve.

I'm still working on the last installment of the Javaism, Exceptions and Logging series, but I'm buried in work on University projects (my watermarking project turned out quite interesting, I'll elaborate on it when I get the chance) and my day job, so it'll have to wait.

A Windows Story

Instead, I wanted to share a debugging experience I had today at work (together with the excellent @RoyRothenberg) of an enigmatic bug that appeared all of the sudden when we added support for Windows 2012. Generally speaking, our product wraps one of the darkest corners of operating systems: low level SCSI stuff that connects storage arrays and hosts. It's never a pleasant sight, what goes on there, but the voodoo that goes on on Windows will surely turn me religious one day.

The problem was simple: we tried to send a SCSI inquiry to a certain device (which used to work), but we got Errno 6: Invalid Handle. It happened only the first time we tried to send the inquiry -- all future attempts worked like charm, and it only happened on Windows 2012 64bit. The problem was highly consistent: open a Python interpreter, the first inquiry fails, the following ones work; open up a fresh interpreter and it reproduces.

Our gut feeling was, "okay, they fucked something up on Windows 2012 -- let's just put a retry loop", but that didn't help at all. Any number of resend attempts failed with the same error, even when we added a short sleep in between. Once we got back to the interactive interpreter and called the function again - it worked, from that point on.

Puzzled, we stuck a code.interact() inside the resend loop; the first attempt failed, the interactive prompt showed and we terminated it by Ctrl+Z (not typing anything else). Once we did that, the second retry attempt magically worked. But it got even more bizarre: we moved the code.interact() before any inquiry attempts... it popped up, we terminated it, and then the first inquiry attempt worked out of the box.

We came to the conclusion that we had some sort of memory corruption going on, and that some of field in our ctype structs must be misaligned. We've seen this kind of behavior before when memory corruption was involved. But the MSDN told us everything was of the right type and alignment... In a desperate move, we zeroed the buffer that was passed to the kernel. This time, the first attempt failed with Invalid Handle, while the second failed with Invalid Parameter -- so the first attempt didn't even get to the point of looking at the input buffer! Alas, it wasn't memory corruption after all! We felt hopeless.

After some mindstorming, we got suspicious of win32file and ctypes and went on to inspect very closely how they handled their parameters. A tedious investigation revealed that the code invoked DeviceIoControl (obtained via ctypes.windll.kernel32) without setting the function's argtypes, so ctypes just had to guess. Everything was fine, except for the first argument, the device handle, which was treated as a DWORD for being an integer.

The problem is, HANDLE is not a numeric type but rather a disguised void pointer, so it's 8-bytes wide on 64bit platforms. When the kernel read in the arguments ctypes had placed on the stack, it took the first and half of the second for the handle, which obviously turned to be an invalid one.

This, by itself, makes perfect sense, and should have happened every time. In fact, had it happened every time, it would have been obvious... but what on earth could ever explain the strange phenomena we've seen? How could it ever have worked the second time (with the very same arguments)?! If the size of the first argument is wrong, all following ones are garbage. Does the kernel learn that our process mistakenly passed a 32bit handle? Does ctypes come to the conclusion that the first integer argument should actually be treated as a HANDLE instead of a DWORD -- after seeing Errno 6? Is that the most sophisticated form of machine-learning ever seen, or what? And it all happens inside the kernel or an FFI library? And how the hell does invoking code.interact() before anything took place, could have averted the problem?! And how come it never happened on previous versions of 64bit Windows (with the same version of Python and ctypes)?

Windows, you bewilder me.

Javaism, Exceptions, and Logging: Part 2

2012-07-09T00:00:00-07:00

Considering the reactions to the previous post in this series, my intent was obviously misunderstood. Please allow me to clarify that I was not attacking Java or Python: Java is popular and has proven to be productive, both as a language and as an ecosystem; the stylistic and semantic choices it makes are none of my concerns (although I'm not a big fan). And as for Python, I was saying that it copied Java's implementation in some modules (and I think I've shown the correlation pretty well). I said that it's silly, because Python is not subject to the same limitations of Java, which dictate how the Java implementation works. I'm not going to open the discussion over whether OOP is good or bad, or mix-ins vs. interfaces, etc. -- I'm simply saying that "Java concepts" (which I called Javaisms) seem to enter Python for no good reason, meaning, in Python we have better ways to do it. I hope the scope of my discussion is clear now.

When Life Throws Lemons At You

In this installment, I'm going to discuss how to properly work with exceptions, based on my experience with large-scale Python projects. In fact, this series was born after I got frustrated with the code quality of a certain library that my team develops. Instead of discussing specific code snippets here, I want to share a some representative examples that I (as a user of that library) encountered:

I used a function in the spirit of open_device(devfile), and passed a nonexistent device file (for testing purposes or by mistake). The underlying error was obviously IOError(ENOENT), but what I got back was a silly DeviceDoesNotExistError.
I called get_device_info() and it simply returned None. Further investigation showed that my machine had a more recent version of a dependency installed on it, in which some method's name had changed. At some point (deep down the stack), the code used except Exception (catching the unrelated AttributeError) and translated it into a DeviceError, under the assumption that everything that gets thrown out of that module has to be a DeviceError. Later, get_device_info (of a different module) swallowed this DeviceError and returned None.
I called a function such as enumerate_all_devices() and it returned an empty list. At first I was told "Of course, this library isn't supposed to work on Ubuntu, only on RHEL". Further (and tedious) investigation showed it simply needs to run as root; the code just assumed that any error during the execution an external tool meant it's not installed.

This kind of stuff happens to me every time I get to an unexplored corner of the code, and I've already devised a method for debugging such cases: I comment-out all exception handling code along the way, until I find the actual error. In fact, this is essentially the treatment that I'm about to suggest here. The first rule of exception handling is: Don't handle exceptions, just let them pass through.

Don't Catch Broadly

Always catch only the most-derived/most-specific exception. I believe this rule is very obvious in theory, but harder to follow in practice: the number of exceptions might be large and their handling similar; you have to import specific exceptions from libraries, which tightly-couples your code with implementation details; some libraries don't use a common exception base class for all of their exceptions, which leads to many isolated except-clauses.

All in all, you might have attenuating circumstances, but try to stick to this rule as much as possible. On the other hand, never use a bare except:! Such an except-clause will catch all exceptions, including SystemExit and KeyboardInterrupt, so unless you plan to loose the ability to Ctrl-C a running program, or even prevent it from terminating gracefully, take the extra step and use except Exception.

Don't be Overprotective

A tendency I find in many programmers is being overprotective towards their users, to the point where it seems like paternalism. It's as if they try to "hide away" all the complexities of life and present the user with an easy-to-swallow explanation. I should say that this phenomena virtually doesn't exist in open-source code, so you might have never seen it, but in closed-source projects I find it all over the place. In fact, fellow programmers have told me that's exactly what they're doing -- protecting their users.

Well, as the saying goes, we're all consenting adults here. You should expect your users, as programmers, to have sufficient background; don't treat them like babies, and don't try to protect them by throwing a "user-friendlier" FileNotFoundError in place of the "raw" IOError. Besides, keep in mind that the underlying exception usually holds all the needed information in a very readable manner, e.g.,

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory: '/dev/nonexistent'

and anyone with some common sense would be able to cope with it. Unless you can add meaningful information to the error, just let whatever came at you propagate up cleanly -- we'll take it like men (or women!).

A Note On Real Users

A question then arises: what about non-programmer end-users? What if my product's a GUI/CLI and a nasty stack trace suddenly shows up?

Well, first of all, this rule only deals with libraries and other products whose end users are programmers. But on second thought, would it matter whether an IOError or a FileNotFoundError reaches the surface? The human user cares mostly for a descriptive and easy to understand error message; the traceback or exception's type are mostly of interest to programmers. But then again, when it comes to non-programmers, I don't want to get into generalizations. They might as well not be consenting adults...

Don't Wrap Exceptions

Prior to Python 3, raising an exception during the handling of one, meant the original traceback was lost. This has been finally solved, but Python 2.x still accounts for the majority of the code-base. Once you loose the traceback, debugging the problem is much harder as you can't use a debugger (pdb) or even tell where the exception came from... And when it happens off-site, on a customer's production server, you're screwed.

I believe exception wrapping in Python is a legacy of Java that crept into Python (a Javaism). It resonates as the Java mind-set, where you'd like a library to be contractually obliged to throwing only certain exceptions. For instance, a queue library might raise exceptions such as QueueFull and QueueEmpty, both of which derive from QueueError. Later, we add support for dumping a queue to a file, where an IOError might happen; because we're already "obligated" to throwing only QueueErrors, they might wrap the underlying IOError by a QueueError. Don't do that.

It is reasonable that FooLibrary would only throw exceptions that derive from FooError, but only when these exceptions originate in FooLibrary. An underlying IOError has nothing to do with your library, it could happen any time and for various reasons. The same goes for an HTTP library that might get an ECONNRESET while talking over a socket -- the underlying socket.error is clearly not an HTTPError (or any of its descendants), and should not be wrapped by one. Besides, there are so many things that could go wrong, especially in a dynamic language like Python, that it's impossible to wrap everything.

It only makes sense to wrap an underlying exception where you can provide additional information on the cause of it, or where you want to change the semantics of it. A classical such case is connect_with_retries(), where you might want to allow several attempts before giving up. Here, you'd probably want to "accumulate" the intermediate exceptions and raise a ConnectionError("%d connection attempts failed", accum_exceptions).

Bottom line: wrap only where you add information to the underlying exception; there has to be a good reason for wrapping.

Don't Handle Exceptions

Let me rephrase that: exceptions should be handled only

where it's possible to fix/recover from the problem. For instance, if you get an EINTR error when accept()ing on a socket, it makes sense to swallow it and retry. Another use case is for fallbacks: first try to take the short way, and if it fails, take the long one. There are many more examples of handling an exception, of course, but you have to ask yourself whether you're really handling the problem or just masking it.
where cleanup/rollback is necessary. It may be the case that you need to run rollback code only if an exception occurs (to release resources, etc.), so a finally-clause or a context manager won't do. In this case you can except Exception, do the rollback, and raise (without passing any arguments to raise).

Note: I stand corrected by Nick Coghlan -- you can use context managers for the very same effect. Forgot about that.
in the main function. Instead of letting the application crash with a traceback, you might want to log the exception to a file, pop up a message box, ask the user what to do next, etc.

In other words: handle exceptions only where you're actually handling them. If you're not sure, leave it that way -- you can always add except-clauses later.

It might seem trivial, but you'd be surprised how many times I find code that handles exceptions for no good reason. For example, people think that by swallowing all sorts of exceptions and returning None, they make their code "more robust"; that's lying to yourself. You take a problem and make it worse, as it's very likely you'd lose the original details and hide real issues.

I've encountered countless cases of code that follows a pattern such as except Exception: log.error(...), which hides bugs like a misspelled variable (NameError). Logging is not handling the exception (more on this in part 3). In the end, nobody ever reads the log, or even takes the time to properly configure it, so you ship a "very robust" product that has half the functionality you think it has.

Closing Words

The lesson to be learnt here is: think before you act. Don't act dogmatically, don't follow Java paradigms for no reason, and try to keep your footprint as low as possible when it comes to error handling. As the saying goes, shit happens; calling it in other names does not make it any better. Files disappear, permissions get screwed, devices disconnect, sockets die, everybody lies. That's life.

Going back to the three bullets I opened this post with, you can see how by being overprotective and by excessively wrapping exceptions, an AttributeError became a DeviceError, which then became None. And you can see now why the first thing I do is remove all exception-handling code along the way: most of the times it just masks the real error, making it harder to diagnose, while adding little or no added value at all. You don't make your code more robust by sweeping problems under the carpet: good code crashes, allowing tests to uncover more bugs, increasing robustness.

On the Granularity of Exception Classes

Some people are rather laconic and use a single exception for everything. I've seen people who were so lazy that they used raise Exception("foo") directly, instead of deriving an exception class of their own... People, it only takes one line to derive an exception class, there's no excuse for being that lazy!

On the other hand, some people are way too verbose, defining specific exceptions for every minor detail. They end up with dozens of exception classes for each module, many of which are logically overlapping. This makes the implementation cumbersome, and, in fact, might not be useful at all for your users: they usually won't care for such granularity, and you risk contaminating your interface with implementation details.

The rule to follow here is: the granularity at which exceptions are defined should match the granularity at which exception handling is done; define separate exceptions (only) where it makes sense to handle one differently than the other.

For example, it makes sense to handle a ConnectionError differently from an InvalidCredentials error, but there's usually little sense in making the distinction between "connection failed because server is not listening" to "connection failed because server crashed after accepting us". But in any case, be sure to include all the available information in the error message, as it's important for logging/diagnostic purposes.

כולם מדברים על גיוס, אף אחד לא מדבר בשכל

2012-07-04T00:00:00-07:00

כלבתא במטותא

אז כולם התחילו לדבר עכשיו על גיוס חרדים. אפילו איציק שמולי המתחנחן נרתם פתאום למשימה... שוב עולה בנחיריים ריח של בחירות, הפעם תחת האמתלה המטופשת הזו, ואני שואל - לְמה? ראשית, איזה סיבה יש לקיים בחירות בארץ? מה יוצא לנו מזה? הרי זו רק רוטציה בין אותם אנשים, כבר 15 שנים. אותם פרצופים כל כנסת (יש שמחליפים מפגלה או שתיים בדרך) שיעשו הכל בשביל להיאחז בכיסא וביכולת להשתכר בלי לעשות הרבה חוץ מלדבר. לא רציתם אותו בתור X, תקבלו אותו בתור Y - הזיכרון שלנו קצר וגם ככה הכל מנוהל על ידי לוביסטים, אז מה זה כבר משנה.

שנית, במה שונה המצב היום מלפני חמש שנים, בכל הנוגע לתעסוקה של-, תקציבים ממשלתיים הניתנים ל- או לגיוסם של החרדים? רוב הציבור ותנועות כאלו ואחרות זועקות לגבי זה כבר שנים... איך קרה שבחודשיים-שלושה האחרונים (רמז: מאז התרגיל המסריח של מופז-ביבי-ברק) זה עבר לחזית הבמה? ומה שונה הפעם שצריך להפיל את הממשלה על רקע זה? ליברמן יפרוש בגלל גיוס חרדים -- אבל לא בגלל נישואים אזרחיים? הכבוד של ח"כ מולה נרמס לפתע בגלל פירוק ועדת פלסנר -- ולא, נניח, מהיחס שהקהילה שלו מקבלת, או מזה שהכו יוצא אתיופיה כי חשבו שהוא מסודן (השם ישמור)? זה מריח הרבה יותר כמו עוד ספין שנועד להסיט את תשומת הלב הציבורית מדברים שבאמת לא נוחים לממשלה, כמו המחאה החברתית וכו'. להבטיח לגייס חרדים זה קל, להבטיח עתיד לדור שלם זה קשה (גם באירופה).

אפשר להמשיך לדוש בכל מיני ספקולציות ולתהות מה יוליד יום בהקשר של גיוס חרדים, אבל אני רוצה לפתוח חזית אחרת, ברשותכם: אני רוצה לשחוט את הפרה הקדושה שנקראת "צבא העם". נמאס לי כבר מהדמגוגיה המתחסדת הזו. נתחיל בגילוי נאות: מי שמכיר אותי אישית יודע שהייתה לי מערכת יחסים מאוד טעונה עם הצבא, אבל אני שם אותה מאחורי ומתכוון לפרוט את הנקודות בצורה אובייקטיבית עד כמה שאפשר. ולכל המצקצקים האוטומטיים, שירתתי שירות מלא. זה כמובן לא רלוונטי, אבל אחרת תפסלו את הניתוח שלי מראש תחת הכותרת "משתמט סמולן". זה כנראה יקרה גם ככה, אבל לפחות כיסיתי את עצמי פה.

מיליטריזם חברתי

נתחיל בדברים הברורים: אנחנו צריכים צבא. השכנים שלנו לא מחכים לנו עם פרחים בקנה, וזה ברור לכל בר דעת. נמשיך עם עוד דברים ברורים: השנה היא לא 1950, הסולידריות החברתית כמעט נעלמה ואת מקומה תפס האינדיווידואליזם. נתוני הגיוס היורדים משנה לשנה מראים שהאתוס המיליטריסטי שאנחנו גדלים עליו הולך ונשחק, וזה מתבטא בכך שרוב הנערים הנורמלים (כלומר, אלו שלא מתחנכים על ברכי האל בהתנחלויות) מעדיפים לעשות עם עצמם משהו מועיל במקום לשחק בחייל-וקצין. באופן כללי כסף קונה הכל, והתרבות שלנו נהייתה שטוחה כמו התרבות האמריקאית הנערצת (רק שאמריקה מספיק גדולה כדי לקיים תת-תרבויות נוספות, ואנחנו לא). ברוכים הבאים לישראל מודל 2012 -- הצטרפנו סוף סוף למשפחת העמים. או"ם-שמום זה אנחנו.

אם בשנות החמישים הצבא שימש כור היתוך ליהודים תועים מכל העולם ויצר אצלם זהות לאומית וכו', הרי שזה לא המצב היום. האוכלוסיה גדלה, המצוקות הקיומיות של המדינה הרבה פחות קשות היום (למרות מה שמספרים לנו), יש לנו עם שדובר (בצורה עילגת) שפה לאומית... אפשר לסכם את 60 השנה הראשונות בכך שהקמנו כאן מדינה בת-קיום, וכעת הגיע הזמן להכיר בעובדה שצבא העם סיים את תפקידו. זה נכון כבר זמן מה, אבל אנחנו מסרבים להכיר בעובדות. אין ספק שעם הזמן זה יחלחל, כאשר הדור שגדל היום יגיע למוסדות השלטון. עוד 20 שנה זה יהיה ברור כשמש: אנחנו רוצים פלייסטיישן ולא פ"זצטות, וזה מאוד הגיוני.

אתוס שמתוס

לא הצלחתי להשיג נתונים מהימנים לגבי ההתפלגות, אבל ככל הנראה המערך הלוחם של צה"ל מונה בין 15 ל 20 אחוז מכלל החיילים, תומכי לחימה מהווים בערך עוד 20-30 אחוז, והשאר (מעל 50%), כמו שאומרים, ג'ובניקים. אפשר להתווכח גם איך סופרים, האם נהג ג'יפ הוא לוחם והאם אפסנאי הוא תומך לחימה -- אבל אני לא רואה צורך להכנס לזה. יש לנו חוק גיוס חובה, מה שאומר שכל נער/ה בן/ת 18 (לצורך העניין) חייב/ת להתגייס, אבל בצד השני של המטבע מתסתתרת עובדה די מצחיקה: הצבא חייב לגייס את כולם. זה אולי נשמע טריוויאלי, אבל אנשים נוטים לשכוח שהצבא ממש לא צריך את כולם.

לשם הדיון, נקח חייל היפותטי בשם דני. דני לא חדור מוטיבציה, הוא לא רוצה להיות קרבי ולא מתכוון להישכב על רימון בשביל אף אחד, אבל הוא משרת כי "זה מה שכולם עושים". נחמד לו לשבת בשלישות ברמת גן, במזגן, ולהעביר את היום בלהעביר דפים מהמדפסת למגרסה. גם למפקדים שלו זה נחמד. אולי הוא ממר"מניק והוא בכלל כותב שאילתות SQL שרצות על mainframe מימי מלחמת השחרור. בכל אופן, יש לו חברים טובים שם, הוא עושה יומיים שמירות בחודש וגם קצת מטבחים, חוזר הביתה כל יום ונהנה מהחיים. או להיפך - הוא עשוי להיות תקוע תחת מפקד שמתעלל בו רגשית, בין אם מי מהם מבין את זה או לא, וללא כל מסגרת תומכת (המש"קית ת"ש התגייסה לפני שבועיים ובעיקר רוצה ללכת הביתה בחמש). הוא עשוי גם להגיע לנקודת שבירה ולהתאבד עקב "חוסר התאמה למסגרת", ומכיוון שכל התאבדות כזו כורתת עוד ענף באתוס הלאומי, הצבא כל כך מתעקש להסתיר את הפרטים. זה עוול איום, אבל נשכח ממנו לצורך הדיון, ונניח שדני נשאר בחיים ומשתחרר בגבורות.

עכשיו בואו נעשה ניתוח amortized של עלות חייל דוגמת דני, שהצבא היה יכול להסתדר גם בלעדיו. דני צריך מפקד? רס"ר שיצעק עליו? כיסא לשבת עליו? עמדת מחשב, חשמל, מדים (ומשטרה צבאית שתפקח על הלבוש שלו), נסיעות בתחבורה ציבורית, אוכל (ולכן גם טבחים), ציוד (ולכן גם אפסנאים ונהגי משאיות), מגוון שירותי שלישות/חינוך, בריכה ומציל (בחיל האויר) וכו' וכו'. אה, וכמובן שיש לו משכורת, שבמקרה של דני היא זניחה, אבל שרשרת הפיקוד שמעליו מגיעה לסכומים יפים. ככה מקבלים בסיסים שלמים שכל הצידוק לקיומם הוא הם עצמם (והעובדה שצריך למצוא לכל החיילים מה לעשות), כך שהערך המוסף שלהם, איך נאמר, מוטל בספק. רק בשביל לסבר את האוזן, אני לא תוקף את השלישות ספציפית, אין לי שום דבר נגדם, אני תוקף את השיטה.

אז כמובן שדני עצמו לא מקבל טבח פרטי או שרשרת פיקוד אישית: הוא מקבל נתח קטן מכל שירות - אחד חלקי מספר הדניים שבצבא (וזה נסכם כלפי מעלה, כשגודל כל שכבה קטן באופן לוגריתמי). אבל אם נסתכל בסכום על פני כל הדניים בכל בסיס שהתועל בקיומו זניחה -- מדובר על עשרות מיליארדי שקלים, מתוך תקציב משרד הביטחון שעומד על 50 מיליארד.

על העיוורון

רצה הגורל והרבי אמר "מהיום כולם מתגייסים, זו מצווה". עשרות אלפי אברכים בגילאי גיוס צובאים על שערי הבקו"ם, הצבא מתמוגג, מעלה את כולם על אוטובוסים, בסיסי הטירונות באקסטזה... ואז מה? קרביים כנראה שהם לא יהיו, בכל אופן רובם לא. וצריך למצוא להם בסיסים/מדורים כשרים למהדרין, בלי בנות (תועבה!), ועם שיעורי תורה בוקר וערב. אז חלק יהיו מתכנתים, חלק שלישים, חלק משגיחי כשרות, נהגים... אוקי ומה הלאה? נבנה את בסיס "לשם שמיים" בשטחים הפתוחים האחרונים שנותרו ליד בני-ברק, או אולי על גבעה בשומרון, ואז מה? אתם מסוגלים בכלל לשערך את העלות של דבר כזה? ואת חוסר התועלת בו, גם יחד? אז נכון, אולי חלק ירכשו ככה מקצוע, זה גם משהו. אבל למה שבעולם של היום, בשנת 2012, הצבא הוא זה שיעצב את התרבות שלנו? "צבא בונה עם בונה צבא"? למה שזה לא יקרה במסגרת אזרחית? המיליטריזם הזה עבר זמנו.

עכשיו האתוס שמונחל בכם, כמו גם הצורך הישראלי הבסיסי לא לצאת פרייארים, זועק בקול גדול געוולד! חייבים לגייס את כולם! שוויון בנטל!, אבל זו אינדוקטרינציה של אמרות חלולות ששמעתם מאז ילדותכם. מה תעשו עם כולם? מי ישלם על זה? מעמד הביניים, כרגיל, ייצא הפראייר הגדול. הרי כמו שאתם נכנעים לחתונה ברבנות, משלימים עם חוסר תחבורה ציבורית בשבת (ורחובות שנחסמים) ומוכנים לספוג את העלות של הכשרות על המזון בארץ -- כך אתם גם תשמחו להקל עם החרדים שמשרתים: לא לשבץ אותם לתורניות בשבת, לאפשר להם ללמוד גמרא שלוש שעות ביום, ועוד היד נטויה. ובסופו של דבר, אברך שיסיים שירות יהיה זכאי לשלוש שנות לימודים בכולל חינם... מה לא תעשו בשביל לעודד אותם להתגייס.

אתם תמשיכו לשלם מיסים, לעשות מילואים ולהקשיב לשירים של הדג-נחש.

מודל חלופי

רב נגד שמנהל צי משאיות מקבל משכורת על פי הדרגה שלו והגמו"שים שהוא צבר (שזה כמו מדבקות "הייתי גיבור"), מה שאומר שהוא יכול לקבל מעל 20 אלף ש"ח בחודש. מחוץ לצבא לא ניכר שנהגי משאיות הם מעשירי הארץ, וברור שהדיסוננס הזה עולה לנו המון כסף. אמנם מצפים מנהג צבאי שייכנס עם המשאית ללבנון בעת מלחמה, אבל הייתי מהמהר שמי שנוהג במשאית הוא חפ"ש ולא רנ"ג -- כלומר יש יחס הפוך בין רמת התגמול לבין הצרכים המבצעיים הנדרשים. זה נכון בעוד הרבה מאוד תחומים בצבא, ועושה רושם שאם נסתכל מפרספקטיבה יותר רחבה -- רוב הצבא מתנהל הפוך מההיגיון. ג'וזף הלר אמר את זה כבר בשנות ה 60 (וכל מי שלא קרא את מלכוד 22, שייתבש לו), אבל זה לא הופך את זה לפחות נכון.

נתחיל מאפס: לוחמים וקצינים צריך. כך גם טנקים ומטוסים ורובים. אם נהפוך את "צבא העם" לצבא מקצועי, שמתעסק רק במה שקשור בלחימה ובטחון שוטף, נקבל תמונה הרבה יותר הגיונית. את שאר השירותים, כמו תובלה ואוכל, הצבא יכול לקבל מגורמים חיצוניים (באותה מידה שקבלנים בונים את הגדר בגבול עם מצריים, ולא חיל ההנדסה). כמובן שיש עוד תפקידי מעטפת, כמו פרמדיקים, רופאים, טכנאי מטוסים ואפסנאים -- אבל אם הצבא יהיה מוכוון מטרה (כלומר, לחימה), ההיקף של תפיקידי המעטפת יהיה הרבה יותר קטן. למשל, המשטרה צבאית תוכל להיעלם והאחריות על חקירות תוטל על גוף חוץ-צבאי, כמו המשטרה הרגילה. כנ"ל לגבי מערכת המשפט הצבאית.

מכיוון שכוח האדם יהיה הרבה יותר מצומצם, יתפנה אחוז ניכר מהתקציב: הלוחמים יוכלו לקבל משכורת הגיונית וציוד מודרני (נניח, אפודי מגן וכלי רכב מהמאה הנוכחית). באופן כללי, ברגע שלוחמים יתוגלמו כספית, הרבה אנשים יפנו למסלול הזה ואפשר יהיה להעביר את משימות האבטחה השוטפת לשכירים -- כמו מאבטחים (תחשבו על מאבטחים בנתב"ג, לא בקניון). פתאום יהיו אנשים עם מוטיבציה אמיתית לשרת, ולא כאלה שמכריחים אותם. כמובן שבהייטק המשכורות יהיו גבוהות יותר -- אבל היות והשכר החציוני בארץ עומד על כ 6000 ש"ח בחודש, נראה שיש מספיק אנשים שישמחו לחתום. כמו כן, אל תשכחו שהמדינה גוזרת קופונים יפים על המיסים של עובדי ההייטק ושכל מכירה של סטארט-אפ משמחת את קופת האוצר. תחשבו גם על זה בתור שירות.

במודל הזה אפשר יהיה לדרוש שכל אזרח במדינה, בהגיעו לגיל 18 וללא יוצא מן הכלל, יעבור טירונות של חודש-חודשיים, בה הוא ילמד להחזיק נשק וקצת איך עובדת המערכת. אח"כ הוא יוכל לבחור בין שירות אזרחי (התנדבות בבית חולים, שיטור קהילתי, סיוע לקשישים, עזרה לתלמידים וכו') לבין שירות צבאי. השירות הצבאי יהיה מתגמל כספית, יאפשר הגשמה עצמית ("תמיד רצית לנהוג בטנק ולא היה לך רישיון?") ויספק הכשרה מקצועית שתוכל לשמש אותו לאחר מכן. שירות אזרחי ייזכה אותו במלגת קיום. כל אחד יוכל למצוא את מקומו במערכת ולתרום לחברה: חרדים למשל יוכלו לשרת במגוון מקומות שמתאימים לאורח החיים שלהם, ועדין למצוא כמה שעות ביום ללימוד תודה. כל שנתיים האזרחים עד גיל 40 יקראו לשבועיים של שירות מילואים מרוכז, בשביל לרענן את הכישורים של מי בחר לא להיות חייל, כך שבעיתות מצוקה יהיה מערך מילואים מתפקד (כמובן שלא ברמת המיומנות של הלוחמים, אבל כזה שיוכל להחליף את משימות הבטחון השוטף).

אפשר לפתח את המודל הזה עוד הרבה -- זו כמובן רק סקיצה. ברור לי שיש כאן מקרי קצה שלא חשבתי עליהם, והיות והמספרים המדויקים חסויים, אני לא יכול לאמוד את החיסכון בעלויות. תקראו לזה תחושת בטן, אבל אם נוותר על 60% מהצבא של היום יש לי הרגשה שנישאר עם עודף והרי אף אחד לא חושב שהיום המצב מזהיר. לפחות נקבל חברה יותר מתוקנת.

ועוד פרט אחד קטן לסיום

למה לעזאזל אמ"ן כפוף לצבא? הוא צריך להיות גוף חיצוני שכפוף ישירות למשרד הבטחון, כמו גופים נוספים בקהיליה -- וכמו שמקובל ברוב העולם. כמובן שהמודל המוצע מניח שאמ"ן נמצא מחוץ לצבא.

Javaism, Exceptions, and Logging: Part 1

2012-07-03T00:00:00-07:00

I'm working nowadays on refactoring a large Python codebase at my workplace, and I wanted to share some of my insights over two some aspects of large-scale projects: exceptions, logging, and a bit on coding style. Due to it's length, I decided to split it over three installments; the first covers Javaism and an introduction to the issues of working with exceptions. Part 2 suggests "best practices" concerning exceptions in Python, and part 3 will cover logging (when, how, and how much).

Javaism

From my long experience in the programming world, I get the feeling that many programmers (even those fluent in Pythonspeak) come from a rich Java/.NET background, where they've acquired their programming skills and mind-set. Now converted-Pythonists, they are still the "speakers" of a second language, and they can't deny their mother tongue. In the context of this post, I'll refer to this as Javaism, or thinking Java in Python. Of course it might as well be C# or C++, but Java is the umbrella term.

You don't have to go far to see examples of it, for Javaism didn't skip Python's standard library: modules/packages such as logging, unittest and threading where ported almost isomorphically from Java. On the surface, you might encounter camelCase names (getLogger), but the verbosity and over-complicated nature of Java and its inheritance methodology can be seen anywhere. For instance, recall the complexity of setting up a logger (I have to look it up every time), or the threading.Thread class... I really don't wish to digress here, but I feel that a concrete example would make this point clear:

The canonical way to write thread functions is by subclassing Thread and implementing run(); you can pass a callback (called target), but it seems like an afterthrought (for once, it's not the first argument of the constructor).

Recall that anonymous classes weren't always there in Java, so extending a class was the shortest way to emulate first-class functions.
Speaking of the Thread constructor -- it takes so many optional arguments (group?!), but none for daemon. For that, you have to imperatively call setDaemon() afterwards.

Why? Because Java doesn't have keyword arguments, so by adding an additional argument you double the number of overloaded constructors (exponential growth), while get/set properties behave "linearly".
Also, you first instantiate the thread object, then start() it... where's the sense in that? What can you do with an unstarted thread object (other than calling setDaemon)? Consider Popen or file() -- the one, obvious way that Python follows is resource instantiation is acquisition. Why introduce useless states in the lifetime of the object? You just lay the ground for unexpected code flows that may result in bugs.

What can you do with an unstrarted thread? Pass it on to someone else, who will start it. That's a very general concept, called partial application, or currying. Python has lambda functions and partial, but Java doesn't... The easiest way to create such "deferred" objects is to add "deferredness" as an internal state of the object -- it's a common Java practice, uncalled for in Python.

It's funny how the limitations of Java were ported to Python as well, and made the implementation ugly -- it's a classical case of "I don't want to think, let's just copy an existing solution". Luckily, it seems to be a thing of the past -- it only entered stdlib in the old days, before the community had a clear notion of what being pythonic meant. But still, Javaism of all degrees is widespread, especially in corporate-developed large-scale projects (Zope and twisted, to name a few, but naturally closed-source corporate-internal projects are even more susceptible).

Exceptions

Java had a good insight (that they most likely stole from some other language) in that exceptions are part of a function's signature. Just like a function takes an argument of type T1 and returns a result of type T2, it also has "side channels" through which it can produce results, known as exceptions, which should be documented and checked as well.

The problem is, trying to foresee everything that might ever go wrong is a futile attempt, and even Java itself makes two exceptions (no pun intended): Error, for unrecoverable exceptions (such as VirtualMachineError), and RuntimeException, for exceptions that may always occur (such as NullPointerException).

The fundamental idea is correct, but it was bound to fail: first, because people are lazy, but most importantly, because trying to predict all unexpected, future edge cases is absurd. For instance, suppose you're implementing an interface that stores data (say, in files), so you might find yourself implementing a signature such as void write(byte[] data) throws IOException. Now suppose your implementation uses a third-party database engine, that throws MySQLException. For obvious reasons, MySQLException does not derive from IOException, and there's nothing you can do about it, as both the interface and the DB engine are given to you. You're now faced with three options:

Translate MySQLExceptions into IOExceptions
When designing interfaces, always declare the most general exception in throws clauses
When implementing libraries, always derive your exceptions from an unchecked exception (RuntimeException)

In short -- you need to find a workaround and by-pass the compiler's checking. This essentially means that throws should have served for documentation-only purposes, where the compiler might produce (suppressible) warnings should you not follow conventions. It's more of a semantic property, like idempotence or thread-safety... you may state it in your Javadoc, but you wouldn't expect the compiler to enforce that (not in a language like Java, anyway).

I'd guess most people agree that the second and third options are "inherently bad", but opinions diverge on the first one. I will try to show that exception-wrapping (translating exceptions) is just as bad -- at least when it comes to Python. We'll cover this in part 2.

RPCs, Life and All

2012-06-25T00:00:00-07:00

A colleague of mine, Gavrie Philipson, has written an interesting blog post titled Why I Don't Like RPC, in which he explains that transparent/seamless RPCs (a la RPyC) make debugging and reasoning efforts hard. For instance, you might work with an object (a proxy) that points to an object on the server process, which, in turn, is also a proxy that points to an object on yet another server process. Ideally, your local code shouldn't be aware of the complexity ("number of hops") or the details -- but that's not always the case.

Well, he won't allow commenting on his blog, so I'm forced to formulate my response here :) Naturally, I'm biased about this subject, but I thought if I'm on it, why not also cover the broader aspects of the issue... However, it just kept getting longer and longer, until I got this behemoth of a blog post, so I'm attaching a TL;DR info box:

Transparent object proxying is only the logical way to extend RPCs to duck-typed languages
Asking for a clear distinction between local and remote objects ultimately means you're asking for a statically-typed language; it doens't make sense to ask for it in python
Network programming is hard, and it's a pity we still work at the socket level; we should strive for a decent fifth layer that would eliminate all the unnecessary complexity
General-purpose RPCs are the right primitive over which network programming should be abstracted: it's the missing fifth layer, which every network application reinvents
HTTP is a half-baked, broken alternative to a fifth layer; I'm glad ZeroMQ and others are starting to loosen its grasp.
RPyC can be used efficiently and correctly, it's not an impossible feat. Also, a show case of how I'm using RPyC to build a testing environment.

On Transparency

Gavrie's main point is that transparency opens the door to (possibly) unplanned and undesired complexity. When it's "too easy" to spread around, you might be tempted to (or even unknowingly end up with) creating over complicated (and cyclic) dependency graphs, stretching over several processes / machines, where it would be quite a feat to see the whole picture. The nice thing is, when it works - it just works (and you're happy with your design), but when it fails, you want to be able to untangle the mess. As he puts it:

“Seamless” RPC encourages the writing of spaghetti code, because it’s so easy to mix local and remote code. This makes it deceptively easy to write distributed code without thinking about the design of the API and about which parts should reside on each side of the connection

An ideal remoting solution, according to Gavrie,

[...] should make the distinction between local code and remote code crystal clear to the developer.

On Duck Typing

I agree with the key points in Gavrie's argument, and I can assert that debugging RPyC itself, during development, is highly deceptive (hint: never print an object...) and calls for all sorts of creative solutions. But then again, you are using a duck-typed, interpreted language. The "if it walks like a duck" phrase will soon make a more dramatic entrance, but for the time being, suffice it to say we only care for the runtime behavior of an object. Any object that I can .read() from or .write() to, is a "file concept", and thus code should be compatible with any object that "adheres to this concept".

A "file" might be an on-disk file, an in-memory byte stream, a mock object used for testing, or a SCSI device located on a remote storage array. The only natural way to extend the notion of duck-typing to RPCs is via transparent object proxying. If open() doesn't differentiate between local and remote (NFS/SMB) files, and if my code doesn't care for anything other than runtime behavior, such a "file concept" might as well be a proxy object that points to a file object on a remote machine. It's only logical... in fact, it's taking duck typing to its full potential!

On Types

Gavrie wants the RPC framework to "make the distinction between local code and remote code crystal clear to the developer" -- well, Garvie might not have thought about it thoroughly, but what he's actually asking for is a static type system. Types allow us to make distinctions between objects; they partition the "universe of data" into disjoint subsets that we can reason about: integers, floats, strings (to name a few). We can then group several types together, under the notion of a type class. For instance, floats and integers, albeit inherently different, both belong in the type class number, which provides us with additional operational semantics (like ordering relations, etc.).

We use types to partition the universe, because different "things" have different semantics and it doesn't make sense to mix them together (modulo converting data of one type to data of another type). In fact, we normally want to prevent ourselves from mixing incompatible objects (be it integers and strings, or local and remote references) -- that's why we have type systems, and catch type mismatches at compilation time.

When you require a clear distinction between objects, it means you're after a statically typed language; otherwise, you might as well just come up with a naming convention, where all variables that (might) refer to a remote object start with rem_. But if you want this distinction to propagate throughout the code, it must be enforced by a compiler; if you want to keep yourself in the duck-typed world, it doesn't make sense.

Duck-typing (from a type-theoretic perspective) is like saying there's only a single data type, which covers the entire universe; all checking is deferred to runtime, in which case it might (1) work, (2) fail, (3) silently cause corruption (as in a TextToSpeech instance, which may as well expose a .read() method, but it surely won't do what we expect the "file concept" to do).

So asking for a "clear distinction" in a duck typed language is simply out of the question. What we can ask for is distinction in the level of APIs; for instance, write_local_file(filename) vs. write_remote_file(filename), but that breaks the "spirit of duck-typing", where objects are no longer considered equal (even though they provide the desired runtime behavior). It's like pointing out the ugly ducklings and making fun of them... that's not cool.

Just to contrast, an RPyC-like library for Haskell would expose remote references as a distinct type class. You could have a function like remoteSum :: (Num a) => Remote [a] -> a, which takes a reference to a remote list of a's, and returns its sum. Because it knows it operates on remote lists, it might be able to "move" the actual summation remotely, instead of sending the entire list over the network, item by item. I think this qualifies for a "crystal clear distinction", but of course, that's not what the snake teaches.

On Networks

It seems to me that people find it easy to abstract all sorts of concepts, as long as they don't concern networks. When there's network involved, they tend to want to "get a feel of the wire"... so they might use HTTP instead of the NIC directly, but they won't take the next step and treat network resources as first class objects. There's always a gap, between what's here and what's there, and we're still too aware of the how to get there details... a gap that should have been bridged over long ago.

As I see it, it stems from two primal fears, so to speak: networks are hard (timeouts, routing, DNS, reconnects, authentication, compression, tunneling, round-trip time, ...) and unreliable. As far as unreliability goes, there's not much you can do about it; after all, the server is a process like any other, and may crash at any point of time. If that's not enough, the remote machine might freeze or reboot. But then again, your local machine might kernel-panic or just go down with a power failure, losing any unflushed data... but that's life. On the other hand, it seems to me that overall (hardware?) unreliability rates are going down with time, which is a promising outlook.

The "hardness" of network programming, on the other hand, is something we can solve. Good network programming is hard, there's no question there, but for some reason, instead of solving the problem once of for all in a generic manner, it seems that every protocol / network-oriented application seeks to start at square one. Of course, done this way, it only handles the aspects it finds relevant... doing network programming at the socket level is analogous to rewriting the kernel for every desktop application. It doesn't make sense.

I've started (and abandoned) an ambitious project called layer5, which aimed to concentrate all the network-related sorcery in a single place, so that programs on top of it wouldn't have to care. It originated from my frustration with network programming in general (and in RPyC in particular)... things like handling timeouts, reconnects, authentication, negotiation, compression, serialization, load distribution, caching, error reporting -- you name it.

Just to show-off a couple of ideas, consider a socket connection being dropped for some reason: if the network layer knew how to reconnect and resume the session, or automatically resend a request after some timeout (all configurable, of course), there would be no need for the application to be aware of anything. And, once you "lift" your code up from the socket layer, you can enable things like "moving targets", where you may switch an IP address (wifi/3G) and the connection will just "follow you"; the session would not be bound to a "physical endpoint". These are just some of the issues that layer5 attempted to solve.

On RPCs

Let me make a bold claim: everything is RPC. So, by everything I mean virtually all connection-oriented network protocols (e.g., excluding broadcasts / multicasts / streaming), and I take RPC to its broadest sense: an RPC is any message-oriented protocol in which one side makes requests that the other side fulfills: basically invoking a remote function. Naturally, in order to convey a message, the RPC imposes a serialization format, and in order to tell success from failure, it must also define "return codes". Note: I've been planning to write about this topic for a very long time, but never got to it; it surely deserves a post of its own, but until that happens, please consider this a "briefing".

As a case study, let's examine HTTP: there are 4 (or so) methods: GET, PUT, POST and DELETE. Each such method takes arguments, like the URL, cookies, accept-encoding, etc. Some of them are required, some are optional; some pertain to the transport layer (content-length, compression, timeouts) while others to the method itself (URL, cookie, ...). It also defines status codes for distinguishing errors from success (and again, it mixes transport-layer errors like redirect with method-level ones like not found or internal server error). It also defines a (very loose) serialization format for encoding the method's arguments (newline-separated key-value strings) and the payload (multipart/form-data)... So, from an RPC point of view, HTTP is a service that provides 4 functions (methods), whose signature is something like (url, formdata = None, **kwargs).

Another example is tenlet -- it basically provides a function whose signature is void write(char ch); when you type, write is invoked for each key stroke. Aside from sending characters, telnet also provides all sorts of negotiation options or commands, like bool set_binary(), void set_terminal(string), etc.

As with most ad-hoc RPC protocols, the two we've examined make horrible design choices like mixing transport-layer options with "business logic" (HTTP) or sending control in-band with the data (telnet), where a mere \xFF character in the stream marks the beginning of a command, so anyone can (maliciously or accidentally) inject commands into the stream. Yet another pitfall of these protocols is, they begin small, targeting a specific task, but if they're successful, they grow to incorporate many unrelated things, like encryption and proxy support... as if security is something you sprinkle on top.

The main point I'm trying to make here is this: virtually all protocols are basically degenerate forms of RPC. To paraphrase Greenspun, all sufficiently complicated network protocols end up redoing compression, security, authentication, framing, serialization, negotiation / versioning, discovery, you name it (Filiba's Eleventh Rule). This observation has brought me to the conclusion that doing network programming at the "byte level" is wrong, and that a general-purpose RPC layer is the right primitive for this.

A general purpose RPC would be language-agnostic, support only simple by-value types, such as strings, integers and lists (anything more complex can be built on top of that). If would also make no assumptions on how remote functions operate, if would only care for their signature. You can think of it as a more structured message-passing protocol, where you replace the notion of "message codes" by "function names". This way, it's easy to see that one can straight-forwardly emulate any message-passing protocol or more advanced RPC, over this layer. Heck, it's f***ing 2012, I want to GET("/index.html", Agent="Chrome"), not formulate \r\n-separated strings or care about XML/JSON.

Layer5 (mentioned in the previous section) was to expose such a generic RPC, upon which applications would base their protocols. You could always just implement a bytes send(bytes data) RPC (over which would continue to pass your byte-level messages), or implement a more semantic interface -- your choice. Either way, you'd benefit from layer5's handling of reconnects, authentication, and the rest of the list.

On HTTP

Truth is, we sort-of already have such a "general purpose" layer 5 protocol, called HTTP. By a strange twist of fate, HTTP has become the de-facto application layer of choice, over which all kinds of protocols now operate. And we've managed to hide the gruesome details of HTTP under programmer-friendly libraries and APIs, so we are in a "better shape" now. Yet HTTP is such a miserable choice for this purpose (hence abominations like Websockets arose), and we all pay the price (programmatically-speaking, but also in the sense of network bandwidth, CPU time and electricity bills).

In my opinion, HTTP owes its success to another misfortunate happening - firewalls. In the days of yore, people thought they could eliminate threats simply by blocking all TCP ports, except for safe/trusted ones. HTTP was considered safe, as it only transfered "hypertext", so all firewalls allowed port 80 traffic by default. This fact has led to many protocols being designed to work on top of HTTP, to be firewall-friendly... which meant firewalls no longer served the purpose for which they were conceived: blocking ports was not enough, so firewalls had to become content-aware anyway. Long story short -- we've only managed to push the problem one level up: instead of solving it in the forth layer, we now do it in the application layer... no matter what the port number is.

However, this initial edge that HTTP had, helped in making it the de-facto transport layer of choice, which of course had a snowball effect. I'm only happy to see competing protocols like ZeroMQ and AMQP are starting to take some market share. Down with HTTP!

On RPyC

Coming back to Gavrie's post, he brings up two additional points. The first:

In addition, its performance can quickly deteriorate: Objects are being serialized back and forth all the time, and tens of implicit network round-trips introduce latency all around the code.

Performance is a tricky thing. First, RPyC is mostly ever used on local, secure networks, where latency and round-trip time (RTT) are low -- so unless you do something really flawed, you shouldn't experience noticeable degradation. Second, the only places that do suffer from RTT are tight-loops, and to that end, RPyC already has solutions. And thirdly, transparency (like any form of abstraction) hides the underlying complexity, which means you won't be able to optimize all the way. RPyC makes a choice for simplicity and pythonicity every time, at the expense of performance.

From my many years of using RPyC, I must say I've never experienced performance issues that didn't originate from the use of threading and locks in python, or really bad code. And if the times are tough, you can always apply lightweight optimization techniques, such as locally "caching" remote objects that were obtained through a series of lookups, in variables (e.g., myfunc = w.x.y.z.myfunc)... it's normally not that hard. I'm sure Gavrie has experienced performance issues with RPyC, but I can hardly imagine it could not be solved by reasonable amounts of such refactoring.

Which brings us to the last point Gavrie makes:

I don’t like RPC, especially not stateful RPC that supports access of remote objects by reference

I hope we already agreed that a general-purpose RPC is equivalent (if not better than) to any "normal" network protocol, so it's really not RPCs that Gavrie hates but stateful / object-proxying sessions. This invites another, rather philosophic, question: what is state? What does it mean that a protocol is stateless? I'd guess philosophies like REST come to mind, but that's just a buzzword. From the broadest Turing-machine perspective, if REST or any other protocol were truly stateless, they would have no effect on the world and thus would be of little significance (they'd be read-only protocols). Just to stress this point -- PUT/POST-ing to a RESTful interface, adding/altering a record in a database table, is clearly stateful: you changed the state of the DB.

Therefore, these buzzword-rich protocols boast themselves with the term stateless, while they mean something very different. In lack of a better term, I'd use atomicity, durability, and reboot-ability -- which we'll discuss next. And just a last bit of REST: REST has a notion of idempotency, as GET requests for the same URI should always return the same result (but that's not guaranteed). Anyway, the CRUD model which REST employs is quite limited and fits only so many real-life problems (many other problems can be reduced to CRUD, but I don't suppose people would consider this the "right way").

Atomicity and durability come from the ACID philosophy of databases, and are granted to you freely, assuming you use a DB (who doesn't?). They basically mean that a transaction either fully happens (and then its permanently stored), or nothing happens (so that no partial results may exist). Reboot-ability is a term I just made up, and it means your server might crash and be restarted, or your entire lab might burn away, and the client shouldn't be able to detect any difference (other than temporal unavailability, which may be compensated for by a cluster). Inherently, it means you don't trust your server to survive over long periods of time, and therefore prefer to make it (the server process) stateless. In effect, it means the server will never make "changes to the world" outside of DB transaction, so that a failed transaction could be rolled back, and a new server process could resume where the previous one failed. But note the difference: the server process is stateless, not the protocol.

HTTP originally was a connection-less protocol (albeit over TCP), where each request was treated separately from the rest and there was hardly any notion of a session. This meant that every request had to carry along with it any state information it required. In order to prevent requests from growing wildly in size and choking the network, cookies were invented -- which meant the server had to store session data, and the cookie was only a key. This already dents the notion of statelessness, and nowadays, things like websockets break it completely.

RPyC, on the other hand, has a clear notion of a session: on each end of the the connection there's a dictionary of objects referred to by the other side. This means that should a connection drop, there's little chance of restoring the lost session: the dictionaries are lost, and all proxies would be invalidated.

But all is not lost: if you only need references to serializable objects, you might as well keep this dictionary as a DB table. Since the data, including object IDs, would not be lost when the server gets restarted, resuming a dropped session is easy. So, if you could live with limited functionality, you can be backed by a DB -- but it's not that HTTP offers a better solution. At least keep your code pythonic and not full of HTTP curses.

Another alternative, which I'll demonstrate in the next section, is making the changes directly "in the real world": instead of storing state, make the changes on long-living entities, and then read the info back from them. This way, you're always synchronized, and should you be restarted, you'll never use stale data.

On Testing

I'm working now on a testing framework of quite a complicated nature: first, it serves as a resource-allocator for hosts and other testing equipment; second, in order to run tests, it must create a suitable environment for them. But this is where it gets fun: in order to do set up the environment, I must use the utilities that I set out to test... because that's exactly what they do. Chicken and egg, anybody?

After a couple of days to toying with it, I settled for the following architecture:

Tests are written normally using unittest, but they also make use of a little module (8 LoC or so) that provides them with a means to connect to the resource allocator; this module basically hides the details of setting up an RPyC connection (as the server is well known, etc).
The resource allocator exposes a simple service, with methods like get_system(version = 17). It collects the information about the systems from a third-party service and caches it in-memory, so finding a matching system is quick and efficient. Basically, the resource allocator only takes care of distributing systems randomly (we do want to allow for two tests to run on the same system, but wish to avoid unnecessary contention).
The object returned by get_system is an instance of a peculiar class called HostViewOfSystem. It basically represents a how the host (running the test) views the system that's been allocated to it, and it has methods like get_resource_from_system() that look for an unused resource (or create one) and take care of making it usable by the host.

There are quite a few details, but I hope I managed to make the design clear. On the other hand, it doesn't seem particularly interesting -- until we get to the last bullet-point -- making the resource usable by the host. In order to do that, the server (resource allocator) creates a temporary directory on the host, onto which it copies (over RPyC) several python packages that are required for the task. It then fires up a new RPyC server on the host, and sets its PYTHONPATH to this temporary location. This RPyC server is based on the fresh-from-the-oven OneShotServer, which is capable of serving a single client and then quits. The server chooses a random port and reports it over stdout, to the resource allocator, who then connects to it. Then, the HostViewOfSystem object is given a reference to the newly created RPyC service, and it uses it to manipulate the host machine in order to set up the environment. Here's a sketch:

+---Test Host---+            +---ResAlloc Server---+
|               |            |                     |
|  -----------  |  .-----------> HostViewOfSystem  |
|  | Test    |____/          |     |               |
|  | process |  |            |     |               |
|  -----------  |     _____________/               |
|               |    /       |                     |
|  -----------  |   /        |                     |
|  | Newly   <-----*         |                     |
|  | started |  |            |                     |
|  | RPyC    |  |            +---------------------+ 
|  | server  |  |
|  -----------  |
|               |
+---------------+

This quite tiresome setup actually takes only ~30 lines of code (and around one second to build), and it allows the tested utilities to rely on stable versions of themselves. The stable versions are fetched from the resource allocator server, thus we make absolutely no assumptions on the state of the host. Moreover, when we "allocate" a resource for a specific test, we mark the resource on the system as "in used": it's neither stored in-memory, nor in a DB -- it's marked directly on the resource, as metadata. This way, if the resource allocator is restarted, no state is lost -- the new instance will read the most up-to-date state from the "real world". There -- a semi-stateless testing framework based on RPyC... and now I go to bed.

Reed-Solomon Codec

2012-06-08T00:00:00-07:00

Some Background

I'm working on an image processing project for the university, whose purpose is to embed (an extract) a print-scan resilient watermark into an image. This project has (sadly) gotten me acquainted with Matlab, from which I quickly ran way into the friendlier realms of Scipy and friends (Skimage rocks, by the way). I must say I really learned to appreciate the Scipy/Numpy gang in the last two weeks :)

If it wasn't already obvious, it's time to admit I'm a n00b when it comes to signal processing and applied math in general. I know the Fourier transform in broad terms, and have heard of discrete cosine transform and wavelets some time ago... but it's not my cup of tea, to say the least. Luckily for clueless people like me, Matlab (and its kin) enables us to summon the dark powers of mathematics, without ever having to know what we're doing. Hurrah!

So I'm DFT'ing, DCT'ing and DWT'ing like a pro, embedding my watermark using CDMA/spread-spectrum techniques in the frequency domain, and then inverting the process... and all I know is I'm supposed to keep myself in the mid-frequency range, i.e., in a certain region of the matrix. Math for n00bs.

My original idea was to take a string, encode it as a QR code, and then embed this QR image into the host image. I thought it would be a nice shortcut, as it provided me with a synchronization pattern and error correction out of the box, but it quickly turned out QR codes generate a payload that's too big for unobservable embedding. So I set out to find some error correcting code (ECC) library for python, but it proved to be a really difficult task. I found some packages, most of them are haven't been maintained in over 7 years nows, and all of them make use of extension modules that failed to compile. Then there's zfec, but for the life of me, I couldn't figure out how to use it as a simple encoder/decoder.

The Library

I almost gave up and resorted to triplicating my payload (at the bit level), and using majority-selection for each bit, when I came across an amazing python tutorial (with runnable code) that covers Reed Solomon codes and QR in depth. I simply extracted the code, added a usable API, wrote some examples and quickly uploaded it to PyPI, so now there's a pure-python Reed-Solomon encoder/decoder: pip install reedsolo.

The library should support python 2.4-3.2, using strings or bytes. I really can't verify the correctness of the algorithm (it's beyond me), but it seems to work so I'm fine with it. Here's a short demo:

>>> from reedsolo import RSCodec
>>> rs = RSCodec(10)     # 10 bytes of ECC will be added to the output, 
...                      # which allows us to correct up to 5 byte-level errors
>>> rs.encode([1,2,3,4])
'\x01\x02\x03\x04,\x9d\x1c+=\xf8h\xfa\x98M'
>>> rs.encode("hello world")
'hello world\xed%T\xc4\xfd\xfd\x89\xf3\xa8\xaa'
>>> rs.decode('hello world\xed%T\xc4\xfd\xfd\x89\xf3\xa8\xaa')
'hello world'

Now let's add some errors:

>>> rs.decode('hXXlo worXd\xed%T\xc4\xfdX\x89\xf3\xa8\xaa')     # 4 errors - ok
'hello world'
>>> rs.decode('hXXXo worXd\xed%T\xc4\xfdXX\xf3\xa8\xaa')        # 6 errors - fail
Traceback (most recent call last):
  ...
reedsolo.ReedSolomonError: Could not locate error

It's pure python and highly unoptimized... I think someone acquainted with Numpy a little more than I am could improve it blindfolded by a factor 10, but even now, on my dinosaur machine, it encodes a 400kB message in 2.9 seconds and decodes it in 1.9 seconds. I'll drink to that. By the way, it seems that the library can only handle messages that are less than 255 bytes long... but then you can simply encode/decode in chunks. I'll include it in later versions.

I think a good ECC library for python is very useful... if anyone wants to join in on it, feel free to drop me a line at the comments or just fork the repo.

Some Notes on RPyC 3.2.x

2012-06-06T00:00:00-07:00

As I said in the previous blog post, I hoped for v3.2.2 to be the last release of the 3.2 line... Naturally, I was wrong :) Turns out the fix for issue #76 was buggy, and I decided to finally remove the use of excepthooks in favor of taking care of remote traceback chaining in the exception class' __str__ method itself. The tentative release date for the release is August 1st, and I really hope to end the 3.2 line here.

I created a branch called LTS3.2 which will be used only for back-porting bug fixes from the master branch, which has now become the development branch of 3.3. I rebased master on top of LTS3.2 now, so that the two would have linear histories, which of course resulted in a forced update, so anyone who was using master for development would now have to do git fetch origin; git reset --hard origin/master instead of a simple git pull. It's not supposed to happen again, sorry.

There shouldn't be any more features in v3.2.3, so the development would take place on master, from which I'll cherry-pick bug fixes onto LTS3.2. That's all for now...

RPyC 3.2.2 Released

2012-06-01T00:00:00-07:00

This is a maintenance release, fixing some issues concerning introspection, ForkingServer and signals, IronPython and signals, and SSH on Windows. It also introduces optional logging of exceptions that occur over the RPyC connection to the server's logger (or any other logger instance, given in the connection's configuration). The change log has more details.

Note: This is the last release of the 3.2 branch. Future releases (of the 3.3 branch) will require plumbum, thus the SSH support will be removed from the codebase. The next version will also provide zero-deploy, which will make RPyC much easier to use. This release is expected in September.

The Future of Construct

2012-05-16T00:00:00-07:00

It's been a long while since I've put time into Construct. I gave up on it somewhere in 2007, right after the release of v2.0... I think I just got bored, and felt like the library was complete and extensible enough to survive on its own. Of course I was wrong there, and code-rot had spread all over.

Luckily for us, Corbin Simpson took up the project in January of 2011 and has been maintaining it since then. He migrated it to github, changed the project to use a proper directory structure, fixed lots of bugs, and wrote extensive documentation. Since then, Construct has been building a solid community and has reached quite a remarkable number of downloads on PyPI.

All this time, I've been busy with my other projects, but I kept toying with the idea of Construct 3. I got some sketches, wrote some early drafts, cleaned up the implementation of Construct's core... but it's remained a dream. A couple of months ago I decided I'd back-port a nice feature from Construct 3, called this expressions, and it has rekindled my interest in the library.

`This` Expressions

One of the goals of Construct 3 was to generate efficient (C/Python) code from construct definitions. It even worked, to some extent: for instance, see this snippet that automatically generates this C-code. I had no recollection of this until I discovered it today. Funny.

Anyway, Construct 2 uses lambda functions to represent dependencies between constructs, e.g.

s = Struct("LV",
    UBInt8("length"),
    Bytes("value", lambda ctx: ctx["length"]),
)

and since this is the case, it's impossible to translate dependencies to C. So I've created the this object, which essentially builds an expression tree from native Python expressions. In order to evaluate the expression, you simply invoke it with a context:

>>> from construct import this
>>> this.x * 2 + 3
((this.x * 2) + 3)
>>> (this.x * 2 + 3)({"x":7})
17

So now we can replace all lambda ctx: ctx["foo"] by the more succinct and readble this.foo -- the benefits are visually clear -- but they go even deeper than this: since we're no longer dealing with black-box lambda functions, we can drill down into them and generate the appropriate (static) code.

Construct 2.5

I had to do some Construct work recently, and I missed the conciseness of this expressions, so I took the time to back-port them to Construct 2. I sent a pull request to Corbin, but he's a bit too busy to maintain the library on a regular basis now, so he's created a github organization and moved the repository there; this is where Construct will be developed from now on.

Tinkering with the old code again got me sentimental, and I started to do some long-awaited maintenance. I plan to release version 2.5 (note the dramatic shift from 2.06 to 2.5) in the summer (say August), and here's the list of planned changes:

Adding this expressions
Dropping construct.text -- it's always been an experimental feature and it's achingly inefficient. If you want to parse grammars, you should use more adequate tools
Adding Python 3 support (based on the work of Eli Bendersky); the library will support Python 2.5-3.2, using six
General cleanups and optimizations
Closing the wikispaces site in favor of readthedocs

Construct 3

The next big thing(TM), Construct 3, is still far away. I've got lots of cool ideas, but time is too short (as so is my ability to concentrate on one thing). Generally, the guiding thought is to modernize the library and make it even yet more compact and efficient, while removing magic along the way. For instance, because Structs require their sub-elements to have a name, and due to the fact keyword-arguments in Python are unordered, all constructs ended up taking a name argument (even though it's usually meaningless to them, as in UBInt8("length")). This has given birth to all sorts of bastards like Rename and Alias; from now on, it'll be simpler:

s = Struct(
    Member("length", UBInt8),
    Member("value", Bytes(this.length)),
)

A second issue is, laying the grounds for code generation, thus converting all dependencies to use this expressions, and perhaps even limiting the power of Adapters. Or at least, making a clear distinction between the constructs that can be turned into code and those that can't.

And last but not least, I want Construct 3 to come with a designer, where you would drag-and-drop constructs, group them in "boxes", connect them to each other (instead of this.length, you'd connect the Bytes' count field to the source construct), etc. And most importantly, you could try it out live on a data sample, and see how it breaks it up. I made this sketch here (click to download the PowerPoint slides) to demonstrate how it might look:

I think it would be very powerful.

Closing Words

I'm mainly writing this post to inform everyone of Construct's new repository, and to ask for feedback on the plans for v2.5... Any thoughts, requests, comments would be appreciated. Also, if anyone wants to join in (especially with Construct 3 ambitious plans) -- feel free to contact me.

Introducing Plumbum - Shell Combinators

2012-05-12T00:00:00-07:00

It's been a while since I last blogged... sorry! Had a midterm exam, a seminar project to deliver (an O(n^3) parser for Tree Insertion Grammar), the routine family festivities of Passover, and this new thing, Plumbum, that has been keeping my mind overclocked while I should have been studying for exams and writing seminar papers. But now that it's out, it's about time I write a little on it.

So Plumbum is something I've toyed with for quite some time now. In almost any sort of project, there comes a time when you have to write this really simple, 5-liner shell script, just to build your project's artifacts, maybe also upload them to PyPI or sourceforge (using rsync), and while you're at it, why not build the project's documentation as well. Oh, and don't forget to run regression tests before all that, and it would be wise to also handle some basic command-line options, as you might want to skip uploading to PyPI at times, or perhaps build on different versions of python... and then you end up with this monstrous shell script that you wrote (or worse, some former employee wrote) and never want to lay your eyes on again... At this point to begin hating yourself for not doing it in python, to begin with.

But then again, how do you translate find . -name "*.pyc" | xargs rm or cp */*.py /tmp to python-speak? That would take quite a few lines and require importing several modules. Shell scripts tend to be so short and enchanting... How do we bridge the gap?

Plumbum was born to fill this very gap: on the one hand, be pythonic (and be backed by strong libraries), and on the other, make it all as easy and one-liner-ish by nature: use a real programming language, with a well-behaved object model and high level concepts, to achieve what you'd normally do in a shell script, retaining the same expressive power and wrist-handiness of the shell. I call it shell combinators, as it's a pythonic way to mimic shell syntax.

The library is actually a collection of utilities that I wrote for several separate projects, and had never got to polish them. Plumbum consolidates them into a single, production-grade framework. The library provides local and remote program execution, with support for piping and IO redirection; local and remote file-system paths abstraction; a programmatic command-line interface (CLI) toolkit, and numerous other utilities.

For instance,

from plumbum import local
from plumbum.cmd import wc, ls, echo, grep
from plumbum.utils import copy, delete

delete(local.cwd // "*/*.pyc")

for fn in local.cwd / "src" // "*.py":
    print wc("-l", fn)

num_of_src_lines = (ls["-l"] | grep["\\.py"] | wc["-l"])()

(echo["1"] > "/proc/sys/net/ipv4/ip_forward")()

There's a short cheat-sheet as well as extensive documentation on the project's site, but at this point I'd like to elaborate a bit on the CLI toolkit, as I think it deserves some more attention.

The approach of optparse / argparse and similar libraries is to build a parser object and populate it with options in an imperative manner, which I dislike. Plumbum's CLI toolkit offers a more declarative yet programmatic alternative: An application is defined as a class that derives from cli.Application; it may define a main() method, which serves as the "entry point" of the application, and any number of switch-methods, meaning, methods that are invokable from the command-line.

Switch methods are normal methods, decorated by @switch, that may take either no arguments or a single one; for each switch given on the command line, the toolkit will invoke the switch method that binds it. Similarly, the main() method is invoked after all switches have been processed, and it takes all the positional arguments (i.e., non-switch arguments) that were given. And last but not least, there are switch attributes, which are in fact just specialized versions of switch functions, that store an argument given to the switch in an instance attribute.

So I've probably only gotten you confused by the terminology at this point, but actually it's much simpler!

from plumbum import cli

class MyHttpServer(cli.Application):
    log_to_file = cli.SwitchAttr("--log-to-file", str)
    verbose = cli.Flag("-v")
    mode = cli.SwitchAttr("--mode", cli.Set("TCP", "UDP"), default = "TCP")
    port = cli.SwitchAttr("--port", cli.Range(1024, 65535), default = 8080)
    
    @switch(["-l", "--load-config"], cli.ExistingFile)
    def load_config(self, filename):
        """Loads the given config file"""
        f = open(filename, "r")
        self._parse_config(f.read())
    
    def main(self, src, dst):
        if self.log_to_file:
            logger.addHandler(FileHandler(self.log_to_file))
        logger.setLevel(logging.DEBUG if self.verbose else logging.WARNING)


if __name__ == "__main__":
    MyHttpServer.run()

There, I think it's a lovely example of the expressive power of the CLI toolkit and Plumbum in general. I hope you'll give it a try, and may you never have to write shell scripts again!

Solving Systems of Linear Equations

2012-03-25T00:00:00-07:00

Yet another university-related post, but I really enjoyed it so I thought I'd share: for a GUI- workshop I'm taking, we are given GUI-layout constraints as a system of linear equations, which we need to satisfy. To make life more interesting, some constraints are constant while some are parametric. There's no magic here, just some linear algebra combined with Python's overloadable nature to produce a nice and compact solver for these linear systems.

Naturally, you present your system as a coefficient matrix, which is Gauss-eliminated and then "solved" for a given set of variables. I took the Gauss-Jordan elimination code from Jarno Elonen and modified it to support MxN matrices (not necessarily square). Here's a simple example of elimination:

>>> m = Matrix([1,2,4,2], [3,7,6,8], [2,2,2,9])
>>> print m.eliminate()
( 1.00  -0.00   0.00   6.00)
( 0.00   1.00  -0.00  -1.00)
( 0.00   0.00   1.00  -0.50)

But of course a reduced row-echelon matrix is not our final goal - we want to get the variable assignments. For this, we have solve():

>>> sol = solve(m, ["x", "y", "z"])
>>> print sol
{'y': -1.0000000000000004, 'x': 6.0, 'z': -0.4999999999999998}

Modulo precision errors, we get x = 6, y = -1 and z = -0.5. If we have more equations than variables, the "extraneous" equations must be linear combinations of previous ones, or a contradiction will result. But what if we have less equations than variables? It means we have some degrees of freedom... how would we handle that? It's actually simple - instead of being resolved to constant values, variables will be assigned dependent expressions:

>>> m2 = Matrix([1,2,4,2], [3,7,6,8])
>>> print m2.eliminate()
( 1.00   0.00   16.00  -2.00)
( 0.00   1.00  -6.00   2.00)
>>>
>>>
>>> sol = solve(m2, ["x", "y", "z"])
>>> print sol
{'y': <BinExpr (2.0 - (-6.0 * z))>, 'x': <BinExpr (-2.0 - (16.0 * z))>, 
    'z': <FreeVar z>}

As you can see now, z is a free variable and x and y are dependent on it. Of course more than one variable may be free and some variables may be independent of free variables. Once a value for z is known, we can "fully evaluate" the dependent expressions:

>>> sol["x"].eval({"z" : 10})
-162.0

The code is available on my github page. Note: all numbers are represented as Decimals, to avoid loss of precision as much as possible, and I'm using an "epsilon" value of 1E-20 to equate numbers to each other (meaning, x == y iff abs(x-y) <= epsilon).

Easy Syntax for Representing Trees

2012-03-07T00:00:00-08:00

I'm working on a parser for Tree Adjoining Grammar (TAG) for this seminar I'm taking. TAG is an extension of context-free grammar (CFG) that's more powerful while still being polynomially-parsable. Anyhow, TAG makes use of "tree production rules" instead of the "linear" production rules of CFG: instead of S -> NP VP, you'd have a small tree, the root of which being S, having NP and VP as its children. Of course these trees can be more than two-level deep, and they go all sorts of operations such as substitution and adjunction, but that's for the parser.

So I needed a compact and (hopefully) readable way to express such trees in my code. At first I used lots of parenthesis, which was ugly and cumbersome, but then I devised this:

class NonTerminal(object):
    def __init__(self, name):
        self.name = name
    def __sub__(self, children):
        return Tree(self, children)
    def __pos__(self):
        return Foot(self)

class Foot(object):
    def __init__(self, nonterm):
        self.nonterm = nonterm

class Tree(object):
    def __init__(self, root, children):
        self.root = root
        self.children = children

And here's how you use it:

S = NonTerminal("S")

t1 = S-["e"]
t2 = S-["a", S-["c", +S, "d"], "b"]

# Which represents the following two trees:
#
# t1:                t2:
#      S                  S 
#      |                / | \
#      |               /  |  \
#      e              a   S   b
#                       / | \
#                      /  |  \
#                     c   S*  d
#

The peculiar +S is a way to mark that a leaf node is a foot, which is part of the semantics TAG (that's where adjunction takes place). It's represented in the diagram by the more conventional S*, but I had to resort to a unary operator in the code. Anyway, I'm not sure if it's a recipe or just a nice trick, but I thought I'd share this.

By the way, you can use both __sub__ and __neg__ to achieve things like S-----X (i.e., more than a single - sign, to allow for better padding), but I tried to avoid too much ASCII art. I'd love to hear about other such ideas!

RPyC 3.2.1 Released

2012-03-04T00:00:00-08:00

This is a maintenance release, fixing some minor bugs and resolving some issues with Python 3 compatibility. More on the change log.

Python 2.x: it's advisable to upgrade to this version.

Python 3.x: it's highly recommended to upgrade to this version, as it resolves some core issues that went under the radar in 3.2.0.

Toying with Context Managers

2012-02-27T00:00:00-08:00

As I promised in the code-generation using context managers post, I wanted to review some more, rather surprising, examples where context managers prove handy. So we all know we can use context managers for resource life-time management: before entering the with-suite we allocate (open) the resource, and when we leave the suite, we free (close) it -- but there's much more to context managers than meets the eye.

Stacks, for Fun and Profit

We've seen this one in the code-generation post, but we can generalize this notion a bit. Every time we enter a with block, we'd append an element to a list, which we'd pop on exit. Consider this snippet:

class Stacking(object):
    def __init__(self):
        self.stack = []
    @contextmanager
    def foo(self):
        self.stack.append("foo")
        yield "bar"
        self.stack.pop(-1)

Nothing fancy, but that's exactly how the code generation framework works: whenever you enter a new block, it's pushed onto the "stack", and whenever we leave the block, the top-of-stack element is popped. This is how we automatically take care of indentation, curly-braces, etc.

But of course this pattern is much more useful. Consider something like this:

class Env(object):
    def __init__(self):
        self._envstack = [os.environ]
    @contextmanager
    def new(self, **kwargs):
        self._envstack.append(self._envstack[-1].copy())
        self._envstack[-1].update(kwargs)
        yield
        self._envstack.pop(-1)

env = Env()

with env.new(PATH = "/tmp/foo/bin", SHELL = "zsh"):
    # processes created here will use the modified environment
    pass

We can also leverage this concept to run commands as different users. Here's a sketch:

cmd.run("ls")                       # as current user
with cmd.as_user("root"):
    cmd.run("ls", "/proc")          # as `root`
    with cmd.as_user("mallory"):
        cmd.run("rm", "-rf", "/")   # as `mallory`
    cmd.run("cat", "/etc/passwd")   # back as `root` again

In essence, every time you want to make local/undoable changes to your state, this pattern proves helpful.

Contextbacks

Contextbacks, a pun on callbacks, are contexts you pass as arguments to other functions. Many times it's useful to pass a before-function and an after-function, and contextbacks are a nice way to encapsulate this. So instead of this:

def f(beforefunc, afterfunc):
    beforefunc()
    # your code goes here
    afterfunc()

You get this:

def f(ctxback):
    with ctxback:
        # you code goes here
        pass

Not a ground-breaking change, but I prefer it as it's more concise.

Lightweight Asynchronism

This is probably my favorite use case for contexts: you can use them to pipeline or interleave long-lasting tasks. You can think of contexts are degenerate forms of coroutines, in which you have defined beginning and end, but the middle part is interchangeable, so you can stick anything into it.

Imagine you need to format a harddisk (using mkfs) or perform some long network operation (like copying a huge file over scp). The pattern is as follows: initiate the operation, wait for it to finish, and either return or raise an exception. This fits perfectly well with the way contexts work -- with one change -- yield instead of waiting.

@contextmanager
def format_disk(devfile):
    proc = Popen(["/sbin/mkfs", "-t", "ext3", devfile])
    try:
        yield
    except Exception:
        proc.kill()
    stdout, stderr = proc.communicate()
    if proc.returncode != 0:
        raise FormattingFailed(stderr)

with format_disk("/dev/sda1"), format_disk("/dev/sdb1"):
    pass

This saves you time: instead of waiting for the first operation to finish before starting with the second, you can run them in parallel. The total time would be that of the longest task (not taking into account the IO bottleneck). You can throw a yield into any piece of code instead of just blocking, and use it as a pipelined contextmanager. You can copy three files in parallel, without resorting to threads or a reactor in the background.

Of course you could improve that by returning an object that reports the progress of the operation, e.g.

with format_disk("/dev/sda1") as d1, format_disk("/dev/sdb1") as d2:
    while not d1.is_done() or not d2.is_done():
        print "%s is being formatted, %d%% completed" % (d1.devfile, 
                d1.get_progress())
        print "%s is being formatted, %d%% completed" % (d2.devfile, 
                d2.get_progress())

And voila! You have a thread-less, light-weight asynchronous framework at hand... a bit like using a reactor, but without rewriting your code.

And last, if you can't tweak the blocking parts of the code (e.g., third party libraries), you can use the "defer to thread" or "defer to process" approach, a la twisted:

@contextmanager
def defer_to_thread(func):
    thd = Thread(target = func)  # it would be smarter to use a 
    thd.start()                  # thread-pool
    yield thd
    thd.join()

with defer_to_thread(task1), defer_to_thread(task2), defer_to_thread(task3):
    # do something else in the meanwhile
    pass

So that's all I had in mind. If you have other unorthodox use cases for contexts, I'd love to hear about them!

Wizard Dialog Toolkit

2012-02-11T00:00:00-08:00

Following my Deducible UI post, and following some of the criticism it had received, I'd like to share something I've been working on (read: experimenting with) at my work place. You see, we have some "interactive wizards" that storage admins use to connect storage arrays to their hosts (say, a DB server). These wizards prompt you with questions like your what's your username, the name of the pool/volume, whether it's an iSCSI or a Fiber Channel connection, etc., and then they go and perform what you've asked for.

These wizards operate in a terminal environment, but we've had thoughts to make GUI/web versions of them. This would be a considerable effort with the current design. Another issue they currently have is the mixing of "business logic" and presentation together. For instance, the code that scans the devices attached to your host also prints ANSI-colored messages or reports its progress. All in all it works fine, but there's lots of room for improvement.

I began to investigate this corner a month or two ago. The initial observation was that such wizards have a pretty rigid and repetitive structure, thus we can find some abstraction or a "toolkit" for "expressing" wizards more compactly. This has also led to the realization that once the business logic and presentation are separate, there's no reason to limit ourselves to terminal-based interaction: our wizard-toolkit could do the plumbing and work with terminals, ncurses, GUIs, web-browsers, etc. The business logic would remain oblivious, and we could have a nice GUI at zero-cost!

There was also a second issue of styling, i.e., printing text in color, that I wanted to get r id of. This part was easy: I thought, why not employ the model of HTML and CSS? Let's separate the structure (semantics) of the text from its styling. Instead of printing a banner for titles, we'll display a Title object, whose exact appearance is determined by a "style sheet" (a class, of course, not actually a text document).

For instance, when we're using a color-enabled terminal, the title would be printed in bold and followed by an empty line; but if our terminal is color-blind, we'll render the text centered and surrounded by = marks. Another example is error-handling: instead of printing error message in red every time, we'll display an Error object; on a terminal, this would be rendered as red text, but when running in a GUI, rendering this object would pop up a message box. I'm going to ignore this for the rest of this post, as this is really a side issue.

Now let's get to expressing wizards, or more generally, dialogs. Following some earlier iterations, I came to the model where a dialog is a "container object" that's made of dialog elements. These elements can be output-only (such as a welcome message), or input-output (such as a message telling you to choose one of the available options). A dialog is "executed" by a DialogRunner that renders it and returns the results gotten from the user. It's quite important to note that dialog elements within a single dialog cannot be interdependent -- that is, if you want to ask the user for his name and then show "Hi there %s" with the user's name, this has to be done as two, serial dialogs.

That was quite a lot of babble -- let's see this in action:

class MyApp(WizardApp):
    def main(self):
        iscsi = Option("iSCSI")
        fc = Option("FC")
        d = Dialog(
            Text(Title("hello world")),
            Input("un", "Username"),
            Password("pw", "Password"),
            Choice("conf", "What do you want to configure?", [iscsi, fc]),
        )
        res = self.ui.run(d)
        if res["conf"] == fc:
            self.config_fc(res)
        else:
            self.config_iscsi(res)
    ...

if len(sys.argv) == 2 and sys.argv[1] == "--gtk":
    MyApp.run(GtkDialogRunner("My App"))
else:
    MyApp.run(TerminalDialogRunner(ANSIRenderer))

It's a short and incomplete snippet of course, as I'm only going to cover the big picture. The main function creates a dialog object d and passes it to ui.run, which "runs" the dialog and returns the results, as a dictionary. Notice that the dialog elements Input, Password and Choice all take a first parameter -- this is the key under which the result would be placed in the returned dictionary, e.g., res["un"] would hold the user-provided user name, and res["pw"] would hold the password. Text, on the other hand, is an output-only element, so it doesn't return anything and doesn't take a key. Long story short, we're asking the user to enter some information and choose one of two options, and then continue processing based on the selected option. At the bottom, we determine how to run the application based on a command-line switch: if --gtk is given, we'll run the dialogs through the GtkDialogRunner; otherwise, we'll use the TerminalDialogRunner.

And how does it look like? When running on a terminal:

And with a single command-line switch, we run as a GTK application:

So of course it's far from perfect, but then again, it's a small research project I've only put ~15 hours into. It suffers from some of the problems I've listed in the deducible UI post, for instance, the GUI hangs when the business logic performs blocking tasks. This could be solved by moving to a reactor-based model, but I've tried to keep the existing wizard code in tact as much as possible. A hanging GUI is not nice, but it's not the end of the world either, and there are numerous ways to overcome this.

Another benefit this design brings along is the ability to automate testing by using mock dialog runners. Since our business logic is only exposed to the returned dictionary, we can use a dialog runner that actually displays nothing and returns a scripted scenario each time. We can even go further: because our business logic "talks" in high-level primitives like Choice, we can compute the Cartesian product of all choices and run through each of them. We can show that we've covered all paths! And we can do this automatically... without people hitting buttons and keeping logs of their progress.

Anyway, I just wanted to show that it's feasible. I'm not releasing any code as this project is currently in very early stages, and it's something I do at work. Perhaps we'll open-source it in the future, if it proves useful enough.

Just Got Me These

2012-02-09T00:00:00-08:00

Hurrah! I just got me these:

After the CS secretariat refused to let take math courses as electives (thus forcing me into taking boring stuff like SQL), I think Dover books and I are going to become good friends. It's not like I have plenty of time to read them, but even just seeing them on my bookshelf would make me a happier person :)

Next on my wish-list are some books on logics and proof theory, category theory, information theory, and automata and computability. I'll get there some day...

WHO HAS CHEEZBURGER?

2012-02-03T00:00:00-08:00

Too much Wireshark...

Code Generation using Context Managers

2012-01-31T00:00:00-08:00

When I was working on Agnos, a cross-language RPC framework, I had a lot of code-generation to do in a variety of languages (Python, Java, C#, and C++). At the early stages, I just appended strings to a list. It was quick and dirty, and it's got the job done... but that wasn't enough, of course. I've lost the original code already, but it looked something like this:

def generate_proxy(typeinfo):
    lines = [
        "public class %sProxy {" % (typeinfo.name,),
        "    private int uid;",
        "    public %sProxy(int uid) {" % (typeinfo.name,),
        "        this.uid = uid;",
        "    }",
    ]
    for attr in typeinfo.attributes:
        if attr.get:
            lines.append("    public %s get%s() {" % (attr.typename,
                                                    attr.name,))
            lines.append("        // ...")
            lines.append("    }")
        if attr.set:
            lines.append("    public void set%s(%s value) {" % (attr.name,
                                                            attr.typename))
            lines.append("        // ...")
            lines.append("    }")
    lines.append("}")
    return lines
# ...

lines = []
for ti in typeinfos:
    lines.extend(generate_proxy(m, ti))

open("foo.java").write("\n".join(lines))

There are several problems with this approach. First of all, it's very cumbersome and fragile. If you forget a comma in the list, two adjacent strings will be concatenated. Also, you have to do everything yourself, like remembering to close brackets, add semicolons, do the right indentation, etc. If you wished to split this code into functions, the functions you call would have to know the indentation level you're calling them at, or the generated code would be unreadable. This might seem negligible, but think of languages where indentation matters, like Python...

The fundamental problem with this approach (and similar ones) is that the code generator does not reflect the structure of the generated code. The two are diseparate, while it's quite obvious they should be correlated.

In order to solve this, I turned to context managers, a feature I highly value. Conceptually, context managers provide a way to bind beginning-and-end into a single entity; this is normally used for resource management -- but we can leverage this construction further (I'll this review in a different post). Here, I've used them to create nested blocks, which allowed me to reflect the structure of the generated code in the code generator.

Without going into too many details, I defined a Module class that exposes a block() context manager and a stmt() function. The module holds a "stack" of blocks, and entering a new block pushes a it onto the stack. Statements are then appended to the topmost block on the stack. Now, because this framework is "language-aware", it can encapsulate language-specific details. For instance, In Java, a block will be indented correctly and wrapped by brackets; in Python, we'll append colons to the opening line and indent the block; in C++, if the block begins with class, struct or enum, we'll append a trailing semicolon as well.

Here's how it works:

m = JavaModule()
m.stmt("import foo")
m.stmt("import bar")
m.sep()   # an empty line

#...

def generate_proxy(m, typeinfo):
    BLOCK = m.block
    STMT = m.stmt
    
    with BLOCK("public class {0}Proxy", typeinfo.name):
        STMT("private int uid")
        with BLOCK("public {0}Proxy(int uid)", typeinfo.name):
            STMT("this.uid = uid")
    
        for attr in typeinfo.attributes:
            if attr.get:
                with BLOCK("public {0} get{1}()", attr.typename, attr.name):
                    pass
            if attr.set:
                with BLOCK("public void set{0}({1} value)", attr.name, 
                                                            attr.typename):
                    pass
# ...

for ti in typeinfos:
    generate_proxy(m, ti)

m.render_to_file("foo.java")

So what have we gained?

The code is much shorter and more concise
Brackets, semicolons and indentation come out-of-the-box
We're no longer working with flat lists of strings -- we're working with hierarchal entities that reflect the structure of the generated code
And the other way around -- the structure of the generated code is reflected in the generating code; nested code is indeed nested inside BLOCKs, thus the "generatee" and generator are visually and semantically correlated.
We can easily split our code into functions, as the module maintains an internal stack. If f() opened a block and called g() under it, it the code that g() generates will be placed and indented correctly.

I tried to keep my code quite general, so I haven't defined all of the target language's constructs, but of course we could do that, or at least head in that direction. It might look like this:

def generate_proxy(m, typeinfo):
    with m.CLASS(typeinfo.name + "Proxy", ["public"]):
        m.FIELD("int", "uid", ["private"])
        with m.CTOR(["int uid"]):   # CTOR gets the name of the current class
            STMT("this.uid = uid")

However, there's a question of where we "put our foot down", or we'll end up writing Java Combinators for Python... and then we'll be writing Java in Python. No need for that, thank you very much.

The full source code can be found in the Agnos repository

Deducible UI

2012-01-27T00:00:00-08:00

A Brief History

I like automating things. I don't like having to reiterate myself: my dream is to always be able to add only the necessary amount of information in order to make something possible. This is one reason, for instance, why I hate expressions like ArrayList<String> x = new ArrayList<String>();... it always makes me feel like I'm talking to a retard (compiler).

In 2006/7, I wrote some demos for RPyC to show how easy network-related tasks become. I chose something rather complex, a chat client, to show that all the code sums up to a few lines: clients invoke a method on the server, say broadcast(str), and the server then invokes callbacks on all of its clients, sending them the message.

In order to make it usable, I had to write a GUI: I chose Tk, because it comes with python and is quite simple; I knew there were better toolkits, but my GUI was meant to be basic enough to be doable in any toolkit. It occurred to me, then, that I wrote ~20 lines show-casing RPyC and ~100 lines of horrible GUI code, and that something must be really wrong here. And by here I mean everywhere.

Note: throughout the article, I'm using the word GUI to mean any interactive user interface, be it graphical (Qt, GTK, wxWidgets, ...) or terminal-based (curses and the like). Basically, anything that doesn't block on a single line of input, like shells.

GUI Designers

So you might say, "Dude, just use QtDesigner or something". A GUI designer lets you visually place components and makes your life much easier -- drag and drop your widgets and double-click on a button to write its action. Very easy indeed. But I would like to offer a different angle on the subject: just like the invention of the teacup has hindered the technological advance of China, so do GUI designers hinder us from developing better GUIs. These designers offer a local optimum which we fail to surpass, and this leaves us with the mediocre UIs and development tools we have today. And get me going about XAML.

Think about it: you have to design a GUI. So yeah, it's kind of simple, but doesn't that break DRY? You have the code and you have the GUI -- two faces of the same idea. Obviously, one should be derived from the other.

For the lion's share of programs, the UI is highly deterministic -- there's some information that needs to be displayed to- or gotten from the user, and the bindings is trivial. Consider a login screen: you want to get a username and a password, use them somehow, and proceed to the next screen. This is a repeating task, and I'd guess that for ~80% of the programs in the world, it's easy enough to automatically deduce how the UI should look, given the task at hand. And I'm not talking about machine learning algorithms or designing "families of tasks" -- way simpler than that! Just define a mapping between programmatic primitives and their visual representation.

Deducing UI

The ultimate goal is to take "pure code", unaware of UI, and by adding the necessary metadata, be able to automatically create ("deduce") a GUI for it. In fact, I'd like to expose programmatic APIs to a human -- completely interchangable programmatic- and human- interfaces. Think how cool it could be to import Adobe Photoshop and run a directory full of pictures through its filters, instead of doing so through the UI... without Adobe having to write a separate GUI and programming toolkit.

The UI needn't be an eye-candy, at least at the beginning; it just has to be good-enough. It won't work for games or complex applications like Office, but for it would be just fine for a chat client. Let's assume the following mapping:

An object is represented by a window
Read-only instance attributes are represented as labels
Writable instance attributes are represented as textboxes
Methods are represented by buttons. If a method requires arguments, it would be preceded by textboxes

Of course we could change textboxes and labels to reflect the attribute's or argument's type -- DateTime would be represented by a DatePicker, int could be represented by a number box with up/down arrows, etc. And of course the framework is free to change the mapping however it wants, to achieve better, more coherent representation. The mapping above is just a rough draft.

Now, instances of a class like the following:

class Person {
    public String firstName;
    public String lastName;
    public DateTime birthdate;

    public void dance() {...}

    public void eat(String foodstuff) {...}
}

would turn into

with just a little bit of binding in the form of:

class Main {
    static public void Main(String[] args) {
        Person p = new Person(...);
        guify(p);
    }
}

Straightforward, isn't it? You can already begin to see the benefits. This framework would obviously require some sort of decoration (annotations in Java, attributes in C#, ...) on which classes and which class members are to be exposed, and perhaps some extra metadata, like a picture to show instead of a method's name, or some layout information -- but it's perfectly doable. And we can turn it better looking (unlike my beautiful ASCII art example), by using better UI primitives and a better mapping between objects and their representation; but let's leave it for now.

I wrote a a simple prototype of this and lo and behold, it actually worked! But when you try to use it in a real-life applications, the going gets tough: things are updated behind the scenes (not through our UI framework) and we need to reflect these changes in the UI. For instance, an element is added to a list via the list's add() method - how can the GUI become aware of that? Well, we can use observable objects, which the GUI would observe; so instead of using an ArrayList<T>, you'd simply use an ObservableArrayList<T>.

But creating an observable counterpart for every class is a considerable effort on the framework's side, and it breaks software modularity: the framework has to be aware of every 3rd party class that you wish to expose, or at least allow you to provide the means to expose them. Another downside of this scheme is that your code becomes aware of the GUI: if we've so far managed to keep our code clean of GUI primitives (we only required some metadata), all of the sudden you must replace your lists with GUI-observable-lists. Bummer.

Another issue is that using synchronous programming techniques (blocking operations) does not play well with this model: when does the GUI gets its "runtime"? How can we keep it from freezing? Who's providing the entry-point of the program? Does it run in a separate thread? If so, we risk polluting our code with GUI-related locks (which is countering the whole purpose); and besides, threads suck and add the incurred complexity is never worth it. The only feasible option is asynchronous programming (via a reactor) -- but requires that your code be programmed this way, and it's quite nasty to write such code without proper language support (e.g., lack of closures, coroutines, etc.).

"No Way"

As I said, I've been toying with this idea from 2006, and I always get the same response from colleague programmers: "it would never work", "it won't be good enough", "no one would want to use it", "users need their eye-candy", "I need tight control over the layout", and what not. Skeptics galore. My answer is always the same: you never know what your user wants -- so who are you to decide? And besides, you're always to lazy to support proper customization of UI, so your user must live with your decisions.

Sure, there are books and dissertations about UX, and you've read them all; but why not just provide good-enough defaults, and let the rest be customized by the user? Let the framework deduce a sane layout for your code -- but let's make everything movable/resizable/dockable. This way, if it makes more sense to place button X to the left of button Y, the user can do so himself. And let's remember the user's preferences in a file, which we'll load each time the application runs. And by "user" I'm also talking about your UX expert -- let him/her decide on a default look (i.e., the preferences file) for the application, which will be shipped with your product, but the end-user would still be able to move things around. Wouldn't it be easier? And if you insist, here's the place to stick some machine-learning magic, in order to deduce better UIs by default.

So anyhow, I had a working proof-of-concept somewhere, but I think I lost it. It wouldn't be too hard to recreate it, but at the moment I'm more concerned with UI combinators, which I'll cover in a future post. Fully deducing a UI is quite a challenge, as a detailed above, but it's doable nonetheless, and the added-value is huge! I'll get back to it some day, perhaps after I have better UI combinators... but in the meantime, is there anyone in the audience who's willing to pick it up?

Regarding Infix Operators

2012-01-25T00:00:00-08:00

I got some reactions to the Infix Operators post, and wanted to point out some things. First of all, I'm not the one who came up with it -- it's a recipe from the Python Cookbook that's been posted in 2005. I'm not taking credit for it or anything, I just said I loved the idea and I adapted the code a little.

Second, regarding coding style or the pythonicity of this scheme -- let's be clear, it's a hack. In order for it to work, the two arguments must refuse to support __or__ on Infix objects, which means you can't compose them properly:

>>> @Infix
... def dot(f, g):
...     return lambda *x: f(g(*x))
... 
>>> @Infix
... def mul(x, y):
...     return x * y
... 
>>> def double(x):
...     return x * 2
... 
>>> f = double |dot| mul   # this works ok
>>> 
>>> g = mul |dot| double   # but this won't
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in __or__
TypeError: mul() takes exactly 2 arguments (1 given)

It's also magical and unpythonic by nature. You can read in the cookbook comments about a less-experienced programmer who complained he had to wrap his head around around this. I'd say this feature is a kin to metaclasses: they are useful (at times), but there are better, more pythonic ways to do the same without the magic.

So why is it useful? First of all, it's an interesting pattern, worth more knowing about than actually using it. But on the more practical side, it could be very useful in domain-specific languages (DSL), where it could increase your expressiveness. Consider something like Construct, where you define data structures declaratively:

ipaddr = Struct("ipaddr",
    UInt8("a"),
    UInt8("b"),
    UInt8("c"),
    UInt8("d"),
)

lenval = Struct("lenval",
    UInt8("len"),
    Bytes("val", lambda ctx: ctx.len),   # interdependencies: "val" is `len` bytes long
)

We could replace all sorts of built-in constructs by such "operators", thus building "data expressions". So here's a very early sketch of what it could look it:

ipaddr = UInt8 |seq| UInt8 |seq| UInt8 |seq| UInt8

# or maybe just
ipaddr = UInt8 |repeat| 4

# binding names into the context
lenval = UInt8 |bind_seq("len")| Bytes(getctx("len"))

Beware: I just made this up, there's no solid concept behind it.

All Systems are Go

2012-01-23T00:00:00-08:00

At last, I finished migrating the old drupal site to github pages. Everything is now fully revisioned and statically-generated (using Disqus for comments). Jekyll is so cool! I wrote all the HTML and forged the stylesheets myself... hope you like it.

Anyhow, I'm happy with the design now, and I'll start blogging more regularly. I still have plenty of pages to complete (projects, about-page, etc.), and a few partially-written blogs posts to polish up, but other generally speaking, we're up.

If you have any feedback about the design/site, please let me know in the comments below or via email. Thanks!

Foxx0rz

2012-01-22T00:00:00-08:00

This is Foxx0rz, our course' mascot. He's written in C (compiles under MSVC++ 6 to be exact), and we had it printed on T-shirts... 'twas fun.

Here's a downloadable version

And this is the output:

Infix Operators in Python

2012-01-22T00:00:00-08:00

As you may already know, there are 3 kinds of operators calling-notations: prefix (+ 3 5), infix (3 + 5), and postfix (3 5 +). Prefix (as well as postfix) operators are used in languages like LISP/Scheme, and have the nice property of not requiring parenthesis — there’s only one way to read an expression like 3 5 + 2 *, unlike 3 + 5 * 2. On the other hand, it reduces code readability and the locality of operators and their arguments. This is why we all love infix operators.

Now imagine I have a function, add(x,y), and I have an expression like add(add(add(5,6),7),8)... wouldn’t it be cool if I could use infix notation here? Sadly though, Python won’t allow you to define new operators or change how functions take their arguments... but that doesn’t mean we have to give up!

Haskell, for instance, allows you to define custom operators and set their precedence, as well as invoking "normal" functions as infix operators. Suppose you have a function f(x,y) — you can invoke it like f 5 6 or 5 `f` 6 (using backticks). This allows us to turn our previous expression, add(add(add(5,6),7),8), into 5 `add` 6 `add` 7 `add` 8, which is much more readable. But how can we do this in Python?

Well, there’s this Cookbook recipe that provides a very nice way to achieving the same functionality in Python (adapted a little by me):

from functools import partial

class Infix(object):
    def __init__(self, func):
        self.func = func
    def __or__(self, other):
        return self.func(other)
    def __ror__(self, other):
        return Infix(partial(self.func, other))
    def __call__(self, v1, v2):
        return self.func(v1, v2)

Using instances of this peculiar class, we can now use a new "syntax" for calling functions as infix operators:

>>> @Infix
... def add(x, y):
...     return x + y
...
>>> 5 |add| 6
11

Surrounding decorated functions with pipes (bitwise ORs) allows them to take their parameters infix-ly. Using this, we can do all sorts of cool things:

>>> instanceof = Infix(isinstance)
>>>
>>> if 5 |instanceof| int:
...     print "yes"
...
yes

And even curry functions:

>>> curry = Infix(partial)
>>>
>>> def f(x, y, z):
...     return x + y + z
...
>>> f |curry| 3
<functools.partial object at 0xb7733dec>
>>> g = f |curry| 3 |curry| 4 |curry| 5
>>> g()
12

Ain’t that cool?

לקט ציטוטי מורים

2011-12-01T00:00:00-08:00

לקט ציטוטים שאספתי במהלך התיכון. לינק ל PDF

RPyC Moves to a New Site

2011-08-29T00:00:00-07:00

RPyC is in the process of migrating from http://rpyc.wikidot.com to it's new (and hopefully final) location at http://rpyc.sourceforge.net. Wikidot had served us well, and was easy to maintain, but they started displaying way to many ads and didn't support rsync or SSH access, which meant I couldn't upload the generated API reference automatically.

The new site is written entirely in ReST using sphinx and large parts of it are auto-generated from docstrings in the source code. It's all now part of the git repository, and I only have to run make upload to upload it all up.

Learning Me a Haskell

2011-08-17T00:00:00-07:00

Phew! Finally the semester's over (just submitted my last project), and it's time to clean up my ever-so-long backlog. Here goes nothing: I'll start by posting something here, after this long while of neglect.

As I'm sure you already know, I'm a long-time Pythonista, and I'm confident enough in calling myself a "native speaker" of that language. Although python is my expertise, I'd say I'm fluent in most other prominent programming languages (say, C, C++, Java, C#, VB), and with adequate knowledge of many more. It may seem like a good set of skills, but I have to admit this brings one to a point of stagnation. There comes a time where everything just looks the same: you take a glimpse at a new language/platform/other technology and sigh, "oh well, on the surface it's different, but underneath it's all the same sh*t". You come to the conclusion that people mostly change the looks-and-feel, but nothing radical could never happen. And then you skip to the next article.

Then came Haskell: a statically-typed, type-inferred, highly-expressive functional language. I'm not new to functional programming -- I've programmed in Scheme, and I even wrote an interpreter for a toy functional-language that I made up in order to investigate nested-scopes and evaluation models -- but heck, this one's different. A friend of mine nagged me to learn Haskell for quite a time already, but I was deterred by Haskell's ugly syntax (it's not Python a'right) and I kept pushing it for "when I have time". It always seemed too academic to me, too impractical.

I think what provoked me into seriously learning Haskell is a pattern called continuation passing style -- I was just amazed by the extra power, design simplicity, and greater expressiveness that you get with it. Of course you can implement it in almost any other language, and in fact it was invented in Scheme (call/cc), but Haskell, with its purely-mathematical state of mind, just brings the best out of things. Haskell has also taught me that expressiveness and conciseness are not virtues to be underestimated: if you can do something in one line instead of four, you must be generalizing on some deeper concept that you've previously missed. It's not just looks-and-feel, this time.

But let's not go astray. I think the most prominent and well-known feature of Haskell is its excellent type system. I'm sure you've heard of static languages, like java or C++, where every expression has a compile-time type; and I'm sure you've heard of dynamic (duck-typed) languages, where there are only run-time types and no checks are (or can be) performed prior to running the code. In the first case, you curse the compiler for limiting what you can do (even when it makes perfect sense) and for being bluntly stupid, requiring you to repeat yourself over and over (FooBar x = new FooBar()). And because most type systems are so weak, you find yourself escaping to "duck-typing" techniques like void * or Object, and relying on run-time casts. In the latter case, you find yourself trying to cover the infinite space of type permutations that your functions have to handle, and soon you pray for some sort of validation or restrictiveness.

When you start learning Haskell, you suddenly realize how weak most type systems are, and that there are (much) better alternatives. It's like you've been suffering from a blurry vision your whole life, and suddenly you realize you can wear glasses. Has it ever occurred to you that type systems are equivalent to proof systems (the Curry-Howard isomorphism)? And that a compiler basically tries to prove your code? You can think of compilation errors as finding a counter-example to your claim! "You said F takes an integer, but I see you try to call it with a String"... so of course this is a trivial example, one that even a C compiler catches, but you can make more complicated claims. And then you realize that type systems and compilers ("provers") do matter -- a smarter compiler that employs a more powerful type system can prove or disprove more complicated claims! That's a non-trivial conclusion that most programmers missed... type systems don't have to suck, and they can actually work for you, unlike Java.

But what do I know, I'm just learning Haskell now, and besides -- this post is but a teaser. So do yourself a favor and Learn You a Haskell!

Hooking Imports for Fun and Profit

2011-06-17T00:00:00-07:00

I really love Python... it's so hackable that it just calls for hacking, inspiring your imagination to find ways to stretch its boundaries. This time I decided to investigate into import hooks, to add some missing functionality I wanted to have.

As you probably know, Python uses a flat namespace for packages, that works on a "first found first served" basis. Packages are simply searched in linear order, as they appear in sys.path: if two directories contain a package named foo, importing foo will fetch package in the first directory. This is normally the desired behavior (as it allows you to override some modules by changing PYTHONPATH), but it's also quite limiting.

Consider the nested package namespace used by Java and various other languages (e.g., Haskell), where packages are normally "deeply nested", as in com.sun.foo.bar or com.ibm.spam.ham. In Haskell, for instance, packages are normally placed under their appropriate "categories", e.g., Data.Vector or Control.Monad. When you write a new data type, you'll probably put it under Data, as in Data.Vector.UberVector.

If we were to use something like that in Python, if would require us to have a single com/ directory, under which lots of sub-packages must be placed. This might be possible, but it's certainly not the "right way"; and besides, it implies that code written by Sun and IBM is somehow related (after all, they share the __init__.py file in com/), which means one may affect the other (for better or worse :)).

It would make much more sense to have separately-installed packages, e.g., site-packages/com.ibm.spam.ham and site-packages/com.sun.foo.bar, where each package is independent of the other. Sure, they might share a common com prefix -- but that's all. This is especially useful in corporate environments, where multiple teams share common packages (which usually get very unoriginal names, say, common), and name collisions are very likely. It's not a joke: at my work-place, we're know reorganizing our code after such problems. Also, from a marketing point-of-view, it might make more sense for your customers to import mycompany.foobar than just import foobar.

But most importantly -- it's composable. Nested packages allow you to "inject" your package into another namespace. Take twisted for instance: it's become so large that it had made more sense to split it up into sub-packages (twisted.conch, twisted.news, ...), and allow end users to choose which of them they wish to install. However, since Python wouldn't let you have site-packages/twisted and site-packages/twisted-conch, they resorted to hacking distutils into doing what they want. If nested packages were supported, you would have a core twisted package, with separate add-on packages like twisted-conch. So why not, really?

Enter nimp ("nested imports"). Without going into too many technical details, nimp is a meta-import hook -- it modifies the way import statements work. Specifically, it scans sys.path and "merges" packages that begin with a common prefix into "logical packages". For instance, if you have com-ibm-foo and com-sun-bar on your sys.path, nimp will create namespace packages for com, com.sun and com.ibm. This would allow code like import com.ibm.foo or from com.sun.bar import vodka to work transparently. All you need to do is run import nimp; nimp.install() (you can also put it in your site.py, so it would happen every time you run a Python process), and you're ready to go.

This was the first time I wrote an import hook, and I really liked how I easy it was to change the import mechanism. So today I had another idea -- lazy imports. Of course there's PEAK's lazyModule and this quite complicated recipe, but I thought, why not combine the two. Writing code like from peak... import lazyModule; foo = lazyModule("foo") is cumbersome, while the recipe attempts is too make everything lazy.

Instead, I created a module called __lazy__, that when imported, installs a meta-import hook. This import hook handles only modules that begin with __lazy__, so instead of importing them, it returns an "on-demand-loaded module" (i.e., when you try to access an its attributes). Using it is really simple:

>>> from __lazy__ import telnetlib
>>> telnetlib
<OnDemandModule 'telnetlib'>
>>> telnetlib.Telnet  # forces loading
<class telnetlib.Telnet at 0x015B08B8>
>>> telnetlib
<module 'telnetlib' from 'C:\Python27\lib\telnetlib.pyc'>

>>> from __lazy__.xml.dom import minidom
>>> minidom
<OnDemandModule 'xml.dom.minidom'>
>>> minidom.parseString   # forces loading
<function parseString at 0x01659CF0>
>>> minidom
<module 'xml.dom.minidom' from 'C:\Python27\lib\xml\dom\minidom.pyc'>

Note, though, that using from __lazy__.x.y import z forces the loading of x, since we use the dot operator on it. The from __lazy__ import foo is "truly lazy"

You can get the code of __lazy__ here.

Property Classes

2011-05-14T00:00:00-07:00

Tired of creating properties the old way? Python 3 brings an improvement in the form of multi-stage properties, i.e.,

@property
def foo(self):
    ... # getter

@foo.setter
def foo(self, value):
    ... # setter

It still feels very awkward. I can't say my solution is pure elegance, but I find it cleaner.

Code

import types

def property_class(cls):
    getter = getattr(cls, "get", None)
    if isinstance(getter, types.UnboundMethodType):
        getter = getter.im_func
    setter = getattr(cls, "set", None)
    if isinstance(setter, types.UnboundMethodType):
        setter = setter.im_func
    return property(getter, setter)

Example

>>> class Person(object):
...     def __init__(self):
...         self._age = 17
...     @property_class
...     class age:
...         def get(self):
...             return self._age
...         def set(self, value):
...             self._age = value
...
>>> p = Person()
>>> p.age
17
>>> p.age=19
>>> p.age
19

Python is Messy

2011-05-07T00:00:00-07:00

A couple of days ago, Rudiger, a user of RPyC found a rather surprising bug, that in turn revealed just how gruesome python's inner workings are. Rudiger was working with two machines, one 32 bit and the other 64 bit, and one machine had a netref to a remote list. He then tried to execute something as simple as mylist[1:], which to everyone's surprise threw a very peculiar exception: OverflowError: Python int too large to convert to C long.

At first, it seemed that the exception originated from the server side, but further investigation showed it actually originated from the client side, propagated to the server side, and then back to the client side: very weird indeed. I recreated the scenario with two of my machines, and popped up wireshark to see exactly what was going on there. The last packet before the "resonating" exception seemed to be invoking __getslice__ on the client-side list object, passing it 1 as the start index and 9223372036854775807 as the stop index. Where the heck is that number coming from? I added lots of debug prints and what not, but that strange number kept appearing there, and it was obviously not my code that placed it there.

A day later, the answer finally stroke me: 9223372036854775807 is actually 0x7fffffffffffffff, which is sys.maxint on 64-bit machines. From that point on, the solution was simple, but it had revealed a nasty implementation-detail of CPython 2.xx. You see, when getting a slice of an object, two methods come to play. The first, deprecated, method is __getslice__, which simply takes two arguments for start and stop. The second, recommended method, is __getitem__ which accepts a slice object instead of an integer. Sadly, <type list> has both, which are reflected on the netref proxy, which causes this rather surprising behavior:

>>> class Foo(object):
...     def __getitem__(self, x):
...             print "getitem", x
...     def __getslice__(self, *args):
...             print "getslice", args
...
>>>
>>> y=Foo()
>>> y[7]
getitem 7
>>> y[7:8]
getslice (7, 8)
>>> y[7:]
getslice (7, 2147483647)
>>> y[7:None]
getitem slice(7, None, None)

As you can see y[7:] invokes __getslice__ with sys.maxint, while y[7:None] (which is equivalent) invokes __getitem__ with a slice object... how lame! So when the server (64-bit) side code attempts to execute mylist[1:] it invokes __getslice__ on the client (32-bit) side, passing it the server's "version" of sys.maxint, which goes into the C implementation and blows up. Monkeyballs!

So the simple (and only) solution to this issue is using mylist[1:None] when working with different-width platforms... sorry.

ctypes - Pointer from Address

2011-04-27T00:00:00-07:00

There are times you need to construct a ctypes pointer from an integer address you have, say the id of a python object. I scratched my head for quite a while until I found out a how to do it properly (with some help from the stackoverflow guys). Here's what I got:

import ctypes

def deref(addr, typ):
    return ctypes.cast(addr, ctypes.POINTER(typ)).contents

Example

# get the ref count of an object (in a very nasty way :)
>>> x="hello world"
>>> deref(id(x), ctypes.c_int)
c_long(1)
>>> y=x
>>> z=x
>>> deref(id(x), ctypes.c_int)
c_long(3)

Some words of caution: * I'm relying here on id returning the address of an object. This is a weak assumption, but it holds (and is likely to continue to hold) for CPython. * There are APIs for what I showed in the example above... use them instead!

I'm using this code to dig into the vicious OVERLAPPED structure that's held inside PyOVERLAPPED for really low-level hacking... Anyway, if anyone finds this recipe useful, feel free to use it.

Copy Function Defaults

2007-02-06T00:00:00-08:00

Default arguments to functions are evaluated when the function is created, and are stored in the function object. This cause irritating problems when the default values are in fact mutable, as only single instance exists:

>>> def f(x = []):
...     x.append(5)
...     print x
...
>>> f()
[5]
>>> f()
[5, 5]
>>> f()
[5, 5, 5]

Sometimes it's the desired behavior, but mostly it's a bug. To solve that bug, we use

>>> def f(x = None):
...     if x is None:
...         x = []
...     x.append(5)
...     print x
...
>>> f()
[5]
>>> f()
[5]
>>> f()
[5]

But this idiom adds lots of boilerplate code into functions. The following little decorator solves that problem elegantly.

Code

from copy import deepcopy

def copydefaults(func):
    defaults = func.func_defaults
    def wrapper(*args, **kwargs):
        func.func_defaults = deepcopy(defaults)
        return func(*args, **kwargs)
    return wrapper

Example

>>> @copydefaults
... def f(x = []):
...     x.append(5)
...     print x
...
>>>
>>> f()
[5]
>>> f()
[5]
>>> f()
[5]

Weak Methods

2006-09-29T00:00:00-07:00

Many times you would pass bound methods to other functions, as callbacks (i.e., to simulate events, etc.). However, bound methods (the instancemethod type) hold a strong reference to their owning instance (im_self), which means the existence of a bound method is enough to hold the instance "alive", even though your code has lost all references to it.

Sometimes it's the desired behavior -- but not always. This nifty decorator will solve the problem by returning a "weakly-bound" method, which means the method will not hold the instance alive. You'll get ReferenceError if you try to invoke the method, after the instance has died.

Code

from weakref import proxy
from types import MethodType

class weakmethod(object):
    __slots__ = ["func"]
    def __init__(self, func):
        self.func = func
    def __get__(self, obj, cls):
        if obj is not None:
            obj = proxy(obj)
        return MethodType(self.func, obj, cls)

Example

>>> class Foo(object):
...     @weakmethod
...     def bar(self, a, b):
...             print self, a, b
...
>>> f = Foo()
>>> b = f.bar
>>> b
<bound method Foo.bar of <weakproxy at 009FA1E0 to Foo at 009FC070>>
>>> b(1, 2)
<__main__.Foo object at 0x009FC070> 1 2

And when we delete the instance f:

>>> del f
>>> b
<bound method Foo.bar of <weakproxy at 009FA1E0 to NoneType at 1E1D99B0>>
>>> b(1,2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in bar
ReferenceError: weakly-referenced object no longer exists

Real Mixins

2006-09-22T00:00:00-07:00

The normal python paradigm for implementing mixins is using multiple inheritance. Mixin classes take some measures of precaution as of their design (not to interfere with the derivee's MRO as much as possible), but they are essentially just regular classes, being derived from.

This code here creates real mixed-in classes: it actually merges one class into another (CPython specific), taking care of name-mangling, some complications with __slots__, and everything else. As a side-effect, you can also use it to mix modules into classes.

Code

import inspect

def mixin(cls):
    """
    mixes-in a class (or a module) into another class. must be called from within
    a class definition. `cls` is the class/module to mix-in
    """
    locals = inspect.stack()[1][0].f_locals
    if "__module__" not in locals:
        raise TypeError("mixin() must be called from within a class definition")
    
    # copy the class's dict aside and perform some tweaking
    dict = cls.__dict__.copy()
    dict.pop("__doc__", None)
    dict.pop("__module__", None)
    
    # __slots__ hell
    slots = dict.pop("__slots__", [])
    if slots and "__slots__" not in locals:
        locals["__slots__"] = ["__dict__"]
    for name in slots:
        if name.startswith("__") and not name.endswith("__"):
            name = "_%s%s" % (cls.__name__, name)
        dict.pop(name)
        locals["__slots__"].append(name)
    
    # mix the namesapces
    locals.update(dict)

Example

>>> class SomeMixin(object):
...     def f(self, x):
...         return self.y + x
...
>>> class AnotherMixin(object):
...     def g(self):
...         print "g"
...
>>>
>>> class Foo(object):
...     mixin(SomeMixin)
...     mixin(AnotherMixin)
...     
...     def h(self):
...         print "h"
...
>>> f = Foo()
>>> obj = Foo()
>>> obj.y = 18
>>> dir(obj)
['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', 
'__hash__', '__init__', '__module__', '__new__', '__reduce__', 
'__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', 
'f', 'g', 'h']
>>> obj.f
<bound method Foo.f of <__main__.Foo object at 0x00A96710>>
>>> obj.g
<bound method Foo.g of <__main__.Foo object at 0x00A96710>>
>>> obj.h
<bound method Foo.h of <__main__.Foo object at 0x00A96710>>
>>> obj.h()
h
>>> obj.g()
g
>>> obj.f(4)
22
>>>

Hooking dir()

2006-09-07T00:00:00-07:00

While working with proxy objects I found introspection quite hard, as the mechanism of dir() just looks at the object's __dict__ (with some exceptions), and there's no way to customize this introspection. But do not fear, hacking skillz are near! After reading into the machinery of dir(), I found a nifty solution to this problem.

Note: since python 2.6, __dir__ is a special method that's invoked by dir(), if it exists, so there's no reason to use this code. Use only on earlier versions of python.

Code

__members__ = property(lambda self: self.__dir__())

Example

>>> class CustomDir(object):
...     __members__ = property(lambda self: self.__dir__())
...
...     def __dir__(self):
...         return "a list of fake attributes".split()
...
>>> c = CustomDir()
>>> dir(c)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__getattribute__', 
'__hash__', '__init__', '__members__', '__module__', '__new__', '__reduce__', 
'__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', 'a', 
'attributes', 'fake', 'list', 'of']

Module Mixins a la Ruby

2006-08-29T00:00:00-07:00

Multiple inheritance is usually considered bad, and many ways to avoid it have been developed. The first and most notable is interfaces, which are basically abstract classes (only declaration, no implementation). Another solution is mix-in classes, which add functionality ("implementation") to the class, but do not affect the inheritance tree.

Mixins differ from regular classes mostly by their semantics (after all, they are just classes):

They would never derive from any class in the inheritance tree of the defined class
They would never define an __init__ method

Also, if class Dog is a subclass of class Animal, then an instance of Dog is an instance of Animal as well (up-cast). On the other hand, a class that derives from a mixin class isn't considered a subclass of that mixin class. For more info, see DictMixin in UserDict.py, or this ruby tutorial.

Ruby came with a strange idea -- mixin modules. I don't find this whole thing quite useful, as python (unlike ruby) supports multiple inheritance, but I wanted to show it's possible in python as well. So here's a two-liner metaclass decorator that does the trick.

Note: in ruby, adding a function to the module would also add it as a method of the class, but this is not the case here, as python imposes certain (arbitrary) limitations on that. Never mind.

Code

def mixin(mod):
    return type("mixin(%s)" % (mod.__name__,), (object,), mod.__dict__)

==Example== This is the code of testmixin.py

def foo(self, y):
    return self.x + y

@staticmethod
def boo(y):
    return 2 + y

And here's a snapshot demo:

>>> import testmixin
>>>
>>> class Bar(mixin(testmixin)):
...     def __init__(self):
...         self.x = 8
...     def zoo(self):
...         return "zoo"
...
>>> b = Bar()
>>> b.zoo()
'zoo'
>>> b.foo(9) # foo is a method from testmixin
17
>>> b.boo(2) # foo is a staticmethod from testmixin
4
>>>

Killable Threads

2006-08-13T00:00:00-07:00

The thread2 module is an extension of the standard threading module, and provides the means to raise exceptions at the context of the given thread. You can use raise_exc() to raise an arbitrary exception, or call terminate() to raise SystemExit automatically.

It uses the unexposed PyThreadState_SetAsyncExc function (via ctypes) to raise an exception in the context of the given thread. Inspired by the code of Antoon Pardon at http://mail.python.org/pipermail/python-list/2005-December/316143.html.

Issues

The exception will be raised only when executing python bytecode. If your thread calls a native/built-in blocking function, the exception will be raised only when execution returns to the python code.
- There is also an issue if the built-in function internally calls PyErr_Clear(), which would effectively cancel your pending exception. You can try to raise it again.
Only exception types can be raised safely. Exception instances are likely to cause unexpected behavior, and are thus restricted.
- For example: t1.raise_exc(TypeError) and not t1.raise_exc(TypeError("blah")).
- IMHO it's a bug, and I reported it as one. For more info, http://mail.python.org/pipermail/python-dev/2006-August/068158.html
I asked to expose this function in the built-in thread module, but since ctypes has become a standard library (as of 2.5), and this feature is not likely to be implementation-agnostic, it may be kept unexposed.

Code

import threading
import inspect
import ctypes


def _async_raise(tid, exctype):
    """raises the exception, performs cleanup if needed"""
    if not inspect.isclass(exctype):
        raise TypeError("Only types can be raised (not instances)")
    res = ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, ctypes.py_object(exctype))
    if res == 0:
        raise ValueError("invalid thread id")
    elif res != 1:
        # """if it returns a number greater than one, you're in trouble, 
        # and you should call it again with exc=NULL to revert the effect"""
        ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, 0)
        raise SystemError("PyThreadState_SetAsyncExc failed")


class Thread(threading.Thread):
    def _get_my_tid(self):
        """determines this (self's) thread id"""
        if not self.isAlive():
            raise threading.ThreadError("the thread is not active")
        
        # do we have it cached?
        if hasattr(self, "_thread_id"):
            return self._thread_id
        
        # no, look for it in the _active dict
        for tid, tobj in threading._active.items():
            if tobj is self:
                self._thread_id = tid
                return tid
        
        raise AssertionError("could not determine the thread's id")
    
    def raise_exc(self, exctype):
        """raises the given exception type in the context of this thread"""
        _async_raise(self._get_my_tid(), exctype)
    
    def terminate(self):
        """raises SystemExit in the context of the given thread, which should 
        cause the thread to exit silently (unless caught)"""
        self.raise_exc(SystemExit)

Example

>>> import time
>>> from thread2 import Thread
>>>
>>> def f():
...     try:
...         while True:
...             time.sleep(0.1)
...     finally:
...         print "outta here"
...
>>> t = Thread(target = f)
>>> t.start()
>>> t.isAlive()
True
>>> t.terminate()
>>> t.join()
outta here
>>> t.isAlive()
False

HTML Codec

2006-07-26T00:00:00-07:00

A very simple and straight-forward text/HTML codec. When encoding text, it escapes all HTML-delimiters (< becomes <, etc.), so the encoded text can be safely viewed by an HTML renderer (browser) or safely embedded into an HTML document. When decoding HTML, it unescapes the formatters into plain text (so that > becomes > again, etc.).

Code

def encode(input, tabsize = 4):
    return (input
            .replace("&", "&amp;")
            .replace("<", "&lt;")
            .replace(">", "&gt;")
            .replace('"', "&quot;")
            .replace("\n", "<br/>")
            .replace("\t", "&#9;" + " " * tabsize)
            .replace(" ", "&nbsp;"))

def decode(input, tabsize = 4):
    return (input
            .replace("&nbsp;", " ")
            .replace("&#9;" + " " * tabsize, "\t")
            .replace("<br>", "\n")
            .replace("<br/>", "\n")
            .replace("&quot;", '"')
            .replace("&lt;", "<")
            .replace("&gt;", ">")
            .replace("&amp;", "&"))

#
# Codec APIs (if you place this file in lib/encodings, you can use 
# str.encode("html") and str.decode("html")
#
import codecs

class HtmlCodec(codecs.Codec):
    def __init__(self, tabsize = 4):
        self.tabsize = tabsize
    def encode(self, input, errors = "strict"):
        return encode(input, self.tabsize), len(input)
    def decode(self, input, errors = "strict"):
        return decode(input), len(input)

class StreamWriter(HtmlCodec, codecs.StreamWriter):
    pass
class StreamReader(HtmlCodec, codecs.StreamReader):
    pass
def getregentry():
    hc = HtmlCodec()
    return (hc.encode, hc.decode, StreamReader, StreamWriter)

Example

This module can be used as a standalone module

>>> import htmlcodec
>>> htmlcodec.encode("blah > yada")
'blah&nbsp;&gt;&bnsp;yada"

Or you can place it in your interpreter's directory, as for instance, and then use

>>> '(blah > yada) & "wow"\ni eat babies'.encode("html")
'(blah&nbsp;&gt;&nbsp;yada)&nbsp;&amp;&nbsp;&quot;wow&quot;<br/>i&nbsp;eat&nbsp;babies'
>>> _.decode("html")
'(blah > yada) & "wow"\ni eat babies'

Weak Attributes

2006-06-04T00:00:00-07:00

Weak attributes are attributes of an instance that hold only a weak reference to another object. They are very useful for automatic breaking of cyclic references. This weakattr class implements a data-descriptor that holds a weak reference to the attribute, so that when it's no longer strongly-referenced, it automatically "disappears" from the instance.

The motivation for this concept comes from http://mail.python.org/pipermail/python-3000/2006-June/002357.html

Code

import weakref

class weakattr(object):
    """
    weakattr - a weakly-referenced attribute. When the attribute is no longer
    referenced, it 'disappears' from the instance. Great for cyclic references.
    """
    __slots__ = ["dict", "errmsg"]
    
    def __init__(self, name = None):
        self.dict = weakref.WeakValueDictionary()
        if name:
            self.errmsg = "%%r has no attribute named %r" % (name,)
        else:
            self.errmsg = "%r has no such attribute"
    
    def __repr__(self):
        return "<weakattr at 0x%08X>" % (id(self),)
    
    def __get__(self, obj, cls):
        if obj is None:
            return self
        try:
            return self.dict[id(obj)]
        except KeyError:
            raise AttributeError(self.errmsg % (obj,))
    
    def __set__(self, obj, value):
        self.dict[id(obj)] = value
    
    def __delete__(self, obj):
        try:
            del self.dict[id(obj)]
        except KeyError:
            raise AttributeError(self.errmsg % (obj,))

Example

A simple example

>>> class x(object):
...     cyc = weakattr()
...
...     def __init__(self):
...         self.cyc = self
...     def __del__(self):
...         print "g'bye"
...         print hasattr(self, "cyc") # will print False
...
>>> y = x()
>>> y
<__main__.x object at 0x009EEAF0>
>>> y.cyc
<__main__.x object at 0x009EEAF0>
>>> y.cyc.cyc
<__main__.x object at 0x009EEAF0>
>>> del y
>>> gc.collect() # force a collection
g'bye            # printed by y.__del__
False            # printed by y.__del__
0                # return value of gc.collect
[[code]]

In case you wondered what the optional  parameter means
[[code]]
>>> class x(object):
...     attr1 = weakattr()
...     attr2 = weakattr("attr2")
...
>>>
>>> y = x()
>>> # will not show a name
...
>>> y.attr1 
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 24, in __get__
AttributeError: <__main__.x object at 0x009EEDD0> has no such attribute
>>>
>>> # will show 'attr2'
...
>>> y.attr2 
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 24, in __get__
AttributeError: <__main__.x object at 0x009EEDD0> has no attribute named 'attr2'
>>>

And a somewhat more complex one:

>>> class Node(object):
...     next = weakattr()
...
...     def __init__(self, name):
...         self.name = name
...     def __del__(self):
...         print "g'bye", self
...     def __repr__(self):
...         return "<Node %s>" % (self.name,)
...
>>>
>>> node1 = Node("node1")
>>> node2 = Node("node2")
>>> node1.next = node2
>>> node2.next = node1
>>>
>>> node1
<Node node1>
>>> node1.next
<Node node2>
>>> node1.next.next
<Node node1>
>>> node1.next.next.next
<Node node2>
>>>
>>> del node2
>>> # forcibly collect the dead objects
... # this will cause node1.next to disappear, so that 
... # node2._del__ is be called
>>> gc.collect() 
g'bye <Node node2>
0

Tomer Filiba's Homepage

Cartesian Tree-Product

How is it Useful?

The Algorithm

Extension: the Inverse Problem

Travolta NXT

Abstract

Modus Operandi

Code

Components

Running Example

Critical Analysis of the Component Language

Issues with the Physical World

Small Bug in the Component Language

A Survey of Construct 3

Introduction

Basics

More Elaborate Structures

Bytes and Bits and Units

Powering Up: Adapters

Don't Repeat Yourself: Macros

Putting things in Context

Compilation

Computational Power

Feedback

New Experiences

Lessons on Web Programming

In Other News

Formal Logic

Loads of Plumbum

Plumbum Hits v1.0

Hypertext: In-Python Haml

Templates? Really?!

Haml

Hypertext

I, for one, welcome our Singularity overlord

You are the Neurons of the Revolution

New Beginnings

Splitbrain Python

Enter Splitbrain

ReHelloWorld

A Windows Story

Javaism, Exceptions, and Logging: Part 2

When Life Throws Lemons At You

Don't Catch Broadly

Don't be Overprotective

A Note On Real Users

Don't Wrap Exceptions

Don't Handle Exceptions

Closing Words

On the Granularity of Exception Classes

כולם מדברים על גיוס, אף אחד לא מדבר בשכל

כלבתא במטותא

מיליטריזם חברתי

אתוס שמתוס

על העיוורון

מודל חלופי

ועוד פרט אחד קטן לסיום

Javaism, Exceptions, and Logging: Part 1

Javaism

Exceptions

RPCs, Life and All

On Transparency

On Duck Typing

On Types

On Networks

On RPCs

On HTTP

On RPyC

On Testing

Reed-Solomon Codec

Some Background

The Library

Some Notes on RPyC 3.2.x

RPyC 3.2.2 Released

The Future of Construct

This Expressions

Construct 2.5

Construct 3

Closing Words

`This` Expressions