Tomer Filiba's Homepage

The Future of Construct

2012-05-16T00:00:00-07:00

It's been a long while since I've put time into Construct. I gave up on it somewhere in 2007, right after the release of v2.0... I think I just got bored, and felt like the library was complete and extensible enough to survive on its own. Of course I was wrong there, and code-rot had spread all over.

Luckily for us, Corbin Simpson took up the project in January of 2011 and has been maintaining it since then. He migrated it to github, changed the project to use a proper directory structure, fixed lots of bugs, and wrote extensive documentation. Since then, Construct has been building a solid community and has reached quite a remarkable number of downloads on PyPI.

All this time, I've been busy with my other projects, but I kept toying with the idea of Construct 3. I got some sketches, wrote some early drafts, cleaned up the implementation of Construct's core... but it's remained a dream. A couple of months ago I decided I'd back-port a nice feature from Construct 3, called this expressions, and it has rekindled my interest in the library.

`This` Expressions

One of the goals of Construct 3 was to generate efficient (C/Python) code from construct definitions. It even worked, to some extent: for instance, see this snippet that automatically generates this C-code. I had no recollection of this until I discovered it today. Funny.

Anyway, Construct 2 uses lambda functions to represent dependencies between constructs, e.g.

s = Struct("LV",
    UBInt8("length"),
    Bytes("value", lambda ctx: ctx["length"]),
)

and since this is the case, it's impossible to translate dependencies to C. So I've created the this object, which essentially builds an expression tree from native Python expressions. In order to evaluate the expression, you simply invoke it with a context:

>>> from construct import this
>>> this.x * 2 + 3
((this.x * 2) + 3)
>>> (this.x * 2 + 3)({"x":7})
17

So now we can replace all lambda ctx: ctx["foo"] by the more succinct and readble this.foo -- the benefits are visually clear -- but they go even deeper than this: since we're no longer dealing with black-box lambda functions, we can drill down into them and generate the appropriate (static) code.

Construct 2.5

I had to do some Construct work recently, and I missed the conciseness of this expressions, so I took the time to back-port them to Construct 2. I sent a pull request to Corbin, but he's a bit too busy to maintain the library on a regular basis now, so he's created a github organization and moved the repository to there; this is where Construct will be developed from now on.

Tinkering with the old code again got me sentimental, and I started to do some long-awaited maintenance. I plan to release version 2.5 (note the dramatic shift from 2.06 to 2.5) in the summer (say August), and here's the list of planned changes:

Adding this expressions
Dropping construct.text -- it's always been an experimental feature and it's achingly inefficient. If you want to parse grammars, you should use more adequate tools
Adding Python 3 support (based on the work of Eli Bendersky); the library will support Python 2.5-3.2, using six
General cleanups and optimizations
Closing the wikispaces site in favor of readthedocs

Construct 3

The next big thing(TM), Construct 3, is still far away. I've got lots of cool ideas, but time is too short (as so is my ability to concentrate on one thing). Generally, the guiding thought is to modernize the library and make it even yet more compact and efficient, while removing magic along the way. For instance, because Structs require their sub-elements to have a name, and due to the fact keyword-arguments in Python are unordered, all constructs ended up taking a name argument (even though it's usually meaningless to them, as in UBInt8("length")). This has given birth to all sorts of bastards like Rename and Alias; from now on, it'll be simpler:

s = Struct(
    Member("length", UBInt8),
    Member("value", Bytes(this.length)),
)

A second issue is, laying the grounds for code generation, thus converting all dependencies to use this expressions, and perhaps even limiting the power of Adapters. Or at least, making a clear distinction between the constructs that can be turned into code and those that can't.

And last but not least, I want Construct 3 to come with a designer, where you would drag-and-drop constructs, group them in "boxes", connect them to each other (instead of this.length, you'd connect the Bytes' count field to the source construct), etc. And most importantly, you could try it out live on a data sample, and see how it breaks it up. I made this sketch here (click to download the PowerPoint slides) to demonstrate how it might look:

I think it would be very powerful.

Closing Words

I'm mainly writing this post to inform everyone of Construct's new repository, and to ask for feedback on the plans for v2.5... Any thoughts, requests, comments would be appreciated. Also, if anyone wants to join in (especially with Construct 3 ambitious plans) -- feel free to contact me.

Introducing Plumbum - Shell Combinators

2012-05-12T00:00:00-07:00

It's been a while since I last blogged... sorry! Had a midterm exam, a seminar project to deliver (an O(n^3) parser for Tree Insertion Grammar), the routine family festivities of Passover, and this new thing, Plumbum, that has been keeping my mind overclocked while I should have been studying for exams and writing seminar papers. But now that it's out, it's about time I write a little on it.

So Plumbum is something I've toyed with for quite some time now. In almost any sort of project, there comes a time when you have to write this really simple, 5-liner shell script, just to build your project's artifacts, maybe also upload them to PyPI or sourceforge (using rsync), and while you're at it, why not build the project's documentation as well. Oh, and don't forget to run regression tests before all that, and it would be wise to also handle some basic command-line options, as you might want to skip uploading to PyPI at times, or perhaps build on different versions of python... and then you end up with this monstrous shell script that you wrote (or worse, some former employee wrote) and never want to lay your eyes on again... At this point to begin hating yourself for not doing it in python, to begin with.

But then again, how do you translate find . -name "*.pyc" | xargs rm or cp */*.py /tmp to python-speak? That would take quite a few lines and require importing several modules. Shell scripts tend to be so short and enchanting... How do we bridge the gap?

Plumbum was born to fill this very gap: on the one hand, be pythonic (and be backed by strong libraries), and on the other, make it all as easy and one-liner-ish by nature: use a real programming language, with a well-behaved object model and high level concepts, to achieve what you'd normally do in a shell script, retaining the same expressive power and wrist-handiness of the shell. I call it shell combinators, as it's a pythonic way to mimic shell syntax.

The library is actually a collection of utilities that I wrote for several separate projects, and had never got to polish them. Plumbum consolidates them into a single, production-grade framework. The library provides local and remote program execution, with support for piping and IO redirection; local and remote file-system paths abstraction; a programmatic command-line interface (CLI) toolkit, and numerous other utilities.

For instance,

from plumbum import local
from plumbum.cmd import wc, ls, echo, grep
from plumbum.utils import copy, delete

delete(local.cwd // "*/*.pyc")

for fn in local.cwd / "src" // "*.py":
    print wc("-l", fn)

num_of_src_lines = (ls["-l"] | grep["\\.py"] | wc["-l"])()

echo["1"] > "/proc/sys/net/ipv4/ip_forward")()

There's a short cheat-sheet as well as extensive documentation on the project's site, but at this point I'd like to elaborate a bit on the CLI toolkit, as I think it deserves some more attention.

The approach of optparse / argparse and similar libraries is to build a parser object and populate it with options in an imperative manner, which I dislike. Plumbum's CLI toolkit offers a more declarative yet programmatic alternative: An application is defined as a class that derives from cli.Application; it may define a main() method, which serves as the "entry point" of the application, and any number of switch-methods, meaning, methods that are invokable from the command-line.

Switch methods are normal methods, decorated by @switch, that may take either no arguments or a single one; for each switch given on the command line, the toolkit will invoke the switch method that binds it. Similarly, the main() method is invoked after all switches have been processed, and it takes all the positional arguments (i.e., non-switch arguments) that were given. And last but not least, there are switch attributes, which are in fact just specialized versions of switch functions, that store an argument given to the switch in an instance attribute.

So I've probably only gotten you confused by the terminology at this point, but actually it's much simpler!

from plumbum import cli

class MyHttpServer(cli.Application):
    log_to_file = cli.SwitchAttr("--log-to-file", str)
    verbose = cli.Flag("-v")
    mode = cli.SwitchAttr("--mode", cli.Set("TCP", "UDP"), default = "TCP")
    port = cli.SwitchAttr("--port", cli.Range(1024, 65535), default = 8080)
    
    @switch(["-l", "--load-config"], cli.ExistingFile)
    def load_config(self, filename):
        """Loads the given config file"""
        f = open(filename, "r")
        self._parse_config(f.read())
        # ...
    
    def main(self, src, dst):
        if self.log_to_file:
            logger.addHandler(FileHandler(self.log_to_file))
        logger.setLevel(logging.DEBUG if self.verbose else logging.WARNING)
        # ...

if __name__ == "__main__":
    MyHttpServer.run()

There, I think it's a lovely example of the expressive power of the CLI toolkit and Plumbum in general. I hope you'll give it a try, and may you never have to write shell scripts again!

Solving Systems of Linear Equations

2012-03-25T00:00:00-07:00

Yet another university-related post, but I really enjoyed it so I thought I'd share: for a GUI- workshop I'm taking, we are given GUI-layout constraints as a system of linear equations, which we need to satisfy. To make life more interesting, some constraints are constant while some are parametric. There's no magic here, just some linear algebra combined with Python's overloadable nature to produce a nice and compact solver for these linear systems.

Naturally, you present your system as a coefficient matrix, which is Gauss-eliminated and then "solved" for a given set of variables. I took the Gauss-Jordan elimination code from Jarno Elonen and modified it to support MxN matrices (not necessarily square). Here's a simple example of elimination:

>>> m = Matrix([1,2,4,2], [3,7,6,8], [2,2,2,9])
>>> print m.eliminate()
( 1.00  -0.00   0.00   6.00)
( 0.00   1.00  -0.00  -1.00)
( 0.00   0.00   1.00  -0.50)

But of course a reduced row-echelon matrix is not our final goal - we want to get the variable assignments. For this, we have solve():

>>> sol = solve(m, ["x", "y", "z"])
>>> print sol
{'y': -1.0000000000000004, 'x': 6.0, 'z': -0.4999999999999998}

Modulo precision errors, we get x = 6, y = -1 and z = -0.5. If we have more equations than variables, the "extraneous" equations must be linear combinations of previous ones, or a contradiction will result. But what if we have less equations than variables? It means we have some degrees of freedom... how would we handle that? It's actually simple - instead of being resolved to constant values, variables will be assigned dependent expressions:

>>> m2 = Matrix([1,2,4,2], [3,7,6,8])
>>> print m2.eliminate()
( 1.00   0.00   16.00  -2.00)
( 0.00   1.00  -6.00   2.00)
>>>
>>>
>>> sol = solve(m2, ["x", "y", "z"])
>>> print sol
{'y': <BinExpr (2.0 - (-6.0 * z))>, 'x': <BinExpr (-2.0 - (16.0 * z))>, 
    'z': <FreeVar z>}

As you can see now, z is a free variable and x and y are dependent on it. Of course more than one variable may be free and some variables may be independent of free variables. Once a value for z is known, we can "fully evaluate" the dependent expressions:

>>> sol["x"].eval({"z" : 10})
-162.0

The code is available on my github page. Note: all numbers are represented as Decimals, to avoid loss of precision as much as possible, and I'm using an "epsilon" value of 1E-20 to equate numbers to each other (meaning, x == y iff abs(x-y) <= epsilon).

Easy Syntax for Representing Trees

2012-03-07T00:00:00-08:00

I'm working on a parser for Tree Adjoining Grammar (TAG) for this seminar I'm taking. TAG is an extension of context-free grammar (CFG) that's more powerful while still being polynomially-parsable. Anyhow, TAG makes use of "tree production rules" instead of the "linear" production rules of CFG: instead of S -> NP VP, you'd have a small tree, the root of which being S, having NP and VP as its children. Of course these trees can be more than two-level deep, and they go all sorts of operations such as substitution and adjunction, but that's for the parser.

So I needed a compact and (hopefully) readable way to express such trees in my code. At first I used lots of parenthesis, which was ugly and cumbersome, but then I devised this:

class NonTerminal(object):
    def __init__(self, name):
        self.name = name
    def __sub__(self, children):
        return Tree(self, children)
    def __pos__(self):
        return Foot(self)

class Foot(object):
    def __init__(self, nonterm):
        self.nonterm = nonterm

class Tree(object):
    def __init__(self, root, children):
        self.root = root
        self.children = children

And here's how you use it:

S = NonTerminal("S")

t1 = S-["e"]
t2 = S-["a", S-["c", +S, "d"], "b"]

# Which represents the following two trees:
#
# t1:                t2:
#      S                  S 
#      |                / | \
#      |               /  |  \
#      e              a   S   b
#                       / | \
#                      /  |  \
#                     c   S*  d
#

The peculiar +S is a way to mark that a leaf node is a foot, which is part of the semantics TAG (that's where adjunction takes place). It's represented in the diagram by the more conventional S*, but I had to resort to a unary operator in the code. Anyway, I'm not sure if it's a recipe or just a nice trick, but I thought I'd share this.

By the way, you can use both __sub__ and __neg__ to achieve things like S-----X (i.e., more than a single - sign, to allow for better padding), but I tried to avoid too much ASCII art. I'd love to hear about other such ideas!

RPyC 3.2.1 Released

2012-03-04T00:00:00-08:00

This is a maintenance release, fixing some minor bugs and resolving some issues with Python 3 compatibility. More on the change log.

Python 2.x: it's advisable to upgrade to this version.

Python 3.x: it's highly recommended to upgrade to this version, as it resolves some core issues that went under the radar in 3.2.0.

Toying with Context Managers

2012-02-27T00:00:00-08:00

As I promised in the code-generation using context managers post, I wanted to review some more, rather surprising, examples where context managers prove handy. So we all know we can use context managers for resource life-time management: before entering the with-suite we allocate (open) the resource, and when we leave the suite, we free (close) it -- but there's much more to context managers than meets the eye.

Stacks, for Fun and Profit

We've seen this one in the code-generation post, but we can generalize this notion a bit. Every time we enter a with block, we'd append an element to a list, which we'd pop on exit. Consider this snippet:

class Stacking(object):
    def __init__(self):
        self.stack = []
    @contextmanager
    def foo(self):
        self.stack.append("foo")
        yield "bar"
        self.stack.pop(-1)

Nothing fancy, but that's exactly how the code generation framework works: whenever you enter a new block, it's pushed onto the "stack", and whenever we leave the block, the top-of-stack element is popped. This is how we automatically take care of indentation, curly-braces, etc.

But of course this pattern is much more useful. Consider something like this:

class Env(object):
    def __init__(self):
        self._envstack = [os.environ]
    @contextmanager
    def new(self, **kwargs):
        self._envstack.append(self._envstack[-1].copy())
        self._envstack[-1].update(kwargs)
        yield
        self._envstack.pop(-1)

env = Env()

with env.new(PATH = "/tmp/foo/bin", SHELL = "zsh"):
    # processes created here will use the modified environment
    pass

We can also leverage this concept to run commands as different users. Here's a sketch:

cmd.run("ls")                       # as current user
with cmd.as_user("root"):
    cmd.run("ls", "/proc")          # as `root`
    with cmd.as_user("mallory"):
        cmd.run("rm", "-rf", "/")   # as `mallory`
    cmd.run("cat", "/etc/passwd")   # back as `root` again

In essence, every time you want to make local/undoable changes to your state, this pattern proves helpful.

Contextbacks

Contextbacks, a pun on callbacks, are contexts you pass as arguments to other functions. Many times it's useful to pass a before-function and an after-function, and contextbacks are a nice way to encapsulate this. So instead of this:

def f(beforefunc, afterfunc):
    beforefunc()
    # your code goes here
    afterfunc()

You get this:

def f(ctxback):
    with ctxback:
        # you code goes here
        pass

Not a ground-breaking change, but I prefer it as it's more concise.

Lightweight Asynchronism

This is probably my favorite use case for contexts: you can use them to pipeline or interleave long-lasting tasks. You can think of contexts are degenerate forms of coroutines, in which you have defined beginning and end, but the middle part is interchangeable, so you can stick anything into it.

Imagine you need to format a harddisk (using mkfs) or perform some long network operation (like copying a huge file over scp). The pattern is as follows: initiate the operation, wait for it to finish, and either return or raise an exception. This fits perfectly well with the way contexts work -- with one change -- yield instead of waiting.

@contextmanager
def format_disk(devfile):
    proc = Popen(["/sbin/mkfs", "-t", "ext3", devfile])
    try:
        yield
    except Exception:
        proc.kill()
    stdout, stderr = proc.communicate()
    if proc.returncode != 0:
        raise FormattingFailed(stderr)

with format_disk("/dev/sda1"), format_disk("/dev/sdb1"):
    pass

This saves you time: instead of waiting for the first operation to finish before starting with the second, you can run them in parallel. The total time would be that of the longest task (not taking into account the IO bottleneck). You can throw a yield into any piece of code instead of just blocking, and use it as a pipelined contextmanager. You can copy three files in parallel, without resorting to threads or a reactor in the background.

Of course you could improve that by returning an object that reports the progress of the operation, e.g.

with format_disk("/dev/sda1") as d1, format_disk("/dev/sdb1") as d2:
    while not d1.is_done() or not d2.is_done():
        print "%s is being formatted, %d%% completed" % (d1.devfile, 
                d1.get_progress())
        print "%s is being formatted, %d%% completed" % (d2.devfile, 
                d2.get_progress())

And voila! You have a thread-less, light-weight asynchronous framework at hand... a bit like using a reactor, but without rewriting your code.

And last, if you can't tweak the blocking parts of the code (e.g., third party libraries), you can use the "defer to thread" or "defer to process" approach, a la twisted:

@contextmanager
def defer_to_thread(func):
    thd = Thread(target = func)  # it would be smarter to use a 
    thd.start()                  # thread-pool
    yield thd
    thd.join()

with defer_to_thread(task1), defer_to_thread(task2), defer_to_thread(task3):
    # do something else in the meanwhile
    pass

So that's all I had in mind. If you have other unorthodox use cases for contexts, I'd love to hear about them!

Wizard Dialog Toolkit

2012-02-11T00:00:00-08:00

Following my Deducible UI post, and following some of the criticism it had received, I'd like to share something I've been working on (read: experimenting with) at my work place. You see, we have some "interactive wizards" that storage admins use to connect storage arrays to their hosts (say, a DB server). These wizards prompt you with questions like your what's your username, the name of the pool/volume, whether it's an iSCSI or a Fiber Channel connection, etc., and then they go and perform what you've asked for.

These wizards operate in a terminal environment, but we've had thoughts to make GUI/web versions of them. This would be a considerable effort with the current design. Another issue they currently have is the mixing of "business logic" and presentation together. For instance, the code that scans the devices attached to your host also prints ANSI-colored messages or reports its progress. All in all it works fine, but there's lots of room for improvement.

I began to investigate this corner a month or two ago. The initial observation was that such wizards have a pretty rigid and repetitive structure, thus we can find some abstraction or a "toolkit" for "expressing" wizards more compactly. This has also led to the realization that once the business logic and presentation are separate, there's no reason to limit ourselves to terminal-based interaction: our wizard-toolkit could do the plumbing and work with terminals, ncurses, GUIs, web-browsers, etc. The business logic would remain oblivious, and we could have a nice GUI at zero-cost!

There was also a second issue of styling, i.e., printing text in color, that I wanted to get r id of. This part was easy: I thought, why not employ the model of HTML and CSS? Let's separate the structure (semantics) of the text from its styling. Instead of printing a banner for titles, we'll display a Title object, whose exact appearance is determined by a "style sheet" (a class, of course, not actually a text document).

For instance, when we're using a color-enabled terminal, the title would be printed in bold and followed by an empty line; but if our terminal is color-blind, we'll render the text centered and surrounded by = marks. Another example is error-handling: instead of printing error message in red every time, we'll display an Error object; on a terminal, this would be rendered as red text, but when running in a GUI, rendering this object would pop up a message box. I'm going to ignore this for the rest of this post, as this is really a side issue.

Now let's get to expressing wizards, or more generally, dialogs. Following some earlier iterations, I came to the model where a dialog is a "container object" that's made of dialog elements. These elements can be output-only (such as a welcome message), or input-output (such as a message telling you to choose one of the available options). A dialog is "executed" by a DialogRunner that renders it and returns the results gotten from the user. It's quite important to note that dialog elements within a single dialog cannot be interdependent -- that is, if you want to ask the user for his name and then show "Hi there %s" with the user's name, this has to be done as two, serial dialogs.

That was quite a lot of babble -- let's see this in action:

class MyApp(WizardApp):
    def main(self):
        iscsi = Option("iSCSI")
        fc = Option("FC")
        d = Dialog(
            Text(Title("hello world")),
            Input("un", "Username"),
            Password("pw", "Password"),
            Choice("conf", "What do you want to configure?", [iscsi, fc]),
        )
        res = self.ui.run(d)
        if res["conf"] == fc:
            self.config_fc(res)
        else:
            self.config_iscsi(res)
    ...

if len(sys.argv) == 2 and sys.argv[1] == "--gtk":
    MyApp.run(GtkDialogRunner("My App"))
else:
    MyApp.run(TerminalDialogRunner(ANSIRenderer))

It's a short and incomplete snippet of course, as I'm only going to cover the big picture. The main function creates a dialog object d and passes it to ui.run, which "runs" the dialog and returns the results, as a dictionary. Notice that the dialog elements Input, Password and Choice all take a first parameter -- this is the key under which the result would be placed in the returned dictionary, e.g., res["un"] would hold the user-provided user name, and res["pw"] would hold the password. Text, on the other hand, is an output-only element, so it doesn't return anything and doesn't take a key. Long story short, we're asking the user to enter some information and choose one of two options, and then continue processing based on the selected option. At the bottom, we determine how to run the application based on a command-line switch: if --gtk is given, we'll run the dialogs through the GtkDialogRunner; otherwise, we'll use the TerminalDialogRunner.

And how does it look like? When running on a terminal:

And with a single command-line switch, we run as a GTK application:

So of course it's far from perfect, but then again, it's a small research project I've only put ~15 hours into. It suffers from some of the problems I've listed in the deducible UI post, for instance, the GUI hangs when the business logic performs blocking tasks. This could be solved by moving to a reactor-based model, but I've tried to keep the existing wizard code in tact as much as possible. A hanging GUI is not nice, but it's not the end of the world either, and there are numerous ways to overcome this.

Another benefit this design brings along is the ability to automate testing by using mock dialog runners. Since our business logic is only exposed to the returned dictionary, we can use a dialog runner that actually displays nothing and returns a scripted scenario each time. We can even go further: because our business logic "talks" in high-level primitives like Choice, we can compute the Cartesian product of all choices and run through each of them. We can show that we've covered all paths! And we can do this automatically... without people hitting buttons and keeping logs of their progress.

Anyway, I just wanted to show that it's feasible. I'm not releasing any code as this project is currently in very early stages, and it's something I do at work. Perhaps we'll open-source it in the future, if it proves useful enough.

Just Got Me These

2012-02-09T00:00:00-08:00

Hurrah! I just got me these:

After the CS secretariat refused to let take math courses as electives (thus forcing me into taking boring stuff like SQL), I think Dover books and I are going to become good friends. It's not like I have plenty of time to read them, but even just seeing them on my bookshelf would make me a happier person :)

Next on my wish-list are some books on logics and proof theory, category theory, information theory, and automata and computability. I'll get there some day...

WHO HAS CHEEZBURGER?

2012-02-03T00:00:00-08:00

Too much Wireshark...

Code Generation using Context Managers

2012-01-31T00:00:00-08:00

When I was working on Agnos, a cross-language RPC framework, I had a lot of code-generation to do in a variety of languages (Python, Java, C#, and C++). At the early stages, I just appended strings to a list. It was quick and dirty, and it's got the job done... but that wasn't enough, of course. I've lost the original code already, but it looked something like this:

def generate_proxy(typeinfo):
    lines = [
        "public class %sProxy {" % (typeinfo.name,),
        "    private int uid;",
        "    public %sProxy(int uid) {" % (typeinfo.name,),
        "        this.uid = uid;",
        "    }",
    ]
    for attr in typeinfo.attributes:
        if attr.get:
            lines.append("    public %s get%s() {" % (attr.typename,
                                                    attr.name,))
            lines.append("        // ...")
            lines.append("    }")
        if attr.set:
            lines.append("    public void set%s(%s value) {" % (attr.name,
                                                            attr.typename))
            lines.append("        // ...")
            lines.append("    }")
    lines.append("}")
    return lines
# ...

lines = []
for ti in typeinfos:
    lines.extend(generate_proxy(m, ti))

open("foo.java").write("\n".join(lines))

There are several problems with this approach. First of all, it's very cumbersome and fragile. If you forget a comma in the list, two adjacent strings will be concatenated. Also, you have to do everything yourself, like remembering to close brackets, add semicolons, do the right indentation, etc. If you wished to split this code into functions, the functions you call would have to know the indentation level you're calling them at, or the generated code would be unreadable. This might seem negligible, but think of languages where indentation matters, like Python...

The fundamental problem with this approach (and similar ones) is that the code generator does not reflect the structure of the generated code. The two are diseparate, while it's quite obvious they should be correlated.

In order to solve this, I turned to context managers, a feature I highly value. Conceptually, context managers provide a way to bind beginning-and-end into a single entity; this is normally used for resource management -- but we can leverage this construction further (I'll this review in a different post). Here, I've used them to create nested blocks, which allowed me to reflect the structure of the generated code in the code generator.

Without going into too many details, I defined a Module class that exposes a block() context manager and a stmt() function. The module holds a "stack" of blocks, and entering a new block pushes a it onto the stack. Statements are then appended to the topmost block on the stack. Now, because this framework is "language-aware", it can encapsulate language-specific details. For instance, In Java, a block will be indented correctly and wrapped by brackets; in Python, we'll append colons to the opening line and indent the block; in C++, if the block begins with class, struct or enum, we'll append a trailing semicolon as well.

Here's how it works:

m = JavaModule()
m.stmt("import foo")
m.stmt("import bar")
m.sep()   # an empty line

#...

def generate_proxy(m, typeinfo):
    BLOCK = m.block
    STMT = m.stmt
    
    with BLOCK("public class {0}Proxy", typeinfo.name):
        STMT("private int uid")
        with BLOCK("public {0}Proxy(int uid)", typeinfo.name):
            STMT("this.uid = uid")
    
        for attr in typeinfo.attributes:
            if attr.get:
                with BLOCK("public {0} get{1}()", attr.typename, attr.name):
                    pass
            if attr.set:
                with BLOCK("public void set{0}({1} value)", attr.name, 
                                                            attr.typename):
                    pass
# ...

for ti in typeinfos:
    generate_proxy(m, ti)

m.render_to_file("foo.java")

So what have we gained?

The code is much shorter and more concise
Brackets, semicolons and indentation come out-of-the-box
We're no longer working with flat lists of strings -- we're working with hierarchal entities that reflect the structure of the generated code
And the other way around -- the structure of the generated code is reflected in the generating code; nested code is indeed nested inside BLOCKs, thus the "generatee" and generator are visually and semantically correlated.
We can easily split our code into functions, as the module maintains an internal stack. If f() opened a block and called g() under it, it the code that g() generates will be placed and indented correctly.

I tried to keep my code quite general, so I haven't defined all of the target language's constructs, but of course we could do that, or at least head in that direction. It might look like this:

def generate_proxy(m, typeinfo):
    with m.CLASS(typeinfo.name + "Proxy", ["public"]):
        m.FIELD("int", "uid", ["private"])
        with m.CTOR(["int uid"]):   # CTOR gets the name of the current class
            STMT("this.uid = uid")

However, there's a question of where we "put our foot down", or we'll end up writing Java Combinators for Python... and then we'll be writing Java in Python. No need for that, thank you very much.

The full source code can be found in the Agnos repository

Deducible UI

2012-01-27T00:00:00-08:00

A Brief History

I like automating things. I don't like having to reiterate myself: my dream is to always be able to add only the necessary amount of information in order to make something possible. This is one reason, for instance, why I hate expressions like ArrayList<String> x = new ArrayList<String>();... it always makes me feel like I'm talking to a retard (compiler).

In 2006/7, I wrote some demos for RPyC to show how easy network-related tasks become. I chose something rather complex, a chat client, to show that all the code sums up to a few lines: clients invoke a method on the server, say broadcast(str), and the server then invokes callbacks on all of its clients, sending them the message.

In order to make it usable, I had to write a GUI: I chose Tk, because it comes with python and is quite simple; I knew there were better toolkits, but my GUI was meant to be basic enough to be doable in any toolkit. It occurred to me, then, that I wrote ~20 lines show-casing RPyC and ~100 lines of horrible GUI code, and that something must be really wrong here. And by here I mean everywhere.

Note: throughout the article, I'm using the word GUI to mean any interactive user interface, be it graphical (Qt, GTK, wxWidgets, ...) or terminal-based (curses and the like). Basically, anything that doesn't block on a single line of input, like shells.

GUI Designers

So you might say, "Dude, just use QtDesigner or something". A GUI designer lets you visually place components and makes your life much easier -- drag and drop your widgets and double-click on a button to write its action. Very easy indeed. But I would like to offer a different angle on the subject: just like the invention of the teacup has hindered the technological advance of China, so do GUI designers hinder us from developing better GUIs. These designers offer a local optimum which we fail to surpass, and this leaves us with the mediocre UIs and development tools we have today. And get me going about XAML.

Think about it: you have to design a GUI. So yeah, it's kind of simple, but doesn't that break DRY? You have the code and you have the GUI -- two faces of the same idea. Obviously, one should be derived from the other.

For the lion's share of programs, the UI is highly deterministic -- there's some information that needs to be displayed to- or gotten from the user, and the bindings is trivial. Consider a login screen: you want to get a username and a password, use them somehow, and proceed to the next screen. This is a repeating task, and I'd guess that for ~80% of the programs in the world, it's easy enough to automatically deduce how the UI should look, given the task at hand. And I'm not talking about machine learning algorithms or designing "families of tasks" -- way simpler than that! Just define a mapping between programmatic primitives and their visual representation.

Deducing UI

The ultimate goal is to take "pure code", unaware of UI, and by adding the necessary metadata, be able to automatically create ("deduce") a GUI for it. In fact, I'd like to expose programmatic APIs to a human -- completely interchangable programmatic- and human- interfaces. Think how cool it could be to import Adobe Photoshop and run a directory full of pictures through its filters, instead of doing so through the UI... without Adobe having to write a separate GUI and programming toolkit.

The UI needn't be an eye-candy, at least at the beginning; it just has to be good-enough. It won't work for games or complex applications like Office, but for it would be just fine for a chat client. Let's assume the following mapping:

An object is represented by a window
Read-only instance attributes are represented as labels
Writable instance attributes are represented as textboxes
Methods are represented by buttons. If a method requires arguments, it would be preceded by textboxes

Of course we could change textboxes and labels to reflect the attribute's or argument's type -- DateTime would be represented by a DatePicker, int could be represented by a number box with up/down arrows, etc. And of course the framework is free to change the mapping however it wants, to achieve better, more coherent representation. The mapping above is just a rough draft.

Now, instances of a class like the following:

class Person {
    public String firstName;
    public String lastName;
    public DateTime birthdate;
    
    public void dance() {...}
    
    public void eat(String foodstuff) {...}
}

would turn into

with just a little bit of binding in the form of:

class Main {
    static public void Main(String[] args) {
        Person p = new Person(...);
        guify(p);
    }
}

Straightforward, isn't it? You can already begin to see the benefits. This framework would obviously require some sort of decoration (annotations in Java, attributes in C#, ...) on which classes and which class members are to be exposed, and perhaps some extra metadata, like a picture to show instead of a method's name, or some layout information -- but it's perfectly doable. And we can turn it better looking (unlike my beautiful ASCII art example), by using better UI primitives and a better mapping between objects and their representation; but let's leave it for now.

I wrote a a simple prototype of this and lo and behold, it actually worked! But when you try to use it in a real-life applications, the going gets tough: things are updated behind the scenes (not through our UI framework) and we need to reflect these changes in the UI. For instance, an element is added to a list via the list's add() method - how can the GUI become aware of that? Well, we can use observable objects, which the GUI would observe; so instead of using an ArrayList<T>, you'd simply use an ObservableArrayList<T>.

But creating an observable counterpart for every class is a considerable effort on the framework's side, and it breaks software modularity: the framework has to be aware of every 3rd party class that you wish to expose, or at least allow you to provide the means to expose them. Another downside of this scheme is that your code becomes aware of the GUI: if we've so far managed to keep our code clean of GUI primitives (we only required some metadata), all of the sudden you must replace your lists with GUI-observable-lists. Bummer.

Another issue is that using synchronous programming techniques (blocking operations) does not play well with this model: when does the GUI gets its "runtime"? How can we keep it from freezing? Who's providing the entry-point of the program? Does it run in a separate thread? If so, we risk polluting our code with GUI-related locks (which is countering the whole purpose); and besides, threads suck and add the incurred complexity is never worth it. The only feasible option is asynchronous programming (via a reactor) -- but requires that your code be programmed this way, and it's quite nasty to write such code without proper language support (e.g., lack of closures, coroutines, etc.).

"No Way"

As I said, I've been toying with this idea from 2006, and I always get the same response from colleague programmers: "it would never work", "it won't be good enough", "no one would want to use it", "users need their eye-candy", "I need tight control over the layout", and what not. Skeptics galore. My answer is always the same: you never know what your user wants -- so who are you to decide? And besides, you're always to lazy to support proper customization of UI, so your user must live with your decisions.

Sure, there are books and dissertations about UX, and you've read them all; but why not just provide good-enough defaults, and let the rest be customized by the user? Let the framework deduce a sane layout for your code -- but let's make everything movable/resizable/dockable. This way, if it makes more sense to place button X to the left of button Y, the user can do so himself. And let's remember the user's preferences in a file, which we'll load each time the application runs. And by "user" I'm also talking about your UX expert -- let him/her decide on a default look (i.e., the preferences file) for the application, which will be shipped with your product, but the end-user would still be able to move things around. Wouldn't it be easier? And if you insist, here's the place to stick some machine-learning magic, in order to deduce better UIs by default.

So anyhow, I had a working proof-of-concept somewhere, but I think I lost it. It wouldn't be too hard to recreate it, but at the moment I'm more concerned with UI combinators, which I'll cover in a future post. Fully deducing a UI is quite a challenge, as a detailed above, but it's doable nonetheless, and the added-value is huge! I'll get back to it some day, perhaps after I have better UI combinators... but in the meantime, is there anyone in the audience who's willing to pick it up?

Regarding Infix Operators

2012-01-25T00:00:00-08:00

I got some reactions to the Infix Operators post, and wanted to point out some things. First of all, I'm not the one who came up with it -- it's a recipe from the Python Cookbook that's been posted in 2005. I'm not taking credit for it or anything, I just said I loved the idea and I adapted the code a little.

Second, regarding coding style or the pythonicity of this scheme -- let's be clear, it's a hack. In order for it to work, the two arguments must refuse to support __or__ on Infix objects, which means you can't compose them properly:

>>> @Infix
... def dot(f, g):
...     return lambda *x: f(g(*x))
... 
>>> @Infix
... def mul(x, y):
...     return x * y
... 
>>> def double(x):
...     return x * 2
... 
>>> f = double |dot| mul   # this works ok
>>> 
>>> g = mul |dot| double   # but this won't
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in __or__
TypeError: mul() takes exactly 2 arguments (1 given)

It's also magical and unpythonic by nature. You can read in the cookbook comments about a less-experienced programmer who complained he had to wrap his head around around this. I'd say this feature is a kin to metaclasses: they are useful (at times), but there are better, more pythonic ways to do the same without the magic.

So why is it useful? First of all, it's an interesting pattern, worth more knowing about than actually using it. But on the more practical side, it could be very useful in domain-specific languages (DSL), where it could increase your expressiveness. Consider something like Construct, where you define data structures declaratively:

ipaddr = Struct("ipaddr",
    UInt8("a"),
    UInt8("b"),
    UInt8("c"),
    UInt8("d"),
)

lenval = Struct("lenval",
    UInt8("len"),
    Bytes("val", lambda ctx: ctx.len),   # interdependencies: "val" is `len` bytes long
)

We could replace all sorts of built-in constructs by such "operators", thus building "data expressions". So here's a very early sketch of what it could look it:

ipaddr = UInt8 |seq| UInt8 |seq| UInt8 |seq| UInt8

# or maybe just
ipaddr = UInt8 |repeat| 4

# binding names into the context
lenval = UInt8 |bind_seq("len")| Bytes(getctx("len"))

Beware: I just made this up, there's no solid concept behind it.

All Systems are Go

2012-01-23T00:00:00-08:00

At last, I finished migrating the old drupal site to github pages. Everything is now fully revisioned and statically-generated (using Disqus for comments). Jekyll is so cool! I wrote all the HTML and forged the stylesheets myself... hope you like it.

Anyhow, I'm happy with the design now, and I'll start blogging more regularly. I still have plenty of pages to complete (projects, about-page, etc.), and a few partially-written blogs posts to polish up, but other generally speaking, we're up.

If you have any feedback about the design/site, please let me know in the comments below or via email. Thanks!

Foxx0rz

2012-01-22T00:00:00-08:00

This is Foxx0rz, our course' mascot. He's written in C (compiles under MSVC++ 6 to be exact), and we had it printed on T-shirts... 'twas fun.

Here's a downloadable version

And this is the output:

Infix Operators in Python

2012-01-22T00:00:00-08:00

As you may already know, there are 3 kinds of operators calling-notations: prefix (+ 3 5), infix (3 + 5), and postfix (3 5 +). Prefix (as well as postfix) operators are used in languages like LISP/Scheme, and have the nice property of not requiring parenthesis — there’s only one way to read an expression like 3 5 + 2 *, unlike 3 + 5 * 2. On the other hand, it reduces code readability and the locality of operators and their arguments. This is why we all love infix operators.

Now imagine I have a function, add(x,y), and I have an expression like add(add(add(5,6),7),8)... wouldn’t it be cool if I could use infix notation here? Sadly though, Python won’t allow you to define new operators or change how functions take their arguments... but that doesn’t mean we have to give up!

Haskell, for instance, allows you to define custom operators and set their precedence, as well as invoking "normal" functions as infix operators. Suppose you have a function f(x,y) — you can invoke it like f 5 6 or 5 `f` 6 (using backticks). This allows us to turn our previous expression, add(add(add(5,6),7),8), into 5 `add` 6 `add` 7 `add` 8, which is much more readable. But how can we do this in Python?

Well, there’s this Cookbook recipe that provides a very nice way to achieving the same functionality in Python (adapted a little by me):

from functools import partial

class Infix(object):
    def __init__(self, func):
        self.func = func
    def __or__(self, other):
        return self.func(other)
    def __ror__(self, other):
        return Infix(partial(self.func, other))
    def __call__(self, v1, v2):
        return self.func(v1, v2)

Using instances of this peculiar class, we can now use a new "syntax" for calling functions as infix operators:

>>> @Infix
... def add(x, y):
...     return x + y
...
>>> 5 |add| 6
11

Surrounding decorated functions with pipes (bitwise ORs) allows them to take their parameters infix-ly. Using this, we can do all sorts of cool things:

>>> instanceof = Infix(isinstance)
>>>
>>> if 5 |instanceof| int:
...     print "yes"
...
yes

And even curry functions:

>>> curry = Infix(partial)
>>>
>>> def f(x, y, z):
...     return x + y + z
...
>>> f |curry| 3
<functools.partial object at 0xb7733dec>
>>> g = f |curry| 3 |curry| 4 |curry| 5
>>> g()
12

Ain’t that cool?

RPyC Moves to a New Site

2011-08-29T00:00:00-07:00

RPyC is in the process of migrating from http://rpyc.wikidot.com to it's new (and hopefully final) location at http://rpyc.sourceforge.net. Wikidot had served us well, and was easy to maintain, but they started displaying way to many ads and didn't support rsync or SSH access, which meant I couldn't upload the generated API reference automatically.

The new site is written entirely in ReST using sphinx and large parts of it are auto-generated from docstrings in the source code. It's all now part of the git repository, and I only have to run make upload to upload it all up.

Learning Me a Haskell

2011-08-17T00:00:00-07:00

Phew! Finally the semester's over (just submitted my last project), and it's time to clean up my ever-so-long backlog. Here goes nothing: I'll start by posting something here, after this long while of neglect.

As I'm sure you already know, I'm a long-time Pythonista, and I'm confident enough in calling myself a "native speaker" of that language. Although python is my expertise, I'd say I'm fluent in most other prominent programming languages (say, C, C++, Java, C#, VB), and with adequate knowledge of many more. It may seem like a good set of skills, but I have to admit this brings one to a point of stagnation. There comes a time where everything just looks the same: you take a glimpse at a new language/platform/other technology and sigh, "oh well, on the surface it's different, but underneath it's all the same sh*t". You come to the conclusion that people mostly change the looks-and-feel, but nothing radical could never happen. And then you skip to the next article.

Then came Haskell: a statically-typed, type-inferred, highly-expressive functional language. I'm not new to functional programming -- I've programmed in Scheme, and I even wrote an interpreter for a toy functional-language that I made up in order to investigate nested-scopes and evaluation models -- but heck, this one's different. A friend of mine nagged me to learn Haskell for quite a time already, but I was deterred by Haskell's ugly syntax (it's not Python a'right) and I kept pushing it for "when I have time". It always seemed too academic to me, too impractical.

I think what provoked me into seriously learning Haskell is a pattern called continuation passing style -- I was just amazed by the extra power, design simplicity, and greater expressiveness that you get with it. Of course you can implement it in almost any other language, and in fact it was invented in Scheme (call/cc), but Haskell, with its purely-mathematical state of mind, just brings the best out of things. Haskell has also taught me that expressiveness and conciseness are not virtues to be underestimated: if you can do something in one line instead of four, you must be generalizing on some deeper concept that you've previously missed. It's not just looks-and-feel, this time.

But let's not go astray. I think the most prominent and well-known feature of Haskell is its excellent type system. I'm sure you've heard of static languages, like java or C++, where every expression has a compile-time type; and I'm sure you've heard of dynamic (duck-typed) languages, where there are only run-time types and no checks are (or can be) performed prior to running the code. In the first case, you curse the compiler for limiting what you can do (even when it makes perfect sense) and for being bluntly stupid, requiring you to repeat yourself over and over (FooBar x = new FooBar()). And because most type systems are so weak, you find yourself escaping to "duck-typing" techniques like void * or Object, and relying on run-time casts. In the latter case, you find yourself trying to cover the infinite space of type permutations that your functions have to handle, and soon you pray for some sort of validation or restrictiveness.

When you start learning Haskell, you suddenly realize how weak most type systems are, and that there are (much) better alternatives. It's like you've been suffering from a blurry vision your whole life, and suddenly you realize you can wear glasses. Has it ever occurred to you that type systems are equivalent to proof systems (the Curry-Howard isomorphism)? And that a compiler basically tries to prove your code? You can think of compilation errors as finding a counter-example to your claim! "You said F takes an integer, but I see you try to call it with a String"... so of course this is a trivial example, one that even a C compiler catches, but you can make more complicated claims. And then you realize that type systems and compilers ("provers") do matter -- a smarter compiler that employs a more powerful type system can prove or disprove more complicated claims! That's a non-trivial conclusion that most programmers missed... type systems don't have to suck, and they can actually work for you, unlike Java.

But what do I know, I'm just learning Haskell now, and besides -- this post is but a teaser. So do yourself a favor and Learn You a Haskell!

Hooking Imports for Fun and Profit

2011-06-17T00:00:00-07:00

I really love Python... it's so hackable that it just calls for hacking, inspiring your imagination to find ways to stretch its boundaries. This time I decided to investigate into import hooks, to add some missing functionality I wanted to have.

As you probably know, Python uses a flat namespace for packages, that works on a "first found first served" basis. Packages are simply searched in linear order, as they appear in sys.path: if two directories contain a package named foo, importing foo will fetch package in the first directory. This is normally the desired behavior (as it allows you to override some modules by changing PYTHONPATH), but it's also quite limiting.

Consider the nested package namespace used by Java and various other languages (e.g., Haskell), where packages are normally "deeply nested", as in com.sun.foo.bar or com.ibm.spam.ham. In Haskell, for instance, packages are normally placed under their appropriate "categories", e.g., Data.Vector or Control.Monad. When you write a new data type, you'll probably put it under Data, as in Data.Vector.UberVector.

If we were to use something like that in Python, if would require us to have a single com/ directory, under which lots of sub-packages must be placed. This might be possible, but it's certainly not the "right way"; and besides, it implies that code written by Sun and IBM is somehow related (after all, they share the __init__.py file in com/), which means one may affect the other (for better or worse :)).

It would make much more sense to have separately-installed packages, e.g., site-packages/com.ibm.spam.ham and site-packages/com.sun.foo.bar, where each package is independent of the other. Sure, they might share a common com prefix -- but that's all. This is especially useful in corporate environments, where multiple teams share common packages (which usually get very unoriginal names, say, common), and name collisions are very likely. It's not a joke: at my work-place, we're know reorganizing our code after such problems. Also, from a marketing point-of-view, it might make more sense for your customers to import mycompany.foobar than just import foobar.

But most importantly -- it's composable. Nested packages allow you to "inject" your package into another namespace. Take twisted for instance: it's become so large that it had made more sense to split it up into sub-packages (twisted.conch, twisted.news, ...), and allow end users to choose which of them they wish to install. However, since Python wouldn't let you have site-packages/twisted and site-packages/twisted-conch, they resorted to hacking distutils into doing what they want. If nested packages were supported, you would have a core twisted package, with separate add-on packages like twisted-conch. So why not, really?

Enter nimp ("nested imports"). Without going into too many technical details, nimp is a meta-import hook -- it modifies the way import statements work. Specifically, it scans sys.path and "merges" packages that begin with a common prefix into "logical packages". For instance, if you have com-ibm-foo and com-sun-bar on your sys.path, nimp will create namespace packages for com, com.sun and com.ibm. This would allow code like import com.ibm.foo or from com.sun.bar import vodka to work transparently. All you need to do is run import nimp; nimp.install() (you can also put it in your site.py, so it would happen every time you run a Python process), and you're ready to go.

This was the first time I wrote an import hook, and I really liked how I easy it was to change the import mechanism. So today I had another idea -- lazy imports. Of course there's PEAK's lazyModule and this quite complicated recipe, but I thought, why not combine the two. Writing code like from peak... import lazyModule; foo = lazyModule("foo") is cumbersome, while the recipe attempts is too make everything lazy.

Instead, I created a module called __lazy__, that when imported, installs a meta-import hook. This import hook handles only modules that begin with __lazy__, so instead of importing them, it returns an "on-demand-loaded module" (i.e., when you try to access an its attributes). Using it is really simple:

>>> from __lazy__ import telnetlib
>>> telnetlib
<OnDemandModule 'telnetlib'>
>>> telnetlib.Telnet  # forces loading
<class telnetlib.Telnet at 0x015B08B8>
>>> telnetlib
<module 'telnetlib' from 'C:\Python27\lib\telnetlib.pyc'>

>>> from __lazy__.xml.dom import minidom
>>> minidom
<OnDemandModule 'xml.dom.minidom'>
>>> minidom.parseString   # forces loading
<function parseString at 0x01659CF0>
>>> minidom
<module 'xml.dom.minidom' from 'C:\Python27\lib\xml\dom\minidom.pyc'>

Note, though, that using from __lazy__.x.y import z forces the loading of x, since we use the dot operator on it. The from __lazy__ import foo is "truly lazy"

You can get the code of __lazy__ here.

Property Classes

2011-05-14T00:00:00-07:00

Tired of creating properties the old way? Python 3 brings an improvement in the form of multi-stage properties, i.e.,

@property
def foo(self):
    ... # getter

@foo.setter
def foo(self, value):
    ... # setter

It still feels very awkward. I can't say my solution is pure elegance, but I find it cleaner.

Code

import types

def property_class(cls):
    getter = getattr(cls, "get", None)
    if isinstance(getter, types.UnboundMethodType):
        getter = getter.im_func
    setter = getattr(cls, "set", None)
    if isinstance(setter, types.UnboundMethodType):
        setter = setter.im_func
    return property(getter, setter)

Example

>>> class Person(object):
...     def __init__(self):
...         self._age = 17
...     @property_class
...     class age:
...         def get(self):
...             return self._age
...         def set(self, value):
...             self._age = value
...
>>> p = Person()
>>> p.age
17
>>> p.age=19
>>> p.age
19

Python is Messy

2011-05-07T00:00:00-07:00

A couple of days ago, Rudiger, a user of RPyC found a rather surprising bug, that in turn revealed just how gruesome python's inner workings are. Rudiger was working with two machines, one 32 bit and the other 64 bit, and one machine had a netref to a remote list. He then tried to execute something as simple as mylist[1:], which to everyone's surprise threw a very peculiar exception: OverflowError: Python int too large to convert to C long.

At first, it seemed that the exception originated from the server side, but further investigation showed it actually originated from the client side, propagated to the server side, and then back to the client side: very weird indeed. I recreated the scenario with two of my machines, and popped up wireshark to see exactly what was going on there. The last packet before the "resonating" exception seemed to be invoking __getslice__ on the client-side list object, passing it 1 as the start index and 9223372036854775807 as the stop index. Where the heck is that number coming from? I added lots of debug prints and what not, but that strange number kept appearing there, and it was obviously not my code that placed it there.

A day later, the answer finally stroke me: 9223372036854775807 is actually 0x7fffffffffffffff, which is sys.maxint on 64-bit machines. From that point on, the solution was simple, but it had revealed a nasty implementation-detail of CPython 2.xx. You see, when getting a slice of an object, two methods come to play. The first, deprecated, method is __getslice__, which simply takes two arguments for start and stop. The second, recommended method, is __getitem__ which accepts a slice object instead of an integer. Sadly, <type list> has both, which are reflected on the netref proxy, which causes this rather surprising behavior:

>>> class Foo(object):
...     def __getitem__(self, x):
...             print "getitem", x
...     def __getslice__(self, *args):
...             print "getslice", args
...
>>>
>>> y=Foo()
>>> y[7]
getitem 7
>>> y[7:8]
getslice (7, 8)
>>> y[7:]
getslice (7, 2147483647)
>>> y[7:None]
getitem slice(7, None, None)

As you can see y[7:] invokes __getslice__ with sys.maxint, while y[7:None] (which is equivalent) invokes __getitem__ with a slice object... how lame! So when the server (64-bit) side code attempts to execute mylist[1:] it invokes __getslice__ on the client (32-bit) side, passing it the server's "version" of sys.maxint, which goes into the C implementation and blows up. Monkeyballs!

So the simple (and only) solution to this issue is using mylist[1:None] when working with different-width platforms... sorry.

ctypes - Pointer from Address

2011-04-27T00:00:00-07:00

There are times you need to construct a ctypes pointer from an integer address you have, say the id of a python object. I scratched my head for quite a while until I found out a how to do it properly (with some help from the stackoverflow guys). Here's what I got:

import ctypes

def deref(addr, typ):
    return ctypes.cast(addr, ctypes.POINTER(typ)).contents

Example

# get the ref count of an object (in a very nasty way :)
>>> x="hello world"
>>> deref(id(x), ctypes.c_int)
c_long(1)
>>> y=x
>>> z=x
>>> deref(id(x), ctypes.c_int)
c_long(3)

Some words of caution: * I'm relying here on id returning the address of an object. This is a weak assumption, but it holds (and is likely to continue to hold) for CPython. * There are APIs for what I showed in the example above... use them instead!

I'm using this code to dig into the vicious OVERLAPPED structure that's held inside PyOVERLAPPED for really low-level hacking... Anyway, if anyone finds this recipe useful, feel free to use it.

Copy Function Defaults

2007-02-06T00:00:00-08:00

Default arguments to functions are evaluated when the function is created, and are stored in the function object. This cause irritating problems when the default values are in fact mutable, as only single instance exists:

>>> def f(x = []):
...     x.append(5)
...     print x
...
>>> f()
[5]
>>> f()
[5, 5]
>>> f()
[5, 5, 5]

Sometimes it's the desired behavior, but mostly it's a bug. To solve that bug, we use

>>> def f(x = None):
...     if x is None:
...         x = []
...     x.append(5)
...     print x
...
>>> f()
[5]
>>> f()
[5]
>>> f()
[5]

But this idiom adds lots of boilerplate code into functions. The following little decorator solves that problem elegantly.

Code

from copy import deepcopy

def copydefaults(func):
    defaults = func.func_defaults
    def wrapper(*args, **kwargs):
        func.func_defaults = deepcopy(defaults)
        return func(*args, **kwargs)
    return wrapper

Example

>>> @copydefaults
... def f(x = []):
...     x.append(5)
...     print x
...
>>>
>>> f()
[5]
>>> f()
[5]
>>> f()
[5]

Weak Methods

2006-09-29T00:00:00-07:00

Many times you would pass bound methods to other functions, as callbacks (i.e., to simulate events, etc.). However, bound methods (the instancemethod type) hold a strong reference to their owning instance (im_self), which means the existence of a bound method is enough to hold the instance "alive", even though your code has lost all references to it.

Sometimes it's the desired behavior -- but not always. This nifty decorator will solve the problem by returning a "weakly-bound" method, which means the method will not hold the instance alive. You'll get ReferenceError if you try to invoke the method, after the instance has died.

Code

from weakref import proxy
from types import MethodType

class weakmethod(object):
    __slots__ = ["func"]
    def __init__(self, func):
        self.func = func
    def __get__(self, obj, cls):
        if obj is not None:
            obj = proxy(obj)
        return MethodType(self.func, obj, cls)

Example

>>> class Foo(object):
...     @weakmethod
...     def bar(self, a, b):
...             print self, a, b
...
>>> f = Foo()
>>> b = f.bar
>>> b
<bound method Foo.bar of <weakproxy at 009FA1E0 to Foo at 009FC070>>
>>> b(1, 2)
<__main__.Foo object at 0x009FC070> 1 2

And when we delete the instance f:

>>> del f
>>> b
<bound method Foo.bar of <weakproxy at 009FA1E0 to NoneType at 1E1D99B0>>
>>> b(1,2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in bar
ReferenceError: weakly-referenced object no longer exists

Real Mixins

2006-09-22T00:00:00-07:00

The normal python paradigm for implementing mixins is using multiple inheritance. Mixin classes take some measures of precaution as of their design (not to interfere with the derivee's MRO as much as possible), but they are essentially just regular classes, being derived from.

This code here creates real mixed-in classes: it actually merges one class into another (CPython specific), taking care of name-mangling, some complications with __slots__, and everything else. As a side-effect, you can also use it to mix modules into classes.

Code

import inspect

def mixin(cls):
    """
    mixes-in a class (or a module) into another class. must be called from within
    a class definition. `cls` is the class/module to mix-in
    """
    locals = inspect.stack()[1][0].f_locals
    if "__module__" not in locals:
        raise TypeError("mixin() must be called from within a class definition")
    
    # copy the class's dict aside and perform some tweaking
    dict = cls.__dict__.copy()
    dict.pop("__doc__", None)
    dict.pop("__module__", None)
    
    # __slots__ hell
    slots = dict.pop("__slots__", [])
    if slots and "__slots__" not in locals:
        locals["__slots__"] = ["__dict__"]
    for name in slots:
        if name.startswith("__") and not name.endswith("__"):
            name = "_%s%s" % (cls.__name__, name)
        dict.pop(name)
        locals["__slots__"].append(name)
    
    # mix the namesapces
    locals.update(dict)

Example

>>> class SomeMixin(object):
...     def f(self, x):
...         return self.y + x
...
>>> class AnotherMixin(object):
...     def g(self):
...         print "g"
...
>>>
>>> class Foo(object):
...     mixin(SomeMixin)
...     mixin(AnotherMixin)
...     
...     def h(self):
...         print "h"
...
>>> f = Foo()
>>> obj = Foo()
>>> obj.y = 18
>>> dir(obj)
['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', 
'__hash__', '__init__', '__module__', '__new__', '__reduce__', 
'__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', 
'f', 'g', 'h']
>>> obj.f
<bound method Foo.f of <__main__.Foo object at 0x00A96710>>
>>> obj.g
<bound method Foo.g of <__main__.Foo object at 0x00A96710>>
>>> obj.h
<bound method Foo.h of <__main__.Foo object at 0x00A96710>>
>>> obj.h()
h
>>> obj.g()
g
>>> obj.f(4)
22
>>>

Hooking dir()

2006-09-07T00:00:00-07:00

While working with proxy objects I found introspection quite hard, as the mechanism of dir() just looks at the object's __dict__ (with some exceptions), and there's no way to customize this introspection. But do not fear, hacking skillz are near! After reading into the machinery of dir(), I found a nifty solution to this problem.

Note: since python 2.6, __dir__ is a special method that's invoked by dir(), if it exists, so there's no reason to use this code. Use only on earlier versions of python.

Code

__members__ = property(lambda self: self.__dir__())

Example

>>> class CustomDir(object):
...     __members__ = property(lambda self: self.__dir__())
...
...     def __dir__(self):
...         return "a list of fake attributes".split()
...
>>> c = CustomDir()
>>> dir(c)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__getattribute__', 
'__hash__', '__init__', '__members__', '__module__', '__new__', '__reduce__', 
'__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', 'a', 
'attributes', 'fake', 'list', 'of']

Module Mixins a la Ruby

2006-08-29T00:00:00-07:00

Multiple inheritance is usually considered bad, and many ways to avoid it have been developed. The first and most notable is interfaces, which are basically abstract classes (only declaration, no implementation). Another solution is mix-in classes, which add functionality ("implementation") to the class, but do not affect the inheritance tree.

Mixins differ from regular classes mostly by their semantics (after all, they are just classes):

They would never derive from any class in the inheritance tree of the defined class
They would never define an __init__ method

Also, if class Dog is a subclass of class Animal, then an instance of Dog is an instance of Animal as well (up-cast). On the other hand, a class that derives from a mixin class isn't considered a subclass of that mixin class. For more info, see DictMixin in UserDict.py, or this ruby tutorial.

Ruby came with a strange idea -- mixin modules. I don't find this whole thing quite useful, as python (unlike ruby) supports multiple inheritance, but I wanted to show it's possible in python as well. So here's a two-liner metaclass decorator that does the trick.

Note: in ruby, adding a function to the module would also add it as a method of the class, but this is not the case here, as python imposes certain (arbitrary) limitations on that. Never mind.

Code

def mixin(mod):
    return type("mixin(%s)" % (mod.__name__,), (object,), mod.__dict__)

==Example== This is the code of testmixin.py

def foo(self, y):
    return self.x + y

@staticmethod
def boo(y):
    return 2 + y

And here's a snapshot demo:

>>> import testmixin
>>>
>>> class Bar(mixin(testmixin)):
...     def __init__(self):
...         self.x = 8
...     def zoo(self):
...         return "zoo"
...
>>> b = Bar()
>>> b.zoo()
'zoo'
>>> b.foo(9) # foo is a method from testmixin
17
>>> b.boo(2) # foo is a staticmethod from testmixin
4
>>>

Killable Threads

2006-08-13T00:00:00-07:00

The thread2 module is an extension of the standard threading module, and provides the means to raise exceptions at the context of the given thread. You can use raise_exc() to raise an arbitrary exception, or call terminate() to raise SystemExit automatically.

It uses the unexposed PyThreadState_SetAsyncExc function (via ctypes) to raise an exception in the context of the given thread. Inspired by the code of Antoon Pardon at http://mail.python.org/pipermail/python-list/2005-December/316143.html.

Issues

The exception will be raised only when executing python bytecode. If your thread calls a native/built-in blocking function, the exception will be raised only when execution returns to the python code.
- There is also an issue if the built-in function internally calls PyErr_Clear(), which would effectively cancel your pending exception. You can try to raise it again.
Only exception types can be raised safely. Exception instances are likely to cause unexpected behavior, and are thus restricted.
- For example: t1.raise_exc(TypeError) and not t1.raise_exc(TypeError("blah")).
- IMHO it's a bug, and I reported it as one. For more info, http://mail.python.org/pipermail/python-dev/2006-August/068158.html
I asked to expose this function in the built-in thread module, but since ctypes has become a standard library (as of 2.5), and this feature is not likely to be implementation-agnostic, it may be kept unexposed.

Code

import threading
import inspect
import ctypes


def _async_raise(tid, exctype):
    """raises the exception, performs cleanup if needed"""
    if not inspect.isclass(exctype):
        raise TypeError("Only types can be raised (not instances)")
    res = ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, ctypes.py_object(exctype))
    if res == 0:
        raise ValueError("invalid thread id")
    elif res != 1:
        # """if it returns a number greater than one, you're in trouble, 
        # and you should call it again with exc=NULL to revert the effect"""
        ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, 0)
        raise SystemError("PyThreadState_SetAsyncExc failed")


class Thread(threading.Thread):
    def _get_my_tid(self):
        """determines this (self's) thread id"""
        if not self.isAlive():
            raise threading.ThreadError("the thread is not active")
        
        # do we have it cached?
        if hasattr(self, "_thread_id"):
            return self._thread_id
        
        # no, look for it in the _active dict
        for tid, tobj in threading._active.items():
            if tobj is self:
                self._thread_id = tid
                return tid
        
        raise AssertionError("could not determine the thread's id")
    
    def raise_exc(self, exctype):
        """raises the given exception type in the context of this thread"""
        _async_raise(self._get_my_tid(), exctype)
    
    def terminate(self):
        """raises SystemExit in the context of the given thread, which should 
        cause the thread to exit silently (unless caught)"""
        self.raise_exc(SystemExit)

Example

>>> import time
>>> from thread2 import Thread
>>>
>>> def f():
...     try:
...         while True:
...             time.sleep(0.1)
...     finally:
...         print "outta here"
...
>>> t = Thread(target = f)
>>> t.start()
>>> t.isAlive()
True
>>> t.terminate()
>>> t.join()
outta here
>>> t.isAlive()
False

HTML Codec

2006-07-26T00:00:00-07:00

A very simple and straight-forward text/HTML codec. When encoding text, it escapes all HTML-delimiters (< becomes <, etc.), so the encoded text can be safely viewed by an HTML renderer (browser) or safely embedded into an HTML document. When decoding HTML, it unescapes the formatters into plain text (so that > becomes > again, etc.).

Code

def encode(input, tabsize = 4):
    return (input
            .replace("&", "&amp;")
            .replace("<", "&lt;")
            .replace(">", "&gt;")
            .replace('"', "&quot;")
            .replace("\n", "<br/>")
            .replace("\t", "&#9;" + " " * tabsize)
            .replace(" ", "&nbsp;"))

def decode(input, tabsize = 4):
    return (input
            .replace("&nbsp;", " ")
            .replace("&#9;" + " " * tabsize, "\t")
            .replace("<br>", "\n")
            .replace("<br/>", "\n")
            .replace("&quot;", '"')
            .replace("&lt;", "<")
            .replace("&gt;", ">")
            .replace("&amp;", "&"))

#
# Codec APIs (if you place this file in lib/encodings, you can use 
# str.encode("html") and str.decode("html")
#
import codecs

class HtmlCodec(codecs.Codec):
    def __init__(self, tabsize = 4):
        self.tabsize = tabsize
    def encode(self, input, errors = "strict"):
        return encode(input, self.tabsize), len(input)
    def decode(self, input, errors = "strict"):
        return decode(input), len(input)

class StreamWriter(HtmlCodec, codecs.StreamWriter):
    pass
class StreamReader(HtmlCodec, codecs.StreamReader):
    pass
def getregentry():
    hc = HtmlCodec()
    return (hc.encode, hc.decode, StreamReader, StreamWriter)

Example

This module can be used as a standalone module

>>> import htmlcodec
>>> htmlcodec.encode("blah > yada")
'blah&nbsp;&gt;&bnsp;yada"

Or you can place it in your interpreter's directory, as for instance, and then use

>>> '(blah > yada) & "wow"\ni eat babies'.encode("html")
'(blah&nbsp;&gt;&nbsp;yada)&nbsp;&amp;&nbsp;&quot;wow&quot;<br/>i&nbsp;eat&nbsp;babies'
>>> _.decode("html")
'(blah > yada) & "wow"\ni eat babies'

Weak Attributes

2006-06-04T00:00:00-07:00

Weak attributes are attributes of an instance that hold only a weak reference to another object. They are very useful for automatic breaking of cyclic references. This weakattr class implements a data-descriptor that holds a weak reference to the attribute, so that when it's no longer strongly-referenced, it automatically "disappears" from the instance.

The motivation for this concept comes from http://mail.python.org/pipermail/python-3000/2006-June/002357.html

Code

import weakref

class weakattr(object):
    """
    weakattr - a weakly-referenced attribute. When the attribute is no longer
    referenced, it 'disappears' from the instance. Great for cyclic references.
    """
    __slots__ = ["dict", "errmsg"]
    
    def __init__(self, name = None):
        self.dict = weakref.WeakValueDictionary()
        if name:
            self.errmsg = "%%r has no attribute named %r" % (name,)
        else:
            self.errmsg = "%r has no such attribute"
    
    def __repr__(self):
        return "<weakattr at 0x%08X>" % (id(self),)
    
    def __get__(self, obj, cls):
        if obj is None:
            return self
        try:
            return self.dict[id(obj)]
        except KeyError:
            raise AttributeError(self.errmsg % (obj,))
    
    def __set__(self, obj, value):
        self.dict[id(obj)] = value
    
    def __delete__(self, obj):
        try:
            del self.dict[id(obj)]
        except KeyError:
            raise AttributeError(self.errmsg % (obj,))

Example

A simple example

>>> class x(object):
...     cyc = weakattr()
...
...     def __init__(self):
...         self.cyc = self
...     def __del__(self):
...         print "g'bye"
...         print hasattr(self, "cyc") # will print False
...
>>> y = x()
>>> y
<__main__.x object at 0x009EEAF0>
>>> y.cyc
<__main__.x object at 0x009EEAF0>
>>> y.cyc.cyc
<__main__.x object at 0x009EEAF0>
>>> del y
>>> gc.collect() # force a collection
g'bye            # printed by y.__del__
False            # printed by y.__del__
0                # return value of gc.collect
[[code]]

In case you wondered what the optional  parameter means
[[code]]
>>> class x(object):
...     attr1 = weakattr()
...     attr2 = weakattr("attr2")
...
>>>
>>> y = x()
>>> # will not show a name
...
>>> y.attr1 
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 24, in __get__
AttributeError: <__main__.x object at 0x009EEDD0> has no such attribute
>>>
>>> # will show 'attr2'
...
>>> y.attr2 
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 24, in __get__
AttributeError: <__main__.x object at 0x009EEDD0> has no attribute named 'attr2'
>>>

And a somewhat more complex one:

>>> class Node(object):
...     next = weakattr()
...
...     def __init__(self, name):
...         self.name = name
...     def __del__(self):
...         print "g'bye", self
...     def __repr__(self):
...         return "<Node %s>" % (self.name,)
...
>>>
>>> node1 = Node("node1")
>>> node2 = Node("node2")
>>> node1.next = node2
>>> node2.next = node1
>>>
>>> node1
<Node node1>
>>> node1.next
<Node node2>
>>> node1.next.next
<Node node1>
>>> node1.next.next.next
<Node node2>
>>>
>>> del node2
>>> # forcibly collect the dead objects
... # this will cause node1.next to disappear, so that 
... # node2._del__ is be called
>>> gc.collect() 
g'bye <Node node2>
0

SeqAttr

2006-06-03T00:00:00-07:00

Sequenced attributes are useful for types that behave as sequences, but provide direct attribute access. For examples of such types, see time.struct_time of the result of os.fstat.

Note: since the inclusion of namedtuple, this recipe is considered deprecated.

==Code==

class seqattr(object):
    __slots__ = ["index"]
    def __init__(self, index):
        self.index= index
    def __repr__(self):
        return "<seqattr(%s)>" % (self.index,)
    def __get__(self, obj, cls):
        if obj is None:
            return self
        return obj[self.index]
    def __set__(self, obj, value):
        obj[self.index] = value

Example

>>> class struct_time(list):
...     year = seqattr(0)
...     month = seqattr(1)
...     day = seqattr(2)
...     hour = seqattr(3)
...     min = seqattr(4)
...     sec = seqattr(5)
...     weakday = seqattr(6)
...     yearday = seqattr(7)
...     daylight_saving = seqattr(8)
...
>>>
>>> struct_time.month
<seqattr(1)>
>>>
>>> s = struct_time((2006, 6, 2, 23, 56, 1, 4, 153, 0))
>>> s
[2006, 6, 2, 23, 56, 1, 4, 153, 0]
>>> s.month
6
>>> s.month = 8
>>> s
[2006, 8, 2, 23, 56, 1, 4, 153, 0]

HElement

2006-05-30T00:00:00-07:00

HElement, or HTML-element, is a simple, straight-forward, and non-magical HTML generation engine. An HElement is made of two parts: attrs - the attributes of the tag, and elems - the sub-elements of the tag. Sub-elements can be either HElements by themselves, or plain objects whose str() is used to embed them into the enclosing element.

No validations are performed on the structure -- it's up to you to create something the browser can chew. If you want to do Table(Html(TextField(name = "blah"))), be my guest. Both pretty (indentation and newlines) and compact HTML can be generated.

Code

#
# module helement.py
# requires the AttrDict recipe
#
from AttrDict import AttrDict


class HElementError(Exception):
    pass

_html_mapping = (
    ("&", "&amp;"),
    ("  ", "&nbsp;&nbsp;"),
    (">", "&gt;"),
    ("<", "&lt;"),
    ('"', "&quot;"),
)

def encode_html(obj):
    text = str(obj)
    for chr, enc in _html_mapping:
        text = text.replace(chr, enc)
    return text

class HElement(object):
    _name = None
    _enclosing = True
    _defaults = {}
    _pretty = True
    
    def __init__(self, *elems, **attrs):
        if self._name is None:
            self.name = self.__class__.__name__.lower()
        else:
            self.name = self._name
        if self._enclosing:
            self.elems = list(elems)
        elif elems:
            raise ElementError("element is not a container")
        self.attrs = AttrDict(self._defaults)
        AttrDict.update(self.attrs, attrs)
    
    def _render(self, nesting, pretty):
        items = []
        if pretty:
            indent = "    " * nesting
            subindent = "    " * (nesting + 1)
            items.append(indent)
        items.append("<")
        items.append(self.name)
        for k, v in self.attrs:
            if isinstance(v, bool):
                if v:
                    items.append(" ")
                    items.append(k)
            else:
                items.append(" ")
                items.append(k)
                items.append('="')
                items.append(encode_html(v))
                items.append('"')
        items.append(">")
        if pretty: 
            items.append("\n")
        if self._enclosing:
            for elem in self.elems:
                if isinstance(elem, HElement):
                    items.extend(elem._render(nesting + 1, pretty))
                elif pretty:
                    for line in encode_html(elem).splitlines():
                        items.append(subindent)
                        items.append(line)
                        items.append("\n")
                else:
                    items.append(" ".join(encode_html(elem).splitlines()))
            if pretty:
                items.append(indent)
            items.append("</")
            items.append(self.name)
            items.append(">")
            if pretty:
                items.append("\n")
        return items
    
    def render(self, nesting = 0, pretty = None):
        if pretty is None:
            pretty = self._pretty
        return "".join(self._render(nesting, pretty))
    
    def __repr__(self):
        return self.render(pretty = False)
    
    __str__ = render


class SimpleElement(HElement): 
    _enclosing = False

# headers
class Root(HElement): _name = "html"
class Head(HElement): pass
class Title(HElement): pass
class Meta(HElement): pass
class Body(HElement): pass

# formatting
class Break(SimpleElement): _name = "br"
class Para(HElement): _name = "p"
class Bold(HElement): _name = "b"
class Italics(HElement): _name = "i"
class Underline(HElement): _name = "u"
class Font(HElement): pass
class BulletGroup(HElement): _name = "ul"
class Bullet(HElement): _name = "li"
class Quote(HElement): _name = "q"
class Pre(HElement): pass
class Mono(HElement): _name = "tt"

# forms
class Form(HElement): pass
class Field(SimpleElement): _name = "input"
class TextField(Field): _defaults = {"type" : "text"}
class PasswordField(Field): _defaults = {"type" : "password"}
class CheckboxField(Field): _defaults = {"type" : "checkbox"}
class RadioButton(Field): _defaults = {"type" : "radio"}
class SubmitButton(Field): _defaults = {"type" : "submit"}
class ResetButton(Field): _defaults = {"type" : "reset"}
class SelectField(HElement): _name = "select"
class OptionField(HElement): _name = "option"
class TextArea(HElement): pass

# tables
class Table(HElement): pass
class Row(HElement): _name = "tr"
class Col(HElement): _name = "td"

Exampel

>>> f = Form(
...     "some text",
...     Table(
...         Row(
...             Col(
...                 "hello moshe",
...                 Bold("kaki\npipi"),
...                 TextField(name = "moshe"),
...                 Break(),
...                 "lala",
...                 bgcolor = "white"),
...         ),
...         Row(
...             Col(
...                 SubmitButton(value = "send")
...             )
...         ),
...         width = "100%",
...     ),
...     action = "login",
...     method = "post"
... )
>>>
>>> f
<form action="login" method="post">some text<table width="100%"><tr><td bgcolo
r="white">hello moshe<b>kaki pipi</b><input type="text" name="moshe"><br>lala<
/td></tr><tr><td><input type="submit" value="send"></td></tr></table></form>

>>> print f
<form action="login" method="post">
    some text
    <table width="100%">
        <tr>
            <td bgcolor="white">
                hello moshe
                <b>
                    kaki
                    pipi
                </b>
                <input type="text" name="moshe">
                <br>
                lala
            </td>
        </tr>
        <tr>
            <td>
                <input type="submit" value="send">
            </td>
        </tr>
    </table>
</form>

>>> f.elems[1].elems.append("and some more text")
>>> f.attrs.method = "get"
>>>
>>> print f
<form action="login" method="get">
    some text
    <table width="100%">
        <tr>
            <td bgcolor="white">
                hello moshe
                <b>
                    kaki
                    pipi
                </b>
                <input type="text" name="moshe">
                <br>
                lala
            </td>
        </tr>
        <tr>
            <td>
                <input type="submit" value="send">
            </td>
        </tr>
        and some more text
    </table>
</form>

AttrDict

2006-05-29T00:00:00-07:00

A dictionary that can be accessed with the attribute notation, as well as the item notation: using AttrDict, a.b is the same as a["b"]. Also, iterating over an AttrDict yields (key, value) pairs (instead of only keys, as in dicts).

Note, however, that because of the unification of getattr and getitem, you cannot access the type's methods, such as get or keys. You can however use the more explicit notation - AttrDict.<<method>>(<<instance>>, params...).

code

#
# module AttrDict.py
#
class AttrDict(dict):
    __slots__ = []
    
    def __delattr__(self, name):
        try:
            del self[name]
        except KeyError:
            raise AttributeError(name)
    
    def __getattribute__(self, name):
        try:
            return self[name]
        except KeyError:
            raise AttributeError(name)
    
    __setattr__ = dict.__setitem__
    __iter__ = dict.iteritems

Example

>>> d = AttrDict()
>>> d
{}
>>> d2 = AttrDict(a=1,b=2)
>>> d2
{'a': 1, 'b': 2}
>>> d.x = 5
>>> d.x
5
>>> d["x"]
5
>>> d["y"] = 7
>>> d
{'y': 7, 'x': 5}
>>>
>>> for k, v in d:
...     print k, "=", v
...
y = 7
x = 5
>>>
>>>
>>>
>>> d.blah = "yada"
>>> d.another_blah = "yada"
>>> d
{'y': 7, 'x': 5, 'blah': 'yada', 'another_blah' : 'yada'}
>>> del d.blah
>>> del d["another_blah"]
>>> d
{'y': 7, 'x': 5}
>>>
>>>
>>>
>>> # this is NOT going to work...
...
>>> d.keys()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 14, in __getattribute__
AttributeError: keys
>>>
>>>
>>> # instead, you'll need to do
...
>>> AttrDict.keys(d)
['y', 'x']
>>>
>>>
>>> AttrDict.update(d, d2)
>>> d
{'y': 7, 'x': 5, 'b': 2, 'a': 1}
>>>

ExceptionContainer

2006-05-28T00:00:00-07:00

An exception baseclass inspired by Container, which takes keyword arguments instead of positional ones, and provides pretty-printing in tracebacks.

Code

#
# module ExceptionContainer.py
# requires the Container module
#
from Container import Container


class ExceptionContainer(Exception):
    def __init__(self, message = "", **kw):
        self.__dict__["__info"] = Container(message = message, **kw)
    def __delattr__(self, name):
        delattr(self.__dict__["__info"], name)
    def __getattr__(self, name):
        return getattr(self.__dict__["__info"], name)
    def __setattr__(self, name, value):
        setattr(self.__dict__["__info"], name, value)
    def __repr__(self):
        return repr(self.__dict__["__info"])
    def __str__(self):
        return str(self.__dict__["__info"])

Example

>>> class FileNotFoundError(ExceptionContainer):
...     pass
...
>>> raise FileNotFoundError("failed to open the file!", filename = "/tmp/blah", errno = 17)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
__main__.FileNotFoundError: Container:
    errno = 17
    filename = '/tmp/blah'
    message = 'failed to open the file!'
>>>
>>> try:
...     raise FileNotFoundError("failed to open the file!", filename = "/tmp/blah", errno = 17)
... except FileNotFoundError, ex:
...     ex.errno = 18
...     raise
...
Traceback (most recent call last):
  File "<stdin>", line 2, in ?
__main__.FileNotFoundError: Container:
    errno = 18
    filename = '/tmp/blah'
    message = 'failed to open the file!'
>>>

Container

2006-05-27T00:00:00-07:00

A general container of attributes, with support for pretty printing. Very useful for debugging, etc. Used extensively in Construct to represent parsed objects.

Code

#
# module Container.py
#
def recursion_lock(retval, lock_name = "____recursion_lock"):
    def decorator(func):
        def wrapper(self, *args, **kw):
            if getattr(self, lock_name, False):
                return retval
            setattr(self, lock_name, True)
            try:
                return func(self, *args, **kw)
            finally:
                setattr(self, lock_name, False)
        return wrapper
    return decorator

class Container(object):
    def __init__(self, **kw):
        self.__dict__.update(kw)
    @recursion_lock("<...>")
    def __repr__(self):
        attrs = sorted("%s = %r" % (k, v) for k, v in self.__dict__.iteritems() if not k.startswith("_"))
        return "%s(%s)" % (self.__class__.__name__, ", ".join(attrs))
    @recursion_lock("<...>")
    def __str__(self, nesting = 1):
        attrs = []
        indentation = "    " * nesting
        for k, v in self.__dict__.iteritems():
            if not k.startswith("_"):
                text = [indentation, k, " = "]
                if isinstance(v, Container):
                    text.append(v.__str__(nesting + 1))
                else:
                    text.append(repr(v))
                attrs.append("".join(text))
        attrs.sort()
        attrs.insert(0, self.__class__.__name__ + ":")
        return "\n".join(attrs)

Example

>>> c = Container(a=1, b="baaa")
>>> c # calls repr()
Container(a = 1, b = 'baaa')
>>> print c # calls str()
Container:
    a = 1
    b = 'baaa'
>>> c.c = 1.23
>>> print c
Container:
    a = 1
    b = 'baaa'
    c = 1.23
>>> c.d = Container(e = 6, f = "g")
>>> c
Container(a = 1, b = 'baaa', c = 1.23, d = Container(e = 6, f = 'g'))
>>> print c
Container:
    a = 1
    b = 'baaa'
    c = 1.23
    d = Container:
        e = 6
        f = 'g'
>>> c.h = c # recursive referencing
>>> c
Container(a = 1, b = 'baaa', c = 1.23, d = Container(e = 6, f = 'g'), h = <...>)
>>> print c
Container:
    a = 1
    b = 'baaa'
    c = 1.23
    d = Container:
        e = 6
        f = 'g'
    h = <...>
>>>

Templite

2006-05-26T00:00:00-07:00

Templite is a fast, light-weight, general purpose, and fully featured templating engine in ~40 lines of code. Unlike many templating engines, this one is not specific to any format (HTML, XML, etc.). Instead, all the text surrounded by ${ and }$ is evaluated as python code. Any python code can be used (import modules, define functions, etc.). In order to emit the rendered text into the template, use the emit or emitf functions.

Security note: templating is mainly used in webservers. Having the template stored on the server is fine, but since the template can contain arbitrary python code (like os.system("rm -rf /")), never accept templites from the client side.

Also note that Templite had given birth to Templite+, which is more sophisticated, and you'll probably want to use the latter.

Features:

All the parsing and compiling is performed at preprocessing, so rendering the text is fast
You can escape the ${ delimiter by $\{, and the }$ delimiter by }\$
Use inline, as t = Templite("the template") or load from a file, as t = Templite.from_file("c:\\thefile.txt") or t = Templite.from_file(open("c:\\thefile.txt"))
Easy rendering via __call__ (less typing)

Code

#
# module: templite.py
#
import re

class Templite(object):
    delimiter = re.compile(r"\$\{(.*?)\}\$", re.DOTALL)
    additional_namespace = {}
    
    def __init__(self, template):
        self.tokens = self.compile(template)
    
    @classmethod
    def from_file(cls, file):
        """
        loads a template from a file. `file` can be either a string, specifying
        a filename, or a file-like object, supporting read() directly
        """
        if isinstance(file, basestring):
            file = open(file)
        return cls(file.read())
    
    @classmethod
    def compile(cls, template):
        tokens = []
        for i, part in enumerate(cls.delimiter.split(template)):
            if i % 2 == 0:
                if not part: continue
                tokens.append((False, part.replace("$\\{", "${")))
            else:
                if not part.strip(): continue
                lines = part.replace("}\\$", "}$").splitlines()
                margin = min(len(l) - len(l.lstrip()) for l in lines if l.strip())
                realigned = "\n".join(l[margin:] for l in lines)
                code = compile(realigned, "<templite %r>" % (realigned[:20],), "exec")
                tokens.append((True, code))
        return tokens
    
    def render(__self, __namespace = {}, **kw):
        """
        renders the template according to the given namespace. 
        __namespace - a dictionary serving as a namespace for evaluation
        **kw - keyword arguments which are added to the namespace
        """
        namespace = {}
        namespace.update(__self.additional_namespace)
        namespace.update(__namespace)
        namespace.update(kw)
        
        def emitter(*args):
            for a in args: output.append(str(a))
        def fmt_emitter(fmt, *args):
            output.append(fmt % args)
        namespace["emit"] = emitter
        namespace["emitf"] = fmt_emitter
        
        output = []
        for is_code, value in __self.tokens:
            if is_code:
                eval(value, namespace)
            else:
                output.append(value)
        return "".join(output)
    
    # shorthand
    __call__ = render

Example

>>> demo = r"""
... <html>
...     <body>
...         ${
...         def say_hello(arg):
...             emit("hello ", arg, "<br>")
...         }$
...
...         <table>
...             ${
...                 for i in range(10):
...                     emit("<tr><td> ")
...                     say_hello(i)
...                     emit(" </tr></td>\n")
...             }$
...         </table>
...
...         ${emit("hi")}$
...
...         tralala ${if x > 7:
...             say_hello("big x")}$ lala
...
...         $\{this is escaped starting delimiter
...
...         ${emit("this }\$ is an escaped ending delimiter")}$
...
...         ${# this is a python comment }$
...
...     </body>
... </html>
... """
>>> t = Templite(demo)
>>> print t(x = 8)

<html>
    <body>


        <table>
            <tr><td> hello 0<br> </tr></td>
<tr><td> hello 1<br> </tr></td>
<tr><td> hello 2<br> </tr></td>
<tr><td> hello 3<br> </tr></td>
<tr><td> hello 4<br> </tr></td>
<tr><td> hello 5<br> </tr></td>
<tr><td> hello 6<br> </tr></td>
<tr><td> hello 7<br> </tr></td>
<tr><td> hello 8<br> </tr></td>
<tr><td> hello 9<br> </tr></td>

        </table>

        hi

        tralala hello big x<br> lala

        ${this is escaped starting delimiter

        this }$ is an escaped ending delimiter



    </body>
</html>

Tomer Filiba's Homepage

The Future of Construct

This Expressions

Construct 2.5

Construct 3

Closing Words

Introducing Plumbum - Shell Combinators

Solving Systems of Linear Equations

Easy Syntax for Representing Trees

RPyC 3.2.1 Released

Toying with Context Managers

Stacks, for Fun and Profit

Contextbacks

Lightweight Asynchronism

Wizard Dialog Toolkit

Just Got Me These

WHO HAS CHEEZBURGER?

Too much Wireshark...

Code Generation using Context Managers

Deducible UI

A Brief History

GUI Designers

Deducing UI

"No Way"

Regarding Infix Operators

See Also

All Systems are Go

Foxx0rz

Infix Operators in Python

RPyC Moves to a New Site

Learning Me a Haskell

Hooking Imports for Fun and Profit

Property Classes

Code

Example

Python is Messy

ctypes - Pointer from Address

Example

Copy Function Defaults

Code

Example

Weak Methods

Code

Example

Real Mixins

Code

Example

Hooking dir()

Code

Example

Module Mixins a la Ruby

Code

Killable Threads

Issues

Code

Example

HTML Codec

Code

Example

Weak Attributes

Code

Example

SeqAttr

Example

HElement

Code

Exampel

AttrDict

code

Example

ExceptionContainer

Code

Example

Container

Code

Example

Templite

Code

Example

`This` Expressions