Matt Brubeck

Discovering Urbit: Functional programming from scratch

2010-03-12T00:00:00-08:00

C. Guy Yarvin is a “good friend” of Mencius Moldbug, a pseudonymous blogger known for iconoclastic novella-length essays on politics and history (and occasionally computer science). Guy recently published, under his own name, a project in language and systems design. His own writing about his work is entertaining but verbose (as Moldbug's readers might expect), so I will attempt to summarize it here.

Nock, Urbit, Watt

First there is Nock, “a tool for defining higher-level languages – comparable to the lambda calculus, but meant as foundational system software rather than foundational metamathematics.” Its primitives include positive integers with equality and increment operators, cons cells with car/cdr/cadr/etc., and a macro for convenient branching. Nock uses trees of integers to represent both code and data. Nock is aggressively tiny; the spec linked above is just 33 terse lines.

Next, Guy provides the rationale for Nock. In short, he asks how a planet-wide computing infrastructure (OS, networking, and languages) would look if designed from first priniciples for robustness and interoperability. The answer he proposes is Urbit: a URI-like namespace distributed globally via content-centric networking, with a feudal structure for top-level names and cryptographic identities. Urbit is a static functional namespace: it is both referentially transparent and monotonic (a name, once bound to a value, cannot be un- or re-bound).

Why does this require a new formal logic and a new programming language? In Urbit, all data and code are distributed via the global namespace. For interoperability, the code must have a standard format. Nock's minimal spec is meant to be an unambiguous, unchanging, totally standardized basis for computation in Urbit. Above it will be Watt, a self-hosting language that compiles to Nock. Urbit itself will be implemented in Watt, so Nock and Watt are designed to treat data as code using metacircular evaluation.

The code

A prototype implementation of Watt is on GitHub. It is not yet self-hosting; the current compiler is written in C. Watt is a functional language with static types called “molds” and a mechanism for explicit lazy evaluation. (I was suprised to find I had accidentally created an incompatible lazy dialect of Nock – despite its goal of unambiguous semantics – just by implementing it in Haskell.)

The code is not fully documented, but the repository contains draft specs for both Watt and Urbit. Beware: the syntax and terminology are a bit unconventional. Guy has offered a few exercises to help get started with Nock and Watt:

The Nock challenge:: Write a decrement operator in Nock, and an interpreter that can evaluate it.
Basic Watt:: Write an integer square root function in Watt.
Advanced Watt:: How would you write a function that tests whether molds A and B are orthogonal (no noun is in both A and B)? Or compatible (any noun in A is also in B)? Are these functions NP-complete? If so, how might one work around this in practice?

If you want to learn more, start with these problems. You can email your solutions to Guy.

Will it work?

I find Urbit intellectually appealing; it is a simple and clean architecture that could potentially replace a lot of complex system software. But can we get there from here?

Guy imagines Urbit as the product of an ages-old Martian civilization:

Since Earth code is fifty years old, and Martian code is fifty million years old, Martian code has been evolving into a big ball of mud for a million times longer than Earth software. (And two million times longer than Windows.) …

Therefore, at some point in Martian history, some abject fsck of a Martian code-monkey must have said: fsck this entire fscking ball of mud. For lo, its defects cannot be summarized; for they exceed the global supply of bullet points; for numerous as the fishes in the sea, like the fishes in the sea they fsck, making more little fscking fishes. For lo, it is fscked, and a big ball of mud. And there is only one thing to do with it: obliterate the trunk, fire the developers, and hire a whole new fscking army of Martian code-monkeys to rewrite the entire fscking thing.

… This is the crucial inference we can draw about Mars: since the Martians had 50 million years to try, in the end they must have succeeded. The result: Martian code, as we know it today. Not enormous and horrible – tiny and diamond-perfect. Moreover, because it is tiny and diamond-perfect, it is perfectly stable and never changes or decays. It neither is a big ball of mud, nor tends to become one. It has achieved its final, permanent and excellent state.

Do Earthlings have the will to throw out the whole ball of mud and start from scratch? I doubt it. We can build Urbit but no one will come, unless it solves some problem radically better than current software. Moldbug thinks feudalism will produce better online reputation, but feudal reputation does not require feudal identity; it is not that much harder to build Moldbug's reputation system on Earth than on Mars. I still have not figured out the killer app that will get early adopters to switch to Urbit.

The network is the human being

2010-03-01T00:00:00-08:00

Nathanael Boehm wrote a nice essay last month called The Future of Employment?, about a disconnect between workers' and employers' views of social networks. (This post is based partly on the ensuing Hacker News thread.) Boehm wrote:

When I need help with a challenge at work or need to run some ideas past people I don’t turn to my co-workers, I look to my network of colleagues beyond the walls of my workplace. Whilst my co-workers might be competent at their job they can’t hope to compete with the hundreds of people I have access to through my social networks.

The late Sun Microsystems taught us that the network is the computer. It's true: we still use non-networked computers for specialized tasks, but nobody wants one on their desk – it's just so useless compared to one that talks to the entire world. Boehm could have titled his essay The Network is the Employee. There are still tasks that people do in isolation, but the ability to contact a network of peers and experts makes the difference in my job, and many others.

Alone together

The lone computer programmer in a small business has thousands of colleagues on Stack Overflow, Reddit, and so on. It's a messy way to find answers, but it's sure better than the days when your only choice was to call tech support – or smack the box with your fist, whichever seemed more useful. I can't begin to list all the problems I've solved and things I've learned by Googling for others with experience, and getting help from a different expert for every problem.

Decades before the web, computer geeks had virtual communities on mailing lists, Usenet, and IRC. Now every job in the world has its corresponding forum. Even the night clerk at the gas station has Not Always Right.

Teaching has long been a solitary profession. Despite working in a crowded classroom, teachers are isolated; they rarely have colleagues observing or participating directly in their work. This has such an impact that teachers are sometimes trained in meditation or reflection techniques, to make up for the lack of external feedback. So I'm curious what happens when teachers start to work together remotely the way programmers do.

You will be assimilated

Boehm's essay also reminded me of a vague sci-fi idea I've been kicking around: the first group minds will evolve from the intersection of Mechanical Turk, virtual assistants, social networking, and augmented reality.

Starting around the 1990s, it was possible to instantly "know" any fact that was published online. Since then, we've increased the amount of content online, our tools for searching it, and ways of connecting to the network. Today we have instant access to almost any published knowledge, anywhere.

There are more people on the net too, and more ways to find and talk to them. Most of us can contact dozens of friends at any given moment, plus friends-of-friends, co-workers, fellow members of communites like Hacker News or MetaFilter, and also complete strangers. Along with raw facts, we have access to vast amounts of human judgement, experience, and skill.

One product of this is the "virtual assistant," who provides a service that was once exclusive to high-powered executives. Now personal assistants can work remotely (often overseas), spread costs by serving many masters, and leverage the internet superpowers listed above. Their services are mostly targeted at small business owners and the Tim Ferriss crowd, but I'm sure someone soon will market virtual assistance to all sorts of other creative workers, teachers, even stay-at-home parents.

So, how long before I can touch a button to let a remote assistant see what I'm seeing in real-time and help me make transportation plans, translate foreign signs and speech, look up emails related to whatever I'm doing or thinking, or even advise me on what to say? Some of these queries will go to my circle of friends, others to the general public, and some to a personal assistant who is paid well to keep up with my specific needs. And that assistant of course will subcontract portions of each job to computer programs, legions of cheap anonymous Turkers, or his or her own network of helpers. At that point, I'm augmenting my own perception, memory, and judgement with a whole network of brains that I carry around, ready to engage with any situation I meet.

If nothing else, I hope someone writes a good sci-fi thriller story in which a rogue virtual assistant manipulates the actions of unsuspecting clients, leading them to some unseen end.

Finding domain names with Node.js

2010-01-13T00:00:00-08:00

I'm working on some ideas for finance or news software that deliberately updates infrequently, so it doesn't reward me for reloading it constantly. I came up with the name "microhertz" to describe the idea. (1 microhertz ≈ once every eleven and a half days.)

As usual when I think of a project name, I did some DNS searches. Unfortunately "microhertz.com" is not available (but "microhertz.org" is). Then I went off on a tangent and got curious about which other SI units are available as domain names.

This was the perfect opportunity to try node.js so I could use its asynchronous DNS library to run dozens of lookups in parallel. I grabbed a list of units and prefixes from NIST and wrote the following script:

var dns = require("dns"), sys = require('sys');

var prefixes = ["yotta", "zetta", "exa", "peta", "tera", "giga", "mega",
  "kilo", "hecto", "deka", "deci", "centi", "milli", "micro", "nano",
  "pico", "femto", "atto", "zepto", "yocto"];

var units = ["meter", "gram", "second", "ampere", "kelvin", "mole",
  "candela", "radian", "steradian", "hertz", "newton", "pascal", "joule",
  "watt", "colomb", "volt", "farad", "ohm", "siemens", "weber", "henry",
  "lumen", "lux", "becquerel", "gray", "sievert", "katal"];

for (var i=0; i<prefixes.length; i++) {
  for (var j=0; j<units.length; j++) {
    checkAvailable(prefixes[i] + units[j] + ".com", sys.puts);
  }
}

function checkAvailable(name, callback) {
  dns.resolve4(name).addErrback(function(e) {
    if (e.errno == dns.NXDOMAIN) callback(name);
  })
}

Out of 540 possible .com names, I found 376 that are available (and 10 more that produced temporary DNS errors, which I haven't investigated). Here are a few interesting ones, with some commentary:

exasecond.com – 32 billion years
petasecond.com – 32 million years
petawatt.com – can be produced for femtoseconds by powerful lasers
terapascal.com
gigakelvin.com – possible temperature of picosecond flashes in sonoluminescence
giganewton.com – 225 million pounds force
gigafarad.com
kilosecond.com – 16 minutes 40 seconds
kilokelvin.com – 1340 degrees Fahrenheit
centiohm.com
millifarad.com
microkelvin.com
picohertz.com – once every 31,689 years
picojoule.com
femtogram.com – mass of a single virus
yoctogram.com – a hydrogen atom weighs 1.66 yoctograms
zeptomole.com – 602 molecules

To get the complete list, just copy the script above to a file, and run it like this: node listnames.js

Along the way I discovered that the API documentation for Node's dns module was out-of-date. This is fixed in my GitHub fork, and I've sent a pull request to the author Ryan Dahl.

Weekend hack: Outline grep

2010-01-12T00:00:00-08:00

I keep almost all of my notes and to-do lists in plain text files, so I can edit and search them with Vim, grep, and other standard Unix tools. I often indent lines in these files to create a simple outline structure, and use the autoindent and foldmethod=indent options to make Vim into a simple outliner.

To get useful output when searching through these outline-structured files, I wrote a simple grep replacement. Given a text file with a Python-style indentation structure, ogrep searches the file for a regular expression. It prints matching lines, with their "parent" lines as context. For example, if input.txt looks like this:

2009-01-01
  New Year's Day!
    No work today.
    Visit with family.
2009-01-02
  Grocery store and library.
2009-01-03
  Stay home.
2009-01-04
  Back to work.
    Remember to set an alarm.

then ogrep work input.txt will produce the following output:

2009-01-01
  New Year's Day!
    No work today.
2009-01-04
  Back to work.

You can download ogrep from the outline-grep repository on GitHub, or just read the literate Haskell file. The code is almost trivial (40 lines of code, plus imports and comments); I'm publishing it just in case anyone else has a use for it, and because some of my friends were curious about how I'm using Haskell. I've now written a few "real-world" Haskell programs (compleat was the first). I'm finding Haskell very well suited to such programs, though this particular one would be equally easy in a language like Perl, Python, or Ruby.

This is a one-off tool to fill a gap in my workflow; there are no configuration options or useful error messages. It would be fairly easy to extend it, though. For example, an option to include children (as well as parents) of matching lines might be handy. I recently realized that ogrep often works for searching through source code too, which may generate some more unexpected use cases.

Android 2.0 uses V8 JavaScript engine

2009-11-06T00:00:00-08:00

Google has not yet released most of the Android 2.0 "eclair" source code, but they did publish source for a very small number of components, including a WebKit snapshot. I was excited to see that the snapshot includes Google's V8 virtual machine. (Previous Android releases used Safari's JavaScriptCore/"SquirrelFish Extreme" VM.) But without the rest of the source tree, there was no way to build and run this on a real Android phone. The SDK includes a binary image that runs only in the qemu-based emulator.

Today I got to try out a Motorola Droid. Here's how its browser compares to Android 1.6 on my HTC Dream (Android Dev Phone / T-Mobile G1) in the V8 Benchmark Suite:

Test	Dream (1.6)	Droid (2.0)	Change
Richards	13.5	15.6	+16%
DeltaBlue	5.23	12.9	+147%
Crypto	13.2	10.9	-17%
RayTrace	10.9	80.1	+635%
EarleyBoyer	23.5	74.7	+218%
RegExp	did not complete	16.5	–
Splay	did not complete	did not complete	–

Some tests (Richards, Crypto) see little or no improvement, while others (DeltaBlue, RayTrace, EarleyBoyer) are dramatically faster. Just for comparison, let's run the same benchmark on Safari 4 (JavaScriptCore) and a Chromium 4 nightly build (V8) on a Mac Pro:

Test	Safari 4	Chromium 4	Change
Richards	4103	4640	+13%
DeltaBlue	3171	4418	+39%
Crypto	3331	3643	+9%
RayTrace	3509	6662	+90%
EarleyBoyer	4737	7643	+61%
RegExp	1268	1187	-6%
Splay	1198	7290	+509%

The precise ratios are different, but the same tests that showed the most improvement from Android 1.6 to 2.0 also show the most improvement from Safari to Chrome. Based on this plus the source code snapshot, I'm pretty sure that Android 2.0 is indeed using V8.

This is exciting news. It makes Droid the first shipping product I know that uses V8 on an ARM processor, although V8 has included an ARM JIT compiler for some time now. [Correction: Palm Pre was first; see the comments below.] For mobile web developers like me, it means we're one step closer to having desktop-quality rich web applications on low-power handheld devices.

Android still lags behind the iPhone in at least one important way for web developers: CSS animation. The iPhone (and Safari on the desktop) provides hardware acceleration for CSS transforms, like this falling leaves demo. On Android, CSS animation is done in software, making it much, much slower. (Even outside the browser, Android's Skia 2D graphics API lacks hardware acceleration. OpenGL is the only way to for Android developers to take advantage of the GPU.) Accelerated animation would really make it possible to write interactive web pages that match the smoothness and responsiveness of native apps.

Final thought: Although the Motorola Droid is still 100 times slower than Chromium on a Mac Pro, it's already faster at some benchmarks than IE8 or Firefox 2 on desktop hardware from just a few years ago.

Update (2010-02-09): Just for comparison, here are numbers for the Google/HTC Nexus One with Android 2.1. The Nexus One is around 2-4 times faster than the Droid at the V8 benchmark suite. It even renders the falling leaves animation at a decent framerate (but still not as smoothly as the GPU-accelerated iPhone).

Test	Droid (2.0)	Nexus One (2.1)	Change
Richards	15.6	52.1	+234%
DeltaBlue	12.9	60.2	+367%
Crypto	10.9	31.7	+191%
RayTrace	80.1	170	+112%
EarleyBoyer	74.7	126	+69%
RegExp	16.5	27.5	+67%
Splay	did not complete	did not complete	–

Compleat: Bash completion for human beings

2009-10-30T00:00:00-07:00

Compleat is an easy, declarative way to add smart tab completion for any command. It's written in Haskell but requires no programming knowledge. See the GitHub repository for a quick description, or read on for a complete explanation.

Background

I'm one of those programmers who loves to carefully tailor my development environment. I do nearly all of my work at the shell or in a text editor, and I've spent a dozen years learning and customizing them to work more quickly and easily.

Most experienced shell users know about programmable completion, which provides smart tab-completion for supported programs like ssh and git. You can also add your own completions for programs that aren't supported—but in my experience, most users never bother.

At Amazon, everyone used Zsh (which has a very powerful but especially baroque completion system) and shared the completion scripts they wrote for our myriad internal tools. Now that I'm in a startup with few other command line die-hards, I'm on my own when it comes to extending my shell.

So I read the fine manual and started writing completions. You can see the script I made for three commands from the Google Android SDK. It's 200 lines of shell code, and fairly straightforward if you happen to be familiar with the Bash completion API. But as I cranked out more and more case statements, I felt there must be a better way...

The Idea

It's not hard to describe the usage of a typical command-line program. There's even a semi-standard format for it, used in man pages and generated by libraries like AutoOpt. For example, here's the usage for android, one of the SDK commands supported by my script:

 android [--silent | --verbose]
   ( list [avd|target]
   | create avd ( --target <target> | --name <name> | --skin <name>
                | --path <file> | --sdcard <file> | --force ) ...
   | move avd (--name <avd> | --rename <new> | --path <file>) ...
   | (delete|update) avd --name <avd>
   | create project ( (--package|--name|--activity|--path) <val>
                    | --target <target> ) ...
   | update project ((--name|--path) <val> | --target <target>) ...
   | update adb )

My idea: What if you could teach the shell to complete a program's arguments just by writing a usage description like this one?

The Solution

With Compleat, you can add completion for any command just by writing a usage description and saving it in a configuration folder. The ten-line description of the android command above generates the same results as my 76-line bash function, and it's so much easier to write and understand!

The syntax should be familiar to long-time Unix users. Optional arguments are enclosed in square brackets; alternate choices are separated by vertical pipes. An ellipsis following an item means it may be repeated, and parentheses group several items into one. Words in angle brackets are parameters for the user to fill in.

Let's look at some more features of the usage format. For programs with complicated arguments, it can be useful to break them down further. You can place alternate usages on their own lines separated by semicolons, like this:

android <opts> list [avd|target];
android <opts> move avd (--name <avd>|--rename <new>|--path <file>)...;
android <opts> (delete|update) avd --name <avd>;

...and so on. Rather than repeat the common options on every line, I used a parameter named "opts". I can define that parameter to be a sub-pattern, which will be used wherever <opts> appears:

opts = [ --silent | --verbose ];

For parameters whose values are not fixed but can be computed by another program, we use a ! symbol followed by a shell command to generate completions, like this:

avd = ! android list avd | grep 'Name:' | cut -f2 -d: ;
target = ! android list target | grep '^id:'| cut -f2 -d' ' ;

Any parameter without a definition will use the shell's built-in completion rules, which suggest matching filenames by default.

The source code is on GitHub. I've been using it for just a week and I'm now writing new usage files for myself almost every day. The README file has more details about the usage syntax, and instructions for installing the software. Give it a try, and please send in any usage files that you want to share! (Questions, bug reports, or patches are also welcome.)

Future Work

For the next release of Compleat, I would like to make installation easier by providing better packaging and pre-compiled binaries; support zsh and other non-bash shells; and write better documentation.

In the long term, I'm thinking about replacing the usage file interpreter with a compiler. The compiler would translate the usage file into shell code, or perhaps another language like C or Haskell. This would potentially improve performance (although speed isn't an issue right now on my development box), and make it easy for usage files to include logic written in the target language. Another idea for the future: What if option-parsing libraries like AutoOpt or the Ruby/Perl/Python equivalents generated completion scripts for every program you wrote?

Final Thoughts

I realized recently that some things I do are so specialized that my parents and non-programmer friends will probably never get them. For example, Compleat is a program to generate programs to help you… run programs? Sigh. Well, maybe someone out there will appreciate it.

Compleat was my weekends/evenings/bus-rides project for the last few weeks (as you can see in the GitHub punch card), and my most fun side project in quite a while. It's the first "real" program I've written in Haskell, though I've been experimenting with the language for a while. Now that I'm comfortable with it, I find that Haskell's particular combination of features works just right to enable quick exploratory programming, while giving a high level of confidence in the behavior of the resulting program. Compleat 1.0 is just 160 lines of Haskell, excluding comments and imports. Every module was completely rewritten at least once as I compared different approaches. (This is much less daunting when the code in question is only a couple dozen lines.) I don't think this particular program would have been quite as easy to write—at least for me—in any of the other platforms I know (including Ruby, Python, Scheme, and C).

I had the idea for Compleat more than a year ago, but at the time I did not know how to implement it easily. I quickly realized that what I wanted to write was a specialized parser generator, and a domain-specific language to go with it. Unfortunately I never took a compiler-design class in school, and had forgotten most of what I learned in my programming languages course. So I began studying parsing algorithms and language implementation, with Compleat as my ultimate goal.

My good friend Josh and his Gazelle parser generator helped inspire me and point me toward other existing work. Compleat actually contains three parsers. The usage file parser and the input line tokenizer are built on the excellent Parsec library. The usage file is then translated into a parser that's built with my own simple set of parser combinators, which were inspired both by Parsec and by the original Monadic Parser Combinators paper by Graham Hutton and Erik Meijer. The simple evaluator for the usage DSL applies what I learned from Jonathan Tang's Write Yourself a Scheme in 48 Hours. And of course Real World Haskell was an essential resource for both the nuts and bolts and the design philosophy of Haskell.

So besides producing a tool that will be useful to me and hopefully others, I also filled in a gap in my CS education, learned some great new languages and tools, and kindled an interest in several new (to me) research areas. It has also renewed my belief in the importance of "academic" knowledge to real engineering problems. I've already come across at least one problem in my day job that I was able to solve faster by implementing a simple parser than I would have a year ago by fumbling with regexes. And I'll be even happier if this inspires some friends or strangers to take a closer look at Haskell, Parsec, or any problem they've thought about and didn't know enough to solve. Yet.

Colophon

2009-10-28T00:00:00-07:00

This site is powered by Jekyll and based on styles and markup by Tom Preston Werner. Comments are run by Disqus and hosting is by Dreamhost.

Before starting this site, I had an Advogato diary for writing about software. I also have a personal journal (mostly interesting to my friends and family).