Matt Brubeck

Android app permissions and Firefox Beta

2011-10-11T10:00:00-07:00

As a non-profit organization, Mozilla has a strong commitment to personal privacy and empowerment. But after we released the last update to Firefox Beta for Android, many people started asking us why Firefox needed access to their phone numbers.

Firefox does not access users’ phone numbers, but it was clear that we needed to address this concern. Where did these questions come from? Here’s the first thing users saw when installing or updating Firefox Beta in the Android Market:

The “Phone Calls” permission was added in the last update to Firefox Beta (but has been since been removed, as I’ll explain below). When users installed that update and tapped on “Phone calls” for more information, they saw this:

Why did Firefox Beta ask for this permission? Firefox did not ever access phone numbers, serial numbers, or phone calls. But it did have code to detect the type of network connection: 2G, 3G, 4G, Wi-Fi, and so on. Firefox or add-ons could use this code to change settings automatically based on network type, for example to use less data on mobile networks.

Unfortunately, this required permission to READ_PHONE_STATE, which also grants access to very sensitive data. We knew this would worry some users, so we immediately started working on explaining how and why Firefox uses various permissions. We now have this information on our support site and will link to it from our Android Market page.

But the reaction to the new permission in Firefox Beta was so strong that we decided to remove that permission completely, along with the code that used it. Now when you go to the Android Market to install Firefox Beta, it will no longer ask to read “phone state and identity.”

Thoughts on Android Permissions

Permissions on Android and similar platforms are not perfect, but they do give users some useful tools to protect themselves. When an app requests only minimal permissions, users know it can do only limited damage if it is buggy or malicious. Recent versions of Android also have well-written explanations of each permission to help users make decisions.

But when an app requests lots of permissions, users have a tough choice. They can grant the permissions, or not use the app at all. This is especially bad for permissions like READ_PHONE_STATE that are needed for some reasonable features but also provide access to sensitive data. Eventually, most people probably get used to granting whatever permissions are requested, especially for apps like Facebook and Netflix that provide unique access to popular services.

Making permissions finer-grained might help (for example, separating “Read phone number” from “Read connection type”), but would also mean longer lists of permissions. That could make users even less likely to read and understand them. Explanations from developers can also help, but only if users trust them to tell the truth. Allowing users to grant or deny individual permissions (perhaps only at the time the app needs them) might help too, or it might just train users to always grant permissions so that apps will stop nagging them.

Aside from these overall design issues, there are also bugs in the developer documentation, and a bug that causes old permissions to stick around even after updating to a new version that doesn’t need them. These little bugs make it harder for developers to do the right thing. Some researchers at UC Berkeley have analyzed the Android source code to produce tools and documentation that fill in some of the gaps for developers.

The good news is that some users are paying attention, and those users make things better for everyone by pressuring developers (like us!) to remove invasive permissions. If you’re one of the Firefox fans who wrote to us about the new permissions in Firefox Beta, thank you! We appreciate it.

Mobile web developers: Your users hate it when you do this

2010-11-19T07:15:00-08:00

Mobile Firefox beta releases include a “Feedback” add-on (like the one in Firefox 4 beta for desktop), which lets users tell us what they think about the new browser. Based on a sample of feedback from mobile beta testers, the most common complaints are about:

Speed
Fitting text to the screen when zoomed in
Mobile vs. desktop versions of web sites

The first two are straightforward, though not necessarily easy. We’re always working on performance, and we have experimental text reflow code (currently available in the Easy Reading add-on). But the last item is more complicated…

Browser detection pitfalls

Web sites can read the User-Agent header sent by your browser to see what browser and OS you are using. Some sites use that information to decide whether to send a “full” version of a web page, or a version formatted for mobile devices.

This can go wrong in several ways. If your browser or device is new, or wasn’t tested when a site was developed, that site has no way of knowing whether it is “mobile.” Users may also change their User-Agent to work around content restrictions or access different media formats. And some sites make incorrect assumptions, like that all browsers with “Android” in their User-Agent string are based on WebKit.

Even when the browser is known, readers and publishers might not agree about whether the mobile or desktop version is better. Based on our feedback, some users want to switch from full sites to mobile sites while others want just the opposite. And some devices, like large touch-screen tablets, combine aspects of handheld and desktop computers.

Solutions

Looking through these complaints, many people are under the mistaken impression that the browser, rather than the web site, decides whether to display mobile-formatted pages. Even the New York Times' David Pogue gets this wrong in his Galaxy Tab review:

When you visit sites like nytimes.com, CNBC.com and Amazon.com, the Galaxy’s browser shows the stripped-down, mobile versions of those sites. According to Samsung, there’s no way to turn that feature off and no way to visit the full-size sites. You can delete the little “m.” in the Web address until you’re blue in the browser, but the Galaxy always puts it right back.

Web developers: your readers are begging us to display your content in their preferred format. We want to help them, but we can’t do it alone.

(I wrote an add-on called Phony that lets mobile Firefox impersonate the User-Agent strings of other browsers. While this improves the experience on some sites, it breaks it on others. Masquerading as another browser can lead sites to serve non-standard markup that do not work in Firefox. Trying to solve this in the browser creates as many problems as it solves.)

Because browser detection is never perfect, web sites should let readers choose between mobile and full content. They can try to guess the right version by default, but please let users opt in or out.

Suggestions for web developers

Here are some first steps typical mobile web sites can take to make their readers happier:

When appropriate, serve the same content to all browsers. You can use stylesheets and scripts to customize your layout for different display sizes, as in this beautiful site by Jon Hicks.
There are valid reasons to use User-Agent sniffing. But if you must use it, test in as many browsers and devices as possible and learn the correct way to detect various browsers. For example, you can detect Gecko-based browsers by looking for Gecko and rv:, and you can detect mobile Firefox by looking for Fennec/.
If a “mobile” user requests a page that isn’t available on your mobile site, don’t just redirect them to an unrelated mobile landing page.
Let users switch from your mobile site to your full site and vice-versa. You can remember users' previous choices for convenience, but let them change their minds.

What's different about Firefox for Android

2010-10-04T16:00:00-07:00

The Mozilla Mobile team has been working for several months on Firefox 4 for Android and Maemo (also known as “Fennec”). Here are some thoughts about the challenges we’ve discovered, how we’ve decided to solve them, and why Firefox is different from other mobile browsers.

Why

People often ask us why Android needs another web browser. These are a few things Firefox does that other Android browsers don’t:

Syncs bookmarks, tabs, history, passwords, and form data to and from your phone. Firefox Sync and the Firefox Awesomebar help you enter URLs and passwords with less typing, and move seamlessly between your desktop and your mobile phone.
Lets anyone write add-ons that can customize any part of the user interface. (Dolphin HD is another Android browser with some great add-ons, but its add-ons are provided by the browser vendor.)
Uses the Jaegermonkey JIT, which is getting faster all the time. It runs JavaScript much faster than the Android 2.1 browser, and is even faster than the Android 2.2 browser on the WebKit’s own SunSpider benchmark.
Supports web technologies like SVG, ECMAScript 5, WebM, and HTTP Strict Transport Security. Firefox for Android currently scores 217 points plus 9 bonus points on html5test.com. (Warning: Those tests can be deceptive; use them as a starting point for comparison only.)

There are several other browsers for Android, but all of them use the built-in WebKit rendering engine (except Opera Mini, which uses a proxy server for rendering). The same is true for iOS, which uses WebKit too – as do the latest versions of BlackBerry, Symbian, and Palm webOS. [Update, November 2010: Opera Mobile for Android is now in beta and uses its own rendering engine too.]

Part of the point of Firefox is to provide alternate capabilities, rather than reuse the built-in ones. Firefox for Android uses same Gecko engine as Firefox 4 for desktop. That’s how it can support features that Android’s WebKit doesn’t, like SVG and ES5. (Of course, WebKit supports some features Gecko doesn’t, which is why it’s great to have a choice.)

Speed and responsiveness

Early Firefox for Android builds were very slow compared to the stock browser. Performance is critical in a mobile browser, and our work in this area is starting to pay off. The new beta version is much speedier, and we have plans to make it even faster.

To make sure Firefox’s interface stays responsive even when pages are rendering, we split the browser into two processes: one for the user interface, and one to render and run scripts in web content. This was part of the Electrolysis project.

To make scrolling and zooming fast, Mozilla’s graphics team has implemented a new “Layers” architecture to allow hardware acceleration and other optimizations. Beta 1 is the first mobile Firefox release to take advantage of this work. Because of Electrolysis, we need to share layers between the UI process and the content process. These cross-process layers allow Firefox to scroll smoothly in response to user input, even if the content process is still busy rendering the page.

These are just the first steps toward making Firefox fast on mobile devices. Upcoming releases will feature OpenGL hardware-accelerated compositing, for improved scrolling of complex pages. They will also load web pages faster, thanks to optimizations of our inter-process communication.

Installation size: Problems and solutions

Not all mobile platforms allow browser apps to include low-level components like JIT compilers. Fortunately there are still platforms like Android, webOS, and Maemo that let apps bundle any libraries they want. But while Android allows us to distribute our own rendering engine and JavaScript compiler, it isn’t really built with apps like Firefox in mind.

Unlike browsers that use the stock WebKit library, Fennec must ship its own rendering engine. Many Android phones were built with just 64 MB to 512 MB of storage for apps. Users who think nothing of a 12 MB download to install Firefox or Chrome on a laptop may think twice before installing it on these phones! Storage space is much larger on newer phones, but this is still an issue for many users.

Even worse, a quirk of the Android NDK means these native libraries are saved twice – both compressed inside the APK and extracted to a folder for loading. For apps like Firefox that are mostly native code, this more than doubles the installation size. Other NDK apps like Google Earth pay the same double storage penalty.

To solve this problem, Mozilla’s Michael Wu wrote a custom dynamic linker that loads libraries from the APK without installing them to a folder. This cut the installed size by more than half, but increased startup time slightly. For newer phones with 1 GB or more of internal storage, we might want to let Firefox take more space but start faster. On phones with less storage, we can use the custom linker to save space.

Firefox 4 beta 1 needed about 40 MB of storage on Android. With the custom linker, Firefox 4 beta 2 takes a fraction of the space, and in Android 2.2 you can move almost all of it to SD.

Hardware compatibility

Stock libraries have another advantage: They can be optimized for specific hardware. In contrast, apps usually come in a single flavor for all devices. Firefox can use ARMv7 features like Thumb-2 and NEON to run as fast as possible on high-end Android phones – but with these optimizations it can’t run at all on low-end hardware. To run optimally on all current hardware, we’d need different builds for different devices. For now we’re focusing on the current high-end phones, which will likely be next year’s mainstream hardware.

Even in this smaller set of devices, we’ve run into problems. Most recently, we discovered that Firefox’s JIT code crashes unpredictably on Samsung Galaxy S phones. This seems to be a bug in the Android 2.1 kernel on these devices. Other developers are seeing similar problems, including crashes in the MonoDroid JIT and in Android’s own Dalvik JIT. For now, Firefox’s JIT features are disabled when running on Galaxy S hardware. This makes JavaScript slower, but a lot more stable. The problem is fixed in leaked Android 2.2 images, so we expect to re-enable the JITs before long.

Competition and choice

Firefox is built by Mozilla, a non-profit organization with a mission to promote openness, innovation, and opportunity on the web. We want our work on the mobile web to benefit everyone, not only Firefox users – just as Firefox on the desktop helped create a new era of innovation and standards for all browsers.

WebKit is an excellent project. But a growing number of mobile sites work only on WebKit. This is dangerously similar to the web ten years ago, when Internet Explorer had an overwhelming market share and many sites used IE-specific markup. That made it hard for other browsers to compete, which killed the incentive for the dominant browser to keep improving.

Upcoming platforms like MeeGo and Windows Phone will give WebKit some new mobile competition – but many users still can’t choose new browser technology without buying new hardware (and often new service contracts). We think you should have a meaningful choice of browsers on your current phone, just like you do on your computer. User choice will encourage all browsers to innovate and learn from each other, so they all improve faster.

Try it out

To check if your phone is compatible with Firefox 4 beta, go to our download page. This is a test release, and we aren’t finished fixing and optimizing it – but we are working hard. Let us know what you think!

We have a lot more work ahead of us. Our next releases will include even more exciting changes like the new Android skin, reduced installation size, and more speed improvements.

Changes for add-ons in Fennec 2.0 alpha 1

2010-09-02T16:11:00-07:00

Update (October 2010): We’ve changed the version number of the next mobile Firefox release from 2.0 to 4.0. The alpha release was numbered 2.0a1, but the first beta release will be numbered 4.0b1.

Last week we released a new alpha version of Firefox for Android and Maemo (a.k.a. Fennec). This release brings some major changes and new features for add-on authors. Our Fennec add-on documentation now has the details you need to start updating your Fennec add-ons or creating new ones.

What’s new for add-ons?

One very big change in Fennec 2.0 is Electrolysis, the project to move content and chrome into separate processes. Any add-on code that interacts with web content through the DOM must now be in a separate script that runs in the content process. For details, see the Electrolysis guide for add-on authors.

Fennec 2.0a1 also features new APIs for extending the context menu and site menu. See the User Interface Guide for links to documentation and example code.

The upcoming beta releases will include even more changes. Add-ons that use Fennec’s panning and zooming features will probably need significant changes for the new graphics code in Fennec 2.0b1. We will also include APIs for for add-ons to customize sharing and other new features. If you are working on an add-on that is affected by these changes, please let us know.

Get started

To start updating or creating your Fennec add-on, download Firefox for Android and Nokia N900 or get the emulator for Mac/Windows/Linux. When you’re ready, update your addons.mozilla.org listing and set the maxVersion to 2.0a1. Or you can start getting ready for beta by setting your maxVersion to 2.0b1pre and keeping up-to-date with our pre-beta nightly builds.

Fennec 2.0 update: The road to alpha

2010-08-12T16:23:00-07:00

Update (October 2010): We’ve changed the version number of the next mobile Firefox release from 2.0 to 4.0. I’ve updated this article to use the new version.

The Mozilla Mobile team has been quiet lately. We’re making a lot of under-the-hood changes for the next version of Fennec (Firefox for mobile), and have been focused on getting basic functionality working again after some major platform changes.

Now things are starting to stabilize, and we are gearing up for an alpha release in just a few weeks. There are still noticeable bugs in our current builds, but it is possible to use them now for testing, add-on development, and regular web browsing (if you don’t mind occasional crashes).

Under the hood

The biggest back-end change in Fennec 4.0 is Electrolysis (a.k.a “e10s”). By moving content rendering and JavaScript into a separate process, e10s allows the Fennec UI to stay responsive while pages are loading. This required us to rewrite large parts of the Fennec UI and platform code, a process that is finally approaching completion.

After the alpha release, the next big platform changes will be related to Layers. Fennec currently handles panning and zooming by dividing pages into “tiles” and rendering them on HTML canvas elements. This works, but it is complicated and not as fast as we’d like. The new layers system will let us replace Fennec’s custom tile management with hardware-accelerated rendering and compositing built into the Firefox 4.0 platform.

Features

We have a bunch of new features planned for the Fennec UI. A few of these have already started to land, so you can try them in nightly builds or the upcoming alpha:

Firefox Sync is now built in – sync tabs, bookmarks, and history from your computer to your phone, no add-on required!
The new Find In Page command is available through the site menu (or by pressing Control+F on a hardware keyboard).
You can now share links through Twitter, Facebook, Google Reader, or email. (The final version of this feature will also let you send links using native Android or Maemo apps.)
Fennec alpha can use your phone’s address book to make it easy to enter phone numbers and email adresses into web forms. (This works on Maemo now; support for Android will be added later.)
We’re adding multi-touch gestures. Pinch zoom has landed for alpha; later releases will also include multi-touch swipe gestures to go to the top or bottom of the current page, or navigate between pages.

The design of these features is not yet final, so their look and feel may change significantly before the final release.

Android

Fennec 1.0 and 1.1 were for available only for Nokia’s Maemo operating system. Fennec 4.0 will run on the Google Android platform, as well as Maemo and its successor MeeGo.

Fennec for Android is brand new, but it is progressing fast. Most of the blocking bugs for alpha 1 have been fixed in the last few days, and the very latest nightly builds are usable for regular browsing, though still rough in places.

Some of our most visible Android bugs were related to keyboard and input method support. Jim Chen’s IME rewrite fixed a lot of these bugs, including a crash on startup with the popular Swype keyboard. There are still a few keyboard bugs left to fix before alpha 1.

Other Android changes, like alert-bar notifications and a new visual theme, will appear in our beta releases this fall.

Add-ons

For Fennec add-ons, the biggest change coming is Electrolysis. Any add-on code that interacts with web content through the DOM must now be in a separate script that runs in the content process. Mark Finkle has written a very useful Electrolysis guide for add-on authors.

In Fennec 1.1 we added the site menu and context menu. Fennec 4 will have improved APIs for add-on authors to add new items to either of those menus. Documentation of the new APIs is coming soon.

Nightly builds

Alpha 1 will go into code freeze as soon as the remaining a1 blocker bugs are fixed. If you want to start testing or developing for Fennec 4 even sooner, you can download a nightly build today. Just remember this is still pre-alpha software, and you should expect bugs.

Maemo users can use the trunk repo to stay up to date with the latest nightly build.
For Android users, the Fennec for Android wiki page has pointers to the latest nightly build, plus a list of known bugs and compatible hardware. (A number of Android blogs have linked recently to random TryServer builds and out-of-date blog posts. Please go to the wiki page instead to get the latest information.)
If you don’t have a compatible Maemo or Android device, you can always download nightly Fennec builds for Mac/Windows/Linux and try out Fennec on your desktop or laptop.

If you have questions, feedback, or bug reports, file them under Fennec in Bugzilla, or come to the #mobile channel on irc.mozilla.org to chat with us!

Implementing the viewport meta tag in Mozilla Fennec

2010-05-10T15:36:00-07:00

The upcoming release of Mobile Firefox (Fennec) 1.1 features improved support for the <meta name="viewport"> tag. Previous version of Fennec supported the width, height, and initial-scale viewport properties, but had problems with some sites designed for iPhone and Android browsers. We now support the same properties Safari does, and we changed Fennec to render mobile sites more consistently on screens of different sizes and resolutions.

touch.facebook.com before:

touch.facebook.com after:

You can see these changes for yourself in the latest Fennec 1.1 or trunk nightly builds.

Background

Mobile browers like Fennec render pages in a virtual “window” (the viewport), usually wider than the screen, so they don’t need to mangle existing layouts by squeezing them into a tiny window. Users can pan and zoom to display different areas of the viewport.

Mobile Safari introduced the “viewport meta tag” to let web developers control the viewport’s size and scale. Many other mobile browsers now support this tag, although it is not part of any web standard. Apple’s documentation does a great job explaining how it works for web developers, but it leaves out some information that would be useful to browser vendors. For example, it says the content attribute is a comma-separated list, but existing browsers and web pages use a mix of commas, semicolons, and spaces as separators.

A pixel is not a pixel

The iPhone and many popular Android phones have 3- to 4-inch screens with 320×480 pixels (~160 dpi). Fennec’s target devices have the same physical size but 480×800 pixels (~240 dpi). Because of this, the last version of Fennec displayed many pages about one third smaller (in physical units) than iPhone or Android. This caused usability and readability problems on many touch-optimized web sites. Peter-Paul Koch wrote about this problem in A pixel is not a pixel is not a pixel.

Fennec 1.1 for Maemo will use 1.5 hardware pixels for each CSS “pixel,” following the lead of the Android browser. This means a site with “initial-scale=1” will render at the same physical size in Fennec for Maemo, Mobile Safari for iPhone, and the Android Browser on both HDPI and MDPI phones. It’s also consistent with the CSS 2.1 specification, which says:

If the pixel density of the output device is very different from that of a typical computer display, the user agent should rescale pixel values. It is recommended that the pixel unit refer to the whole number of device pixels that best approximates the reference pixel. It is recommended that the reference pixel be the visual angle of one pixel on a device with a pixel density of 96dpi and a distance from the reader of an arm’s length.

This change only affects web pages that explicitly set the viewport size or scale. The pixel ratio is 1.5 applies only if the viewport scale is set to 1. The size of a “pixel” on any page changes with the zoom level, and the default zoom level for most pages in Fennec has not changed.

On 240-dpi screens, pages with initial-scale=1 will effectively be zoomed to 150% by both Fennec and Android WebKit. Their text will be smooth and crisp, but their bitmap images will probably not take advantage of the full screen resolution. To get sharper images on these screens, mobile web developers can create images at 150% of their final size (or 200%, to support the rumored 320-dpi iPhone) and then scale them down using HTML/CSS.

WebKit on Android supports an additional undocumented target-densityDpi property, to let web developers override the CSS-to-device pixel ratio. Fennec doesn’t support this property now, but if we see a compelling need for it (or if it becomes part of a documented standard) then we might implement it too.

Right now Fennec uses the same default ratio of 1.5 on all devices. (This is a hidden preference that can be changed in about:config or by an add-on.) Later we’ll need to change this – as well as many other parts of Fennec’s user interface – to choose the correct size automatically, depending on the screen density.

Viewport width and screen width

Many sites set their viewport to width=320, initial-scale=1 to fit precisely onto the iPhone display in portrait mode. As mentioned above, this caused problems when Fennec 1.0 rendered these sites, especially in landscape mode. To fix this, Fennec 1.1 will expand the viewport width if necessary to fill the screen at the requested scale. This matches the behavior of Android and Mobile Safari, and is especially useful on large-screen devices like the iPad. (Allen Pike’s Choosing a viewport for iPad sites has a good explanation for web developers.)

We also added support for minimum-scale, maximum-scale, and user-scalable, with defaults and limits similar to Safari’s. These properties affect the initial scale and width as well as limiting zooming after the page is loaded.

Standards

<meta name="viewport"> is a good example of browsers innovating exactly how Sachin Agarwal thinks they should. It was implemented by a single browser, used by web developers, and copied by other browsers without waiting for any standards organization. It has clearly improved on earlier solutions like MobileOptimized and HandheldFriendly.

Now that viewport metadata has proved to be a useful extension to HTML, I think it is worth standardizing. According to the HTML5 spec, new names for the meta element should first be registered on the WHATWG wiki and then be ratified through the W3C standards process. If anyone at Mozilla or elsewhere is working on a standard specification for viewport metadata, please let me know.

Fennec on Android: user feedback and next steps

2010-04-30T00:00:00-07:00

Last month I joined Mozilla as a UI engineer on the Fennec (Mobile Firefox) project. Firefox is already available for Nokia’s Maemo platform, and now a group of Mozilla programmers are porting it to Android. This Tuesday they asked for feedback on an early preview build.

Until now, the only people working on Firefox for Android were back-end (platform) developers. This week was the first time most other people – including me – got to try it out. We front-end developers and designers are now starting to adapt the user interface to Android. For now it uses the look and feel of Firefox for Maemo.

Because we are an open source project, we like to share our work even at this early stage of development. While I wasn’t directly involved in the Android development effort, I spent some of my spare time this week talking to users via Twitter and our Android feedback group. Here’s what I heard, in rough order of importance to users, plus some information on our future plans.¹

Zoom and multi-touch: Pinch zoom gestures are coming! We are reviewing a patch for animated multi-touch (pinch) zooming on Qt-based devices, and testing similar code on Android. (Maemo devices have no multi-touch, so we use their volume buttons to zoom. That code hasn’t been ported to Android, so only double-tap zoom was working in the preview build.)
We also had some requests to fit text to the screen when zoomed in, like the Android browser. Today Brad Lassey and Ben Stover released the Easy Reading add-on that does exactly that. We might make this a built-in option in Fennec once it is fast and reliable enough.
Menu and Back buttons: The preview build did not handle Android’s standard hardware buttons, but code is now checked in to support the back button and the menu and search buttons. We’ll continue to refine the way Fennec uses these buttons.
Size: A ten megabyte download (over 30 MB installed) is not huge for a desktop browser, but it’s hefty for a mobile app – especially on Android, where apps are saved to limited onboard memory.
Shrinking Fennec is possible, but not trivial. Some of the library and toolkit code in our build is probably unused and could be removed. And we could try minifying our JavaScript source, like many websites do. Michael Wu hopes current efforts like Omnijar and Thumb-2 (bug 563751) will cut the installed size approximately in half.
Users also reported that our Fennec build did not work with the feature in some custom Android ROMs to move apps to the SD card. Mozilla’s Android devs are working on a fix for this. It will be nice someday when app storage on Android is as plentiful as it is on other mobile platforms.
Hardware compatibility: There are a lot of different Android phones out there. Some of them won’t run Fennec because they still have Android 1.5 or 1.6. We hope this will be fixed by the hardware vendors soon, since we currently rely on some Android 2.0 APIs. Other devices failed for different reasons, possibly related to insufficient RAM or incompatible OpenGL APIs. We will need to optimize Firefox’s memory footprint on Android, and test on a wider selection of devices, perhaps with help from Firefox users.
Here’s a list of supported hardware.
Keyboard problems: There were many problems with the software keyboard working intermittently or not at all, especially in landscape orientation. There were also problems with Shift and Alt keys on some hardware keyboards. I haven’t heard any news about of these bugs, but we know we need to fix them quickly.
Speed: Strangely, we had some users calling Fennec slooooooow and others calling it fast as hell (and those tweets were sent just one minute apart)!
Once a page is loaded, Fennec is pretty speedy. It’s faster than the Android browser in some areas, and slower in others. But it’s definitely choppy while a page is still loading or complex scripts are running. To fix this, our next major release of Fennec will include Electrolysis. This gives Firefox a multi-process architecture much like Google Chrome, and ensures that the browser always stays responsive.
Electrolysis requires many changes to our code, so it may be a couple of months before it appears in usable Fennec builds. In the meantime, Mozilla is working on many other performance improvements. This work will also speed up Firefox for desktop computers – I’ve been using the FF4 nightly builds, and they are already much snappier than the Firefox 3.5 I was using before.
We’ve also checked in some simple changes to improve perceived speed, like better feedback when pages start loading.
Crashing bugs: Users were generally forgiving of crashes and other obvious bugs, to be expected at this stage of development. We will of course fix any such bugs as fast as possible.
Add-ons: We’re just starting Fennec 1.1 beta testing, and most of our add-ons are not yet updated for version 1.1. Unfortunately, this meant that many add-ons were not available to our first Android previewers. This should be fixed over the next few weeks.
Add-ons are easily Firefox’s biggest advantage over other mobile browsers. For the first time I can easily customize my phone’s browser exactly how I want. I’ve already written two Fennec add-ons, Read Later and Show Image Title. And there are many great add-ons from other developers to choose from.
User interface: Feedback on our UI was generally positive. Most users said panning to reveal the toolbars felt natural and easy. I think Madhava and Sean have done a great job with the design. This will get even better as we take advantage of Android features like the hardware buttons, integration with other activities, voice input, and the notification bar.
Flash: The Flash plugin is not yet included in our Android builds, but it will be supported eventually. Firefox for Maemo already works with Flash, although enabling it does cause performance problems on some sites. (We are working on fixing that with major changes to our graphics code.)
Web site compatibility: Fennec renders almost all pages the same as desktop Firefox. Users did report problems entering data on some pages, and found that most sites do not have mobile versions targeted at Firefox. One of my personal goals is to make Fennec compatible with more mobile sites, and to give web developers the support they need to make sites work great in Fennec. I’ll write much more about this in future articles.

We don’t have a regular schedule yet for releasing new builds on Android. Once we get the code merged and automated build servers configured, we’ll publish nightly builds of Firefox for Android alongside our Maemo and desktop nightlies. Later this year we will have alpha and beta versions, and with luck a stable release. Until then, you can follow @MozMobile or Vlad (@vvuk) to hear about any new previews.

Please remember I am still new to the project, and cannot speak for the whole team. This is a personal blog, not a Firefox roadmap! ↩

Headless Web Workers: Does the web need background apps?

2010-04-22T00:00:00-07:00

At my last job, I created several web applications designed to replace built-in apps on mobile phones. While modern browsers and HTML5 made this incredibly easy in many ways, we still ended up writing native (i.e. non-web) code for most of our applications. There were a few different areas where the browser alone didn’t meet our needs, but one that I found suprisingly common was background processing.

Consider the following mobile applications:

Calendar or clock with alarms.
E-book reader that syncs content from a server.
IM or email client that notifies the user of new messages.
Shopping list that pops up whenever you are near the store.

Ideally, each of these apps will perform some actions even when the user does not have it open. (Background processing is not strictly necessary for the e-reader, but it would be useful to ensure the library is up-to-date even when opened in a place with no network connection.)

You can’t do this with a web app. Web Workers don’t solve the problem, because they run only while the web page is open. What we need are headless web workers.¹

Update (2010-04-26): Gordon Anderson points out in comments that members of Google’s Chromium/Chrome OS projects have made a very similar proposal they call Persistent SharedWorkers.

The API

Headless workers could use almost the same API as Web Workers. Instead of responding to messages from a web page, they would listen to events from the host system (browser or OS). These events might include time intervals, power-on/resume, changes in network connection, geographic locations, or “push” notifications from a remote server.

The event-driven architecture of JavaScript in the browser allows the host system a high level of discretion over resource consumption. There’s no special code needed to suspend processes and later restore their state, because JavaScript workers are naturally inactive between events. The host can provide limits on CPU or memory usage per event, with a separate message to notify processes whose handlers were aborted. And it can limit the number of concurrent processes by choosing when to dispatch events to listeners. Some listeners could even be disabled completely at times (like if the device is busy or the battery is low), and notified later of the events they missed.

This is almost a return to the old days of cooperative multitasking. Mobile computing is definitely driving everyone towards higher-level process control in the OS, and different assumptions for applications. It’s not surprising that my whole proposal resembles Android and iPhone 4.0 multitasking in several ways, since I’ve been doing development on Android for the last 18 months and encountering many of the same issues.

The UI

Headless workers do need some way to interact with the user. They could display standard system notifications (via Growl on the Mac, libnotify on Ubuntu, the status bar in Android, etc.) using W3C Web Notifications, which already have an experimental implementation in Chrome.

Users also needs to know which sites have background tasks installed. Headless workers could be represented by icons in a standard location (perhaps a toolbar in desktop browsers, or the home screen on a mobile device). The icons could display ambient status; clicking one would reveal a menu with options to configure or remove it.

Questions

This proposal might be hard to standardize, especially where it’s tied to specific OS capabilities. For now I’m just curious: would it be useful? You can write a native app or a browser extension to solve this problem today. But would it be worthwhile to have a standard, cross-platform way to do it? Has anyone else run into problems that this approach could solve?

Because all web standards should have names that sound like Harry Potter creatures. ↩

Reading List: a Mobile Firefox extension

2010-04-18T00:00:00-07:00

Update (October 2010): I changed the name of the add-on from “Read Later” to “Reading List” to avoid confusion with the popular service and Firefox extension called Read It Later.

Hello, Planet Mozilla! I’m Matt Brubeck, the newest member of the Mobile Firefox (Fennec) front-end team. I’m working remotely from Seattle, but you can find me in #mobile during the North American day, or follow me on Buzz/Twitter/etc.

Fennec is a new browser built on the Mozilla platform and sharing much of Firefox’s code and features, but with a UI designed from the ground up for touchscreen devices. It’s shipping now for Nokia Maemo, and builds should be available very soon (weeks rather than months, I hope) for Android 2.x.

To help myself learn Fennec and XUL, I wrote a simple extension called Reading List. Like Marco Arment’s Instapaper service it stores a list of web pages so you can return to them later. Unlike Instapaper, my extension does not save pages to a remote server. Instead, it uses your mobile device’s storage, so you can view saved pages offline. I use code from Arc90’s Readability bookmarklet to extract the main content from the page, save it, and present it in a simple mobile-friendly layout.

One thing the extension can’t do (which Instapaper and other services can) is synchronize saved pages between computers. This would be a great feature for a Mobile Firefox add-on, but writing my own sync service is more work than I want to put into this little side project. A future version may use Weave to sync saved pages, if the size of the data is not a problem.

If you are using a recent Fennec 1.1 build, try out Reading List and let me know what you think. And if you’re a developer, you can look at the source code to see how a simple Fennec extension works.

Discovering Urbit: Functional programming from scratch

2010-03-12T00:00:00-08:00

C. Guy Yarvin is a “good friend” of Mencius Moldbug, a pseudonymous blogger known for iconoclastic novella-length essays on politics and history (and occasionally computer science). Guy recently published, under his own name, a project in language and systems design. His own writing about his work is entertaining but verbose (as Moldbug’s readers might expect), so I will attempt to summarize it here.

Nock, Urbit, Watt

First there is Nock, “a tool for defining higher-level languages – comparable to the lambda calculus, but meant as foundational system software rather than foundational metamathematics.” Its primitives include positive integers with equality and increment operators, cons cells with car/cdr/cadr/etc., and a macro for convenient branching. Nock uses trees of integers to represent both code and data. Nock is aggressively tiny; the spec linked above is just 33 terse lines.

Next, Guy provides the rationale for Nock. In short, he asks how a planet-wide computing infrastructure (OS, networking, and languages) would look if designed from first priniciples for robustness and interoperability. The answer he proposes is Urbit: a URI-like namespace distributed globally via content-centric networking, with a feudal structure for top-level names and cryptographic identities. Urbit is a static functional namespace: it is both referentially transparent and monotonic (a name, once bound to a value, cannot be un- or re-bound).

Why does this require a new formal logic and a new programming language? In Urbit, all data and code are distributed via the global namespace. For interoperability, the code must have a standard format. Nock’s minimal spec is meant to be an unambiguous, unchanging, totally standardized basis for computation in Urbit. Above it will be Watt, a self-hosting language that compiles to Nock. Urbit itself will be implemented in Watt, so Nock and Watt are designed to treat data as code using metacircular evaluation.

The code

A prototype implementation of Watt is on GitHub. It is not yet self-hosting; the current compiler is written in C. Watt is a functional language with static types called “molds” and a mechanism for explicit lazy evaluation. (I was suprised to find I had accidentally created an incompatible lazy dialect of Nock – despite its goal of unambiguous semantics – just by implementing it in Haskell.)

The code is not fully documented, but the repository contains draft specs for both Watt and Urbit. Beware: the syntax and terminology are a bit unconventional. Guy has offered a few exercises to help get started with Nock and Watt:

The Nock challenge:: Write a decrement operator in Nock, and an interpreter that can evaluate it.
Basic Watt:: Write an integer square root function in Watt.
Advanced Watt:: How would you write a function that tests whether molds A and B are orthogonal (no noun is in both A and B)? Or compatible (any noun in A is also in B)? Are these functions NP-complete? If so, how might one work around this in practice?

If you want to learn more, start with these problems. You can email your solutions to Guy.

Will it work?

I find Urbit intellectually appealing; it is a simple and clean architecture that could potentially replace a lot of complex system software. But can we get there from here?

Guy imagines Urbit as the product of an ages-old Martian civilization:

Since Earth code is fifty years old, and Martian code is fifty million years old, Martian code has been evolving into a big ball of mud for a million times longer than Earth software. (And two million times longer than Windows.) …

Therefore, at some point in Martian history, some abject fsck of a Martian code-monkey must have said: fsck this entire fscking ball of mud. For lo, its defects cannot be summarized; for they exceed the global supply of bullet points; for numerous as the fishes in the sea, like the fishes in the sea they fsck, making more little fscking fishes. For lo, it is fscked, and a big ball of mud. And there is only one thing to do with it: obliterate the trunk, fire the developers, and hire a whole new fscking army of Martian code-monkeys to rewrite the entire fscking thing.

… This is the crucial inference we can draw about Mars: since the Martians had 50 million years to try, in the end they must have succeeded. The result: Martian code, as we know it today. Not enormous and horrible – tiny and diamond-perfect. Moreover, because it is tiny and diamond-perfect, it is perfectly stable and never changes or decays. It neither is a big ball of mud, nor tends to become one. It has achieved its final, permanent and excellent state.

Do Earthlings have the will to throw out the whole ball of mud and start from scratch? I doubt it. We can build Urbit but no one will come, unless it solves some problem radically better than current software. Moldbug thinks feudalism will produce better online reputation, but feudal reputation does not require feudal identity; it is not that much harder to build Moldbug’s reputation system on Earth than on Mars. I still have not figured out the killer app that will get early adopters to switch to Urbit.

The network is the human being

2010-03-01T00:00:00-08:00

Nathanael Boehm wrote a nice essay last month called The Future of Employment?, about a disconnect between workers' and employers' views of social networks. (This post is based partly on the ensuing Hacker News thread.) Boehm wrote:

When I need help with a challenge at work or need to run some ideas past people I don’t turn to my co-workers, I look to my network of colleagues beyond the walls of my workplace. Whilst my co-workers might be competent at their job they can’t hope to compete with the hundreds of people I have access to through my social networks.

The late Sun Microsystems taught us that the network is the computer. It’s true: we still use non-networked computers for specialized tasks, but nobody wants one on their desk – it’s just so useless compared to one that talks to the entire world. Boehm could have titled his essay The Network is the Employee. There are still tasks that people do in isolation, but the ability to contact a network of peers and experts makes the difference in my job, and many others.

Alone together

The lone computer programmer in a small business has thousands of colleagues on Stack Overflow, Reddit, and so on. It’s a messy way to find answers, but it’s sure better than the days when your only choice was to call tech support – or smack the box with your fist, whichever seemed more useful. I can’t begin to list all the problems I’ve solved and things I’ve learned by Googling for others with experience, and getting help from a different expert for every problem.

Decades before the web, computer geeks had virtual communities on mailing lists, Usenet, and IRC. Now every job in the world has its corresponding forum. Even the night clerk at the gas station has Not Always Right.

Teaching has long been a solitary profession. Despite working in a crowded classroom, teachers are isolated; they rarely have colleagues observing or participating directly in their work. This has such an impact that teachers are sometimes trained in meditation or reflection techniques, to make up for the lack of external feedback. So I’m curious what happens when teachers start to work together remotely the way programmers do.

You will be assimilated

Boehm’s essay also reminded me of a vague sci-fi idea I’ve been kicking around: the first group minds will evolve from the intersection of Mechanical Turk, virtual assistants, social networking, and augmented reality.

Starting around the 1990s, it was possible to instantly “know” any fact that was published online. Since then, we’ve increased the amount of content online, our tools for searching it, and ways of connecting to the network. Today we have instant access to almost any published knowledge, anywhere.

There are more people on the net too, and more ways to find and talk to them. Most of us can contact dozens of friends at any given moment, plus friends-of-friends, co-workers, fellow members of communites like Hacker News or MetaFilter, and also complete strangers. Along with raw facts, we have access to vast amounts of human judgement, experience, and skill.

One product of this is the “virtual assistant,” who provides a service that was once exclusive to high-powered executives. Now personal assistants can work remotely (often overseas), spread costs by serving many masters, and leverage the internet superpowers listed above. Their services are mostly targeted at small business owners and the Tim Ferriss crowd, but I’m sure someone soon will market virtual assistance to all sorts of other creative workers, teachers, even stay-at-home parents.

So, how long before I can touch a button to let a remote assistant see what I’m seeing in real-time and help me make transportation plans, translate foreign signs and speech, look up emails related to whatever I’m doing or thinking, or even advise me on what to say? Some of these queries will go to my circle of friends, others to the general public, and some to a personal assistant who is paid well to keep up with my specific needs. And that assistant of course will subcontract portions of each job to computer programs, legions of cheap anonymous Turkers, or his or her own network of helpers. At that point, I’m augmenting my own perception, memory, and judgement with a whole network of brains that I carry around, ready to engage with any situation I meet.

If nothing else, I hope someone writes a good sci-fi thriller story in which a rogue virtual assistant manipulates the actions of unsuspecting clients, leading them to some unseen end.

Finding domain names with Node.js

2010-01-13T00:00:00-08:00

I’m working on some ideas for finance or news software that deliberately updates infrequently, so it doesn’t reward me for reloading it constantly. I came up with the name “microhertz” to describe the idea. (1 microhertz ≈ once every eleven and a half days.)

As usual when I think of a project name, I did some DNS searches. Unfortunately “microhertz.com” is not available (but “microhertz.org” is). Then I went off on a tangent and got curious about which other SI units are available as domain names.

This was the perfect opportunity to try node.js so I could use its asynchronous DNS library to run dozens of lookups in parallel. I grabbed a list of units and prefixes from NIST and wrote the following script:

var dns = require("dns"), sys = require('sys');

var prefixes = ["yotta", "zetta", "exa", "peta", "tera", "giga", "mega",
  "kilo", "hecto", "deka", "deci", "centi", "milli", "micro", "nano",
  "pico", "femto", "atto", "zepto", "yocto"];

var units = ["meter", "gram", "second", "ampere", "kelvin", "mole",
  "candela", "radian", "steradian", "hertz", "newton", "pascal", "joule",
  "watt", "colomb", "volt", "farad", "ohm", "siemens", "weber", "henry",
  "lumen", "lux", "becquerel", "gray", "sievert", "katal"];

for (var i=0; i<prefixes.length; i++) {
  for (var j=0; j<units.length; j++) {
    checkAvailable(prefixes[i] + units[j] + ".com", sys.puts);
  }
}

function checkAvailable(name, callback) {
  dns.resolve4(name).addErrback(function(e) {
    if (e.errno == dns.NXDOMAIN) callback(name);
  })
}

Out of 540 possible .com names, I found 376 that are available (and 10 more that produced temporary DNS errors, which I haven’t investigated). Here are a few interesting ones, with some commentary:

exasecond.com – 32 billion years
petasecond.com – 32 million years
petawatt.com – can be produced for femtoseconds by powerful lasers
terapascal.com
gigakelvin.com – possible temperature of picosecond flashes in sonoluminescence
giganewton.com – 225 million pounds force
gigafarad.com
kilosecond.com – 16 minutes 40 seconds
kilokelvin.com – 1340 degrees Fahrenheit
centiohm.com
millifarad.com
microkelvin.com
picohertz.com – once every 31,689 years
picojoule.com
femtogram.com – mass of a single virus
yoctogram.com – a hydrogen atom weighs 1.66 yoctograms
zeptomole.com – 602 molecules

To get the complete list, just copy the script above to a file, and run it like this: node listnames.js

Along the way I discovered that the API documentation for Node’s dns module was out-of-date. This is fixed in my GitHub fork, and I’ve sent a pull request to the author Ryan Dahl.

Weekend hack: Outline grep

2010-01-12T00:00:00-08:00

I keep almost all of my notes and to-do lists in plain text files, so I can edit and search them with Vim, grep, and other standard Unix tools. I often indent lines in these files to create a simple outline structure, and use the autoindent and foldmethod=indent options to make Vim into a simple outliner.

To get useful output when searching through these outline-structured files, I wrote a simple grep replacement. Given a text file with a Python-style indentation structure, ogrep searches the file for a regular expression. It prints matching lines, with their “parent” lines as context. For example, if input.txt looks like this:

2009-01-01
  New Year's Day!
    No work today.
    Visit with family.
2009-01-02
  Grocery store and library.
2009-01-03
  Stay home.
2009-01-04
  Back to work.
    Remember to set an alarm.

then ogrep work input.txt will produce the following output:

2009-01-01
  New Year's Day!
    No work today.
2009-01-04
  Back to work.

You can download ogrep from the outline-grep repository on GitHub, or just read the literate Haskell file. The code is almost trivial (40 lines of code, plus imports and comments); I’m publishing it just in case anyone else has a use for it, and because some of my friends were curious about how I’m using Haskell. I’ve now written a few “real-world” Haskell programs (compleat was the first). I’m finding Haskell very well suited to such programs, though this particular one would be equally easy in a language like Perl, Python, or Ruby.

This is a one-off tool to fill a gap in my workflow; there are no configuration options or useful error messages. It would be fairly easy to extend it, though. For example, an option to include children (as well as parents) of matching lines might be handy. I recently realized that ogrep often works for searching through source code too, which may generate some more unexpected use cases.

Android 2.0 uses V8 JavaScript engine

2009-11-06T00:00:00-08:00

Google has not yet released most of the Android 2.0 “eclair” source code, but they did publish source for a very small number of components, including a WebKit snapshot. I was excited to see that the snapshot includes Google’s V8 virtual machine. (Previous Android releases used Safari’s JavaScriptCore/“SquirrelFish Extreme” VM.) But without the rest of the source tree, there was no way to build and run this on a real Android phone. The SDK includes a binary image that runs only in the qemu-based emulator.

Today I got to try out a Motorola Droid. Here’s how its browser compares to Android 1.6 on my HTC Dream (Android Dev Phone / T-Mobile G1) in the V8 Benchmark Suite:

Test	Dream (1.6)	Droid (2.0)	Change
Richards	13.5	15.6	+16%
DeltaBlue	5.23	12.9	+147%
Crypto	13.2	10.9	-17%
RayTrace	10.9	80.1	+635%
EarleyBoyer	23.5	74.7	+218%
RegExp	did not complete	16.5	–
Splay	did not complete	did not complete	–

Some tests (Richards, Crypto) see little or no improvement, while others (DeltaBlue, RayTrace, EarleyBoyer) are dramatically faster. Just for comparison, let’s run the same benchmark on Safari 4 (JavaScriptCore) and a Chromium 4 nightly build (V8) on a Mac Pro:

Test	Safari 4	Chromium 4	Change
Richards	4103	4640	+13%
DeltaBlue	3171	4418	+39%
Crypto	3331	3643	+9%
RayTrace	3509	6662	+90%
EarleyBoyer	4737	7643	+61%
RegExp	1268	1187	-6%
Splay	1198	7290	+509%

The precise ratios are different, but the same tests that showed the most improvement from Android 1.6 to 2.0 also show the most improvement from Safari to Chrome. Based on this plus the source code snapshot, I’m pretty sure that Android 2.0 is indeed using V8.

This is exciting news. It makes Droid the first shipping product I know that uses V8 on an ARM processor, although V8 has included an ARM JIT compiler for some time now. [Correction: Palm Pre was first; see the comments below.] For mobile web developers like me, it means we’re one step closer to having desktop-quality rich web applications on low-power handheld devices.

Android still lags behind the iPhone in at least one important way for web developers: CSS animation. The iPhone (and Safari on the desktop) provides hardware acceleration for CSS transforms, like this falling leaves demo. On Android, CSS animation is done in software, making it much, much slower. (Even outside the browser, Android’s Skia 2D graphics API lacks hardware acceleration. OpenGL is the only way to for Android developers to take advantage of the GPU.) Accelerated animation would really make it possible to write interactive web pages that match the smoothness and responsiveness of native apps.

Final thought: Although the Motorola Droid is still 100 times slower than Chromium on a Mac Pro, it’s already faster at some benchmarks than IE8 or Firefox 2 on desktop hardware from just a few years ago.

Update (2010-02-09): Just for comparison, here are numbers for the Google/HTC Nexus One with Android 2.1. The Nexus One is around 2-4 times faster than the Droid at the V8 benchmark suite. It even renders the falling leaves animation at a decent framerate (but still not as smoothly as the GPU-accelerated iPhone).

Test	Droid (2.0)	Nexus One (2.1)	Change
Richards	15.6	52.1	+234%
DeltaBlue	12.9	60.2	+367%
Crypto	10.9	31.7	+191%
RayTrace	80.1	170	+112%
EarleyBoyer	74.7	126	+69%
RegExp	16.5	27.5	+67%
Splay	did not complete	did not complete	–

Compleat: Bash completion for human beings

2009-10-30T00:00:00-07:00

Compleat is an easy, declarative way to add smart tab completion for any command. It’s written in Haskell but requires no programming knowledge. See the GitHub repository for a quick description, or read on for a complete explanation.

Background

I’m one of those programmers who loves to carefully tailor my development environment. I do nearly all of my work at the shell or in a text editor, and I’ve spent a dozen years learning and customizing them to work more quickly and easily.

Most experienced shell users know about programmable completion, which provides smart tab-completion for supported programs like ssh and git. You can also add your own completions for programs that aren’t supported—but in my experience, most users never bother.

At Amazon, everyone used Zsh (which has a very powerful but especially baroque completion system) and shared the completion scripts they wrote for our myriad internal tools. Now that I’m in a startup with few other command line die-hards, I’m on my own when it comes to extending my shell.

So I read the fine manual and started writing completions. You can see the script I made for three commands from the Google Android SDK. It’s 200 lines of shell code, and fairly straightforward if you happen to be familiar with the Bash completion API. But as I cranked out more and more case statements, I felt there must be a better way…

The Idea

It’s not hard to describe the usage of a typical command-line program. There’s even a semi-standard format for it, used in man pages and generated by libraries like AutoOpt. For example, here’s the usage for android, one of the SDK commands supported by my script:

 android [--silent | --verbose]
   ( list [avd|target]
   | create avd ( --target <target> | --name <name> | --skin <name>
                | --path <file> | --sdcard <file> | --force ) ...
   | move avd (--name <avd> | --rename <new> | --path <file>) ...
   | (delete|update) avd --name <avd>
   | create project ( (--package|--name|--activity|--path) <val>
                    | --target <target> ) ...
   | update project ((--name|--path) <val> | --target <target>) ...
   | update adb )

My idea: What if you could teach the shell to complete a program’s arguments just by writing a usage description like this one?

The Solution

With Compleat, you can add completion for any command just by writing a usage description and saving it in a configuration folder. The ten-line description of the android command above generates the same results as my 76-line bash function, and it’s so much easier to write and understand!

The syntax should be familiar to long-time Unix users. Optional arguments are enclosed in square brackets; alternate choices are separated by vertical pipes. An ellipsis following an item means it may be repeated, and parentheses group several items into one. Words in angle brackets are parameters for the user to fill in.

Let’s look at some more features of the usage format. For programs with complicated arguments, it can be useful to break them down further. You can place alternate usages on their own lines separated by semicolons, like this:

android <opts> list [avd|target];
android <opts> move avd (--name <avd>|--rename <new>|--path <file>)...;
android <opts> (delete|update) avd --name <avd>;

…and so on. Rather than repeat the common options on every line, I used a parameter named “opts”. I can define that parameter to be a sub-pattern, which will be used wherever <opts> appears:

opts = [ --silent | --verbose ];

For parameters whose values are not fixed but can be computed by another program, we use a ! symbol followed by a shell command to generate completions, like this:

avd = ! android list avd | grep 'Name:' | cut -f2 -d: ;
target = ! android list target | grep '^id:'| cut -f2 -d' ' ;

Any parameter without a definition will use the shell’s built-in completion rules, which suggest matching filenames by default.

The source code is on GitHub. I’ve been using it for just a week and I’m now writing new usage files for myself almost every day. The README file has more details about the usage syntax, and instructions for installing the software. Give it a try, and please send in any usage files that you want to share! (Questions, bug reports, or patches are also welcome.)

Future Work

For the next release of Compleat, I would like to make installation easier by providing better packaging and pre-compiled binaries; support zsh and other non-bash shells; and write better documentation.

In the long term, I’m thinking about replacing the usage file interpreter with a compiler. The compiler would translate the usage file into shell code, or perhaps another language like C or Haskell. This would potentially improve performance (although speed isn’t an issue right now on my development box), and make it easy for usage files to include logic written in the target language. Another idea for the future: What if option-parsing libraries like AutoOpt or the Ruby/Perl/Python equivalents generated completion scripts for every program you wrote?

Final Thoughts

I realized recently that some things I do are so specialized that my parents and non-programmer friends will probably never get them. For example, Compleat is a program to generate programs to help you… run programs? Sigh. Well, maybe someone out there will appreciate it.

Compleat was my weekends/evenings/bus-rides project for the last few weeks (as you can see in the GitHub punch card), and my most fun side project in quite a while. It’s the first “real” program I’ve written in Haskell, though I’ve been experimenting with the language for a while. Now that I’m comfortable with it, I find that Haskell’s particular combination of features works just right to enable quick exploratory programming, while giving a high level of confidence in the behavior of the resulting program. Compleat 1.0 is just 160 lines of Haskell, excluding comments and imports. Every module was completely rewritten at least once as I compared different approaches. (This is much less daunting when the code in question is only a couple dozen lines.) I don’t think this particular program would have been quite as easy to write—at least for me—in any of the other platforms I know (including Ruby, Python, Scheme, and C).

I had the idea for Compleat more than a year ago, but at the time I did not know how to implement it easily. I quickly realized that what I wanted to write was a specialized parser generator, and a domain-specific language to go with it. Unfortunately I never took a compiler-design class in school, and had forgotten most of what I learned in my programming languages course. So I began studying parsing algorithms and language implementation, with Compleat as my ultimate goal.

My good friend Josh and his Gazelle parser generator helped inspire me and point me toward other existing work. Compleat actually contains three parsers. The usage file parser and the input line tokenizer are built on the excellent Parsec library. The usage file is then translated into a parser that’s built with my own simple set of parser combinators, which were inspired both by Parsec and by the original Monadic Parser Combinators paper by Graham Hutton and Erik Meijer. The simple evaluator for the usage DSL applies what I learned from Jonathan Tang’s Write Yourself a Scheme in 48 Hours. And of course Real World Haskell was an essential resource for both the nuts and bolts and the design philosophy of Haskell.

So besides producing a tool that will be useful to me and hopefully others, I also filled in a gap in my CS education, learned some great new languages and tools, and kindled an interest in several new (to me) research areas. It has also renewed my belief in the importance of “academic” knowledge to real engineering problems. I’ve already come across at least one problem in my day job that I was able to solve faster by implementing a simple parser than I would have a year ago by fumbling with regexes. And I’ll be even happier if this inspires some friends or strangers to take a closer look at Haskell, Parsec, or any problem they’ve thought about and didn’t know enough to solve. Yet.

Matt Brubeck

Android app permissions and Firefox Beta

Thoughts on Android Permissions

Mobile web developers: Your users hate it when you do this

Browser detection pitfalls

Solutions

Suggestions for web developers

Further reading

What's different about Firefox for Android

Why

Speed and responsiveness

Installation size: Problems and solutions

Hardware compatibility

Competition and choice

Try it out

Changes for add-ons in Fennec 2.0 alpha 1

What’s new for add-ons?

Get started

Fennec 2.0 update: The road to alpha

Under the hood

Features

Android

Add-ons

Nightly builds

Implementing the viewport meta tag in Mozilla Fennec

Background

A pixel is not a pixel

Viewport width and screen width

Standards

Fennec on Android: user feedback and next steps

Headless Web Workers: Does the web need background apps?

The API

The UI

Questions

Reading List: a Mobile Firefox extension

Discovering Urbit: Functional programming from scratch

Nock, Urbit, Watt

The code

Will it work?

The network is the human being

Alone together

You will be assimilated

Finding domain names with Node.js

Weekend hack: Outline grep

Android 2.0 uses V8 JavaScript engine

Compleat: Bash completion for human beings

Background

The Idea

The Solution

Future Work

Final Thoughts