Phil Toland

I have been back in sunny San Antonio for over a week now, and I have had a chance to reflect on this year’s Erlang Factory in San Francisco. The conference was great in almost every respect, and the folks at Erlang Solutions deserve a lot of credit for pulling off a great conference. In particular, Francesco Cesarini went out of his way to make sure all of the delegates felt welcome and that the conference ran smoothly.

The hotel where the conference was held was the Marines Memorial Club and Hotel in downtown San Francisco. As the name suggests, the hotel is a memorial to U.S. Marines who have died fighting in foreign wars. It is part hotel and part museum. There is a lot of character and a sense of history that you don’t normally find at a hotel and conference venue.

Many people that I have talked to suggest that the best part of technology conferences is the “hallway track” where you meet new people and discuss different aspects of the technology. This was certainly true at Erlang Factory, especially since there were social events scheduled for Thursday, Friday and Saturday evenings. I often fell into conversations with other delegates over breakfast or lunch on topics ranging from the talks we just saw to interesting uses of Erlang. It certainly helps to be able to put faces with names that you may have only seen on emails or on Twitter.

My talk covering our Erlang work at Rackspace was well reveived by the attendees. I had some interesting conversations afterwards with delegates who had faced similar problems. On Friday evening I sat on a panel discussing running Erlang on a Platform as a Service (PaaS). There was some interesting back and forth between the members of the panel and some in the audience. Deploying an Erlang application to a PaaS is only part of the problem; current PaaS implementations do not allow doing the sorts of things that make Erlang stand out like distributed applications and hot code upgrades.

I think that I got more out of Erlang Factory than any conference I have attended in recent years. The Erlang community is still small enough that conferences like Erlang Factory can have a huge impact by getting people together under one roof. Kenji Rikitake makes a strong argument for the value of Erlang Factory to the community and I couldn’t agree more. I am looking forward to next year.

[NOTE: This post originally appeared on the Rackspace blog.]

My father used to be an auto mechanic. Every so often the Cornwell or Snap-On tool truck would come by his garage and he would wander through it like a kid in a candy store. To this day he has large rolling toolboxes full of the mysterious implements of car repair. Each tool was useful in some situation where it made his job easier in some cases, and doable in others.

I am a software developer and I collect programming languages as my father collected tools. I once went to the local library and checked out old manuals for COBOL and FORTRAN; not because I wanted to write any code with them (perish the thought!) but because I wanted to know what makes them tick. I originally approached the Erlang programming language with the same sense of curiosity and detachment. There wasn’t a lot of information about Erlang out in the wild at the time and it wasn’t clear to me what Erlang might bring to the table.

Then in 2007 an interesting thing happened. I was attending RailsConf in Portland, Ore. and the Pragmatic Programmers had announced that they were publishing Programming Erlang by Joe Armstrong, the father of Erlang. There was quite a lot of chatter in the halls and before talks about this weird “new” programming language. The attendees of RailsConf 2007 represented early adopters and enthusiasts who would be comfortable exploring little known languages. It was becoming clear that Erlang might complement Ruby development nicely being strong in the areas where Ruby lacked and vice versa. I bought the book when it was released and was very intrigued.

Erlang was developed in the 1980s as a programming language to be used in telephony equipment an environment where reliability is a far greater concern than raw computational speed. In order to achieve high reliability, Erlang uses lightweight processes that are completely isolated from each other. These independent processes communicate by passing messages back and forth. If a process fails, it does not disrupt the entire system. There is a sophisticated supervision framework that can detect when a process crashes and react appropriately by, for example, restarting the process that crashed. A nice side effect of this architecture is that Erlang scales very well across multiple cores in a single server or across multiple servers in a cluster. The lightweight processes in Erlang do not share anything, so the process you are passing messages to can be located on the same machine running on a different core or on another machine in the cluster entirely.

Perhaps the most famous Erlang project is the AXD301 ATM switch developed at Ericsson in the late 1990’s. The code for the switch consisted of over a million lines of Erlang code and reportedly reached nine “9”s of uptime. While using Erlang doesn’t automatically guarantee that level of reliability, it is a glimpse of what is possible. Concurrency and reliability are baked into Erlang in a way that is different than most other languages I have worked with. This leads developers to think differently about application architecture, especially when it comes to error handling and failures within a large, distributed system.

In 2008 I was given the opportunity to put Erlang to the test in an application I was building for my employer at the time. On that project I worked on a small team consisting of myself and two other developers and we were able to deliver a solid system in a relatively short amount of time. Erlang allowed us to focus on the business problem and not the incidentals. I have chosen to work with Erlang ever since then because it is both powerful and empowering.

Many of today’s most interesting problems in software development are also the most challenging. Concurrency in a multicore world and reliability when applications are growing ever larger and ever more complex must rank up there, along with naming and cache invalidation as some of the most challenging problems in modern software development. Erlang is a pragmatic programming language aimed squarely at those problems. These traits make Erlang an excellent choice for building backend systems and APIs. For Example, the Riak and CouchDB databases are both implemented in Erlang as well as the RabbitMQ message queue. The Webmachine library makes writing ReSTful APIs a snap.

It is cliche these days to suggest that programmers should use the right tool for the job. However, another saying popular among programmers is that when all you have is a hammer every problem looks like a nail. Therefore, if developers are to use the best tool for the job they need to have a full toolbox. Erlang is a tool that deserves a place in every developer’s toolbox. While Erlang is not a tool that is appropriate for every situation, in the situations where it is a good fit, Erlang can make a developer’s job much easier, and in some cases doable.

I will be presenting at the Lone Star Ruby Conference this year with my coworker Mike (a.k.a. @lifeinzembla on Twitter). The talk is titled “Polyglot Parallelism: A Case Study in Using Erlang and Ruby at Rackspace” and will cover our experience rebuilding some older systems in Erlang. We plan to get into when you should use Erlang to augment your existing code as well as what benefits you can expect from Erlang and what pitfalls to avoid. The full abstract is below:

Two years ago Rackspace had a problem: how do we backup 20K network devices, in 8 datacenters, across 3 continents, with less than a 1% failure rate — every single day? Many solutions were tried and found wanting: a pure Perl solution, a vendor solution and then one in Ruby, none worked well enough. They were not fast enough or they were not reliable enough, or they were not transparent enough when things went wrong.

Now we all love Ruby but good Rubyists know that it is not always the best tool for the job. After re-examining the problem we decided to rewrite the application in a mixture of Erlang and Ruby. By exploiting the strengths of both — Erlang’s astonishing support for parallelism and Ruby’s strengths in web development — the problem was solved.

In this talk we’ll get down and dirty with the details: the problems we faced and how we solved them. We’ll cover the application architecture, how Ruby and Erlang work together, and the Erlang approach to asynchronous operations (hint: it does not involve callbacks). So come on by and find out how you can get these two great languages to work together.

Phil Toland does not have an impressive beard, but he has been working with Erlang since 2008 when he cut his Erlang teeth developing Yet Another Dynamo Clone. These days he works at Rackspace automating network devices against their will and often his own better judgement.

Mike Sassak is Internet famous for his observational skills and his impressive beard. Also, he works at Rackspace and contributes to Cucumber.

I needed to encrypt some text before storing it in the database. Since I needed to get the original back at some point I settled on Blowfish and AES as good candidates. There are a couple gems with provide this type of encryption, but I could not get either one to work.

Eventually I came across this snippet and this blog post. It turns out that it is trivial to use the Ruby OpenSSL library to do encryption. I cleaned up the code from the first snippet a bit:

require 'openssl'

module Blowfish
   def self.cipher(mode, key, data)
     cipher = OpenSSL::Cipher::Cipher.new('bf-cbc').send(mode)
     cipher.key = Digest::SHA256.digest(key)
     cipher.update(data) << cipher.final
   end

   def self.encrypt(key, data)
     cipher(:encrypt, key, data)
   end

   def self.decrypt(key, text)
     cipher(:decrypt, key, text)
   end
 end

 if $0 == __FILE__
   p "text" == Blowfish.decrypt("key", Blowfish.encrypt("key", "text"))
 end

The Obama administration is announcing steep increases in the federal CAFE standards. Under the new standards average fuel efficiency will increase from 27.5 to 35.5 MPG. The administration is positioning it as a win for everybody: the government continues to assert its control over the auto industry, environmentalists get thrown a bone and the auto manufacturers get big “green” subsidies. It is good for everyone but consumers, anyway. The auto manufacturers will of course pass the cost of these changes along to consumers to the tune of $1000 per car. John Stossel makes an insightful observation:

In a press release, the administration admits that the changes will increase the cost of the typical car by $1,000. But they say it will save each driver $3,000 over the life of the car. Are they right? I don’t know, but if these savings really exist, why does the government need to FORCE people to take them?

If automakers wanted to make cars like this, they would do it on their own.

The problem is that Americans generally don’t want fuel efficient vehicles. They want SUVs, minivans, luxury and sports cars. The CAFE standards contributed to the downfall of the Big 3 automakers last year as they churned out crappy fuel efficient econoboxes that nobody wanted so that they could continue selling the SUVs people did want. I don’t see that changing significantly unless gas gets way more expensive.

ISO OOXML convener: Microsoft’s format “heading for failure”:

Although Microsoft’s Office Open XML (OOXML) document format became an ISO standard two years ago, the company still hasn’t built any software that truly complies with the standard. Microsoft Office 2010, which is expected to be released later this year, implements the deprecated “transitional” version of the format and is not compatible with the version that was crafted by the standards body.

Everyone who is surprised by this please raise your hand. Yeah, that is what I thought. I was skeptical of the ISO OOXML standardization process from the beginning. Microsoft has never been big on respecting standards, and it appears that is true even when they create the standard themselves.

I have release version 0.4.6 of Patron, my HTTP client library for Ruby. The big news for this release is a dramatic speedup for uploads. There are still a few bugs outstanding and I hope to have a new release ready soon. You can get Patron via rubygems with gem install patron or get the source from Github.

When I decided to move my blog from Wordpress to Tumblr I did a lot of searching to find a tool that would import my content. I found a couple of solutions, but none that worked for me. After taking a look at Tumblr’s API docs I decided to throw together a simple script that would parse a Wordpress export file and post the content to Tumblr. It turned out to be pretty easy. Since this is one of those scratch an itch type of solutions it is biased a bit towards the type of content I needed to import (mostly plain posts formatted with Markdown). I am posting it in the hopes that it will be useful, but as always, caveat programmor.

This is going to be one of those embarrassingly meta posts where I talk about the technology behind this site. Rather than drag this out needlessly I will get right to the point and say that I have fully moved my blog over from a self-hosted Wordpress install to Tumblr. Actually, it is a little more complicated than that. I had three sites where I theoretically posted content: my main blog at fiatdev.com, my personal site at philtoland.com and my Tumblr blog at tumbl.fiatdev.com. While I have not posted regularly for a couple of years, it became clear to me that I simply had too many buckets for online content. It was time to simplify and so I moved the content from all three sites into my Tumblr blog.

Quite frankly, I am tired of having to maintain a blog engine. That is something that was kinda fun when I started but has become a chore for me. I began to obsess over the Wordpress engine itself while not writing any actual content. I like Tumblr because it is hosted and simple. Oh, and the bookmarklet is “teh awesome” as the kids say these days. Tumblr’s bookmarklet and simplicity remove the friction from posting content (hence the title of this post).

Now, with all of that having been said, we will see if any of this results in more content being posted here or not.

I recently reset my iPhone in an attempt to clear up some weird issues I was having. It wasn’t until afterwards that I realized that I had forgotten to pull my photos off of the phone first. After doing some searching I discovered that the iPhone backup files are located at:

/Users/<user>/Library/Application Support/MobileSync/Backup

I then found the bkupextract.pl script that attempts to extract all of the sqlite3 databases from the backup files. I modified the script to also extract the plist, JPEG and PNG files and posted it to a gist. My Perl-Fu is weak, so no complaining about the quality of the code, but it found my photos.

The 0.4 version of Patron has been released. This version addresses feedback from the initial public release and fixes a few minor bugs:

• Fixed a bug in the Patron.version method
• Added HTTP proxy support
• Enabled SSL and Win32 support
• Added ability to download data to a file with GET
• Fixed a bug where header values were prefixed with a space
• Made the Session#request method public
• Added support for the WebDAV COPY method
• Added support for uploading files with POST and PUT

In addition to all of this, Patron has a new home page at http://toland.github.com/patron/.

So, what’s next for Patron? Well, I would like to put out a 0.5 release in the next few weeks with some refactoring and API tuning. Also, I would like to implement the rest of the WebDAV methods. I am toying with the idea of creating a net/http compatible interface so that Patron can be used as a drop-in replacement. Not sure if that is going to go anywhere, though.

Once I am happy with the API and documentation I will release 1.0. Other than that, I try to respond to bug reports and feature requests in a timely manner. I originally wrote Patron to scratch my own itch, and that itch has been scratched. At this point I am looking to make Patron as broadly useful as I can, and your feedback helps with that.

Patches and contributed documentation are always welcome. In particular, if anyone feels like trying to get this working on Windows, I will be happy to integrate the resulting patches and publish a binary gem.

One thing that I did not like about working with the Nitrogen web framework is the way that view rendering is handled. Nitrogen calls the function main/0 in your web modules which, by convention, returns an instance of the template record. This record points to a template file that has HTML code with a little Erlang mixed in. Essentially, you can call functions in your web module and whatever content those functions return is interpolated into your template.

This all makes it sound a little more complicated than it really is. Here is a little “hello world” sample to illustrate how it works. First, the web_page Erlang module defines the main/0 function and a couple of callouts:

-module(web_page).
-include("wf.inc").
-compile(export_all).

main() -> #template { file="./wwwroot/template.html" }.
title() -> "Hello, World!".
content() -> #h1 { text="Hello, World!" }.

This will cause Nitrogen to render the template.html file.

<html>
<head>
<title>[[[page:title()]]]</title>
</head>
<body>
[[[page:content()]]]
</body>
</html>

The callouts in the template will call the respective functions in the web_page module and we end up with a page that displays a bold “Hello, World!”. The template file is really analogous to a Rails layout and the content that Rails puts into a view template is returned from the callouts using a markup DSL based on the Erlang record syntax.

This template mechanism is simple, but it becomes very limiting when working on a non-trivial application. Nitrogen mixes controller functionality with presentation in a way that is reminiscent of PHP or JSP. Also, you cannot compose templates and the markup DSL doesn’t cover all of the HTML elements and attributes.

I ran into this problem while working on a Nitrogen powered site and decided that I just couldn’t live with the default template system. After a little digging around I discovered ErlyDTL which implements a substantial subset of the Django Template Language in Erlang. It turns out that using ErlyDTL in a Nitrogen site is quite simple.

The first thing I needed was a utility function to render ErlyDTL templates. Nitrogen includes implicit elements for the notification flash and Javascript code for all of that AJAX goodnes which I needed to replicate in my render function.

-module(my_util).
-compile(export_all).

template_path(Template) ->
    filename:join([code:priv_dir(my_app), templates, Template]).

render_template(Name, Variables) ->
    Template = template_path(Name ++ ".html"),
    Mod = list_to_atom(Name ++ "_template"),
    ok = erlydtl:compile(Template, Mod),
    {ok, Body} = Mod:render([{flash, element_flash:render()},
                             {script, wf_script:get_script()}
                             | Variables]),
    Body.

Now, I can replace the simple module from earlier with this new one:

-module(web_page).
-include("wf.inc").
-compile(export_all).

main() ->
    my_util:render_template("template", [
          {title, "Hello, World!"},
          {content, content()}]).

content() -> "Hello, World!".       

My template file is now equally simple:

<html>
<head>
<title>{{ title }}</title>
</head>
<body>
<h1>{{ content }}</h1>    
</body>
</html>

All of the markup is now in the template file, and the module is concerned only with getting the content from somewhere and making it available to the view. This also allows us to use the full power of ErlyDTL: we can compose templates and perform simple operations on the content to be displayed. Here is an example from the ErlyDTL home page:

Welcome back, {{ name }}!

You have {{ friends|length }} friends: {{ friends|join:", " }}

Have some primes:
{# this is exciting #}
{% for i in primes %}
    {{ i }}
{% endfor %}

There is one drawback to this approach. The Nitrogen DSL includes helpers for generating links that work with its event system. When using ErlyDTL we have to create those links manually, but it isn’t hard. The key thing to remember is that the link id is the event name that will be passed to your module. Here are a couple of examples:

<a id="do_it" href="javascript:">Do It!</a>
<button id="do_it" class="button submit">Do It!</button>

The extra flexibility of DTL is certainly worth the small amount of extra work.