Erlang Inside

The Erlang Factory’s 2010 conference is March 25th and 26th in San Francisco, with the university three days before, starting the 22nd.  The conference is at the Hilton San Francisco Airport, with three tracks, each on a different theme. The number of tracks gives attendees an unusually broad set of choices for talks. When attending conferences I use the times when one or both talks are uninteresting to check email and generally hang out with people. Occasionally there are two talks I want to see at once so I’ll pick one and then duck out to catch the end of the other. Joe Armstrong and Nick Gerakine’s simultaneous talks on Day 2, for example. However, in this case there will at times be three interesting subjects running concurrently, which is only appropriate for a conference on Erlang.

Francesco C: Erlang Factory is becoming a very popular event. We are expecting an increase in participants, somewhere between 150 – 200 delegates. We have seen a growing trend, irrespective of the state of the economy.

FC: It varies. Last year, we had students who never used Erlang in a commercial project but were interested in learning more. They were interacting with people who have been working with Erlang full time for 15 years. Even with this huge gap, you did not notice any divide. That’s the beauty of our conferences; Erlang enthusiasts from architects to newbies have an opportunity to exchange knowledge, ideas, experience and have a great time together. Of those attending, about 50% came from the west coast, 17% from the east coast, 15% from the mid-west and 18% from the rest of the world, mainly Europe, but also from Canada, Central and South America.

FC: About 20% attend the Erlang University. The value of attending the courses in this way is that they are condensed versions of the 5-day courses and allow delegates to attend the training as well as the conference. Furthermore, having the training together with the conference provides the opportunity to discussions and networking, multiplying the value many times. New for this year is a course by Basho Technologies‘ very own Kevin Smith on Web development in Erlang. This course will run alongside the OTP Express course, taught by Erlang Solutions’ training manager Henry Nystrom, Erlang Express, taught by Erlang Programming O’Reilly Author Simon Thompson and Quick Check Express, taught by Prof. John Hughes and Prof. Thomas Arts.

FC: It felt very strange being referred to the Old Guard last year. But during the first day of the conference, I heard newcomers comment on approachable and friendly everyone was. As one of the speakers put it, the Erlang community seems to lack Prima Donnas…. Which is an advantage.  Gone are the days when you knew everyone on the erlang-questions mailing list personally, but the community still feels small and welcoming. Where else do you see inventors of the language helping newbies on mailing lists? Those who have been around a while get excited at all the new projects and companies adopting Erlang, and will help out to ensure it happens.  This is what we have all collectively been working for, and seeing presentations from facebook, E*Trade, Yahoo! or SAP (To mention but a few of the success stories) makes us all very proud.

FC: The community seems to be growing the fastest in North America, Eastern Europe and China, but have no hard facts to back this up other than tweets, blog posts and websites. In 2004, it was easy to derive by examining the email addresses of the erlang-questions mailing list subscribers. Today, the majority of subscribers seems to be using a gmail account. We are aggregating concrete data from Erlang related websites, book sales and other sources which we hope to present in San Francisco.

FC: It is very varied. In the US, we have a lot of private individuals attending the training courses and the conferences. In Europe, it is mainly companies; everything from one man band consultancies, small start-ups to multinational corporations. This difference is cultural, where the responsibility is moved from the employee to the employer. These companies, in turn, are using Erlang to solve problems related to distribution, concurrency, reliability and scaling on multicore. Five to ten years ago, Erlang was primarily used within Telecoms. Today, you will find it in finance, banking, messaging, web development, as well as in cloud computing backbones. Another emerging area covers test automation and property based testing.

FC: The Erlang Factory is a conference not only for Erlang Developers but also for non technical people, including managers, testers and system architects. We advocate the right tool for the right job, so expect to meet Ruby and Haskell enthusiasts alongside java and python developers, all there to learn more. If you are thinking of introducing Erlang in your organization or just want to learn more about it, this is the event to be. This is the major difference with the User Conferences, which are primarily aimed at developers.

FC: Those using Erlang on a daily basis will be able to get the latest news directly from the horses’ mouth, meeting members of the community they have previously only met in ascii format. With the ratio of four delegates to every speaker, the Erlang factory gives everyone the opportunity to network, provide feedback on tools and libraries, and discuss ideas, projects and solutions. Many have even found new jobs this way.

FC: Don’t hesitate to try it out one year and we promise you will come back for more, maybe not only as a delegate, but also as a Speaker. With Joe Armstrong and Steve Vinoski in San Francisco, and Robert Virding and Martin Odersky in London being just some of the 35+ speakers at each event who have confirmed, the Erlang Factory has now become the largest Erlang conference around. We are also working hard to reduce costs so as to enable more people to attend. This is, something which in San Francisco has been reflected not only in reduced Early Bird rates, but also in the conference hotel room rate, which we managed to negotiate down by $70 per night compared to last year.

Remember tomorrow is the last day to get signed up for $500…

Mustache is an Erlang port of the Ruby framework of the same name, originally written by Chris Wanstrath, one of the founders of GitHub. A powerful feature of this type of framework is the ability to write a ‘template’ that you could use in a Rails application and then reuse in an Erlang application with no changes to that file. The ‘view’, which in MVC terms is more like the ‘controller’ would have to be re-written, but this would greatly simplify the work of porting an application, OR simply running a part of application in a different framework for performance reasons.

Chris calls Mustache “a replacement for your views”. Instead of views consisting of ERB or HAML with random helpers and arbitrary logic, your views are broken into two parts: a Ruby class (or Erlang module) and an HTML template.

They call the Ruby/Erlang part the “view” and the HTML template the “template.”

All your logic, decisions, and code is contained in your view. All your markup is contained in your template. The template does nothing but reference methods in your view.

This strict separation makes it easier to write clean templates, easier to test your views, and more fun to work on your app’s front end.

Check it out!

In the interest of publishing more often and covering smaller topics, lets review web-server-agnostic middleware for Erlang.

The current standard for Erlang web application middleware is EWGI, inspired by Python’s PEP 333 and providing similar functionality as Ruby’s Rack. All major Erlang web servers support EWGI, and most web frameworks.

Created by Rusty Klophaus, SimpleBridge “takes the pain out of coding to multiple Erlang web servers by creating a standardized interface.” It is used by the soon to be released new version of The Nitrogen Web Framework. According to Rusty, SimpleBridge has some improvements over EWGI, such as a smaller code base; more easily extendable (Takes about 150 lines to add support for a new HTTP server, vs. ~350 for EWGI; support for multipart file uploads, with size limits and handle-able errors; static file support; more specific interface functions for getting and setting cookies.

However, based on my experience with Rack, Multipart file uploads, static file support, and anything but basic cookie manipulation are all best handled by a web server such as nginx or a reverse proxy like HAProxy anyway. Perhaps it is better that EWGI tries to do less in this case…

Let’s look at a simple EWGI example with MochiWeb (shamelessly pulled from the docs)…

First we startup mochiweb calling the loop/1 function. Mochiweb will call our loop/1 function whenever a request is received.

In loop/1 we setup a call to our simple implementation of the EWGI interface.

Now the EWGI implementation is called with a tuple that aways starts with ewgi_context, and contains a Request and a Response. This code returns a 5-tuple response (further details in the documentation).

EWGI solves the problem of creating a standard interface to middleware components. It has become the standard, and the power of such standards lies in their implementations’ ubiquity. Making your own hit web framework at home in your spare time has never been easier.

The Go language, getting a lot of attention because it was recently open sourced by Google, seems to be a sort of programmers’ Rorschach test in that everyone calls it a replacement for C++/Python/Erlang/Your Language Here… My sense is it wouldn’t be getting that much attention if it were released apart from Google. But thinking about Go for the past few days made me realize that the most important part about Go is actually its “Erlang-ness”. Everything else it provides is nice if you’re a C++ developer doing systems programming already, but not critical. But the goroutine/channel piece is the part that actually differentiates the language as something worth investigating that you can’t easily work around with the existing toolset.

Would it replace Erlang for many applications? Probably not, because though the coroutines will be powerful in certain systems, Go is not a platform like Erlang, with all the benefits of hot swappable code, supervisor processes,hibernating processes, etc… To a non-developer I would describe Go as containing some of the building blocks that make Erlang a great language and platform, but ultimately Go is targeting lower-level systems development. That type of coding wouldn’t be a good fit for Erlang anyway.

So where would they overlap? Probably most likely the two languages would be in applications that are not too OS-level but not too application-centric where OTP or hot-swapping code, multi-machine communication, etc would be requirements. The most obvious example would be web infrastructure. Would Go be a good fit for writing a web server vs Yaws or Mochiweb? Probably. Would I want to use Go to write my own cross-node communication that i get for free with Erlang? Probably not.

So I’m realizing that Go provides a service to the Erlang community in that it validates the fundamentals of the Erlang platform. Goroutines are equivalent to Erlang processes:

Why goroutines instead of threads?

Goroutines are part of making concurrency easy to use. The idea, which has been around for a while, is to multiplex independently executing functions—coroutines, really—onto a set of threads. When a coroutine blocks, such as by calling a blocking system call, the run-time automatically moves other coroutines on the same operating system thread to a different, runnable thread so they won’t be blocked. The programmer sees none of this, which is the point. The result, which we call goroutines, can be very cheap: unless they spend a lot of time in long-running system calls, they cost little more than the memory for the stack.

And Channels to Erlang messages: ( see slide 28 from this deck)… very cool stuff and further evidence of the slow but inexorable march toward mainstream multi-core programming.

Tim over at timyang.net has an interesting ‘hello world’ web server test between Erlang, Go, Java, and C. The C code is an nginx plugin, Java uses the MINA framework; Erlang and Go are small standalone apps. Java ‘won’, with Erlang in second place. Interesting article and the first I’ve seen comparing Go and Erlang performance.

Playdar, an open-source ‘music content resolver service’ that finds music on your local computers or your friends’ computers, then streams that music to you, rewrote the code in Erlang. From the original C++ codebase, Richard Jones reduced the lines of code by 75% – from 8000 to 2000 lines of code… The code is on github and the old source is available if you want to compare the two. The biggest hurdle Richard has yet to overcome is the redistributable binary is about 10MB vs 2.5 MB for the C++ version. Does Erlang have a future in desktop applications outside of CouchDB?

As Erlang usage grows the documentation surrounding the platform is being subjected to much more scrutiny. The existing Erlang documentation site is difficult to navigate and I often find myself hunting for a particular module by guessing the module name in the address bar of my browser.

However, Dale Harvey’s erldocs.com takes a lot of the pain out of the old documentation – it has an ‘ajaxy’ fast search, where you can search for a particular function, a module, or both; but it doesn’t sacrifice state in favor of interactivity, because the pages are still static html.

Overall erldocs.com is a great improvement on the presentation of Erlang documentation.

My only quibble is there’s no link to Erlang Inside on the erldocs home page

Welcome to the second of the erlang:lists series where we list some interesting happenings in the world of Erlang.

erlang:lists(NewErlangIRC) – Kevin Smith just launched a new IRC channel – #erlang-otp. Have an Erlang question you need answered quick? Wishing you had a friendly place to hang out online with other Erlangers?  Tired of either the silence or trolling on #erlang? If you answered yes to any of these consider hanging out in #erlang-otp, the friendly Erlang IRC channel. Erlang newbies are especially welcome!

erlang:lists(BuildYourNextWebAppWithErlang) – Steve Vinoski has a new article on “Build Your Next Web App With Erlang” – the latest downloadable pdf in his Functional Web series.

In the spirit of NetFlix, but 1000 times less lucrative, Joel Reymont, a freelance developer who lives in Tenerife, one of the Canary Islands, has issued an interesting challenge. Not able to get his multicast Erlang code to send 20,000 messages in less than four seconds, he asked the erlang-questions forum for help for the last few weeks. I’ve been impressed by the forum’s support but no one could solve the issue. So, he proposed a challenge – $1000 to the first person who can get the broadcast under one second, $500 if under two seconds.

In his words,

The Ultimate Erlang Challenge is on:

http://github.com/tinycode/janus/tree/master

Be the first to improve Janus to unicast to 20K clients with a consistent maximum latency of less than 1s and grab a $1000 bounty.

Be the first to get the maximum latency consistently under 2s and claim $500.

Develop on any hardware but test on Amazon EC2 for final numbers.

Welcome to the start of the erlang:lists series where we list some interesting happenings in the world of Erlang.

erlang:lists(CouchDBNaked) – Harish Mallipeddi, a performance engineer at Yahoo, has put together a great post on the internals of CouchDB, and how to even use some of the couch source to build your own B-Tree based mini-application.

erlang:lists(ErlangFactory) – Slides and videos are up from the 2009 London Erlang Factory conference. About half have videos and most have slides available. Not sure yet if they will all have videos or if there are reasons for not showing the films for some speakers.

Each week I’ll mention a few must-see articles, blog entries or videos. If you have something to bring to my attention – contact me at chad at inakanetworks com.