Erlang Inside

I took a minute from getting ready for my talk at Erlang Factory (and working on the day job) to talk with Marc Worrell, Lead Architect of Zotonic – a new Content Management System written entirely in Erlang. This is a brief overview of the system focused on the ‘why’ – the technical details are well covered on their site, but I wanted to understand the motivations behind a company taking on a project like this. Marc also runs his own programming consultancy, WhatWebWhat.

I’m excited to experiment with Zotonic, but I frankly don’t have a downloaded copy of Postgres lying around. I’m planning on getting that installed in order to try it out and will report back.

Here is the interview, as usual, emphasis mine.

1. Hi Marc, can you tell me the impetus for Zotonic?

I used to be the CTO at Mediamatic. Mediamatic built and maintains the community management system Anymeta. Anymeta grew from a simple library to a full fledged community management system. For that we added more and more interactivity and connectivity. Which brought us in a head-on collision with PHP.

PHP is far from stellar in handling many small requests, can’t keep connections open and gets really slow when your codebase grows. Though we need open connections to user agents, XMPP connectivity and quick turnaround on Ajax requests. All are challenging for PHP. So we ended up adding more and more system services to make PHP perform and have the system support our requirements.

Growing a site with PHP starts with opcode caches, memcached, a modified memcached with dependency checking, extra servers and then more servers. Finally adding XMPP server and client processes written in Python. A whole lot of extra software.

And then we were still rewriting PHP tables into code so that it performed better on application load.

Frustration, especially considering my background with building embedded development platforms. Embedded systems and the programming language C show what you can do with the available CPU power.

After I left Mediamatic I started ask myself how to do this better, how to support those multi-core cpus, how to use those gigabytes of memory and how to have all the connectivity a modern website (information hub) needs.

I checked out Ruby on Rails, I checked Django and some other frameworks. They promise a lot. They fall short when you benchmark their efficiency on modern cpus and server systems.

That is when I rediscovered Erlang. The book from Joe really brought me to the realisation that Erlang could solve most, if not all, the problems I was encountering. And his approach to failure was nothing short of a revelation to me.

This lay the seed for Zotonic, as there was nothing else.

2. The Zotonic team seems to have a lot of experience, what brought you together, and what do you do now? Is Zotonic your full-time venture or are you doing something else?

We all know each other from Mediamatic. Tim and Arjan still work for Mediamatic, Peet is a former colleague and now works for Mirabeau. I am currently a freelancer. Zotonic is my tool to deliver great looking web sites that perform well. You could say that it is my full time occupation. Tim and Arjan are part time working on Zotonic.

We get more and more project requests, so we are happy that we did make the right decision to start Zotonic and make it Open Source as well.

3. Why Erlang? How does it make a CMS easier if most will be non-distributed/one-box systems, and Erlang isn’t known for Raw I/O performance? How are you taking advantage of Erlang’s sweet-spot of easy distributed message passing?

We are now using Erlang’s excellent failure handling and multiprocessing capabilities. I think that the failure handling and with it the possibility to make robust systems is one of the often overlooked features of Erlang, when not the most important.

We don’t use the distributed nature of Erlang, yet. But that might change quite soon.

That text processing is essential for web apps is a misconception. Most web apps only parse requests, fetch some data, concatenate a whole lot of text chunks and then output them over a socket back to the user agent. You don’t need text processing for that. In fact on low end machines Erlang is quite capable of saturating a gigabit connection. Informal benchmarks gave more than 6000 page requests per second (famous helloworld templates) on a low end server. That should be enough.

To achieve this performance we made some rather big changes to Webmachine. Basho will be looking into incorporating those changes into the mainstream Webmachine code.

And programming in a functional language is fun

4. NoSQL support? is it important for a CMS? Are there plans to support Riak or Couch or MongoDB?

NoSQL is really the story of the day. A lot of people see it as a means to an end. Some think you need it for performance. That is not true. An external party did some benchmarking on Zotonic and expected the database to be the main bottle neck. It was not, by far.

As I often say, the best way to achieve high performance is to not hit the database. Add some caching and the current RDBMS products can bring you a long way, especially on modern hardware. When you don’t need millions of objects and 99.99% uptime guarantees then I suggest that you just use a RDBMS. Your data will be safe and the administrators of your customer will be happy that they can access the data easily.

Having said that, we are looking into Riak and some other distributed stores. We have some projects at hand that require a scale and uptime that will make us move to a distributed system with elegant failure handling. Only with these requirements a distributed store becomes important.

5. How do you guys use Zotonic yourselves? From an end-user perspective, how does it differ from Wordpress or Drupal?

A first impression is important. And the first impression of Zotonic is its speed. I had people asking things like “stop… did I really see five pages just now?” They are used to the slowness of Wordpress and Drupal, at least when not using Google, Yahoo or one of the other big sites.

Over time there will be another big difference: realtime. We are in the process of adding more and more realtime features. They enable realtime communicationa and federation of data. It doesn’t matter where your data (profile, blog etc.) is, it matters that it is up to date and can be published where you want it.

For ourselves, we are moving all our sites, blogs, and whatnot to Zotonic. It is more fun in Zotonic and we are eager to use the PubSub capabilities Arjan just added.

6. From the IT-guy perspective, why should he roll out Zotonic?

You need less hardware for more visitors.

Let’s give an example.

We made the site for the New Island Festival, a cultural festival in New York. At some point we were seeing sustained traffic of 2 mbit. What made that traffic impressive was that the site was hosted on a small VPS and only using 5-10% cpu. We have seen other requirements for PHP sites

7. What are the largest deployments so far?

The New Island Festival was a nice test bed for the number of visitors. A bit bigger, content-wise, site we just finished is the site for the Mediafonds in the Netherlands. This site has around 25000 pages.

We are now working on a project with a projected page count of around a million and later we will have a project that requires a ten fold higher page count.

8. Roadmap – what are your plans for the future of Zotonic?

We have now scheduled or are planning:
- redesign of the admin environment.
- redesign of the access permission system to better support user generated content.
- extended support of XMPP PubSub, also sharing images and other metadata.
- realtime tools for server monitoring.
- support for distributed Zotonic nodes (still with a central store).
- support for a distributed, fault tolerant data and file store.

And many smaller changes that will make the system better for building the websites our customers and programmers love.

9. MySQL Support? Is it planned?

No, we don’t plan MySQL support.  Though it should be doable with the new binary driver. Right now we are focusing on the distributed version with a k/v store.

The Erlang Factory’s 2010 conference is March 25th and 26th in San Francisco, with the university three days before, starting the 22nd.  The conference is at the Hilton San Francisco Airport, with three tracks, each on a different theme. The number of tracks gives attendees an unusually broad set of choices for talks. When attending conferences I use the times when one or both talks are uninteresting to check email and generally hang out with people. Occasionally there are two talks I want to see at once so I’ll pick one and then duck out to catch the end of the other. Joe Armstrong and Nick Gerakine’s simultaneous talks on Day 2, for example. However, in this case there will at times be three interesting subjects running concurrently, which is only appropriate for a conference on Erlang.

Francesco C: Erlang Factory is becoming a very popular event. We are expecting an increase in participants, somewhere between 150 – 200 delegates. We have seen a growing trend, irrespective of the state of the economy.

FC: It varies. Last year, we had students who never used Erlang in a commercial project but were interested in learning more. They were interacting with people who have been working with Erlang full time for 15 years. Even with this huge gap, you did not notice any divide. That’s the beauty of our conferences; Erlang enthusiasts from architects to newbies have an opportunity to exchange knowledge, ideas, experience and have a great time together. Of those attending, about 50% came from the west coast, 17% from the east coast, 15% from the mid-west and 18% from the rest of the world, mainly Europe, but also from Canada, Central and South America.

FC: About 20% attend the Erlang University. The value of attending the courses in this way is that they are condensed versions of the 5-day courses and allow delegates to attend the training as well as the conference. Furthermore, having the training together with the conference provides the opportunity to discussions and networking, multiplying the value many times. New for this year is a course by Basho Technologies‘ very own Kevin Smith on Web development in Erlang. This course will run alongside the OTP Express course, taught by Erlang Solutions’ training manager Henry Nystrom, Erlang Express, taught by Erlang Programming O’Reilly Author Simon Thompson and Quick Check Express, taught by Prof. John Hughes and Prof. Thomas Arts.

FC: It felt very strange being referred to the Old Guard last year. But during the first day of the conference, I heard newcomers comment on approachable and friendly everyone was. As one of the speakers put it, the Erlang community seems to lack Prima Donnas…. Which is an advantage.  Gone are the days when you knew everyone on the erlang-questions mailing list personally, but the community still feels small and welcoming. Where else do you see inventors of the language helping newbies on mailing lists? Those who have been around a while get excited at all the new projects and companies adopting Erlang, and will help out to ensure it happens.  This is what we have all collectively been working for, and seeing presentations from facebook, E*Trade, Yahoo! or SAP (To mention but a few of the success stories) makes us all very proud.

FC: The community seems to be growing the fastest in North America, Eastern Europe and China, but have no hard facts to back this up other than tweets, blog posts and websites. In 2004, it was easy to derive by examining the email addresses of the erlang-questions mailing list subscribers. Today, the majority of subscribers seems to be using a gmail account. We are aggregating concrete data from Erlang related websites, book sales and other sources which we hope to present in San Francisco.

FC: It is very varied. In the US, we have a lot of private individuals attending the training courses and the conferences. In Europe, it is mainly companies; everything from one man band consultancies, small start-ups to multinational corporations. This difference is cultural, where the responsibility is moved from the employee to the employer. These companies, in turn, are using Erlang to solve problems related to distribution, concurrency, reliability and scaling on multicore. Five to ten years ago, Erlang was primarily used within Telecoms. Today, you will find it in finance, banking, messaging, web development, as well as in cloud computing backbones. Another emerging area covers test automation and property based testing.

FC: The Erlang Factory is a conference not only for Erlang Developers but also for non technical people, including managers, testers and system architects. We advocate the right tool for the right job, so expect to meet Ruby and Haskell enthusiasts alongside java and python developers, all there to learn more. If you are thinking of introducing Erlang in your organization or just want to learn more about it, this is the event to be. This is the major difference with the User Conferences, which are primarily aimed at developers.

FC: Those using Erlang on a daily basis will be able to get the latest news directly from the horses’ mouth, meeting members of the community they have previously only met in ascii format. With the ratio of four delegates to every speaker, the Erlang factory gives everyone the opportunity to network, provide feedback on tools and libraries, and discuss ideas, projects and solutions. Many have even found new jobs this way.

FC: Don’t hesitate to try it out one year and we promise you will come back for more, maybe not only as a delegate, but also as a Speaker. With Joe Armstrong and Steve Vinoski in San Francisco, and Robert Virding and Martin Odersky in London being just some of the 35+ speakers at each event who have confirmed, the Erlang Factory has now become the largest Erlang conference around. We are also working hard to reduce costs so as to enable more people to attend. This is, something which in San Francisco has been reflected not only in reduced Early Bird rates, but also in the conference hotel room rate, which we managed to negotiate down by $70 per night compared to last year.

Remember tomorrow is the last day to get signed up for $500…

Mustache is an Erlang port of the Ruby framework of the same name, originally written by Chris Wanstrath, one of the founders of GitHub. A powerful feature of this type of framework is the ability to write a ‘template’ that you could use in a Rails application and then reuse in an Erlang application with no changes to that file. The ‘view’, which in MVC terms is more like the ‘controller’ would have to be re-written, but this would greatly simplify the work of porting an application, OR simply running a part of application in a different framework for performance reasons.

Chris calls Mustache “a replacement for your views”. Instead of views consisting of ERB or HAML with random helpers and arbitrary logic, your views are broken into two parts: a Ruby class (or Erlang module) and an HTML template.

They call the Ruby/Erlang part the “view” and the HTML template the “template.”

All your logic, decisions, and code is contained in your view. All your markup is contained in your template. The template does nothing but reference methods in your view.

This strict separation makes it easier to write clean templates, easier to test your views, and more fun to work on your app’s front end.

Check it out!

In the interest of publishing more often and covering smaller topics, lets review web-server-agnostic middleware for Erlang.

The current standard for Erlang web application middleware is EWGI, inspired by Python’s PEP 333 and providing similar functionality as Ruby’s Rack. All major Erlang web servers support EWGI, and most web frameworks.

Created by Rusty Klophaus, SimpleBridge “takes the pain out of coding to multiple Erlang web servers by creating a standardized interface.” It is used by the soon to be released new version of The Nitrogen Web Framework. According to Rusty, SimpleBridge has some improvements over EWGI, such as a smaller code base; more easily extendable (Takes about 150 lines to add support for a new HTTP server, vs. ~350 for EWGI; support for multipart file uploads, with size limits and handle-able errors; static file support; more specific interface functions for getting and setting cookies.

However, based on my experience with Rack, Multipart file uploads, static file support, and anything but basic cookie manipulation are all best handled by a web server such as nginx or a reverse proxy like HAProxy anyway. Perhaps it is better that EWGI tries to do less in this case…

Let’s look at a simple EWGI example with MochiWeb (shamelessly pulled from the docs)…

First we startup mochiweb calling the loop/1 function. Mochiweb will call our loop/1 function whenever a request is received.

In loop/1 we setup a call to our simple implementation of the EWGI interface.

Now the EWGI implementation is called with a tuple that aways starts with ewgi_context, and contains a Request and a Response. This code returns a 5-tuple response (further details in the documentation).

EWGI solves the problem of creating a standard interface to middleware components. It has become the standard, and the power of such standards lies in their implementations’ ubiquity. Making your own hit web framework at home in your spare time has never been easier.

The Go language, getting a lot of attention because it was recently open sourced by Google, seems to be a sort of programmers’ Rorschach test in that everyone calls it a replacement for C++/Python/Erlang/Your Language Here… My sense is it wouldn’t be getting that much attention if it were released apart from Google. But thinking about Go for the past few days made me realize that the most important part about Go is actually its “Erlang-ness”. Everything else it provides is nice if you’re a C++ developer doing systems programming already, but not critical. But the goroutine/channel piece is the part that actually differentiates the language as something worth investigating that you can’t easily work around with the existing toolset.

Would it replace Erlang for many applications? Probably not, because though the coroutines will be powerful in certain systems, Go is not a platform like Erlang, with all the benefits of hot swappable code, supervisor processes,hibernating processes, etc… To a non-developer I would describe Go as containing some of the building blocks that make Erlang a great language and platform, but ultimately Go is targeting lower-level systems development. That type of coding wouldn’t be a good fit for Erlang anyway.

So where would they overlap? Probably most likely the two languages would be in applications that are not too OS-level but not too application-centric where OTP or hot-swapping code, multi-machine communication, etc would be requirements. The most obvious example would be web infrastructure. Would Go be a good fit for writing a web server vs Yaws or Mochiweb? Probably. Would I want to use Go to write my own cross-node communication that i get for free with Erlang? Probably not.

So I’m realizing that Go provides a service to the Erlang community in that it validates the fundamentals of the Erlang platform. Goroutines are equivalent to Erlang processes:

Why goroutines instead of threads?

Goroutines are part of making concurrency easy to use. The idea, which has been around for a while, is to multiplex independently executing functions—coroutines, really—onto a set of threads. When a coroutine blocks, such as by calling a blocking system call, the run-time automatically moves other coroutines on the same operating system thread to a different, runnable thread so they won’t be blocked. The programmer sees none of this, which is the point. The result, which we call goroutines, can be very cheap: unless they spend a lot of time in long-running system calls, they cost little more than the memory for the stack.

And Channels to Erlang messages: ( see slide 28 from this deck)… very cool stuff and further evidence of the slow but inexorable march toward mainstream multi-core programming.

Tim over at timyang.net has an interesting ‘hello world’ web server test between Erlang, Go, Java, and C. The C code is an nginx plugin, Java uses the MINA framework; Erlang and Go are small standalone apps. Java ‘won’, with Erlang in second place. Interesting article and the first I’ve seen comparing Go and Erlang performance.

Playdar, an open-source ‘music content resolver service’ that finds music on your local computers or your friends’ computers, then streams that music to you, rewrote the code in Erlang. From the original C++ codebase, Richard Jones reduced the lines of code by 75% – from 8000 to 2000 lines of code… The code is on github and the old source is available if you want to compare the two. The biggest hurdle Richard has yet to overcome is the redistributable binary is about 10MB vs 2.5 MB for the C++ version. Does Erlang have a future in desktop applications outside of CouchDB?

As Erlang usage grows the documentation surrounding the platform is being subjected to much more scrutiny. The existing Erlang documentation site is difficult to navigate and I often find myself hunting for a particular module by guessing the module name in the address bar of my browser.

However, Dale Harvey’s erldocs.com takes a lot of the pain out of the old documentation – it has an ‘ajaxy’ fast search, where you can search for a particular function, a module, or both; but it doesn’t sacrifice state in favor of interactivity, because the pages are still static html.

Overall erldocs.com is a great improvement on the presentation of Erlang documentation.

My only quibble is there’s no link to Erlang Inside on the erldocs home page

Welcome to the second of the erlang:lists series where we list some interesting happenings in the world of Erlang.

erlang:lists(NewErlangIRC) – Kevin Smith just launched a new IRC channel – #erlang-otp. Have an Erlang question you need answered quick? Wishing you had a friendly place to hang out online with other Erlangers?  Tired of either the silence or trolling on #erlang? If you answered yes to any of these consider hanging out in #erlang-otp, the friendly Erlang IRC channel. Erlang newbies are especially welcome!

erlang:lists(BuildYourNextWebAppWithErlang) – Steve Vinoski has a new article on “Build Your Next Web App With Erlang” – the latest downloadable pdf in his Functional Web series.

In the spirit of NetFlix, but 1000 times less lucrative, Joel Reymont, a freelance developer who lives in Tenerife, one of the Canary Islands, has issued an interesting challenge. Not able to get his multicast Erlang code to send 20,000 messages in less than four seconds, he asked the erlang-questions forum for help for the last few weeks. I’ve been impressed by the forum’s support but no one could solve the issue. So, he proposed a challenge – $1000 to the first person who can get the broadcast under one second, $500 if under two seconds.

In his words,

The Ultimate Erlang Challenge is on:

http://github.com/tinycode/janus/tree/master

Be the first to improve Janus to unicast to 20K clients with a consistent maximum latency of less than 1s and grab a $1000 bounty.

Be the first to get the maximum latency consistently under 2s and claim $500.

Develop on any hardware but test on Amazon EC2 for final numbers.