Interview with Tino Breddin: the Social Science of Tech Communities

Tino Breddin, embedded Erlang developer, takes some time to talk with us.

Why are communities such as Erlang’s and Ruby’s so different? What makes them approchable or not? Is the US the problem? Should we forcibly relocate all Ruby developers to Sweden? Tino talks about these questions and more in this fascinating interview.

Tino Breddin is a Systems Engineer at Erlang Solutions Ltd where he spends quality time on building scalable, highly-reliable systems for messaging and data storage. When not using Erlang he prefers using Python or Ruby for anything which needs to be automated. Previously Tino worked at the research labs of SAP Labs LLC in Palo Alto and SAP AG in Dresden focusing on massively scalable systems development. We talked over Skype two weeks ago about his experience with Erlang and his talk at the San Francisco Bay Erlang Conference starting next week.

Chad (Erlang Inside): Well thanks for the call… So are you based in Sweden?

Tino: No, I’m based in Germany. In fact, I’m the only one.

Chad: How long have you worked for Erlang Solutions?

Tino: I started one and a half years ago working for Erlang Solutions and before that I worked for SAP, one of the larger German companies, so I used to live in California. I kind of moved from Java to Erlang. In the beginning when I worked at SAP there was a division called SAP research. They were focused on what the company would need in five years in terms of technology or systems. As part of one project which I worked on in California, I was looking at alternative technologies, and before I joined my team lead picked Erlang for our project… I think because he met Francesco [ed - Founder of Erlang Solutions] at some point as the technology fit the research goal of the project, so we started using it in a very niche project at SAP research (nobody else was using it) and we tried to advertise it at the company as well.

Chad: How did evangelizing Erlang inside SAP go?

Tino: That’s always a challenge, I suppose. I don’t know whether you’ve been in this situation, but it’s similar to trying to get people to use other niche technologies such as Haskell. If you try to advertise such a technology from a tech point of view, most managers will say, “let’s stick to the old stuff which has been proven and we know how it works”. So we quickly realized that it was a dead end to try to advertise the language itself or even OTP, so in the end we just decided to try to advertise the results, which I think were awesome. We made a lot of small prototypes which showed that the technology was superior to what was used before. This was something people could understand, see it in the numbers and metrics.

Chad: Tell me a bit more about the research you did and specifics on the projects you were working on?

Tino: I was focused on small prototypes around getting messages, evaluating the type of message and routing to some destination or creating new messages. These systems were meant to be very big, scalable, and integrated with other systems, and Erlang was a good fit. I also worked on some data processing with huge amounts of data and that’s also where Erlang came into play.

Chad: You said you did quite a bit of Java and Python in the past. How do you feel Erlang compares and contrasts to those languages?

Tino: Just as a recap I used Java a lot before I started to work on Erlang fulltime, mainly because it was the language we used at work for various projects. I did and do use Python and Ruby a lot fo anything else – more scripting, deployment scripts, etc. and I think for me personally those two languages have a nice style that fit in really well. I think that Erlang is really good for anything I need to do in terms of backend processing, whether it is some system which is TCP based and needs to route messages or some data processing system; whereas Ruby and Python are in my opinion are really strong in the web front and so if I had to do a web application I would still use one of those languages – Django or Rails.

For Java, I think it falls into the same kind of category as Erlang. I wouldn’t use it for web front end stuff. It has it’s advantages (or at least the Java Virtual Machine has it’s advantages over Erlang and the BEAM machine in certain respects – though the BEAM has advantages as well), so I think one should really evaluate the use case, and I wouldn’t say Erlang is always the better choice. And that’s also where I think technologies like Erjang will really change the picture of how people think of Erlang as a language.

Chad: Yes. Have you used Erjang at all? Last year at the conference in San Francisco I met Kresten and saw him working on it. It was pretty early but I’m curious if people are using it now.

Tino: Well I don’t know of any production use of Erjang but I spoke to one of the core committers to it a few weeks ago – a guy from Germany as well and he said that they are at a stage now where they’re trying to get a really cool use case going or at least show some unique usage of mature Erlang frameworks such as Riak for instance but I don’t think there’s anything in production yet that uses Erjang.

Chad: What do you think about Reia since you have an interest and background in scripting languages?

Tino: Well I think that there are two sides of the coin (at least) in my perception. I think it’s cool that people are building things on top of BEAM, which has a different appeal to people in terms of syntax and semantics, but I personally don’t feel attracted to it. I quickly got over the point where the language syntax of Erlang annoyed me; that’s not a problem anymore, so I pretty much stick to Erlang. So I haven’t even tried Reia at this point.

Chad: To me the syntax of Erlang is something I actually like about it once you get used to it.

Tino: I think you are right, it’s because it’s different. When you think about other languages they’re often very similar. I feel at least that when I switch from one language to the other I always mix certain syntactical aspects. But when you switch from Erlang to Python you won’t mix syntactic aspects because the two are so different. I see that as a strength.

Chad: I haven’t met a lot of people that are excited about scripting languages. We have some Erlang developers here and I keep threatening people here that we are going to use Reia on a project, and they’re not happy with me. There is part of me that’s a language evangelist that likes Reia – it’s easier to convince people to use a Ruby-like scripting language that runs on BEAM than it is to get them into Erlang. My sense that there may be people that will find over time that like it better, so it probably just needs some time and attention.

Let’s talk about your talk. I know you were involved in continuous integration, but what’s your talk on?

Tino: I mainly want to talk about the ecosystem around Erlang.

Chad: Oh, okay, great.

Tino: You are also involved with Ruby right?

Chad: Yes.

Tino: You can probably understand my reasoning. In my opinion the various communities around Ruby and Python, Erlang and other languages, are very different. That stems from how they evolved and where they came from and where they are going. In my opinion, the Erlang community has really made a big effort and big jump over the last two years, and I just want to explain how that ecosystem has evolved up until now, and where I think it’s headed over the next two years, and how that compares to other communities, and what benefit that brings to the user.

Chad: And when you say how that compares to other communities, do you mean from mostly a technology perspective or from a personal and group dynamics per? Or both?

Tino: I’m more focused on the group dynamics and what the relationship between contributors and users is and how easy it is to get in touch and kind of be a part of the community. So from a technology point of view I think that the communities are very similar. There might be some different processes in place but overall they’re similar.

I really want to focus on how people can get into the community. You know when you start using Erlang it might feel like the community is small, you know it only has one community website and one mailing list. But if you dig a little bit deeper, it might look from the outside a bit small, but there is a lot of knowledge and a lot of people involved in it so you can get a lot out of the community if you really want to.

Chad: Right.

Tino: So I kind of talked a little bit about that at the Erlang Factory Lite in Munich which we do whenever there is some interest. That was a small meeting for 4-5 hours where I talked about the ecosystem and the response was really good. People who were mostly new to Erlang really liked this overview as they instantly saw how the whole dynamic worked and they saw how to get information and how they can actually get in touch with people.

Chad: That sounds like a great idea. This is actually a pet topic of mine, so I’m really interested to hear the talk.

Tino: Well you’re doing kind of the same thing with your website right?

Chad: That’s really the goal. As we’ve gotten bigger with Inaka we haven’t had a lot of time to focus on it and update the website but we hope to change that this year. But my goal has been to evangelise a platform and a language to people that aren’t familiar with it. Which is kind of why often with my questions I do bring up scripting languages and I ask questions that a lot of the more “hard core” Erlang guys are asking “What? Why are you bring that up?” and they’re just uninterested in a more web perspective. Whereas I feel like if you’re talking to the average Ruby or Python developer (perhaps not quite as much but to some extent), you’re going to be talking to people building websites and doing Facebook integrations, etc, and so that’s the commercial reality of software development for a lot of developers, and so I think it’s important to talk about Erlang from this perspective.

Chad: So I don’t want to get into too much on your talk but why is it that, for instance with Ruby it’s full of these sort of “rock star” guys with near cult-like followings and Erlang is basically a lot of serious legends but the general feeling is that they’re quite down to earth and very easy to get along with and very open. Do you have a theory on why that is?

Tino: I guess everybody has a theory about that but also I think that the people who are attracted to Ruby, at least some of them or a significant part of the community are more opinionated. They do like to go out and shout that Ruby is the best thing that ever happened to software development. And that’s not how Erlang people operate, at least the “legends” you were referring to.

Part of that might also be the origin of most of the people. Swedes are not the most opinionated or loud people out there so whereas -- not to be offensive -- but people from the US are more open and like to talk (which is not a bad thing)…

Chad: Hah right…The Benjamin Franklin style self-promotion.

Tino: And that’s the difference…

Chad: Yeah but Ruby is Japanese… and frankly Rails was invented by a Dane… so they took root outside the domain of their creators.

Tino: That’s true but he [ed - DHH, Rails founder] moved to Chicago and the inventor is very opinionated as well and he kick started the community there. Whereas the community around Erlang started and still is focused around Sweden. So now there are more and more people in the US getting interested in Erlang and you instantly see that there is more and more hype generated as people in the US take it up. They are more open and talk more about their opinions, at least on the web, and you see companies like Basho and the guys around OTP in Action. You have the Erlang Cam…

Chad: What’s that?

Tino: Erlang Camp.

Chad: Oh I thought you were referring to some camera that, perhaps, followed Joe Armstrong around or something. Eric and Martin from OTP in Action put Erlang Camp together. Got it.

Tino: It feels like when the Ruby community got started and was very open and going out and saying they’re a new technology and then new frameworks came and it now feels like Erlang is moving in the same direction. Which is great because it means that more people are learning the technology and are getting to have a choice.

Chad: Yes. That’s really cool. I agree with what you are saying though I think it’s a mystery to some extent why Ruby went the way it did; I have more Ruby developers currently than Erlang here but we are slowly converting them… We often have projects where we use Ruby on the front end or for admin and we use Erlang on the back end.

Tino: And I think that’s the perfect use case for the two technologies.

Chad: Have you done any of that type of work where you’re mixing those languages or Python?

Tino: Oh yeah I’m trying to evangelize this kind of use case and setup all the time.

Chad: When you do, how do you tell people that you communicate between them, technically? Do you make Python look like an Erlang node or HTTP or BERT and Ernie?

Tino: It really depends on the use case. Generally HTTP though there are times we use the database and have some kind of notification mechanism to tell the Erlang subsystem or backend that something has changed. But most often I stick to HTTP because it’s just so simple and any component can talk HTTP.

Chad: Any other questions we should discuss?

Tino: Well you asked me one here we didn’t answer – what are some underrated technologies in Erlang? I think there are two kinds of people. #1, the kind who likes to use print statements and when there is a bug try to narrow it down some way or another with them, and #2 the others who try to debug, and that's where tracer comes in really handy, especially when you’re running a distributed system, because then printing out values just doesn’t make any sense anymore – it just becomes too hard to monitor a distributed system in that respect. So tracer and the dbg module are things which have really bad documentation but should be used more. And I think they’re one of those things where you really need somebody to get you started. You can just sit next to someone and watch how they use it and therefore learn from them. dbg is one of those system tools that is just so well hidden that way too few people use it.

Chad: Yeah I used it once and it was useful and then every time I need to use it again I cannot remember how to use it so I have to start over.

Tino: The function names are nice abbreviations.

So the second one is kind of a my colleague Ulf Weiger is one of the more experienced Erlang users I would say…

Chad: [Laughs]

Tino: And a couple of projects where we were thinking about how to store data, most of the time we end up using Mnesia. But the public perception of Mnesia – it doesn’t have a good reputation, it’s documentation is sub-par and it’s really hard to figure out how it does things because it’s fairly complicated. I usually use the comparison where when you think about a database, when you want to store data you use the database, and it’s a black box. you don’t start digging around in the storage subsystem of mysql, whereas with mnesia its just a component which you use to create a database system. It’s really an embedded component so you have to work hard how to wrap your head around how you would use Mnesia.. and that’s the hard part. So I hope someone will write a book about it and people will see the light and use it to go on and do more cool stuff.

Chad: That’s interesting. I’ve had trouble with Mnesia, we’ve used it for a few projects. The problem would be we would have a small database and maybe we would shut the node down poorly and then it would take 25 minutes to boot the server and it wasn’t that much data, and we kind of said to ourselves, if we have that problem with test data, in production where we gather about 1GB of data a day of Twitter data, which frankly isn’t that much... There were just too many warts related to startup time and maintenance. You kind of have to be a db expert to run it, so I’m curious do you feel the same thing or do you often feel like you’re working with people who can deal with the ‘warts’.

Tino: No I think you’re right in that you need a little bit more expertise in terms of database systems, and how do you architect your database. We often use it so I have a good feeling about using mnesia but I have the same observation as you do – it’s not always a good fit. Especially when you get started for the first or second time, you need a lot of time to figure out how the little features fit together and shouldn’t be afraid to just dig down into the code because this is for the most part how I learned how mnesia works. And this is just natural because it’s a component which you need to embed and add your code on top of it in order to make it really useful for you. So out of the box it most often doesn’t fit into any system – you have to engineer some layer on top of it to make it do what you need it to do. And to do just exactly that thing we need to step up and create better documentation and a better awareness of the technology and I think Ulf does a good job evangelizing the technology. So hopefully at some point he gets a chance to write a book about it.

Chad: That would be a useful book. As the Erlang community grows I think you’re finding that there is now room for specialty books that there wasn’t before. The OTP book is a good example.

OK that’s really helpful and a good interview.

Tino: Well, you said you have some Ruby devs being converted to Erlang devs – that probably already makes you the largest Erlang development shop in South America.

Chad: Hah well there used to be about 10 developers at a company here that only did Erlang and for some reason they disbanded. One of those guys is here so we often joke that there are 9 former Erlang samurai warriors wandering the desert of Argentina like “Ronin”, and we must find them. So there are some Erlang guys here, and a lot of shops that are using Erlang. Hopefully we’ll have enough projects to round all of them up.

Well thanks for the interview – I’m really looking to seeing you in San Francisco in a few weeks!


Follow Tino on twitter at @tolbrino and follow me at @chaddepue. Get an alert when we publish a new story by following @erlang_inside.