Erlang Inside Rotating Header Image

Erlang and Test Driven Design – an Interview with Fred Hebert

Fred Hebert is an Erlang enthusiast based in the unexplored Northern parts of Quebec (relatively speaking). He is so enthusiastic, in fact, that he started writing Learn You Some Erlang for great good! a free online book designed to teach Erlang, functional programming and basic computer science concepts.

He started his career working with backend services for medium to large scale web sites, but php just did not do it. In 2010 Erlang Solutions came to the rescue! He is now working with course development and e-learning while remaining true to web applications, this time, written in Erlang.

Chad: Ok so, first thing I would say is welcome to Erlang Inside. So tell me a little bit about yourself and your background and how you first started programming?

Fred: Well, I didn’t want to start programming in the first place. I started studying in multi-media in the first place. I wanted to be graphic designer and one the classes we had was about programming Javascript and PHP. And so I really liking programming from that point on because I found that it had creative aspects I was looking for in a job; but also better salaries in general than designing. At least here [Canada] that’s the case. And I kind of just always liked the idea of doing the problem solving part of it and from that point on I just started studying on my own. At some point I picked up Scheme and a bunch of other languages and got a job as a web developer and working on back-end services for a dating website. And at this place I got at some point into a research and development group where my boss asked me to look into Erlang because Facebook started using Erlang around that point and I just decided “Yeah, ok I’ll look into Erlang.” And the group was kind of the funny thing because my boss-when I went on sick leave one of my co-workers went on a parental leave so with my little team I was alone for like 3 months on the same level as the executive board of members of the company. So nobody could tell me what to do I just had 3 months to look into Erlang and play with it and try a bunch of different things.

Chad: And this was in Canada or in the U.S. or where?

Fred: Oh, in Canada. So yeah, I learned it that way and at some point I was just looking for a project of some sort and decided well I’ll write a book about it. That way I will force myself to learn really everything correctly and it will be something creative. I will be able to do a little bit of web stuff that I like draw a little bit and get better at writing and speaking English basically.

Chad: That’s cool. So where are you from originally Fred, are you from Quebec?

Fred: Yeah, I’m from Quebec. I grew up speaking French so I’m a true French-Canadian there.

Chad: So basically in 3 months you started working on learning on your own and basically said hey I’m going to write a book and that book was “Learn You Some Erlang?”

Fred: Yeah, that’s about it and at some point I got bored with my job that I had and I just went on Twitter and asked Francesco [Cesarini] because I knew he wrote the book and worked for Erlang solutions. Francesco wrote at some point “We’re looking for developers in the sub-Saharan Africa” for some kind of contract and I just sent him a reply saying “If you’re looking for someone in Canada tell me.” And I got the job that way. So that’s how I really got into Erlang and working with that. It’s a bit of luck and… Yeah, a little bit of work and a little bit of luck.

Chad: That’s great, and how long have you been working for Erlang solutions then?

Fred: It’s going to be close to a year now. I started what was it, late September beginning of October maybe it was October and November last year.

Chad: And how long has “Learn You Some Erlang” been out? Well, I know it’s been a work in progress for quite awhile. But when was the first part of it released?

Fred: On Learn You Some Erlang, I think I started working on the first plans sometime in April two years ago and the first 3 chapters were published I think in July two years ago. So I’ve been working on it for more than two years and that’s probably the project that I’ve had for the longest time in my life.

Chad: Ok. Since we’re already talking about learn you some Erlang let’s kind of keep it going on that. Has anybody given you a hard time about adopting the style of Learn you some Haskell?

Fred: No, the thing is the guy from Learn you A Haskell is a guy who’s username is Bonus on the SomethingAwful forums. And he was into programming people there sometime on the same IRC channels and I talked to him and at some point he presented the book in the forums and when I was looking for ideas I’ve never written a book before or really anything this close to being that extensive. So I just looked at his book and I said “Well I could do the same for Erlang as he did for Haskell.” And I basically asked him “Would you be alright with me doing that?” And I started by just copying the basic structure of the index, introductions starting out modules and that kind of thing. Actually that really helped me a lot because all directions of where do I start and what kind of style do I take and what kind of tone do I take. I could just inspire myself from the stuff he did and he always left me a lot of liberty about that he never came and told me “Well you should change this” and that kind of stuff. He always told me that he thought it was great. It was more help than anything else just because I could steal some ideas and inspire myself.

Chad: That’s great. So is it complete or is there more to come?

Fred: Well, I signed a book deal about it so it’s going to be published at some point. So by February I’m supposed to be completely done with it.

Chad: Who did you sign the deal with if you don’t mind me asking?

Fred: It’s No Starch Press the same people who published “Learn You A Haskell.”

Chad: Oh, cool.

Fred: So what I want to cover I’m working right now on chapters on application of dates and release of dates. So app-ups and rel-ups. I want to work on ETS; I want to put a chapter on testing type checking — these kind of things. I want to do ports and sockets. What else do I want to do? I have a bunch of ideas I have something like 6 or 7 more chapters that I want to do. I still need to cover distributed Erlang and distributed applications. I’m showing releases but I’m showing no distributions as of now.

Chad: And so is ESL sponsoring this thing or is it more…

Fred: No, I always wanted to keep everything independent sometimes people offer me donations and I never wanted to take them. The reason being that as soon as someone starts paying for product, they can feel not entitled but kind of responsible for it and can ask to change the content; and I always wanted to have full control on that kind of thing until it’s done so I want to keep all sponsorship advertisements and donations out of the process.

Chad: Ok, that makes sense. So do you have any download stats? And page views? Have you had a lot of people give you feedback or thanks or what?

Fred: Let me see, I’ll just go look at the stats. But as far as feedback goes and these kinds of things I get a lot of it. Mostly positive, I guess people who don’t like it just don’t talk to me in the first place. So the general opinion that people mention to me seems to be rather positive about it. That’s the reason probably why I kept going for this one. About feedback yeah, I’m getting a few emails every week. Maybe sometimes about fixes, sometimes it’s questions about the content sometimes it’s people needing a little bit more guidance on ISU or reading a tutorial that point. So it’s really nice for that. Let me see for the stats I need to configure some stuff because I’m a bit paranoid about people sensing data on the server. The stats have been going up all the time except for the days where I will be on and I will get user peaks and I will get 20,000 new page views a day and that kind of stuff. I think for the first year the average had to be from 400 to 600 unique page views per day. Last year it went up to something like 600 or 700 and right now I think I’m going about a 1000 a day.

Chad: That’s pretty good. Well let’s talk a little bit about test driven development. Tell me your interests and what test driven development is to you. That’s a funny question but… Yeah, first like TDD at a high level and then we’ll talk about it specifically with Erlang.

Fred: I have got to say I had been doing very little test driven development before writing and teaching the course about it. So when I had to write and teach a course I had absorbed a lot of it applied to experiences I had before and that kind of thing. And this is not a problem with TDD. The problem was the fact at the places I worked before had very little tests. So when I got all this information about TDD and these kinds of things. I thought it really could really solve the problems we had many times before and it’s kind of alright to write tests after. The TDD improves you and this is one of the things I mentioned the most in my course. TDD is not about testing – its about design and then the tests are just a side effect of designing your stuff properly.

The big advantage that you have is that writing tests after the fact is pretty boring and so everyone just kind of botches up the work and nobody does it seriously when you write the tests later. So the only fun time to write tests (to me at least), is to do it when I’m creating the code because creating and writing the code is what’s fun. So if I write the tests at the same time – I’m going to write good tests. If I write them after the fact I’m going to be bored and do a crappy job in general.
The second point is that writing the test first makes me think of how am I going to use the code. What kind of functions do I want to have? What do I want the interface to be like? What kind of arguments am I going to pass in there? And this forces you to take the perspective of the user of the code rather than the perspective of the writer of the code. When you write the code you sometimes try to just make the cleanest thing that can work and then you end up having something that’s a bit more ugly when looking at it from the outside. And then sometimes you’ll have code that’s hard to test because you wrote it for it to be easy to write and read and not to be easy to use. And to me that’s the real advantage and that’s why it’s designed – you’re taking your specifications what you need to be done-“I want this feature or this and that way and that other way.” And then once you have these features and you’re thinking about that you just turn the specification into to codes to some extent and I wanted to return this kind of value and this is the structure I expect to have. Another advantage is always on re-factoring, which will mean that there’s some confidence in your code. You can change the code and be sure that to some extent it still works the way it did before. Or it’s still broken the same way it was before. Because that’s one of the risks you’re too confident in your tests and then you just jump off the bridge thinking that you’ll land fine and then you die or something like that.

Chad: So that’s what would happen to you if you don’t do test driven development. So let’s talk a little bit about how that applies to Erlang then.

PropEr

Fred: Ok, well for Erlang I must say we have fantastic tools to do testing we’ve got a few as frame work common test which have pretty bad documentation but is a pretty awesome framework for testing we’ve got EUnit, we’ve got PropEr and quick check. We’ve got tools like meck which is pretty damn neat mocking tool I’ll talk more about it if we have time later. But it’s a really great tool.

In any case, for Erlang people tend to test sometimes after the fact. There’s a few reasons for this. One we’ve got the shell and people tend to test a lot of their stuff and the behaviors in the shell. You’ve got stuff that’s timing dependent and parallel code that’s hard and difficult tests. And then you’ve got somewhat of a tiny culture about tests – we don’t care that much about testing sometimes. We don’t see testing frameworks as one of the tools necessary to work with Erlang. And I mean a lot of programs work fine without it.
But, when we look at something like the OTP releases the whole code is entirely tested, but external code is often not.
Finally, processes are ‘shared nothing’, so it’s sometimes hard to test that kind of code without altering it to be able to test it in the first place. That’s one of the big issues for people having a hard time testing Erlang code.

Chad: What’s do you think is the most popular test tool for Erlang currently?

Fred: EUnit is by far the most used one I’m pretty sure of that. PropEr is getting a lot of attention since the conference is in San Francisco or London where they were announcing some features that were in there. PropEr is absolutely interesting for property-based testing. So that’s one of the strengths of Erlang there. But yeah, EUnit is most used I’d say then PropEr then I will have to say QuickCheck and maybe common tests in last and then you have test server which is kind of a low level common test off that’s going to be the least used. And then some people have their own little frameworks that they wrote out.

Chad: And do you feel like PropEr will eventually take over EUnit or do you feel like it’s more of a niche?

Fred: I’m thinking that they have different purposes. You can test a lot of stuff with EUnit you can test a lot of stuff with PropEr but some of the code doesn’t have very clearly defined properties. Sometimes you just want to test if I’m receiving true or false I’m expecting this kind of that kind of output and it’s not worth it to write sometimes a property about something that doesn’t have that much of a big space of what can be returned and what can be input into the function. I’m thinking usually the kind of stuff I like to do with proper-everything that can be defined not as an interface but as a behavior. I don’t mean OTP behavior but one the examples in my courses is when you’re writing a website or a blog you have something called a slug and the slug is that part of the URL where all the special characters are replaces by dashes and you only keep the ASCII stuff and that kind of thing. And so for that one you will have pretty clear properties you will have I’m putting in a string and I’m getting a string back. I will have only characters between this range so integers. Then I will have A to Z, “Z” (or “Zed” depending on where you’re from). And then you will have dashes but not two dashes one next to the other and then you have these kind of properties and the input you can put in the function is pretty damn wide. It can be any kind of string, you can have EUnit code stuff and you have to handle it right. And so for that kind of stuff I think property based testing is pretty fantastic. You can just generate random input get it out. Data structures are another great application because data structures will have properties when you insert or you get a tree of that shape then it’s supposed to be transformed in that matter in respect this property — everything is ordered and that order matters. So that’s great for property based testing.

But then you might be working with something that’s not as good for that. I could say I’m working testing a finite state machine. And I’m testing a function and while I’m passing the arguments for some kind of list or I’m passing a name I expect the process to be registered with that name. So at that point it’s necessarily worth it, I think, to test it with proper because a names a name and you put the properties you want and then what you want to test is the actually registration or data process that’s found and for this kind of thing I think you can feel safe enough with EUnit anytime.

Chad: So ok let’s talk a little bit about test driven development with such process and state focused applications like gen_fsm. Let me think about how to ask this question better. If I’m doing test driven development with a website it’s very easy to test a rails website. I can test my models with Test::Unit or rspec; I can test the controllers using something like Cucumber or Selenium. If I’m testing something like a gen_fsm or a gen_server – well, you know the irony of Erlang being sort of no shared state is that those things are all about shared state. They’re all about internal state – and so what I’ve found is that my EUnit tests spend a lot of time mocking the inputs and the database layer and so I spend a lot of time building a lot of time building wrappers around the gen_server or the gen_fsm. And a lot of my tests are actually testing things like timing, testing things like moving through a certain set of states and there’s a lot of complexity to that. So I’m curious how you treat doing Erlang test driven development different from say traditional web test driven development.

Fred: First off, you mentioned testing internal state and that’s something I disagree about doing. I feel it’s sometimes much better to just test from the API because the problem is that if you are testing on the internal data structures the internal state these kinds of things when you are to change your implementation of the code you need to change the tests. And the tests are kind of useless from that point on. You cannot use them to re-factor code to change how it works or to compare to what you had because it was tied how the code is built itself. So I tend to avoid and just test the interface of the module so only the exported functions these kind of things. But to get back into something like gen_fsm there are two approaches that I think work rather well. First of all you can just test with the callback functions so I’m just faking the input in the state to handle call or in that case it’s handle sync events let’s say. And so I will put the input I expect them to get I will put the current state I’m supposed to have. I will put the current state data I’m supposed to have. Then I will look at the output and all the internal messages going on. So that’s something that you can just taking the callback functions and testing what they do themselves. Something easy that you can do is use something I’ll just tie it in is the function called sys:get_status and that one is for all the OTP behaviors. When you do that, you will receive the whole data structure of the internal state of the gen_fsm server finite state machines are. So by calling this function you can say at this time currently in time this is the state of my finite state machine or gen_server. And then you can look at the internals that way without breaking the API.

Chad: Interesting.

Fred: So that’s pretty useful. A third way to do things, it’s a bit more complex because it’s playing with the global state of the VM is using tracing because in Erlang everything at all can be traced. Sometimes I will just call function- I was testing the other day for Erlang solutions E-learning offers and certifications stuff. There’s an exercise about process rings and so to run the process ring the thing I did was found the function traced whatever column of messages that were in there and analyzed that I got all the properties I wanted. Do I get enough processes? Do I get enough messages? And then you can draw a graph of where the messages were coming from and going to. And by doing it with tracing then you can get a lot of insight about what’s going on in the VM as a whole. So that can be a useful way to do it

Chad: Do you ever do tracing with automated test development where you’re actually looking for a certain set of states in the trace or is this more normally a manual process?

Fred: Well what do you mean?

Chad: In other words let’s say you’re running a EUnit test and turning tracing on the process and then receiving the trace and then actually parsing it to make sure the right thing happens? Or is this more that you turn tracing on and then you’re watching it manually to make sure that you know you’re…

Fred: Oh no, when I’m doing automated testing it’s always automated testing. When I trace it there is the option to send the messages to a given process.

Chad: Exactly right. That’s what I mean – how are you doing that?

Fred: The test process will gather messages and sometimes I just do it once in the shell and “I say ok that’s the kind of behavior I expect” and sometimes I find a bug right in the shell. And I will say “these are the kind of events I expect.” And so I will write them down and the more I do this kind of testing the less I use Erlang shell. I rarely use it except to just compile stuff and then run the tests with itself. Because all the behavior I want to test is already in the test so I no longer need to open the shell test it wait for the time, see if everything’s alright.

Chad: Ok, that seems like a great approach and actually that’s really interesting because I can’t think of another platform where you can actually turn tracing on so easily and test what’s happening inside of another process or an object other than Erlang.

Fred: Yeah, I mean there has to be ways to do it other languages in some places but it’s not always obvious or thought the same way. I mean Erlang was made to run forever so you don’t want to stop your program you might want to debug it in production so this is something that not many run times or languages have ready or easily available. Some guy from IRC a guy I’ve worked with a lot said he worked in embedded software and sometimes you needed to have dedicated hardware to that kind of tracing and it would cost them thousands and thousands of dollars and we just had it for free. So that’s pretty neat.

Fred: One of the other advantages that we have is with something like meck mocking tools in general get kind of weird when you have to work with types and then if you have classes in an object or in languages then you have to work with inheritance and making sure that everything respects the right API and then you have to private the public method and it can make it very complex to do mocking properly. And when I’m looking at tools like meck well the way it works is it can just swap the module from the runtime system in Erlang; replace it with whatever it thinks should be doing or mocking it and then does it transparently. And that way you don’t have to find a secret way to change the class something it just leaves your code the way it is and just changes the global environment of the VM by loading a different module than the one you usually have. Now you’ve got your mock going you can have passthrough calls. You can check the whole history of the calls you’ve made to that module so it makes tracing/mocking very easy including outputting stuff. I used it sometimes to test file format; am I outputting the right kind of content. You can just mock the ideal module you need un-stick it you mock it and then you’re able to say “well I did output the right stuff” and that’s not something that’s often easy to do or to test when you’re just writing straight out to the VOS with a system call.

Chad: How often do you use meck with your tests?

Fred: Well, I use it frequently whenever I’m getting into modules that have dependencies so I have a gen_fsm that’s calling some kind of server and I don’t want to depend on the server to make my stuff work. So I will just mock the other server maybe its registration. So you get rid of that stuff and you pretend it has the right behavior. So that’s when I will usually use it. I don’t use it in “Learn You Some Erlang” because I only tend to stick to what’s inside the Erlang distribution for that one.

Chad: If someone is writing Erlang code now and maybe they want to get into doing test driven development what tips you would give them to get started? Maybe they’re familiar with Erlang but they’re not an expert. They’ve written some gen_servers they’re familiar with OTP and they want to get into doing more TDD style development. What’s the most important thing you would tell them to do?

Fred: First I’ll tell them to acknowledge the tools that they have available. You’ve got EUnit that does very well for unit testing; when you get to integration testing and system testing then common test starts to be very interesting. You can do distribution testing something that works. Many notes start them for you. Have more complex hierarchies. When you have something that’s more of an algorithm then I would go with PropEr and these kind of things. So you’ve got a bunch of tools and the right tool for the right job based on that. Then once you have an idea of what these tools are I’d say try to find where you’re problem fits into that what are you trying to test. If it’s a module or something like that I would just say “well test it the same way you would test it in the shell.” The quick way that I like to do it that way is that if it works in the shell for you then it should work in the code. And the only thing you do is a take a little bit more time to type that code in the module and then after that it’s always automated so you don’t have to type that in the shell anymore. But you still get the same kind of tests. So to me that’s the easiest way. Do whatever you were doing in the shell do it in the module. Don’t care about it after that.

Chad: Is there any tool out there that you know of that does equivalent of AutoTest? Are you familiar with AutoTest and Ruby where you can basically run a series of tests and then if any fail anytime you make changes to a particular file it only re-runs the tests that would be affected by the chance you made. In other words it sort of eliminates having to run through your entire test suite while you’re doing coding and debugging.

Fred: No, I’m not aware of that.

Chad: Yeah, I was curious I haven’t heard anything.

Fred: It should be kind of simple to write one if you’re using Emake files you can use make all load and then itself putting you all the modules that need to be recompiled. So you could probably look to these modules there if you capture the output from the made call and then just run the test related to these files you could do it. That’s something neat about the Erlang testing tools. It can all be used together because everything has to evaulate to something. EUnit finds a mistake it’s returned false. So I can run EUnit tests from with common tests and these EUnit tests can be running proper tests and one inside the other like that. So there’s a way to use them and build tools on top of them that could do it. With a little bit of trickery, a few hundred lines of code and you will have that tool.

Chad: OK that makes sense. I want to talk a little bit about continuous integration. Are you guys doing a lot with CI and does that fit into your testing philosophy?

Fred: Well, I’m not very active on the programming side of Erlang solutions right now. I’ve been spending a long time writing course material, updating it and these kind of things. And I’m starting to work a little bit on helping people on E-learning and that kind of stuff. But I know that we have this project I forget the project name. At Erlang Factory in San Francisco Tino Bredden was presenting it. We are running CI servers and hoping to do it as some kind of platform as a service something like that. And Erlang solutions was working on a solution to let anyone run a code on our continuous integration servers and to be able to see the results on there then you could have different plans; free stuff, paying stuff, and all these kinds of things. I’m not part of the project so I don’t know all the details but we’re working on that and general testing stuff. I know that we have a few internal projects I think working with that one. So we do have a continuous integration server. If I recall for a while we were using the EUnit “sure fire” module if I remember what I saw. And so you could get the XML outputs that you needed with I think it was “Hudson” which is now “Jenkins.”

Chad: Are there tools that you don’t have that you wish existed or anything that is missing in the Erlang toolset?

Fred: Yeah, there is a tool I’ve had in development for a long time but I never find the time probably never will. It’s a bit tricky. But if you work with Python you have something called “Doc Tests” and so the way Doc Tests work is that you type in what you want to happen in the Python shell and then you type in the output that you expect and you can test against that. And what I thought about in Erlang it would be absolutely neat if we could have a shell recorder because everything is Erlang calls. So it’s easy to get the timing in there. You could start something called the “Shell Recorder” it just binds itself to the Erlang shell and would record all of the commands you send in there, records all the output that’s coming out and then saves that into a file that you can edit. And then you can just run the shell session over and over again — this kind of thing. So you save all the input save all the outputs and then run it again to see if it still fits the previous recording. So you could just test code the recording that way. And that will be a very easy way to test output of the code, failures of the code and that kind of thing without much of a problem. So this is something I would find pretty neat and I’ve had that in the back of my mind forever and I’ll probably never do it. But I think it would be pretty nice.

Chad: That’s great, I really like that idea.

Fred: Yeah, I like it too and then sometimes I find it less useful because the more it goes the less time running code into the shell I’m just running my test suites. To some extent I’m losing the need for it but before knew I all the tools we had it did look like something very interesting especially for testing something for cross-notes.

Chad: Ok, I think that’s actually a brilliant idea I wish I had time to work on something like that. Ok, so any other questions we had on TDD-yeah I think that was all the main questions I had, anything else that we’ve missed?

Fred: Well we had introduction to different tools. No I think that might sound like most of it from that part I guess. I mean you’ve got all these other tools that are not about testing but are still about support and all the environment you would quickly mention tracing tools you’ve got a bunch of them you’ve got the logger which is part of the language this kind of thing. One thing I like to say is that the best you can have is when you put your code in production. And so you have to be able to diagnose problems in productions, be able to fix them, and be able to work with them. And that’s something that you can do in Erlang that’s hard to do with different things different software in these cases.

Chad: Just a couple of practical questions. How often do you declare the types all of your methods up front?

Fred: Well, I have two types of codes usually I have prototype code and then I have production code or project code. And I don’t think prototyping is exclusive with testing. Testing is usually useful when you know the properties of your software “what’s going to happen.” If you don’t know the properties writing tests will be kind of useless and get you nowhere. Maybe it will help you find some of the properties you need. But then, I prefer to do prototyping for that. So when I’m doing prototyping and just banging out code trying to see the mechanics the easiest approach I’m taking “does it feel right or does it feel wrong.” In these cases I won’t comment the code that much. I won’t be typing it. I won’t be running spec checks. I’ll be OK with just leaving -compile(export_all)’s all over the place. Whenever I’m doing something that’s more about production though I tend to try to type all the functions, declare all the types, have EDoc in there provide a test going with it. The few side projects I have tend to respect this kind of thing. But yeah, I tend to type my stuff. Type checking is worth the effort in my opinion. Even if it’s not enforced at run time for Erlang it’s still worth the investment in time. Especially since now you can use it with PropEr and if you’re using just broad types and that’s something that needs a sequence of function calls. Then now you can use the type declaration as a way to generate data for your properties and your tests. It’s getting absolutely useful; now add to that the type declarations are useful in EDoc. There’s no reason not to have them anymore.

Chad: That makes sense.

Fred: I mean you have to make it worth your investment in time. And that’s the same with testing, if you think that your test will be extremely tedious to write and you’re never going to use them because you can’t trust your tests… For a lot of these things sometimes it’s not writing tests and when it’s code that you will only run for yourself a few times and you don’t plan to ever use or change after that. And it’s not in public stuff it’s kind of worthless to write tests sometimes because you don’t need them because you don’t need the software that much in the first place. So there’s a yield there between the time you take for the tests and the time you will take to maintain it. And if you know it’s just going to be a one-time code and it doesn’t have high impacts in production I’m more likely to test it in the shell and not write tests for that. If it’s something I’m going to distribute I don’t want to be the guy who kills someone because my software was used and had a defect in the hospital or something like that.

Chad: Just one other thing and I might move this question to a different part of the interview; but if you find a bug in production do you write tests first to reproduce the bug and then apply the fix?

Fred: Yes and no. Sometimes when I’m just hunting the bug-first of all I don’t try these things in production. I will try to get the production system and then I will try in development because I don’t want to trash the data base by accident like I did at some point. But I’m going to try in development and then I’m just going to head into code and I’m a bit of an old school debugger even though I can’t be old school I’m too young for that. I’m debugging a lot of file formats, output statements, E-Code, prints and things like that. I’m using that all the time to debug my stuff and see the sequence of events. With Erlang I’ve got the option to trace instead of inserting that so I’m saving some time some time. But I’m going to just insert stuff in the code directly then find something and say “Oh that’s really the mistake I have.” Then I try to do a quick fix in it and see “Oh yeah I solved the issue by doing that” so I probably solved the problem. Then what I do is a I reverse the quick fix, go write a test, make sure my test finds the failure I spotted in the shell or something like that. And then I enable my quick fix back again and see if the test works now. So I’m testing for the broken condition at first and then I’m testing for the repaired condition. But I’m going to find the issue I’m trying to solve before testing. If I’m just running the test first of all and I don’t understand the context in which it happens I’m going to miss a lot of stuff because in Erlang and any kind of large software a lot of issues you will have will have to do with some kind of global state that has side effects in place you didn’t except. So Haskell people will tell you if you had monads you will not have the problem. In this case it can be configuration it can be these kinds of things and I want to have the system running and not just tests in the file to be able to target the problem with more precision in the first place.

Chad: What about testing and debugging?

Fred: Well, sometimes stuff is just a lost code. You just can’t reproduce it all and you know it’s some kind of hidden bug and the fact that you’re looking at it is keeping you from replicating the bug. And in these cases I fire up the printer, print the code and then just highlight it and check it and do it logically and just trace it by hand. And I’m a visual guy so sometimes I will just draw bubbles and arrows and these are the directions and then this message goes to this other bubble which is this other process. But wait a minute this kind of message came from that place and just looking at the code and analyzing it is usually my last resort. I had to do it once or twice before. But it doesn’t happen much. The biggest bug I had was actually PHP. It was a problem with a chrome job running on a table that was not locked properly. So the bug would only happen once a day five minutes before midnight. In some certain contexts not all of the time; so you had to look at the code and say “wait a minute this table is used by this other bit of code, and this chrome job is only running when this kind of threshold is on the servers.” And then you simulate that and it takes you forever and in some cases like that at the point where you’re just frustrated pulling your hair out of your hair and just have no idea then I print the code look at it and that’s really the best way to find any kind of bug. But it’s just so time consuming to do.

Chad: I have a question for you related to algorithms. So a lot of Erlang products are more what I would say are algorithm heavy programs. In other words you’re doing stuff that has a lot of complexity to the processing. Are you familiar with the guy, Peter Norvig , with the TDD failure of his Sudoku solver? Sound familiar?

Fred: Yeah the Sudoku solving and I think the big issue there–point I was proving is that there are two kinds of ways to attack TDD. There’s I think in the small method you don’t know much about your problem and you’re using TDD to guide a design. And then you’ve got the TDD where you know you’re going to need these kinds of things so you know in that event you’ve got to write a mailing system and that you’re boss just bought something like a fan mail server costing thousands of dollars. And you had this knowledge beforehand. You’re going to have to take two different approaches one is more about discovery the other is more about writing a specification as code. And when you do something with Sudoku there are explicit rules already known and techniques to solve these problems. And it’s kind of I wouldn’t stay stupid but misguided to jump head first into the problem without understanding it fully and what needs to be done to solve the problem. When you know there’s a mathematical reason or mathematical principal underneath, good techniques that have been proven over the years it’s useless to jump head first and ignore all the previous knowledge and just do it guided by tests. And I think that the way Peter Norvig did it with testing the stuff could have worked with TDD. He had all these ideas that I’m going to represent the grid that way I’m going to use this method that was used by expert Sudoku solvers. And you could have written a test to do that rather than doing the other thing; where the guy was just saying “how am I going to represent the grid.” So if you know how to solve the problem then you will save a whole bunch of tests. You document it of course “this is the technique I am using I found it at this place.” But then the test you have can be based on that knowledge rather than trying to brute force your way through the missing knowledge. The same prototyping is not mutually exclusive with TDD both can work together to help you guide your design. I don’t think that knowing or starting with knowledge on how to solve the problem is a bad idea.

Chad: Yeah, I think that’s a great point to me. What I think the problem is that people sometimes think of TDD as being a sort of formula for finding emergent algorithms versus being something that allows you to just make sure that algorithm that you already come up with on your own or that you’re coming up with as a part of the coding is correct. So it makes a lot of sense.

Fred: Right, that depends I think they’re a different school of thought. You have the idea of extreme programming the TDD extreme programming way; extreme programming way sometimes tends to do that more. My personal opinion is that it’s not always efficient. There are some kinds of problems where it will help. If you’re working with business rules I guess it can makes sense to do it that way. You know the business rules in advance. You know the kind of output you need. You have no idea of the systems that are going to be implemented sometimes it can be interesting to search based on that. But when it’s something that knowledge exists algorithms, Sudoku is some kind of algorithm there. We had a talk in London. Some guy was doing maze solving and sometimes you know there are some kinds of known ways to solve mazes. I think it’s counterproductive to just ignore the existing knowledge and think that you can be faster than all the people that have been working on that kind of stuff for years before you.

Chad: Ok that makes sense, cool. Well anything else you want to cover continuous integration, test driven development, I’m trying to think…

Fred: Oh yeah, testing with side effects this is something I see a lot of people do. This is something like focusing for common tests but I like common tests and I’m going to solve it that way. Sometimes I will see people write EUnit tests and then you need to write files on disk you need to start remote notes you need to run the same test 50 times to do some kind of stress measure on your testing stuff. Or you need to be scaffolding I need to start second application on the site and communicate with that one. When you do that with EUnit you have to build all that scaffolding yourself; writing some kind of function that will let you write random files that didn’t exist before and for these kinds of tests you’re getting into something called system testing; where you’re testing the whole system and interactions between different components. Common tests are very good for that. It’s already creating all these things like scratch directories, configurations for tests. You’re going to need passwords that can change depending on the environment you’re running the tests in. You can create different configurations pass that to the test functions going to run with them. It can start remote notes, let’s do write scratch files and scratch directories that you can look at later and do all your file operations that has side effects. You can do them there without breaking the system too much unless you’re doing something global with the system. But if you’re writing just a given path then you can pass that path to common test there. And with all that kind of system testing common tests is really worth it because it’s going to save sometime once you know it to work on these kinds of things. Erlang is totally ready to do this kind of entire software testing and I know some people in some enterprises have been using common test to do distributed testing on C code. Just because in those different protocols it was just easier to distribute Erlang through testing and loads and that kind of stuff than maintaining the C frame work they had that had hundreds of thousands of lines of code. It just scratched that.

Chad: So you can use the Erlang common test for the actual test framework.

Fred: Yeah, but because common test comes with ways to communicate from SSH to HTTP and SNMP agents in there all that kind of stuff. So when you’re communicating with hardware and firmware written in C then sometimes it just simulates the client with common tests; use all the stuff that’s already there and pull the programs as a black box testing in that matter. It’s very neat for everything that’s moderately large to test with many attractions common test is becoming a very neat tool.

Chad: Ok that’s great, and do you guys have a particular set of tools that you recommend at Erlang solutions is there a particular tool you recommend or is it more up to the client?

Fred: I have not done a lot of consulting. Usually if the client doesn’t have an idea of what they’re going to do I figure we’re just going to recommend the tools we know ourselves. Someone on the team is going to be productive with what they know. But I think it’s important that if the client has a given setup and their team is comfortable with that and it works-if it doesn’t work you have to correct that- but if it works for them it’s a bit counterproductive to change their methodology if they are on a deadline. You can show them new tools but you don’t want to tear the whole enterprise down just to bring it up and then be losing time.
Sometimes people tell me “should I start the project in Erlang?” And I ask them “what kind of language does your team know?” And they know Ruby or they know JavaScript or they know Python and they don’t have much time so no just write it in the language that you know. You’re better off sometimes having an adequate solution in the language you know and know how to maintain rather than a good one that you cannot work with because you don’t know how it works.

Chad: That’s great, well listen thanks so much Fred, I really appreciate it.

Fred: You mentioned a few questions in IRC.

Chad: Well, yeah the last thing I was going to ask was on IRC. You spent a lot of time on there and you’re a really valuable resource. Is there anything you want to mention about that?

Fred: I would say to people is if you need Erlang help it’s really the first place to go to get any help quickly. For quick help I would say IRC first and when you can’t get the answer for IRC usually the mailing list will have all the Erlang old guards. The really smart people, the people from Ericsson who maintain it and know all of the ins and outs of the language are going to be hanging on the mailing list. And sometimes people on IRC say “well we don’t have the mailing list” but IRC’s definitely becoming a very neat place to ask for help. I think it’s progressively taking sites like “Trafficzip” and that kind of thing. Which had a bunch of recipes but then if one of the recipes is not there then you head to the mailing list right now people can head to IRC and just ask for help directly there. There is almost someone 24/7 there just lurking. You may wait for half an hour but usually you might have a response there.

Chad: Thanks and Cheers.

Fred: Cheers.

One Day Erlang Conference in Washington DC – December 3rd

The Erlang DC Conference is early December! There is a great list of speakers already and they are looking for additional speakers. Sponsors include Basho, AOL, and Erlang Solutions. They’re on twitter at @erlangdc, and the site is at erlangdc.com

Erlang User Conference 3-4 Nov

The EUC in Stockholm is in full effect this week – keep an eye on twitter #erlang and #euc for news.

Scaling Erlang

Good write up by Inaka’s Fernando Benavides on Scaling Erlang Web Applications; follow along as he takes a sample app from 1000 to 100,000 concurrent users.

Spawnfest 2011 in 15 days – “Rails Rumble” of the Erlang world – 48 hour contest to make the best Erlang powered app

Spawnfest is the annual 48 hour Erlang development competition where you can compete against other developers in building the best, most breathtaking Erlang application, library or anything you like! It’s July 9th and 10th anywhere in the world.

They say, “We hope Spawnfest will help bring the Erlang developer community together and get people interested in our favorite programming language.”

We (Inaka) are planning on sponsoring a team and are getting excited about the competition.

If you’ve seen Rails Rumbles of years past, the amount that a talented team can accomplish in one weekend is absolutely astounding. What will the teams at Spawnfest accomplish? How many teams have signed up? Will they work in a massively parallel manner? We’ll have more in the next few days about the event.

Follow them @spawnfest

Motivated Reasoning and Erlang vs Python vs Node

A comparison between Misultin, Mochiweb, Cowboy, Node.JS, and Tornado. Tests such as this one tend to focus on either raw parsing speed or total number of concurrent connections, and which is more important depends on the ultimate application. Or they perfectly tune the tester’s favorite framework and use a stock configuration for the other competitors. For instance this test doesn’t use multiple cores (which is one of the points of Erlang) and the test server was running locally on the same box as the test client, which could affect the results in unexpected ways. Ironically, Erlang often does poorly in this style of benchmark as it’s not known as being good at raw language performance and even worse at string parsing. However in this case it comes out on top… and would have been much stronger had it been able to run on multiple cores.

Framework benchmarks are a joy to me for a different reason, however. In theory, dispassionate analysis should prevail in benchmark blog posts. In this case Roberto seems to do a good job of testing even-handedly. But what exactly is the goal of the test? Faster ≠ better; and Erlang is rarely called fast. Pretty ≠ better; and Erlang is never the belle at the ball. Would Roberto have published the results if Misultin were slower? I’m not implying he wouldn’t, but (and this is not to pick on him), I’ve never seen a framework battle royale where the author of one of the contenders published the results and their framework didn’t win. More informed readers please enlighten me if there are obvious examples.

Incidentally, one of the things I found fascinating about Tim Bray’s Wide Finder was that he didn’t really have a dog in that fight… so the results seemed…objective. I can’t fault his motives even though I can still wonder what exactly was the point of the project.

Taking a step back, I am always struck by the comment threads below these posts. They remind me that most people deal with insecurity; most people self-identify with groups for (often) arbitrary reasons, and most pick a group and then reason backwards as to why they’re a member of that group or school or team or church. Their motivations are mostly unclear internally, much less to others around them. This is so common in the software world – see my recent interview with Tino Bredden on Erlang other communities where we discuss some of this. Incidentally, I could easily see myself running a blog on programming sociology (though I’m not a socialogist), because I find group behavior, particularly group geek behavior (as I am one), fascinating.

We are all somewhat impervious to new information, preferring the beliefs in which we are already invested. We often ignore new contradictory information, actively argue against it or discount its source, all in an effort to maintain existing evaluations. Reasoning away contradictions this way is psychologically easier than revising our feelings. In this sense, our emotions color how we perceive “facts.” – David P. Redlawsk

What’s the point? In this project, Erlang seemed faster, uglier (according to some), and had one hand tied behind it’s back, as it wasn’t allowed to use multiple cores. Does that make me feel happy or sad? Does that make me feel like I need to justify my platform or library? Do I feel myself engaging in Motivated Reasoning? Or should I take a step back and realize it probably doesn’t matter either way…and that my response to the test says more about me than about my choice of framework.

Erlang in Haskell

Haskellers who want to experiment with the concepts of Erlang in Haskell can now do so with an experimental Erlang-like distributed computing framework called ‘remote’. It’s quite basic but covers concurrency, selective receive of messages, local and global registration of processes, and trapping exits. Related Microsoft Research Paper.

Memory Models in Erlang vs Java

A view of Erlang (focused on the memory model) from a “Java Code Geek” – http://www.javacodegeeks.com/2011/04/erlang-vs-java-memory-architecture.html

Interview with Tino Breddin, Embedded Erlang developer, on the social science of tech communities.

Why are communities such as Erlang’s and Ruby’s so different? What makes them approchable or not? Is America the problem? Should we forcibly relocate all Ruby developers to Sweden? Tino talks about that question and others in this fascinating interview.

Follow Tino on twitter at @tolbrino and follow me at @chaddepue. Get an alert when we publish a new story by following @erlang_inside.

Tino Breddin is a Systems Engineer at Erlang Solutions Ltd where he spends quality time on building scalable, highly-reliable systems for messaging and data storage. When not using Erlang he prefers using Python or Ruby for anything which needs to be automated. Previously Tino worked at the research labs of SAP Labs LLC in Palo Alto and SAP AG in Dresden focusing on massively scalable systems development. We talked over skype two weeks ago about his experience with Erlang and his talk at the San Francisco Bay Erlang conference starting next week.

Chad (Erlang Inside): Well thanks for the call… So are you based in Sweden?

Tino: No, I’m based in Germany, in fact I’m the only one.

Chad: So, how long have you worked for Erlang Solutions?

Tino: I started one and a half years ago working for Erlang Solutions and before that I worked for SAP, one of the larger German companies, so I used to live in California. I kind of moved from Java to Erlang. In the beginning when I worked at SAP there was a division called SAP research. They were focused on what the company would need in five years in terms of technology or systems. As part of one project which I worked on in California, I was looking at alternative technologies, and before I joined my team lead picked Erlang for our project… I think because he met Francesco [ed - Founder of Erlang Solutions] at some point as the technology fit the research goal of the project, so we started using it in a very niche project at SAP research (nobody else was using it) and we tried to advertise it at the company as well.

Chad: How did evangelizing Erlang inside SAP go?

Tino: That’s always a challenge, I suppose. I don’t know whether you’ve been in this situation, but it’s similar to trying to get people to use other niche technologies such as Haskell. If you try to advertise such a technology from a tech point of view, most managers will say, “let’s stick to the old stuff which has been proven and we know how it works”. So we quickly realized that it was a dead end to try to advertise the language itself or even OTP, so in the end we just decided to try to advertise the results, which I think were awesome. We made a lot of small prototypes which showed that the technology was superior to what was used before. This was something people could understand, see it in the numbers and metrics.

Chad: Tell me a bit more about the research you did and specifics on the projects you were working on?

Tino: I was focused on small prototypes around getting messages, evaluating the type of message and routing to some destination or creating new messages. These systems were meant to be very big, scalable, and integrated with other systems, and Erlang was a good fit. I also worked on some data processing with huge amounts of data and that’s also where Erlang came into play.

Chad: You said you did quite a bit of Java and Python in the past. How do you feel Erlang compares and contrasts to those languages?

Tino: Just as a recap I used Java a lot before I started to work on Erlang fulltime, mainly because it was the language we used at work for various projects. I did and do use python and ruby a lot fo anything else – more scripting, deployment scripts, etc. and I think for me personally those two languages have a nice style that fit in really well. I think that Erlang is really good for anything I need to do in terms of backend processing, whether it is some system which is TCP based and needs to route messages or some data processing system; whereas Ruby and Python are in my opinion are really strong in the web front and so if I had to do a web application I would still use one of those languages – Django or Rails.

Tino: For Java, I think it falls into the same kind of category as Erlang. I wouldn’t use it for web front end stuff. It has it’s advantages (or at least the Java Virtual Machine has it’s advantages over Erlang and the BEAM machine in certain respects – though the BEAM has advantages as well), so I think one should really evaluate the use case, and I wouldn’t say Erlang is always the better choice. And that’s also where I think technologies like Erjang will really change the picture of how people think of Erlang as a language

Chad: Yes. Have you used Erjang at all? Last year at the conference in San Francisco I met Kresten and saw him working on it. It was pretty early but I’m curious if people are using it now.

Tino: Well I don’t know of any production use of Erjang but I spoke to one of the core committers to it a few weeks ago – a guy from Germany as well and he said that they are at a stage now where they’re trying to get a really cool use case going or at least show some unique usage of mature Erlang frameworks such as Riak for instance but I doin’t thin there’s anything in production yet that uses Erjang.

Chad: What do you think about Reia since you have an interest and background in scripting languages?

Tino: Well I think that there are two sides of the coin (at least) in my perception. I think it’s cool that people are building things on top of BEAM, which has a different appeal to people in terms of syntax and semantics, but I personally don’t feel attracted to it. I quickly got over the point where the language syntax of Erlang annoyed me; that’s not a problem anymore, so I pretty much stick to Erlang. So I haven’t even tried Reia at this point.

Chad: To me the syntax of Erlang is something I actually like about it once you get used to it.

Tino: I think you are right, it’s because it’s different. When you think about other languages they’re often very similar. I feel at least that when I switch from one language to the other I always mix certain syntactical aspects. But when you switch form Erlang to python you won’t mix syntactic aspects because the two are so different. I see that as a strength.

Chad: I haven’t met a lot of people that are excited about scripting languages. We have some Erlang developers here and I keep threatening people here that we are going to use Reia on a project, and they’re not happy with me. There is part of me that’s a language evangelist that likes Reia – it’s easier to convince people to use a Ruby like scripting language that runs on BEAM than it is to get them into Erlang. My sense that there may be people that will find over time that like it better, so it probably just needs some time and attention.

Chad: Let’s talk about your talk. I know you were involved in continuous integration, but what’s your talk on?

Tino: I mainly want to talk about the ecosystem around Erlang.

Chad: Oh, ok, great.

Tino: You are also involved with ruby right?

Chad: Yes

Tino: You can probably understand my reasoning. In my opinion the various communities around Ruby and Python, Erlang and other languages, are very different. That stems from how they evolved and where they came from and where they are going. In my opinion, the Erlang community has really made a big effort and big jump over the last two years, and I just want to explain how that ecosystem has evolved up until now, and where I think it’s headed over the next two years, and how that compares to other communities, and what benefit that brings to the user.

Chad: And when you say how that compares to other communities, do you mean from mostly a technology perspective or from a personal and group dynamics per? Or both?

Tino: I’m more focused on the group dynamics and what the relationship between contributors and users is and how easy it is to get in touch and kind of be a part of the community. So from a technology point of view I think that the communities are very similar. There might be some different process in place but overall they’re similar.

Tino: I really want to focus on how people can get into the community. you know when you start using Erlang it might feel like the community is small, you know it only has one community website and one mailing list. But if you dig a little bit deeper, it might look from the outside, a bit small, but there is a lot of knowledge and a lot of people involved in it so you can get a lot out of the community if you really want to.

Chad: Right

Tino: So I kind of talked a little bit about that at the Erlang Factory Lite in Munich which we do whenever there is some interest. That was a small meeting for 4-5 hours where I talked about the ecosystem and the response was really good. People who were mostly new to Erlang really liked this overview as they instantly saw how the whole dynamic worked and they saw how to get information and how they can actually get in touch with people.

Chad: That sounds like a great idea. This is actually a pet topic of mine, so I’m really interested to hear the talk.

Tino: Well you’re doing kind of the same thing with your website right?

Chad: That’s really the goal. As we’ve gotten bigger with Inaka we haven’t had a lot of time to focus on it and update the website but we hope to change that this year. But my goal has been to evangelise a platform and a language to people that aren’t familiar with it. Which is kind of why often with my questions I do bring up scripting languages and I ask questions that a lot of the more “hard core” Erlang guys are asking “what? why are you bring that up?” and they’re just uninterested in a more “web perspective” where as I feel like if you’re talking to the average ruby or python developer (perhaps not quite as much but to some extent), you’re going to be talking to people building websites and doing Facebook integrations, etc, and so that’s the commercial reality of software development for a lot of developers, and so I think it’s important to talk about Erlang from this perspective.

Chad: So I don’t want to get into too much on your talk but why is it that, for instance with Ruby it’s full of these sort of “rock star” guys with near cult-like followings and Erlang is basically a lot of serious legends but the general feeling is that they’re quite down to earth and very easy to get along with and very open. Do you have a theory on why that is?

Tino: I guess everybody has a theory about that but also I think that the people who are attracted to ruby, at least some of them or a significant part of the community are more opinionated. they do like to go out and shout that Ruby is the best thing that ever happened to software development. And that’s not how Erlang people operate, at least the “legends” you were referring to.

Tino: Part of that might also be the origin of most of the people. Swedes are not the most opinionated or loud people out there so whereas not to be offensive but people from the US are more open and like to talk (which is not a bad thing)…

Chad: Hah Right…The Benjamin Franklin style self promotion.

Tino: and that’s the difference…

Chad: Yeah but Ruby is Japanese… and frankly Rails was invented by a Dane… so they took root outside the domain of their creators.

Tino: That’s true but he [ed - DHH, Rails founder] moved to Chicago and the inventor is very opinionated as well and he kick started the community there. Whereas the community around Erlang started and still is focused around Sweden. So now there are more and more people in the US getting interested in Erlang and you instantly see that there is more and more hype generated as people in the US take it up. They are more open and talk more about their opinions, at least on the web, and you see companies like Basho and the guys around OTP in Action. You have the Erlang Cam…

Chad: What’s that?

Tino: Erlang Camp.

Chad: Oh I thought you were referring to some camera that, perhaps, followed Joe Armstrong around or something. Eric and Martin from OTP in Action put Erlang Camp together. Got it.

Tino: It feels like when the Ruby community got started and was very open and going out and saying they’re a new technology and then new frameworks came and it now feels like Erlang is moving in the same direction. Which is great because it means that more people are learning the technology and are getting to have a choice.

Chad: Yes. That’s really cool. I agree with what you are saying though I think it’s a mystery to some extent why Ruby went the way it did; I have more Ruby developers currently than Erlang here but we are slowly converting them… We often have projects where we use Ruby on the front end or for admin and we use Erlang on the back end.

Tino: And I think that’s the perfect use case for the two technologies.

Chad: Have you done any of that type of work where you’re mixing those languages or Python?

Tino: Oh yeah I’m trying to evangelize this kind of use case and setup all the time.

Chad: When you do, how do you tell people that you communicate between them, technically? Do you make Python look like an Erlang node or HTTP or BERT and Ernie?

Tino: It really depends on the use case. Generally HTTP though there are times we use the database and have some kind of notification mechanism to tell the Erlang subsystem or backend that something has changed. But most often I stick to HTTP because it’s just so simple and any component can talk HTTP.

Chad: Any other questions we should discuss?

Tino: Well you asked me one here we didn’t answer – what are some underrated technologies in Erlang? I think there are two kinds of people. #1, the kind who likes to use print statements and when there is a bug try to narrow it down some way or another with them, and #2 the others who try to debug, and thats where tracer comes in really handy, especially when you’re running a distributed system, because then printing out values just doesn’t make any sense anymore – it just becomes too hard to monitor a distributed system in that respect. So tracer and the dbg module are things which have really bad documentation but should be used more. And I think they’re one of those things where you really need somebody to get you started. You can just sit next to someone and watch how they use it and therefore learn from them. dbg is one of those system tools that is just so well hidden that way too few people use it.

Chad: Yeah I used it once and it was useful and then every time I need to use it again I cannot remember how to use it so I have to start over …

Tino: The function names are nice abbreviations…

Tino: So the second one is kind of a my colleague Ulf Weiger is one of the more experienced Erlang users I would say…

Chad: Laughs

Tino: And a couple of projects where we were thinking about how to store data, most of the time we end up using Mnesia. But the public perception of mnesia – it doesn’t have a good reputation, it’s documentation is sub-par and it’s really hard to figure out how it does things because it’s fairly complicated. I usually use the comparison where when you think about a database, when you want to store data you use the database, and it’s a black box. you don’t start digging around in the storage subsystem of mysql, whereas with mnesia its just a component which you use to create a database system. It’s really an embedded component so you have to work hard how to wrap you head around how you would use mnesia.. and that’s the hard part. So I hope someone will write a book about it and people will see the light and use it to go on and do more cool stuff.

Chad: That’s interesting. I’ve had trouble with mnesia, we’ve used it for a few projects. the problem would be we would have a small database and maybe we would shut the node down poorly and then it would take 25 minutes to boot the server and it wasn’t that much data. and we kind of said to ourselves, if we have that problem with test data, in production where we gather about 1GB of data a day of twitter data, which frankly isn’t that much.. There were just too many warts related to startup time and maintenance. You kind of have to be a db expert to run it, so I’m curious do you feel the same thing or do you often feel like you’re working with people who can deal with the ‘warts’.

Tino: No I think you’re right in that you need a little bit more expertise in terms of database systems, and how do you architect your database. We often use it so I have a good feeling about using mnesia but I have the same observation as you do – it’s not always a good fit. Especially when you get started for the first or second time, you need a lot of time to figure out how the little features fit together and shouldn’t be afraid to just dig down into the code because this is for the most part how I learned how mnesia works. And this is just natural because it’s a component which you need to embed and add your code on top of it in order to make it really useful for you. So out of the box it most often doesn’t fit into any system – you have to engineer some layer on top of it to make it do what you need it to do. And to do just exactly that thing we need to step up and create better documentation and a better awareness of the technology and I think Ulf does a good job evangelizing the technology. So hopefully at some point he gets a chance to write a book about it.

Chad: That would be a useful book. as the Erlang community grows I think you’re finding that there is now room for specialty books that tehre wasn’t before – the OTP book is a good example.

Chad: OK that’s really helpful and a good interview.

Tino: Well, you said you have some Ruby devs being converted to Erlang devs – that probably already makes you the largest Erlang development shop in South America.

Chad: Hah well there used to be about 10 developers at a company here that only did Erlang and for some reason they disbanded. One of those guys is here so we often joke that there are 9 former Erlang samurai warriors wandering the desert of Argentina like “Ronin”, and we must find them. So there are some Erlang guys here, and a lot of shops that are using Erlang. Hopefully we’ll have enough projects to round all of them up.

Chad: Well thanks for the interview – I’m really looking to seeing you in San Francisco in a few weeks!

Interview with Kostis Sagonas – On Erlang tools, type systems, and how HiPE compares to JIT

I’m pleased that we have a fascinating interview with Kostis Sagonas – leader of the HiPE team, one of the creators of Dialyzer, and a speaker at the SF Bay Erlang Factory… This is a bit longer than normal but absolutely worth the read – we cover Erlang’s unique type system, Java’s JIT vs Erlang/OTP’s HiPE, tools such as Dialyzer, and why we will have a future post on Language Creators who have never used a debugger in their language. As usual I transcribed a Skype call so feel free to submit errors and I’ll correct them in the original for the next day or so.

EI: Maybe it’s a good place to start to just have you go over a quick background on yourself. And, I know your history just from looking at your personal website, but anything that you wanted to kind of say, particularly about how you got into Erlang would be really helpful.

Kostis Sagonas: Ok. So, I’m in academia and I’ve been in academia I guess for 20 years now. So when I was an undergraduate student I got fascinated by programming languages and Prolog. It was pretty hot at that time so I chose to do my thesis work, PhD work, on implementing extensions of the Warren Abstract Machine for Prolog.

EI: Ok.

Kostis Sagonas: So, I was very interested in the implementation of high level languages, the declarative languages. So then I found a position in Sweden as an assistant professor then. There was a very exciting project here in the department on, during the high performance Erlang compiler. So, since my interest was in implementing declarative languages and Erlang had a lot of similarities with Prolog I said, “This is for me,” so, I led that group.

EI: What do you like about Erlang? What, what was interesting to you about it, originally?

Kostis Sagonas: It was interesting because it was, it was a language that had potential to succeed in some areas where Prolog did not materialize its full potential. So, it was, a very down-to-Earth language, it, even back then had an interesting user base in industry. It was “for real,” so to say.

EI: Right.

Kostis Sagonas: And well, there was local industry supporting it and, we seemed to have an implementation that was the most advanced implementation apparently at the time. The compiler back then wasn’t really mature, but we could with effort become part of the Erlang distribution. I think HiPE has been part of Erlang/OTP since 2001 or 2.

EI: Did you have a hard time convincing Ericsson, at the time, to add HiPE to, to the language or to the platform, or? What was the response when it was suggested to add it?

Kostis Sagonas: So, I think Ericsson has been very, very positive for what we are doing at Uppsala university from the start. So they have been extremely supportive. On the other hand, I think that they are a bit, I think that the word is ‘conservative’ since they do not have the expertise to, be supporting the compiler they were a bit reluctant to announce it or to announce it as a supported component to their customers. They were extremely happy having it, HiPE the compiler in the distribution as an experimental addition at first, and then as a component that, is on the same status as many other applications that exist in OTP.

Kostis Sagonas: I think that they are still a bit reluctant to say that they’re, that’s supported. Basically because they don’t have the expertise or anybody working at OTP who understands the guts of the compiler.

Kostis Sagonas: The OTP team has been supportive from the start and for the last 3 years now. The HiPE compiler is being used more and more. It’s actually being used every time you use dialyzer for a big set of files. It automatically compiles a lot of key files for dialyzer to native code. There have been a lot of projects that I know that are using HiPE. Especially in France, they seem to be an active community there that is using HiPE all the time. So it’s, it’s I think has come to the point that it’s mature enough to be used in many applications that require the extra speed that HiPE gives them.

EI: Right. Well, it seems it’s become quite popular just from talking to people in the community. Also I have a few developers at Inaka (which is my consulting company) and they’ve been quite happy with it in terms of performance. So, I’m curious, when you started HiPE, was this was sort of an academic project, was it sponsored by Ericsson or was there a connection to Ericsson, or was it totally separate research project only?

Kostis Sagonas: So, when I started HiPE was in a bit of a, how do I describe this? So, there were two students at Uppsala University developing HiPE as their master thesis. Then it was a bit left on its own in some sense. Basically because the first, when I came in, the version of HiPE we had was based on JAM [ed- Joe's Abstract Machine]. This was at exactly the time that JAM was being obsoleted, taken out from the Erlang/OTP distribution, and BEAM was left as the only machine. So it was a time that BEAM started being the default, and probably the only machine back then. So I realized the danger there. We were going to be left a bit behind and so the first thing I did was to actually write the translation from BEAM to HiPE’s intermediate code presentation. It was actually my second program in Erlang I’ve ever written. The first one was – it’s still in the distribution, by the way, the first one – is the BEAM disassembler. It was really, really fun to write that program. It is still in a file (called beam_disasm.erl) under the compiler application

EI: So most people’s first program is factorial or something. So that’s that’s impressive. (laughs)

Kostis Sagonas: So it was even more impressive because there was no documentation about the BEAM formats

Kostis Sagonas: So, I was just guessing what Erlang’s byte codes could possibly be.

Kostis Sagonas: So it was really fun, writing that program. And you can still see my code. I think it’s still pretty good for being my first Erlang program!

Kostis Sagonas: The language, that I actually had no—I never read an Erlang book, believe it or not.

Kostis Sagonas: So, I picked it up from just programming.

EI: Actually, I don’t know when the first Erlang book came out because the Pragmatic Programmer’s was quite a bit after that.

Kostis Sagonas: Yes, yea, well, it, there had been an Erlang book back then, the first Erlang book, but it was very hard to find, even still back then.

Kostis Sagonas: So, yeah-

EI: That’s funny. Actually just a couple more questions on HiPE. Did you, so it currently-

Kostis Sagonas: Sorry, sorry, I did not actually answer your original question. So—

EI: Yes—

Kostis Sagonas: So, actually, we the university and Ericsson were part of the same project, sponsored actually by research foundations back in Sweden, that was in Sweden at that time, called, NUTEK [ed: currently VINNOVA]. NUTEK was a foundation that was giving money to encourage collaboration between academia and industry. So most of the sponsoring of the HiPE system, came from NUTEK. Now there were some matching funds from Ericsson, but I think it was a small percentage in actual money. But Ericsson was actually sponsoring HiPE back then.

EI: So, that’s helpful, actually. The next question I have is that I’m sort of curious about the design decisions made, with HiPE. I don’t know a lot about compilation to native code from bytecode –

Kostis Sagonas: Don’t worry, just shoot the question and, we’ll correct it.

EI: Well, I’m curious why you decided to use an intermediate format versus going directly to the native code and could you explain that and the reasons behind it? Because, you already had a virtual machine format so it’s kind of interesting.

Kostis Sagonas: So, we actually use two different formats inside HiPE. There is an intermediate code presentation which is much closer to Erlang and then there is another RTL—so-called Register Transfer Language—which is quite similar to GCC’s RTL, but not the same. So the, need for—the reason for these two representations has to do with the difference in the representation for Erlang terms in the intermediate code format all the terms are just like Erlang terms—all the variables contain Erlang terms. In the RTL we use the explicit representation in memory, so they are tagged by use.

Kostis Sagonas: So technically that’s the big difference between these representations. So we start from the BEAM format which we just translate to some representation of our own. Mostly because we want to be doing some optimizations there. Now, BEAM does some of its optimizations while it compiles, and then it does some very heavy optimization when it loads the file.

EI: Okay

Kostis Sagonas: So you see the BEAM loader it’s actually a Turing complete piece of software there. It does magic when it loads the file.

EI: Right. And, actually I saw a post by you on the Erlang questions forum where you— someone asked you about— because I was going to ask you about this – LLVM and is that kind of a future direction and then I saw you wrote a response to someone where you mentioned that the BEAM loader is doing quite a bit of magic and that affects the reasons why you would use a particular format or not. So are there sort of design decisions, with the overall Erlang platform that—maybe for historical reasons—make the job of the HiPE compiler much more difficult than for instance something like the Java JIT.

Kostis Sagonas: Absolutely. Absolutely. So, there has been— so Erlang file use certain properties in writing Erlang programs more than others. So, one of them is this ability to do change the code at any point. This basically means that it’s very difficult to do global optimizations. To do inter-module optimizations. Even across functions it’s pretty difficult to do this sort of thing because you need mechanism also to undo this optimizations if a new version of the module gets loaded which violates the assumption you had before.

EI: Wow. Okay.

Kostis Sagonas: Okay, so you need this. It doesn’t mean it’s impossible. Just means that it’s very very difficult.

EI: Right and how much of that is related to the thing that you just said in terms of optimizations is related to the lack of a static type system per se or—

Kostis Sagonas: It’s not so much the lack of the static type system is the ability to then load something which is completely different that what you had before.

Kostis Sagonas: Now, the—of course if we had the perfect type information, a very strict type system, we could take advantage of it. But actually I don’t think even in this case we would be very, very careful because you could always load a new version of the module that violated the assumptions when you made the optimization. So, they value this sort of thing much more than some possible performance improvement that you could get.

EI: Okay, yeah, that’s very interesting.

Kostis Sagonas: Yes. So, I always thought it was a bit weird to be loading just one module at a time. And one of the things that I many times thought was the ability to declare a set of modules as an application and then do this whole code loading at the application level as opposed to at the just-one-module level.

Kostis Sagonas: And my understand is that most projects out there actually do it this way. They don’t just update a single module, they update a set of modules. But, in the language, actually, you can just update a single module.

EI: Okay

Kostis Sagonas: I think it’s a remnant from the time where machines were slower and it took considerably longer to update a whole set of modules.

EI: I see, that makes sense. It’s more a historical artifact than it is—

Kostis Sagonas: I think it is more historical artifact. If one designed a new Erlang right now, I think it would make more sense to declare— to have a way of declaring certain things as an application and then loading them all at the same time.

EI: Right. What else would you change if you were to build New Erlang today? Kind of keeping in mind the similar goals of the immutability and message passing and some of the robustness of Erlang. What would you change with those constraints?

Kostis Sagonas: It’s a very interesting question. So, I think I’m a guy that likes to do small changes at a time, as opposed to doing grand changes to the language. One thing that I would definitely do is I would be more aggressive in taking out certain remnants of the past from Erlang which are ugly; You carry a lot of legacy code and just support weird constructs that were added to the language once-upon-a-time and by now there are better ways of doing these things. For example there is still support for calling tuples of pairs of module name and a function name and calling that sort of thing with some arguments. This is an remnant of the period before Funs existed in the language. It’s still supported.

EI: I see

Kostis Sagonas: There are some other examples I can give you where some things that were basically a mistake are still part of the language.

EI: Interesting. So you would get rid of the module-function-argument syntax and just pass functions, as one example.

Kostis Sagonas: Actually, it’s not the M/F/A syntax. It’s the module-function which you can then use as a Fun. Very few programmers use this. But, it’s still part of the language.

EI: Right.

Kostis Sagonas: This is just an example, of course, okay? There are other things that are remnants of the past and they they don’t — they’re not removed from the language due to backwards-compatibility arguments.

EI: Right. Related to that — and I ask this question often of people — what do you think about the record syntax and would you change that? Do you think it’s a problem in the language?

Kostis Sagonas: I haven’t been bothered by the record syntax so much as other people. Basically because I’m using records in a very disciplined way. So, these days whenever I go to use a record for some data structure I try to create a abstract data type, a module, that basically has a, in the Java jargon, getters and setters. And these are the functions that get exported from the ADT module. I even declare these data type as opaque because there is a type syntax for doing that sort of thing. And then I have dialyzer check that I’m not violating the opacity of this term anywhere else. So, records are great if you use them in this way. They are pretty nasty if you try to mix tuples where you explicitly pattern match on the representation, or use element, which is really really bad.

EI: Right.

Kostis Sagonas: That’s really the problem.

EI: Very interesting. So, how often do you use parameterized modules, and what do you think about those? And I actually specifically asking because one of my developers asked me, he said “oh you’re going to talk to Kostis, you should ask him about why dialyzer doesn’t support paramaterized modules.”

Kostis Sagonas: Actually, I think this is wrong. Dialyzer supports paramaterized modules. So we have support on this since, oh, two years ago?

EI: Oh, interesting, ’cause he told me he had some problems recently using it, so I’ll have to ask him—

Kostis Sagonas: Now, it’s conceivable that there are some problems. But, in principle there is support for this.

EI: Okay

Kostis Sagonas: Now, there might be some bugs, obviously we have not verified dialyzer, it’s a very complex program to do that sort of thing. But, if there are any issues I think he should just send a bug report. There is support for that sort of thing. Now, I think parameterized modules are an interesting addition. People especially coming from a more object-oriented background, they find them are more familiar to them, the whole idea is more familiar to them. Personally, I’ve not written anything that used paramaterized modules, ever. I didn’t really feel the need for it. But, I think it’s a bit of a chicken and egg problem. As long as there are not that many users the Erlang/OTP Team is not so keen in making this official and as long as it’s not official there’s very little incentive to change all that to understand parameterized modules, which makes them even less popular, and you see that this is a bit of a vicious circle.

EI: Yes, right.

Kostis Sagonas: But I actually have no strong opinion either in favor or against them.

EI: I found often Erlang developers are quite opinionated about them and don’t like them, and it’s very interesting. Well, let’s talk really quickly about a couple of HiPE questions and I want to talk a bit about dialyzer a bit more, and then talk about your talk at Erlang Factory and then we will be done. With HiPe I was wondering if you could talk a bit about comparing it to the JVM, to the JIT in the JVM and I don’t know if you saw my question three here about sort of, you know, how it compares with regard to the idea that there are these thousands of man-hours going into the Java HotSpot VM. Do you have any thoughts on that and how HiPE compares and how it fits into the world of bytecode compilers?

Kostis Sagonas: So. At the end of the day the only reason to do native code compilation is to get performance. So, the performance that HiPE gives you is not limited due to the fact that we’ve not spent thousands of man-years to do optimizations. But it’s constrained by certain decisions at the language level.

Kostis Sagonas: It’s also constrained by certain decisions that the HiPE compiler has done. In HiPE you can take a single function and compile only that to native code. Now, obviously there is very little that you can do in these situations since you do not have global information to do global optimizations. So it’s actually this problem that doesn’t give performance that is say, an order of magnitude better than the BEAM compiler. On the other hand I’ve not been impressed with the speed-ups that the JIT or HotSpot Java implementations achieved. My experience is that they typically get between three and four times speed-up over the JVM code. And, it’s about the same speed-up that one gets from HiPE. So, if I have to code a number — now obviously it depends on the application but we get something like two and a half to three and a half times better speed than BEAM and BEAM is actually very well-engineered. So, I don’t think that by going to something like the JVM that you get so much improvement in performance. So yes, it may be many many thousands of man-years there, but I don’t think they were very well spent. I think Erlang/OTP has achieved that with fewer man-years.

EI: Right. That’s very interesting. So what’s the future for HiPE? Is it kind of done? I know it’s not your focus now.

Kostis Sagonas: The last five years it hasn’t been my main focus. But the things are slightly changing. Perhaps I shouldn’t be announcing this right now, but I do have a project with two students of mine and a colleague to have an LLVM backend for HiPE. The compilation through LLVM. Now, it’s very early in the process to say anything more than that. But, we’ll see what speed-up we get.

EI: Oh, interesting.

Kostis Sagonas: So I hope I will have something more complete to say on this, perhaps in the London Factory. It will be very early to say something about it in San Francisco.

EI: That’s very interesting. Yeah, I saw in that post that you mentioned you had a few students researching it but then it sounded like they didn’t have time before they had to move on.

Kostis Sagonas: Yeah, but it was also two years ago and LLVM was not so mature back then. So understanding is that currently there is a lot of effort around LLVM. And, we’ll see how this turns out.

EI: So, let’s talk then quickly about dialyzer. Do you find people reacting negatively because it sort of bolts a type system on top of Erlang, or do you find people like it? What’s the feeling in the community about it?

Kostis Sagonas: I think you should ask somebody else about the feeling of the community but I think…

EI: (laughs)

Kostis Sagonas: —because I’m obviously biased here. But I think dialyzer is an extremely cool idea. I’m obviously biased, I have to say this, but personally I do not see and disadvantage whatsoever in dialyzer.

Kostis Sagonas: Because, it’s actually not right that it imposes a type system on top of a language. On the contrary, dialyzer can be used without any type annotations whatsoever. So you can use dialyzer with your program as you wrote it, without a single type declaration or a single spec, and it will find bugs in your program just by typing dialyzer and giving it filenames afterwards. So I don’t really see any reason not to use it. Now, if you want to get the most out of dialyzer then you will have to supply some information, but you can supply as much information as you want or feel like. My personal view is that the types are very good for documentation purposes. So, I think there is an advantage in using types for catching some programming errors. What most people don’t realize is that you need a different type system for doing optimizations and for catching programming errors, and Dialyzer has taken the approach of using a type system only to catch programming errors. So the types that you can write are very very relaxed. Actually they are types that you cannot write in any other statically typed language that I know. You cannot say that something has a type of “any” in most other languages. So—

EI: Hmm, interesting. So it’s different than saying a type like a variant or Objective-C id. It’s essentially saying it can truly be any type.

Kostis Sagonas: Yes, it can truly be any type. Or you can have arbitrary unions. You can mix without using any constructors. You can mix integers and floats. Integers and lists. Anything basically. So—

EI: That’s a good point. That’s quite different than any other types system I’ve ever used.

Kostis Sagonas: It’s very very different than any other type system out there. And it’s actually something that I’ve not seen—the approach is something that I’ve not seen anywhere else. So it’s fundamentally a different approach than anything that has been tried out there for adding types to a dynamically typed language.

EI: Can you quickly explain for a non-academic what “success type” is and how that’s different than a normal type in another system?

Kostis Sagonas: So, success typings try to approximate the set of values for which a function will return something; will not throw an exception.

EI: I see. Okay. That makes sense.

Kostis Sagonas: So, they are trying to find—So, in typical type systems—Sorry I’m a bit slow here because I try to seek how to say it without being too technical.

EI: It’s mainly that I think this is really interesting because there’s not a lot of explanation of this out on the Web, so feel free to kind of say it in more technical terms.

Kostis Sagonas: So the typical type systems want to guarantee type safety. Okay? Which means that they artificially restrict the users of functions in order to guarantee type safety. Now, Erlang is a dynamically typed language, alright, but it’s also a type-safe language. The only thing that can go wrong is that you get an exception during runtime. You are not going to get a segmentation fault because of using a wrong type. Okay? So, success typings try to find all information they can find about when a function will not throw an exception. That’s where the “success” comes from. When it will return, it will succeed in its evaluation.

EI: Okay. Perfect. Yeah, that’s helpful, and I saw at least a couple references to your paper —you wrote a paper on that, I guess, on success types. [ed - "Practical Type Inference Based on Success Typings"] and it’s on my reading list here. Where do you think dialyzer is headed? Is there any future development happening there?

Kostis Sagonas: Oh, there is a LOT of development happening there. So, we have added some components recently to detect also concurrency errors that might exist. So there is already something in the distribution that detects race conditions you may have in Erlang. And we have another analysis that finds message passing errors, in applications. But one of the nice things about success typings is that you can make them stronger and stronger and stronger in each pass and you will see that dialyzer detects more and more errors as you are using a newer OTP version. We can do that sort of thing because we can improve on the inference, we can make it more fine-grained. So we have already some future additions that detects some really really interesting errors.

Kostis Sagonas: You will see this in I think a year.

EI: So it sounds like you’re working on things that won’t go into production use for quite a long time?

Kostis Sagonas: One of the reasons for this sort of thing is that we are very conservative; so we want to guarantee that Dialyzer is never wrong; that it does not produce any false positives. So we thoroughly test every release. Also, we would like to, before we announce, before we put something in the OTP distribution, we want to eliminate all errors of that sort that exist in OTP itself. And that takes a bit of time.

EI: I see. That makes sense. So, some sort of “dog-fooding” the product before it goes out.

Kostis Sagonas: Yes

EI: That makes sense. So, anything else you want to cover on Dialyzer or HiPE?

Kostis Sagonas: There have been a lot of positive stories about Dialyzer users. I got a very very nice mail quite recently from a guy that I have a lot of respect for his programming ability. He’s working a company – I will not even mention the company. They have been using older OTP version recently and they switched to R14 just a month ago. And, he gave Dialyzer a run on his codebase. His codebase was around 50,000 lines of code and he was very positively surprised about the things that Dialyzer discovered. So, I think there is very little reason not to use Dialyzer these days, because it takes absolutely no effort from the programmer.

Kostis Sagonas: It takes effort to correct the results!

EI: That’s the thing! When we first ran Dialyzer on our product we were about to release, we found a lot of issues. So, we’ve had a lot of good results with it.

EI: Any thoughts on Reia or Effene or Lisp-Flavored-Erlang?
Kostis: I haven’t had the time to seriously look into them. I think science is (and this is not my quote but I will use it anyway) I think science — and you can extend that to programming—is like sex. It might have some practical consequences but that’s not the reason why we do it. Okay, so as long as these people have fun in developing these things, I think it’s great. So, they are interesting efforts and I think that the community should look into them. Now, my view is that not all of them will obviously survive, or they will be come so interesting that many people will be using them. But, it’s nice to be having a lot of other developers using Erlang as a target language.

EI: Right
Kostis: Now, I would have liked to see some communities around these efforts and not being a single man’s project. But I guess people are just busy doing things, not only for fun but for money. The Erlang programming community is, these days, very successful in writing real applications, and they only do things like LISP-flavored Erlang or Reia

EI: Yes, exactly, it’s a hobby.
Kostis: It’s a hobby.

EI: As a final question, I thought it was kind of funny that in last year’s Erlang Factory, Joe Armstrong said he doesn’t, he’s never, used the Erlang debugger. And, it made me think –

Kostis Sagonas: So actually, I’m in the same category. I’ve never used the Erlang debugger. I don’t know how to use the Erlang debugger. I really do not—I’ve never used this. I never felt the need in a language like Erlang, where the exceptions you get are so detailed you use anything but io:format for debugging.

EI: Interesting

Kostis Sagonas: Some of my students have used the debugger and they say it’s great. I’ve never used it. It’s amazing. So, although I don’t always agree with Joe, we are in the same team there.

EI: Okay. It feels like I should write an article called, “Erlang legends who have never used the debugger”. Or perhaps a comparison of major languages and whether or not you need to use the debugger to write in them. So, I guess related to that then, is there a tool that you think people should be using, other than Dialyzer, that they aren’t using right now?

Kostis Sagonas: So, that’s exactly what my talk was going to be about.

EI: Okay! Well, you don’t have to reveal your whole talk but—

Kostis Sagonas: No, I will not reveal my whole talk, definitely not. But I think that the era where one fired up an editor and then wrote a program, used a compiler to compile this program, and wrote some unit tests for this, is long gone. So, one needs more modern tools for development of programs. And a big part of my effort over the last seven or eight years has been to develop such tools. Dialyzer is one of these tools. I’m going to talk about half a dozen tools there, and there are some other tools that are in the pipe-line. Some of them are known to some parts of the community, some other ones will be revealed at the San Francisco Erlang Factory. I will not say more.

EI: Well that’s a good way to end the interview. I appreciate your taking the time, and I’m hoping to make it to San Francisco, so we’ll see.

Kostis Sagonas: It’s actually going to be my first time in the Erlang Factory San Francisco. I’ve been to the London. And I think it’s great because it’s going to be many more Americans than Europeans there. It’s in Silicon Valley and it will be very nice to see the interest that exists around Erlang in the Silicon Valley community. So I’m really really looking forward to that trip. It so happens that these days I’m doing a lot of traveling, and especially my March schedule is a mess. But, I’m really really looking forward to going and spending as many days as I can in the Erlang Factory and in the University in San Francisco. I think it will be a very great event.

EI: Well, last year it was amazing so I’m looking forward to it. Well, thanks a lot Kostis.

Kostis Sagonas: Okay. Sounds good.