Fred Hebert is an Erlang enthusiast based in the unexplored Northern parts of Quebec (relatively speaking). He is so enthusiastic, in fact, that he started writing Learn You Some Erlang for great good! a free online book designed to teach Erlang, functional programming and basic computer science concepts.
He started his career working with backend services for medium to large scale web sites, but php just did not do it. In 2010 Erlang Solutions came to the rescue! He is now working with course development and e-learning while remaining true to web applications, this time, written in Erlang.
Chad: Ok so, first thing I would say is welcome to Erlang Inside. So tell me a little bit about yourself and your background and how you first started programming?
Fred: Well, I didn’t want to start programming in the first place. I started studying in multi-media in the first place. I wanted to be graphic designer and one the classes we had was about programming Javascript and PHP. And so I really liking programming from that point on because I found that it had creative aspects I was looking for in a job; but also better salaries in general than designing. At least here [Canada] that’s the case. And I kind of just always liked the idea of doing the problem solving part of it and from that point on I just started studying on my own. At some point I picked up Scheme and a bunch of other languages and got a job as a web developer and working on back-end services for a dating website. And at this place I got at some point into a research and development group where my boss asked me to look into Erlang because Facebook started using Erlang around that point and I just decided “Yeah, ok I’ll look into Erlang.” And the group was kind of the funny thing because my boss-when I went on sick leave one of my co-workers went on a parental leave so with my little team I was alone for like 3 months on the same level as the executive board of members of the company. So nobody could tell me what to do I just had 3 months to look into Erlang and play with it and try a bunch of different things.
Chad: And this was in Canada or in the U.S. or where?
Fred: Oh, in Canada. So yeah, I learned it that way and at some point I was just looking for a project of some sort and decided well I’ll write a book about it. That way I will force myself to learn really everything correctly and it will be something creative. I will be able to do a little bit of web stuff that I like draw a little bit and get better at writing and speaking English basically.
Chad: That’s cool. So where are you from originally Fred, are you from Quebec?
Fred: Yeah, I’m from Quebec. I grew up speaking French so I’m a true French-Canadian there.
Chad: So basically in 3 months you started working on learning on your own and basically said hey I’m going to write a book and that book was “Learn You Some Erlang?”
Fred: Yeah, that’s about it and at some point I got bored with my job that I had and I just went on Twitter and asked Francesco [Cesarini] because I knew he wrote the book and worked for Erlang solutions. Francesco wrote at some point “We’re looking for developers in the sub-Saharan Africa” for some kind of contract and I just sent him a reply saying “If you’re looking for someone in Canada tell me.” And I got the job that way. So that’s how I really got into Erlang and working with that. It’s a bit of luck and… Yeah, a little bit of work and a little bit of luck.
Chad: That’s great, and how long have you been working for Erlang solutions then?
Fred: It’s going to be close to a year now. I started what was it, late September beginning of October maybe it was October and November last year.
Chad: And how long has “Learn You Some Erlang” been out? Well, I know it’s been a work in progress for quite awhile. But when was the first part of it released?
Fred: On Learn You Some Erlang, I think I started working on the first plans sometime in April two years ago and the first 3 chapters were published I think in July two years ago. So I’ve been working on it for more than two years and that’s probably the project that I’ve had for the longest time in my life.
Chad: Ok. Since we’re already talking about learn you some Erlang let’s kind of keep it going on that. Has anybody given you a hard time about adopting the style of Learn you some Haskell?
Fred: No, the thing is the guy from Learn you A Haskell is a guy who’s username is Bonus on the SomethingAwful forums. And he was into programming people there sometime on the same IRC channels and I talked to him and at some point he presented the book in the forums and when I was looking for ideas I’ve never written a book before or really anything this close to being that extensive. So I just looked at his book and I said “Well I could do the same for Erlang as he did for Haskell.” And I basically asked him “Would you be alright with me doing that?” And I started by just copying the basic structure of the index, introductions starting out modules and that kind of thing. Actually that really helped me a lot because all directions of where do I start and what kind of style do I take and what kind of tone do I take. I could just inspire myself from the stuff he did and he always left me a lot of liberty about that he never came and told me “Well you should change this” and that kind of stuff. He always told me that he thought it was great. It was more help than anything else just because I could steal some ideas and inspire myself.
Chad: That’s great. So is it complete or is there more to come?
Fred: Well, I signed a book deal about it so it’s going to be published at some point. So by February I’m supposed to be completely done with it.
Chad: Who did you sign the deal with if you don’t mind me asking?
Fred: It’s No Starch Press the same people who published “Learn You A Haskell.”
Chad: Oh, cool.
Fred: So what I want to cover I’m working right now on chapters on application of dates and release of dates. So app-ups and rel-ups. I want to work on ETS; I want to put a chapter on testing type checking — these kind of things. I want to do ports and sockets. What else do I want to do? I have a bunch of ideas I have something like 6 or 7 more chapters that I want to do. I still need to cover distributed Erlang and distributed applications. I’m showing releases but I’m showing no distributions as of now.
Chad: And so is ESL sponsoring this thing or is it more…
Fred: No, I always wanted to keep everything independent sometimes people offer me donations and I never wanted to take them. The reason being that as soon as someone starts paying for product, they can feel not entitled but kind of responsible for it and can ask to change the content; and I always wanted to have full control on that kind of thing until it’s done so I want to keep all sponsorship advertisements and donations out of the process.
Chad: Ok, that makes sense. So do you have any download stats? And page views? Have you had a lot of people give you feedback or thanks or what?
Fred: Let me see, I’ll just go look at the stats. But as far as feedback goes and these kinds of things I get a lot of it. Mostly positive, I guess people who don’t like it just don’t talk to me in the first place. So the general opinion that people mention to me seems to be rather positive about it. That’s the reason probably why I kept going for this one. About feedback yeah, I’m getting a few emails every week. Maybe sometimes about fixes, sometimes it’s questions about the content sometimes it’s people needing a little bit more guidance on ISU or reading a tutorial that point. So it’s really nice for that. Let me see for the stats I need to configure some stuff because I’m a bit paranoid about people sensing data on the server. The stats have been going up all the time except for the days where I will be on and I will get user peaks and I will get 20,000 new page views a day and that kind of stuff. I think for the first year the average had to be from 400 to 600 unique page views per day. Last year it went up to something like 600 or 700 and right now I think I’m going about a 1000 a day.
Chad: That’s pretty good. Well let’s talk a little bit about test driven development. Tell me your interests and what test driven development is to you. That’s a funny question but… Yeah, first like TDD at a high level and then we’ll talk about it specifically with Erlang.
Fred: I have got to say I had been doing very little test driven development before writing and teaching the course about it. So when I had to write and teach a course I had absorbed a lot of it applied to experiences I had before and that kind of thing. And this is not a problem with TDD. The problem was the fact at the places I worked before had very little tests. So when I got all this information about TDD and these kinds of things. I thought it really could really solve the problems we had many times before and it’s kind of alright to write tests after. The TDD improves you and this is one of the things I mentioned the most in my course. TDD is not about testing – its about design and then the tests are just a side effect of designing your stuff properly.
The big advantage that you have is that writing tests after the fact is pretty boring and so everyone just kind of botches up the work and nobody does it seriously when you write the tests later. So the only fun time to write tests (to me at least), is to do it when I’m creating the code because creating and writing the code is what’s fun. So if I write the tests at the same time – I’m going to write good tests. If I write them after the fact I’m going to be bored and do a crappy job in general.
The second point is that writing the test first makes me think of how am I going to use the code. What kind of functions do I want to have? What do I want the interface to be like? What kind of arguments am I going to pass in there? And this forces you to take the perspective of the user of the code rather than the perspective of the writer of the code. When you write the code you sometimes try to just make the cleanest thing that can work and then you end up having something that’s a bit more ugly when looking at it from the outside. And then sometimes you’ll have code that’s hard to test because you wrote it for it to be easy to write and read and not to be easy to use. And to me that’s the real advantage and that’s why it’s designed – you’re taking your specifications what you need to be done-“I want this feature or this and that way and that other way.” And then once you have these features and you’re thinking about that you just turn the specification into to codes to some extent and I wanted to return this kind of value and this is the structure I expect to have. Another advantage is always on re-factoring, which will mean that there’s some confidence in your code. You can change the code and be sure that to some extent it still works the way it did before. Or it’s still broken the same way it was before. Because that’s one of the risks you’re too confident in your tests and then you just jump off the bridge thinking that you’ll land fine and then you die or something like that.
Chad: So that’s what would happen to you if you don’t do test driven development. So let’s talk a little bit about how that applies to Erlang then.
Fred: Ok, well for Erlang I must say we have fantastic tools to do testing we’ve got a few as frame work common test which have pretty bad documentation but is a pretty awesome framework for testing we’ve got EUnit, we’ve got PropEr and quick check. We’ve got tools like meck which is pretty damn neat mocking tool I’ll talk more about it if we have time later. But it’s a really great tool.
In any case, for Erlang people tend to test sometimes after the fact. There’s a few reasons for this. One we’ve got the shell and people tend to test a lot of their stuff and the behaviors in the shell. You’ve got stuff that’s timing dependent and parallel code that’s hard and difficult tests. And then you’ve got somewhat of a tiny culture about tests – we don’t care that much about testing sometimes. We don’t see testing frameworks as one of the tools necessary to work with Erlang. And I mean a lot of programs work fine without it.
But, when we look at something like the OTP releases the whole code is entirely tested, but external code is often not.
Finally, processes are ‘shared nothing’, so it’s sometimes hard to test that kind of code without altering it to be able to test it in the first place. That’s one of the big issues for people having a hard time testing Erlang code.
Chad: What’s do you think is the most popular test tool for Erlang currently?
Fred: EUnit is by far the most used one I’m pretty sure of that. PropEr is getting a lot of attention since the conference is in San Francisco or London where they were announcing some features that were in there. PropEr is absolutely interesting for property-based testing. So that’s one of the strengths of Erlang there. But yeah, EUnit is most used I’d say then PropEr then I will have to say QuickCheck and maybe common tests in last and then you have test server which is kind of a low level common test off that’s going to be the least used. And then some people have their own little frameworks that they wrote out.
Chad: And do you feel like PropEr will eventually take over EUnit or do you feel like it’s more of a niche?
Fred: I’m thinking that they have different purposes. You can test a lot of stuff with EUnit you can test a lot of stuff with PropEr but some of the code doesn’t have very clearly defined properties. Sometimes you just want to test if I’m receiving true or false I’m expecting this kind of that kind of output and it’s not worth it to write sometimes a property about something that doesn’t have that much of a big space of what can be returned and what can be input into the function. I’m thinking usually the kind of stuff I like to do with proper-everything that can be defined not as an interface but as a behavior. I don’t mean OTP behavior but one the examples in my courses is when you’re writing a website or a blog you have something called a slug and the slug is that part of the URL where all the special characters are replaces by dashes and you only keep the ASCII stuff and that kind of thing. And so for that one you will have pretty clear properties you will have I’m putting in a string and I’m getting a string back. I will have only characters between this range so integers. Then I will have A to Z, “Z” (or “Zed” depending on where you’re from). And then you will have dashes but not two dashes one next to the other and then you have these kind of properties and the input you can put in the function is pretty damn wide. It can be any kind of string, you can have EUnit code stuff and you have to handle it right. And so for that kind of stuff I think property based testing is pretty fantastic. You can just generate random input get it out. Data structures are another great application because data structures will have properties when you insert or you get a tree of that shape then it’s supposed to be transformed in that matter in respect this property — everything is ordered and that order matters. So that’s great for property based testing.
But then you might be working with something that’s not as good for that. I could say I’m working testing a finite state machine. And I’m testing a function and while I’m passing the arguments for some kind of list or I’m passing a name I expect the process to be registered with that name. So at that point it’s necessarily worth it, I think, to test it with proper because a names a name and you put the properties you want and then what you want to test is the actually registration or data process that’s found and for this kind of thing I think you can feel safe enough with EUnit anytime.
Chad: So ok let’s talk a little bit about test driven development with such process and state focused applications like gen_fsm. Let me think about how to ask this question better. If I’m doing test driven development with a website it’s very easy to test a rails website. I can test my models with Test::Unit or rspec; I can test the controllers using something like Cucumber or Selenium. If I’m testing something like a gen_fsm or a gen_server – well, you know the irony of Erlang being sort of no shared state is that those things are all about shared state. They’re all about internal state – and so what I’ve found is that my EUnit tests spend a lot of time mocking the inputs and the database layer and so I spend a lot of time building a lot of time building wrappers around the gen_server or the gen_fsm. And a lot of my tests are actually testing things like timing, testing things like moving through a certain set of states and there’s a lot of complexity to that. So I’m curious how you treat doing Erlang test driven development different from say traditional web test driven development.
Fred: First off, you mentioned testing internal state and that’s something I disagree about doing. I feel it’s sometimes much better to just test from the API because the problem is that if you are testing on the internal data structures the internal state these kinds of things when you are to change your implementation of the code you need to change the tests. And the tests are kind of useless from that point on. You cannot use them to re-factor code to change how it works or to compare to what you had because it was tied how the code is built itself. So I tend to avoid and just test the interface of the module so only the exported functions these kind of things. But to get back into something like gen_fsm there are two approaches that I think work rather well. First of all you can just test with the callback functions so I’m just faking the input in the state to handle call or in that case it’s handle sync events let’s say. And so I will put the input I expect them to get I will put the current state I’m supposed to have. I will put the current state data I’m supposed to have. Then I will look at the output and all the internal messages going on. So that’s something that you can just taking the callback functions and testing what they do themselves. Something easy that you can do is use something I’ll just tie it in is the function called sys:get_status and that one is for all the OTP behaviors. When you do that, you will receive the whole data structure of the internal state of the gen_fsm server finite state machines are. So by calling this function you can say at this time currently in time this is the state of my finite state machine or gen_server. And then you can look at the internals that way without breaking the API.
Chad: Interesting.
Fred: So that’s pretty useful. A third way to do things, it’s a bit more complex because it’s playing with the global state of the VM is using tracing because in Erlang everything at all can be traced. Sometimes I will just call function- I was testing the other day for Erlang solutions E-learning offers and certifications stuff. There’s an exercise about process rings and so to run the process ring the thing I did was found the function traced whatever column of messages that were in there and analyzed that I got all the properties I wanted. Do I get enough processes? Do I get enough messages? And then you can draw a graph of where the messages were coming from and going to. And by doing it with tracing then you can get a lot of insight about what’s going on in the VM as a whole. So that can be a useful way to do it
Chad: Do you ever do tracing with automated test development where you’re actually looking for a certain set of states in the trace or is this more normally a manual process?
Fred: Well what do you mean?
Chad: In other words let’s say you’re running a EUnit test and turning tracing on the process and then receiving the trace and then actually parsing it to make sure the right thing happens? Or is this more that you turn tracing on and then you’re watching it manually to make sure that you know you’re…
Fred: Oh no, when I’m doing automated testing it’s always automated testing. When I trace it there is the option to send the messages to a given process.
Chad: Exactly right. That’s what I mean – how are you doing that?
Fred: The test process will gather messages and sometimes I just do it once in the shell and “I say ok that’s the kind of behavior I expect” and sometimes I find a bug right in the shell. And I will say “these are the kind of events I expect.” And so I will write them down and the more I do this kind of testing the less I use Erlang shell. I rarely use it except to just compile stuff and then run the tests with itself. Because all the behavior I want to test is already in the test so I no longer need to open the shell test it wait for the time, see if everything’s alright.
Chad: Ok, that seems like a great approach and actually that’s really interesting because I can’t think of another platform where you can actually turn tracing on so easily and test what’s happening inside of another process or an object other than Erlang.
Fred: Yeah, I mean there has to be ways to do it other languages in some places but it’s not always obvious or thought the same way. I mean Erlang was made to run forever so you don’t want to stop your program you might want to debug it in production so this is something that not many run times or languages have ready or easily available. Some guy from IRC a guy I’ve worked with a lot said he worked in embedded software and sometimes you needed to have dedicated hardware to that kind of tracing and it would cost them thousands and thousands of dollars and we just had it for free. So that’s pretty neat.
Fred: One of the other advantages that we have is with something like meck mocking tools in general get kind of weird when you have to work with types and then if you have classes in an object or in languages then you have to work with inheritance and making sure that everything respects the right API and then you have to private the public method and it can make it very complex to do mocking properly. And when I’m looking at tools like meck well the way it works is it can just swap the module from the runtime system in Erlang; replace it with whatever it thinks should be doing or mocking it and then does it transparently. And that way you don’t have to find a secret way to change the class something it just leaves your code the way it is and just changes the global environment of the VM by loading a different module than the one you usually have. Now you’ve got your mock going you can have passthrough calls. You can check the whole history of the calls you’ve made to that module so it makes tracing/mocking very easy including outputting stuff. I used it sometimes to test file format; am I outputting the right kind of content. You can just mock the ideal module you need un-stick it you mock it and then you’re able to say “well I did output the right stuff” and that’s not something that’s often easy to do or to test when you’re just writing straight out to the VOS with a system call.
Chad: How often do you use meck with your tests?
Fred: Well, I use it frequently whenever I’m getting into modules that have dependencies so I have a gen_fsm that’s calling some kind of server and I don’t want to depend on the server to make my stuff work. So I will just mock the other server maybe its registration. So you get rid of that stuff and you pretend it has the right behavior. So that’s when I will usually use it. I don’t use it in “Learn You Some Erlang” because I only tend to stick to what’s inside the Erlang distribution for that one.
Chad: If someone is writing Erlang code now and maybe they want to get into doing test driven development what tips you would give them to get started? Maybe they’re familiar with Erlang but they’re not an expert. They’ve written some gen_servers they’re familiar with OTP and they want to get into doing more TDD style development. What’s the most important thing you would tell them to do?
Fred: First I’ll tell them to acknowledge the tools that they have available. You’ve got EUnit that does very well for unit testing; when you get to integration testing and system testing then common test starts to be very interesting. You can do distribution testing something that works. Many notes start them for you. Have more complex hierarchies. When you have something that’s more of an algorithm then I would go with PropEr and these kind of things. So you’ve got a bunch of tools and the right tool for the right job based on that. Then once you have an idea of what these tools are I’d say try to find where you’re problem fits into that what are you trying to test. If it’s a module or something like that I would just say “well test it the same way you would test it in the shell.” The quick way that I like to do it that way is that if it works in the shell for you then it should work in the code. And the only thing you do is a take a little bit more time to type that code in the module and then after that it’s always automated so you don’t have to type that in the shell anymore. But you still get the same kind of tests. So to me that’s the easiest way. Do whatever you were doing in the shell do it in the module. Don’t care about it after that.
Chad: Is there any tool out there that you know of that does equivalent of AutoTest? Are you familiar with AutoTest and Ruby where you can basically run a series of tests and then if any fail anytime you make changes to a particular file it only re-runs the tests that would be affected by the chance you made. In other words it sort of eliminates having to run through your entire test suite while you’re doing coding and debugging.
Fred: No, I’m not aware of that.
Chad: Yeah, I was curious I haven’t heard anything.
Fred: It should be kind of simple to write one if you’re using Emake files you can use make all load and then itself putting you all the modules that need to be recompiled. So you could probably look to these modules there if you capture the output from the made call and then just run the test related to these files you could do it. That’s something neat about the Erlang testing tools. It can all be used together because everything has to evaulate to something. EUnit finds a mistake it’s returned false. So I can run EUnit tests from with common tests and these EUnit tests can be running proper tests and one inside the other like that. So there’s a way to use them and build tools on top of them that could do it. With a little bit of trickery, a few hundred lines of code and you will have that tool.
Chad: OK that makes sense. I want to talk a little bit about continuous integration. Are you guys doing a lot with CI and does that fit into your testing philosophy?
Fred: Well, I’m not very active on the programming side of Erlang solutions right now. I’ve been spending a long time writing course material, updating it and these kind of things. And I’m starting to work a little bit on helping people on E-learning and that kind of stuff. But I know that we have this project I forget the project name. At Erlang Factory in San Francisco Tino Bredden was presenting it. We are running CI servers and hoping to do it as some kind of platform as a service something like that. And Erlang solutions was working on a solution to let anyone run a code on our continuous integration servers and to be able to see the results on there then you could have different plans; free stuff, paying stuff, and all these kinds of things. I’m not part of the project so I don’t know all the details but we’re working on that and general testing stuff. I know that we have a few internal projects I think working with that one. So we do have a continuous integration server. If I recall for a while we were using the EUnit “sure fire” module if I remember what I saw. And so you could get the XML outputs that you needed with I think it was “Hudson” which is now “Jenkins.”
Chad: Are there tools that you don’t have that you wish existed or anything that is missing in the Erlang toolset?
Fred: Yeah, there is a tool I’ve had in development for a long time but I never find the time probably never will. It’s a bit tricky. But if you work with Python you have something called “Doc Tests” and so the way Doc Tests work is that you type in what you want to happen in the Python shell and then you type in the output that you expect and you can test against that. And what I thought about in Erlang it would be absolutely neat if we could have a shell recorder because everything is Erlang calls. So it’s easy to get the timing in there. You could start something called the “Shell Recorder” it just binds itself to the Erlang shell and would record all of the commands you send in there, records all the output that’s coming out and then saves that into a file that you can edit. And then you can just run the shell session over and over again — this kind of thing. So you save all the input save all the outputs and then run it again to see if it still fits the previous recording. So you could just test code the recording that way. And that will be a very easy way to test output of the code, failures of the code and that kind of thing without much of a problem. So this is something I would find pretty neat and I’ve had that in the back of my mind forever and I’ll probably never do it. But I think it would be pretty nice.
Chad: That’s great, I really like that idea.
Fred: Yeah, I like it too and then sometimes I find it less useful because the more it goes the less time running code into the shell I’m just running my test suites. To some extent I’m losing the need for it but before knew I all the tools we had it did look like something very interesting especially for testing something for cross-notes.
Chad: Ok, I think that’s actually a brilliant idea I wish I had time to work on something like that. Ok, so any other questions we had on TDD-yeah I think that was all the main questions I had, anything else that we’ve missed?
Fred: Well we had introduction to different tools. No I think that might sound like most of it from that part I guess. I mean you’ve got all these other tools that are not about testing but are still about support and all the environment you would quickly mention tracing tools you’ve got a bunch of them you’ve got the logger which is part of the language this kind of thing. One thing I like to say is that the best you can have is when you put your code in production. And so you have to be able to diagnose problems in productions, be able to fix them, and be able to work with them. And that’s something that you can do in Erlang that’s hard to do with different things different software in these cases.
Chad: Just a couple of practical questions. How often do you declare the types all of your methods up front?
Fred: Well, I have two types of codes usually I have prototype code and then I have production code or project code. And I don’t think prototyping is exclusive with testing. Testing is usually useful when you know the properties of your software “what’s going to happen.” If you don’t know the properties writing tests will be kind of useless and get you nowhere. Maybe it will help you find some of the properties you need. But then, I prefer to do prototyping for that. So when I’m doing prototyping and just banging out code trying to see the mechanics the easiest approach I’m taking “does it feel right or does it feel wrong.” In these cases I won’t comment the code that much. I won’t be typing it. I won’t be running spec checks. I’ll be OK with just leaving -compile(export_all)’s all over the place. Whenever I’m doing something that’s more about production though I tend to try to type all the functions, declare all the types, have EDoc in there provide a test going with it. The few side projects I have tend to respect this kind of thing. But yeah, I tend to type my stuff. Type checking is worth the effort in my opinion. Even if it’s not enforced at run time for Erlang it’s still worth the investment in time. Especially since now you can use it with PropEr and if you’re using just broad types and that’s something that needs a sequence of function calls. Then now you can use the type declaration as a way to generate data for your properties and your tests. It’s getting absolutely useful; now add to that the type declarations are useful in EDoc. There’s no reason not to have them anymore.
Chad: That makes sense.
Fred: I mean you have to make it worth your investment in time. And that’s the same with testing, if you think that your test will be extremely tedious to write and you’re never going to use them because you can’t trust your tests… For a lot of these things sometimes it’s not writing tests and when it’s code that you will only run for yourself a few times and you don’t plan to ever use or change after that. And it’s not in public stuff it’s kind of worthless to write tests sometimes because you don’t need them because you don’t need the software that much in the first place. So there’s a yield there between the time you take for the tests and the time you will take to maintain it. And if you know it’s just going to be a one-time code and it doesn’t have high impacts in production I’m more likely to test it in the shell and not write tests for that. If it’s something I’m going to distribute I don’t want to be the guy who kills someone because my software was used and had a defect in the hospital or something like that.
Chad: Just one other thing and I might move this question to a different part of the interview; but if you find a bug in production do you write tests first to reproduce the bug and then apply the fix?
Fred: Yes and no. Sometimes when I’m just hunting the bug-first of all I don’t try these things in production. I will try to get the production system and then I will try in development because I don’t want to trash the data base by accident like I did at some point. But I’m going to try in development and then I’m just going to head into code and I’m a bit of an old school debugger even though I can’t be old school I’m too young for that. I’m debugging a lot of file formats, output statements, E-Code, prints and things like that. I’m using that all the time to debug my stuff and see the sequence of events. With Erlang I’ve got the option to trace instead of inserting that so I’m saving some time some time. But I’m going to just insert stuff in the code directly then find something and say “Oh that’s really the mistake I have.” Then I try to do a quick fix in it and see “Oh yeah I solved the issue by doing that” so I probably solved the problem. Then what I do is a I reverse the quick fix, go write a test, make sure my test finds the failure I spotted in the shell or something like that. And then I enable my quick fix back again and see if the test works now. So I’m testing for the broken condition at first and then I’m testing for the repaired condition. But I’m going to find the issue I’m trying to solve before testing. If I’m just running the test first of all and I don’t understand the context in which it happens I’m going to miss a lot of stuff because in Erlang and any kind of large software a lot of issues you will have will have to do with some kind of global state that has side effects in place you didn’t except. So Haskell people will tell you if you had monads you will not have the problem. In this case it can be configuration it can be these kinds of things and I want to have the system running and not just tests in the file to be able to target the problem with more precision in the first place.
Chad: What about testing and debugging?
Fred: Well, sometimes stuff is just a lost code. You just can’t reproduce it all and you know it’s some kind of hidden bug and the fact that you’re looking at it is keeping you from replicating the bug. And in these cases I fire up the printer, print the code and then just highlight it and check it and do it logically and just trace it by hand. And I’m a visual guy so sometimes I will just draw bubbles and arrows and these are the directions and then this message goes to this other bubble which is this other process. But wait a minute this kind of message came from that place and just looking at the code and analyzing it is usually my last resort. I had to do it once or twice before. But it doesn’t happen much. The biggest bug I had was actually PHP. It was a problem with a chrome job running on a table that was not locked properly. So the bug would only happen once a day five minutes before midnight. In some certain contexts not all of the time; so you had to look at the code and say “wait a minute this table is used by this other bit of code, and this chrome job is only running when this kind of threshold is on the servers.” And then you simulate that and it takes you forever and in some cases like that at the point where you’re just frustrated pulling your hair out of your hair and just have no idea then I print the code look at it and that’s really the best way to find any kind of bug. But it’s just so time consuming to do.
Chad: I have a question for you related to algorithms. So a lot of Erlang products are more what I would say are algorithm heavy programs. In other words you’re doing stuff that has a lot of complexity to the processing. Are you familiar with the guy, Peter Norvig , with the TDD failure of his Sudoku solver? Sound familiar?
Fred: Yeah the Sudoku solving and I think the big issue there–point I was proving is that there are two kinds of ways to attack TDD. There’s I think in the small method you don’t know much about your problem and you’re using TDD to guide a design. And then you’ve got the TDD where you know you’re going to need these kinds of things so you know in that event you’ve got to write a mailing system and that you’re boss just bought something like a fan mail server costing thousands of dollars. And you had this knowledge beforehand. You’re going to have to take two different approaches one is more about discovery the other is more about writing a specification as code. And when you do something with Sudoku there are explicit rules already known and techniques to solve these problems. And it’s kind of I wouldn’t stay stupid but misguided to jump head first into the problem without understanding it fully and what needs to be done to solve the problem. When you know there’s a mathematical reason or mathematical principal underneath, good techniques that have been proven over the years it’s useless to jump head first and ignore all the previous knowledge and just do it guided by tests. And I think that the way Peter Norvig did it with testing the stuff could have worked with TDD. He had all these ideas that I’m going to represent the grid that way I’m going to use this method that was used by expert Sudoku solvers. And you could have written a test to do that rather than doing the other thing; where the guy was just saying “how am I going to represent the grid.” So if you know how to solve the problem then you will save a whole bunch of tests. You document it of course “this is the technique I am using I found it at this place.” But then the test you have can be based on that knowledge rather than trying to brute force your way through the missing knowledge. The same prototyping is not mutually exclusive with TDD both can work together to help you guide your design. I don’t think that knowing or starting with knowledge on how to solve the problem is a bad idea.
Chad: Yeah, I think that’s a great point to me. What I think the problem is that people sometimes think of TDD as being a sort of formula for finding emergent algorithms versus being something that allows you to just make sure that algorithm that you already come up with on your own or that you’re coming up with as a part of the coding is correct. So it makes a lot of sense.
Fred: Right, that depends I think they’re a different school of thought. You have the idea of extreme programming the TDD extreme programming way; extreme programming way sometimes tends to do that more. My personal opinion is that it’s not always efficient. There are some kinds of problems where it will help. If you’re working with business rules I guess it can makes sense to do it that way. You know the business rules in advance. You know the kind of output you need. You have no idea of the systems that are going to be implemented sometimes it can be interesting to search based on that. But when it’s something that knowledge exists algorithms, Sudoku is some kind of algorithm there. We had a talk in London. Some guy was doing maze solving and sometimes you know there are some kinds of known ways to solve mazes. I think it’s counterproductive to just ignore the existing knowledge and think that you can be faster than all the people that have been working on that kind of stuff for years before you.
Chad: Ok that makes sense, cool. Well anything else you want to cover continuous integration, test driven development, I’m trying to think…
Fred: Oh yeah, testing with side effects this is something I see a lot of people do. This is something like focusing for common tests but I like common tests and I’m going to solve it that way. Sometimes I will see people write EUnit tests and then you need to write files on disk you need to start remote notes you need to run the same test 50 times to do some kind of stress measure on your testing stuff. Or you need to be scaffolding I need to start second application on the site and communicate with that one. When you do that with EUnit you have to build all that scaffolding yourself; writing some kind of function that will let you write random files that didn’t exist before and for these kinds of tests you’re getting into something called system testing; where you’re testing the whole system and interactions between different components. Common tests are very good for that. It’s already creating all these things like scratch directories, configurations for tests. You’re going to need passwords that can change depending on the environment you’re running the tests in. You can create different configurations pass that to the test functions going to run with them. It can start remote notes, let’s do write scratch files and scratch directories that you can look at later and do all your file operations that has side effects. You can do them there without breaking the system too much unless you’re doing something global with the system. But if you’re writing just a given path then you can pass that path to common test there. And with all that kind of system testing common tests is really worth it because it’s going to save sometime once you know it to work on these kinds of things. Erlang is totally ready to do this kind of entire software testing and I know some people in some enterprises have been using common test to do distributed testing on C code. Just because in those different protocols it was just easier to distribute Erlang through testing and loads and that kind of stuff than maintaining the C frame work they had that had hundreds of thousands of lines of code. It just scratched that.
Chad: So you can use the Erlang common test for the actual test framework.
Fred: Yeah, but because common test comes with ways to communicate from SSH to HTTP and SNMP agents in there all that kind of stuff. So when you’re communicating with hardware and firmware written in C then sometimes it just simulates the client with common tests; use all the stuff that’s already there and pull the programs as a black box testing in that matter. It’s very neat for everything that’s moderately large to test with many attractions common test is becoming a very neat tool.
Chad: Ok that’s great, and do you guys have a particular set of tools that you recommend at Erlang solutions is there a particular tool you recommend or is it more up to the client?
Fred: I have not done a lot of consulting. Usually if the client doesn’t have an idea of what they’re going to do I figure we’re just going to recommend the tools we know ourselves. Someone on the team is going to be productive with what they know. But I think it’s important that if the client has a given setup and their team is comfortable with that and it works-if it doesn’t work you have to correct that- but if it works for them it’s a bit counterproductive to change their methodology if they are on a deadline. You can show them new tools but you don’t want to tear the whole enterprise down just to bring it up and then be losing time.
Sometimes people tell me “should I start the project in Erlang?” And I ask them “what kind of language does your team know?” And they know Ruby or they know JavaScript or they know Python and they don’t have much time so no just write it in the language that you know. You’re better off sometimes having an adequate solution in the language you know and know how to maintain rather than a good one that you cannot work with because you don’t know how it works.
Chad: That’s great, well listen thanks so much Fred, I really appreciate it.
Fred: You mentioned a few questions in IRC.
Chad: Well, yeah the last thing I was going to ask was on IRC. You spent a lot of time on there and you’re a really valuable resource. Is there anything you want to mention about that?
Fred: I would say to people is if you need Erlang help it’s really the first place to go to get any help quickly. For quick help I would say IRC first and when you can’t get the answer for IRC usually the mailing list will have all the Erlang old guards. The really smart people, the people from Ericsson who maintain it and know all of the ins and outs of the language are going to be hanging on the mailing list. And sometimes people on IRC say “well we don’t have the mailing list” but IRC’s definitely becoming a very neat place to ask for help. I think it’s progressively taking sites like “Trafficzip” and that kind of thing. Which had a bunch of recipes but then if one of the recipes is not there then you head to the mailing list right now people can head to IRC and just ask for help directly there. There is almost someone 24/7 there just lurking. You may wait for half an hour but usually you might have a response there.
Chad: Thanks and Cheers.
Fred: Cheers.