Taming software development with teamwork

Danny Dig and Ameya Ketkar develop tools to help programmers create software


How are Oregon State researchers impacting the apps on your phone? Danny Dig and his team have been improving software for companies like Microsoft, IBM, Intel, Boeing, and Google for years by developing tools that find and fix bugs. Dig is an associate professor of computer science at Oregon State and an adjunct professor at University of Illinois at Urbana-Champaign.

Season number
Season 5
Episode number

[MUSIC: "The Ether Bunny" by Eyes Closed Audio used with permission of a Creative Commons Attribution License.  ]

NARRATOR: From the College of Engineering at Oregon State University, this is Engineering Out Loud.

ROBERTSON: Johanna’s in the house! Johanna did you want to introduce yourself?

CARSON: Yes, I’m Johanna Carson and I’m with the School of Civil and Construction Engineering. And on the podcast team I do photography and visuals so I haven’t been in the booth in a while.

ROBERTSON: Well, welcome back. And I’m Rachel Robertson. So Johanna, last time you were here we were talking about networking speed and how it can affect the performance of your phone. (And listeners, if you want to check that one out it was Season 4, Episode 2.) And today we have another issue that can affect the performance of your phone apps — the software itself. So, software has become pretty critical in our lives these days and it can be frustrating and almost nerve wracking when it doesn’t work.

CARSON: I think it’s interesting how quickly we have become so dependent upon software from when you get in your car to drive somewhere – I can’t remember the last time I pulled out a paper map to examine it, or even looked online at a map. I’m using this kind of software on the go.

ROBERTSON: Right. Exactly. And I almost have a panic if I feel like, oh no, my battery is dying. I’m not going to know where to go. I’m going to get lost in the city.  

CARSON: You’ll be in the abyss.

ROBERTSON: Right and so having apps we can rely on to work when we need them and not freeze or crash at the wrong moment is pretty important to most people. But few of us think about what goes on behind the scenes. The task of creating reliable software has become increasingly difficult as software has become more complex. In fact, it is estimated that the software behind all the Google apps spans two billion lines of code. I can’t even imagine what that looks like.

[MUSIC: “Calm” by Mortal Thing used with permission of the artist.]

CARSON: It’s hard to fathom.

ROBERTSON: And those two billion lines of code are being modified by 25,000 Google engineers. That’s pretty crazy. And I think a lot of people don’t realize that software is written by teams of people. Or in Google’s case teams of teams.

CARSON: Yeah, that makes me realize that I really understand how all of that is created. You know, I haven’t done programming myself but I have quite a few family members who do that so I’m familiar with that in rough terms but I don’t really understand how the process is done by the software engineers. In my mind I picture someone sitting down and it would be similar to composing a piece of music or blocking in a painting and then creating a painting.

ROBERTSON:  That’s good. Good imagery. And I think a lot of people may think of it the same way. It may have been like that at one time, but I talked to Danny Dig, who is an associate professor of computer science here, and he explained how complicated software development has become, and why it has created challenges for developing bug free code.  

DIG: So, a widespread myth is that software engineering and software development is writing brand new programs from a scratch which couldn't be farther away from the truth. When you think about all these large software systems that we have today — think about Microsoft Office Suite or everything we have on the smartphone — those are gigantic large projects no one single human being  alone or single engineer writes them from scratch. In fact, 90% of software development costs arise from changing existing code — not from writing brand new code — by changing existing code.

ROBERTSON: Danny’s area of research is software engineering. In particular, much of his work involves developing tools to help software developers fix bugs that can make your phone app or other software slow down or freeze. The tools are to benefit software developers by finding errors and inefficiencies in their code, but users benefit indirectly by having software that is more reliable, faster, and more secure.

DIG: We have been very fortunate over the years that we have had lots of our research incorporated into the official release of highly popular software tools that are used by millions of Java programmers or C Sharp programmers who work on Microsoft platform are used every day. And that's to me the greatest reward and the greatest sense of accomplishment of doing research is whether the industry adopts and uses our inventions because in the end it's not just about us but it's: How can we make other people better? How can we make them more productive? How can we empower them?

[MUSIC: “Harps Uplifting” by Mortal Thing used with permission of the artist.]

ROBERTSON: Just as programming is not a solitary experience anymore, neither is research. And Danny has a team of students (both graduate and undergraduate) who work with him. And the story I want to tell you today is about a project by one of his graduate students that caught the attention of a Google engineer.

KETKAR: I am Ameya Ketkar.

ROBERTSON: Ameya was inspired to come to Oregon State to pursue research when he was working in industry.

KETKAR: For a couple years I worked as a developer at a bank and while working I realized the value of having the right tools in hand and how important the tools are to a developer. I love tools, like, I think tools are what makes a developer a better developer.

ROBERTSON: Well, that sounds pretty cool, but if you are like me, you have no idea what a tool for software development could be. So, I asked Ameya and Danny to explain how tools work.

KETKAR: So, we've got source code which is the program itself. So, we develop tools which analyzes the source code and then edits the source code in a very interactive manner.

DIG: And we are very keen on saying this is Interactive. So, don’t believe the tool replaces the human, the tool is mostly augmenting the human. The tool finds out where are problems in this particular usage of this new construct that the Java programming language introduced and in the end it is ultimately the decision of the human whether to accept that recommendation and to allow the tool to also change the code.

It's a nice combination of where we have a lot of automation but at the same time relying on the expertise and the insights from the human gives the tool a lot more leeway. The tool can be much more aggressive in the kinds of edits that it suggests to the programmer because in the end the programmer has to decide whether to apply or not apply those edits.

KETKAR: So, actually it's like Grammarly for source code. Like those applications that you use on your text while you write which goes and checks your grammar. Have you written it correctly? Or it suggests: You could write it this way. So basically we do that for source code, in a way. We try to improve the readability of the text.

DIG: So people, you know, people are using editors and when you're using the editor it sometimes it highlights errors that you make in the text when you're editing the text but so now we have both highlighting and finding those errors especially those that have performance implications but also fixing them so it's not fully automatic. But, you know we ask them this is the fix -- do you want to apply this fix?

ROBERTSON: But we are getting ahead of ourselves a little because before they even get to creating the tools, they have to first figure out what problems need to be solved.

DIG: So our process is something that I call problem driven. It's a problem driven research approach in which we first spend a significant amount of time figuring out what's a problem that software developers experience in practice. And then to find this problem we first conduct one or several formative studies where we analyze we look at pain points. Essentially, this is a discovery process where we look at the pain points.

ROBERTSON: Danny and his students use both qualitative and quantitative methods to find where the problems are. They use surveys and semi-structured interviews to give them initial direction on where the developers are reporting issues — their pain points, as Danny calls them. But then they back that up with evidence by looking for errors in the source code that correspond to the reported issues.

DIG: Once we nail down what is the problem then we work on what is a solution that solves this particular problem. And now when we come up with a solution it fits naturally fits like almost like a perfectly matched glove on your hand. And the reason why we have then such a high adoption rate of our research and practice is because we really spend a lot of time understanding what is the problem and figuring out what is the heart, what is the essence of the problem before we design a solution. What I have found over the years is this gives us a significant chance of coming up with research that is widely adopted in practice and, as I said, this is the number one value that I care about whether we produce research that makes a difference for the lives of software practitioners.

ROBERTSON: So, as we talked about earlier, the way they do that is by creating tools for software developers. What Ameya creates are two major tools: one that reads and analyzes the source code, and one that actually does the edits. These two tools communicate with each other, so they are essentially two tools in one. But you may be wondering, as I was, why these tools are necessary.

DIG: When software grows at gigantic proportions that have millions or tens or hundreds of millions of lines of source code, the code can easily become what’s called a spaghetti code. Where it's so intertwined you make a small tiny change here that has huge ramifications somewhere else in the program and this is the number root cause for software bugs. So that's the complexity, that's the challenge in our research is: How do we verify? How do we make sure that this change that we are carrying out with the tool doesn't have any unintended consequences. So it is systematic, it only changes what we desire to change and it doesn't have any unintended consequences because of all the dependencies of these code modules and the way how they are intertwined with many other parts of the software. So that's one of the most fun part in our research is taming these complexities. The code can be so complex and so tangled and: How do you make changes in this gigantic scale of spider web?

ROBERTSON: It’s one of those situations where a computer can outperform a human. The tool can more efficiently analyze the code and find what are called a corner cases – bugs that will only show up only in certain conditions. And it can also look at all the ramifications of a change. Ameya describes the issue of making changes in software as being analogous to making a big change to a novel such as changing the protagonist in a story from male to female.

KETKAR: Now I’ll have to change everywhere I used “he,” I’ll have to change it to “she.” Everywhere I had used “his” I'll have to use “her.” Now imagine doing this manually for let’s say for a JK Rowling book — like a huge book. Like I just tomorrow I decide Harry Potter was female. It's a tough task to do it manually so in some way we're trying to do that in some way we're trying to do that for source code. So sometimes if you want to make a big change like change the gender of the main protagonist you're going to end up doing a lot of work.

[MUSIC: “Morning Stroll,” Josh Kirsch/Media Right Productions, used with permission of a Creative Commons Attribution-NonCommercial License]

ROBERTSON: Although software does not normally need to change its gender — there are certainly some analogous situations. Consider that the hardware in our phones is constantly changing. All of your mobile apps have to adapt or die. Some changes may be surface changes like “he” to “she,” but there are some changes that go deep into the code. You can imagine that if Harry Potter is now female that may also require some changes to the plot. That was the kind of issue that Ameya uncovered in a product called Apache Cassandra which is an open-source database used for managing large sets of data.   

JIRSA: Huge, huge, huge capacity, and if you need more capacity you just add more machines.

ROBERTSON: That’s Jeff Jirsa, who is an Apache Cassandra committer and member of the Apache Cassandra project management committee. I learned that a committer is a person who can modify source code for an open-source project. Open source means the code is available for anyone to look at and suggest changes. So, Jeff receives changes from people who submit them and after evaluating them decides whether or not they will become part of the software. So, he is the person who “commits” the changes to the central repository. And in this case he is the one who received several patches from Ameya which we will get to in a minute. But first a little more about what Cassandra is.

JIRSA: It's used by probably thousands of companies, including some very big companies. You know, the people that have talked about using it include people like Facebook, Instagram, eBay, Walmart, Netflix talks about it a lot, Apple talks about it a lot, Microsoft has talked about it. So, there's probably dozens of very large companies that use it, and thousands of people, big and small, that use this database. But it's completely open-source and completely free, so anyone can basically download it and use it, and anyone can suggest changes back to the community.

ROBERTSON: And here’s where we get to that spider web of code that Danny mentioned.

JIRSA: One of the beautiful things about open-source software is that anyone can contribute. The downside of that is you occasionally get people who contribute code that is non-optimized. And what Danny's team was able to do was actually find some cases where people used features of the programming language in ways that are perfectly correct but slow — and in particular they generate more objects than they should, which ultimately goes down to slowing down the database.

ROBERTSON: Now this is where Ameya comes in.

KETKAR: I have submitted more than 30 changes to the Cassandra project.

ROBERTSON: And one of those changes was at the plot level (from our analogy). Here’s Jeff to explain.

JIRSA: And they fixed probably 20 or 30 different places, but one of them in particular was deep inside the database and actually is very important for performance.

ROBERTSON: Okay, and did that surprise you that they were able to find that?

JIRSA: I wouldn't have expected that situation to exist there. I'm not completely surprised, just due to the way the software evolves over time, with multiple contributors and many of them being volunteers, it’s not hugely surprising that an inefficiency like that would sneak in. But it was definitely nice that they did find it because it was unlikely that anyone else would have ever gone and looked for it and found it.

[MUSIC: “Morning Stroll,” Josh Kirsch/Media Right Productions, used with permission of a Creative Commons Attribution-NonCommercial License]

ROBERTSON: Submitting patches to open-source projects is one way that Danny and his team are able to have their research go out into the world, but the other way they do it is to work directly with software engineers inside companies. In the past, Danny has worked with Microsoft, IBM, Intel, Boeing, and Google. Danny usually cultivates these relationships with companies, but at recent conference where they were presenting their research on the Cassandra project, they were surprised to have a Google engineer in the audience who took a keen interest in their work.

DIG: So it turned out that this problem of fixing this misusage of this particular was important for the team, and for the code that they have at Google. So we started to talk more and he was very excited. He talked with us I think for about half an hour at the end of the conference, at the end of our presentation which is not typical, it's unusual for someone to show that level of excitement and interest on a project. So that was for us a nice challenge: to say how can we make work in an environment where the code is so large that it doesn't fit on any one single computer. So that was for us a nice challenge and of course the potential of having an impact and having our tool being used by engineers who work on Google codebase that was another big draw for us.

ROBERTSON: Partnerships and collaborations are essential for Danny’s research and he is always out connecting with people in industry to find out what are their major challenges and what they see as big upcoming opportunities.

DIG: So during those trips, I have been at more than 30 companies and I asked them what they are excited about the next big growth of computing and everyone talks about being excited about using intelligence that is personalized.

ROBERTSON: So, think of Siri, Alexa, or Cortana except better. An assistant that can anticipate your needs. Danny’s vision is big, really big.

DIG: The size and the complexity of this big dream is something beyond what any single researcher or what any single university what any single company can accomplish on their own. So it requires a team effort. So I have put together first a team of researchers from several universities and also a consortium of companies. So right now we have more than 30 companies who are interested to join us with this consortium.

ROBERTSON: Danny’s big dream is still a work in progress and he is currently working on securing funding for the consortium which he hopes will become an established center for Internet of Things development.

DIG: I'm super excited myself. I think if you can use devices to learn about us and to augment us so that … again, I'm not saying that they should replace us I'm not talking about a world where the robots are going to take over and take our jobs and are going to automate us away they are going to remove us. But I'm excited about the world where software intelligence and all this Internet of Things and all these smart objects are going to augment, are going to take us to another level of productivity, are going to probably going to semi-automate the lowest repetitive jobs but are going to help us to maximize our time, to help us to excel, you know, help us to go to go just beyond what we could do on our own. So this is what I'm excited about how can we augment human intelligence it's not how about we can replace humans remove them out of the equation, but how can we augment human intelligence with this huge intelligence that we can harness from all these devices that are monitoring our lives.

[MUSIC: “Harps Uplifting” by Mortal Thing used with permission of the artist.]

ROBERTSON: That’s it, my friends. I hope you enjoyed learning more about how researchers at Oregon State are applying their work to real software problems out there.

[SOUND EFFECT: Timpani roll in D from freeSFX.co.uk, used with permission of Creative Commons Public Domain]

And here’s Johanna …

CARSON: This episode was produced and hosted by Rachel Robertson. Audio editing by Brian Blythe. Our intro music is “The Ether Bunny” by Eyes Closed Audio on SoundCloud and used with permission of a Creative Commons attribution license. Other music and effects in this episode were also used with appropriate licenses. You can find the links on our website. For more episodes, visit engineeringoutloud.oregonstate.edu or subscribe by searching “Engineering Out Loud” on your favorite podcast app.

ROBERTSON: Thanks Johanna!

CARSON: Anytime! Get me out of here.

BRIAN BLYTHE: That’s where we are ending it.