Software Architecture in an Agile World

So what is Architecture? I like the quote by Ralph Johnson because it’s a clear and succinct definition:

Another quote I like that stresses the importance of thinking ahead:

So, according to Johnson and Thomas we want to get some bits and pieces right early in the project. We do want to think ahead before we build something, but we don’t want to fall into the trap of designing a system that, by the time it’s built, won’t fit what the business needs anymore.

On the other hand we have an agile process where we need to deliver something 'of value' to the customer every couple of weeks. How do we reconcile these two, since sitting in front of a whiteboard for days doesn’t really produce anything of direct value to the customer?

One of the hardest parts of being a software engineer is explaining what it is that you actually do. When we explain that we 'write code' it sounds like a simple sequential process. On one end coffee goes in, on the other end lines of code come out. Right? To a certain extent this is what we do; we break down problems into smaller problems, imagine solutions to these problems, implement code that solves them, and hey presto we have a shiny new login screen.

Here it’s not hard to draw a parallel between writing recipes in a cookbook and writing software. After all we write step-by-step instructions for a CPU (which is like a cook but without any sense of humour). If we write a recipe for a pizza we start with the dough, add tomato sauce, herbs, toppings, etc. Any cook can follow this recipe.

Software is similar: we write instructions for a CPU but don’t even have to do it in the native language of our CPU. We have higher level languages that let us express ourselves better which then get translated to instructions. This means we can tell our cook to make pizza dough instead of having to explain all the individual steps of assembling the dough from flour, water and yeast.

Where this comparison starts to break down is how incredibly complex our recipes become. I think most of us have a few go-to recipes they know by heart. But it is simply impossible to know every single line of code in a non-trivial application. What’s worse; it is not possible for a single person to know what a complex system does exactly. Systems often end up doing things other than what we assume they do.

To give a rough idea of how complex software is, let’s take a look at a really simple piece of software many of you might have used: Pokémon Red. The source of this game is available on GitHub. 177,469 lines of glorious black and white nostalgia.

Can you imagine a cooking recipe that has 177 thousand steps? A software engineer can; they work with these types of recipes daily. Fortunately we don’t have to deal with assembly anymore, the languages we write software in (be it Java, C++ or Kotlin) are a lot more expressive. To write Pokémon Red in C code you would roughly need 'only' 44 thousand lines of code.

On my previous project, a microservice architecture with well over 20 services, the largest microservice had 50 thousand lines of Java code. It was too large for a microservice (and later fortunately got split up), but that is still a LOT of code for a service that just had to deal with the single task of letting users add bank accounts to the application.

Just the services that I worked on myself had 196 thousand lines of code in total. To compare; the massive Guava framework has 769 thousand lines of code. The Spring Framework has one million lines of code. Can you imagine writing a pizza recipe with a million steps?

Someone thought that it would be cool to be able to play a game inside another game. This person took it upon himself to port the 177 thousand lines of Pokémon Red code, by hand, to 357 thousand command blocks. It 'only' took him 21 months. You can see it in action in this video on YouTube:

The entire video is 29 minutes. I would strongly urge you to watch the entire thing since he gives a great introduction on why and how he did all this, but if you skip to 11:15 they’ll go down into the actual 'code' of the game and you are greeted by this in-game view:

All those green and orange blocks are Minecraft command blocks. They can be programmed with a single line of 'code' to handle really simple tasks. Because these tasks are always simple and he needed to create workarounds for missing features (such as using player scoreboards as data storage) he needed 357 thousand command blocks to handle the 177 thousand lines of Pokémon Red code.

I don’t know about you, but I was awestruck when I saw the video. Not only because of the enormous amount of effort that went into it, but also because it shows complexity so well. It’s beautiful.

Software is its own worst enemy. As a Software Engineer I dislike software; I try to write as little of it as possible. When software inside the system has become obsolete, it will be removed. If the software doesn’t serve a purpose it only adds negative value. Some of the best sprints I had were ones where I ended up with a negative amount of lines of code produced.

This is incredibly counter-intuitive to almost anyone who’s not an experienced software engineer. Can you imagine being paid as a recipe writer to just remove recipes? It doesn’t make much sense for a recipe writer; but for us it definitely does.

A great example of how complexity is harmful is in the Minecraft video. Around 14:50 the author is explaining how he uses coloured wool blocks to encode data (for example whether a move is a 'Fire' or 'Water' move). When talking about a second layer of these blocks he mentions that he doesn’t quite remember exactly what these blocks do anymore.

So imagine this. This guy is a brilliant programmer who wrote all this stuff by himself. It’s not code someone else wrote. The 'code' is right there in front of him. He’s not a junior developer either, he designed the entire system by himself. But he doesn’t know what it does; he’s just guessing it!

We humans are incapable of handling this level of complexity. This is why we need to build abstractions of this complexity, without it we can’t even reason about it. Or to quote Dijkstra:

Our puny brains can’t fit all this information. If there are things you haven’t used for a while, like what those blocks are used for, our brain tosses it out. This is why we put lines of code into functions, functions into components, components into modules, modules into systems, systems into architectures. We need higher and higher levels of abstraction to even be able to reason about the complexity of these bits of code we deal with on a day-to-day basis.

In the video you see that the developer did this too. The blocks are stacked into columns and rows that make up functions. Similar functions are grouped together into modules. And these modules are then grouped together into overall systems (animation, input and game rules). This is not code that randomly grew; he did up-front design.

And this is why, no matter how senior your development team is, you need this form of abstraction and communication. Without it you end up with a disjointed unmaintainable mess of spaghetti no one dares to touch. Software will always grow, and we need to guide this growth in order to keep this complexity manageable.

So for me, architecture boils down to maintaining this complexity. We can’t just decide to stop developing software. Very few businesses that deal with real problems for real users are ever going to accept us telling them the software is 'done'. This is why big design up front can never work; we can’t ever predict the future. You could pretend you can (and many people do); but then you’re lying to yourself as much as you’re lying to whoever pays your salary.

And on the other end; we need to guide the growth of the software in a fashion that keeps complexity in check. As I said in a previous post; Software Architect is not so much a job title as it is a role for a certain set of Software Engineers. In my opinion you can’t be a Software Architect without being a Software Engineer; if you want to reason about high level abstractions you need to understand the underlying systems.

So does every Software Engineer make a good Software Architect? No; many are very content working on the part of the system they are responsible for and don’t want to have to think about the whole of the system. Which is fine too; where architects often need a more shallow broad focus, engineering complex bits needs a deep narrow focus. You can’t have both; no one has a deep broad knowledge of everything. Even Einstein needed the help from Grossmann on the harder bits of math for his general theory of relativity.

That does however often lead to conflicts. Those interests don’t always align. A developer or team might want to trial a completely different programming language. Why not right? Languages are just tools. Sure. But another technology stack means not just a different language but also different tools, libraries, frameworks. Deployment will be different too. Integrating that stack into existing CI/CD systems is often an afterthought.

So as an architect you will need to first and foremost give these developers a voice. Don’t shoot down every experiment they want to take on; guide them. Help them with for example a simple checklist of 'things' the new system has to support: IDE support, testing libraries, integration into CI/CD, logging, monitoring, tracing, etc. If these things are a lot of work, or difficult to get working with the existing systems, the developer will see this soon enough. And if it works fine, and the development group as a whole wants to support it (you can’t expect a Java developer to want to go and maintain a JavaScript back-end 'just because'), why not support these choices?

Create your own tech radar of technologies, tools, libraries and frameworks you do and do not want to support.

I started out this post with the question whether Architecture adds direct customer value. The answer is that it doesn’t. Not doing architecture on the other hand adds negative value. Not doing architecture means complexity grows out of control. Development will slow down, the amount of bugs in production will start to rise, and customers will become unhappy.

Some decisions are easy to change. If I want to add ham to a pizza I can just add ham to it. If I want to remove salami I can easily remove it. But some decisions are a lot harder. If I have a pizza and instead of a tomato sauce base I want to use a sour cream base, I can’t just keep the same toppings. Sour cream pizza with chicken and spinach? Delicious. Tomato pizza with ham, cheese and pineapple? Delicious (I know, I’m a heretic). Sour cream pizza with ham, cheese and pineapple? I don’t know about you but I can’t imagine it being tasty.

This is why we need architecture, it’s the decisions that we want to get right from the start that are the most important ones. When we swap out the foundation of our pizza we well need to refactor all the ingredients that depend on the taste and mouthfeel of that foundation to work.

Combining the two is all about balance and communication. One one hand you have teams with experienced senior engineers who constantly make architectural decisions within their team, often by discussing these issues between them. On the other hand you have people in the architect role to help them fit those decisions into the overall plan of the entire system. So one of the responsibilities as an architect is to create a layer of communication over those teams.

It’s common for this to happen automatically in an ad-hoc manner but this often needs additional structure. In my current project, as well as in my previous project, we had a short weekly meeting with architects and engineers from the teams sitting together to discuss the cross-cutting architectural concerns.

So as an architect you should be a facilitator and mentor at least as much as a gatekeeper. This often includes communicating with all stakeholders, technical and non-technical, why certain decisions were made. Ivory tower architecture where a software architect dictates his view without any dialog never worked and will never work.

In an agile world we still need architecture; now even more than ever because we can’t predict where we are going in any exact manner. Without architecture, without thinking ahead of the impact of our decisions on the system as a whole, complexity will grow into chaos. No matter how experienced the engineers in your teams are; with a disjointed vision on the system the system will become a disjointed mess.

As an architect you will need to see your role as one where you, from experience, facilitate communication over those teams on the topics that affect the system and it s complexity. This mostly involves listening and guiding. Sometimes it involves hitting the brake and having tough discussions. But, like software engineering itself, it all boils down to communication.

As a Software Architect it’s my goal to facilitate this communication with my own set of tools. So please set up architecture meetings with a well defined agenda (open to anyone to add topics) which result in discussions that lead to clear decisions and goals over the teams. Help developers by implementing changes yourself. Set up a technology radar, and guide teams to add their tools to that radar. Give them tools to document decisions and set up objective tests whenever possible.

And most importantly; stay humble and work with your team.