Scaling the Codebase (Part 2): Modular Code Architecture

This is the second post in a four part blog series:

1. Introduction
2. Modular Code Architecture (this part)
3. Reducing Developer Friction
4. Wrapping everything in a Monorepo

An interesting thing to note about how Shawn worked was how easy it was for her to enter new areas of the code. Another interesting thing to notice was her lack of fear: she deployed code she’d never seen before, and minutes after changing it. At Roundforest Engineering, we believe that one of the most important things to avoid is fear of change. If we fear changing the code, we can’t really provide the agility of change that the business needs.

(not sure who Shawn is? Or what I'm talking about? Read the first part of this blog series in here)

In addition, the most important thing to note: she didn’t need to understand everything in the codebase in order to work with the codebase. And when she worked on it, she didn’t have to negotiate with others in order to work with the code. Independence! It’s a core value of Roundforest Engineering: the independence to work on code and not pass through various processes that try to ensure quality, but only add to the friction of the process.

No matter how large our codebase is, Shawn could navigate it and add to it easily. This property of a codebase is crucial when the codebase starts to grow. To keep the code understandable you need to “scale” the company’s code. You need to keep it from turning into a big ball of spaghetti code.

How do we scale a company’s code? How can we have many developers work on the same codebase, without turning it into one big ball of unmaintainable spaghetti code? How can we have both agility, which Roundforest values, and code quality, which is a necessity of any robust engineering culture?

The answer is always the same: modularize it. Turn it into a set of loosely coupled modules. Most companies implement this, but only in the micro. In the design of the features. So they build classes and whatnot to modularize parts of the code. It’s part of what is called in the industry “clean code”. And there’s lots of information on the internet about how to write clean code.

This is nice (and important!), but it doesn’t really solve the problem, because the problem is a macro problem. We can have a huge codebase of clean code, but the modularization in the large isn’t there, which will still leave us with a big ball of spaghetti code.

The solution Roundforest chose is to modularize in the large. To have a modular code architecture. And for the code to be modular, we’d want the architecture itself to be modular, so we use:

1. Microservices in the backend
2. Microfrontends in the frontend

By creating an application from a set of microservices and microfrontends, we are already on the way to modularize the application into a set of highly independent and loosely coupled modules (the microservices/microfrontends).

Microservices are great for modularization. Our developer, Shawn, could understand the microservice code without the need to understand everything about the app. She could even change it and know that her changes are local. She deployed it without being afraid that it will affect other parts of the application.

The interesting thing is that microservices and microfrontend can scale the codebase even if you’re not doing “clean code”! Let’s take microservices to the extreme: let’s imagine microservices that are independently coded, independently tested, independently deployed, and architectured correctly to be loosely coupled, but where the developers of each microservice are really bad and do not follow the clean code principles, and each microservice is a bunch of spaghetti code.

We’re still OK. A new developer looking to change or add some code to the microservice, doesn’t need to understand the whole set of code. They only need to understand the code of the microservice, and even if it’s spaghetti code, it’s still not a lot of code!

Of course, if it is clean, well tested code, it’s even better. But in a monolithic codebase, hundreds of thousands of lines of clean code won’t help, because you still need to understand the whole before diving into a specific area of code. And you still need to deploy everything even if you changed a small part of it.

A word about microfrontends

Microfrontends are a new concept in the frontend world. Martin Fowler defines them as "an architectural style where independently deliverable frontend applications are composed into a greater whole". This is a great definition as it combines the idea of composing applications from smaller components, and of deploying (delivering) the components independently of one another.

The common thinking currently is that each microfrontend should be manifested as a separate browser component that runs in the browser independently of the other microfrontend browser components. See this article as an example of this kind of thinking.

At Roundforest we believe that the walls between the different components in the browser are too thin and generate components that are not loosely coupled, and thus difficult to change. We chose an alternative way to modularize the application, a modularization that is natural in the web world: at Roundforest, each page in the application is a different microfrontend.

Microfrontends at Roundforest will be the subject of a separate blog post sometime in the future.

Modularizing the Code

Now it’s time to talk about the three pillars of development at Roundforest: modularizing the code, reducing developer friction, and the monorepo.

But first, a word about “modules” in the JavaScript world.

Roundforest develops in JavaScript and TypeScript, both backend and frontend. In JavaScript (and thus in TypeScript), a module usually refers to a file that is imported (or required). But the module we are discussing in the blog post is not that, but rather is the unit of modularization in modular code. In JavaScript, what we refer to in this blog post as a “module” is actually a package. In JavaScript, it is a package that is the unit of modularization.

So for the rest of this blog post, we will be using “Modules” and “Packages” interchangeably. In this blog post, “Module” does not mean a JavaScript file, but rather a unit of modularization.

Back to our pillars. We’ll start with modularizing the code. In the previous section, we talked about modularizing the architecture using microservices and microfrontends. Is this enough? A microservice and microfrontend architecture? Nope. Two additional things are needed for a complete solution to scaling the codebase:

• Modules should be independent
• Library Modules

Let’s discuss those.

Modules Should Be Independent

This is crucial. Microservice/microfrontend packages should be:

• Independently coded
• Independently tested
• Independently built, and…
• Independently deployed

Independence means that a developer can go into the codebase of a package, and only that codebase, and after a short time of reading the code, can start coding. Why does independence help with that?

• Because the package code is independent of other package’s code, it can be understood independently. And because there’s not a lot of it, it can be easily understood.
• Because the package tests are independent of other packages, they can be easily run, and because there’s not a lot of code, the tests can be easily understood, and even more importantly: run quickly (the upper limit at Roundforest would be 2-3 minutes).
• Because the tests are easily run, and the build time is small, they can be easily built.
• Because packages are independent, deployment is not a problem and can occur whenever we want and however much we want.

Moreover, because the deployment is independent, after coding and running the tests, building and deploying the microservice or microfrontend can be done independently of others. So a developer can wrap their head around everything—coding, testing, and deploying—pretty easily.

Independence is VERY important, and a modular code architecture is not modular if the packages are not independent. If the packages are not independent, then we’re back in a monolithic codebase, so it is VERY important to isolate the code of the different packages.

It is the independence of the packages that empowers developers to do changes in the code without the need for various gatekeeping processes that slow down the development process and hamper the ability of the business to move forward quickly.
Independence of packages also enables another important aspect that is important to us at Roundforest: the ability to experiment with technologies, play around with them, reject the bad ones, and accept the good ones. Since each package is independent, we can try things at a package level, which is simple and fast. At Roundforest we do not shy away from these experiments. On the contrary, we value them and believe they are important for a healthy engineering culture.

Library Modules

But what if microservices or microfrontends have common code? For example, all microfrontends need a shared set of components (buttons, tabs, form controls…). Microservices also have common code, such as error handling, logging, and other concerns. Because all Roundforest modules are independent, we cannot just have a folder full of shared components that all the microfrontends/microservices reference.

We could, of course, copy/paste the shared code into all the microfrontends/microservices, but that’s just, well, not done!

Instead, some of the packages are not microservices or microfrontends. They’re just library packages that package a set of functions/classes/components into an NPM package that gets published to our private NPM registry. Any microfrontend or microservice package that wants those shared components just references them in the package.json.

The fact that a microservice is dependent on a library package does not mean that it has stopped being independent! There is a dependency on the NPM package, but there is no dependency on the code. This is not a theoretical difference. Depending on the NPM package means that it depends on the API of that library, and does not need to concern itself with how it is implemented, how it is built, or how it is tested. The dependence is an API dependence, and not a code dependence.

For example, if I change the microfrontend code, then I don’t need to build and run the tests on the shared components (you do have tests on the shared component library, right? And visual tests?). That’s a whopping time saver. This also means that a developer that needs to concentrate on the microfrontend code doesn’t need to also think of the shared components.

The inverse is also true, and just as important: if I change the shared library package, it has its own set of tests, and I can develop it independently of any other package, without worrying that I will break packages that are dependent on it. The independent testing here is very important for a shared library. Many times I’ve seen shared libraries that rely on the application itself to be tested, because anyway all the tests are run, even when only the shared library is changed. This is bad for many reasons:

• Test times becomes a problem
• The shared library becomes more and more coupled to the application
• It is difficult to understand and develop the shared library separately from the application.

So if you have a shared library, and if it’s not independent, you’re not gaining a lot from it. At least not as much as a Roundforest developer!

Next

Is Modular Code Architecture enough? No! We need mechanisms that remove developer friction. Read all about it in the next part of the series, here.

(Want to be as independent a software developers as ours are? Check out our open positions here)

photo by Raphael Koh on Unsplash
This post also appears on Gil Tayar's private blog: http://gils-blog.tayar.org/posts/scaling-the-codebase-avoiding-spaghetti-code-part-2/