This is the first post in a four part blog series:
- Introduction (this part)
- Modular Code Architecture
- Reducing Developer Friction
- Wrapping everything in a Monorepo
How does one build a scalable development team? How does one ensure that even if the size of the codebase of a company grows and the size of the development team grows, the development velocity stays the same and does not deteriorate?
Everybody’s been there: a company that started lean and fast, but now has tens or hundreds of developers, millions of lines of code, and a development velocity that went down the drain. Everybody’s stuck in a viscous fluid, only able to move slowly.
Roundforest is an agile company. And, no, not a “Scrum” company, but a truly agile one that values a discovery process in which we implement rapid changes to the product and use a/b testing to figure out which changes are the best to the user. In this environment, using regular development processes, spaghetti code becomes inevitable.
How do we avoid this scenario? And how do we avoid it while still allowing for rapid changes?
That’s the question I’ve been pondering and dealing with for the past 20 years: since being CTO of my own startup in the early 2000s, through seeing how Wix deals with the problem, implementing a solution to the problem at Applitools, and refining the methodology and tools here at Roundforest.
The solution has three main pillars:
- Modular code architecture
- Reducing development friction
- Monorepo
TL;DR
(if you don't want to read the whole thing, here's a succint summary)
At Roundforest, we pride ourselves on code that grows and grows without turning into a big ball of spaghetti code. We do this by using a modular code architecture that enables the code to be split into many independent packages that are easily developed.
This enables the development to be focused, controlled, fast, safe, and frequent, and in general is part of our goal to remove the friction from the development process.
And all this we do in a monorepo, which houses all the company’s code.
A Day in the Life of a Roundforest Developer
Before we start, let’s peek over the shoulder of a Roundforest developer while she’s doing her daily work.
Good morning! Shawn, a hypothetical software developer at Roundforest starts her day at the coffee machine. But let's ignore those parts of the day, shall we? 😁
Shawn’s task is to fix a bug in our “Best.io” app, an app that shows recommendations for products in categories like “Laptops”, “Vacuum cleaners”, and “Headphones”. This app not only shows the categories, but also searches the Internet to look for information about how good the products are, stores information from Amazon and other marketplaces about the product, and runs complex algorithms that take all this information and figures out which are the best products for each category.
Shawn figures out that the problem is in the “Amazon Product Catalog” microservice. She knows that all Roundforest code is in one repository, which she has on her machine, so she can access the code with no problem. She pops open the Visual Studio Code workspace which harbors all the backend code of the “Best.io” app, and searches for the “amazon-product-catalog” folder.
She starts digging into the code, but realizes that the bug is actually in a library package that the microservice uses. She loves the fact that the microservice code is divided into a few packages, because this ensures that the build times of the microservice itself amounts to only one or two minutes.
She looks into the library package that has the problematic code. She’s never seen the library's code before, but she knows that it’s just a library package full of functions that the microservice uses. Fairly easy to understand, and the existing tests make it even easier to understand. She knows the problem happens when a certain function is called with a specific set of values, but why is it failing?! She doesn’t know, so she writes a test that reproduces the problem. It’s really easy. Again, because it’s a small library that can be easily understood.
She runs the test, and it fails! Yay! Now she can just debug the function using a debugger. She quickly finds the bug and fixes it. The test passes! Now she can create a new version of the library with the bug fix, and use that new version in the microservice.
She runs a command that builds, tests, and publishes the library to our NPM private registry, and commits her code. The build runs locally, and because the library is small, runs in under 30 seconds. She is confident that the library is working correctly even with her fix because the tests in the library are comprehensive. How can they not be?
Why does she publish the library? Can’t she just use the library folder in the microservice? She could! But that means that the two folders (the library and the microservice) are not independent of each other. And anything that is not independent, is coupled
Now she can go to the microservice code. Because we’re in a monorepo, it’s just a “cd” away, and Visual Studio Code already has the code in the window. All she needs to do to incorporate the bug fix is to update the package dependencies to use the new library. This is anyway done automatically whenever we build the microservice. So she builds the microservice. This builds the code and the docker image. It also runs the tests, and so Shawn is comfortable knowing that her library code change didn’t break anything.
The microservice build took under two minutes. How come? Well, it’s a small service, and parts of the code is in library packages that are already built and tested. The build also publishes the docker image to Roundforest’s registry, so she’s ready to deploy the package. This she does by “cd”-in to the microservice’s deployment package and running the deploy command. This updates the Kubernetes YAML files to the new version of the microservice, and deploys the YAML file, which updates the microservice in production. The whole deployment took around a minute.
Shawn is pretty confident that the build is OK, because production tests are automatically run as part of the deployment, and those passed. But just to be sure, she does a manual check that ensures her that the fix is in production and working. She also opens up the dashboard for a few minutes and monitors the microservice to be sure that no errors are happening.
Now… where’s that cup of coffee she needs?
Next
Wonder why she's working this way? How did she achieve this nice workflow that she has? Read on in the next part of the series, here.
(We're hiring more "Shawn's" to our team. Interested? Check out our open positions here)
image by freeimageslive.co.uk - creator
This post also appears on Gil Tayar's private blog: http://gils-blog.tayar.org/posts/scaling-the-codebase-avoiding-spaghetti-code-part-1/
