Introduced in 1982, the ZX Spectrum was a computer that sparked the home computer revolution in Britain and, due to its affordability, in many other European countries. The basic version had 16 KB of RAM, while the enhanced version boasted an immense 48 KB.

Though I’ve never owned a ZX Spectrum, it is close to my heart. It was the first computer I had a chance to work with. The computer club I was going to as a kid had a dozen of these computers, and we couldn’t wait for the teacher to finally stop talking and let us play some games.

Fast forward 40 years, and while browsing eBay, a brand-new replacement case for a ZX Spectrum showed up in my search results. The memories came back, and I impulse-bought it immediately. I didn’t have any plan for it, but once I got it, I knew I had to put it to good use, i.e., build a ZX Spectrum replica.

Getting started

A nice case is a good start, but what goes inside is key. I found a few interesting options, but the Harlequin 128K Kit from ByteDelight was by far the best. This kit contained everything I needed to build a fully functional ZX Spectrum.

When I received my Harlequin 128K Kit, I felt overwhelmed. While the ZX Spectrum had only a handful of chips (not counting RAM), this kit had more than 50. There is a good reason for this - the kit uses widely available, off-the-shelf chips to emulate the ZX Spectrum’s ULA (Uncommitted Logic Array) specialized chip that went out of production almost 40 years ago.

I had never done much soldering, and the 50+ chips alone meant hundreds of solder joints. But there were also resistors, diodes, transistors, etc. The website promised it would be a fun challenge and that basic skills and creativity were enough to pull it off. These were encouraging words, but what the website didn’t mention was that a single mistake would ruin it all.

To make the assembly process easier, the kit parts were divided into 40 or so clearly labeled bags and provided with the recommended installation order. The detailed assembly guide included additional hints and called out easy-to-miss gotchas. Given my skill level, I found the kit’s organization extremely helpful.

First Try

The instructions said the soldering would take a few hours. Indeed, I spent a few hours a day for a week building the board. I followed the assembly steps very closely because I knew troubleshooting any non-obvious issues would be beyond my abilities. Eventually, I emerged from my garage with the board, anxious to try it.

I connected it, and what I saw made my heart sink.

If the board had been completely dead, I might have hoped I had missed a connection, but what I saw indicated a subtle issue I doubted I could fix. A quick Internet search only confirmed my suspicion. A few people reported similar symptoms, but no one could propose a reliable fix.

I verified that all the chips were inserted correctly and pressed them down firmly to ensure good contact. I inspected my soldering but didn’t find any missed solder joints. I double-checked the jumpers and found that a couple of them controlling the video settings were misconfigured. I corrected them, but nothing changed. Frustrated, I gave up.

Round 2

The problem haunted me. A week later, I went over everything again and confirmed I hadn’t missed anything. As I was out of ideas, I started playing with jumpers for ROM selection. To my astonishment, one of my changes produced this screen:

This meant the board had been working fine all along! It was just trying to run some garbage instead of a valid ROM program.

ROM

I turned to the guide to understand what was happening. Due to copyright, the kit doesn’t ship with the original ZX Spectrum ROM. Instead, it comes with the AM29F040B EEPROM, which is big enough to hold up to 8 different ROM images. The board uses jumpers to select the image to run. Apparently, some of the banks in my EEPROM didn’t have a valid ROM image, and my initial configuration happened to point to one of these banks.

Now that I understood the issue, the fix was easy: I needed to burn the ZX Spectrum ROM into my EEPROM. The only problem was that I didn’t have an EEPROM programmer. After some research, I settled on the XGecu T48 EEPROM programmer. When I got it, I was unpleasantly surprised: the official software only supported Windows, but I am a Mac user.

I sacrificed a lot for this project, but having to install Parallels would be too much. Fortunately, I found an open-source command-line tool called minipro that offered what I needed.

The Harlequin 128K board I built was compatible with both ZX Spectrum and ZX Spectrum 128. Because my EEPROM could hold multiple ROM images, I burned them both so I could easily switch between them.

Visual Minipro

While the minipro tool did the job, setting it up and understanding the options took some effort. A UI version of the tool would make everything so much easier. As nothing like this existed, I decided to build it myself. Because my tool is a minipro GUI wrapper, I called it Visual Minipro and made it available on the Mac App Store.

Finishing the build

I was delighted to see my board up and running, but two problems bothered me: the video quality was pretty bad, and I couldn’t use a joystick.

Video

The Harlequin 128K board offers two video outputs: composite and RGB. The RGB signal is intended for use with SCART, which has never been a thing in the US. Since none of my monitors supported SCART, composite video felt like a better choice. Besides, exposing the RGB socket would require cutting a hole in my case, which I didn’t want to do.

To get NTSC timings over composite video, I installed the NTSC crystal that came with the kit. It did the trick, but the result wasn’t great. The picture on my 1084S monitor was blurry, and when I tried to convert it to HDMI using my Retrotink 2X Pro converter, I got no color.

Joystick

In the 8-bit era, a joystick was a must-have device. Most games couldn’t even be played without a joystick. While the ZX Spectrum didn’t support a joystick out of the box, third-party joystick interfaces quickly filled the gap. The most popular one was the Kempston joystick interface.

The Harlequin 128K board has built-in support for the Kempston joystick interface. This is nice, but there are two problems: the ZX Spectrum case lacks a factory-made hole for a joystick port, and the board lacks mounting points for the DB9 joystick socket. So, even if I had modified the case, that case might not have been strong enough to provide solid support for the joystick port. Besides, I knew the DIY joystick hole would look ugly. So, I decided to look for something different.

ZX VGA Joy to the rescue

One popular way to get HDMI output on the ZX Spectrum is ZX-HD, an HDMI interface for the ZX Spectrum. It is a fine device, but it wouldn’t solve my joystick problem. So, I kept looking for alternatives and found something that fit my needs perfectly: ZX VGA Joy. This device could output digital video over HDMI and provided a joystick port compatible with the Kempston interface. As a bonus, it also had the reset button.

I ordered it, and couldn’t wait to try it. When it finally arrived, I wasn’t able to make it work with my setup reliably - the screen would go black soon after turning the Spectrum on. Quick debugging proved this was an issue with the computer - the composite video worked just fine. After reading more about the inner workings of ZX VGA Joy, I got a hunch about what was happening. ZX VGA Joy relies on PAL timings to generate the HDMI output. But my computer had the NTSC crystal installed. My hypothesis was that this difference was responsible for the glitch because the PAL and NTSC frequencies differ.

My guess was correct. Replacing the NTSC crystal with a PAL crystal indeed fixed the blank-screen problem, but it came at a cost. After installing the PAL crystal, I lost the ability to use the composite video. This wasn’t a big deal, though. Given the low quality of the composite signal, I wouldn’t want to use it anyway.

Final result

Here is the final result after putting everything together (with a modded Nintendo NES controller in lieu of a joystick and the original SimCity game humming happily on 48 KB of RAM).

Conclusion

There is no practical reason to build a 40-plus-year-old computer today. To be honest, even when I was starting, I didn’t expect to really use it. However, the process of building it, the interesting technical challenges, and the satisfaction of seeing it come to life were all well worth it.

To make sure I am making steady progress, I challenged myself to write one piece each week. I decided to write in public to guarantee the quality of my posts. My reasoning was simple: publishing something subpar because I didn't put the honest effort into it would feel very uncomfortable, and I wouldn't do it.

Choosing a newsletter as my medium was an easy decision. With 25 years of experience as a Software Engineer, I have plenty to share. Additionally, a newsletter was the ideal format for maintaining quality and consistency.

Today, I am thrilled to report that I successfully completed my challenge. I published my first post on 11/29/2023 - exactly a year ago, and I wrote a new post every week after that.

What I learned

My journey taught me a few important lessons about writing. Here they are:

Writing takes a lot of time

I knew that writing was time-consuming as I had written tens of posts on my personal blog before. The big difference was that before, I could spend weeks on a single post, but now I was on a schedule. I had about a week to write and publish a post. I found this surprisingly challenging. Initially, I could barely keep up with my plan despite spending most of my weekends writing.

Writing consistently is hard

Consistently doing anything that demands a significant amount of time is difficult. Writing is no different. It is so easy to justify giving up when a family or a job emergency happens (that tree that fell on our house was no fun, and the clean-up took weeks). It is so tempting to take a week or two off for a family vacation. But I knew that skipping once would only make it easier to skip again.

The English language

In the past year, thanks to intentional practice, I learned more about written English than I had since I moved to the United States in 2006. My two main resources were books (including "On Writing Well" by William Zinser and "The Elements of Style" by William Strunk Jr. and E.B. White) and Grammarly. (Grammarly proved especially helpful in identifying mistakes I didn't realize I was making consistently.) I also used Chat GPT a bit. It was useful in some aspects, like finding synonyms and generating images, but I wouldn't say I liked the content it produced. It was either very pompous or too vulgar. And it just didn't sound close to how I write.

The results

Here are the results my practice yielded:

• writing a post takes me now half the time it took when I started

• newer posts read easier than the first ones I published

• my writing at work became more concise and better tailored for the audience

I also have a library of posts to which I can point my mentees and team members.

What's next?

Even though I completed my writing challenge, it doesn't mean my writing couldn't improve further. However, going forward, I am going to make it less intense.

The weekly writing cadence left me little free time to do other things. I have a few topics I have wanted to research for a long time but couldn't put enough continuous effort into making solid progress. I also want to spend more time on some of the hobbies I neglected in the past 12 months. To make room for this, I decided to publish my newsletter less frequently. Furthermore, some upcoming posts might relate to my research topics instead of the Software Engineering career development advice I have been focusing on until now. I hope they will still interest you despite being different. Thank you in advance for your readership.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

Software is no different. The complexity of software systems has been only growing. As a result, the ability to quickly build mental models and use them to reason about these systems is an important software engineering skill.

Building good mental models is important because they allow us to better communicate with other software engineers, explain our ideas, and understand the impact of our changes on our software.

Building mental models

Reasoning about software systems requires building mental models that represent them. The granularity of these models depends on their purpose. Often, one level of granularity is not sufficient. The most effective software engineers can quickly build a few models and switch between them depending on the situation.

For example, if your team owns a few services, you can draw a lines-and-boxes diagram where boxes represent services and lines represent dependencies. You can then zoom in and build a model for each service. These service-level models could illustrate interactions between the libraries a service consists of. If you want more details, you can create a class diagram. You can continue zooming in and focus on methods, code blocks, statements, etc.

Each of these models describes your system at a different level of granularity and has a unique set of applications. The high-level model could be useful to troubleshoot larger outages (or when talking with your director) but is unlikely to help you fix a small bug. The more detailed models are best suited for solving gnarly issues but won't be helpful when explaining your infrastructure to other teams.

Building models covering different aspects of the same system is also common. If you want to analyze your system from the security perspective, your model will include different details than when focusing on performance.

Caveats

Mental models are so natural to people that we often forget about their flaws.

All models are wrong

Models, by definition, ignore details. As they only capture certain aspects of reality, they are inaccurate. Furthermore, some relevant information is often omitted because it doesn't fit the model. Edge cases are an excellent example.

For instance, when software developers explain how their code works, they rarely mention error cases. They focus on their ifs and fors and the program flow but omit exceptions because exceptions make their model murkier. The problem is that error scenarios are important. Incorrect or missing error handling is a common cause of system outages.

Models get more wrong with time

The world, including our software systems, is in constant flux. But mental models don't automatically keep up with changing reality. Outdated mental models lead to misunderstandings and bad decisions.

I experienced this very problem recently. I started working on a feature that depended on a system I had never seen changing. Everything was going swimmingly, and I only needed to tie up a few loose ends related to some new data requirements. But when doing this, I discovered, to my dismay, that the system I depended on had recently changed. It had been updated to accommodate the same data requirements I struggled with. Making my feature work now required additional information I didn't have. Plumbing this data meant revisiting my implementation. Working off of an outdated mental model cost me implementing my feature twice.

No two models are identical

Building mental models requires deciding which details are important depending on the purpose of the model. However, even if the purpose of the model is well understood, different people will consider different information relevant.

Also, mental models built at different times will naturally differ because they capture different system versions.

The interesting fact about this phenomenon is that the overlap between mental models built by different people is usually significant. The differences are often discovered unexpectedly, e.g., when discussing small but important details.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

Most developers learn this truth the hard way. I did when I implemented a feature based on incorrect assumptions, and the only way to save it was to rewrite it.

While junior engineers often fall into the trap of jumping straight to coding without thinking about the problem, more senior engineers fall into a different trap: overthinking.

Senior developers are routinely responsible for projects requiring a handful of engineers and a few months to complete. There is no way to execute these projects without careful thinking and planning. Even if you wanted to start coding on the first day, you couldn't. You simply wouldn't know where to start.

The top priority in the initial phases of bigger software projects is to sort out the ambiguity that blocks development. The most common way to do this is to devise a design that will guide the implementation.

The tricky part is that figuring out the design is by itself an ambiguous problem. There is usually more than one way to implement the solution, and many constraints and requirements are unclear. Even estimating how long it will take to prepare the design can be difficult.

All these uncertainties put pressure on the engineer(s) responsible for the project. Making a bad decision can lead to wasted time (amplified by the size of the team) or even a project failure.

When stakes are high, it is natural to proceed carefully, explore potential problems, and work with others to identify gaps that may otherwise go unnoticed. However, setting limits for these activities is crucial. Failing to do so will inevitably lead to overthinking, which can also put the project at risk due to:

• Delays - analyzing every imaginable scenario takes a long time and will eat into development time, making meeting expected timelines impossible.

• Overengineering - trying to address minor or hypothetical issues leads to overly complex designs that are hard to implement and expensive to maintain.

• Missed opportunity - endless discussions, revisions, and feedback rounds steal time engineers could spend on other project activities or elsewhere.

How do you know if you are overthinking?

One of the problems with overthinking is that the line between productive analysis and overthinking is thin, and it is easy to miss this line. However, there are signs of getting there:

• Although all critical requirements have been satisfied, new requirements are being added.

• The same topics continue to be discussed repeatedly without reaching any resolution.

• Edge cases, minor issues, and esoteric scenarios start to dominate the discussion.

• The debate moves to future scenarios that are out of the scope of the project at hand.

What to deal with overthinking?

It is important to understand that the goal of the planning and design phases is not to identify and solve all possible problems. First, it is impossible even to list all the issues. No matter how much time you spend thinking you will miss something. Second, many problems will never materialize, or if they do, their impact will be minimal. To focus on what's important, create a list of requirements, decide which are critical (the list should be short), and satisfy those. Leave out the remaining ones and revisit them if they become a major problem.

Many problems have no one correct or even best solution. No amount of debate is going to change that. It may take time to arrive at this conclusion, but once it is settled, you must pick one of the alternatives and live with the consequences.

If you are not sure, prototype. An hour of coding can save you a few hours of meetings. Code does win arguments.

Design for your current needs. If you don't operate at Facebook's scale, don't design for Facebook's scale. Don't build abstractions for your hypothetical future scenarios.

Accept the fact that things may not work out. Fortunately, you are not pouring concrete. This is software, and the ability to modify it is one of its greatest advantages. You can build small and evolve your solution when your needs grow.

Build vertically. A small feature working end-to-end will allow you to discover issues early and course-correct. If you build horizontally, you won't see gaps until very late when all the layers are ready. Fixing bigger problems at this stage will be an undertaking.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

Ramping up on your new team’s code base is difficult, but knowing how to work effectively in it is crucial to your success. So, how do you do it quickly?

My mid- and early senior developer years were intense. Due to a mix of reorgs and personal interests, I found myself on a new team every year or so. As a result, I had to learn new codebases in quick succession. They included .NET System.Xml, OData, Entity Framework, Entity Framework Designer, ASP.Net SignalR, ASP.Net Core, and the Alexa mobile app, and most of them were over one hundred thousand lines.

The first couple of transitions were slow and overwhelming. But they helped me develop a strategy I used later to quickly get up to speed on new codebases.

Get your hands dirty ASAP

The sooner you start working actively with the code base, the sooner you will get productive. Reading documentation can be helpful, but nothing can replace the hands-on experience.

When I joined Amazon, I asked my manager to assign me a simple bug to fix on my first day. Even though the fix amounted to a single line of code, it took me a few days to send it for review. It might seem long, but this quest was not about fixing the bug. Rather, it was about forcing myself to set up my development environment, learn how to build and run the code, write unit tests, and debug our product.

Review code

Each team has its way of letting members know there is a new PR (Pull Request) for review. It could be email, chat, or review tool notifications. Whatever it is, subscribe to this channel and start reviewing PRs. You won’t understand much initially, but you may still catch some bugs (e.g., off-by-one errors). More importantly, these reviews will allow you to ask questions and gather a broader context of what the team is working on.

Identify code that matters

The 80/20 rule does apply to codebases. This is especially noticeable in the bigger ones, where most code changes developers make are concentrated in one area. Knowing which code it is allows you to focus your ramp-up on the area you are likely to work on soon.

There are a couple of easy ways to tell which code is in the top 20%:

• paying attention to code reviews

• checking the commit history

Take notes, draw diagrams

I discovered that taking notes and drawing diagrams is a very effective way to grok the most complex parts of the code. Call graphs, class diagrams, and dependency graphs all help organize the information and are great reference material in case you need to refresh your memory.

After I joined Amazon, I drew diagrams of a few areas of code I couldn’t understand. They became an instant hit. Even people who had been on the team for much longer than me wanted a copy.

Debug code

Reading a new codebase is like reading a book in a language you barely know. You can do it, but it is excruciating.

For me, one of the best ways to overcome this is to step through the code with the debugger. Initially, I have no idea what I am looking at. But if I follow the same code path a few times, I begin to recognize code I have already seen, and soon, everything starts to fall into place.

To get the most out of my debugging sessions, I do two things:

• I continuously inspect the state - the stack trace, parameters, and local and class variables

• I take notes and draw diagrams

Using the debugger to learn the code base is slower and narrower than reading the code. But it is also much deeper. In fact, when debugging, I often realize that the understanding I gained from reading the code was incomplete or sometimes even incorrect.

Read documentation

Reading documentation can help accelerate your onboarding. It could be especially useful when it comes to the high-level architecture and concepts that are hard to deduce from code.

However, you should take documentation with a grain of salt. It is often sparse and outdated. But this could be good news for you. Updating the documentation as part of your ramp-up could be a great contribution to your new team.

Join on-call rotation

Joining the on-call rotation to accelerate team onboarding might sound extreme but I did it on my first team at Facebook. I did it because I was concerned that my ramp-up was slow, so I decided to push myself. I learned more about our services this week than in weeks before. After my shift ended, I knew what services our team owned, their dependencies and where to find their code. Alerts immediately pointed me to the hot code paths, and troubleshooting issues forced me to dig into the code.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

If not coding, then what?

If coding is not the skill to grow beyond senior levels, what skill is it?

This question has no correct answer, as no single skill can elevate you to the staff+ levels.

However, software development is a team sport, and successful senior engineers must focus on many areas besides coding. They are often responsible for projects spanning one or more teams. They drive the design, collaborate with partner teams, communicate progress, etc. Doing all this work effectively requires good communication skills, especially writing (which I consider one of the Universal Skills.)

Why writing?

Writing clarifies thinking and promotes the exploration of ideas. I don't know how many times I thought I understood something, only to struggle to summarize it in writing. But once I succeeded, I had a much deeper grasp of the concept and noticed new insights I hadn't considered before.

Thanks to its durability and asynchronous nature, writing is also a great way to scale. You can write something once and refer to it later. Your readers can benefit from it even if you are not around. Here are a few examples from the software engineering field:

• Project execution plans are useful for aligning all interested parties: the team that will execute the project, partner teams, your manager, etc., without having to talk to them individually.

• Documentation helps avoid explaining the same concepts again and again. It protects the team from scrambling when a key team member leaves the project or the team (see also: bus factor)

• Design documents allow for gathering feedback without holding a meeting for all interested parties. They are also an invaluable resource to understand why certain design choices were made and what alternatives were considered.

Writing is difficult

Writing is not natural for most people. Making the content clear, concise, and well-organized is grueling work.

I often see software developers dismay when I ask them to write a rollout plan or a design doc. Some tell me they were relieved to graduate from college because it meant they would never have to write again, and I am shuttering their world.

There are also other reasons why writing is difficult. Many developers have to write in a non-native language. But even native speakers often struggle because the way of writing they learned at school does not serve them at work.

Opportunities to practice writing

Even though writing becomes important gradually, it doesn't mean you should wait to improve it. On the contrary, the sooner you start, the better. Fortunately, every developer has plenty of opportunities to practice writing on the job.

Emails

Emails are everyone's bread and butter these days. However, many emails are hard to read and understand and, as a result, fail to achieve their goal.

In my first job, our manager asked us to send a weekly email summarizing what we worked on and accomplished in the past week. I was proud of my reports: they were very detailed and explained everything. Despite these emails, my manager kept asking me what I had been working on. When I saw one of these emails years later, I understood. He never read them. I couldn't blame him - it was an unbearable wall of text.

Memos / Announcements

Posts, memos, and announcements meant for a wide audience need to be tailored to that audience. Otherwise, readers won't understand them and will give up reading them.

I recently read a post from my co-worker reporting on the status of our project. The audience of this post was broad (more than 150 people) and included managers, directors, and partner teams. The technical details in this post left me lost despite my heavy involvement in this project. I can only guess what others took away from this post.

Design documents

Good design documents explain complex topics using simple language. This combination makes them hard to write, but the payoff is worth the effort. Confusing design documents lead to lengthy discussions, feedback on unimportant matters, and frustration.

I once asked a junior engineer to write a design document explaining how he plans to implement a feature we promised to deliver. What I got was an untitled Google Doc with no text and two pictures - a diagram and "The Starry Night" by van Gogh. While I have nothing against "The Starry Night", the document didn't give me the faintest idea about the design of the feature, assumptions, and considered alternatives.

Code review feedback

The main purpose of sending code for review is to gather feedback. But giving short, clear, and actionable feedback professionally is an art. The conclusion: if you want to improve your writing, you should review a lot of code (and provide feedback).

Code comments

I am not a huge fan of writing code comments, but in some situations, they are warranted. Unfortunately, many code comments are so poorly written that it is sometimes hard to tell if they are there to help you or make you more confused.

The main challenge with code comments is that they need to be short to not overshadow the code but must clearly explain intricate ideas that the code cannot express. These requirements make writing code comments good practice.

Documentation

Writing documentation is one of the least favorite tasks software developers want to do. Yet, it often is one of the most impactful they can do. Good documentation helps put out on-call fires faster, makes onboarding new team members easier, and reduces randomization caused by repeatedly answering the same questions. By writing documentation, you help your team achieve more and polish your writing skills.

Bug reports

If you want someone to do something for you, you need to make it as easy as possible for them to do it. If you don't, what you are asking for will take a long time or will never get done.

This rule applies perfectly to bug reports. If you encounter a bug that blocks your work, writing a clear bug report dramatically increases the chances of getting the issue fixed. Despite this, many reported bugs are incomprehensible.

At Microsoft, I worked on a few high-profile open-source projects like Entity Framework or ASP.Net Core. As thousands of developers used our products, we received a decent number of bug reports. Unfortunately, we often couldn't understand what issue was being reported, how to reproduce it, and the expected behavior. Following up on these issues was painful. The back-and-forth took weeks. The "bugs" slipped from release to release while we were waiting for the details we requested. Eventually, we closed most of these bugs without resolution as it was hard to prioritize them over other issues we could immediately investigate and fix.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

• Inability to execute independently: making even the smallest decision without involving your manager or the tech lead will be hard if you don't understand the bigger picture. You will get stuck if you can't get hold of them. It will also be difficult for you to demonstrate you know how to solve problems at your level and are ready for bigger challenges.

• Communication gap: your manager or senior engineer may unintentionally give you incomplete or incorrect information. If you don't have enough context, it will be hard to notice this. You will struggle to complete the task, but once you finally do, it may turn out that what you built is not what they hoped for and needs to be redone.

• Hindered innovation: unawareness of where your work fits limits your ability to propose solutions beyond what you're asked to do. Sometimes, the approach you're instructed to follow may not be the best solution to the problem, but exploring alternatives is impossible if you don't understand the broader context.

• Incorrect prioritization: working on a task without knowing its purpose may lead to neglecting this task and unknowingly delaying work that depends on it.

How to understand the bigger picture?

The easiest way to understand where your work fits is to ask your manager or the tech lead. They are responsible for what the team needs to deliver, so they should be able to explain this instantly.

Your question may, however, come to them as a surprise. They probably assume everyone on the team already understands the purpose of their work. In my experience, this is not always the case. The bigger and more complex the project, the harder it is to connect the dots.

When talking to your manager or tech lead, you can start small, but it is important to go deep. Start asking about your task. You may hear that it contributes to a project the team is working on. An answer like this is not very helpful but could be a great starting point. It allows you to drive the discussion further and ask more interesting questions like:

• Why are we working on this project? Why is it important?

• What metrics is this work expected to move, and how?

• How does it support the company's goals and priorities?

• What projects did we decide not to pursue to fund this work (a.k.a. opportunity cost)?

A different way to understand where your work fits might be by talking to your product manager or people from the UX (User Experience) or marketing team. Because of their different perspective, they can teach you things you would never learn from fellow engineers. The challenge with this approach, however, is that you need to be able to explain your role in the project to them.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

The three-line execution graph

Executing multiple projects of the same priority at the same time looks like this:

Initially, there are two projects: Project A and Project B. You start working on Project A, but after a while, you receive a call from the customer inquiring about the progress of Project B. To make this customer happy, you switch to project B. In the meantime, a new interesting project, C, pops up. It is cool and seems small, so you pick it up. Your manager realizes that project A is dragging and asks about it. You somehow manage to finish your toy project C and move to A, which is well past the deadline. Then you pick B again.

If you didn't jump from project to project, the execution could look like this:

If you compare these graphs, three things stand out in the second scenario:

• Overall, the execution took less time. Resuming a paused project requires time to remember where the project was left off and get in the groove, i.e., to switch the context, which is time-consuming.

• Projects A and B finished much quicker than in the first case. While you didn't make customer B feel good by saying you were working on their project, ultimately, the project was completed sooner. In fact, both projects, A and B, were finished much sooner than they would have if you bounced between them.

• Project C came in late, so it should wait unless it is a much higher priority than other projects. Otherwise, it disrupts the execution of these projects.

I know that life is not that simple. Completely avoiding context switching is rarely possible. But if you can limit it, your productivity will dramatically increase.

Does it mean you should only work on one thing at a time?

In the past I thought a good solution for junior engineers to combat context switching was to ask them to work on just one project at a time. But this idea has a serious drawback - projects often get stuck due to factors outside our control. If this happens and there is no backup, idling until the issue gets resolved is a waste of time.

Having two projects with different priorities works best. You execute on the higher-priority project whenever you can. If you can't, you turn to the other project until the main project gets unblocked.

What I like about this approach is that it is always clear what to work on: the higher priority project wins unless working on it is impossible.

Falling back to the lower priority project means there might be some context switching. While it is not ideal, it is better than idly waiting until the issues blocking the main project are resolved.

But my TL (Tech Lead) always works on five projects!

Indeed, experienced senior and staff engineers often work on a few projects at the same time. In my experience, however, it is a different kind of work. It might be preparing a high-level design, working on an alignment with a partner team, breaking projects into smaller tasks, and tracking the progress.

The secret is that most of these activities don't require as much focus as coding. Handling a few of them at the same time is much more manageable because the cost of switching contexts is much lower.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

Some team members were still around, and I turned to them for help. I learned that the impacted service shipped merely a week before, and barely anyone knew how it worked. The person who wrote and shipped it was on sick leave.

It took us a few hours to figure out what was happening and to mitigate the outage. This work made one thing apparent - this service was not ready for the prime time.

In the week following the incident, we filled the gaps we had found during the outage. Our main goal was to ensure that future on-calls wouldn't have to scramble when encountering issues with this service.

But the bigger question left unanswered was: how can we avoid similar issues with any new service or feature we will ship in the future?

The idea we came up with was the Service Readiness Checklist.

What is the Service Readiness Checklist?

The Readiness Checklist is a checklist that contains requirements each service (or a bigger feature) needs to meet to be considered ready to ship. It serves two purposes:

• to guarantee that none of the aspects related to operating the service have been forgotten

• to make it clear who is responsible for ensuring that requirements have been reviewed and met

When we are close to shipping, we create a task that contains a copy of the readiness checklist and assign it to the engineer driving the project. They become responsible for ensuring all requirements on the checklist.

Having one engineer responsible for the checklist helps avoid situations where some requirements fall through the cracks because everyone thought someone else was taking care of them. The primary job of this engineer is to ensure all checkboxes are checked. They may do the work themselves if they choose to or assign items to people involved in the project and coordinate the work.

Occasionally, the checklist owner may decide that some requirements are inapplicable. For example, the checklist may call for setting up deployment, but there is nothing to do if the existing deployment infrastructure automatically covers it.

The checklist will usually contain more than ten requirements. They are all obvious, but it is easy to miss some just because of how many there are.

Example readiness checklist

There is no single readiness checklist that would work for every team because each team operates differently. They all follow different processes and have their own ways of running their code and detecting and troubleshooting outages. There is, however, a common subset of requirements that can be a starting point for a team-specific readiness checklist:

☐ Has the service/feature been introduced to the on-call?
☐ Has sufficient documentation been created for the service? Does it contain information about dependencies, including the on-calls who own them?
☐ Does the service have working dashboards?
☐ Have alerts been created and tested?
☐ Does the service/feature have runbooks (a.k.a. playbooks)?
☐ Has the service been load tested?
☐ Is logging for the service/feature enabled at the appropriate level?
☐ Is automated deployment configured?
☐ Does the service/feature have sufficient test coverage?
☐ Has a rollout plan been developed?

Success story

Our team was tasked to solve a relatively big problem on tight timelines. The solution required building a pipeline of a few services. Because we didn't have enough people to implement this infrastructure within the allotted amount of time, we asked for help. Soon after, a few engineers temporarily joined our team. We were worried, however, that this partnership may not work out because of the differences in our engineering cultures. The Service Readiness Checklist was one of the things (others included coding guidelines, interface-based programming, etc.) that helped set clear expectations. With both teams on the same page, the collaboration was smooth, and we shipped the project on time.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

I divide skills into three main categories: company-specific, job-specific, and universal. This categorization makes it easy for me to decide where to invest my time when it comes to skill development.

Company-specific skills

Each company has internal infrastructure, processes, and tools. Knowing them before joining the company is impossible, but everyone must learn them after joining.

Being strategic about what to focus on can save you a lot of time. Company-specific knowledge and skills are, by definition, not transferable. You need to learn them to do your job, but they become useless the moment you move on.

My strategy is to have a solid understanding of company-specific tools and processes but not dive too deeply unless I have to. Being in the dark will slow me down, but drilling into all the details is not much better. Most of these things constantly change, and I will quickly forget what I don't use. Instead, my time is better spent developing other, more general skills.

When I worked at Microsoft, I had a colleague named Ben. Ben was the most knowledgeable person I knew when it came to the .NET Framework build system. Although he was not on the build team, he knew all the scripts, hacks, and environment variables used in this build system. Acquiring and keeping this knowledge fresh ate much of Ben's time. Having Ben around was great for the team. We could (and constantly did) ping him for help with build-related issues. Eventually, Ben moved to a different team outside our organization. His expertise evaporated instantly, even though he didn't go far.

Job-specific skills

Job-specific skills are usually the skills that get us hired. Companies look for people who can hit the ground running, and having skills in demand increases the chances of getting hired substantially.

Job-specific skills are transferable. You learned them before getting hired and may use them in your next job. There is a caveat, though. It is only true if you keep your skills up-to-date. Knowing React and knowing React-as-of-four-years-ago is not the same thing.

I learned that trying to keep skills sharp just by using them at work doesn't always work. Companies rarely move as fast as technology. For instance, the product you work on might have been on the bleeding edge a few years ago. But it got stuck there because there was never a good enough business reason to migrate it to the newest framework version. You still need to maintain it but can't use the latest features.

Another thing to pay attention to is the signs that the technology you specialize in is becoming obsolete. Some technologies are more durable, but some can fade fast. You don't want to wake up one day only to realize that everyone except you has moved on.

CoffeeScript was one of the most followed projects on GitHub in the early to mid-2010s. It was incredibly popular, and there was a lot of hype around it. Today, hardly anyone remembers it.

Even if you got hired for your job-specific skills, it doesn't mean you can't learn new skills on the job. If you want to develop a new skill, you may consider joining a new project or moving to a different team. It isn't always easy because you don't have the skills they need, but it is doable - especially if you are known as someone who learns fast.

And the biggest secret: not only do you develop a new skill that may land you your next job, but you are also paid to do this.

Universal skills

Finally, there are the universal skills. These are skills that never become obsolete. You can use them at your current job, at your next job, or for non-work-related purposes. You can also apply them instantly.

These are skills like writing, effective time management, delivering great presentations, etc.

The biggest problem with universal skills is that developing them never seems urgent. As a software developer, you will not lose your job because you write lengthy emails. But eventually, you may hit a glass ceiling and realize that what inhibits you is not coding but the lack of these universal skills.

—

* - citation needed

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

They are time-consuming and require significant mental effort.

Code reviewing is also not taught at school, and figuring it out on your own is hard work. In this post, I share five crucial habits all great code reviewers I know have in common.

Make the code review about the code

Code reviews are about the code, not the person who wrote the code. If you think the proposed change is incorrect or have suggestions to improve it, then, by all means, leave your comments (just remember to make them professional and high-quality). However, leave out comments that don’t relate to the code under review.

Understand the code and ask questions when in doubt

Understanding the code under review is the foundation of a solid code review.

It is also the most difficult part.

Whenever you have doubts about proposed changes, you should ask the author for clarification. You might not be aware of an assumption the author is making or need help understanding how their changes fit in. However, the difficulty in grasping the changes is also often a sign of a mistake.

Asking a question will prompt the author to explain their thought process. As a result, they will either answer the question and clear up your doubts or realize that something is indeed wrong and needs to be fixed.

Be clear about your expectations

Code reviews can generate a wide variety of comments. Some are nits that you would like to see fixed but are not real issues. Some, however, identify serious problems that must be corrected before merging. If you leave a comment, make sure that the author understands which category it falls into. Doing this will save you and the PR author time and frustration.

Sometimes, you may take on a PR that is outside of your area of expertise. You may realize it only after you already left some comments. You are now in a weird situation: the author expects you to finish the review and approve the PR, but you don’t feel confident you can. If this happens, instead of accepting the change you don’t understand, it is better to leave a comment recommending the author get a review from someone more familiar with the code they are changing.

Side note: reviewing code that is outside of your area of expertise is a good thing. Even if you don’t understand the change enough to approve it, you can still provide useful feedback, identify bugs, and learn something. Just make sure the author does not expect to get your approval.

Cross-check with the existing code

Code reviews show only code that changed. Most of the time, it is sufficient. But sometimes, to understand the change fully, you need to check how the new code works with the code not included in the review because it hasn’t changed.

This idea may be obvious to most developers, but I was surprised to meet some who had never considered checking the existing code when reviewing PRs. In my experience, the biggest surprises are caused by not what’s included in the PR but by what’s missing.

Occasionally, examining the existing code may not give you all the answers. The ultimate weapon for these situations is to use a debugger. You can check out the PR branch and step through the code. It should resolve all your doubts. I resort to a debugger very rarely. It’s almost always easier and faster to ask the author.

Resist the “stamp” pressure

The pressure to merge changes quickly for projects on tight timelines is high. And it only grows as the deadline nears.

During the end game, engineers enter a Pull Request frenzy. Eventually, due to the number of PRs, code reviews often become a bottleneck. So, the engineers try to unblock themselves by asking to “stamp the diff” (i.e., approve changes without looking).

On the one hand it is understandable - no one wants to miss the deadline. On the other hand, these are the times when code changes need even more scrutiny than usual. Due to the time pressure, most PRs are coded very hastily. The changes may not be validated thoroughly (or at all) and the stress only increases the likelihood of mistakes.

While a proper code review takes time, merging code with issues a review could have caught is more costly. At best, fixing the problem will require sending a new PR (which, by the way, will trigger a review). At worst, an embarrassing bug ships to customers.

I remember when one of my teams worked extremely hard to finish a project on time. We were very close, and then one of the team members dropped a 1000-line PR a few hours before the deadline. The manager was trying hard to find someone willing to approve the PR. It wasn’t easy because the PR had a bunch of red flags, like spotty test coverage or many TODO comments, but he eventually succeeded. As soon as our product shipped, we started getting reports from angry customers complaining that important scenarios stopped working. We found that the hastily merged PR was the culprit. The team scrambled to fix the issues, but the damage was done. This one PR cost us the reputation we’d been building for a long time.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

1. Unpredictability

Unpredictability is the number one reason for on-call anxiety. You might be responsible for a wide range of services. They may break anytime for various reasons like network issues, deployments, failing dependencies, shared infra outages, data center drains (a.k.a. storms), excavators damaging fiber-optic cables (real!), etc. On-calls, especially new ones, worry they won't know what to do if they get an alert. How would they figure out what broke? How would they come up with a fix?

What to do about it?

With experience, the unpredictability aspect of the on-call gets easier. But even for the most seasoned on-call engineers, handling an outage can be difficult without the proper tools like:

• Easy-to-navigate dashboards that allow to tell quickly if a service is working correctly and identify problematic areas in case of failures

• Playbooks (a.k.a. runbooks) explaining troubleshooting and mitigation steps

• Documentation describing the service and its dependencies, including the relevant on-call rotations to reach out if necessary

Having a team eager to jump in and help mitigate an outage quickly is priceless. Your team members understand some areas better than you. Knowing they have your back is reassuring.

2. Too many alerts and incidents

The second most common reason engineers fear their on-call is a never-ending litany of alerts, requests, and tasks. If you get a new alert when you barely finished acknowledging a previous one and are also expected to handle customer tickets and deal with requests from other teams, fretting your on-call is understandable. The exhaustion is usually exacerbated by the feeling of not doing a decent job. I was on a rotation like this once. After a while, I realized that everyone, not only me, was overwhelmed. Even though we toiled long hours, most alerts were ignored, customer tickets remained answered, and requests from other teams were only handled after they escalated them to the manager.

What to do about it?

There is no way a single person can fix a very heavy on-call by themselves. They won't have the time during their shift, and by the time the shift ends, they will be so fed up that they won't want to hear about anything on-call-related. There are, however, a few low-hanging fruits that can help improve the quality of the on-call quickly:

• Delete alerts - find routinely ignored alerts and determine if they're useful. If they aren't - delete them.

• Tune noisy but useful alerts - adjust thresholds and windows for flapping alerts, alerts that fire prematurely, and short-lived alerts.

• Get a secondary on-call - a second person could help handle tasks the primary on-call does not have the capacity to deal with (e.g., customer tickets). This could be only temporary.

These ideas can alleviate on-call pain but are unlikely to fix a bad on-call for good. Improving a heavy on-call requires identifying and addressing problems at their source and demands effort from the entire team to maintain on-call quality. I wrote a post dedicated to this topic. Take a look.

3. Middle-of-the-night alerts

Many on-call rotations are 24/7. The on-call is responsible for dealing with incidents promptly, even if they happen in the middle of the night. Waking to an alert is not fun, and if it happens regularly, it is a valid reason to resent being on-call.

What to do about it?

While it may not be possible to avoid all middle-of-the-night alerts, there might be some actions you can take to reduce the disruption. A lot will depend on your specific situation, but here are some ideas:

• Check your dashboards in the evening and address any issues that could raise an alert.

• Increase alert thresholds outside working hours. If your traffic is cyclical - e.g., you have much lower traffic at night because most requests come from one timezone - you may be able to relax thresholds outside working hours. Even if an incident happens, its impact will be smaller. Also, alerts get much noisier if the traffic volume is low (e.g., if you get ten requests in an hour and one fails, you might get an alert due to a 10% error rate).

• Disable alerts at night, if you can. Some outages won't cause any impact unless they last for a long time. For instance, our team was responsible for a service that would work fine even if one of its dependencies was down for a day. This 24-hour grace period allowed us to turn off alerts at night.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

Code reviews

Code reviews are the easiest way to learn from others. Unfortunately, many developers treat code reviews as a chore. They want to spend as little time as possible on them. This way, they lose great learning opportunities.

What makes code reviews special is their interactivity. They allow asking the author about their choices and discuss alternatives they considered. Other participants often leave interesting comments or propose novel alternative solutions.

Reading other developers' code

I learned a lot about programming by reading code written by other developers. I often want to know how a feature or library I use works. Usually, the fastest way to get this information is by inspecting its code.

For instance, TypeScript supported some features, e.g., async/await, before they were added to JavaScript. I was curious how it was possible. So, I wrote short TypeScript snippets and checked how they were transpiled.

Your company's codebase is another great resource to learn from. When I get stuck, I often search my company repository to check how other developers solved a similar problem. Our repo is big, so if I can't find anything useful, I am almost sure what I am trying to do is questionable.

I use the same strategy for my side projects but search GitHub.

If you find reading other developers' code challenging (I sure did), take a look at this post.

Debugging

Stepping through the code, analyzing the stack trace, and inspecting variables will allow you to understand important nuances and easy-to-miss details much deeper. I often fire a debugger if I can't answer all my questions after reading the code.

"Borrowing" code

Let's be honest. Not all code needs to be written from scratch. Sometimes, we just need a boilerplate. But sometimes, we don't know how to solve a problem. In these cases, copying and adapting code is often faster (and easier). It could be the code you wrote in the past or someone else's code, e.g., copied from StackOverflow (I have yet to find a software developer who never copied code from StackOverflow.) AI-powered programming tools are built on this idea. Tools like Github Copilot ask you to constantly vet and adapt the code they generate. Here is the thing, though. You'll learn nothing if you don't try to understand why the code you copied works or can't correctly adapt it.

Programming contests

Advent of Code taught me a lot. It is a light programming contest that takes place every December and consists of a series of small programming puzzles that can be solved in any programming language. I find it an excellent way to keep my coding skills sharp.

Solving Advent of Code problems is a good exercise, but examining other participants' solutions is where the real learnings are. And quite frankly, it can be a humbling experience. The different techniques the participants use to solve the problems can be astonishing. I remember being proud of my ultra-short, 30-line-long solution, only to see someone else solve the same problem in the same programming language with just two lines of code because they used a clever idea.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

And I agree - tests are important.

But they won't identify many issues code reviews will. Here are a few examples.

Unwanted dependencies

Developers often add dependencies they could easily do without. Bringing in libraries that are megabytes in size only to use one small function or relying on packages like true may not be worth the cost. Code reviews are a great opportunity to spot new dependencies and discuss the value they bring.

Potential performance issues

In my experience, most automated tests use only basic test inputs. For instance, tests for code that operates on arrays rarely use arrays with more than a few items. These inputs might be sufficient to test the basic functionality but won't put the code under stress.

Code reviews allow spotting suboptimal algorithms whose execution time rapidly grows with the input size or scenarios prone to combinatorial explosion.

The latter bit our team not too long ago. When reviewing a code change, one of the reviewers mentioned the possibility of a combinatorial explosion. After discussing it with the author, they concluded it would never happen. Fast forward a few weeks, and our service occasionally uses 100% CPU before it crashes due to running out of memory. Guess what? The hypothetical scenario did happen. Had we analyzed the concern mentioned in the code review more thoroughly, we would've avoided the problem completely.

Code complexity and readability

Computers execute all code with the same ease. They don't care what it looks like. Humans are different. The more complex the code, the harder it is to understand and correctly modify. Code review is the best time to identify code that will become a maintenance nightmare due to its complexity and poor readability.

Missing test coverage

The purpose of automated tests is to flag bugs and regressions. But how do we ensure that these tests exist in the first place? Through a code review! If test coverage for a proposed change is insufficient, it is usually enough to ask in a code review for improving it.

Bugs? Yes, sometimes.

Code reviews do find bugs. They don't find all of them, but any bug caught before it reaches the repository is a win. Code reviews are one of the first stages in the software development process making them the earliest chance to catch bugs. And the sooner a bug is found, the cheaper it is to fix.

Conclusion

Tests are not a replacement for code reviews. However, code reviews are also not a replacement for tests. These are two different tools. Even though there might be some overlap, they have different purposes. Having both helps maintain high-quality code.

Storytime

One memorable issue I found through a code review was the questionable use of regular expressions. My experience taught me to be careful with regular expressions because even innocent-looking ones can lead to serious performance issues. But when I saw the proposed change, I was speechless: the code generated regular expressions on the fly in a loop and executed them.

At that time, I was nineteen years into my Software Engineering career (including 11 years at Microsoft), and I hadn't seen a problem whose solution would require generating regular expressions on the fly. I didn't even completely understand what the code did, but I was convinced this code should never be checked in (despite passing tests). After digging deeper into the problem, we found that a single `for` loop with two indices could solve it. If not for the code review, this change would've made it to millions of phones, including, perhaps, yours, because this app is very popular and has hundreds of millions of downloads across iOS and Android.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

Easy access to all code

Meta's monorepo contains most of the company's code. Any developer working at Meta has access to it. We can search it, read it, and check the commit history. We also can, and frequently do, modify code code managed by other teams.

This easy access to all the code is great for the developer's productivity. Engineers can understand their dependencies deeper, debug issues across the entire stack, and implement features or bug fixes regardless of who manages the code. All this is available at their fingertips. They can hit the ground quickly without talking to other teams, reading their out-of-date wikis, and spending time figuring out how to clone and build their code.

Linear commit history

Meta's monorepo does not use branches, so the commit history is linear. The linear commiit history saves engineers from having to reverse engineer a London Tube Map-like merge history to determine if a given commit's snapshot contains their changes. With linear commit history, answering this question boils down to comparing commit times.

No versioning

Versioning is one of the most complex problems when working with multiple repos. Each repo is independent, and teams are free to decide which versions of dependencies they want to adopt. However, because each repo evolves at its own pace, different repos will inevitably end up with different versions of the same package. These inconsistencies lead to situations where a project may contain more than one version of the same dependency, but no single version works for everyone.

I experienced this firsthand during my time at Amazon. I was working on the Alexa app, which consisted of tens of packages, each pulling in at least a few dependencies. It was a versioning hell: conflicts were common, and resolving them was difficult. For example - one package used an older dependency because a newer version contained a bug. Another package, however, required the latest version because older versions lacked the needed features.

A monorepo solves versioning issues in a simple way: there is no versioning. All code is built together, so each package or project has only one version for a given commit.

Atomic commits

Monorepos allow atomic cross-project commits. Developers can rename classes or change function signatures without breaking code or tests. They just need to fix all the code affected by their change in the same commit.

Doing the same is impossible in a multi-repo environment. Introducing breaking changes is either safe but slow (as it requires multiple commits for a proper migration) or fast but at the expense of broken builds.

This problem plagued the ASP.NET Core project in its early days (ProjectK anyone?). The team was working on getting abstractions right, so the foundational interfaces constantly changed. Many packages (each in its repo) implemented or used these interfaces. Whenever they changed, most repos stopped compiling and needed fixes.

Build

Builds in monorepos are conceptually simple: all code in the repo is built at a given commit.

This approach makes it possible to quickly tell what's included in the build and create bundles where all build artifacts match.

While the idea is simple, building the entire monorepo becomes increasingly challenging as the repository grows. Compiling big monorepos, like Meta's, in a reasonable time is impossible without specialized build tools and massive infra.

Multiple repos make creating a list of matching packages surprisingly hard. I learned this when working on ASP.NET Core. The framework initially consisted of a couple of dozen of repos. Our build servers were constantly grinding because of what we called "build waves." A build wave was initiated by a single commit that triggered a build. When this build finished, it triggered builds in repos depending on it. This process continued until all repos were built. Not only was this process slow and fragile, but with a steady stream of commits across all the repos, producing a list of matching packages was difficult.

The ASP.NET Core team eventually consolidated all the code in a single repository adopting the monorepo approach. This change happened after I left the team, but I believe the challenges behind getting fast and consistent builds were an important reason.

What are the problems with monorepos?

If monorepos are so great, why isn't everyone using them? There are several reasons.

Scale

Scale poses the biggest challenge for monorepos. Meta's repository is counted in terabytes and receives thousands of commits each day. Detecting conflicts and ensuring that all changes are merged correctly and don't break the build without hurting developers' productivity is tough. As most off-the-shelf tools cannot handle this scale, Meta has many dedicated teams that maintain the build infrastructure. Sometimes, they need to go to great lengths to do their job. Here is an example:

Back in 2013, tooling teams ran a simulation that showed that in a few years, basic git commands would take 45 minutes to execute if the repo continued to grow at the rate it did. It was unacceptable, so Facebook engineers turned to Git folks to solve this problem. At that time, Git was uninterested in modifying their SCM (Source Code Management) to support such a big repo. The Mercurial (hg) team, however, was more receptive. With significant contributions from Facebook, it rearchitected Mercurial to meet Facebook's requirements. This is why Meta (a.k.a. Facebook) uses Mercurial (hg) as its source control.

Granular project permissions

Monorepos make accessing any code in the repository easy, which is great for developers' productivity. However, companies often have sensitive code only selected developers should be able to access. This requirement goes against the idea of the monorepo, which aims to make all code easily accessible. As a result, enforcing access to code in a monorepo is problematic. Creating separate repos for sensitive projects is also not ideal, especially if these projects use the common infrastructure the monorepo provides for free.

Release management

A common strategy to maintain multiple releases is to create a branch for each release. Follow-ups (e.g., bug fixes) can be merged to these branches without bringing unrelated changes that could destabilize the release. This strategy won't work in monorepos with a linear history.

I must admit that I don't know how teams that ship their products publicly handle their releases. Our team owns a few services we deploy to production frequently. If we find an issue, we roll back our deployment and fix the bug forward.

A single commit can break the build

Because for monorepos, the entire codebase is built at a given commit, merging a mistake that causes compilation errors will break the build. These situations happen despite the tooling that is supposed to prevent them. In practice, this is only rarely a problem. Developers are only affected if the project that doesn't compile is a dependency. And even then, they can workaround the problem by working off of an older commit until the breakage is fixed.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

Wikipedia defines gold plating as: “the phenomenon of working on a project or task past the point of diminishing returns.” While the article talks about gold plating in the context of project management, the same phenomenon occurs in software development under a more familiar name: unnecessary refactoring.

The feedback I got from the senior engineer was that he noticed a pattern where I continued working on code that was already finished. I was polishing tests without covering new scenarios, changing perfectly fine variable or method names, or making functions slightly shorter.

I felt offended - I was making the code better!

How could a senior software engineer not see it?

How could they be against improving code?

So, he asked me to explain how my changes improved the code. I couldn’t. Only then did I realize he was right. I had to admit the new code was a bit different, but it wasn’t objectively better.

But then he went further and asked me how my changes impacted the team. I got confused: why would these small changes affect the team? It turned out they could and in a few ways.

I didn’t use my time effectively.

I spent time working on unimportant changes instead of taking on work that mattered. Therefore, someone else had to pick up tasks I could work on. If I did it, we could fix more bugs, implement more features, or ship faster. I also hurt myself - important work is usually a good learning opportunity and can lead to a quicker promotion, but I chose to pass on it.

I stole time from team members.

Code reviews were standard practice on every team I worked on in the past 20 years. Reviewing even small changes requires time. By requesting reviews of unneeded changes, I demanded that my team members spend time on trivialities.

Changing any code can lead to merge conflicts and disrupt other developer’s work. Sometimes, it is unavoidable. But it is not fun when changes no one needs cause conflicts.

I occasionally introduced issues.

A few times, my gold plating resulted in bugs. I missed an edge case in the new code, and somehow, tests didn’t catch it. The bug would break the build or make it to production. Having to justify fixing issues introduced by unnecessary changes is always embarrassing.

Not all refactoring is gold plating.

I am not trying to convince anyone that refactoring, in general, is a waste of time. In most cases, it’s the opposite. Refactoring code often aims to simplify implementing future changes, remove duplication, or make code more understandable. Sometimes, especially when deadlines loom, developers (myself included) make shortcuts or introduce hacks that are a ticking bomb. Removing them is the right thing to do, substantially improving the code quality. These kinds of refactoring are not gold plating. Gold plating is about changes we could live without without anyone noticing them.

Now that you know what gold plating is, whenever you decide to refactor some code, you should ask yourself: “Is it a real improvement, or am I just gold plating?”

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

Before Microsoft, I worked in a small start-up, and our projects didn't exceed tens of thousands of lines of code. We could open, edit, and compile these projects directly in the IDE (Integrated Development Environment). My new team's codebase had a few hundred thousand lines written in several programming languages. It was about 15 years old and used pretty much all possible technologies Microsoft invented in those years. Compiling it successfully was impossible without setting tens of environment variables and using magic command line incantations. No single IDE could handle this. It took me a few weeks before I began to feel comfortable with this codebase and all the tools I had to use for development.

This was almost twenty years ago, and since then, I have worked in several other big codebases, including .NET Framework, Visual Studio, ASP.NET Core, Amazon's codebase, and Meta's (Facebook's) mono repo. Even though all these codebases were different, they had many similar challenges, most of which could be overcome using similar tactics.

Trying to understand all code is futile

A single person cannot deeply understand a codebase that has a few hundred thousand lines. But this is not the only challenge. Large codebases are not static. They often receive hundreds of contributions each day, so they evolve rapidly.

On the bright side, understanding all the code is not necessary. Rather, it is better to have a very good understanding of the area your team works on and a decent knowledge of the areas your code interacts with.

Code searching

It's hard to be productive if you can't search code. But it gets exponentially harder if you can't even find the repo. And this was my experience during my years at Microsoft.

At that time, each team managed its codebase and source control individually, but there wasn't any tool to find these repositories. The internal search returned an incomplete list of, often outdated, wikis. The easiest way to find code was to first find the team responsible for it and then get all the details from them.

(Around the time I was leaving Microsoft, it implemented its new engineering system, 1ES (One Engineering System), which I am sure brought significant improvements.)

Searching large codebases on a dev machine may not be an option. Cloning the entire codebase to a dev box may not be feasible, especially if the codebase consists of thousands of federated repos, like Amazon's. Even if cloning is possible, tools such as grep are often too slow. This is why most big codebases have dedicated tools that make searching the code fast. Many of them also support following references, which is extremely helpful.

One factor that tremendously simplifies searching the code is formatting. If coding style is not enforced, finding anything is almost impossible. Searching a uniformly formatted codebase is much easier. This is why implementing a tool that enforces coding style is a good investment.

Build system complexity

Understanding the build system is key to being productive when working with big codebases.

Big codebases tend to have extremely complex build systems, often consisting of custom scripts, one-off tools, and specialized extensions stitched together to do the job. Off-the-shelf developer tools (e.g., IDEs) rarely can handle this complexity. Developers may struggle for days when they encounter a build system issue.

Many big companies have built their own tools to reign in this complexity and make it easier and faster for developers to work on large, multi-language code bases. Meta has buck Amazon has brazil, and Google has bazel. But from my experience, especially, with brazil, these tools also have some rough edges, so understanding how they work can go a long way.

The development environment is constantly in flux

Due to the number of engineers working in large codebases, even small productivity improvements can yield savings measured in engineering years. Maintainers work all the time to identify and fix bottlenecks. Because of this, the developer environment changes constantly, and the transitions are often not smooth, ironically resulting in lost productivity.

In 2019, Facebook decided to move away from Nuclide as its main IDE and migrate to VS Code. As a fan and an early adopter of VS Code (I even created an extension, and it was only in 2015!) I welcomed this change. But the ride was bumpy. The command I used the most (a few times per hour) during the first year was: Developer: Reload Windows. I had to use Vim or go back to Nuclide multiple times because VS Code stopped working. The early versions were bare - it took more than two years to bring all the features Nuclide offered to VS Code.

(To clarify, the tooling team did an awesome job. It supported both IDEs during the migration and put immense effort into making this migration successful. And it paid off—today, our VS Code is very stable, constantly gets new features, and is a pleasure to work with.)

Slow builds

Compiling large codebases takes time. Fortunately, you never need to do it yourself. In most cases, you only need to build and integrate with your product the sub-project you modified. However, even these steps can take considerable time despite the miracles that build engineers perform.

Legacy code

The codebases of many successful products that have been around for decades (e.g., Microsoft Windows) are big. They grow organically over the years thanks to the contributions of hundreds or thousands of developers who merge code daily. New releases are developed by expanding previous releases. Consequently, large codebases accumulate a lot of legacy code that almost no one is familiar with. I am sure some of the code I considered legacy when I joined Microsoft twenty years ago is still around because the product I worked on is still on the market.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

The on-call onboarding process is the same for each new team member. It consists of the following steps:

1. Regular ramp-up

2. On-call overview

3. Shadow shift

4. Reverse shadow shift

5. First solo shift

Let's look into each of these steps in more detail.

Regular ramp-up

The regular ramp-up aims to help new team members familiarize themselves with the problems the team is solving and teach them how to work effectively in the team's codebase. We want new colleagues to work on the code they will be responsible for when they are on call later. This approach allows them to acquire basic context that will be useful for maintaining this code and troubleshooting issues.

On-call overview

Regular ramp-up is rarely sufficient for new people to grasp the entire infra the team is responsible for. And knowing this infra is just the tip of the iceberg. There is much more an effective on-call needs to be familiar with, for instance:

• what are the dependencies, and what is the impact of their failures

• how to find dashboards and use them for debugging

• where to find the documentation (e.g., runbooks)

• expectations, e.g., is the on-call responsible for alerts raised outside working hours

• how to do deployments and rollbacks

• tools used to troubleshoot and fix issues

• standard operating procedures

• and more

On our team, we organize knowledge-sharing sessions that give new team members an overview of all these areas. We record these sessions to make revisiting unclear topics easy.

Shadow on-call shift

During the shadow on-call shift, the on-call-in-training (a.k.a. secondary on-call) shadows an experienced on-call (a.k.a. primary on-call). Both on-calls are subscribed to all tasks and alerts, but resolving issues is the primary on-call's responsibility. The primary on-call is expected to show the secondary on-call how to deal with outages. This is usually limited to problems occurring during working hours. Finally, the primary on-call can ask the secondary on-call to handle non-critical tasks, providing guidance as needed.

Reverse shadow on-call shift

After the shadow shift, things get real: the on-call in training becomes the primary on-call. They are now responsible for handling all alerts, tasks, deployments, etc. However, they are not alone—they have an experienced on-call having their back during the entire shift.

We schedule shadow and reverse shadow shifts back-to-back. This way, everything the on-call-in-training learned during the first shift is fresh when they become the primary on-call.

First solo shift

Once shadowing is complete, we add the new team member to the on-call rotation. We add them to the queue's end, giving them additional time to learn more about our systems and the infrastructure.

In addition to training new on-calls, our team maintains a chat to discuss on-call problems and get help when resolving issues. Both new and experienced on-calls regularly use this chat when they are stuck because they know someone will be there to help them.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

With no choice but to listen to Taylor Swift, I wanted to know if her songs had anything to say about Software Engineering. Surprisingly, the answer was "yes." Here is what I've discovered.

On being a Senior Software Engineer

I found this gem in the song Karma. It is a perfect description of a Senior Software Engineer. I don't know any who wouldn't identify with it:

Ask me what I learned from all those years
Ask me what I earned from all those tears
Ask me why so many fade but I'm still here
I'm still here

Gnarly bugs

When it comes to bugs, every developer remembers one they struggled with for a long time and became obsessed with fixing. The song Tell Me Why articulates this feeling extremely well.

Why do you have to make me feel small
So you can feel whole inside?
Why do you have to put down my dreams
So you're the only thing on my mind?

Botched Deployments

Remember the time your push brought the entire production down? You can hear about it in the song Ivy.

So yeah, it's a fire
It's a goddamn blaze in the dark
And you started it
You started it

Aggressive schedules

In the Long Story Short, Taylor Swift sings about how developers feel after delivering a critical project on impossible-to-meet timelines. And it's exactly what you'd expect.

Long story short, it was a bad time
Long story short, I survived

Career Development

When I consider all the career conversations I've had in the past 20+ years, the I Forgot That You Existed song perfectly reflects my experience.

Sent me a clear message
Taught me some hard lessons
I just forget what they were
It's all just a blur

I had my domestic Taylor Swift SME (Subject Matter Expert) review this post. She didn't agree with any of my interpretations, so you should, perhaps, take them with a grain of salt, too.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel

But this is great news for you. All these factors make improving on-call a great career opportunity. Here are a few reasons:

• Team-wide impact. Making the on-call better increases work satisfaction for everyone on the team.

• Finding work is easy. No on-call is perfect. There's always something to fix.

• No competition. Most engineers consider work related to on-call uninteresting, so you can fully own the entire area. As a result, your scope might be bigger than any other development work you own.

Getting started

It is difficult to propose meaningful improvements to your team's on-call before your first shift. You need to become familiar with your team's on-call responsibilities and problems before trying to make it better.

Once you have a few shifts under your belt, you should know the most problematic areas. Come up with a few concrete actions to remedy the biggest issues. This list doesn't have to be complete to get started. Some examples include tuning (or deleting) the noisiest alerts, refactoring fragile code, or automating time-consuming manual tasks.

Talk to your manager about the improvements you want to make. No manager who cares about their team would refuse the offer to improve the team's on-call. If the timing is not right (e.g., your team is closing a big release), ask your manager when a better time would be. Mention that you may need their help to ensure the participation of all team members.

Set your expectations right. Despite the improvements, don't expect your team members to suddenly start loving their on-call. It's a win if they stop dreading it.

Execution

From my experience, the two most effective ways to improve the on-call is to have regular (e.g., twice a year) fixathons combined with ongoing maintenance.

During a fixathon, the entire team spends a few days fixing the biggest on-call issues. In most cases, these will be issues that started occurring since the previous fixathon but weren't taken care of by on-calls during their shifts. You may need to work closely with your manager to ensure the entire team's participation, especially at the beginning.

Ongoing maintenance involves fixing problems as they arise, usually done by the person on call. As some shifts are heavier than others, the on-call may not always be able to address all issues.

Your role

Before talking about what your role is, let's talk about what your role isn't.

Your role isn't to single-handedly fix all on-call issues.

This approach doesn't scale. If you try it, you will eventually burn out, struggling to do two full-time jobs simultaneously: your regular responsibilities and fixing on-call issues. The worst part is that your team members won't feel responsible for maintaining the on-call quality. They might even care less because now somebody is fixing issues for them.

While you should still participate in fixing on-call issues, your main role is to:

• organize fixathons - identify the most pressing issues and distribute issues for the team to work on, track progress, and measure the improvement

• ensure on-calls are addressing issues they encountered during their shifts

• build tools - e.g., dashboards to monitor the quality of the on-call or queries that allow to identify the biggest problems quickly

If you do this consistently, your team members will eventually find fixing on-call issues natural.

Skills you will learn

Driving on-call improvements will help you hone a few skills that are key for successful senior and even staff engineers:

• leading without authority - as the owner of the on-call improvement area you're responsible for coming up with the plan and leading its execution

• scaling through others - because you involve the entire team, you can get much more done than if you did it yourself

• influencing the engineering culture of the team - ingraining a sense of responsibility for the on-call quality in team members is an impactful change

• holding people accountable - making sure everyone does their part is always a challenge

• identifying problems worth solving - instead of being told what problems to solve, you are responsible for finding these problems and deciding if they are worth solving

Expanding your scope

Once you start seeing the results of your work, you can take it further to expand your scope.

You can become the engineer who manages the on-call rotation for your team. This work doesn't take a lot of time but can save a lot of headaches for your manager. The typical responsibilities include:

• managing the on-call schedule

• organizing onboarding new team members to the on-call rotation

• helping figure out shift swaps and substitutions

Another way to increase your scope is to share your experience with other teams. You can organize talks showing what you did, the results you achieved, and what worked and what didn't. You can also generalize the tools you built so that other teams can use them.

If you found this useful, please share it with a friend and consider subscribing if you haven’t already.

Thanks for reading!

-Pawel