Which Scale?

Without context, the correct answer to most questions is “It depends”. This doesn’t make for an interesting conversation, hence people assume some context and it is usually the one they live in. It is the main reason why internet conversations break down: implicit ideas don't travel well over the wire.

Context shapes language as well, which is why you sometimes have to lean on another one to capture an idea perfectly. The nation of process masters¹ have one that resonates with me well: Fachidiot. In German, “Fach” stands for work or occupation and an idiot doesn’t need translation. It captures well a certain kind of people that perceive and frame any situation through the prism of their work. We all do it to a certain degree, yet some occupations are more susceptible to this than others and only some people master this approach enough to be completely useless in other areas of life.

Scale Is The Defining Context

In software development, scale dictates everything. Decisions about code quality, project organization, legal and regulatory requirements, appropriate languages, licenses you can use, system requirements, team communication patterns, development lifecycle and zillions of other things shall depend on the scale of the endeavor. That is, are you learning to code in your basement, running your hobby project, running a company eshop, developing a SaaS application, providing an embedded system for a pacemaker,writing a Linux kernel, shipping a database, running a cloud, providing a payroll for civic service, developing spyware for the NSA or doing actual rocket science?

When making decisions, there are two fallacies I see running around.

Fallacy 1: WYSIATI

“What you see is all there is” bias which translates into the inability to give advice to anyone working in a different environment. I see this especially across the regulation divide. Developers in unregulated fields can’t imagine having their hands tied by such restrictions. Their advice to one in a regulated industry is perceived as ridiculous carelessness. Even better: they could put people in jail.

After working in a regulated field, developers often forget how to fly free and the amount of analysis and preparation will bring unregulated dev teams to a halt.

When giving or receiving advice, make sure the experience is relevant. Keeping track of what you don’t know is hard, but without it, you are just hand-waving.

Fallacy 2: The Complexity Fetish

Smaller scale is perceived as inferior. This translates into the inability to receive insight from a smaller-scale environment. Any form of advice or learning from such is off the table a priori as “this wouldn’t work for us”².

On the other hand, any learning from environments operating on a higher scale is accepted naïvely. I believe this is the main driver behind FAANG being such an influential trend-setter in this new iteration of “nobody got fired for choosing IBM”.

Bias is natural. It stems from the way we learn and the way we are rewarded. When people join companies early, their career grows together with the company.

But during such growth, you don’t necessarily develop to be a faster sprinter—you change your discipline to run a decathlon instead. This is exactly what’s needed—but I see people freaking out about some missing piece of decathlon equipment when they are asked just to sprint again. It is why the top BigCorp talent may not fare well in your startup.

Scale Singularity

It is telling that up until now, I’ve used “scale” as if it was a single defining quality—the way it’s often used in conversations. The façets of scale are so rarely mentioned I think they are worth pointing out.

The Data Scale

How much data do you handle? It spans three categories:

• “Fits Excel”
• “Big Data” that any public cloud can handle
• “This requires thinking”

Each of them has two dimensions: the sheer amount of records and the size of an individual item. For example signal or sensor processing crosses boundaries with the amount of metadata; modern video can push you on the size. Challenges for each are different.

The Processing Power Scale

How many TFLOPs do you require? Challenges include proximity of the processing power to the data.

Processing power is easiest to put a price tag on, making it easier to decide whether it’s worth investing in making a program more efficient.

And some projects can use all the processing power you give them³.

The Processing Complexity Scale

How complex is your data processing? A lot of software can be reduced to an overly complicated ETL pipeline. I do not mean it in a derogatory way: maintaining a complicated ETL is a shitload of work.

And it is a scale and complexity in itself. Beside the code implications, it makes all other complexities go way worse, especially if the source of the data lies outside of your team’s boundary.

The Organization Size Scale

How many people are in your company? The more people, the more complicated everything is. There are books and movements and hacks on how to pretend it’s not the case, but none of them work.

On the other end of the spectrum, staying small limits what you can do. Pick your poison.

The Constraint Scale

What are your regulatory and legal requirements? You don’t operate in a vacuum. The country and industry you are in imposes restrictions and constraints. Those exist on a scale and have huge consequences on other scale axes.

A lot of multinational company evilness is a direct result of going up this scale. When your processes have to be a sum of all laws and regulations of all countries you operate in, you are essentially a bureaucracy superpower. Good luck.

The Usage Scale

How many people use your services? How many are users and how many are customers? Servicing one enterprise customer looks very different than serving one million users, although the revenue may be the same.

(users or customers, impact on support etc)

The Network Traffic Scale

What is your normal and peak traffic throughput? Beyond the naïve approach, network constraints get very interesting very quickly. And both throughput and latency can push you into large-scale regional deployment with a scale on its own.

Other

Based on your domain, I bet you can amend this list by your own dozens of points. Yes, this list is not exhaustive. It should rather show how talking about scale as a singular scalar value is less than helpful.

Handling Scale Without Complexity

I define excellent handling of scale as “supporting scaling up on an axis while limiting the increase in complexity.” I mostly don’t know how to do this.

A natural approach to handling complexity is “divide and conquer:” break down the complex problem into simpler problems that can be solved and synthesized a complete solution. Being a Fachidiot myself, I see a great parallel with how microservices work: the complexity has been pushed from “solving the problem” step into the “communication and synthesis” step, making it less visible, harder to track and often harder to define.

The “make sure partial solutions solve the complete problem” part is implemented as “the process”. There is a company that makes software for that⁴ and is notorious for lukewarm reception beyond the top management.

It Depends

Scale is specific to your domain and your company. The best generic guidance I can provide is “determine the best fit for the current environment and use your good judgement and prediction upon encountering future inflection points.”

So when a scaling question comes, I ask: “Which scale?”

Thanks to Mark Foster for feedback and extensive corrections.

---

1. They run the world with it. And nobody else would come with such an aptly named language as Allgemeiner Berichts-Aufbereitungs-Prozessor (generic report preparation processor); reading out loud “Advanced Business Application Programming” just doesn’t roll off your tongue in the same way. ↩︎

2. But how do you know? There should be a data answer. NNALD can help. ↩︎

3. Beside projects artificially designed to function as indirect heating (like Bitcoin), useful examples include physics, like weather models. ↩︎

4. The background behind the old enterprise adage “If your company survives SAP implementation, it will survive anything” comes from a difference between a process and a reality. Handling that would make for a different essay. ↩︎