
The Cost of Configuration
Written by a human, for humans ❤️
Every. Single. Bit. Matters. You change one bit, behavior changes. This is true for tiny things like a bit flip in a machine-language opcode, and also holds when flipping a feature flag enablement in an enterprise SaaS platform.
As I argued in a previous post, the behavior of a software platform is the sum of its code plus its configuration. And every single bit matters in both.
Testing code is not a new problem. Even with the advances in AI coding and automated code generation, software testing is an undecidable problem. But at the very least, if we’ve built our platform as a multi-tenant solution, we run the same code for every customer, and therefore we have a single piece of code that we need to worry about.
Bear with me as I know I’m oversimplifying. Even a “single piece of code” can be composed of multiple modules, written in different languages, running on different platforms. And, more importantly, evolving every time we merge a new pull request. But still… with enough test cases we can highly increase the confidence that our code will do what we want.
And yet, in a software platform, that’s only half of the equation. We still have to test our configuration.
But there’s a catch. It’s not our configuration. Some of it may be, such as things like feature flag settings, environment variables and deployment settings. But in any software platform there are also major parts of the configuration that are done by each tenant itself.
In an e-commerce platform, this configuration can define things like the promo codes or the payment and financing options. In an integrations platform, the configuration dictates the flows and connectors that each tenant configured, as well as the schedules or webhooks that are used to trigger them.
So how do we test this config? We can build this into levels going from the lowest level of effort to the most sophisticated. This is inspired by the Richardson Maturity Model, heavily used to describe the maturity of REST APIs.
Level 0: Testing Some Configurations (and Hoping for the Best)
We certainly won’t be able to test all configurations. But we should strive to at least test some of them. I know this sounds very basic, but many platforms have a lot of automated unit testing for the platform software components, but hardly any testing at all for those components in unison with real configuration.
This is especially hard for new products. If we’ve just built a shiny new platform and we still don’t have any real customers, we can only hope to test some configurations that we think are relevant.
As an example, in an integration platform we should strive to test each connector at least once. This does not guarantee that we will test those connectors the way a real customer would use them, though. But it’s a start.
Level 1: Testing Representative Configurations
Once we start getting tenants and we start taking a look at how they use our platform, we should aim to learn from that and test accordingly.
Sounds easy, but it’s not so simple. First of all, we need proper instrumentation to discover which configuration shapes are more common than others. In the e-commerce platform example, this could be as simple as a pie chart with how many tenants use what types of discount codes. For example, if we know that 90% of customers use single-use discount codes, we should focus a significant amount of testing on those.
But in a more complex example such as the integrations platform, the shape of the configuration could be as complex as an XML that defines a pipeline-like flow of data across various connectors, with if/else and for-like constructs. In other words, our configuration is a Turing-complete DSL! Finding representative “programs” in that DSL is not an easy task.
More importantly, once we know how to find which configurations are more common than others, we need to be able to distill them to minimalistic scenarios that don’t bleed tenant configuration data. We can’t just copy the config from one tenant and use it as a test case. Something I’ve learnt working in software platforms is that customers’ configurations are hardly ever minimal or distilled (topic for another day, I guess!). So we’ll need to boil them down to its core to make sure we’re testing exactly what we want, no more and no less.
Level 2: Testing Representative Configurations Over Time
Knowing what to test is not enough. We also need to know when to start testing a new config pattern that just emerged and when to stop testing configuration patterns that fell out of fashion. Tenant configurations will evolve over time. If our company goes more into enterprise customers, we’ll find more complex configurations compared to the ones we were testing a few quarters ago. Similarly, if our company decides to go for smaller customers (AKA “down market”) then we need to be ready to test those simpler but equally important configuration patterns as well.
We can do this by investing some time in automating the configuration patterns discovery process. Easier said than done, as this involves a combination of data gathering, distillation and synthesis. With AI this can be more streamlined now and there are fewer excuses not to do it relatively often, especially for fast-growing platforms in dynamic verticals like agentic platforms.
Level 3: Testing Real Configurations
What if I told you that you can (and sometimes should!) test on real customer data? Before you start cancelling me, hear me out. This is the ultimate level when it comes to testing platform configurations, and it doesn’t come without its challenges, but it’s worth doing for select key areas in which the configuration space is enormous and in which regressions are expensive but easy to spot.
One perfect example is custom reports. Any big platform has them and they’re usually quite complex in nature. Which columns, which data to put in them, aggregations, time slices, filters, etc. But reports have two very nice traits: they don’t mutate your data and they’re easy to compare byte-for-byte for differences.
The overall idea is to copy a sampled (e.g., decimated) real customer dataset, anonymize it and then execute its configuration (e.g., its custom reports) in a sandboxed environment. If the current production codebase yields a different behavior when compared to the release candidate (e.g., the Total items sold column renders 15 instead of 14), then we know that there is something to revise before proceeding with the release. Maybe its a time slicer algorithm bug, maybe its an aggregate function bug. But somewhere there’s something worth looking at.
This is a pretty expensive process and it also requires involving the legal team to make sure that we’re managing our customers’ data correctly and within the terms of their contract agreements. But, applied with care, it can greatly increase our confidence and will also automatically reflect new trends in config.
Final Thoughts
I’m not advocating to reach Level 3 on all of our platform components. But we need to start proactively thinking about Configuration Testing. Have a strategy for it. See in which level you land for each of your platform configurable components. Can you go one level higher? Is it feasible? What’s the cost of doing it and what’s the cost of not doing it?
Testing our platform configuration is, in my opinion, the biggest configuration cost. It’s not the only one and in future posts we’ll go over some of those other cost aspects to platform configuration.
Written by a human, for humans ❤️
Comments