← All writing

Production lessons

Feature Flags: when to use, when to skip

Rollout and kill-switch flags buy control over releases and incidents, but they add code and test cost. When to flag integrations and when to skip domain logic.

What are feature flags, really? They help us in developing our systems to reduce the blast radius of deployments. They enable multiple deployments of one feature without showing this unfinished feature to the user (aka rollout flags). Flags also may help in building some kind of guards in our code, which allow us to turn off particular features without deploying applications (in case of incidents, aka kill switch).

Flags should protect us and make our life easier.

Cons?

  • Introduce overall complexity to the system
  • Introduce complexity to the unit and other types of tests
  • If a flag is of type rollout, it should be removed after release. That is additional work.

It’s not a free gift.

Pros?

  • Separation of deployments and releases
  • Delivering feature partially
  • Progressive feature rollout
  • Ability to turn off a feature in case of incidents
  • Ability to turn off a feature in case of buggy deployments
  • Ability to release a feature to a single user on production

As you can see, we gain a lot.

How to implement them?

Feature flags can be implemented in many ways. I recommend using one of the SaaS platforms, such as Flagsmith, Unleash, ConfigCat, or Onion Switch. We get an administration panel from them for managing flags, which helps us to switch them without deploying or restarting applications.

In the code this is a simple if statement:


public async Task<OrderDto> GetOrderAsync(
    int orderId, 
    cancellationToken cancellationToken)
{
    var isFlagEnabled = await _featureFlags.IsEnabledAsync(
        "ORDERS_READ_MODEL", 
        cancellationToken
    );
    return isFlagEnabled 
        ? await GetOrderNewPathAsync(orderId, cancellationToken) 
        : await GetOrderOldPathAsync(orderId, cancellationToken);
}

Flags vs life — rollout flags

This is when hard parts come. Feature flags are a super tool, but we have to remember that it is not a free gift, we have to pay for it, usually the currency is our time.

A rollout flag is a special type of flag that helps us with developing particular functionality. This flag should be removed from code after the release of the whole feature. Another benefit is enabling separate deployments for features, progressive release, and rollback if needed.

When can we skip the feature flag?

The decision to skip a feature flag can be when we have changes in code we fully control (e.g., domain or application layer changes). It may appear that testing such flags costs much:

  • Complexity of unit tests
  • Complexity of integration tests
  • Complexity of manual tests
  • Lots of ifs in code
  • After release lots of if branches to remove

My experience says that if we change our own application logic, it is not always necessary to implement a feature flag. The reason is simple: there is less upside. What can go wrong?

  • Some calculations go wrong
  • Some saves go wrong

But we should have unit tests and integration tests for that feature. It is already tested and perhaps all is green. In my opinion there is no justified reason for implementing flags in places where the risk is low. In this case we control the logic of the domain or application layer 100%.

To sum up, we can skip feature flags for internal logic changes.

When should we implement flags?

We may know in advance that we don’t want to show a particular feature to the users or we want to deploy it separately across many deployments.

We may have another situation: we don’t know which option of our new feature is better, so in this case we may want to introduce flags (not really rollout, but experimental) that will have many states (Option A, Option B, etc.) across different types of users.

In practice, there are lots of dangerous situations in which we want to have feature flags implemented - integrations (HTTP, async API, etc.). Usually we hear that they are the hardest part in development and there is a big reason for that. They may behave in unpredictable ways, totally different than we expected, and even break rules, patterns and best practices that were widely accepted by the IT community. It adds additional problems.

What are the kinds of integrations that we should cover by feature flags?

  • HTTP clients (gRPC, and others)
    • Moving to new endpoints from old
    • Retiring some endpoints
    • Adding new fields to request or response
  • Async API (RabbitMQ, ActiveMQ, Azure Service Bus, etc.)
    • Moving to new event
    • New event consuming
    • Adding new fields to the DTO of consumed event
    • Changing field behavior, for example: moving from string to nullable string, a situation where we were using an empty string and now we moved to null or filled string

Above are problems that I found in my career, we should be prepared for them. Some of them can be addressed by contract testing, but when we don’t have them we should introduce flags for these cases.

Typical cases in the real world are adding new integrations, when it is a totally new integration on a totally new endpoint, then I don’t see a justified reason for introducing the flag (unless we need a kill switch so users cannot execute an endpoint directly, but that is a kill switch flag type rather). The reason is simple: this feature is not being used yet, so the risk is close to zero that something will go wrong. The situation may change when our frontend wants to integrate with our BFF, then on the frontend side we should assume potential problems and introduce the flags there (the rule of integration with HTTP client).

Above are situations that can be hard to decide whether we need a flag or not. There are also cases where we know in advance that we may want to introduce the flags. That may be a situation where we make a refactor and we introduce a read model, but after deployment the read model is not properly propagated, so we need to delay the switching of our flow from old source to new source. So we need a runtime switch that will allow us to switch to a new source when the read model data source will be properly propagated. To be honest, this can be done in another way, without feature flags: using separate deployments, first of them may introduce a read model and fill it with data, and others may start using this new source. We have to choose our preferred way.

We have to consider other challenges. We may find a situation where we have to update important infrastructure dependencies. In such cases we should also use feature flags. The branch-by-abstraction pattern is usually helpful.

In the end, we have to remember about one thing. After we deploy the flag and release the feature, we have to remove the flag from the code. If we don’t do that, our system will start to get complex, additional noise and unnecessary code that is not being used will appear and we will be in charge of supporting it.

Kill switch - another type of flag

There is another type of flag that is called kill switch - the lifecycle of such flags matches that of our system, we never remove them. Its responsibility is to turn off that functionality in critical moments, such as: a feature that is throwing ‘red errors’, data breach, etc. For an e-shop it may be the page ‘My Returns’, which had a data breach and users may see data of other users — we have a quick way to remove the problem.

In my opinion each bigger module/feature should have an entry flag that will help us turn it off. We do that usually per whole page or functionality.

Making choices

Making choices between go and no-go with flags is always a trade-off. On one side we have some peace, we can turn the feature off in some cases. On the other hand, it adds additional code complexity, later testing and finally an additional job which is removing the rollout flag after all work. We have to choose carefully the way we want to go, to avoid problems in later stages of feature development.

TL;DR

Flags give us lots of peace but they add additional work also. However, we gain enough that it is worth sometimes sacrificing a bit of time to have more control over the application and make our work more fluent, as no flags sometimes stop our deployments. We have to choose when to introduce a flag and when to not do that.

If we follow the rule of flagging integrations each time (HTTP clients, message brokers, etc.) and other bigger infrastructural dependencies, we can get a clear yes-or-no answer about whether to introduce flags, every time.

If we change the business logic that we fully control and we have big coverage by automation tests, then we may want to skip the flag, as the risk is low.

We have to remember that adding lots of small changes (e.g. wide refactor deployed frequently), for which we introduce new flags each time, we get stuck in flag complexity and flag-removal work - it may take much time. We need to carefully weigh our decisions.

Comments

Join the conversation using your GitHub account. You can also browse the discussions on GitHub.