← All writing

Distributed systems

Outbox and Inbox: how I introduced reliable messaging

Lost events, duplicate processing, and stuck orders: why I use outbox and inbox patterns and introduce them gradually into existing systems.

I think the starting point for distributed systems is usually message brokers and their events. Event processing should usually be done in a reliable way. If not, well, this may be a conscious decision or an unconscious one.

What does it look like without any thinking about reliability? Just look on that: Sending a message? Sure!

public sealed class ConfirmOrderFromEventCommandHandler(
    IOrderRepository orders,
    ServiceBusClient serviceBusClient)
    : IRequestHandler<ConfirmOrderFromEventCommand>
{
    public async Task Handle(
        ConfirmOrderFromEventCommand request, 
        CancellationToken cancellationToken)
    {
        var order = await orders.GetByIdAsync(request.OrderId, cancellationToken);
        if (order is null) return;

        order.MarkConfirmed();
        await orders.SaveChangesAsync(cancellationToken);

        await using ServiceBusSender sender = serviceBusClient.CreateSender("order-confirmed");

        var integrationEvent = new OrderConfirmedIntegrationEvent(
            order.Id,
            order.CustomerEmail,
            order.Amount,
            DateTimeOffset.UtcNow);

        var body = JsonSerializer.Serialize(integrationEvent);
        var message = new ServiceBusMessage
        {
            Body = BinaryData.FromString(body),
            ContentType = MediaTypeNames.Application.Json,
            Subject = nameof(OrderConfirmedIntegrationEvent)
        };

        await sender.SendMessageAsync(message, cancellationToken);
    }
}

Receiving? Just do it!

public sealed class NaiveMessageConsumer(
    ServiceBusClient serviceBusClient,
    IServiceScopeFactory scopeFactory,
    ILogger<NaiveMessageConsumer> logger)
    : BackgroundService
{

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await using ServiceBusProcessor processor = serviceBusClient.CreateProcessor(
            "order-created",
            new ServiceBusProcessorOptions { AutoCompleteMessages = false });

        processor.ProcessMessageAsync += ProcessMessageAsync;
        processor.ProcessErrorAsync += args =>
        {
            logger.LogError(args.Exception, "Service Bus processor error.");
            return Task.CompletedTask;
        };

        await processor.StartProcessingAsync(stoppingToken);
    }

    private async Task ProcessMessageAsync(ProcessMessageEventArgs args)
    {
        using var scope = scopeFactory.CreateScope();
        var mediator = scope.ServiceProvider.GetRequiredService<IMediator>();

        if (args.Message.Subject != nameof(OrderCreatedIntegrationEvent))
            return;

        var evt = JsonSerializer.Deserialize<OrderCreatedIntegrationEvent>(
            args.Message.Body.ToString()
        )!;
        await mediator.Send(
            new ConfirmOrderFromEventCommand(evt.OrderId), 
            args.CancellationToken
        );

        await args.CompleteMessageAsync(args.Message, args.CancellationToken);
    }
}

I found that many systems don’t use a reliable approach, but they should, and it may be painful. Maybe not at the start, but I’m 100% sure that eventually there will be that painful situation.

Business will ask: developers, what are you doing? A deploy? Why does it hurt? Developers answer that they are doing nothing right now.

What could go wrong? Well, deeply hidden bugs or even infrastructure failure.

Why existing approach is not enough

You may wonder: what could go wrong? Businesses could ask: why the additional work? It’s working now. Well, just wait for it ;>

Infrastructure failure? No problem, let’s process events once again!

  • Additional orders
  • Additional transfer
  • Additional bank account charge
  • Wrong numbers in the statistics
  • Angry business

Misconfigured system? No problem, here we go again:

  • Additional orders
  • Additional transfer
  • Additional bank account charge
  • Wrong numbers in the statistics
  • Angry business

That’s only one side - receiving. We have another - sending.

Imagine this situation: you buy your dream MacBook. You can’t wait, that was your dream. And finally, you did it! Initial status of the order: in progress.

  • 1st day - in progress
  • 2nd day - in progress
  • 3rd day - in progress
  • Xth day - in progress

What happened? The order was saved in the order microservice, but the event was not published, which was necessary for the delivery microservice. The order is stuck. This is an easy situation:

  1. You make a transaction in the database and save everything.
  2. You send an event (it is not published because of some random error). And you are ready for failure. 😉

Maybe a different way? No problem.

  1. Sending an event before database commit (your order will be delivered).
  2. Transaction commit - and boom! Network error. 😀

What happened? Well, it depends on what we were doing. I could imagine a situation where no payment was made and your MacBook would be delivered (free is always a fair price 😉).

What is the solution?

So simple, why bother?

You may wonder why I started to talk about such trivial topics. Well, in my opinion, implementation is not the problem. There is something deeper and scarier…

I found lots of teams that deliver message-driven systems and have never implemented or even heard of inbox or outbox. What’s more, they don’t even think about such approaches to make it more reliable.

Those topics are very important and shouldn’t be omitted from message-based systems. Everyone working on them should have it in mind and remember that there is something more than processing messages and delivering business needs.

Those are non-functional requirements that should be covered because when they are missed, scary things may happen.

Why Inbox and Outbox

Let’s start first with the outbox. In my opinion, that is usually the way to go and the default approach.

An alternative may be Debezium - events produced from the database transaction log.

We will stick to the outbox. You may ask how to implement it. Easy, make an additional table (e.g., Events) with the content to send and with delivery attributes.

How to work with that? You start your transaction with buying your MacBook, in that transaction create what you need in the microservice database, and in the same transaction save the event that has to be sent.

What happened? You have an event to send and the order details saved. In one transaction. ACID.

What next? Create a job (it may be KEDA or another type of job) that claims messages to send from that table to your broker, sends them, and then removes them from the database. Error? No problem, retry. Message sent twice? No problem, set deduplication on your broker.

Your broker doesn’t support deduplication? No problem, implement an inbox.

The inbox allows you to implement deduplication and safe retry as well (as it is decoupled from the broker).

Without an inbox, you would have to make your event handlers idempotent, and you would have to think about it all the time. With an inbox, you get deduplication at that layer.

How does it work?

  1. Receiving a message from the broker.
  2. Save the message to a table in the database.
  3. Commit the message to the broker.
  4. Claim the message from the database.
  5. Process the message.
  6. Depending on the result: mark the message as read, as a permanent failure, as a temporary failure, or dead-letter it.

With that, you get reliability of delivering events (idempotent consumers, deduplication). You have to remember that every message should have an idempotency key that is stored in a table column marked as unique to get deduplication. Usually, the deduplication key is a simple GUID/UUID.

What may be hard? Well, imagine a situation where you have to process events in the proper order. You claim them one by one, and you have a few replicas of your job. This is a situation where you could process messages out of order. So you should have some correlation key.

Introducing the patterns into an existing system

How to introduce it into the existing system? Let’s imagine a situation where you have no reliability, no outbox, no inbox.

Start simply with small steps. Rome wasn’t built in a day.

You have two ways to go with these patterns:

  • Select some inbox/outbox library.
  • Create your own.

Don’t have the time to change all events? Do it when receiving or sending an event in the next feature. One by one, progressively.

Scared of concurrency? Use leader election so that one replica will be processing, or use pessimistic concurrency with lease-based locking on that row.

Or maybe you see someone sending an event? Or receiving? Make a comment in a pull request.

Introducing these patterns is not hard, it is a go or no-go, but I strongly recommend it.

What it changes

It changes a lot! This is a big deal. With just that, our systems start to be better and more resistant to errors and failures.

After a few months, you will see fewer problems in PROD. Your supervisor will be thankful (I hope 😉).

And… This is not theory, I have seen that in my career. Such small changes, and they give a lot.

Remember, this is not a free lunch. You have to support that code later. If you go with your own implementation, you will probably have to make performance improvements: as your system grows, your need for performance will also grow.

With libraries, you will probably get generally fine-tuned performance. But if you have a really big need for performance, there may be a ceiling, and implementing the pattern yourself may be a better choice.

TL;DR

If your system doesn’t support any reliable messaging, in my opinion, it is a good idea to start with outbox and inbox.

Implementation of these patterns is relatively simple. This is a go-or-not-go decision.

Not going will force you into fixing bugs from production. Going will give you a bit of peace.

The main idea of these patterns is a database table between you and the broker.

  • When sending, save to the outbox table in a database transaction with the other data, then send from a job.
  • When receiving, save to a table in the database, then a job will claim messages from the table and process them.

Just start.

P.S. Remember that outbox and inbox are only a part of messaging reliability. Don’t stop, it’s just a start. 😉 P.S. 2. Please don’t stop at the outbox, it is not enough! Inbox is also important.

Comments

Join the conversation using your GitHub account. You can also browse the discussions on GitHub.