~/BIKUNDO
Back to blog

#Architecture#Microservices#Elixir#USSD#Strangler Fig

Breaking up a monolith one route at a time

Lessons from moving an SMS, USSD and web learning platform off a legacy monolith: why USSD traffic strains a request-per-worker stack, routing traffic out gradually, Elixir session processes, and keeping data consistent between services.

Eneza Education delivers learning content over SMS, USSD and the web, mostly to students on basic phones, through partnerships with carriers like Safaricom, MTN and Tigo. As engineering manager I led a team of 15 engineers through moving that platform off a legacy monolith and onto Node.js, Elixir and Python services.

This post isn’t an argument for microservices. A monolith is the right default for most teams. It’s about what to do once one part of the system has needs the rest of it can’t meet, and how to split that part out without stopping a platform people use every day.

Why USSD traffic strains a request-per-worker stack

USSD looks simple from the outside. A user dials a code, gets a text menu, replies with a number, and gets the next menu. Under the hood, the gateway turns every one of those steps into an HTTP request to your server, and it waits on you. Africa’s Talking, for example, POSTs sessionId, phoneNumber, serviceCode and text to your callback URL and expects a reply within 10 seconds. The reply has to start with CON if the session continues or END if it’s finished. Direct carrier integrations differ in the details, but the shape is the same.

Now put that traffic on a classic PHP-FPM setup, where each request occupies a worker for its whole lifetime. Most of the time that’s fine. The trouble starts when something downstream slows down: a payment provider, an SMS gateway, one bad query. Every request that touches it now holds a worker for seconds instead of milliseconds. The pool fills up, new requests queue behind it, and everything that shares the pool, including the web app, slows down with it.

The lesson I took from this: the fix is isolation, not a faster language. The traffic with the tightest deadline shouldn’t share a failure domain with everything else.

Route traffic out one path at a time

The strangler fig pattern gets its name from a vine that grows around a tree until it can stand on its own. In practice it means you put a proxy in front of the monolith and move one route at a time to new code, while everything else keeps working as it did.

upstream legacy_app  { server legacy:9000; }
upstream ussd_service { server ussd:4000; }

server {
    location /ussd/ {
        proxy_pass http://ussd_service;
        proxy_read_timeout 8s;
    }

    location / {
        proxy_pass http://legacy_app;
    }
}

Two things make this approach work. The rollback is one line in a config file, so you can be bold about the first move. And each route you move is a complete, testable unit, so you’re never halfway through a big-bang rewrite with nothing to show for it.

Choosing what to move first matters more than how. Good candidates have high load or tight latency needs, a narrow interface and few tables shared with the rest of the system. A USSD handler meets all three. The admin back office usually meets none of them, so leave it in the monolith for a long time, possibly forever.

Start stateless, then add a process per session

One detail of the USSD protocol changes the design: the gateway sends the whole path the user has taken in text, joined with *. A user who picked option 1 and then option 2 sends "1*2". That means many menus can be a pure function of the input, with no server-side session state at all. Start there. It’s the easiest version to test and scale.

You need state once a session has to hold data you don’t want to fetch on every step: a quiz loaded at the start, answers collected along the way, a timer of your own. This is where the BEAM earns its keep. An Elixir process is cheap enough to run one per active session, and if one crashes on bad input, its supervisor cleans it up without affecting any other session.

defmodule Ussd.Session do
  use GenServer, restart: :temporary

  @idle_timeout :timer.seconds(30)

  def start_link({session_id, phone}) do
    GenServer.start_link(__MODULE__, phone, name: via(session_id))
  end

  # Called by the HTTP handler for every gateway request.
  def handle_input(session_id, phone, text) do
    pid =
      case DynamicSupervisor.start_child(Ussd.Sessions, {__MODULE__, {session_id, phone}}) do
        {:ok, pid} -> pid
        {:error, {:already_started, pid}} -> pid
      end

    # Stay well inside the gateway's 10-second limit.
    GenServer.call(pid, {:input, text}, 5_000)
  end

  @impl true
  def init(phone), do: {:ok, %{phone: phone, quiz: nil}, @idle_timeout}

  @impl true
  def handle_call({:input, text}, _from, state) do
    steps = if text == "", do: [], else: String.split(text, "*")
    {reply, state} = Ussd.Menu.render(steps, state)
    # Returning the timeout again resets the idle timer on every step.
    {:reply, reply, state, @idle_timeout}
  end

  @impl true
  def handle_info(:timeout, state), do: {:stop, :normal, state}

  defp via(session_id), do: {:via, Registry, {Ussd.Registry, session_id}}
end

A few choices in there are deliberate. restart: :temporary means a session that ends or crashes is gone for good, which is what you want, because the gateway will have moved on anyway. Starting the process through DynamicSupervisor.start_child/2 and accepting :already_started handles the case where two requests for the same session arrive at once. And the idle timeout is returned from every callback, so an abandoned session cleans itself up rather than leaking.

The hard part is the data

Splitting code is the easy half. The shared database is where these migrations stall. Two rules kept ours manageable:

  1. Every table has exactly one owner. Other services read it through the owner’s API or a read-only replica, and they never write to it.
  2. Changes that other services care about travel as events, not as rows they poll.

Events bring their own failure mode. If a service updates its database and then publishes an event, a crash between the two leaves the system inconsistent. The transactional outbox pattern closes that gap by writing the event to a table in the same transaction as the change itself:

BEGIN;
UPDATE quiz_attempts SET status = 'completed' WHERE id = 81;
INSERT INTO outbox (id, type, payload)
VALUES ('6f1c2e9a', 'QuizCompleted', '{"attempt_id": 81}');
COMMIT;

A small relay process reads unsent rows, publishes them to whatever broker you use and marks them sent. Several relays can share the work safely by claiming rows with SELECT ... FOR UPDATE SKIP LOCKED, which both PostgreSQL and MySQL 8 support.

The relay guarantees at-least-once delivery, which means consumers will sometimes see the same event twice. So every consumer records what it has processed, in the same transaction as its own side effect:

BEGIN;
INSERT INTO processed_events (event_id, consumer)
VALUES ('6f1c2e9a', 'rewards');   -- unique (event_id, consumer)
UPDATE learner_points SET points = points + 10 WHERE learner_id = 4928;
COMMIT;

If the insert hits the unique constraint, the event has already been handled. The transaction rolls back and nothing is applied twice. It’s not clever, and that’s why it works.

What I’d tell a team starting this today

Every new service adds a deploy pipeline, a dashboard, an on-call rotation and, if it’s a new language, a hiring problem. Moving to three languages has to buy you something specific for each one. Write that reason down before you start.

Plan the rollback before the migration. If moving a route back to the monolith takes more than a config change and a deploy, the step is too big.

And measure the thing you’re trying to fix. “The monolith is slow” isn’t a goal. “USSD replies stay well under the gateway’s limit even when the payment provider is slow” is one. You’ll know when you’ve hit it, and you’ll know when to stop.