VISend JD
← Blog

Huy Bạch · 6 min read
  • Migration
  • Story

A TODO had been in the codebase longer than the newest person on the team. This is about it, and about the next migration.

There's a comment in the codebase that everyone new to the team asks about:

// TODO: delete this field once the migration is done

That line had been there longer than the newest person on the team. The migration finished long ago. But the field was still there. Nobody dared to delete it.

Why nobody dared

One quiet day, I tried grepping for that field's name. The results were longer than I expected. Here, an if: use the new data if it's there, otherwise read the old field. There, another if, written slightly differently. Somewhere else, a third flavor of fallback.

Each if made sense at the time it was written. Whoever needed it wrote it right where they needed it, to be quick. But taken together, nobody could answer the simplest question: if we delete this field, what breaks?

And when nobody can answer that, the default answer is always: just leave it.

The next migration

A few months later, we had to move another kind of data to a new structure. This one was bigger, touching the part that nearly every request goes through. I looked at that TODO and promised myself: this time, no line like that gets left behind.

The first job wasn't writing code. It was counting. How many places read the old data, and where from. The initial number made the team a bit nervous. But on closer look, most of them read through the same intermediate layer, not straight from the database. Keep that layer's shape unchanged, and they would never know a move was underway. The part that really had to change was much smaller than I thought.

The second job: the transition logic was allowed to live in exactly one place. A single module knew about both the new data and the old. Everything else called through it. In the task's definition of done, I added one line: no fallback lives outside this module. At review time, grep.

Backfill night

The scariest part of any migration is the backfill, when all the old data moves to the new place. Anyone who's been at this a while has probably had a night of watching a script run, praying it doesn't die halfway.

This time we split the data into small ranges, each range its own job in the queue, with a fixed ID. If a range failed, only that range reran. Rerunning the whole thing from the start produced the same result as before.

On the real night, one range failed. It retried on its own, then finished. I found out the next morning, reading the logs over a cup of coffee. It was the first migration night I got a full night's sleep.

The day the field goes

We didn't do it big-bang. The old field stayed as a safety net. Deploy the new code, write to both places in parallel, run the backfill, reconcile the data on production. Only once everything matched would we think about deleting.

That day hasn't come yet. But when it does, I know that PR will touch exactly one file: remove the fallback branch in the transition module, plus a migration that drops the old field. Review will probably take under five minutes.

And there won't be a TODO left behind for the next new person to ask about.

Contact

Looking for a Tech Lead?

I want to own the technical direction and help a team ship products that matter.

huybn.dev97@gmail.com