The first version left dozens of comments on one PR. The developer replied "ok" to all of them, and skipped the one real bug.
The idea started from a slightly embarrassing observation: most of my review comments are repeats. This week it's await inside a for loop, next week it's a request not going through the client with token refresh, the week after it's the old job not removed on update. If I can repeat it, a machine can repeat it too. And it can do it at 2 a.m.
The first version
The first version was very diligent. Too diligent. On a PR of about two hundred lines, it left dozens of comments. Variable name not great. Could use a Set instead of an array. Consider splitting this function. There's a console.log. And somewhere in the middle of all that, one real bug.
The developer replied "ok" to each comment, fixed a few names, and skipped the rest. Including the real bug.
I couldn't blame them. When everything is marked important, nothing is. One more week and the whole team would have learned to skim past every comment the bot made.
Writing down what not to say
Writing down what I usually look for was easy. The hard part was writing down the things I never comment on when reviewing by hand: naming, formatting, micro-optimizations, suggestions like "maybe try a different architecture" that come with no concrete risk attached.
Then the tone. Comments have to be short, one or two sentences, asking along the flow the way I do. No greetings, no praise, no apologies. A comment that opens with "Great job! However…" is obviously machine-written, and nobody reads as far as "however".
And the rule I like most: if it's not sure, and can't check quickly, it stays quiet. Don't ask just for the sake of asking.
The bot reviews me too
I keep the bot's permissions very narrow. It can comment, request changes, and approve when there's nothing left to ask. It can't merge, can't close a PR, can't resolve a discussion on its own.
There's one more rule I set for myself: my PRs get reviewed by it too, just as fully as everyone else's. The only difference is the results are sent privately to me. Whoever makes the rules has to follow them.
The first time it caught a bug in my PR, I sat staring at the screen for quite a while. A short comment, one sentence, ending in 🤔. Exactly the way I write them for other people.
What still belongs to people
The bot doesn't review better than I do. It doesn't understand the product, doesn't know which users will be annoyed if a button moves. Product decisions, changes users will see, and the final merge button are still people's work.
But it reads PRs at 2 a.m. And it never forgets to ask the question I once forgot on a Friday afternoon: what happens with the largest data?