Triage, Investigation, Reproduction: The unspoken monotony

Most of the hours in support go into a loop nobody talks about: triage, investigate, reproduce.
Then, again.

Every support engineer knows the moment. A ticket lands at 8:47 on a Monday: "Since the update last week, our integration fails at random with 401 errors. Please fix ASAP."
No endpoint, no request ID, no timestamp. Just urgency.

What follows never makes the conference talk or the resolution note. It's the same loop as last week: triage, investigate, reproduce. Then again for the next ticket. This is the unspoken monotony of support, where most of the hours go and most of the expertise lives.

The ticket will eventually close with one line: "Root cause: clock drift on customer's new worker node. Resolved." Here's what that line hides.

Triage: reading what wasn't written

The loudest word in the ticket is "update." The tempting move is to pull the release notes and hunt for a regression. The word that matters is quieter: "random." A broken credential fails every request. When only some fail, those requests differ from the rest in a way nobody has spotted yet.

So the reply skips the generic checklist and asks:

What share of requests fail, and when did it start?
Can you send request IDs for a few failed calls?
How many servers run the integration?
What changed in your environment around then?

The answers: about a third of calls fail, starting Tuesday. Three worker nodes. On Tuesday, one was replaced with a VM built from a new image. The "update" was a guess; the customer saw our release notes the same week and connected the dots.

None of that was in the ticket. All of it matters. Triage is turning an anxious sentence into a problem statement, and it's monotonous because the questions are nearly always the same.

Investigation: archaeology in someone else's stack

The failed request IDs lead straight to the authentication logs:

08:12:03Z WARN token_rejected client=acme-sync-prod reason="iat in future" leeway=60s src=10.4.2.17
08:12:41Z INFO token_issued   client=acme-sync-prod src=10.4.2.15

Every rejection comes from one node, and every rejected token claims to be about three minutes from the future. The service tolerates 60 seconds of drift. One command on the customer's side seals it:

$ timedatectl
System clock synchronized: no
NTP service: inactive

A search of old tickets finds last year's twin, filed under completely different words: "tokens expire the moment we get them." Same cause: a hardened VM image with time sync switched off.

One log query, one command, one old ticket written well enough to find. The monotony is doing this every time, including when the old ticket just says "fixed, closed."

Reproduction: rebuilding someone else's world

A theory only becomes a finding once it reproduces, and that means recreating the customer's world on your own machine.

The first attempt fails, because every test machine keeps perfect time. So the engineer writes a small client that lies about the clock. At 45 seconds ahead, everything works. At 90, every request fails with the customer's exact log line. Theory confirmed, threshold found, our release cleared.

And that's the easy version. A sign-in loop hitting only some users might mean matching an identity provider, a session policy, a strict browser, and a proxy that inspects traffic. Here, "works on my machine" is the problem. It's also the stage most often skipped under pressure, which is how teams ship workarounds for bugs they never understood.

Why nobody talks about it

Dashboards track response time and resolution time, not the six hours spent reproducing. Good stories are about the clever fix, not the day spent building an environment and failing to reproduce an issue you have received 3 emails within 5 hours. And saying "I spent all day rebuilding an environment" sounds like complaining or an excuse.

But invisible work doesn't get planned, staffed, or improved. The engineers stuck in the loop just burn out quietly.

Making the monotony pay

You can't eliminate the loop. You can stop paying for it twice.

Ask specific questions. "Please provide more details" helps nobody. "Send two failed request IDs and tell us how many servers you run" often settles it in one exchange.
Write investigation notes worth finding. The clock drift case was fast because someone, a year earlier, wrote more than "fixed."
Keep reproduction kits. Sample projects per platform, a clock-faking client, a VM image with time sync off. The first repro is expensive; the second should be cheap.
Document repeat offenders. Three reproductions of the same cause belong in a troubleshooting article, not in three engineers' heads.
Measure the phases, not just the outcome. Tagging where the time went shows exactly where tooling pays back first.
Automate the gathering, not the judgment. Let tools pull logs, find similar tickets, and scaffold repros. A tool that leaps from symptom to answer is just skipping the loop, and the customer pays when it's wrong.

The monotony is the craft

Great support engineers aren't defined by the occasional brilliant diagnosis. They're defined by running the same loop carefully, hundreds of times: reading what the customer didn't say, ignoring the loudest word in the ticket, and refusing to call it fixed until they've watched it break.

That loop is monotonous. It's also where the value is. The least we can do is talk about it.