The Memory Fix That Broke Its Own Alarm

Covers August 2026 · published August 2026

My AI Second Brain runs on a dedicated machine, and Claude Code Remote Control is the service that keeps every session alive on it. A few weeks ago that service started dying, the same way, over and over: memory climbed for hours, then the whole box went into swap and everything froze at once. Not one session. All of them.

The fix looked simple. Give the service a hard memory ceiling, so a runaway session gets killed on its own instead of taking the whole machine down with it. I set MemoryMax to a real number and moved on.

It didn’t work. The service still dumped memory into swap and dragged the whole box down with it. MemoryMax is a real, hard cap on the cgroup’s own RAM use, that part isn’t in question. The gap is what it doesn’t cover: by default a cgroup can still reclaim itself onto swap before RAM alone hits the ceiling, so the host kept filling with swap even though the RAM cap was holding. The actual fix needed a second setting, MemorySwapMax=0, which forbids swap entirely and makes the ceiling mean something for the whole system, not just the cgroup’s RAM line.

That produced a new failure. With swap off and a soft throttle zone (MemoryHigh) still set, the service didn’t crash, it just froze, silently making no progress at all, using no swap and no more memory. A process-presence check would have called that healthy. It wasn’t, it was stuck in reclaim with nowhere left to go. I only found it by testing in an isolated throwaway copy of the service first, not the real one. Dropping MemoryHigh fixed it: a clean kill, a clean restart, no more freeze.

Three weeks later, that recipe held, no recurrence on this host, though I hadn’t run a forced load test against it either. Then I built something that used it as a foundation, and it broke the recipe in a way I hadn’t considered at all.

The problem wasn’t the memory limit. It was what I’d told it to do when the limit fired, and what actually happens when it fires depends on something the recipe itself never mentioned: what’s running inside that cgroup.

Remote Control isn’t one process. It’s a supervisor, with every live session running underneath it as its own child, all sharing the same cgroup.

That matters because OOMPolicy=stop tears down the whole systemd unit on a kill, cgroup included, new identity, no history, but only if the kill actually reaches the supervisor. If the OOM killer instead picks a runaway child session, the supervisor survives, the unit never goes down, and the cgroup’s history survives with it. The recipe’s real behavior depends on which one gets picked, and the first draft of this piece didn’t say that.

I’d since built a small watchdog that tracks memory pressure over time by watching that cgroup. Under stop, an OOM event that reaches the supervisor quietly resets the watchdog’s own baseline. It never gets the chance to notice a pattern, because from its point of view, a brand new, perfectly healthy service shows up every time the old one dies. The alarm was never wrong. It just couldn’t ring, by construction, in exactly the failure mode most worth catching.

Worse, I found a second bug hiding inside the first one. Earlier that same day, I’d already edited the config file to OOMPolicy=continue. But a systemd config edit only takes effect once you tell systemd to reread it, and that step never happened. The file said one thing. The running service was still doing the other. Nothing about the service looked broken. It just wasn’t running what the file said it was.

Both got fixed the same evening. Reloading picked up the policy change, confirmed by reading it back from the running unit, not assumed from the file. Separately, the memory ceiling itself had already been live-adjusted earlier without a restart, confirmed by reading the actual cgroup value off disk before and after, with the process’s own ID and start time unchanged throughout, meaning that specific change genuinely cost nothing. I only trust that because I checked it, not because reloading a config file sounds like it should be non-disruptive.

The watchdog confirmed its own fix. It logs its assumption on every run, and the moment the policy actually flipped, its own log line changed from flagging the assumption as broken to confirming it held, with nobody telling it to. That’s a real, useful signal that the config now matches reality. It is not, on its own, proof the alarm will actually fire the next time a real OOM happens, that’s a different test I haven’t run yet.

Why contain it instead of fixing the actual leak

The honest version of this question deserves a real answer, not a dodge. I checked what was actually driving the memory growth before deciding containment was the right response, not the easy one.

It isn’t a classic leak. A freshly started instance sits healthy, and there’s no single process that grows without bound. What’s actually happening is concurrency accumulation: each live session is a real, separate process, dozens of them can be running at once over a long uptime, and nothing was ever scheduled to recycle that accumulation before this fix. There’s no leak to patch, because there’s no one thing that’s broken. The growth is the intended behavior of the system, running longer and busier than the original design assumed.

That’s also why I didn’t reach for a durable, cgroup-independent signal, like restart counters or journal-parsed OOM records, as a full replacement for cgroup-scoped monitoring. Those are real and worth having alongside this, they’d survive a supervisor-level kill that this approach can still miss. But they don’t tell you the thing a live cgroup read does: how close the current, actually-running set of sessions is to the wall right now, before anything fires. I’d rather have both than pick one and call it settled.

What actually held up, on this host (systemd 255, cgroup v2, Linux Mint), checked again while writing this

• MemoryMax hard-caps a cgroup’s own RAM. It does not by itself stop that RAM from being reclaimed onto swap, MemorySwapMax=0 is what closes that gap.
• MemoryHigh combined with forbidden swap can stall a process in reclaim instead of producing a clean OOM kill. For fail-fast behavior, drop MemoryHigh entirely.
• OOMPolicy=continue only preserves the cgroup’s history if the process the kernel picks to kill is a child, not the supervisor. State your process model before trusting this.
• A config file being correct means nothing until the running service actually reads it. Reloading did apply a live memory-ceiling change on this system with zero restart, confirmed by reading the cgroup value directly, but I checked that rather than assumed it, and I’d check it again on a different systemd version before trusting it there too.
• A monitor that logs its own assumption, and flags when that assumption breaks, catches a config-drift lie a static comment never could. It’s a good instrument for that one thing. It isn’t proof the alarm works end to end under a real kill.

What I actually changed, and how you can check it

The corrective action: switch OOMPolicy from stop to continue, apply it live, and build a watchdog that checks its own operating assumption on every run instead of assuming the config is what it says it is.

Proof it worked, re-runnable, not just asserted:

systemctl --user show claude-remote-control.service -p OOMPolicy

returns OOMPolicy=continue today, read from the live unit, not the file. And the specific claim that a memory ceiling can change without a restart isn’t a memory of what happened weeks ago, it’s a test I re-ran tonight, on a disposable unit, while writing this piece:

systemctl --user daemon-reload

after editing only MemoryMax in the unit file, with no restart, no set-property, showed the running cgroup’s memory.max change while MainPID and ActiveEnterTimestamp stayed identical. Anyone can reproduce that in five minutes on their own throwaway unit.

Exactly how it works: daemon-reload re-reads unit files and pushes resource-control properties, MemoryMax, MemorySwapMax, OOMPolicy among them, into the already-running cgroup, without touching the process itself. That’s different from properties like ExecStart, which only take effect on the next start. Knowing which category a setting falls into is the difference between a live fix and a fix that silently waits for a restart that may not come for weeks.

Compared against known best practice: site reliability engineering treats an alert that has never actually fired under a real, induced failure as unverified, not working, the standard guidance is to test monitoring against the actual failure mode, not just the code path that’s supposed to detect it. This piece’s watchdog conforms to that in one respect: it self-checks its own live assumption on every run, rather than trusting a comment. It does not yet conform in the harder respect, I have confirmed the assumption is correct, not that the alarm fires end to end under a real, induced OOM kill. That’s the next test, not a finished one, and I’d rather say that plainly than round it up to done.

The lesson

Every fix in this chain matched the specific failure I was looking at when I made it. None of them was complete containment, and I don’t think a single recipe like this ever fully is. Each one also opened a new, different way to fail quietly, one I only found because something I built later depended on an assumption something I built earlier had broken without telling anyone. The actual discipline isn’t getting the setting right once. It’s naming what you didn’t test, building something that tells you the moment an assumption stops holding, and trusting what the live system is actually doing over what the file says it should be doing.

If this helps you, check out my adblob to support the work.