Nobody calls it murder when the victim is still technically alive.
That's the trick. That's what makes this one beautiful, if you're the kind of person who can admire a crime while you're standing in the middle of it. The servers were up. The applications were running. Green lights across the board on every dashboard that mattered to anyone who mattered. And underneath all of it, ninety-two machines were suffocating in broad daylight.
The ticket said "intermittent out of memory errors." WebSphere crashing in the night. Application teams logging in the next morning to find their JVMs dead on the floor, heap exhaustion in the logs, no explanation. They'd restart the service. It would run fine. They'd file a ticket. The ticket would get closed. Can't reproduce.
Because here's the thing about this particular murder — the crime scene cleaned itself.
Every time someone logged into the guest to investigate, LabManager noticed. An active session. A human at the wheel. Priority restored. And the balloon driver — the thing that had been squeezing the life out of that VM's memory all night — quietly deflated. Gave the pages back. Smiled and straightened its tie. By the time the admin had the console open and was checking memory counters, everything looked fine. Plenty of headroom. Must have been a bad deploy. Must have been a memory leak. Must have been Java.
Java's always good for a fall.
I didn't walk in knowing anything. You never do. You walk in with a badge that doesn't open half the doors and a contract that says you're here to help, which means nobody wants to talk to you. So you do what you always do on the first week of a new job in a place that doesn't trust you yet. You walk the halls. You buy coffee for people who know things. You ask questions that sound dumb on purpose, because smart questions make people defensive and dumb questions make them generous.
"So how many VMs are we running in this environment?"
The number came back like a weather report. Casual. Ordinary. Ninety-two, give or take.
I kept my face still. Then I found one of the engineers in the break room. Good kid, hadn't learned yet to be careful about what he said to contractors.
"What are the specs on the Lab Manager hosts?"
He told me. Memory, cores, the whole sheet. Didn't think twice about it.
I went back to my desk and did the math on the back of a legal pad. Ninety-two VMs. The memory each one had been promised. The physical memory actually installed in the hosts underneath them. The numbers didn't add up. They didn't almost not add up. They were off by a factor that made my chest tight.
The house had sold every room twice. Maybe three times. And every guest thought they had the place to themselves.
I knew exactly what the crime was. Proving it was going to take ten months.
I should tell you about Jimmy.
Every job has a Jimmy. The mistake is thinking that tells you what kind.
This one? Not dangerous. Not stupid, exactly. Just... there. Always there. Standing close enough to hear the answer, far enough away to deny he didn't already know it. Jimmy's gift — his only gift — was knowing who to ask. Not what to ask. Not why to ask. Just who. He'd sidle up while you were elbow-deep in a problem, ask a question that sounded casual, nod like he was confirming something he'd already suspected, and then twenty minutes later you'd hear your own words coming out of his mouth in a meeting with someone who mattered.
A parrot with a lanyard.
Jimmy even called me from lunch at his Red Hat training bootcamp. Not after class. Not later that night with notes and questions. From lunch. While the instructor's words were still warm on the plate. He'd step outside with a sandwich in one hand and a phone in the other and ask me what the hell the thing he'd just been told actually meant. I'd explain it. Hypervisors. Memory pressure. Linux internals. Whatever chapter they were on that day. He'd listen, make the right little noises, then go back inside with my explanation tucked under his hat like it came with the course materials.
I should have charged tuition.
Was he helpful? Hard to say. He was always around. Always offering to carry something, relay something, "loop someone in." Like a sidekick you never hired and can't shake. A bad tail in a cheap suit who thinks he's your partner because he's been following you long enough. You'd look up from your desk and there he'd be, lingering at the edge of your cube with that helpful smile, and you'd think: what did I just say out loud?
Jimmy had been on the team before I got there. He'd be on the team after I left. Men like Jimmy always are. They survive because they're load-bearing walls in someone else's org chart — not because they hold anything up, but because nobody's sure what happens if you remove them.
The balloon driver. vmmemctl. Sitting in every guest like a sleeper agent, doing exactly what it was designed to do: when the host runs out of physical memory, it inflates inside the guest, claims pages the guest thought were free, and hands them back to the hypervisor. The guest doesn't crash. It doesn't throw an error. It just... takes. Quietly. Until there's nothing left for WebSphere to allocate, and the JVM does the only thing it can do — it dies.
And then the body disappears.
Because LabManager is watching. It knows who's active and who's idle. A VM running a batch job at 3 AM with nobody logged in? That's idle. That's the weakest gazelle in the herd, and the balloon driver takes from the weak to feed the strong. But the moment an admin opens a console session to investigate, the guest lights up as active again. LabManager shuffles the deck. The balloon deflates. Memory returns like it was never gone. The admin stares at clean counters, shrugs, and closes the ticket.
The perfect crime. Not because the evidence was hidden — because the evidence fled the scene every time a witness arrived.
The only way to catch it was to watch without being seen. Low-impact cron jobs. Scripts so small and quiet that LabManager didn't care about them. Didn't register them as activity. Didn't reprioritize the guest. Just a little heartbeat in the background, writing numbers to a log file nobody asked for, on a schedule nobody approved, capturing the real-time state of a machine that was being robbed in its sleep.
That's how you build a case against a killer who knows when you're watching.
Jimmy watched me build the scripts. Asked what I was doing. I told him. He nodded slowly, the way he always did, like he was filing it under a heading only he could see. I didn't think about it. You never think about Jimmy. That's his whole trick.
I charted a week of data. The picture was a staircase going the wrong direction. Real memory climbing in jagged steps from nearly zero to twelve gigs while virtual memory stayed pinned at sixteen — a promise the host had no intention of keeping. Every vertical drop in the line was a VM getting kicked sideways in the night. Restarts. VMotions. The hypervisor playing a shell game with memory it had already spent three times over.
Overcommitment. The dirtiest word in virtualization, and nobody wanted to say it.
I said it. I said it in a meeting, with charts. I said it with timestamps, with balloon driver metrics, with correlation data that mapped every WebSphere crash to a memory reclamation event on the host. I built the case the way you build any case — one piece at a time, in a database nobody asked me to build, using tools nobody gave me permission to use.
Ten months. Not to find the crime. I found the crime in a week with arithmetic and a legal pad. Ten months to prove it to people who had every reason not to believe me. Because believing me meant admitting the environment they'd designed, built, and signed off on was killing WebSphere in the dark and erasing the fingerprints before morning.
I presented the findings myself. Stood in front of the room with my Access database and my pivot charts and my ten months of balloon driver data and I laid it out cold. The overcommitment ratios. The memory staircase. The correlation between vmmemctl inflation events and every single WebSphere crash they'd been hand-waving for a year. Every recommendation backed by a vendor citation. Every finding linked to a timestamp they could verify themselves.
The room got quiet the way rooms get quiet when the argument is over and everyone knows it.
The VM admins had been the muscle — the ones who kept the lie alive. Every time an app team opened a ticket, the admins would log into the guest, watch the balloon deflate on cue, and close the case. Can't reproduce. They weren't lying. They were telling the truth about what they saw. They just couldn't see it, because looking at it made it go away. They weren't evil. That's the thing about henchmen. They never think they're henchmen. They think the environment is fine because the dashboard says it's fine, and the dashboard says it's fine because they built the dashboard to measure the wrong things.
But now the real dashboard was up on the projector, and the numbers didn't leave when you looked at them.
They had to act. There was nothing left to close, no ticket to bury, no counter to point at. The evidence was in the room and it wasn't going anywhere.
The fix was simple, the way all fixes are simple once you've spent a year proving the problem exists. Dedicated ESX host. Move the critical VMs off the overcommitted cluster. Stop letting the balloon driver run a protection racket on machines that were supposed to be production.
They moved the containers. The problems vanished. The software shipped.
And then a new role opened up. Administration of the now-stable environment. The environment I'd diagnosed, instrumented, prosecuted, and saved. A real position. Permanent. The kind of thing a contractor dreams about — the door from the outside to the inside, earned the hard way.
Jimmy found me at my desk. Leaned on the cube wall with that easy grin.
"You're a shoe-in," he said. "Don't even worry about it. I'll put in a good word."
And I believed him. That's the part that still gets me. Not that Jimmy did what Jimmy did — Jimmy always does what Jimmy does. It's that I sat there and relaxed. Let my guard down. Stopped campaigning for a job I'd already earned with ten months of receipts because a man who'd been watching me work told me to take it easy, he had my back, the fix was in.
The fix was in, all right.
Jimmy took the job.
Of course he did. Jimmy had been standing next to the work for so long that from a distance, in the right light, it looked like he'd been doing it. He knew the vocabulary because he'd heard me use it. He knew the architecture because he'd asked me to explain it. He knew the timeline, the milestones, the vendor citations — because Jimmy always knows. That's his whole job. Not to understand the thing. To understand who understands the thing, and to be standing close enough when the music stops.
I should have known better. You always should have known better, afterward. That's what afterward is for.
I built my dashboard in Microsoft Access. Yeah. I know. But I didn't have budget, I didn't have authority, and I didn't have time to fight a purchase order through a procurement system designed by people who think "monitoring" is something you do to children. I had VBScript, I had WMI, I had cron, and I had a man's stubbornness about being right.
Tables for app memory, CPU, netstat, ping, and the star witness: vmmemctl. Queries that cross-referenced balloon inflation events against WebSphere crash times. Forms that let me slice by environment, by time window, by host. Pivot charts that told the story so clearly that the only possible response was to either fix the problem or fire me.
Somewhere in a drawer, that database is still sitting. Its pivot charts still loaded. Its story already told. And Jimmy's in the next room, managing the environment it saved, wearing the job like a coat someone left on the back of a chair.
The balloon driver is still out there too. Sitting in the kernel, polite as a butler, waiting for the host to ask it for a favor. And when it does, your server won't crash. Your alerts won't fire. Your WebSphere will just die at 3 AM with nobody watching, and by the time you log in to check, the room will be clean, the bed will be made, and there won't be a scratch on the place.
You'll file a ticket. They'll close it. Can't reproduce.
And somewhere nearby, Jimmy will be nodding. Taking notes. Asking just the right question. Telling you to relax. Telling you he'll put in a good word.
Don't worry about a thing.
Case closed.