skip to content

Search
Syspirit
EN

#Patching a million VMs in a week: OVH's Januscape story

OVH had to patch its whole KVM fleet in a hurry after a critical flaw. A look at a fascinating write-up: canary, stop thresholds, anti-affinity and very real headaches.

8 min read Karl Certa
#virtualization #kvm #ovh #security #review #cloud #infra
Network operations crisis room at night, teams of engineers facing a wall of screens during the Januscape patching campaign
AI-generated image

Picture the scene. Tuesday, July 7th 2026, early afternoon, a security alert comes in. A critical flaw in KVM, the virtualization engine that runs almost all of your instances. Tens of thousands of hypervisors, around one million customer VMs sitting on top of them. And an exploit that crashes an unpatched host in two minutes flat. You have a few days to fix everything, not a few months.

That’s exactly what happened to OVHcloud, with the Januscape flaw (CVE-2026-53359). The good news is that they published a detailed write-up about the whole operation, signed by Julien Levrard, their CISO.

What interests me here isn’t really the flaw itself. It’s the logistics. How do you patch a fleet like this in a hurry, worldwide, without breaking everything.

The flaw in 30 seconds

Just enough to understand what’s at stake, without diving into the guts of the kernel.

Januscape is a flaw in KVM, the layer that isolates VMs from each other on the same physical server. Normally, each VM is locked inside its own bubble: it can’t see the host running it, nor its neighbours. Januscape breaks that bubble.

Here’s how the attack works: a customer who is root on their own VM can exploit a memory management bug in the kernel to escape their machine and take control of the physical server. Once on the host, they can crash it, or go poking around in the VMs of other customers running right next to them. This is called a guest-to-host escape, and on shared hosting, it’s the worst case there is.

The technical detail, a use-after-free in KVM’s shadow-paging, is well explained here if you’re interested.

Three things that explain the urgency:

  • The exploit is reproducible: an internal test crashed an unpatched host in about two minutes.
  • Every x86 Linux kernel before the fix commit is affected, whatever the distro.
  • At the time, no exploit code allowing a host takeover was public. But everyone knew it was only a matter of time.

So the question wasn’t whether to patch, but how to do it at this scale, knowing that zero customer impact was simply impossible.

Five options on the table

This is the part I find the most instructive, because it shows the reasoning. OVH listed five ways to deal with the problem, and ruled out the first four one by one:

  1. Wait for the official patched kernels. That meant depending on someone else’s timeline and staying exposed for an uncontrolled amount of time. Ruled out fast.
  2. Live patch (patching the kernel on the fly). This requires enabling an option that lowers the kernel’s hardening and limits detection if it gets compromised. Too risky at fleet scale, especially if a takeover exploit came out. No.
  3. Disable nested virtualization. This makes the exploit useless, but it breaks live-migration and the impact on customer usage was impossible to gauge. Ruled out.
  4. Live-migrate the VMs to already-patched hosts. Great for service continuity, no impact on the VMs. Except that copying a million VMs from host to host takes months, not days. Kept for a few critical VMs only.
  5. Backport the patch into their Debian kernels and reboot every host. The option they went with.
OVH sketch summarizing the five mitigation options considered for the Januscape flaw

Original sketch from OVHcloud’s write-up.

The part worth pausing on is the decision behind option 5: unilateral patching with controlled impact. OVH decided to patch and reboot without waiting for each customer’s individual consent, knowing full well that some services would go down.

The reasoning fits in three lines: not patching leaves the whole fleet exposed to a critical flaw; handling it case by case would stretch the timeline and leave most hosts vulnerable for weeks; acting fast and globally protects the most people, even if it temporarily hits a minority.

As a customer, getting your VM rebooted without asking for anything stings. But given the alternative, it’s hard to blame them. The priority was no longer to avoid impact, but to minimize it, spread it out and make it predictable.

How it worked: follow the sun and canary

The operation ran follow the sun: a crisis team active 24/7, rotating by geographic zone. Each region took over during its own local morning and handed the context to the next. Three sync points a day to keep the teams aligned (NOC, experts, support, security).

Sydney was used as the pilot region. Why? Few hosts, so limited risk, and above all the local off-peak window (night in Sydney) lines up perfectly with office hours in Europe. Ideal for validating the runbook under real conditions, at small scale, before rolling it out everywhere. A canary, basically, but at the scale of a whole datacenter.

Once the procedure was stable on Sydney, follow the sun kicked in: first European wave the same evening, then region after region.

The order wasn’t uniform. VPS first, Public Cloud second, because the exposure isn’t the same: on VPS, the impact per reboot stays contained, while a Public Cloud region packs thousands of customers and critical instances (databases, load balancers, message queues). For Public Cloud, they started with the least dense regions, then worked at a finer granularity on the big ones.

The safeguards that make the difference

Two mechanisms are worth a look, and they scale down to any size.

Stop thresholds. Each reboot wave had a stop threshold: if too many hosts were down at the same time (15 in the big regions, 5 elsewhere), the wave paused. A safety catch so you don’t keep rebooting while the on-site technicians are still struggling to bring the previous machines back.

Anti-affinity. The real risk for a customer isn’t the reboot itself, it’s seeing all their instances go down at once when they had deliberately spread them out to be resilient. So the orchestrators never rebooted two hosts of the same customer at the same time. Best-effort, not guaranteed 100%, but enough to make the impact absorbable at the application level.

The headaches, because there are always some

What I like about this write-up is that it doesn’t sugar-coat anything. OVH was transparent about the various problems they hit and what their operation involved:

  • VMs not restarting. From the very first European wave, libvirt-guests was conflicting with Nova and shutting instances down without telling the API. Fix: disable and mask the service before reboot.
  • Data corruption. Some forced reboots landed right in the middle of a disk write. Fix: extend the graceful shutdown window to 60 seconds before the kill.
  • Hardware giving out. When you reboot tens of thousands of machines, something is bound to break. On the first night, 20 to 30 hosts out of 6,000 didn’t come back on their own: dead memory sticks, CMOS batteries to remove, network interfaces down. Technicians were on standby at every site for everything no tool can do in their place.

That last point is one we often forget when we think “cloud fleet”. Behind the orchestrator, there are people in the datacenters, ready to jump on a rack at 3am.

Communication during the crisis

Communicating widely during the operation was a dilemma: spelling out the rollout sequence meant handing attackers the map of the hosts still vulnerable, and possibly pushing curious people to “test” the available exploit. So OVH chose discretion during execution.

In practice: no public status page at first, a targeted and progressive communication, sent only to customers whose instances were on hosts scheduled for reboot, region by region.

It wasn’t perfect, and they say so. For big regions like GRA6 (nearly 90,000 customers affected), mass emailing was ruled out to avoid a flood of tickets. Some messages never arrived (outdated addresses, filtered notifications). The pivot: a conditional banner in the Manager, via feature flipping, shown if the logged-in customer was one of the impacted accounts. Built and deployed mid-operation.

Conclusion

This write-up is genuinely interesting. You can see the team coordination, the cool heads and the method it takes in a company that sells cloud and high availability. And for them, it was a first:

Patching and restarting every Public Cloud and VPS host had never been done before under this kind of time pressure.

In just 11 days, honestly, that’s respectable. They accept from the start that there will be side effects, but zero impact was never the goal, as they say themselves:

Rebooting a fleet of this global scale with zero impact was not an achievable goal. The objective was therefore the smallest possible impact compatible with the security of the entire fleet.

What stays with me most is their choice to move quietly. No big public announcement, just a targeted region-by-region communication, so as not to hand attackers the list of hosts still vulnerable. Pulling off an operation this size with such a small impact, and almost on the down-low, I find that pretty impressive.

And given the pace at which kernel flaws keep coming out, they warn that this will happen again. They plan to do better next time, especially on informing customers ahead of time.

Sources: OVHcloud’s write-up by Julien Levrard, spotted via The Register.