Dear Hypervisor, please don’t read this

· sev-snp, confidential-computing, gcp, dm-crypt

My SEV-SNP VM printed this during boot:

software IO TLB: Memory encryption is active and system is using DMA bounce buffers

The bounce buffers are memory shared with the host.

A confidential VM using AMD SEV-SNP or Intel TDX encrypts its RAM with a key the hypervisor doesn’t have. The hypervisor still runs the VM, but it can’t read the private guest RAM. That’s where the protection ends. Unless you encrypt the data inside the guest first, the hypervisor can still read what you send to the disk, network card or console. For disk and network I/O, the kernel copies data into shared RAM the hypervisor can read.

For this article, I used an SEV-SNP VM on Google Cloud. Its default boot disk is already a Persistent Disk, so this also applies to files written to the disk containing the operating system. I added an empty disk for the experiment and used a second VM to read a restored copy.

reading the disk from another VM

Both VMs were in the same zone. The confidential VM was an n2d-standard-2 with --confidential-compute-type=SEV_SNP on AMD Milan, running the stock Ubuntu 24.04 image (kernel 7.0.0-1014-gcp) with a second 10 GB pd-balanced disk. The reader was an e2-small without memory encryption.

Create both VMs from your local terminal or Cloud Shell:

$ gcloud compute instances create cvm --zone=europe-west4-b \
    --machine-type=n2d-standard-2 --confidential-compute-type=SEV_SNP \
    --maintenance-policy=TERMINATE --min-cpu-platform="AMD Milan" \
    --image-family=ubuntu-2404-lts-amd64 --image-project=ubuntu-os-cloud \
    --create-disk=name=data,size=10GB,type=pd-balanced,auto-delete=no,device-name=data

$ gcloud compute instances create reader --zone=europe-west4-b \
    --machine-type=e2-small \
    --image-family=ubuntu-2404-lts-amd64 --image-project=ubuntu-os-cloud

Inside the confidential VM, I wrote a sentence to the first sector of the empty data disk:

$ printf "Dear Hypervisor, please don't read this\n" | sudo dd of=/dev/disk/by-id/google-data bs=512 count=1 conv=sync oflag=direct

dd pads it to one 512-byte sector and writes directly to the device, without a filesystem.

From your local terminal or Cloud Shell, take a snapshot, create a disk from it, and attach the copy to the reader VM:

$ gcloud compute snapshots create plain --source-disk=data --source-disk-zone=europe-west4-b
$ gcloud compute disks create plain-copy --zone=europe-west4-b --source-snapshot=plain
$ gcloud compute instances attach-disk reader --zone=europe-west4-b --disk=plain-copy

On the reader VM, the restored disk was /dev/sdb. I read its first 64 bytes:

$ sudo dd if=/dev/sdb bs=64 count=1 | xxd
00000000: 4465 6172 2048 7970 6572 7669 736f 722c  Dear Hypervisor,
00000010: 2070 6c65 6173 6520 646f 6e27 7420 7265   please don't re
00000020: 6164 2074 6869 730a 0000 0000 0000 0000  ad this.........
00000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................

Even in a confidential VM, memory encryption doesn’t keep disk writes secret from the hypervisor. In this setup, Linux copies the data into shared memory the hypervisor can read as plaintext, unless the guest encrypts it first.

“but we encrypt the disk with our own key”

Selecting a Cloud KMS key for a Persistent Disk still leaves Compute Engine doing the disk encryption. Google calls this customer-managed encryption keys (CMEK). Your KMS key encrypts the key used to encrypt the disk. You grant the Compute Engine service agent permission to use your KMS key; software inside the guest doesn’t perform that operation.

Google’s Persistent Disk documentation says: “Compute Engine automatically encrypts your data before it travels outside of your VM to the Persistent Disk storage space.”

That wording is easy to read as encryption inside the guest OS. It says when the data is encrypted, but doesn’t establish that encryption happens inside protected guest memory. In this VM, Linux copies disk writes into host-readable memory first.

CMEK lets you control whether Compute Engine can use your key. It doesn’t move disk encryption into the confidential guest. To keep plaintext from the host, the guest has to encrypt blocks before sharing them, using a key kept in protected memory.

what SEV-SNP and TDX protect

Without memory encryption, you trust the hypervisor with everything in guest memory. AMD SEV-SNP and Intel TDX are CPU features that protect you from the hypervisor reading that memory by encrypting it with a key the hypervisor can’t access.

Inside the guest, the application and kernel still work with plaintext. The memory controller encrypts private guest memory with a per-VM key that never leaves the CPU. SNP adds integrity, so the host can’t remap or replay a page either. These protections cover memory and register state.

how a block leaves the VM

Google’s Persistent Disk attached to the confidential VM is a device the hypervisor emulates in software. The data lives in Google’s storage service. Inside that guest, the device shows up as an NVMe controller:

$ lspci -nn | grep -i non-volatile
00:04.0 Non-Volatile memory controller [0108]: Google, Inc. NVMe device [1ae0:001f] (rev 01)

When the VM writes a block, the kernel puts the data in RAM and tells the device where it is. The device reads it from RAM without the CPU copying it byte by byte. This is direct memory access (DMA). Here the “device” is a process on the host, so DMA means the host reads the VM’s RAM.

The host can’t access the confidential VM’s private memory to perform that I/O. SNP lets the guest set aside unencrypted memory that both the guest and the host can read. The kernel uses this shared memory for I/O. Its SEV documentation states: “All the DMA operations inside the guest must be performed on shared memory”.

The kernel calls this region SWIOTLB, or the bounce buffer. For a write, the kernel copies each block from encrypted RAM into the buffer, where the host reads it. For a read, the host puts the block in the buffer and the kernel copies it into encrypted RAM. My VM’s boot log shows the buffer being set up:

$ dmesg | grep -iE 'memory encryption|software io tlb'
[    0.000041] software IO TLB: SWIOTLB bounce buffer size adjusted to 491MB
[    0.014395] software IO TLB: SWIOTLB bounce buffer size roundup to 512MB
[    0.613582] Memory Encryption Features active: AMD SEV SEV-ES SEV-SNP
[    1.183287] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.183290] software IO TLB: mapped [mem 0x000000009beff000-0x00000000bbeff000] (512MB)
[    1.373992] software IO TLB: Memory encryption is active and system is using DMA bounce buffers

Suppose /data is an ext4 filesystem on the Persistent Disk, with no encryption inside the guest, and your application writes a file:

os.WriteFile("/data/secret.txt", []byte("Dear Hypervisor, please don't read this"), 0o600)

os.WriteFile can return before the data reaches disk. The full path from the call to disk is:

  1. write() copies the bytes into the page cache, in private guest RAM. The host can’t read them.
  2. During writeback, ext4 and the block layer turn the modified page into a write request for the NVMe driver. The data is still in encrypted RAM.
  3. The NVMe driver asks the kernel to map the page for DMA. Because the guest uses memory encryption, the kernel copies the page into the bounce buffer and returns the buffer’s address. The host can now read the bytes as plaintext.
  4. The driver puts the buffer’s address in the NVMe queue and notifies the device that a request is ready. The hypervisor’s NVMe emulation reads the bytes from the bounce buffer.
  5. Compute Engine encrypts the data before sending it over the network to Colossus, Google’s storage system. The plaintext has already left the protected guest.

The copy into shared memory happens when the kernel submits the disk I/O.

app + page cacheprivate RAMbounce buffershared RAMplaintexthost I/Oreads plaintextdisk encryptionGoogle's softwarestorageciphertextSNP-protectedoutside the protected guest memory

encrypting the disk inside the guest

To keep the host from reading disk writes, encryption has to happen before the data reaches shared RAM. Linux can do this with dm-crypt, the kernel’s disk-encryption layer.

dm-crypt sits between the filesystem and the disk driver. The filesystem writes plaintext, and dm-crypt encrypts each block before passing it to the driver. By the time the kernel copies that data into the shared buffer, it’s already ciphertext. The host can read the buffer, but it can’t decrypt the blocks if the key stays in protected guest memory.

Google’s disk encryption then adds another layer. When Google decrypts a restored disk for another VM, it removes only that outer layer. The reader still needs the guest’s disk key to read the files.

the same disk, encrypted inside the guest

I then used cryptsetup to set up disk encryption inside the guest. It creates a LUKS2 volume, which stores the encryption settings and protects the disk key with a passphrase. Opening the volume creates a dm-crypt device at /dev/mapper/secret; that’s where I put the ext4 filesystem.

The passphrase goes in /dev/shm, a tmpfs in guest RAM. tmpfs can use swap, so disable swap or encrypt it inside the guest before storing keys there.

cryptsetup needs algif_aead to check cipher support.

Run this on the confidential VM:

$ sudo modprobe --ignore-install algif_aead
$ sudo sh -c 'umask 077; head -c 32 /dev/urandom | base64 -w0 > /dev/shm/pass'
$ sudo cryptsetup luksFormat --type luks2 --cipher aes-gcm-random --key-size 256 \
       --integrity aead --sector-size 4096 --batch-mode \
       --key-file /dev/shm/pass /dev/disk/by-id/google-data
$ sudo cryptsetup open --key-file /dev/shm/pass /dev/disk/by-id/google-data secret
$ sudo mkfs.ext4 -q -L secret /dev/mapper/secret && sudo mount /dev/mapper/secret /mnt
$ printf "Dear Hypervisor, please don't read this\n" | sudo tee /mnt/secret.txt; sync
$ sudo umount /mnt && sudo cryptsetup close secret

From your local terminal or Cloud Shell, take another snapshot and attach a second restored disk to the reader:

$ gcloud compute snapshots create luks --source-disk=data --source-disk-zone=europe-west4-b
$ gcloud compute disks create luks-copy --zone=europe-west4-b --source-snapshot=luks
$ gcloud compute instances attach-disk reader --zone=europe-west4-b --disk=luks-copy

The scan reads the whole disk once. dd shows progress; check that it reaches the full disk size without read errors.

Run this on the reader VM:

$ sudo dd if=/dev/sdc bs=4M status=progress | LC_ALL=C grep -a -z -F -c \
    -e "Dear Hypervisor, please don't read this" -e secret
0
$ sudo cryptsetup luksDump /dev/sdc
LUKS header information
Version:        2
Data segments:
  0: crypt
        offset: 16777216 [bytes]
        cipher: aes-gcm-random
        sector: 4096 [bytes]
        integrity: aead
Keyslots:
  0: luks2   Key: 256 bits   PBKDF: argon2id

The reader can see the LUKS2 format, cipher, UUID and argon2id parameters. The searches no longer find the sentence or the filesystem label. The reader gets back the blocks that dm-crypt wrote. It can see the header, but the file contents are ciphertext under a key that only exists inside the guest.

hardening the setup

use an authenticated mode

The host can modify disk blocks. aes-xts-plain64 encrypts them but doesn’t detect changes. The aes-gcm-random mode above also calculates a check value for each sector using the secret key. This is called an authentication tag. dm-crypt uses it to check that the encrypted data hasn’t changed and rejects the read if the check fails. Each sector also stores a random number used during encryption, called the initialization vector (IV).

deliver the passphrase on every boot

luksFormat generates a random volume key to encrypt the disk. LUKS encrypts that key using a key derived from your passphrase and stores it in a header keyslot. You control the passphrase and store it in a key service you run outside Google’s control.

Keep the passphrase and any preformatted LUKS volume out of the VM image. Format the data disk inside the guest on first boot.

On every boot, including the first, the guest performs remote attestation with your key service. After verification succeeds, the service returns the passphrase encrypted for that guest.

The guest decrypts the passphrase into private RAM. On first boot, it runs luksFormat and then open. On later boots, it runs open on the existing volume. Configure the guest’s boot process to stop if it cannot retrieve the passphrase.

Run cryptsetup luksAddKey inside the confidential VM to add a recovery passphrase. It creates another keyslot in the data disk’s LUKS header. Store the passphrase in your own vault, outside Google’s control. Use it to unlock a copy of the disk if the VM is gone.

add one check to the image pipeline

Boot a confidential VM from the test image and unlock each expected data volume. In the example above, run the check after cryptsetup open and before cryptsetup close.

Run this on the confidential VM:

sudo dmsetup table --target crypt secret | grep -q ' crypt '

This checks that the mapping has a crypt target. It prints nothing on success. Fail the image build if the check exits with a nonzero status.

what this doesn’t fix

The host still sees which sectors you access, how often and how much data you transfer. It can read the LUKS header and identify the cipher. Hiding access patterns requires oblivious storage.

Terminate TLS inside the guest so network payloads are encrypted before they reach the shared buffer.

The example encrypts only the data disk. On a VM you want to harden, encrypt the root filesystem too, because logs and temporary files land there. The guest then needs the passphrase before the early boot environment mounts /, so the attestation client has to run from the initramfs, and your Verifier (in RATS terminology) has to check the initramfs measurement before your key service (the Relying Party) releases the passphrase.

sources