Blogging about open source virtualization

Switched my blog from wordpress to jekyll.

Automatic import handled most of content. Didn’t cover syntax highlight (extra wordpress plugin) though, so that needed some manual invention. Also copying over the (few) images was a manual process.

Configuration isn’t imported automatically, but that is just editing a few lines in _config.yml. Permalinks can be configured to be compatible to wordpress without much trouble, so all the article links stay valid.

My blog is stored in git now. Everything is just static pages. No database needed. No user data stored anywhere.

I’m pretty pleased with the result.

• Linux kernel 4.15,
• QEMU v2.11, and
• libvirt v4.0.

Why vfio-ccw?

How does this work?

Why subchannels and not ccw devices?

How is an actual I/O request processed?

Isn't that all a bit too synchronous?

Where is the IOMMU for this?

Does this cover everything supported by the architecture?

How can I try this out?

What's the deal with that "unrestricted cssids" thing?

What are the plans for the future?

• Added support for an interactive bootloader. As always, we strongly recommend to use the existing support in libvirt.
  To enable/disable, add the following element to your guest definition:
  
     <os>
       <bootmenu enable=’yes|no’ timeout=’n’/>
       ...
     </os>
  
  The timeout parameter specifies a timeout in milliseconds after which the default entry is chosen.
  Alternatively, set attribute loadparm to PROMPT to enable the boot menu without timeout in the respective disk's element:
  
     <disk ...>
       <boot order=’1’ loadparm=’PROMPT’/>
       ...
     </disk>
  
  Example:
  To enable the boot menu for 32 seconds for a guest using a libvirt
  domain XML format follows:
  
     <domain type=’kvm’>
       <os>
         <bootmenu enable=’yes’ timeout=’32000’/>
         ...
       </os>
• Exposure of guest crash information: When a guest is started using libvirt and crashes due to disabled wait, wrong interrupts or a program check loop, libvirt will print the information to the guest’s log, typically located at /var/log/libvirt/qemu.
  E.g. a crash due to a disabled wait results in an entry as follows:
  
     s390: psw-mask=’0xXXXXXXXXXXXXXXXX’, psw-addr=’0xXXXXXXXXXX
           XXXXXX’,crash reason: disabled wait
  
  Requires  libvirt v4.2.
• Added support for guests with more than 8TB of memory.

FROM fedora
RUN dnf install -y libguestfs libguestfs-tools-c virt-v2v \
                   libvirt-daemon libvirt-daemon-config-network

# https://bugzilla.redhat.com/show_bug.cgi?id=1045069
RUN useradd -ms /bin/bash v2v
USER v2v
WORKDIR /home/v2v

# This is required for virt-v2v because neither systemd nor
# root libvirtd runs, and therefore there is no virbr0, and
# therefore virt-v2v cannot set up the network through libvirt.
ENV LIBGUESTFS_BACKEND direct

For the QEMU 2.12 release we added support for a model of the Raspberry Pi 3 board (thanks to everybody involved in developing and upstreaming that code). The model is sufficient to boot a Debian image, so I wanted to write up how to do that.

Things to know before you start

Before I start, some warnings about the current state of the QEMU emulation of this board:

• We don’t emulate the boot rom, so QEMU will not automatically boot from an SD card image. You need to manually extract the kernel, initrd and device tree blob from the SD image first. I’ll talk about how to do that below.
• We don’t have an emulation of the BCM2835 USB controller. This means that there is no networking support, because on the raspi devices the ethernet hangs off the USB controller.
• Our raspi3 model will only boot AArch64 (64-bit) kernels. If you want to boot a 32-bit kernel you should use the “raspi2” board model.
• The QEMU model is missing models of some devices, and others are guesswork due to a lack of documentation of the hardware; so although the kernel I tested here will boot, it’s quite possible that other kernels may fail.

You’ll need the following things on your host system:

• QEMU version 2.12 or better
• libguestfs (on Debian and Ubuntu, install the libguestfs-tools package)

Getting the image

I’m using the unofficial preview images described on the Debian wiki.

$ wget https://people.debian.org/~stapelberg/raspberrypi3/2018-01-08/2018-01-08-raspberry-pi-3-buster-PREVIEW.img.xz
$ xz -d 2018-01-08-raspberry-pi-3-buster-PREVIEW.img.xz

Extracting the guest boot partition contents

I use libguestfs to extract files from the guest SD card image. There are other ways to do this but I think libguestfs is the easiest to use. First, check that libguestfs is working on your system:

$ virt-filesystems -a 2018-01-08-raspberry-pi-3-buster-PREVIEW.img
/dev/sda1
/dev/sda2

If this doesn’t work, then you should sort that out first. A couple of common reasons I’ve seen:

• if you’re on Ubuntu then your kernels in /boot are installed not-world-readable; you can fix this with sudo chmod 644 /boot/vmlinuz*
• if you’re running Virtualbox on the same host it will interfere with libguestfs’s attempt to run KVM; you can fix that by exiting Virtualbox

Now you can ask libguestfs to extract the contents of the boot partition:

$ mkdir bootpart
$ guestfish --ro -a 2018-01-08-raspberry-pi-3-buster-PREVIEW.img -m /dev/sda1

Then at the guestfish prompt type:

copy-out / bootpart/
quit

This should have copied various files into the bootpart/ subdirectory.

Run the guest image

You should now be able to run the guest image:

$ qemu-system-aarch64 \
  -kernel bootpart/vmlinuz-4.14.0-3-arm64 \
  -initrd bootpart/initrd.img-4.14.0-3-arm64 \
  -dtb bootpart/bcm2837-rpi-3-b.dtb \
  -M raspi3 -m 1024 \
  -serial stdio \
  -append "rw earlycon=pl011,0x3f201000 console=ttyAMA0 loglevel=8 root=/dev/mmcblk0p2 fsck.repair=yes net.ifnames=0 rootwait memtest=1" \
  -drive file=2018-01-08-raspberry-pi-3-buster-PREVIEW.img,format=raw,if=sd

and have it boot to a login prompt (the root password for this Debian image is “raspberry”).

There will be several WARNING logs and backtraces printed by the kernel as it starts; these will have a backtrace like this:

[  145.157957] [] uart_get_baud_rate+0xe4/0x188
[  145.158349] [] pl011_set_termios+0x60/0x348
[  145.158733] [] uart_change_speed.isra.3+0x50/0x130
[  145.159147] [] uart_set_termios+0x7c/0x180
[  145.159570] [] tty_set_termios+0x168/0x200
[  145.159976] [] set_termios+0x2b0/0x338
[  145.160647] [] tty_mode_ioctl+0x358/0x590
[  145.161127] [] n_tty_ioctl_helper+0x54/0x168
[  145.161521] [] n_tty_ioctl+0xd4/0x1a0
[  145.161883] [] tty_ioctl+0x150/0xac0
[  145.162255] [] do_vfs_ioctl+0xc4/0x768
[  145.162620] [] SyS_ioctl+0x8c/0xa8

These are ugly but harmless. (The underlying cause is that QEMU doesn’t implement the undocumented ‘cprman’ clock control hardware, and so Linux thinks that the UART is running at a zero baud rate and complains.)

We’d like to announce the availability of the QEMU 2.12.0 release. This release contains 2700+ commits from 204 authors.

You can grab the tarball from our download page. The full list of changes are available in the Wiki.

Highlights include:

• Spectre/Meltdown mitigation support for x86/pseries/s390 guests. For more details see: https://www.qemu.org/2018/02/14/qemu-2-11-1-and-spectre-update/
• Numerous block support improvements, including support for directly interacting with userspace NVMe driver, and general improvements to NBD server/client including more efficient reads of sparse files
• Networking support for VMWare paravirtualized RDMA device (RDMA HCA and Soft-RoCE supported), CAN bus support via Linux SocketCAN and SJA1000-based PCI interfaces, and general improvements for dual-stack IPv4/IPv6 environments
• GUI security/bug fixes, dmabufs support for GTK/Spice.
• Better IPMI support for Platform Events and SEL logging in internal BMC emulation
• SMBIOS support for “OEM Strings”, which can be used for automating guest image activation without relying on network-based querying
• Disk cache information via virtio-balloon
• ARM: AArch64 new instructions for FCMA/RDM and SIMD/FP16/crypto/complex number extensions
• ARM: initial support for Raspberry Pi 3 machine type
• ARM: Corex-M33/Armv8-M emulation via new mps2-an505 board and many other improvements for M profile emulation
• HPPA: support for full machine emulation (hppa-softmmu)
• PowerPC: PPC4xx emulation improvements, including I2C bus support
• PowerPC: new Sam460ex machine type
• PowerPC: significant TCG performance improvements
• PowerPC: pseries: support for Spectre/Meltdown mitigations
• RISC-V: new RISC-V target via “spike_v1.9.1”, “spike_v1.10”, and “virt” machine types
• s390: non-virtual devices no longer require dedicated channel subsystem and guest support for multiple CSSs
• s390: general PCI improvements, MSI-X support for virtio-pci devices
• s390: improved TCG emulation support
• s390: KVM support for systems larger than 7.999TB
• SPARC: sun4u power device emulation
• SPARC: improved trace-event support and emulation/debug fixes
• Tricore: new instruction variants for JEQ/JNE and 64-bit MOV
• x86: Intel IOMMU support for 48-bit addresses
• Xtensa: backend now uses libisa for instruction decoding/disassebly
• Xtensa: multi-threaded TCG support and noMMU configuration variants
• and lots more…

Thank you to everyone involved!

My name is Yonathan Bettan and I work at RedHat in the virtualization KVM team.

This blog has 2 main purposes, the first is to give you an idea of what VirtIO is and why should we use it when the second is to serve as a step-by-step guide describing how to write a VirtIO device from zero with some code examples.

In addition, I will write a SIMPLE VirtIO example and documented device that you will be able to find in the Qemu project. Clone https://github.com/qemu/qemu.git repo for the full project.

Motivation

Let us start with a NIC (Network Interface Controller) as an example to understand better virtualization. A NIC is responsible for transmitting and receiving packets through the network. The received packets are written into memory and the packets to be sent are copied from the memory to the NIC for transmission with CPU intervention or without (DMA). When the NIC finishes a specific task it sends an interrupt to the OS.

If we want a physical machine to have a network connection we will have to buy a NIC and with the same logic if we want a virtual machine (VM) to have a network connection we will need to supply a virtual NIC.

One possible solution is to make the hypervisor fully emulate the NIC according to its spec – Virtual device.

When a package is sent on the guest OS it is sent to virtual NIC (vNIC). For each byte of data we will get:

A virtual interrupt will be generated ==> a VMexit will occur ==> the hypervisor will send the data to the physical NIC (pNIC) ==> the pNIC will interrupt the host OS when it finishes the transaction ==> the hypervisor will finally interrupt the guest OS to notify the transaction is finish.

We can see here a function of a NIC driver which its purpose is to read data from the device into a buffer.

NOTE: even if we use MMIO instead of PIO we still have a limitation on the MMIO write size and each MMIO write generates a VMexit so we still may have multiply VMexits.

The main benefits, in this case, is that the OS stays unchanged because the virtual device acts like a physical device so the already-written NIC driver does the job correctly on the emulated device. From the other hand it works slowly since each access to the vNIC generate a VMexit for each byte (as pNIC would have done) but in the reality this is not a real device (only code variables) so we don’t need to VMexit on each byte, instead we can just write the whole buffer and generate a single VMexit.

Another possible solution is to give the guest direct access to the pNIC – Device assignment.

When a package is sent on the guest OS it is sent to the vNIC. For each byte of data we will get:

The data is sent directly to the pNIC without the hypervisor intervention ==> the pNIC will interrupt the guest OS directly when it finishes the transaction.

Now we have max performance that the HW can supply but we need separate pNIC for each guest and another one to the host, this becomes expensive.

The tradeoff between Virtual devices and Device assignment is Paravirtual devices and its protocol VirtIO.

This case is quite similar to the Virtual device case except for 2 facts, the first is that the emulated device don’t pretend to act like a real device (no need to send virtual interrupts for each byte written but only a single virtual interrupt once the whole buffer is written) and the second is that we now have to write a new driver since the original driver no longer feets with the emulated HW.

We can now see the same function of a NIC  new driver.

Another reason to use VirtIO devices is that Linux support multiple hypervisors such as KVM, Xen, VMWare etc. Therefore we have drivers for each one of them. VirtIO provides Driver-unification, a uniform ABI for all those hypervisors. An ABI is an interface at the compiler level and describes how do parameters are passed to function (register\stack), how interrupts are propagated etc. VirtIO also provides device check and configuration.

Virtualization VS Paravirtualization

Sources

• https://www.ibm.com/developerworks/library/l-virtio/
• https://www.cs.cmu.edu/~412/lectures/Virtio_2015-10-14.pdf

Probably not worth a post, as it’s mentioned in the readme, but CRIO was recently updated in minikube v0.26.0 which now makes it work like a charm.

When updating to 0.26 make sure to update the minikube binary, but also the docker-machine-driver-kvm2 binary.

Like in the past it is possible to switch to CRIO using

$ minikube start --container-runtime=cri-o
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.
$

However, my favorit launch line is:

minikube start --container-runtime=cri-o --network-plugin=cni --bootstrapper=kubeadm --vm-driver=kvm2

Which will use CRIO as the container runtime, CNI for networking, kubeadm for bringing up kube inside a KVM VM.

"The IBM Z platforms provide the HiperSockets technology feature for high-speed communications. This paper documents how to set up and configure KVM virtual machines to use HiperSockets with IP routing capabilities of the TCP/IP stack.
It provides a Network Performance comparison between various network configurations and illustrates how HiperSockets can achieve greater performance for many workload types, across a wide range of data-flow patterns, compared with using an OSA 10GbE card."

Availability across multiple architectures
To further support customer choice in computing architecture, Red Hat Enterprise Linux 7.5 is simultaneously available across all supported architectures, including x86, IBM Power, IBM z Systems, and 64-bit Arm.

It took more than a year from the first working patches to the upstream merge. But now it's finally done. The linux kernel 4.16 (released on easter weekend) has the kernel-side code needed. The qemu code has been merged too (for gtk and spice user interfaces) and will be in the upcoming 2.12 release which is in code freeze right now. The 2.12 release candidates already have the code, so you can grab one if you don't want wait for the final release to play with this.

The vgpu code in the intel driver is off by default and must be enabled via module option. And, while being at it, also suggest to load the kvmgt module. So I've dropped a config file with these lines ...

options i915 enable_gvt=1
softdep i915 pre: kvmgt

... into /etc/modprobe.d/. For some reason dracut didn't pick the changes up even after regenerating the initrd. Because of that I've blacklisted the intel driver (rd.driver.blacklist=i915 on the kernel command line) so the driver gets loaded later, after mounting the root filesystem, and modprobe actually sets the parameter.

With that in place you should have a /sys/class/mdev_bus directory with the intel gpu in there. You can create vgpu devices now. Check the mediated device documentation for details.

One final thing to take care of: Currently using gvt mlocks all guest memory. For that work the mlock limit (ulimit -l) must be high enough, otherwise the vgpu will not work correctly and you'll see a scrambled display. Limit can be configured in /etc/security/limits.conf.

Now lets use our new vgpu with qemu:

qemu-system-x86_64 \
     -enable-kvm \
     -m 1G \
     -nodefaults \
     -M graphics=off \
     -serial stdio \
     -display gtk,gl=on \
     -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/UUID,display=on \
     -cdrom /vmdisk/iso/Fedora-Workstation-Live-x86_64-27-1.6.iso

Details on the non-obvious qemu switches:

-nodefaults

Do not create default devices (such as vga and nic).

-M graphics=off

Hint for the firmware that the guest runs without a graphical display. This enables serial console support in seabios. We use this here because the vgpu has no firmware support (i.e. no vgabios), therefore nothing is visible on the display until the i915 kernel module loads.

-display gtk,gl=on

Use gtk display, enable opengl.

-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/UUID,display=on

Add the vgpu to the guest, enable the display. Of course you have to replace UUID with your device.

libvirt support is still being worked on. Most bits are there, but some little details are missing. For example there is no way (yet) to tell libvirt the guest doesn't need an emulated vga device, so you'll end up with two spice windows, one for the emulated vga and one for the vgpu. Other than that things are working pretty straight forward. You need spice with opengl support enabled:

<graphics type='spice'>
  <listen type='none'/>
  <gl enable='yes'/>
</graphics>

And the vgpu must be added of course:

<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci'>
  <source>
    <address uuid='UUID'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</hostdev>

Then you can start the domain. Use "virt-viewer --attach guest" to connect to the guest. Note that guest and virt-viewer must run on the same machine, sending the vgpu display to a remote machine does not yet work.

I/O Devices

• Instruction Execution Protection
• SIMD Extensions
• STHYI Support
• Transparent activation of new hardware features: This will enable a set of performance-oriented hardware facilities of z14 that do not require hypervisor changes.
  This includes the following facilities:
• miscellaneous-instruction-extensions facility 2
• entropy-encoding compression facility
• order-preserving-compression facility

I’m pleased to announce a new release of GTK-VNC, version 0.7.2. The release focus is on bug fixing, and addresses an important regression in TLS handling from the previous release.

• Deprecated the manual python2 binding in favour of GObject introspection. It will be deleted in the next release.
• Emit led state notification on connect
• Fix incorrect copyright notices
• Simplify shifted-tab key handling
• Don’t short circuit TLS credential request
• Improve check for keymap under XWayland
• Update doap description of project
• Modernize RPM specfile

Thanks to all those who reported bugs and provides patches that went into this new release.

• Instruction Execution Protection
  This feature provides KVM hypervisor support for the Instruction Execution Protection (IEP) facility in the z14. The IEP prevents code execution from memory regions marked as non-executable, improving the security model.
  Other than activating/deactivating this feature in the applicable the CPU models in QEMU (which holds true for most hardware-related features on IBM Z in general), there are no switches associated with this feature.
  Requires Linux kernel 4.11 in the KVM host and guests, as well as QEMU v2.10 (host only).
  In the z14 CPU model, the respective feature is:
    iep       Instruction-execution-protection facility
• SIMD Extensions
  Following up to the SIMD instructions as introduced with the previous z13 model, the new z14 provides further vector instructions, which can again be used in KVM guests.
  These new vector instructions can be used to improve decimal calculations as well as for implementing high performance variants of certain cryptographic operations.
  Requires Linux kernel 4.11 as well as QEMU v2.10 in the KVM host, and binaries or a respective Java Runtime Environment in guests using the new vector instructions.
  In the z14 CPU model, the respective feature is:
    vxpd      Vector packed decimal facility
    vxeh      Vector enhancements facility
• Keyless Guest Support
  This feature supports the so-called Keyless Subset (KSS) facility, a new feature of the z14 hardware. With the KSS facility enabled, a host is not required to perform the (costly) storage key initialization and management for KVM guests, unless a guest issues a storage key instruction.
  Requires Linux kernel 4.12 in the KVM host. As for the guests, note that starting with SLES12SP1, RHEL7.2 and Ubuntu 16.04, Linux on IBM Z does not issue any storage key operations anymore.
  This feature does not have a separate entry in the z14 CPU model.
  
• CPUMF Basic Sample Configuration Level Indication
  Basic mode samples as defined in "The Load-Program-Parameter and the CPU-Measurement Facilities" (SA23-2260) do not provide an indication whether the sample was taken in a KVM host or guest. Beginning with z14, the hardware provides an indication of the configuration level (level of SIE, e.g. LPAR or KVM). This item exploits this information to make the perf guest/host decision reliable.
  Requires Linux kernel 4.12 in the KVM host.
  There is no separate entry in the z14 CPU model, since this feature applies to the host only.
• Semaphore assist
  Improves performance of semaphore locks.
  Requires Linux kernel 4.7 and QEMU v2.10 in the KVM host. Exploitation in Linux kernels in guests is still in progress here, scheduled for 4.14.
  In the z14 CPU model, the respective feature is:
    sema      Semaphore-assist facility
• Guarded storage
  This feature is specifically aimed at Java Virtual Machines running in KVM guests to run with fewer and shorter pauses for garbage collection.
  Requires Linux kernel 4.12 and QEMU 2.10 in the KVM host, and a Java Runtime Environment with respective support in the guests.
  In the z14 CPU model, the respective feature is:
    gs        Guarded-storage facility
• MSA Updates
  z14 introduces 3 new Message Security Assists (MSA) for the following functionalities:
      MSA6: SHA3 hashing
      MSA7: A True Random  Number Generator (TRNG)
      MSA8: The CIPHER MESSAGE WITH AUTHENTICATION instruction,
                  which provides support for the Galois-counter-mode (GCM)
  MSA6 and MSA 7 require Linux kernel 4.7, while MSA8 requires Linux kernel 4.12. All require QEMU v2.10 in the KVM host. These features can be exploited in KVM guests' kernels and userspace applications independently (i.e. a KVM guest's userspace applications can take advantage of these features irrespective of the guest's kernel version).
  In the z14 CPU model, the respective features are:
    msa6      Message-security-assist-extension 6 facility
    msa7      Message-security-assist-extension 7 facility
    msa8      Message-security-assist-extension 8 facility
• Compression enhancements
  New instructions improve compression capabilities and performance.
  Requires Linux kernel 4.7 in the KVM host.
  In the z14 CPU model, the respective features are:
    opc       Order Preserving Compression facility
    eec       Entropy encoding compression facility
• Miscellaneous instructions
  Details on these instructions are to be published in the forthcoming z14 Principles of Operation (PoP).
  Requires Linux kernel 4.7 and QEMU 2.10 in the KVM host, and binaries that were compiled for the z14 instruction set using binutils v2.28 and gcc v7.1 in the guests.
  In the z14 CPU model, the respective feature is:
    minste2   Miscellaneous-instruction-extensions facility 2

You have this image, of a VM, which you want to run - alongside containers - why? - well, you need it. Some people would say it’s dope, but sometimes you really need it, because it has an app you want to integrate with pods.

Here is how you can do this with KubeVirt.

1 Deploy KubeVirt

Deploy KubeVirt on your cluster - or follow the demo guide to setup a fresh minikube cluster.

2 Download Ubuntu

While KubeVirt comes up (use kubectl get --all-namespaces pods), download Ubuntu Server

3 Install kubectl plugin

Make sure to have the latest or recent kubectl tool installed, and install the pvc plugin:

curl -L https://github.com/fabiand/kubectl-plugin-pvc/raw/master/install.sh | bash

4 Create disk

Upload the Ubuntu server image:

$ kubectl plugin pvc create ubuntu1704 1Gi $PWD/ubuntu-17.04-server-amd64.iso disk.img
Creating PVC
persistentvolumeclaim "ubuntu1704" created
Populating PVC
pod "ubuntu1704" created
total 701444
701444 -rw-rw-r--    1 1000     1000      685.0M Aug 25  2017 disk.img
Cleanup
pod "ubuntu1704" deleted

5 Create and launch VM

Create a VM:

$ kubectl apply -f -
apiVersion: kubevirt.io/v1alpha1
kind: VirtualMachinePreset
metadata:
  name: large
spec:
  selector:
    matchLabels:
      kubevirt.io/size: large
  domain:
    resources:
      requests:
        memory: 1Gi
---
apiVersion: kubevirt.io/v1alpha1
kind: OfflineVirtualMachine
metadata:
  name: ubuntu
spec:
  running: true
  selector:
    matchLabels:
      guest: ubuntu
  template:
    metadata:
      labels: 
        guest: ubuntu
        kubevirt.io/size: large
    spec:
      domain:
        devices:
          disks:
            - name: ubuntu
              volumeName: ubuntu
              disk:
                bus: virtio
      volumes:
        - name: ubuntu
          claimName: ubuntu1710

6 Connect to VM

$ ./virtctl-v0.3.0-linux-amd64 vnc --kubeconfig ~/.kube/config ubuntu

Final notes - This is booting the Ubuntu ISO image. But this flow should work for existing images, which might be much more useful.

KubeVirt becomes usable. And to make it easier to use it would be nice to be able to import existing VMs. After all migration is a strong point of KubeVirt.

virt-v2v is the tool of choice to convert some guest to run on the KVM hypervisor. What a great fit.

Thus recently I started a little POC to check if this would really work.

This post is just to wrap it up, as I just tagged v0.2.0 and finished a nice OVA import.

What the POC does:

• Take an URL pointing to an OVA
• Download and convert the OVA to a domxml and raw disk image
• Create a PVC and move the raw disk image to it
• Create an OfflineVirtualMachine from the domxml using xslt

This is pretty straight forward and currently living in a Job which can be found here: https://github.com/fabiand/v2v-job

It’s actually using an OpenShift Template, but only works on Kubernetes so far, because I didn’t finish the RBAC profiles. However, using the oc tool you can even run it on Kubernetes without Template support by using:

$ oc process --local -f manifests/template.yaml \
    -p SOURCE_TYPE=ova \
    -p SOURCE_NAME=http://192.168.42.1:8000/my.ova \
  | kubectl apply -f -
serviceaccount "kubevirt-privileged" created
job "v2v" created

The interested reader can take a peek at the whole process in this log.

And btw - This little but awesome patch on libguestfs by Pino - will help this job to auto-detect - well, guess - the the guest operating system and set the OfflineVirtualMachine annotations correctly, in order to then - at runtime - apply the right VirtualMachinePresets, in order to launch the guest with optimized defaults.

OpenStack community released ‘queens’ version this week. IMHO it is quite important moment for AArch64 community as well because it works out of the box for us.

Gone are things like setting hw_firmware_type=uefi for each image you upload to Glance — Nova assumes UEFI to be the default firmware on AArch64 (unless you set the variable to different value for some reason). This simplifies things as users does not have to worry about and we should have less support questions on new setups of Linaro Developer Cloud (which will be based on ‘Queens’ instead of ‘Newton’).

There is a working graphical console if your guest image uses properly configured kernel (4.14 from Debian/stretch-backports works fine, 4.4 from Ubuntu/xenial (used by CirrOS) does not have graphics enabled). Handy feature which we were asked already by some users.

Sad thing is state of live migration on AArch64. It simply does not work through the whole stack (Nova, libvirt, QEMU) because we have no idea what exactly cpu we are running on and how it is compatible with other cpu cores. In theory live migration between same type of processors (like XGene1 -> XGene1) should be possible but we do not have even that level of information available. More information can be found in bug 1430987 reported against libvirt.

Less sad part? We set cpu_model to ‘host-passthrough’ by default now (in Nova) so nevermind which deployment method is used it should work out of the box.

When it comes to building (Kolla) and deploying (Kolla Ansible) most of the changes were done during Pike cycle. During Queens’ one most of the changes were small tweaks here and there. I think that our biggest change was convincing everyone in Kolla(-ansible) to migrate from MariaDB 10.0.x (usually from external repositories) to 10.1.x taken from distribution (Debian) or from RDO.

What will Rocky bring? Better hotplug for PCI Express machines (AArch64/virt, x86/q35 models) is one thing. I hope that live migration stuff situation will improve as well.

First post for quite some time. A side effect of being busy to get streamline our KubeVirt user experience.

KubeVirt v0.3.0 was not released at the beginnig of the month.

That release was intended to be a little bigger, because it included a large architecture change (to the good). The change itself was amazingly friendly and went in without much problems - even if it took some time.

But, the work which was building upon this patch in the storage and network areas was delayed and didn’t make it in time. Thus we skipped the release in order to let storage and network catch up.

The important thing about these two areas is, that KubeVirt was able to connect a VM to a network, and was able to boot of a iSCSI target, but this was not really tightly integrated with Kubernetes.

Now, just this week two patches landed which actually do integrate these areas with Kubernetes.

Storage

The first is storage - mainly written by Artyom, and finalized by David - which allows a user to use a persistent volume as the backing storage for a VM disk:

metadata:
  name: myvm
apiVersion: kubevirt.io/v1alpha1
kind: VirtualMachine
spec:
  domain:
    devices:
      disks:
      - name: mypvcdisk
        volumeName: mypvc
        lun: {}
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: mypvc

This means that any storage solution supported by Kubernetes to provide PVs can be used to store virtual machine images. This is a big step forward in terms of compatibility.

This actually works by taking this claim, and attaching it to the VM’s pod definition, in order to let the kubelet then mount the respective volume to the VM’s pod. Once that is done, KubeVirt will take care to connect the disk image within that PV to the VM itself. This is only possible because the architecture change caused libvirt to run inside every VM pod, and thus allow the VM to consume the pod resources.

Side note, another project is in progress to actually let a user upload a virtual machine disk to the cluster in a convenient way: https://github.com/kubevirt/containerized-data-importer.

Network

The second change is about network which Vladik worked on for some time. This change also required the architectural changes, in order to allow the VM and libvirt to consume the pod’s network resource.

Just like with pods the user does not need to do anything to get basic network connectivity. KubeVirt will connect the VM to the NIC of the pod in order to give it the most compatible intergation. Thus now you are able to expose a TCP or UDP port of the VM to the outside world using regular Kubernetes Services.

A side note here is that despite this integration we are now looking to enhance this further to allow the usage of side cars like Istio.

Alpha Release

The three changes - and their delay - caused the delay of v0.3.0 - which will now be released in the beginnig of March. But we have done a few pre-releases in order to allow interested users to try this code right now:

KubeVirt v0.3.0-alpha.3 is the most recent alpha release and should work fairly well.

More

But these items were just a small fraction of what we have been doing.

If you look at the kubevirt org on GitHub you will notice many more repositories there, covering storage, cockpit, and deployment with ansible - and it will be another post to write about how all of this is fitting together.

Welcome aboard!

KubeVirt is really speeding up and we are still looking for support. So if you are interested in working on a bleeding edge project tightly coupled with Kubernetes, but also having it’s own notion, and an great team, then just reach out to me.

While working on a major re-factor of QEMU’s softfloat code I’ve been doing a lot of benchmarking. It can be quite tedious work as you need to be careful you’ve run the correct steps on the correct binaries and keeping notes is important. It is a task that cries out for scripting but that in itself can be a compromise as you end up stitching a pipeline of commands together in something like perl. You may script it all in a language designed for this sort of thing like R but then find your final upload step is a pain to implement.

One solution to this is to use a literate programming workbook like this. Literate programming is a style where you interleave your code with natural prose describing the steps you go through. This is different from simply having well commented code in a source tree. For one thing you do not have to leap around a large code base as everything you need is on the file you are reading, from top to bottom. There are many solutions out there including various python based examples. Of course being a happy Emacs user I use one of its stand-out features org-mode which comes with multi-language org-babel support. This allows me to document my benchmarking while scripting up the steps in a variety of “languages” depending on the my needs at the time. Let’s take a look at the first section:

1 Binaries To Test

Here we have several tables of binaries to test. We refer to the
current benchmarking set from the next stage, Run Benchmark.

For a final test we might compare the system QEMU with a reference
build as well as our current build.

Binary	title
/usr/bin/qemu-aarch64	system-2.5.log
~/lsrc/qemu/qemu-builddirs/arm-targets.build/aarch64-linux-user/qemu-aarch64	master.log
~/lsrc/qemu/qemu.git/aarch64-linux-user/qemu-aarch64	softfloat-v4.log

Well that is certainly fairly explanatory. These are named org-mode tables which can be referred to in other code snippets and passed in as variables. So the next job is to run the benchmark itself:

2 Run Benchmark

This runs the benchmark against each binary we have selected above.

    import subprocess
    import os

    runs=[]

    for qemu,logname in files:
    cmd="taskset -c 0 %s ./vector-benchmark -n %s | tee %s" % (qemu, tests, logname)
        subprocess.call(cmd, shell=True)
        runs.append(logname)

        return runs
        

So why use python as the test runner? Well truth is whenever I end up munging arrays in shell script I forget the syntax and end up jumping through all sorts of hoops. Easier just to have some simple python. I use python again later to read the data back into an org-table so I can pass it to the next step, graphing:

set title "Vector Benchmark Results (lower is better)"
set style data histograms
set style fill solid 1.0 border lt -1

set xtics rotate by 90 right
set yrange [:]
set xlabel noenhanced
set ylabel "nsecs/Kop" noenhanced
set xtics noenhanced
set ytics noenhanced
set boxwidth 1
set xtics format ""
set xtics scale 0
set grid ytics
set term pngcairo size 1200,500

plot for [i=2:5] data using i:xtic(1) title columnhead

This is a GNU Plot script which takes the data and plots an image from it. org-mode takes care of the details of marshalling the table data into GNU Plot so all this script is really concerned with is setting styles and titles. The language is capable of some fairly advanced stuff but I could always pre-process the data with something else if I needed to.

Finally I need to upload my graph to an image hosting service to share with my colleges. This can be done with a elaborate curl command but I have another trick at my disposal thanks to the excellent restclient-mode. This mode is actually designed for interactive debugging of REST APIs but it is also easily to use from an org-mode source block. So the whole thing looks like a HTTP session:

:client_id = feedbeef

# Upload images to imgur
POST https://api.imgur.com/3/image
Authorization: Client-ID :client_id
Content-type: image/png

< benchmark.png

Finally because the above dumps all the headers when run (which is very handy for debugging) I actually only want the URL in most cases. I can do this simply enough in elisp:

#+name: post-to-imgur
#+begin_src emacs-lisp :var json-string=upload-to-imgur()
  (when (string-match
         (rx "link" (one-or-more (any "\":" whitespace))
             (group (one-or-more (not (any "\"")))))
         json-string)
    (match-string 1 json-string))
#+end_src

The :var line calls the restclient-mode function automatically and passes it the result which it can then extract the final URL from.

And there you have it, my entire benchmarking workflow document in a single file which I can read through tweaking each step as I go. This isn’t the first time I’ve done this sort of thing. As I use org-mode extensively as a logbook to keep track of my upstream work I’ve slowly grown a series of scripts for common tasks. For example every patch series and pull request I post is done via org. I keep the whole thing in a git repository so each time I finish a sequence I can commit the results into the repository as a permanent record of what steps I ran.

If you want even more inspiration I suggest you look at John Kitchen’s scimax work. As a publishing scientist he makes extensive use of org-mode when writing his papers. He is able to include the main prose with the code to plot the graphs and tables in a single source document from which his camera ready documents are generated. Should he ever need to reproduce any work his exact steps are all there in the source document. Yet another example of why org-mode is awesome

You run VM instance. Nevermind is it part of OpenStack setup or just local one started using Boxes, virt-manager, virsh or other that kind of fronted to libvirt daemon. And then you want to add some virtual hardware to it. And another card and one more controller…

Easy to imagine scenario, right? What can go wrong, you say? “No more available PCI slots.” message can happen. On second/third card/controller… But how? Why?

Like I wrote in one of my previous posts most of VM instances are 90s pc hardware virtual boxes. With simple PCI bus which accepts several cards to be added/removed at any moment.

But not on AArch64 architecture. Nor on x86-64 with Q35 machine type. What is a difference? Both are PCI Express machines. And by default they have far too small amount of pcie slots (called pcie-root-port in qemu/libvirt language). More about PCI Express support can be found in PCI topology and hotplug page of libvirt documentation.

So I wrote a patch to Nova to make sure that enough slots will be available. And then started testing. Tried few different approaches, discussed with upstream libvirt developers about ways of solving the problem and finally we selected the one and only proper way of doing it. Then discussed failures with UEFI developers. And went for help to Qemu authors. And explained what I want to achieve and why to everyone in each of those four projects. At some point I had seen pcie-root-port things everywhere…

Turned out that the method of fixing it is kind of simple: we have to create whole pcie structure with root port and slots. This tells libvirt to not try any automatic adding of slots (which may be tricky if not configured properly as you may end with too small amount of slots for basic addons).

Then I went with idea of using insane values. VM with one hundred PCIe slots? Sure. So I made one, booted it and then something weird happen: landed in UEFI shell instead of getting system booted. Why? How? Where is my storage? Network? Etc?

Turns out that Qemu has limits. And libvirt has limits… All ports/slots went into one bus and memory for MMCONFIG and/or I/O space was gone. There are two interesting threads about it on qemu-devel mailing list.

So I added magic number into my patch: 28 — this amount of pcie-root-port entries in my aarch64 VM instance was giving me bootable system. Have to check it on x86-64/q35 setup still but it should be more or less the same. I expect this patch to land in ‘Rocky’ (the next OpenStack release) and probably will have to find a way to get it into ‘Queens’ as well because this is what we are planning to use for next edition of Linaro Developer Cloud.

Conclusion? Hotplug may be complicated. But issues with it can be solved.

Oddities of PCI on s390x architecture

Oddity #1: No MMIO, but instructions

Oddity #2: No user space I/O

Oddity #3: No topology, but FID and UID

Implications for the QEMU implementation of PCI on s390x

Instruction handlers

Modelling and (lack of) topology

How can I try this out?

s390x-softmmu/qemu-system-s390x -M s390-ccw-virtio,accel=tcg -cpu qemu,zpci=on (...) -device zpci,uid=12,fid=2,target=vpci02,id=zpci2 -device virtio-net-pci,id="vpci02",addr=0x2

KVM virtualization is now supported on IBM z Systems. However, this feature is only available in the newly introduced user space based on kernel version 4.14, provided by the kernel-alt packages.

A previous post detailed how QEMU/KVM might be affected by Spectre/Meltdown attacks, and what the plan was to mitigate them in QEMU 2.11.1 (and eventually QEMU 2.12).

QEMU 2.11.1 is now available, and contains the aforementioned mitigation functionality for x86 guests, along with additional mitigation functionality for pseries and s390x guests (ARM guests do not currently require additional QEMU patches). However, enabling this functionality requires additional configuration beyond just updating QEMU, which we want to address with this post.

Please note that QEMU/KVM has at least the same requirements as other unprivileged processes running on the host with regard to Spectre/Meltdown mitigation. What is being addressed here is enabling a guest operating system to enable the same (or similar) mitigations to protect itself from unprivileged guest processes running under the guest operating system. Thus, the patches/requirements listed here are specific to that goal and should not be regarded as the full set of requirements to enable mitigations on the host side (though in some cases there is some overlap between the two with regard to required patches/etc).

Also please note that this is a best-effort from the QEMU/KVM community, and these mitigations rely on a mix of additional kernel/firmware/microcode updates that are in some cases not yet available publicly, or may not yet be implemented in some distros, so users are highly encouraged to consult with their respective vendors/distros to confirm whether all the required components are in place. We do our best to highlight the requirements here, but this may not be an exhaustive list.

Enabling mitigation features for x86 KVM guests

Note: these mitigations are known to cause some performance degradation for certain workloads (whether used on host or guest), and for some Intel architectures alternative solutions like retpoline-based kernels may be available which may provide similar levels of mitigation with reduced performance impact. Please check with your distro/vendor to see what options are available to you.

For x86 guests there are 2 additional CPU flags associated with Spectre/Meltdown mitigation: spec-ctrl, and ibpb:

• spec-ctrl: exposes Indirect Branch Restricted Speculation (IBRS)
• ibpb: exposes Indirect Branch Prediction Barriers

These flags expose additional functionality made available through new microcode updates for certain Intel/AMD processors that can be used to mitigate various attack vectors related to Spectre. (Meltdown mitigation via KPTI does not require additional CPU functionality or microcode, and does not require an updated QEMU, only the related guest/host kernel patches).

Utilizing this functionality requires guest/host kernel updates, as well as microcode updates for Intel and recent AMD processors. The status of these kernel patches upstream is still in flux, but most supported distros have some form of the patches that is sufficient to make use of the functionality. The current status/availability of microcode updates depends on your CPU architecture/model. Please check with your vendor/distro to confirm these prerequisites are available/installed.

Generally, for Intel CPUs with updated microcode, spec-ctrl will enable both IBRS and IBPB functionality. For AMD EPYC processors, ibpb can be used to enable IBPB specifically, and is thought to be sufficient by itself for that particular architecture.

These flags can be set in a similar manner as other CPU flags, i.e.:

qemu-system-x86_64 -cpu qemu64,+spec-ctrl,... ...
qemu-system-x86_64 -cpu IvyBridge,+spec-ctrl,... ...
qemu-system-x86_64 -cpu EPYC,+ibpb,... ...
etc...

Additionally, for management stacks that lack support for setting specific CPU flags, a set of new CPU types have been added which enable the appropriate CPU flags automatically:

qemu-system-x86_64 -cpu Nehalem-IBRS ...
qemu-system-x86_64 -cpu Westmere-IBRS ...
qemu-system-x86_64 -cpu SandyBridge-IBRS ...
qemu-system-x86_64 -cpu IvyBridge-IBRS ...
qemu-system-x86_64 -cpu Haswell-IBRS ...
qemu-system-x86_64 -cpu Haswell-noTSX-IBRS ...
qemu-system-x86_64 -cpu Broadwell-IBRS ...
qemu-system-x86_64 -cpu Broadwell-noTSX-IBRS ...
qemu-system-x86_64 -cpu Skylake-Client-IBRS ...
qemu-system-x86_64 -cpu Skylake-Server-IBRS ...
qemu-system-x86_64 -cpu EPYC-IBPB ...

With these settings enabled, guests may still require additional configuration to enable IBRS/IBPB, which may vary somewhat from one distro to another. For RHEL guests, the following resource may be useful:

• https://access.redhat.com/articles/3311301

With regard to migration compatibility, spec-ctrl/ibrs (or the corresponding CPU type) should be set the same on both source/target to maintain compatibility. Thus, guests will need to be rebooted to make use of the new features.

Enabling mitigation features for pseries KVM guests

For pseries guests there are 3 tri-state -machine options/capabilities relating to Spectre/Meltdown mitigation: cap-cfpc, cap-sbbc, cap-ibs, which each correspond to a set of host machine capabilities advertised by the KVM kernel module in new/patched host kernels that can be used to mitigate various aspects of Spectre/Meltdown:

• cap-cfpc: Cache Flush on Privilege Change
• cap-sbbc: Speculation Barrier Bounds Checking
• cap-ibs: Indirect Branch Serialisation

Each option can be set to one of “broken”, “workaround”, or “fixed”, which correspond, respectively, to instructing the guest whether the host is vulnerable, has OS-level workarounds available, or has hardware/firmware that does not require OS-level workarounds. Based on these options, QEMU will perform checks to validate whether the specified settings are available on the current host and pass these settings on to the guest kernel. At a minimum, any setting other than “broken” will require a host kernel that has some form of the following patches:

commit 3214d01f139b7544e870fc0b7fcce8da13c1cb51
KVM: PPC: Book3S: Provide information about hardware/firmware CVE workarounds

commit 191eccb1580939fb0d47deb405b82a85b0379070
powerpc/pseries: Add H_GET_CPU_CHARACTERISTICS flags & wrapper

and whether a host will support “workaround” and “fixed” settings for each option will depend on the hardware/firmware level of the host system.

In turn, to make use of “workaround” or “fixed” settings for each option, the guest kernel will require at least the following set of patches:

• https://lists.ozlabs.org/pipermail/linuxppc-dev/2018-January/167455.html

These are available upstream and have been backported to a number of stable kernels. Please check with your vendor/distro to confirm the required hardware/firmware and guest kernel patches are available/installed.

All three options, cap-cfpc, cap-sbbc, and cap-ibs default to “broken” to maintain compatibility with previous versions of QEMU and unpatched host kernels. To enable them you must start QEMU with the desired mitigation strategy specified explicitly. For example:

qemu-system-ppc64 ... \
  -machine pseries-2.11,cap-cfpc=workaround,cap-sbbc=workaround,cap-ibs=fixed

With regard to migration compatibility, setting any of these features to a value other than “broken” will require an identical setting for that option on the source/destination guest. To enable these settings your guests will need to be rebooted at some point.

Enabling mitigation features for s390x KVM guests

For s390x guests there are 2 CPU feature bits relating to Spectre/Meltdown:

• bpb: Branch prediction blocking
• ppa15: PPA15 is installed

bpb requires a host kernel patched with:

commit 35b3fde6203b932b2b1a5b53b3d8808abc9c4f60
KVM: s390: wire up bpb feature

and both bpb and ppa15 require a firmware with the appropriate support level as well as guest kernel patches to enable the functionality within guests. Please check with your distro/vendor to confirm.

Both bpb and ppa15 are enabled by default when using “-cpu host” and when the host kernels supports these facilities. For other CPU models, the flags have to be set manually. For example:

qemu-system-s390x -M s390-ccw-virtio-2.11 ... \
  -cpu zEC12,bpb=on,ppa15=on

With regard to migration, enabling bpb or ppa15 feature flags requires that the source/target also has those flags enabled. Since this is enabled by default for ‘-cpu host’ (when available on the host), you must ensure that bpb=off,ppa15=off is used if you wish to maintain migration compatibility with existing guests when using ‘-cpu host’, or take steps to reboot guests with bpb/ppa15 enabled prior to migration.

After a looong time things are finally landing upstream. Both vfio API update for vgpu local display support and support for that in the intel drm driver have been merged during the 4.16 merge window. Should not take too long to get the qemu changes merged upstream now, if all goes well the next release (2.12) will have it.

Tina Zhang wrote a nice blog article with both technical background and instructions for those who want play with it.