Fedora 28 was released last week, so here are the fresh Fedora 29 images for qemu.

As usual the images don't have a root password. You have to set one using virt-customize -a --root-password "password:<secret>", otherwise you can't login after boot.

Some images use grub2 as bootloader, some use systemd-boot. The filename indicates which uses which. The x86_64 and i686 images can be booted with both uefi and bios firmware. The arm images come as grub2 variant only. systemd-boot doesn't support 32bit arm and crashes on 64bit arm.

The images can also be booted as container, using systemd-nspawn --boot --image <file>, but you have to convert them to raw first as systemd-nspawn can't handle qcow2.

The 32bit arm image (armhfp) isn't there because doesn't boot for me. Seems the fedora grub2-efi.armhfp package has trouble booting the kernel in qemu (with edk2 firmware). To be investigated if I find some time. Note: The fedora 28 image uses a custom grub2-efi.armhfp package as fedora didn't ship grub2-efi.armhfp in version 28.

The images have been created with imagefish.

• Linux kernel 4.18,
• QEMU v2.12, and
• libvirt v4.6.

I gave a talk about security in QEMU at KVM Forum 2018. It covers the architecture of QEMU and focusses on the attack surfaces that are exposed to guests. I hope it will be useful to anyone auditing or writing device emulation code. It also describes the key design principles for isolating the QEMU process and limiting the damage that can be done if a guest escapes.

The video of the talk is now available:

The slides are available here (PDF).

lets start with some history ...

The original VGA

It was introduced by IBM in 1987. It had a bunch of new features, and also included old ones which where already present in the predecessor devices CGA and EGA, including:

• text modes (80x25, also 80x50 using a smaller font)
• 16 color mode (640x480, 4 bit per color, one plane per bit)
• 256 color mode (320x240, 8 bit per color)
• various tweaks you can do, like enabling double scan or split screen.

The VGA has 256k of video memory and it is accessed using a memory window at 0xa0000. It is not possible to access all video memory at the same time, you have to set bank registers to map the piece of memory you want access into the window.

All vga devices emulated by qemu support this.

Super VGA

In the early 90ies various enhanced VGA cards, typically named "Super VGA" (abbreviated SVGA) became available from various vendors. The cirrus vga emulated by qemu is a typical SVGA card which was quite popular back then. They add various new features:

• more video memory, which in turn allows for:
• higher resolutions for 256 color modes.
• more colors (64k, 16 bit per pixel).
• even more colors (16M, 24 or 32 bit per pixel).
• linear framebuffer, so you can access all video memory at the same time, without having to bank-switch the video memory into the rather small window at 0xa0000.
• 2D acceleration (cirrus blitter for example).

SVGA in qemu

All SVGA devices in qemu (except cirrus) have support for the bochs display interface. That interface was implemented by the bochs emulator first (this is where the name comes from). It was implemented in qemu too. For the qemu standard vga it is the primary interface. qxl-vga, virtio-vga and vmsvga support the bochs dispi interface when they are in vga compatibility mode, which is typically the case at boot, before the guest os loads the native display driver.

The bochs display interface is a paravirtual interface, with just the bare essentials to set video modes on a virtual display device. There are no registers for clock rate and other timing stuff for example.

Traditionally the bochs display interface uses I/O ports 0x1ce (bochs register index) and 0x1cf (bochs register data), As both registers are 16bit the data registers is unaligned, which does not work on non-x86 archs, so 0x1d0 is supported as data port too.

Graphics usage by modern guests

Lets have a look at what modern guests are doing in the graphics field:

• Everything but 32 bit true color modes is pretty much unused. The only exception is 16 bit true color modes which are still used sometimes in resource-constrained environments (raspberry pi for example).
• 2D acceleration is dead. It's either software rendering into a dumb framebuffer, or using the 3D engine for 2D rendering.
• text mode is used only with BIOS firmware, and even then only at boot (bootloader, vgacon until the kms driver loads). UEFI goes straight to graphics mode.
• Banked video memory access is dead. Text mode still uses the 0xa0000 window, but the text buffer is small enough that there is no bank switching needed.

So, we have a lot of rather complex code to emulate features not used at all by modern guests. There have been security bugs in the past in that complex but largely unused code ...

So, can we simplify things?

Turns out: yes, we can. First step already happened in qemu 1.3. The qemu stdvga got a MMIO bar. The MMIO bar can be used as alternative way to access the vga registers and also the bochs dispi interface registers.

OVMF (UEFI implementation for qemu) uses the MMIO bar. The bochs-drm.ko linux kms driver uses the MMIO bar too. In fact, both use the bochs display interface registers only, except for setting the unblank bit so the screen will not stay black.

So, the guest code already ignores the vga emulation. Cool. We can build on that.

Introducing -device bochs-display

New display device. Merged in qemu 3.0. Featues:

• No VGA compatibility. PCI class is display/other instead of display/vga.
• It has a stdvga-style MMIO bar. The vga registers are not available of course. Otherwise the register interface is identical to the stdvga though.
• Implemented from scratch, no code sharing with vga. Code size is an order of magnitude smaller when compared to vga.
• No I/O ports needed. You can plug it into an PCIe slot.
• OVMF supports it.
• bochs-drm.ko supports it too.

So, all set for UEFI guests. You can switch from stdvga to bochs-display, and everything continues to work fine.

But what about BIOS and text mode?

Accessing the vga hardware directly for text mode is rare these days. Typically seabios and linux boot loaders call vgabios functions to render text on the display. So, we can hook in there and support text rendering without the hardware actually having text mode support. A very simliar approach is taken by sgabios, to redirect vga text output to the serial line.

Luckily we are not the first ones facing that problem. coreboot can initialize the graphics hardware and setup a framebuffer with the native display resolution. Having to switch back to text mode when running seabios as coreboot payload is not exactly nice. So, there is a vgabios variant for coreboot which renders text to a framebuffer.

So, take that, tweak the initialization code to program the bochs dispi interface instead of looking for a framebuffer setup by coreboot, and we are ready to go. Seabios boot messages show up on the bochs-display framebuffer. Yay!

This will work out-of-the-box in qemu 3.1. The vgabios is already present in qemu 3.0, but due to a bug it is not installed by default, it must be copyed over manually to get things going.

There are some drawbacks, which may or may not be a problem depending on your use case:

• linux vgacon does not work due to direct vga hardware access. So you have to use vesafb or just live with not having early boot messages. Once bochs-drm.ko loads fbcon will be functional.
• The vgabios uses a fixed 1024x768 resolution and does not support switching modes after initialization. Reason is that the initialization code runs in big real mode, so accessing the mmio bar is easy then. That is not the case for vgabios function calls though. Resolutions smaller than 1024x768 are allowed by vgabios and will simply use the upper left corner of the display.

That's it. Enjoy the new legacy-free display device.

KVM virtualization is now supported on IBM z Systems. However, this feature is only available in the newly introduced user space based on kernel version 4.14, provided by the kernel-alt packages.

virt-builder is a tool for rapidly creating customized Linux images. Recently I’ve added support for Windows although for rather obvious licensing reasons we cannot distribute the Windows templates which would be needed to provide Windows support for everyone. However you can build your own Windows templates as described here and then:

$ virt-builder -l | grep windows
windows-10.0-server      x86_64     Windows Server 2016 (x86_64)
windows-6.2-server       x86_64     Windows Server 2012 (x86_64)
windows-6.3-server       x86_64     Windows Server 2012 R2 (x86_64)
$ virt-builder windows-6.3-server
[   0.6] Downloading: http://xx/builder/windows-6.3-server.xz
[   5.1] Planning how to build this image
[   5.1] Uncompressing
[  60.1] Opening the new disk
[  77.6] Setting a random seed
virt-builder: warning: random seed could not be set for this type of guest
virt-builder: warning: passwords could not be set for this type of guest
[  77.6] Finishing off
                   Output file: windows-6.3-server.img
                   Output size: 10.0G
                 Output format: raw
            Total usable space: 9.7G
                    Free space: 3.5G (36%)

To build a Windows template repository you will need the latest libguestfs sources checked out from https://github.com/libguestfs/libguestfs and you will also need a suitable Windows Volume License, KMS or MSDN developer subscription. Also the final Windows templates are at least ten times larger than Linux templates, so virt-builder operations take correspondingly longer and use lots more disk space.

First download install ISOs for the Windows guests you want to use.

After cloning the latest libguestfs sources, go into the builder/templates subdirectory. Edit the top of the make-template.ml script to set the path which contains the Windows ISOs. You will also possibly need to edit the names of the ISOs later in the script.

Build a template, eg:

$ ../../run ./make-template.ml windows 2k12 x86_64

You’ll need to read the script to understand what the arguments do. The script will ask you for the product key, where you should enter the volume license key or your MSDN key.

Each time you run the script successfully you’ll end up with two files called something like:

windows-6.2-server.xz
windows-6.2-server.index-fragment

The version numbers are Windows internal version numbers.

After you’ve created templates for all the Windows guest types you need, copy them to any (private) web server, and concatenate all the index fragments into the final index file:

$ cat *.index-fragment > index

Finally create a virt-builder repo file pointing to this index file:

# cat /etc/virt-builder/repos.d/windows.conf
[windows]
uri=http://xx/builder/index

You can now create Windows guests in virt-builder. However note they are not sysprepped. We can’t do this because it requires some Windows tooling. So while these guests are good for small tests and similar, they’re not suitable for creating actual Windows long-lived VMs. To do that you will need to add a sysprep.exe step somewhere in the template creation process.

Note that this is a blog post for (new) QEMU developers. If you are just interested in using QEMU, you can certainly skip this text. Otherwise, in case you have ever been in touch with the QEMU device model (“qdev”), you are likely aware of the basic qdev code boilerplate already:

static void mydev_realize(DeviceState *dev, Error **errp)
{
    /* callback function that is run during device "realization" */
}

static void mydev_instance_init(Object *obj)
{
    /* callback function that is run during device instance init */
}

static Property mydev_properties[] = {
    DEFINE_PROP_xxx("myprop", MyDevState, field, ...),
    /* ... */
    DEFINE_PROP_END_OF_LIST(),
};

static void mydev_class_init(ObjectClass *oc, void *data)
{
    DeviceClass *dc = DEVICE_CLASS(oc);

    dc->realize = mydev_realize;
    dc->desc = "My cool device";
    dc->props = mydev_properties;
    /* ... and other device class setup code ... */
}

static const TypeInfo mydev_info = {
    .name          = TYPE_MYDEV,
    .parent        = TYPE_SYS_BUS_DEVICE,  /* or something else */
    .instance_size = sizeof(mydev_state),
    .instance_init = mydev_instance_init,
    .class_init    = mydev_class_init,
};

static void mydev_register_types(void)
{
    type_register_static(&mydev_info);
}

type_init(mydev_register_types)

There are three different initialization functions involved here, the class_init, the instance_init and the realize function. While it is quite obvious to distinguish the class_init function from the two others (it is used for initializing the class data, not the data that is used for an instance … this is similar to the object model with classes and instances in C++), I initially always wondered about the difference between the instance_init() and the realize() functions. Having fixed quite a lot of related bugs in the past months in the QEMU code base, I now know that a lot of other people are also not properly aware of the difference here, so I think it is now time to write down some information that I’m now aware of, to make sure that I don’t forget about this again, and maybe help others to avoid related bugs in the future ;-)

First it is of course always a good idea to have a look at the documentation. While the documentation of TypeInfo (where instance_init() is defined) is not very helpful to understand the differences, the documentation of DeviceClass (where realize() is defined) has some more useful information: You can learn here that the object instantiation is done first, before the device is realized, i.e. the instance_init() function is called first, and the realize() function is called afterwards. The former must not fail, while the latter can return an error to its caller via a pointer to an “Error” object pointer.

So the basic idea here is that device objects are first instantiated, then these objects can be inspected for their interfaces and their creators can set up their properties to configure their settings and wire them up with other devices, before the device finally becomes “active” by being realized. It is important here to notice that devices can be instantiated (and also finalized) without being realized! This happens for example if the device is introspected: If you enter for example device_add xyz,help at the HMP monitor, or if you send the device-list-properties QOM command to QEMU to retrieve the device’s properties, QEMU creates a temporary instance of the device to query the properties of the object, without realizing it. The object gets destroyed (“finalized”) immediately afterwards.

Knowing this, you can avoid a set of bugs which could be found with a couple of devices in the past:

• If you want your device to provide properties for other parts of the QEMU code or for the users, and you want to add those properties via one of the many object_property_add*() functions of QEMU (instead of using the “props” field of the DeviceClass), then you should do this in the instance_init() and not in the realize() function. Otherwise the properties won’t show up when the user runs --device xyz,help or the device-list-properties QOM command to get some information about your device.

• instance_init() functions must really never fail, i.e. also not call abort() or exit(). Otherwise QEMU can terminate unexpectedly when a user simply wanted to have a look at the list of device properties with device_add xyz,help or the device-list-properties QOM command. If your device cannot work in certain circumstances, check for the error condition in the realize() function instead and return with an appropriate error there.

• Never assume that your device is always instantiated only with the machine that it was designed for. It’s of course a good idea to set the “user_creatable = false” flag in the DeviceClass of your device if your device cannot be plugged in arbitrary machines. But device introspection can still happen at any time, with any machine. So if you wrote a device called “mydev-a” that only works with --machine A, the user still can start QEMU with the option --machine B instead and then run device_add mydev-a,help or the device-list-properties QOM command. The instance_init() function of your device will be called to create a temporary instance of your device, even though the base machine is B and not A here. So you especially should take care to not depend on the availability of certain buses or other devices in the instance_init() function, nor use things like serial_hd() or nd_table[] in your instance_init() function, since these might (and should) have been used by the machine init function already. If your device needs to be wired up, provide properties as interfaces to the outside and let the creator of your device (e.g. the machine init code) wire your device between the device instantiation and the realize phase instead.

• Make sure that your device leaves a clean state after a temporary instance is destroyed again, i.e. don’t assume that there will be only one instance of your device which is created at the beginning right after QEMU has been started and is destroyed at the very end before QEMU terminates. Thus do not assume that the things that you do in your instance_init() don’t need explicit clean-up since the device instance will only be destroyed when QEMU terminates. Device instances can be created and destroyed at any time, so when the device is finalized, you must not leave any dangling pointers or references to your device behind you, e.g. in the QOM tree. When you create other objects in your instance_init() function, make sure to set proper parents of these objects or use an instance_finalize() function, so that the created objects get cleaned up correctly again when your device is destroyed.

All in all, if you write code for a new QEMU device, it is likely a good idea to use the instance_init() function only for e.g. creating properties and other things that are required before device realization, and then do the main work in the realize() function instead.

• TOD-Clock Epoch Extension Support: Extends the TOD clock beyond the year 2042.
• Setting sysctl vm.allocate_pgste is now superfluous.
• Netboot: The network boot firmware sets the client architecture option (93) in the DHCP request to 0x1f ("s390 Basic"). This allows a DHCP server to deliver the correct boot image for IBM Z guests. This is useful in situations where a single DHCP server has to provide network boot images for multiple architectures, e.g. for the purpose of installing operating systems.
• Added support for virtio-input-ccw and virtio-gpu-ccw. These newly supported devices lay the foundation for applications that require graphical interfaces, which thereby become usable from remote via VNC or SPICE.
  Here is a sample XML snippet for a guest definition:
  
      <input type='keyboard' bus='virtio'/>
      <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
        <listen type='address' address='0.0.0.0'/>
      </graphics>
      <video>
        <model type='virtio' heads='1' primary='yes'/>
      </video>

• A new CPU model representing IBM z14 Model ZR1 was added:
  14ZR1 (long name: IBM z14 Model ZR1 GA1).
• Re-use your existing infrastructure for LPAR installs by utilizing the newly added support for .INS files in network boot.

A collection of tips on using usb with qemu.

Picking a host adapter

The short answer for this one is: Unless you are running an operating system museum just use -device qemu-xhci.

Any recent operating system should support xhci out-of-the box. The only OS without xhci support which is still in widespread use is Windows 7.

In case your qemu version doesn't support qemu-xhci you can use nec-usb-xhci instead.

The -usb command line switch adds usb controllers matching the emulated hardware platform. So for the 'pc' machine type (emulating a 20+ year old i440FX chipset) this is a uhci host adapter (supporting usb1). For the 'q35' machine type (emulating a almost 10 year old Q35 chipset) it is ehci (for usb2 devices) with uhci companions (for usb1 devices). This is what you can use when running old guests which lack xhci support.

When using xhci you should better not use -usb, because you would get two usb busses then. Which is a valid configuration, but requires naming the usb host adapter and specifying the usb bus when adding usb devices if you want avoid qemu picking a random usb bus:

-device qemu-xhci,id=xhci -device usb-tablet,bus=xhci.0

With a single usb bus you can just say -device usb-tablet and be done with it.

Not enough usb ports?

Qemu can emulate an usb hub (-device usb-hub). But the hub supports usb1 only, so you should avoid using it. Better solution is to just increase the number of root ports. xhci has four root ports by default, but it supports up to 15 ports. And in case this still isn't enough a second xhci adapter can be added to the virtual machine.

To create a host adapter with 8 ports use -device qemu-xhci,p2=8,p3=8. The libvirt configuration is:

<controller type='usb' model='qemu-xhci' ports='8'/>

In case you wonder why qemu-xhci needs both p2 and p3 parameters: p2 specifies the number of usb2 ports (which support usb1 too), and p3 specifies the number of usb3 ports. It is possible to assign different counts here. When using -device qemu-xhci,p2=8,p3=4 you'll get an xhci adapter where ports 1-4 support both usb2 and usb3 and ports 5-8 are usb2-only. Can be used to force a usb3-capable usb device into usb2 mode by plugging it into a usb2-only xhci port. There should rarely be a need to actually do that in practice though.

I’m pleased to announce a new release of GTK-VNC, version 0.9.0. This is a cleanup/modernization release. Note that the next release (1.0.0) will drop support for GTK-2

• Requires gnutls >= 3.1.18
• Requires libgcrypt >= 1.5.0
• Requires glib2 >= 2.42.0
• Use libgcrypt for DES routines
• Add missing cipher close calls in ARD auth
• Check for errors after reading mslogon params
• Support newer UltraVNC mslogon auth type code
• Avoid divide by zero in mslogin auth from bogus params
• Re-allow python2 accidentally blocked when removing python binding

Thanks to all those who reported bugs and provides patches that went into this new release.

We’d like to announce the availability of the QEMU 3.0.0 release. This release contains 2300+ commits from 169 authors.

A note from the maintainer: Why 3.0? Well, we felt that our version numbers were getting a bit unwieldy, and since this year is QEMU’s 15th birthday it seemed like a good excuse to roll over the major digit. Going forward we plan to increment the major version once a year, for the first release of the year. Don’t read too much into it: it doesn’t imply a drastic compatibility break. Rumours of our triskaidekaphobia have been greatly exaggerated ;-)

You can grab the tarball from our download page. The full list of changes are available in the Wiki.

Highlights include:

• Support for additional x86/AMD mitigations against Speculative Store Bypass (Spectre Variant 4, CVE-2018-3639)
• Improved support for nested KVM guests running on Hyper-V
• Block device support for active disk-mirroring, which avoids convergence issues which may arise when doing passive/background mirroring of busy devices
• Improved support for AHCI emulation, SCSI emulation, and persistent reservations / cluster management
• OpenGL ES support for SDL front-end, additional framebuffer device options for early boot display without using legacy VGA emulation
• Live migration support for TPM TIS devices, capping bandwidth usage during post-copy migration, and recovering from a failed post-copy migration
• Improved latency when using user-mode networking / SLIRP
• ARM: support for SMMUv3 IOMMU when using ‘virt’ machine type
• ARM: v8M extensions for VLLDM and VLSTM floating-point instructions, and improved support for AArch64 v8.2 FP16 extensions
• ARM: support for Scalable Vector Extensions in linux-user mode
• Microblaze: support for 64-bit address sizes and translation bug fixes
• PowerPC: PMU support for mac99 machine type and improvements for Uninorth PCI host bridge emulation for Mac machine types
• PowerPC: preliminary support for emulating POWER9 hash MMU mode when using powernv machine type
• RISC-V: improvement for privileged ISA emulation
• s390: support for z14 ZR1 CPU model
• s390: bpb/ppa15 Spectre mitigations enabled by default for z196 and later CPU models
• s390: support for configuring consoles via -serial options
• and lots more…

Thank you to everyone involved!

I’m pleased to announce a new release of GTK-VNC, version 0.8.0. This is a small maintenance release tidying up some loose ends

• Deleted the python2 binding in favour of GObject introspection
• Pull in latest keycodemapdb content
• Disable/fix -Wcast-function-type warnings

Thanks to all those who reported bugs and provides patches that went into this new release.

CPU models

I am happy to announce a new bugfix release of virt-viewer 7.0 (gpg), including experimental Windows installers for Win x86 MSI (gpg) and Win x64 MSI (gpg). The virsh and virt-viewer binaries in the Windows builds should now successfully connect to libvirtd, following fixes to libvirt’s mingw port.

Signatures are created with key DAF3 A6FD B26B 6291 2D0E 8E3F BE86 EBB4 1510 4FDF (4096R)

All historical releases are available from:

http://virt-manager.org/download/

Changes in this release include:

• Require spice-gtk >= 0.35
• Clarify error message when no running VM is found
• Improve check for libgovirt requirements
• Support “-” as a URI for input connection file
• Remove support for spice controller interface
• Misc man page fixes
• Lookup win32 translations relative to install dir
• Position connect window in center not top-left
• Misc fixes for ovirt foreign menu support

Thanks to everyone who contributed towards this release.

Virglrenderer provides OpenGL acceleration to a guest running on QEMU.

My current GSoC project is to add support for the Vulkan API.

Vulkan is drastically different to OpenGL. Thus, this addition is not straight-forward. My current idea is to add an alternative path for Vulkan. Currently, two different states are kept, one for OpenGL, and one for Vulkan. Commands will either go to the OpenGL or Vulkan front-end.

For now, only compute shaders are supported. The work is divided in two parts: a Vulkan ICD in MESA, and a new front-end for Virgl and vtest.

If you have any feedback, do not hesitate !

This experiment can be tested using this repository. If you have an Intel driver in use, you might be able to use the Dockerfile provided.

Each part is also available independently:

• MESA
• VirglRenderer
• Vulkan compute sample

With the various CPU hardware vulnerabilities reported this year, guest CPU configuration is now a security critical task. This blog post contains content I’ve written that is on its way to become part of the QEMU documentation.

QEMU / KVM virtualization supports two ways to configure CPU models

Host passthrough

This passes the host CPU model features, model, stepping, exactly to the guest. Note that KVM may filter out some host CPU model features if they cannot be supported with virtualization. Live migration is unsafe when this mode is used as libvirt / QEMU cannot guarantee a stable CPU is exposed to the guest across hosts. This is the recommended CPU to use, provided live migration is not required.

Named model

QEMU comes with a number of predefined named CPU models, that typically refer to specific generations of hardware released by Intel and AMD. These allow the guest VMs to have a degree of isolation from the host CPU, allowing greater flexibility in live migrating between hosts with differing hardware.

In both cases, it is possible to optionally add or remove individual CPU features, to alter what is presented to the guest by default.

Libvirt supports a third way to configure CPU models known as “Host model”. This uses the QEMU “Named model” feature, automatically picking a CPU model that is similar the host CPU, and then adding extra features to approximate the host model as closely as possible. This does not guarantee the CPU family, stepping, etc will precisely match the host CPU, as they would with “Host passthrough”, but gives much of the benefit of passthrough, while making live migration safe.

Recommendations for KVM CPU model configuration on x86 hosts

The information that follows provides recommendations for configuring CPU models on x86 hosts. The goals are to maximise performance, while protecting guest OS against various CPU hardware flaws, and optionally enabling live migration between hosts with hetergeneous CPU models.

Preferred CPU models for Intel x86 hosts

The following CPU models are preferred for use on Intel hosts. Administrators / applications are recommended to use the CPU model that matches the generation of the host CPUs in use. In a deployment with a mixture of host CPU models between machines, if live migration compatibility is required, use the newest CPU model that is compatible across all desired hosts.

Skylake-Server

Skylake-Server-IBRS

Intel Xeon Processor (Skylake, 2016)

Skylake-Client

Skylake-Client-IBRS

Intel Core Processor (Skylake, 2015)

Broadwell

Broadwell-IBRS

Broadwell-noTSX

Broadwell-noTSX-IBRS

Intel Core Processor (Broadwell, 2014)

Haswell

Haswell-IBRS

Haswell-noTSX

Haswell-noTSX-IBRS

Intel Core Processor (Haswell, 2013)

IvyBridge

IvyBridge-IBRS

Intel Xeon E3-12xx v2 (Ivy Bridge, 2012)

SandyBridge

SandyBridge-IBRS

Intel Xeon E312xx (Sandy Bridge, 2011)

Westmere

Westmere-IBRS

Westmere E56xx/L56xx/X56xx (Nehalem-C, 2010)

Nehalem

Nehalem-IBRS

Intel Core i7 9xx (Nehalem Class Core i7, 2008)

Penryn

Intel Core 2 Duo P9xxx (Penryn Class Core 2, 2007)

Conroe

Intel Celeron_4x0 (Conroe/Merom Class Core 2, 2006)

Important CPU features for Intel x86 hosts

The following are important CPU features that should be used on Intel x86 hosts, when available in the host CPU. Some of them require explicit configuration to enable, as they are not included by default in some, or all, of the named CPU models listed above. In general all of these features are included if using “Host passthrough” or “Host model”.

pcid

Recommended to mitigate the cost of the Meltdown (CVE-2017-5754) fix. Included by default in Haswell, Broadwell & Skylake Intel CPU models. Should be explicitly turned on for Westmere, SandyBridge, and IvyBridge Intel CPU models. Note that some desktop/mobile Westmere CPUs cannot support this feature.

spec-ctrl

Required to enable the Spectre (CVE-2017-5753 and CVE-2017-5715) fix, in cases where retpolines are not sufficient. Included by default in Intel CPU models with -IBRS suffix. Must be explicitly turned on for Intel CPU models without -IBRS suffix. Requires the host CPU microcode to support this feature before it can be used for guest CPUs.

ssbd

Required to enable the CVE-2018-3639 fix. Not included by default in any Intel CPU model. Must be explicitly turned on for all Intel CPU models. Requires the host CPU microcode to support this feature before it can be used for guest CPUs.

pdpe1gb

Recommended to allow guest OS to use 1GB size pages.Not included by default in any Intel CPU model. Should be explicitly turned on for all Intel CPU models. Note that not all CPU hardware will support this feature.

Preferred CPU models for AMD x86 hosts

The following CPU models are preferred for use on Intel hosts. Administrators / applications are recommended to use the CPU model that matches the generation of the host CPUs in use. In a deployment with a mixture of host CPU models between machines, if live migration compatibility is required, use the newest CPU model that is compatible across all desired hosts.

EPYC

EPYC-IBPB

AMD EPYC Processor (2017)

Opteron_G5

AMD Opteron 63xx class CPU (2012)

Opteron_G4

AMD Opteron 62xx class CPU (2011)

Opteron_G3

AMD Opteron 23xx (Gen 3 Class Opteron, 2009)

Opteron_G2

AMD Opteron 22xx (Gen 2 Class Opteron, 2006)

Opteron_G1

AMD Opteron 240 (Gen 1 Class Opteron, 2004)

Important CPU features for AMD x86 hosts

The following are important CPU features that should be used on AMD x86 hosts, when available in the host CPU. Some of them require explicit configuration to enable, as they are not included by default in some, or all, of the named CPU models listed above. In general all of these features are included if using “Host passthrough” or “Host model”.

ibpb

Required to enable the Spectre (CVE-2017-5753 and CVE-2017-5715) fix, in cases where retpolines are not sufficient. Included by default in AMD CPU models with -IBPB suffix. Must be explicitly turned on for AMD CPU models without -IBPB suffix. Requires the host CPU microcode to support this feature before it can be used for guest CPUs.

virt-ssbd

Required to enable the CVE-2018-3639 fix. Not included by default in any AMD CPU model. Must be explicitly turned on for all AMD CPU models. This should be provided to guests, even if amd-ssbd is also provided, for maximum guest compatibility. Note for some QEMU / libvirt versions, this must be force enabled when when using “Host model”, because this is a virtual feature that doesn’t exist in the physical host CPUs.

amd-ssbd

Required to enable the CVE-2018-3639 fix. Not included by default in any AMD CPU model. Must be explicitly turned on for all AMD CPU models. This provides higher performance than virt-ssbd so should be exposed to guests whenever available in the host. virt-ssbd should none the less also be exposed for maximum guest compatability as some kernels only know about virt-ssbd.

amd-no-ssb

Recommended to indicate the host is not vulnerable CVE-2018-3639. Not included by default in any AMD CPU model. Future hardware genarations of CPU will not be vulnerable to CVE-2018-3639, and thus the guest should be told not to enable its mitigations, by exposing amd-no-ssb. This is mutually exclusive with virt-ssbd and amd-ssbd.

pdpe1gb

Recommended to allow guest OS to use 1GB size pages. Not included by default in any AMD CPU model. Should be explicitly turned on for all AMD CPU models. Note that not all CPU hardware will support this feature.

Default x86 CPU models

The default QEMU CPU models are designed such that they can run on all hosts. If an application does not wish to do perform any host compatibility checks before launching guests, the default is guaranteed to work.

The default CPU models will, however, leave the guest OS vulnerable to various CPU hardware flaws, so their use is strongly discouraged. Applications should follow the earlier guidance to setup a better CPU configuration, with host passthrough recommended if live migration is not needed.

qemu32

qemu64

QEMU Virtual CPU version 2.5+ (32 & 64 bit variants). qemu64 is used for x86_64 guests and qemu32 is used for i686 guests, when no -cpu argument is given to QEMU, or no <cpu> is provided in libvirt XML.

Other non-recommended x86 CPUs

The following CPUs models are compatible with most AMD and Intel x86 hosts, but their usage is discouraged, as they expose a very limited featureset, which prevents guests having optimal performance.

kvm32

kvm64

Common KVM processor (32 & 64 bit variants). Legacy models just for historical compatibility with ancient QEMU versions.

486

athlon

phenom

coreduo

core2duo

n270

pentium

pentium2

pentium3

Various very old x86 CPU models, mostly predating the introduction of hardware assisted virtualization, that should thus not be required for running virtual machines.

Syntax for configuring CPU models

The example below illustrate the approach to configuring the various CPU models / features in QEMU and libvirt

QEMU command line

Host passthrough

   $ qemu-system-x86_64 -cpu host

With feature customization:

   $ qemu-system-x86_64 -cpu host,-vmx,...

Named CPU models

   $ qemu-system-x86_64 -cpu Westmere

With feature customization:

   $ qemu-system-x86_64 -cpu Westmere,+pcid,...

Libvirt guest XML

Host passthrough

   <cpu mode='host-passthrough'/>

With feature customization:

   <cpu mode='host-passthrough'>
       <feature name="vmx" policy="disable"/>
       ...
   </cpu>

Host model

   <cpu mode='host-model'/>

With feature customization:

   <cpu mode='host-model'>
       <feature name="vmx" policy="disable"/>
       ...
   </cpu>

Named model

   <cpu mode='custom'>
       <model>Westmere</model>
   </cpu>

With feature customization:

   <cpu mode='custom'>
       <model>Westmere</model>
       <feature name="pcid" policy="require"/>
       ...
   </cpu>

 

Ever since I started working on QEMU, a small directory called tests/tcg has been in a perpetually broken state. It contains tests that exercise QEMU’s ability to work across architectures using the power of the Tiny Code Generator. However as these tests needed to be compiled for the guest architectures and not the host architecture—this is known as cross-compiling—most developers never ran them. As the tests were hardly ever built inevitably a certain amount of bit-rot set in.

Cross Compilers

In the old days, cross-compilation setups were almost all hand-crafted affairs which involved building versions of binutils, gcc and a basic libc. If you couldn’t get someone to give you a pre-built tarball, it was something you laboured through once and hopefully never had to touch again. There were even dedicated scripts like crosstool-ng which attempted to make the process of patching and configuring your toolchain easier.

While the distributions have improved their support for cross compilers over the years, there are still plenty of variations in how they are deployed. It is hard for a project like QEMU which has to build on a wide range of operating systems and architectures to seamlessly use any given distributions compiler setup. However for those with cross compilers at hand configure now accepts two additional flags:

--cross-cc-$(ARCH)
--cross-cc-flags-$(ARCH)

With a compiler specified for each guest architecture you want to test the build system can now build and run the tests. For developers that don’t have cross compilers around, they can take advantage of QEMU’s docker images.

Enter Docker Containers

If you work in IT you would be hard pressed not to have noticed the hype around Docker and the concept of containerisation over the last few years. Put simply containers allow you to define a known working set of software that gets run in an isolated environment for a given task. While this has many uses for QEMU it allows us to define build environments that any developer can run without having to mess around with their preferred host setup.

Over the last few years QEMU’s build system has been expanding the number of docker images it supports. Most of this has been in service of our CI testing such as Patchew and Shippable but any developer with a docker setup can run the exact same images. For example if you want to check your patches won’t break when compiled on a 32 bit ARM system you can run:

make docker-test-build@debian-armhf-cross J=n

instead of tracking down a piece of ARM hardware to actually build on. Run make docker in your source tree to see the range of builds and tests it can support.

make check-tcg

With the latest work merged into master we can now take advantage of either hand-configured and Docker-based cross compilers to build test cases for TCG again. To run the TCG tests after you have built QEMU:

make check-tcg

and the build system will build and run all the tests it can for your configured targets.

Rules for tests/tcg

So now we have the infrastructure in place to add new tests what rules need to be followed to add new tests?

Well the first thing to note is currently all the tests are for the linux-user variant of QEMU. This means the tests are all currently user-space tests that have access to the Linux syscall ABI.

Another thing to note is the tests are separate from the rest of the QEMU test infrastructure. To keep things simple they are compiled as standalone “static” binaries. As the cross-compilation setup can be quite rudimentary for some of the rarer architectures we only compile against a standard libc. There is no support for linking to other libraries like for example glib. Thread and maths support is part of glibc so shouldn’t be a problem.

Finally when writing new tests consider if it really is architecture specific or can be added to tests/tcg/multiarch. The multiarch tests are re-built for every supported architecture and should be the default place for anything that tests syscalls or other common parts of the code base.

What’s next

My hope with this work is we can start adding more tests to systematically defend functionality in linux-user. In fact I hope the first port of call to reproducing a crash would be writing a test case that can be added to our growing library of tests.

Another thing that needs sorting out is getting toolchains for all of the less common architectures. The current work relies heavily on the excellent work of the Debian toolchain team in making multiarch aware cross compilers available in their distribution. However QEMU supports a lot more architectures than QEMU, some only as system emulations. In principle supporting them is as easy as adding another docker recipe but it might be these recipes end up having to compile the compilers from source.

The tests/tcg directory still contains a number of source files we don’t build.

The cris and openrisc directories contain user-space tests which just need the support of a toolchain and the relevant Makefile plumbing to be added.

The lm32, mips and xtensa targets have a set of tests that need a system emulator. Aside from adding the compilers as docker images some additional work is needed to handle the differences between plain linux-user tests which can simply return an exit code to getting the results from a qemu-system emulation. Some architectures have semi-hosting support already for this while others report their test status over a simple serial link which will need to be parsed and handled in custom versions of the run-%: rule.

• KVM on IBM Z (Session #23164): Latest news on our development work with the open source community
• KVM on Z Performance Analysis with Open Source Tooling (Session #23167): We will take a look at several tools and techniques to investigate KVM performance on IBM Z.
  

If you used QEMU in the past, you are probably familiar with the -net command line option, which can be used to configure a network connection for the guest, or with with the -netdev option, which configures a network back-end. Yet, QEMU v2.12 introduces a third way to configure NICs, the -nic option.

The ChangeLog of QEMU v2.12 says that -nic can “quickly create a network front-end (emulated NIC) and a host back-end”. But why did QEMU need yet another way to configure the network, and how does it compare with -net and -netdev? To answer these questions, we need to look at the model behind network virtualization in QEMU.

As hinted by the ChangeLog entry, a network interface consists of two separate entities:

1. The emulated hardware that the guest sees, i.e. the so-called NIC (network interface controller). On systems that support PCI cards, these typically could be an e1000 network card, a rtl8139 network card or a virtio-net device. This entity is also called the “front-end”.

2. The network back-end on the host side, i.e. the interface that QEMU uses to exchange network packets with the outside (like other QEMU instances or other real hosts in your intranet or the internet). The common host back-ends are the “user” (a.k.a. SLIRP) back-end which provides access to the host’s network via NAT, the “tap” back-end which allows the guest to directly access the host’s network, or the “socket” back-end which can be used to connect multiple QEMU instances to simulate a shared network for their guests.

Based on this, it is already possible to define the most obvious difference between -net, -netdev and -nic: the -net option can create either a front-end or a back-end (and also does other things); -netdev can only create a back-end; while a single occurrence of -nic will create both a front-end and a back-end. But for the non-obvious differences, we also need to have a detailed look at the -net and -netdev options first …

The legacy -net option

QEMU’s initial way of configuring the network for the guest was the -net option. The emulated NIC hardware can be chosen with the -net nic,model=xyz,... parameter, and the host back-end with the -net <backend>,... parameter (e.g. -net user for the SLIRP back-end). However, the emulated NIC and the host back-end are not directly connected. They are rather both connected to an emulated hub (called “vlan” in older versions of QEMU). Therefore, if you start QEMU with -net nic,model=e1000 -net user -net nic,model=virtio -net tap for example, you get a setup where all the front-ends and back-ends are connected together via a hub:

That means the e1000 NIC also gets the network traffic from the virtio-net NIC and both host back-ends… this is probably not what the users expected; it’s more likely that they wanted two separate networks in the guest, one for each NIC. Because -net always connects its NIC to a hub, you would have to tell QEMU to use two separate hubs, using the “vlan” parameter. For example -net nic,model=e1000,vlan=0 -net user,vlan=0 -net nic,model=virtio,vlan=1 -net tap,vlan=1 moves the virtio-net NIC and the “tap” back-end to a second hub (with ID #1).

Please note that the “vlan” parameter will be dropped in QEMU v3.0 since the term was rather confusing (it’s not related to IEEE 802.1Q for example) and caused a lot of misconfigurations in the past. Additional hubs can still be instantiated with -netdev (or -nic) and the special “hubport” back-end. The -net option itself will still stay around since it is still useful if you only want to use one front-end and one back-end together, or if you want to tunnel the traffic of multiple NICs through one back-end only (something like -net nic,model=e1000 -net nic,model=virtio -net l2tpv3,... for example).

The modern -netdev option

Beside the confusing “vlan” parameter of the -net option, there is one more major drawback with -net: the emulated hub between the NIC and the back-end gets in the way when the NIC front-end has to work closely together with the host back-end. For example, vhost acceleration cannot be enabled if you create a virtio-net device with -net nic,model=virtio.

To configure a network connection where the emulated NIC is directly connected to a host network back-end, without a hub in between, the well-established solution is to use the -netdev option for the back-end, together with -device for the front-end. Assuming that you want to configure the same devices as in the -net example above, you could use -netdev user,id=n1 -device e1000,netdev=n1 -netdev tap,id=n2 -device virtio-net,netdev=n2. This will give you straight 1:1 connections between the NICs and the host back-ends:

Note that you can also still connect the devices to a hub with the special -netdev hubport back-end, but in most of the normal use cases, the use of a hub is not required anymore.

Now while -netdev together with -device provide a very flexible and extensive way to configure a network connection, there are still two drawbacks with this option pair which prevented us from deprecating the legacy -net option completely:

1. The -device option can only be used for pluggable NICs. Boards (e.g. embedded boards) which feature an on-board NIC cannot be configured with -device yet, so -net nic,netdev=<id> must be used here instead.

2. In some cases, the -net option is easier to use (less to type). For example, assuming you want to set up a “tap” network connection and your default scripts /etc/qemu-ifup and -down are already in place, it’s enough to type -net nic -net tap to start your guest. To do the same with -netdev, you always have to specify an ID here, too, for example like this: -netdev tap,id=n1 -device e1000,netdev=n1.

The new -nic option

Looking at the disadvantages listed above, users could benefit from a convenience option that:

• is easier to use (and shorter to type) than -netdev <backend>,id=<id> -device <dev>,netdev=<id>
• can be used to configure on-board / non-pluggable NICs, too
• does not place a hub between the NIC and the host back-end.

This is where the new -nic option kicks in: this option can be used to configure both the guest’s NIC hardware and the host back-end in one go. For example, instead of -netdev tap,id=n1 -device e1000,netdev=n1 you can simply type -nic tap,model=e1000. If you don’t care about the exact NIC model type, you can even omit the model=... parameter and type -nic tap. This is even shorter and more convenient than the previous shortest way of typing -net nic -net tap. To get a list of NIC models that you can use with this option, you can simply run QEMU with -nic model=help.

Beside being easier to use, the -nic option can be used to configure on-board NICs, too (just like the -net option). For machines that have on-board NICs, the first -nic option configures the first on-board NIC, the second -nic option configures the second on-board NIC, and so forth.

Conclusion

• The new -nic option gives you an easy and quick way to configure the networking of your guest.
• For more detailed configuration, e.g. when you need to tweak the details of the emulated NIC hardware, you can use -device together with -netdev.
• The -net option should be avoided these days unless you really want to configure a set-up with a hub between the front-ends and back-ends.

Fedora 28 was released a few weeks ago. New Fedora 28 images are finally uploaded now.

There are no raspberry pi images any more. Just use the standard Fedora arm images, they work just fine for both arm (rpi 2) and aarch64 (rpi 3).

There efi images are for qemu. Some use grub2 as bootloader, some use systemd-boot. The filename indicates which uses which. The efi images can also be booted as container, using systemd-nspawn --boot --image <file>, but you have to convert them to raw first as systemd-nspawn can't handle qcow2.

As usual the images don't have a root password. You have to set one using virt-customize -a --root-password "password:<secret>", otherwise you can't login after boot.

The images have been created with imagefish.