virtio-net is completely broken in terms of LSO / RSO #1131
Assignees
Labels
Bug
Bug
Comments
|
I think i can confirm this bug. However, i suspect this might be due to a newer Windows (or host) upgrade breaking things. My wife uses a Windows VM on our "NAS", where she puts all her photos. The resulting network-performance was fine, reading new photos and exporting finished ones (or the backup of the library) was pleasantly fast and the limiting factor have been the disks. However, i went ahead and upgraded Windows to 11 and upgraded the host from 6.8.1 to 6.10.2. Since then, importing new photos is very slow, the library backup gets cancelled because of network errors. RX seems to be better than TX, in other Words. Currently i am running Host: VM: Since this upgrade, especially TX is problematic. If one uses wireshark to look at the network-traffic caused by the VM - with all acceleration features enabled - i get frequent "duplicate ACKs" and "Out of order" errors. My current test-file was a 2GB .zip and i was unable to copy it onto the nas (TX if seen from the Windows 11 VM) without windows cancelling the transfer because of errors. Currently, i have disabled every offloading feature of the driver using the windows device properties, which seems to have fixed the problem. Since this was working at one point in time, there seems to be a regression somewhere. If i throw a LinuxLive .iso in the CD-ROM drive of the VM and rerun the tests i get good performance, stable connections and wireshark does not complain. If i can provide anything to help to narrow that problem down, i am happy to help/test/... Thank you for your work on the virtio-drivers, have a nice day, |
|
Can you please attach complete libvirt XML or complete qemu command line? |
|
Please also run "ethtool -k" on the host and the target machine |
|
I can confirm this issue on the following System: QEMU Version: 9.0.2 Host: Gentoo Linux, Kernel 6.10.4 Guest OS(es): Windows 10 and 11 with latest updates using latest VirtIO drivers (0.1.262-2) All guests are connected to the same bridge ( "br1" no hardware device attached) ethtool -K br1 tso off or ethtool -K vnet(x) tso off fixes the issue. No modifications on any guest needed. |
|
Windows VM: <domain type="kvm">
<name>win10</name>
<uuid>ca6d862f-1a43-4106-96bd-45d444819735</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/10"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">8388608</memory>
<currentMemory unit="KiB">8388608</currentMemory>
<memoryBacking>
<hugepages/>
<nosharepages/>
<locked/>
</memoryBacking>
<vcpu placement="static">2</vcpu>
<os>
<type arch="x86_64" machine="pc-q35-8.2">hvm</type>
<boot dev="hd"/>
</os>
<features>
<acpi/>
<apic/>
<hyperv mode="custom">
<relaxed state="on"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
</hyperv>
<vmport state="off"/>
</features>
<cpu mode="host-passthrough" check="none" migratable="on"/>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="hypervclock" present="yes"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/mnt/ssd/imgs/Win10_English_x64.iso"/>
<target dev="sdb" bus="sata"/>
<readonly/>
<address type="drive" controller="0" bus="0" target="0" unit="1"/>
</disk>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/mnt/ssd/imgs/virtio-win-0.1.262.iso"/>
<target dev="sdc" bus="sata"/>
<readonly/>
<address type="drive" controller="0" bus="0" target="0" unit="2"/>
</disk>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2"/>
<source file="/mnt/nvme/kvm/win10.qcow2"/>
<target dev="vda" bus="virtio"/>
<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci" ports="15">
<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</controller>
<controller type="pci" index="0" model="pcie-root"/>
<controller type="pci" index="1" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="1" port="0x10"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="2" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="2" port="0x11"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
</controller>
<controller type="pci" index="3" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="3" port="0x12"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
</controller>
<controller type="pci" index="4" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="4" port="0x13"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
</controller>
<controller type="pci" index="5" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="5" port="0x14"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
</controller>
<controller type="pci" index="6" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="6" port="0x15"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
</controller>
<controller type="pci" index="7" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="7" port="0x16"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
</controller>
<controller type="pci" index="8" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="8" port="0x17"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
</controller>
<controller type="pci" index="9" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="9" port="0x18"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="10" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="10" port="0x19"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
</controller>
<controller type="pci" index="11" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="11" port="0x1a"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
</controller>
<controller type="pci" index="12" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="12" port="0x1b"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
</controller>
<controller type="pci" index="13" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="13" port="0x1c"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
</controller>
<controller type="pci" index="14" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="14" port="0x1d"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
</controller>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
</controller>
<controller type="virtio-serial" index="0">
<address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</controller>
<interface type="direct">
<mac address="52:54:00:2f:7f:ff"/>
<source dev="enp4s0" mode="bridge"/>
<model type="virtio"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
<interface type="bridge">
<mac address="52:54:00:85:fc:71"/>
<source bridge="bridge0"/>
<model type="virtio"/>
<driver name="vhost" txmode="iothread" ioeventfd="on" event_idx="on" queues="12">
<host csum="on" gso="on" tso4="on" tso6="on" ecn="on" ufo="on" mrg_rxbuf="on"/>
<guest csum="on" tso4="on" tso6="on" ecn="on" ufo="on"/>
</driver>
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</interface>
<serial type="pty">
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
</serial>
<console type="pty">
<target type="serial" port="0"/>
</console>
<channel type="spicevmc">
<target type="virtio" name="com.redhat.spice.0"/>
<address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>
<input type="tablet" bus="usb">
<address type="usb" bus="0" port="1"/>
</input>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<graphics type="spice" autoport="yes">
<listen type="address"/>
<image compression="off"/>
</graphics>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
</sound>
<audio id="1" type="spice"/>
<video>
<model type="virtio" heads="1" primary="yes"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
</video>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="2"/>
</redirdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="3"/>
</redirdev>
<watchdog model="itco" action="reset"/>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</memballoon>
</devices>
</domain>
Linux VM: <domain type="kvm">
<name>OSIRIS</name>
<uuid>75cdf1f7-d812-47b2-9424-d3229e7e8f8f</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://ubuntu.com/ubuntu/22.04"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">16777216</memory>
<currentMemory unit="KiB">16777216</currentMemory>
<memoryBacking>
<hugepages/>
<nosharepages/>
<locked/>
<source type="memfd"/>
<access mode="shared"/>
</memoryBacking>
<vcpu placement="static">4</vcpu>
<os firmware="efi">
<type arch="x86_64" machine="pc-q35-8.2">hvm</type>
<firmware>
<feature enabled="yes" name="enrolled-keys"/>
<feature enabled="yes" name="secure-boot"/>
</firmware>
<loader readonly="yes" secure="yes" type="pflash" format="qcow2">/usr/share/edk2/ovmf/OVMF_CODE_4M.secboot.qcow2</loader>
<nvram template="/usr/share/edk2/ovmf/OVMF_VARS_4M.secboot.qcow2" format="qcow2">/var/lib/libvirt/qemu/nvram/OSIRIS_VARS.qcow2</nvram>
<boot dev="hd"/>
</os>
<features>
<acpi/>
<apic/>
<vmport state="off"/>
<smm state="on"/>
</features>
<cpu mode="host-passthrough" check="none" migratable="on"/>
<clock offset="utc">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2"/>
<source file="/mnt/nvme/vms/OSIRIS.qcow2"/>
<target dev="vda" bus="virtio"/>
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci" ports="15">
<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</controller>
<controller type="pci" index="0" model="pcie-root"/>
<controller type="pci" index="1" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="1" port="0x10"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="2" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="2" port="0x11"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
</controller>
<controller type="pci" index="3" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="3" port="0x12"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
</controller>
<controller type="pci" index="4" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="4" port="0x13"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
</controller>
<controller type="pci" index="5" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="5" port="0x14"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
</controller>
<controller type="pci" index="6" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="6" port="0x15"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
</controller>
<controller type="pci" index="7" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="7" port="0x16"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
</controller>
<controller type="pci" index="8" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="8" port="0x17"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
</controller>
<controller type="pci" index="9" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="9" port="0x18"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="10" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="10" port="0x19"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
</controller>
<controller type="pci" index="11" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="11" port="0x1a"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
</controller>
<controller type="pci" index="12" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="12" port="0x1b"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
</controller>
<controller type="pci" index="13" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="13" port="0x1c"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
</controller>
<controller type="pci" index="14" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="14" port="0x1d"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
</controller>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
</controller>
<controller type="virtio-serial" index="0">
<address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</controller>
<controller type="scsi" index="0" model="virtio-scsi">
<address type="pci" domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
</controller>
<filesystem type="mount" accessmode="passthrough">
<driver type="virtiofs"/>
<source dir="/home/marcin"/>
<target dir="DRAGONFLY marcin"/>
<address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
</filesystem>
<interface type="direct">
<mac address="00:0c:29:04:1e:11"/>
<source dev="enp4s0" mode="bridge"/>
<model type="virtio"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
<interface type="bridge">
<mac address="00:0c:29:04:1e:1b"/>
<source bridge="bridge0"/>
<model type="virtio"/>
<address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</interface>
<serial type="pty">
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
</serial>
<console type="pty">
<target type="serial" port="0"/>
</console>
<channel type="unix">
<target type="virtio" name="org.qemu.guest_agent.0"/>
<address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>
<channel type="spicevmc">
<target type="virtio" name="com.redhat.spice.0"/>
<address type="virtio-serial" controller="0" bus="0" port="2"/>
</channel>
<input type="tablet" bus="usb">
<address type="usb" bus="0" port="1"/>
</input>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<graphics type="spice" autoport="yes">
<listen type="address"/>
<image compression="off"/>
</graphics>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
</sound>
<audio id="1" type="spice"/>
<video>
<model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
</video>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="2"/>
</redirdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="3"/>
</redirdev>
<watchdog model="itco" action="reset"/>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</memballoon>
<rng model="virtio">
<backend model="random">/dev/urandom</backend>
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</rng>
</devices>
</domain>Here's
EDIT: Previously I only included Linux guest interface options, now I also added Windows vnet data. |
|
A bug was fixed in 6.10.5 that looks like this. It was introduced in 6.10.3. |
|
@anthonyde Thanks! @TripleEmcoder @brotkastn - can you test with the updated kernel? |
|
Yes, thank you! I am running 6.10.6 at the moment and Linux/FreeBSD -> Windows with TSO Same improvements for the reverse direction, Windows -> Linux with LSO |
|
I can confirm that this issue is gone when running 6.10.6. copying files from a Windows 10 / 11 host to a TNS host works as espected The second issue, that the copy process from Windows 10 or 11 Host to the TNS host didn't start according to the Windows copy dialog is also gone. Exactly the same as disabling TSO on "br1" or the Windows hosts vnet* device in earlier kernels. |
|
@TripleEmcoder Please feel free to close the issue if it's not actual anymore |
I can also confirm that upgrading to 6.10.6 fixes the issue here. Thank you everyone who helped to track this down :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
(or I am very bad at configuring KVM / QEMU / Windows, but I am using the defaults mostly for the repro here)
Describe the bug
To get acceptable TX performance in a Windows machine using a virtio NIC one must disable Large Send Offload in the adapter properties. This increases TX speed from ca. 1Mbps to 2Gbps and affects both guest-to-host, guest-to-guest traffic. There are numerous reports of this on the internet, I can probably link a few threads if needed.
To get acceptable performance when on the RX-side of transfers coming in from other guests one must disable TCP Segment Offload on those machines - or perhaps there is a way to do it on the
vnetXinterface on the host, but I haven't discovered one. This increases transfers to Windows to 3Gbps from the original ca. 15Mbps when Linux transmits or 100Kbps (sic!) when data originates from FreeBSD.To Reproduce
Expected behavior
On Windows virtio NICs should provide workable speeds in their default configuration (a few Gbps feels adequate). Currently in the default setup speeds allow SSH sessions, but SMB file transfers from TrueNAS (FreeBSD) fail completely, same any attempt to access its web UI.
Ideally Windows NICs would provide the similar performance to virtio on Linux, which provides raw speeds in the 20Gbps range both for host+guest and guest+guest traffic in both directions, which requires working LSO and RSO it seems.
The above has been tested on:
Host:
VM:
Additional context
Full table of measurements:
For the example cell of
3.5G / ~0the table reads as follows: when runningiperf3 -son TrueNAS andiperf3 -c <IP>on Windows 11 you get 3.5Gbps of outgoing traffic and then with--reversethe transfer is basically stalled (measured 50-150Kbps).To see the basiline capabilities of the IPv4 stack I also measeured in-VM localhost-loopback speeds.
There are similar values on the other side of the diagonal for when
-sand-c <IP>were called in reverse, to see if the socket setups have any influence on the values.ifconfig vnet1 -tsoethtool -K enp7s0 tso offThe text was updated successfully, but these errors were encountered: