ひまわり

はやく人間になりたい

QEMU: RISC-V 64のhypervisor extensionを有効にするほうほう

RISC-V

もくじ

2019/09/15追記: githubにrepositoryができて，ツールのビルド手順がwikiにできた github.com

ことの始まり

今進行形でLinux KVMにRISC-Vのpatchを入れようとしている人たちがいる

lkml.org

現状 under reviewだけど、このパッチはv5とある通り、改定が繰り返されている KVMに関することの、初出は, 2019/07/29で lkml.org

これを最初見たときに, RISC-V Privileged ISA Specificationには、確かに hypervisorに関する記述（それでも、まだdraft (2019/08/25日現在))はあるが、
それを実装したエミュレータやプロセッサは存在するのだろうかという疑問があった

どうもpatchのコメントを見るに、QEMU RISC-Vに、hypervisorの機能を実装し、それを用いて動かすということらしい
OpenSBI(firmware), xvisor(hypervisor), kvmtool(kvm client)についても、軽く調べたが、それは別に記事にまとめようと思う
上のpatchも気になるが、まずQEMUの実装を手元で動かしてみようと思う

QEMU

先のLinuxへのpatchのコメントに、手元で動かす方法が簡単にあった
QEMUに手を加えたものをGithubで開発しており、今本家のQEMUにpatchを送ってunder reviewとのことらしい

GitHub - alistair23/qemu at mainline/alistair/riscv-hyp-ext-v0.4.next

[Qemu-devel] [RFC v1 00/23] Add RISC-V Hypervisor Extension

つい先日QEMUのv4.1.0が出ていたので、もしやと思ってCHANGELOGを見に行ってしまったが、流石にまだだった

ゴール

動かしていろいろしたいが、この記事でのゴールは, Guest Linux をQEMU RISC-V64上で動かし、
そのGuestからRISC-V extension (M, F, D, Qとか)の識別子にH(Hypervisor)があること確認する事とする

要は、guest linuxのkernel logに下のようなメッセージが見えるということである

[    0.000000] riscv: ISA extensions acdfhimsu

一番右の文字列が拡張を表していて、hypervisorに関する拡張がプロセッサに実装されていない場合は、以下のようになる

[    0.000000] riscv: ISA extensions acdfimsu

わかりずらいが、"h"が居ない

準備

自分の環境は、Arch Linuxなので、試す人は、異なるところが環境であるかも

  1. QEMUを手元に持ってくる
$ git clone git@github.com:alistair23/qemu.git
$ git checkout -b rv64_hyv remotes/origin/mainline/alistair/riscv-hyp-ext-v0.4.next
$ ./configure --list-target=riscv64-softmmu 
$ make -j `nproc`

qemu/riscv64-softmmu/qemu-system-riscv64 ができる

  1. cross compileの環境の用意
$ git clone --recursive https://github.com/riscv/riscv-gnu-toolchain
$ ./configure --prefix=${install path} --enable-linux
$ make -j `nproc`
$ make install
  1. Firmware(OpenSBI)の用意
$ git clone https://github.com/riscv/opensbi.git
$ git checkout -b hyp_ext_v1 remotes/origin/hyp_ext_changes_v1
$ # setup the PATH environment variable in order to use RISC-V gcc, etc
$ export CROSS_COMPILE=riscv64-unknwon-linux-gnu-
$ make PLATFORM=qemu/virt -I=${install dir} install

  1. Rootfsの用意 KVMをkernelに内蔵してしまえばいらないが、せっかくなので起動した後に、
    dmesgを詳しく眺めたり, sysとかの下を見たりするためにbusybox使って用意する

  2. roofsとなるディレクトリの用意 

      $ mkdir ${initramfs_dir}
  $ mkdir -p ${initramfs_dir}/{bin, lib, sbin, dev}

  3. busybox 

    $ curl -L http://busybox.net/downloads/busybox-1.21.1.tar.bz2 | tar -xj
$ make allnoconfig
$ make menuconfig # enable static link build and some applets which are init, ash, dmesg, less, etc
$ make -j nproc
$ make CONFIG_PREFIX=${initramfs_dir} install

  4. Guest Linuxの用意

$ git clone https://github.com/avpatel/linux.git
$ make ARCH=riscv defconfig
$ make ARCH=riscv -j `nproc` CROSS_COMPILE=riscv64-unknown-linux-gnu- Image
$ make ARCH=riscv -j `nproc` CROSS_COMPILE=riscv64-unknown-linux-gnu- INSTALL_MOD_PATH=${initramfs_dir} modules_install
  1. initramfsの生成
$ cd ${initramfs_dir}
$ find . | cpio -o -H newc | gzip > ../initrd.img

起動

qemu-system-riscv64 \
        -monitor null \
        -cpu rv64,x-h=true \
        -m 512M \
        -display none \
        -serial mon:stdio \
        -M virt \
        -kernel local/platform/qemu/virt/firmware/fw_jump.elf \
        -device loader,file=linux/arch/riscv/boot/Image,addr=0x80200000 \
        -append "console=ttyS0 earlycon=sbi" \
        -initrd ./initrd.img \
  • boot log
qemu-system-riscv64: warning: No -bios option specified. Not loading a firmware.
qemu-system-riscv64: warning: This default will change in a future QEMU release. Please use the -bios option to avoid breakages when this happens.
qemu-system-riscv64: warning: See QEMU's deprecation documentation for details.

OpenSBI v0.4-28-g07f78e0 (Aug 24 2019 23:00:12)
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name          : QEMU Virt Machine
Platform HART Features : RV64ACDFHIMSU
Platform Max HARTs     : 8
Current Hart           : 0
Firmware Base          : 0x80000000
Firmware Size          : 112 KB
Runtime SBI Version    : 0.1

PMP0: 0x0000000080000000-0x000000008001ffff (A)
PMP1: 0x0000000000000000-0xffffffffffffffff (A,R,W,X)
[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[    0.000000] Linux version 5.3.0-rc5-00771-g1b24712aa35b (hima@arch) (gcc version 8.3.0 (GCC)) #3 SMP Sat Aug 24 22:32:46 JST 2019
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] Initial ramdisk at: 0x(____ptrval____) (2186569 bytes)
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000080200000-0x000000009fffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080200000-0x000000009fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x000000009fffffff]
[    0.000000] software IO TLB: mapped [mem 0x9b8fb000-0x9f8fb000] (64MB)
[    0.000000] riscv: ISA extensions acdfhimsu
[    0.000000] riscv: ELF capabilities acdfim
[    0.000000] percpu: Embedded 18 pages/cpu s34776 r8192 d30760 u73728
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 128775
[    0.000000] Kernel command line: root=/dev/ram rw console=ttyS0 earlycon=sbi
[    0.000000] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    0.000000] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.000000] Sorting __ex_table...
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 437296K/522240K available (6053K kernel code, 381K rwdata, 1886K rodata, 213K init, 305K bss, 84944K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=1.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.000000] NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0
[    0.000000] plic: mapped 53 interrupts with 1 handlers for 2 contexts.
[    0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x24e6a1710, max_idle_ns: 440795202120 ns
[    0.000330] sched_clock: 64 bits at 10MHz, resolution 100ns, wraps every 4398046511100ns
[    0.005481] Console: colour dummy device 80x25
[    0.009865] Calibrating delay loop (skipped), value calculated using timer frequency.. 20.00 BogoMIPS (lpj=40000)
[    0.010840] pid_max: default: 32768 minimum: 301
[    0.013178] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes, linear)
[    0.013802] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes, linear)
[    0.047420] rcu: Hierarchical SRCU implementation.
[    0.050989] smp: Bringing up secondary CPUs ...
[    0.051561] smp: Brought up 1 node, 1 CPU
[    0.092081] devtmpfs: initialized
[    0.103312] random: get_random_u32 called from bucket_table_alloc.isra.10+0x4e/0x160 with crng_init=0
[    0.106640] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.107616] futex hash table entries: 256 (order: 2, 16384 bytes, linear)
[    0.111381] NET: Registered protocol family 16
[    0.161110] vgaarb: loaded
[    0.164297] SCSI subsystem initialized
[    0.167398] usbcore: registered new interface driver usbfs
[    0.168074] usbcore: registered new interface driver hub
[    0.168649] usbcore: registered new device driver usb
[    0.177815] clocksource: Switched to clocksource riscv_clocksource
[    0.205067] NET: Registered protocol family 2
[    0.209833] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.210514] TCP established hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.211209] TCP bind hash table entries: 4096 (order: 4, 65536 bytes, linear)
[    0.211819] TCP: Hash tables configured (established 4096 bind 4096)
[    0.214047] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.214701] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.216495] NET: Registered protocol family 1
[    0.220686] RPC: Registered named UNIX socket transport module.
[    0.221157] RPC: Registered udp transport module.
[    0.221789] RPC: Registered tcp transport module.
[    0.222131] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.222662] PCI: CLS 0 bytes, default 64
[    0.227504] Unpacking initramfs...
[    0.427787] Freeing initrd memory: 2132K
[    0.428605] kvm [1]: hypervisor extension available
[    0.428989] kvm [1]: host has 14 VMID bits
[    0.433104] workingset: timestamp_bits=62 max_order=17 bucket_order=0
[    0.446603] NFS: Registering the id_resolver key type
[    0.447776] Key type id_resolver registered
[    0.448119] Key type id_legacy registered
[    0.448512] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    0.449980] 9p: Installing v9fs 9p2000 file system support
[    0.451533] NET: Registered protocol family 38
[    0.452109] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.452692] io scheduler mq-deadline registered
[    0.453100] io scheduler kyber registered
[    0.455802] pci-host-generic 30000000.pci: host bridge /soc/pci@30000000 ranges:
[    0.456835] pci-host-generic 30000000.pci:    IO 0x03000000..0x0300ffff -> 0x00000000
[    0.458022] pci-host-generic 30000000.pci:   MEM 0x40000000..0x7fffffff -> 0x40000000
[    0.460116] pci-host-generic 30000000.pci: ECAM at [mem 0x30000000-0x3fffffff] for [bus 00-ff]
[    0.461752] pci-host-generic 30000000.pci: PCI host bridge to bus 0000:00
[    0.462381] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.462849] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    0.463291] pci_bus 0000:00: root bus resource [mem 0x40000000-0x7fffffff]
[    0.464618] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000
[    0.609131] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    0.616741] printk: console [ttyS0] disabled
[    0.618205] 10000000.uart: ttyS0 at MMIO 0x10000000 (irq = 10, base_baud = 230400) is a 16550A
[    0.619862] printk: console [ttyS0] enabled
[    0.619862] printk: console [ttyS0] enabled
[    0.620593] printk: bootconsole [sbi0] disabled
[    0.620593] printk: bootconsole [sbi0] disabled
[    0.623483] [drm] radeon kernel modesetting enabled.
[    0.639863] loop: module loaded
[    0.642528] libphy: Fixed MDIO Bus: probed
[    0.643506] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    0.643814] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    0.644391] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.644761] ehci-pci: EHCI PCI platform driver
[    0.645194] ehci-platform: EHCI generic platform driver
[    0.645853] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.646235] ohci-pci: OHCI PCI platform driver
[    0.646685] ohci-platform: OHCI generic platform driver
[    0.647488] usbcore: registered new interface driver uas
[    0.647976] usbcore: registered new interface driver usb-storage
[    0.649509] mousedev: PS/2 mouse device common for all mice
[    0.651177] usbcore: registered new interface driver usbhid
[    0.651469] usbhid: USB HID core driver
[    0.653237] NET: Registered protocol family 10
[    0.659911] Segment Routing with IPv6
[    0.660491] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
[    0.663586] NET: Registered protocol family 17
[    0.665134] 9pnet: Installing 9P2000 support
[    0.665889] Key type dns_resolver registered
[    0.699386] Freeing unused kernel memory: 212K
[    0.699673] This architecture does not have kernel memory protection.
[    0.700041] Run /init as init process
#
  • ISA extensionの確認
$ dmesg | grep "riscv: ISA"
[    0.000000] riscv: ISA extensions acdfhimsu
$ dmesg | grep kvm 
[    0.337890] kvm [1]: hypervisor extension available
[    0.338435] kvm [1]: host has 14 VMID bits

これで、 hypervisor extensionが有効なqemuが用意できた

TODO

コマンド等が雑にかいてあるので、気をつけて
OpenSBI, kvmtool, QEMUのパッチの内容、RISC-V KVMについて等について、今後まとめたい

これで、あそべそうだ

2019/09/02: 実際にゲストうごかした