### Install QEMU Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come Build and install QEMU from source. Ensure you have the necessary build tools and dependencies. The `--enable-slirp` option is used for network acceleration. ```shell mkdir build \ cd build \ ../configure --enable-slirp \ make -j \ sudo make install \ ``` -------------------------------- ### Install Debian VM with QEMU Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come Install the Debian VM using the downloaded ISO and the created disk image. This command boots from the CD-ROM ('-boot d'), allocates 4GB of RAM ('-m 4G'), configures a network interface with host forwarding for SSH ('-netdev user,id=net0,hostfwd=tcp::10022-:22'), and enables KVM acceleration ('-accel kvm'). ```shell qemu-system-x86_64 -boot d -cdrom debian-12.2.0-amd64-netinst.iso -m 4G \ -device e1000,netdev=net0,mac=52:54:00:12:34:56 -netdev user,id=net0,hostfwd=tcp::10022-:22 \ -hda mydisk.img -accel kvm ``` -------------------------------- ### Clone QEMU Repository Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come Clone the QEMU project repository to your local machine to begin the installation process. ```git git clone https://gitlab.com/qemu-project/qemu.git ``` -------------------------------- ### Launch Debian VM Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come Launch the installed Debian VM with 8GB of RAM and KVM acceleration enabled. This command assumes the VM has already been installed on 'mydisk.img'. ```shell qemu-system-x86_64 -hda mydisk.img -m 8G -accel kvm ``` -------------------------------- ### Build VM with CXL PMEM and VMEM Devices Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come Use this command to launch a QEMU VM with a complex CXL setup including persistent and volatile memory, interleaved devices, and multiple NUMA nodes. Ensure necessary CXL and NVdimm options are enabled in the machine configuration. ```bash qemu-system-x86_64 -hda img/snapshot.img -accel kvm \ -machine q35,nvdimm=on,cxl=on \ -device e1000,netdev=net0,mac=52:54:00:12:34:56 \ -netdev user,id=net0,hostfwd=tcp::10022-:22 \ -m 4G,slots=8,maxmem=10G \ -smp cpus=8,cores=2,sockets=4 \ -object memory-backend-ram,size=1G,id=ram0 \ -object memory-backend-ram,size=1G,id=ram1 \ -object memory-backend-ram,size=1G,id=ram2 \ -object memory-backend-ram,size=1G,id=ram3 \ -object memory-backend-ram,id=cxl-mem0,share=on,size=256M \ -object memory-backend-ram,id=cxl-mem1,share=on,size=256M \ -object memory-backend-ram,id=cxl-mem2,share=on,size=256M \ -object memory-backend-ram,id=cxl-mem3,share=on,size=256M \ -object memory-backend-file,id=cxl-mem4,share=on,mem-path=/tmp/cxltest.raw,size=256M \ -object memory-backend-file,id=cxl-lsa4,share=on,mem-path=/tmp/lsa.raw,size=256M \ -numa node,nodeid=0,cpus=0-1,memdev=ram0 \ -numa node,nodeid=1,cpus=2-3,memdev=ram1 \ -numa node,nodeid=2,cpus=4-5,memdev=ram2 \ -numa node,nodeid=3,cpus=6-7,memdev=ram3 \ -device pxb-cxl,numa_node=0,bus_nr=24,bus=pcie.0,id=pxb-cxl.1 \ -device pxb-cxl,numa_node=1,bus_nr=32,bus=pcie.0,id=pxb-cxl.2 \ -device pxb-cxl,numa_node=3,bus_nr=40,bus=pcie.0,id=pxb-cxl.3 \ -device cxl-rp,port=0,bus=pxb-cxl.1,id=root_port1,chassis=0,slot=0 \ -device cxl-rp,port=1,bus=pxb-cxl.1,id=root_port2,chassis=0,slot=3 \ -device cxl-rp,port=2,bus=pxb-cxl.2,id=root_port3,chassis=1,slot=0 \ -device cxl-rp,port=3,bus=pxb-cxl.2,id=root_port4,chassis=1,slot=5 \ -device cxl-rp,port=0,bus=pxb-cxl.3,id=root_port5,chassis=2,slot=0 \ -device cxl-upstream,bus=root_port1,id=us0 \ -device cxl-upstream,bus=root_port3,id=us1 \ -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=7 \ -device cxl-type3,bus=swport0,volatile-memdev=cxl-mem0,id=cxl-vmem0 \ -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=8 \ -device cxl-type3,bus=swport1,volatile-memdev=cxl-mem1,id=cxl-vmem1 \ -device cxl-downstream,port=2,bus=us1,id=swport2,chassis=1,slot=9 \ -device cxl-type3,bus=swport2,volatile-memdev=cxl-mem2,id=cxl-vmem2 \ -device cxl-downstream,port=3,bus=us1,id=swport3,chassis=1,slot=10 \ -device cxl-type3,bus=swport3,volatile-memdev=cxl-mem3,id=cxl-vmem3 \ -device cxl-type3,bus=root_port5,persistent-memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem0 \ -M cxl-fmw.0.targets.0=pxb-cxl.1,cxl-fmw.0.size=4G,cxl-fmw.1.targets.0=pxb-cxl.2,cxl-fmw.1.size=4G,cxl-fmw.2.targets.0=pxb-cxl.3, cxl-fmw.2.size=512M ``` -------------------------------- ### QEMU Configuration for Complex Memory Setup Source: https://github.com/open-mpi/hwloc/wiki/Simulating-complex-memory-with-Qemu This QEMU command line simulates a system with two sockets, each having a CXL switch with multiple devices and memory regions. It configures NUMA nodes, CXL memory backends (RAM, file, persistent), and defines the memory fabric (FWM) targets and sizes. ```bash qemu-system-x86_64 \ -machine q35,accel=kvm,nvdimm=on,cxl=on \ -drive if=pflash,format=raw,file=$FILES/OVMF.fd \ -drive media=disk,format=qcow2,file=$FILES/efi.qcow2 \ -device e1000,netdev=net0,mac=52:54:00:12:34:56 \ -netdev user,id=net0,hostfwd=tcp::10022-:22 \ -m 2G,slots=8,maxmem=6G \ -smp cpus=4,cores=2,sockets=2 \ -object memory-backend-ram,size=1G,id=mem0 \ -numa node,nodeid=0,cpus=0-1,memdev=mem0 \ -object memory-backend-ram,size=1G,id=mem1 \ -numa node,nodeid=1,cpus=2-3,memdev=mem1 \ \ -object memory-backend-ram,id=cxl-mem0,share=on,size=256M \ -device pxb-cxl,numa_node=0,bus_nr=16,bus=pcie.0,id=cxl.0 \ -device cxl-rp,port=0,bus=cxl.0,id=root_port16,chassis=0,slot=0 \ -device cxl-type3,bus=root_port16,volatile-memdev=cxl-mem0,id=cxl-mem0 \ \ -object memory-backend-file,id=cxl-pmem1,share=on,mem-path=/tmp/cxltest1.raw,size=256M \ -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa1.raw,size=256M \ -object memory-backend-ram,id=cxl-mem2,share=on,size=256M \ -object memory-backend-file,id=cxl-pmem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M \ -object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M \ -device pxb-cxl,numa_node=0,bus_nr=24,bus=pcie.0,id=cxl.1 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port24,chassis=1,slot=0 \ -device cxl-upstream,bus=root_port24,id=sw0 \ -device cxl-downstream,port=1,bus=sw0,id=sw0port1,chassis=1,slot=1 \ -device cxl-type3,bus=sw0port1,persistent-memdev=cxl-pmem1,lsa=cxl-lsa1,id=cxl-pmem1 \ -device cxl-downstream,port=2,bus=sw0,id=sw0port2,chassis=1,slot=2 \ -device cxl-type3,bus=sw0port2,volatile-memdev=cxl-mem2,persistent-memdev=cxl-pmem2,lsa=cxl-lsa2,id=cxl-pmem2 \ \ -device pxb-cxl,numa_node=1,bus_nr=32,bus=pcie.0,id=cxl.2 \ -device cxl-rp,port=0,bus=cxl.2,id=root_port32,chassis=2,slot=0 \ -device cxl-upstream,bus=root_port32,id=sw1 \ -device cxl-downstream,port=0,bus=sw1,id=sw1port0,chassis=2,slot=1 \ -object memory-backend-ram,id=cxl-mem3,share=on,size=256M \ -device cxl-type3,bus=sw1port0,volatile-memdev=cxl-mem3,id=cxl-mem3 \ -device cxl-downstream,port=1,bus=sw1,id=sw1port1,chassis=2,slot=2 \ -object memory-backend-ram,id=cxl-mem4,share=on,size=256M \ -device cxl-type3,bus=sw1port1,volatile-memdev=cxl-mem4,id=cxl-mem4 \ \ -object memory-backend-file,id=cxl-pmem5,share=on,mem-path=/tmp/cxltest5.raw,size=256M \ -object memory-backend-file,id=cxl-lsa5,share=on,mem-path=/tmp/lsa5.raw,size=256M \ -device pxb-cxl,numa_node=1,bus_nr=40,bus=pcie.0,id=cxl.3 \ -device cxl-rp,port=0,bus=cxl.3,id=root_port40,chassis=3,slot=0 \ -device cxl-type3,bus=root_port40,persistent-memdev=cxl-pmem5,lsa=cxl-lsa5,id=cxl-pmem5 \ -M \ cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=4G, cxl-fmw.1.targets.0=cxl.1,cxl-fmw.1.size=4G, cxl-fmw.2.targets.0=cxl.2,cxl-fmw.2.size=4G, cxl-fmw.3.targets.0=cxl.3,cxl-fmw.3.size=4G ``` -------------------------------- ### Install HWLOC Plugins Source: https://github.com/open-mpi/hwloc/blob/master/contrib/windows-cmake/CMakeLists.txt Installs hwloc plugins based on the HWLOC_ENABLED_PLUGINS_LIST. Libraries are placed in lib/hwloc, archives in lib/hwloc, and runtime binaries in bin/hwloc. ```cmake foreach(plugin IN LISTS HWLOC_ENABLED_PLUGINS_LIST) install(TARGETS hwloc_${plugin} LIBRARY DESTINATION lib/hwloc ARCHIVE DESTINATION lib/hwloc RUNTIME DESTINATION bin/hwloc ) endforeach() ``` -------------------------------- ### Install HWLOC Headers Source: https://github.com/open-mpi/hwloc/blob/master/contrib/windows-cmake/CMakeLists.txt Installs hwloc header files if HWLOC_SKIP_INCLUDES is not set. This includes the main hwloc.h, all .h files in the include directory, and autogenerated config.h. ```cmake if(NOT HWLOC_SKIP_INCLUDES) install(FILES ${TOPDIR}/include/hwloc.h TYPE INCLUDE) install(DIRECTORY ${TOPDIR}/include/hwloc TYPE INCLUDE FILES_MATCHING PATTERN "*.h") install(FILES ${CMAKE_CURRENT_BINARY_DIR}/include/hwloc/autogen/config.h DESTINATION include/hwloc/autogen) endif() ``` -------------------------------- ### Download Debian Netinstall ISO Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come Download the Debian 12.2.0 amd64 netinstall ISO image to your current directory. This image will be used to install the virtual machine. ```shell wget https://www.debian.org/distrib/netinst/debian-12.2.0-amd64-netinst.iso . ``` -------------------------------- ### Install hwloc Target Source: https://github.com/open-mpi/hwloc/blob/master/contrib/windows-cmake/CMakeLists.txt Installs the main hwloc library target. This makes the compiled library available for use by other projects or for system-wide installation. ```cmake install(TARGETS hwloc) ``` -------------------------------- ### List Active NVDIMM Devices Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come Run this command inside the VM to list active and enabled NVDIMM devices and their configurations. This helps verify the NVDIMM setup. ```bash ndctl list -NRD ``` -------------------------------- ### QEMU: Single PMEM Device on Single 4-Core Socket Source: https://github.com/open-mpi/hwloc/wiki/Simulating-complex-memory-with-Qemu Configures a single Persistent Memory (PMEM) device using CXL. This setup requires memory-backend-file objects for both the memory and its label storage, along with a Fixed Memory Window (FWM) to map the memory. ```bash qemu-system-x86_64 \ -machine q35,accel=kvm,nvdimm=on,cxl=on \ -drive if=pflash,format=raw,file=$FILES/OVMF.fd \ -drive media=disk,format=qcow2,file=$FILES/efi.qcow2 \ -device e1000,netdev=net0,mac=52:54:00:12:34:56 \ -netdev user,id=net0,hostfwd=tcp::10022-:22 \ -m 4G,slots=8,maxmem=8G \ -smp 4 \ -object memory-backend-ram,size=4G,id=mem0 \ -numa node,nodeid=0,cpus=0-3,memdev=mem0 \ -object memory-backend-file,id=pmem0,share=on,mem-path=/tmp/cxltest.raw,size=256M \ -object memory-backend-file,id=cxl-lsa0,share=on,mem-path=/tmp/lsa.raw,size=256M \ -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ -device cxl-type3,bus=root_port13,persistent-memdev=pmem0,lsa=cxl-lsa0,id=cxl-pmem0 \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G ``` -------------------------------- ### Install LSTOPO Tool Source: https://github.com/open-mpi/hwloc/blob/master/contrib/windows-cmake/CMakeLists.txt Conditionally installs the lstopo tool if HWLOC_SKIP_LSTOPO is not set. The target to be installed is specified by the LSTOPOS variable. ```cmake if(NOT HWLOC_SKIP_LSTOPO) install(TARGETS ${LSTOPOS}) endif() ``` -------------------------------- ### Install HWLOC Tools Source: https://github.com/open-mpi/hwloc/blob/master/contrib/windows-cmake/CMakeLists.txt Conditionally installs hwloc tools if HWLOC_SKIP_TOOLS is not set. The targets to be installed are specified by the TOOLS variable. ```cmake if(NOT HWLOC_SKIP_TOOLS) install(TARGETS ${TOOLS}) endif() ``` -------------------------------- ### Install MSVC PDB Files Source: https://github.com/open-mpi/hwloc/blob/master/contrib/windows-cmake/CMakeLists.txt Conditionally installs PDB (Program Database) files for debugging on MSVC builds when shared libraries are enabled. These files are marked as optional. ```cmake if(MSVC AND BUILD_SHARED_LIBS) install(FILES $ TYPE BIN OPTIONAL) endif() ``` -------------------------------- ### QEMU: Two RAM Devices on Different PXBs Source: https://github.com/open-mpi/hwloc/wiki/Simulating-complex-memory-with-Qemu Configures two volatile RAM devices attached to different CXL PXBs. Each RAM device requires a separate memory-backend-ram object and is attached via its own CXL root port on a distinct PXB. This setup uses two Fixed Memory Windows (FWM), one for each PXB. ```bash qemu-system-x86_64 \ -machine q35,accel=kvm,nvdimm=on,cxl=on \ -drive if=pflash,format=raw,file=$FILES/OVMF.fd \ -drive media=disk,format=qcow2,file=$FILES/efi.qcow2 \ -device e1000,netdev=net0,mac=52:54:00:12:34:56 \ -netdev user,id=net0,hostfwd=tcp::10022-:22 \ -m 2G,slots=8,maxmem=6G \ -smp cpus=4,cores=2,sockets=2 \ -object memory-backend-ram,size=1G,id=mem0 \ -numa node,nodeid=0,cpus=0-1,memdev=mem0 \ -object memory-backend-ram,size=1G,id=mem1 \ -numa node,nodeid=1,cpus=2-3,memdev=mem1 \ -object memory-backend-ram,id=vmem0,share=on,size=256M \ -device pxb-cxl,numa_node=0,bus_nr=12,bus=pcie.0,id=cxl.1 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \ -object memory-backend-ram,id=vmem1,share=on,size=256M \ -device pxb-cxl,numa_node=1,bus_nr=14,bus=pcie.0,id=cxl.2 \ -device cxl-rp,port=0,bus=cxl.2,id=root_port14,chassis=1,slot=2 \ -device cxl-type3,bus=root_port14,volatile-memdev=vmem1,id=cxl-vmem1 \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.1.targets.0=cxl.2,cxl-fmw.1.size=4G ``` -------------------------------- ### Configure QEMU HMAT Performance Attributes Source: https://github.com/open-mpi/hwloc/wiki/Simulating-complex-memory-with-Qemu Set Host-side Memory Attributes Table (HMAT) latency and bandwidth for NUMA nodes in QEMU. This example configures two CPU nodes and a third node with shared performance characteristics. ```bash qemu-system-x86_64 -accel kvm \ -machine pc,hmat=on \ -drive if=pflash,format=raw,file=./OVMF.fd \ -drive media=disk,format=qcow2,file=efi.qcow2 \ -smp 4 \ -m 3G \ -object memory-backend-ram,size=1G,id=ram0 \ -object memory-backend-ram,size=1G,id=ram1 \ -object memory-backend-ram,size=1G,id=ram2 \ -numa node,nodeid=0,memdev=ram0,cpus=0-1 \ -numa node,nodeid=1,memdev=ram1,cpus=2-3 \ -numa node,nodeid=2,memdev=ram2 \ -numa hmat-lb,initiator=0,target=0,hierarchy=memory,data-type=access-latency,latency=10 \ -numa hmat-lb,initiator=0,target=0,hierarchy=memory,data-type=access-bandwidth,bandwidth=10485760 \ -numa hmat-lb,initiator=0,target=1,hierarchy=memory,data-type=access-latency,latency=20 \ -numa hmat-lb,initiator=0,target=1,hierarchy=memory,data-type=access-bandwidth,bandwidth=5242880 \ -numa hmat-lb,initiator=0,target=2,hierarchy=memory,data-type=access-latency,latency=30 \ -numa hmat-lb,initiator=0,target=2,hierarchy=memory,data-type=access-bandwidth,bandwidth=1048576 \ -numa hmat-lb,initiator=1,target=0,hierarchy=memory,data-type=access-latency,latency=20 \ -numa hmat-lb,initiator=1,target=0,hierarchy=memory,data-type=access-bandwidth,bandwidth=5242880 \ -numa hmat-lb,initiator=1,target=1,hierarchy=memory,data-type=access-latency,latency=10 \ -numa hmat-lb,initiator=1,target=1,hierarchy=memory,data-type=access-bandwidth,bandwidth=10485760 \ -numa hmat-lb,initiator=1,target=2,hierarchy=memory,data-type=access-latency,latency=30 \ -numa hmat-lb,initiator=1,target=2,hierarchy=memory,data-type=access-bandwidth,bandwidth=1048576 ``` -------------------------------- ### QEMU: Two RAM Devices on Single PXB Source: https://github.com/open-mpi/hwloc/wiki/Simulating-complex-memory-with-Qemu Configures two volatile RAM devices attached to the same CXL PXB, each connected via a separate root port. This setup uses a single Fixed Memory Window (FWM) that can be configured as a single interleaved region or two separate regions. Note that Linux may require a second RP for correct decoder propagation. ```bash qemu-system-x86_64 \ -machine q35,accel=kvm,nvdimm=on,cxl=on \ -drive if=pflash,format=raw,file=$FILES/OVMF.fd \ -drive media=disk,format=qcow2,file=$FILES/efi.qcow2 \ -device e1000,netdev=net0,mac=52:54:00:12:34:56 \ -netdev user,id=net0,hostfwd=tcp::10022-:22 \ -m 2G,slots=8,maxmem=6G \ -smp cpus=4,cores=2,sockets=2 \ -object memory-backend-ram,size=1G,id=mem0 \ -numa node,nodeid=0,cpus=0-1,memdev=mem0 \ -object memory-backend-ram,size=1G,id=mem1 \ -numa node,nodeid=1,cpus=2-3,memdev=mem1 \ -object memory-backend-ram,id=vmem0,share=on,size=256M \ -object memory-backend-ram,id=vmem1,share=on,size=256M \ -device pxb-cxl,numa_node=0,bus_nr=12,bus=pcie.0,id=cxl.1 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ -device cxl-type3,bus=root_port13,volatile-memdev=vmem0,id=cxl-vmem0 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port14,chassis=1,slot=2 \ -device cxl-type3,bus=root_port14,volatile-memdev=vmem1,id=cxl-vmem1 \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G ``` -------------------------------- ### Read Specific Mapping Details Source: https://github.com/open-mpi/hwloc/wiki/Simulating-complex-memory-with-Qemu Access individual mapping files (e.g., 'mapping0') to get details about the memory device, its offset, size, and flags. ```bash % cat /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0012:00/ndbus0/region2/mapping0 nmem0,0,1073741824,0 ``` -------------------------------- ### Configure QEMU NUMA SLIT Distances Source: https://github.com/open-mpi/hwloc/wiki/Simulating-complex-memory-with-Qemu Define the System Local Interconnect (SLIT) distances between NUMA nodes in QEMU using the '-numa dist' option. This example sets specific distances for nodes 0 through 4. ```bash -numa dist,src=0,dst=0,val=10 -numa dist,src=0,dst=1,val=20 -numa dist,src=0,dst=2,val=12 -numa dist,src=0,dst=3,val=22 -numa dist,src=0,dst=4,val=15 \ -numa dist,src=1,dst=0,val=20 -numa dist,src=1,dst=1,val=10 -numa dist,src=1,dst=2,val=22 -numa dist,src=1,dst=3,val=12 -numa dist,src=1,dst=4,val=25 \ -numa dist,src=2,dst=0,val=12 -numa dist,src=2,dst=1,val=22 -numa dist,src=2,dst=2,val=10 -numa dist,src=2,dst=3,val=25 -numa dist,src=2,dst=4,val=30 \ -numa dist,src=3,dst=0,val=22 -numa dist,src=3,dst=1,val=12 -numa dist,src=3,dst=2,val=25 -numa dist,src=3,dst=3,val=10 -numa dist,src=3,dst=4,val=30 \ -numa dist,src=4,dst=0,val=15 -numa dist,src=4,dst=1,val=25 -numa dist,src=4,dst=2,val=30 -numa dist,src=4,dst=3,val=30 -numa dist,src=4,dst=4,val=10 ``` -------------------------------- ### Configure VM with CXL Memory and Devices Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come Use this command to launch a QEMU virtual machine with CXL memory, PXB, and CXL root ports. Ensure bus_nr values are unique to avoid conflicts. ```bash qemu-system-x86_64 -hda img/snapshot.img -accel kvm \ -machine q35,nvdimm=on,cxl=on \ -device e1000,netdev=net0,mac=52:54:00:12:34:56 \ -netdev user,id=net0,hostfwd=tcp::10022-:22 \ -m 2G,slots=8,maxmem=10G \ -smp cpus=4,cores=2,sockets=2 \ -object memory-backend-ram,size=1G,id=ram0 \ -object memory-backend-ram,size=1G,id=ram1 \ -object memory-backend-ram,id=cxl-mem0,share=on,size=256M \ -object memory-backend-ram,id=cxl-mem1,share=on,size=256M \ -object memory-backend-ram,id=cxl-mem2,share=on,size=256M \ -object memory-backend-ram,id=cxl-mem3,share=on,size=256M \ -numa node,nodeid=0,cpus=0-1,memdev=ram0 \ -numa node,nodeid=1,cpus=2-3,memdev=ram1 \ -device pxb-cxl,numa_node=0,bus_nr=24,bus=pcie.0,id=pxb-cxl.1 \ -device pxb-cxl,numa_node=1,bus_nr=32,bus=pcie.0,id=pxb-cxl.2 \ -device cxl-rp,port=0,bus=pxb-cxl.1,id=root_port1,chassis=0,slot=0 \ -device cxl-rp,port=1,bus=pxb-cxl.1,id=root_port2,chassis=0,slot=1 \ -device cxl-rp,port=2,bus=pxb-cxl.2,id=root_port3,chassis=1,slot=0 \ -device cxl-rp,port=3,bus=pxb-cxl.2,id=root_port4,chassis=1,slot=2 \ -device cxl-upstream,bus=root_port1,id=us0 \ -device cxl-upstream,bus=root_port3,id=us1 \ -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=3 \ -device cxl-type3,bus=swport0,volatile-memdev=cxl-mem0,id=cxl-vmem0 \ -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=4 \ -device cxl-type3,bus=swport1,volatile-memdev=cxl-mem1,id=cxl-vmem1 \ -device cxl-downstream,port=2,bus=us1,id=swport2,chassis=1,slot=5 \ -device cxl-type3,bus=swport2,volatile-memdev=cxl-mem2,id=cxl-vmem2 \ -device cxl-downstream,port=3,bus=us1,id=swport3,chassis=1,slot=6 \ -device cxl-type3,bus=swport3,volatile-memdev=cxl-mem3,id=cxl-vmem3 \ -M cxl-fmw.0.targets.0=pxb-cxl.1,cxl-fmw.0.size=4G,cxl-fmw.1.targets.0=pxb-cxl.2,cxl-fmw.1.size=4G ``` -------------------------------- ### Run QEMU for Headless VM with NUMA and NVMe Configuration Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come This script launches a QEMU virtual machine with specific hardware configurations including NUMA nodes, memory backends, and NVMe devices, running in a headless mode. Press Ctrl+A+X to exit. ```bash #!/bin/bash echo "... Running qemu ..." qemu-system-x86_64 \ -hda /mnt/scratch/pclouzet/qemu/img/new_clean.img \ -device e1000,netdev=net0,mac=52:54:00:12:34:56 \ -netdev user,id=net0,hostfwd=tcp::10022-:22 \ -accel kvm \ -machine q35,nvdimm=on,cxl=off \ -m size=8G,slots=1,maxmem=9G \ -smp cpus=16 \ -object memory-backend-ram,size=4G,id=ram0 \ -object memory-backend-ram,size=2G,id=ram1 \ -object memory-backend-ram,size=1G,id=ram2 \ -object memory-backend-ram,size=1G,id=ram3 \ -numa node,nodeid=0,memdev=ram0,cpus=0-3 \ -numa node,nodeid=1,memdev=ram1,cpus=4-7 \ -numa node,nodeid=2,memdev=ram2,cpus=8-11 \ -numa node,nodeid=3,memdev=ram3,cpus=12-15 \ -object memory-backend-ram,size=1G,id=nvdimm0,share=on \ -device nvdimm,id=nvdimm0,memdev=nvdimm0,unarmed=off,node=4 \ -numa node,nodeid=4 \ -nographic ``` -------------------------------- ### Create QEMU Disk Image Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come Create a new disk image file in qcow2 format named 'mydisk.img' with a size of 20GB. This image will store the virtual machine's operating system and data. ```shell qemu-img create -f qcow2 mydisk.img 20G ``` -------------------------------- ### Configure and Build Static Library and Programs Source: https://github.com/open-mpi/hwloc/wiki/StaticBuild Use these commands to configure hwloc for a static build and then build fully static programs. Ensure LDFLAGS are set correctly for both steps. ```sh $ ./configure --enable-static --disable-shared LDFLAGS="-static" $ make LDFLAGS=-all-static ``` ```sh $ ./configure --enable-static --disable-shared LDFLAGS="--static" ``` -------------------------------- ### Configure Qemu with 2 NUMAs + 2 CPU-less NUMA Source: https://github.com/open-mpi/hwloc/wiki/Simulating-complex-memory-with-Qemu Use this command to set up Qemu with multiple NUMA nodes, including CPU-less nodes, for complex memory simulations. Requires OVMF for UEFI booting. ```bash kvm \ -drive if=pflash,format=raw,file=./OVMF.fd \ -drive media=disk,format=qcow2,file=efi.qcow2 \ -smp 4 -m 6G \ -object memory-backend-ram,size=3G,id=m0 \ -object memory-backend-ram,size=1G,id=m1 \ -object memory-backend-ram,size=1G,id=m2 \ -object memory-backend-ram,size=1G,id=m3 \ -numa node,nodeid=0,memdev=m0,cpus=0-1 \ -numa node,nodeid=1,memdev=m1,cpus=2-3 \ -numa node,nodeid=2,memdev=m2 \ -numa node,nodeid=3,memdev=m3 ``` -------------------------------- ### Configure Qemu with NVDIMMs Source: https://github.com/open-mpi/hwloc/wiki/Simulating-complex-memory-with-Qemu Set up Qemu to include NVDIMMs by enabling the 'nvdimm' machine option and configuring memory slots and backend file objects. Ensure 'maxmem' in '-m' accounts for both RAM and NVDIMM size. ```bash kvm \ -machine pc,nvdimm=on \ -drive if=pflash,format=raw,file=./OVMF.fd \ -drive media=disk,format=qcow2,file=efi.qcow2 \ -smp 4 \ -m 6G,slots=1,maxmem=7G \ -object memory-backend-ram,size=3G,id=ram0 \ -object memory-backend-ram,size=1G,id=ram1 \ -object memory-backend-ram,size=1G,id=ram2 \ -object memory-backend-ram,size=1G,id=ram3 \ -numa node,nodeid=0,memdev=ram0,cpus=0-1 \ -numa node,nodeid=1,memdev=ram1,cpus=2-3 \ -numa node,nodeid=2,memdev=ram2 \ -numa node,nodeid=3,memdev=ram3 \ -numa node,nodeid=4 \ -object memory-backend-file,id=nvdimm1,share=on,mem-path=nvdimm.img,size=1G \ -device nvdimm,id=nvdimm1,memdev=nvdimm1,unarmed=off,node=4 ``` -------------------------------- ### Create QEMU Snapshot Image Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come Create a snapshot of the 'mydisk.img' file. The new snapshot image 'snapshot.img' will be a copy-on-write image based on 'mydisk.img'. It is recommended not to modify the original 'mydisk.img' after creating a snapshot to prevent corruption. ```shell qemu-img create -f qcow2 -b mydisk.img -F qcow2 snapshot.img ``` -------------------------------- ### Reconfigure DAX Device as System RAM Source: https://github.com/open-mpi/hwloc/wiki/Simulating-complex-memory-with-Qemu Use 'daxctl reconfigure-device' to change the mode of a DAX device, for example, to 'system-ram', making it usable as regular memory. ```bash % daxctl reconfigure-device --mode=system-ram dax2.0 ``` -------------------------------- ### Automate VM Tasks with rc.local Script Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come This shell script runs as root on VM startup to download, build, and test hwloc, then copies logs and shuts down the VM. Ensure the host IP and login are correctly set. ```shell #!/bin/sh # Guest and Host share the same login. login=pclouzet ip_adress_of_host=192.168.134.40 set -e cd /home/$login rm -rf archive* # Get hwloc, unzip wget https://ci.inria.fr/hwloc/job/basic/job/master/lastSuccessfulBuild/artifact/*zip*/archive.zip unzip archive.zip cd archive filename=$(ls hwloc*.tar.bz2) tar xf $filename hwloc_dir="${filename%.tar.bz2}" cd $hwloc_dir # Manual Install ./configure >> log.txt make >> log.txt # Test ./utils/lstopo/lstopo-no-graphics >> log.txt # Copy log file and send it back to host cp log.txt /home/$login chown $login /home/$login/log.txt scp /home/$login/log.txt pclouzet@$ip_adress_of_host:/mnt/scratch/$login/qemu/scripts/hwloc_test # Shutdown guest halt ``` -------------------------------- ### Configure CXL Persistent Memory in QEMU Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come Launch a QEMU VM with CXL enabled, setting up a CXL hostbridge, root-port, and a Type 3 persistent memory device. This requires memory backends for both the persistent memory and its Label Storage Area (LSA), and a Fixed Memory Window (FMW). ```bash qemu-system-x86_64 -hda img/snapshot.img -accel kvm \ -machine q35,nvdimm=on,cxl=on \ -device e1000,netdev=net0,mac=52:54:00:12:34:56 \ -netdev user,id=net0,hostfwd=tcp::10022-:22 \ -m 4G,slots=8,maxmem=8G \ -smp 4 \ -object memory-backend-ram,size=4G,id=mem0 \ -numa node,nodeid=0,cpus=0-3,memdev=mem0 \ -object memory-backend-file,id=pmem0,share=on,mem-path=/tmp/cxltest.raw,size=256M \ -object memory-backend-file,id=cxl-lsa0,share=on,mem-path=/tmp/lsa.raw,size=256M \ -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \ -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \ -device cxl-type3,bus=root_port13,persistent-memdev=pmem0,lsa=cxl-lsa0,id=cxl-pmem0 \ -M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G ``` -------------------------------- ### Add NVDIMM Node to QEMU VM Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come Use this command to launch a QEMU VM with NVDIMM support enabled, specifying memory backends and NUMA node configurations. Ensure the image paths for NVDIMM and LSA are correct. ```bash qemu-system-x86_64 -hda img/snapshot.img -accel kvm \ -device e1000,netdev=net0,mac=52:54:00:12:34:56 -netdev user,id=net0,hostfwd=tcp::10022-:22 \ -machine pc,nvdimm=on \ -m 8G,slots=1,maxmem=9G \ -smp cpus=16 \ -object memory-backend-ram,size=4G,id=ram0 \ -object memory-backend-ram,size=2G,id=ram1 \ -object memory-backend-ram,size=1G,id=ram2 \ -object memory-backend-ram,size=1G,id=ram3 \ -device nvdimm,id=nvdimm1,memdev=nvdimm1,unarmed=off,node=4 \ -object memory-backend-file,id=nvdimm1,share=on,mem-path=img/nvdimm.img,size=1G \ -numa node,nodeid=0,memdev=ram0,cpus=0-3 \ -numa node,nodeid=1,memdev=ram1,cpus=4-7 \ -numa node,nodeid=2,memdev=ram2,cpus=8-11 \ -numa node,nodeid=3,memdev=ram3,cpus=12-15 \ -numa node,nodeid=4 ``` -------------------------------- ### List CXL Memory Devices in VM Source: https://github.com/open-mpi/hwloc/wiki/The-shape-of-VM-to-come Use this command inside the VM to list available CXL memory devices and their properties. ```json [ { "memdev":"mem1", "ram_size":268435456, "serial":0, "numa_node":1, "host":"0000:23:00.0" }, { "memdev":"mem0", "ram_size":268435456, "serial":0, "numa_node":1, "host":"0000:24:00.0" }, { "memdev":"mem2", "ram_size":268435456, "serial":0, "numa_node":0, "host":"0000:1b:00.0" }, { "memdev":"mem3", "ram_size":268435456, "serial":0, "numa_node":0, "host":"0000:1c:00.0" } ] ``` -------------------------------- ### Set Source File and Include Directories Source: https://github.com/open-mpi/hwloc/blob/master/contrib/android/AndroidApp/lstopo/src/main/cpp/CMakeLists.txt Configures the build by setting a source file and including header directories. Ensure these paths are correct relative to your project structure. ```cmake set(../../../../../../../hwloc/topology.c) ``` ```cmake include_directories(../../../../../include) ``` ```cmake include_directories(../../../../../../../include) ``` ```cmake include_directories(../../../../../../../utils/hwloc) ```