znvmewipeseals/znvme
A userspace NVMe driver written in Zig.
Dependencies
cc5c6a5d71a317ed4b0ad776842d1d0655f72d0aREADME
znvme
A userspace NVMe driver implemented in Zig, leveraging VFIO for direct hardware access.
Features
- Direct access to NVMe device registers and features from user space.
- Secure operation without sudo by using VFIO.
- Easy integration with Zig applications.
Requirements
- Zig 0.14.1 or later
- Linux kernel with
vfio-pcisupport- IOMMU enabled in the kernel (Intel VT-d or AMD-Vi)
- NVMe devices connected to the system
Installation
To use this library, you need to clone the repository and build the Zig package.
git clone https://github.com/wipeseals/znvme.git
cd znvme
zig build test
Usage
Basic Example
TODO
Advanced Example
TODO
IOMMU Setup
Ensure the kernel command line includes intel_iommu=on or amd_iommu=on depending on your CPU architecture.
cat /proc/cmdline
sudo dmesg | grep -e DMAR -e IOMMU
user in 🌐 nbg9 in znvme on develop [$✘?] via ↯ v0.14.1 via ❄️ impure (nix-shell-env)
❯ sudo dmesg | grep -e DMAR -e IOMMU
(snip)
[ 0.005995] ACPI: DMAR 0x00000000725CE000 000088 (v02 INTEL EDK2 00000002 01000013)
[ 0.006037] ACPI: Reserving DMAR table memory at [mem 0xXXXXXXXX-0xXXXXXXXX]
[ 0.027230] DMAR: IOMMU enabled
[ 0.079635] DMAR: Host address width 39
For Intel CPUs
Add intel_iommu=on to the kernel command line in your bootloader configuration (e.g., GRUB):
sudo vim /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on"
Then update GRUB
sudo update-grub
VFIO Setup
To access NVMe devices from user space, you need to switch the kernel driver from nvme to vfio-pci.
Driver switching scripts are available in the misc/ directory.
Listing NVMe devices
First, check the NVMe devices connected to your system.
The following command lists devices currently bound to the nvme driver:
user in 🌐 nbg9 in znvme on master [!?] via ↯ v0.14.1 via ❄️ impure (nix-shell-env)
❯ sudo ./misc/list.sh
# Listing NVMe devices
Node Generic SN Model Namespace Usage Format FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1 /dev/ng0n1 Z4HF716NF88S KIOXIA-EXCERIA with Heatsink SSD 0x1 1.02 TB / 1.02 TB 512 B + 0 B AJRA4101
/dev/nvme1n1 /dev/ng1n1 201008800819 WDC WD BLACK SDBPNTY-512G-1106 0x1 512.11 GB / 512.11 GB 512 B + 0 B HPS2
/dev/nvme2n1 /dev/ng2n1 TTSMA2513X09608 TWSC TSC3AN512-F8T40S 0x1 512.11 GB / 512.11 GB 512 B + 0 B SN13126
# Listing NVMe devices with PCI addresses, VID, PID, and manufacturer/model info
PCI Address: 01:00.0, VID: 15b7, PID: 5006, Info: Sandisk Corp SanDisk Extreme Pro / WD Black SN750 / PC SN730 / Red SN700 NVMe SSD
PCI Address: 06:00.0, VID: 1e0f, PID: 000d, Info: KIOXIA Corporation NVMe SSD Controller XG7
PCI Address: 07:00.0, VID: 1e4b, PID: 1202, Info: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1202 (DRAM-less) (rev 01)
You can also check each device's PCI address, vendor ID, and product ID.
This information is required for driver switching.
Binding to the vfio-pci driver
Unbind the NVMe device from the nvme driver and bind it to the vfio-pci driver.
Example for switching the device at PCI address 0000:06:00.0:
user in 🌐 nbg9 in znvme on master [!?] via ↯ v0.14.1 via ❄️ impure (nix-shell-env)
❯ sudo ./misc/bind_vfio.sh 0000:06:00.0 1e0f 000d
Unbound 0000:06:00.0 from nvme driver.
grep: /sys/bus/pci/drivers/vfio-pci/new_id: Permission denied
Bound 0000:06:00.0 to vfio-pci driver.
Bound 0000:06:00.0 to vfio-pci driver.
Successfully bound 0000:06:00.0 to vfio-pci driver.
06:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD Controller XG7
Subsystem: KIOXIA Corporation NVMe SSD Controller XG7
Kernel driver in use: vfio-pci
Kernel modules: nvme
If you see Permission denied for /sys/bus/pci/drivers/vfio-pci/new_id, make sure you are running as root.
After binding, confirm Kernel driver in use: vfio-pci.
Unbinding from the vfio-pci driver and rebinding to the nvme driver
After your work, you can return the device to the original nvme driver:
user in 🌐 nbg9 in znvme on master [!?] via ↯ v0.14.1 via ❄️ impure (nix-shell-env)
❯ sudo ./misc/unbind_vfio.sh 0000:06:00.0
Unbound 0000:06:00.0 from vfio-pci driver.
Bound 0000:06:00.0 to nvme driver.
Successfully bound 0000:06:00.0 to nvme driver.
06:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD Controller XG7
Subsystem: KIOXIA Corporation NVMe SSD Controller XG7
Kernel driver in use: nvme
Kernel modules: nvme
References
License
This project is licensed under the MIT License. See the LICENSE file for details.