nvme
— NVM Express
core driver
To compile this driver into your kernel, place the following line
in your kernel configuration file:
device nvme
Or, to load the driver as a module at boot, place the following
line in
loader.conf(5):
Most users will also want to enable
nvd(4) or
nda(4) to
expose NVM Express namespaces as disk devices which can be partitioned. Note
that in NVM Express terms, a namespace is roughly equivalent to a SCSI
LUN.
The nvme
driver provides support for NVM
Express (NVMe) controllers, such as:
- Hardware initialization
- Per-CPU IO queue pairs
- API for registering NVMe namespace consumers such as
nvd(4)
or
nda(4)
- API for submitting NVM commands to namespaces
- Ioctls for controller and namespace configuration and management
The nvme
driver creates controller device
nodes in the format /dev/nvmeX and namespace device
nodes in the format /dev/nvmeXnsY. Note that the NVM
Express specification starts numbering namespaces at 1, not 0, and this
driver follows that convention.
By default, nvme
will create an I/O queue
pair for each CPU, provided enough MSI-X vectors and NVMe queue pairs can be
allocated. If not enough vectors or queue pairs are available, nvme(4) will
use a smaller number of queue pairs and assign multiple CPUs per queue
pair.
To force a single I/O queue pair shared by all CPUs, set the
following tunable value in
loader.conf(5):
hw.nvme.per_cpu_io_queues=0
To assign more than one CPU per I/O queue pair, thereby reducing
the number of MSI-X vectors consumed by the device, set the following
tunable value in
loader.conf(5):
hw.nvme.min_cpus_per_ioq=X
To force legacy interrupts for all nvme
driver instances, set the following tunable value in
loader.conf(5):
Note that use of INTx implies disabling of per-CPU I/O queue
pairs.
To control maximum amount of system RAM in bytes to use as Host
Memory Buffer for capable devices, set the following tunable:
The default value is 5% of physical memory size per device.
The
nvd(4)
driver is used to provide a disk driver to the system by default. The
nda(4)
driver can also be used instead. The
nvd(4)
driver performs better with smaller transactions and few TRIM commands. It
sends all commands directly to the drive immediately. The
nda(4)
driver performs better with larger transactions and also collapses TRIM
commands giving better performance. It can queue commands to the drive;
combine BIO_DELETE
commands into a single trip; and
use the CAM I/O scheduler to bias one type of operation over another. To
select the
nda(4)
driver, set the following tunable value in
loader.conf(5):
This value may also be set in the kernel config file with
When there is an error, nvme
prints only
the most relevant information about the command by default. To enable
dumping of all information about the command, set the following tunable
value in
loader.conf(5):
hw.nvme.verbose_cmd_dump=1
Prior versions of the driver reset the card twice on boot. This
proved to be unnecessary and inefficient, so the driver now resets drive
controller only once. The old behavior may be restored in the kernel config
file with
The following controller-level sysctls are currently
implemented:
- dev.nvme.0.num_cpus_per_ioq
- (R) Number of CPUs associated with each I/O queue pair.
- dev.nvme.0.int_coal_time
- (R/W) Interrupt coalescing timer period in microseconds. Set to 0 to
disable.
- dev.nvme.0.int_coal_threshold
- (R/W) Interrupt coalescing threshold in number of command completions. Set
to 0 to disable.
The following queue pair-level sysctls are currently implemented.
Admin queue sysctls take the format of dev.nvme.0.adminq and I/O queue
sysctls take the format of dev.nvme.0.ioq0.
- dev.nvme.0.ioq0.num_entries
- (R) Number of entries in this queue pair's command and completion
queue.
- dev.nvme.0.ioq0.num_tr
- (R) Number of nvme_tracker structures currently allocated for this queue
pair.
- dev.nvme.0.ioq0.num_prp_list
- (R) Number of nvme_prp_list structures currently allocated for this queue
pair.
- dev.nvme.0.ioq0.sq_head
- (R) Current location of the submission queue head pointer as observed by
the driver. The head pointer is incremented by the controller as it takes
commands off of the submission queue.
- dev.nvme.0.ioq0.sq_tail
- (R) Current location of the submission queue tail pointer as observed by
the driver. The driver increments the tail pointer after writing a command
into the submission queue to signal that a new command is ready to be
processed.
- dev.nvme.0.ioq0.cq_head
- (R) Current location of the completion queue head pointer as observed by
the driver. The driver increments the head pointer after finishing with a
completion entry that was posted by the controller.
- dev.nvme.0.ioq0.num_cmds
- (R) Number of commands that have been submitted on this queue pair.
- dev.nvme.0.ioq0.dump_debug
- (W) Writing 1 to this sysctl will dump the full contents of the submission
and completion queues to the console.
In addition to the typical pci attachment, the
nvme
driver supports attaching to a
ahci(4)
device. Intel's Rapid Storage Technology (RST) hides the nvme device behind
the AHCI device due to limitations in Windows. However, this effectively
hides it from the FreeBSD kernel. To work around
this limitation, FreeBSD detects that the AHCI
device supports RST and when it is enabled. See
ahci(4)
for more details.
The nvme
driver first appeared in
FreeBSD 9.2.