| The MSI Driver Guide HOWTO
| Tom L Nguyen firstname.lastname@example.org
| Revised Feb 12, 2004 by Martine Silbermann
| email: Martine.Silbermann@hp.com
| Revised Jun 25, 2004 by Tom L Nguyen
| Revised Jul 9, 2008 by Matthew Wilcox <email@example.com>
| Copyright 2003, 2008 Intel Corporation
|1. About this guide
|This guide describes the basics of Message Signaled Interrupts (MSIs),
|the advantages of using MSI over traditional interrupt mechanisms, how
|to change your driver to use MSI or MSI-X and some basic diagnostics to
|try if a device doesn't support MSIs.
|2. What are MSIs?
|A Message Signaled Interrupt is a write from the device to a special
|address which causes an interrupt to be received by the CPU.
|The MSI capability was first specified in PCI 2.2 and was later enhanced
|in PCI 3.0 to allow each interrupt to be masked individually. The MSI-X
|capability was also introduced with PCI 3.0. It supports more interrupts
|per device than MSI and allows interrupts to be independently configured.
|Devices may support both MSI and MSI-X, but only one can be enabled at
|3. Why use MSIs?
|There are three reasons why using MSIs can give an advantage over
|traditional pin-based interrupts.
|Pin-based PCI interrupts are often shared amongst several devices.
|To support this, the kernel must call each interrupt handler associated
|with an interrupt, which leads to reduced performance for the system as
|a whole. MSIs are never shared, so this problem cannot arise.
|When a device writes data to memory, then raises a pin-based interrupt,
|it is possible that the interrupt may arrive before all the data has
|arrived in memory (this becomes more likely with devices behind PCI-PCI
|bridges). In order to ensure that all the data has arrived in memory,
|the interrupt handler must read a register on the device which raised
|the interrupt. PCI transaction ordering rules require that all the data
|arrive in memory before the value may be returned from the register.
|Using MSIs avoids this problem as the interrupt-generating write cannot
|pass the data writes, so by the time the interrupt is raised, the driver
|knows that all the data has arrived in memory.
|PCI devices can only support a single pin-based interrupt per function.
|Often drivers have to query the device to find out what event has
|occurred, slowing down interrupt handling for the common case. With
|MSIs, a device can support more interrupts, allowing each interrupt
|to be specialised to a different purpose. One possible design gives
|infrequent conditions (such as errors) their own interrupt which allows
|the driver to handle the normal interrupt handling path more efficiently.
|Other possible designs include giving one interrupt to each packet queue
|in a network card or each port in a storage controller.
|4. How to use MSIs
|PCI devices are initialised to use pin-based interrupts. The device
|driver has to set up the device to use MSI or MSI-X. Not all machines
|support MSIs correctly, and for those machines, the APIs described below
|will simply fail and the device will continue to use pin-based interrupts.
|4.1 Include kernel support for MSIs
|To support MSI or MSI-X, the kernel must be built with the CONFIG_PCI_MSI
|option enabled. This option is only available on some architectures,
|and it may depend on some other options also being set. For example,
|on x86, you must also enable X86_UP_APIC or SMP in order to see the
|4.2 Using MSI
|Most of the hard work is done for the driver in the PCI layer. The driver
|simply has to request that the PCI layer set up the MSI capability for this
|To automatically use MSI or MSI-X interrupt vectors, use the following
| int pci_alloc_irq_vectors(struct pci_dev *dev, unsigned int min_vecs,
| unsigned int max_vecs, unsigned int flags);
|which allocates up to max_vecs interrupt vectors for a PCI device. It
|returns the number of vectors allocated or a negative error. If the device
|has a requirements for a minimum number of vectors the driver can pass a
|min_vecs argument set to this limit, and the PCI core will return -ENOSPC
|if it can't meet the minimum number of vectors.
|The flags argument is used to specify which type of interrupt can be used
|by the device and the driver (PCI_IRQ_LEGACY, PCI_IRQ_MSI, PCI_IRQ_MSIX).
|A convenient short-hand (PCI_IRQ_ALL_TYPES) is also available to ask for
|any possible kind of interrupt. If the PCI_IRQ_AFFINITY flag is set,
|pci_alloc_irq_vectors() will spread the interrupts around the available CPUs.
|To get the Linux IRQ numbers passed to request_irq() and free_irq() and the
|vectors, use the following function:
| int pci_irq_vector(struct pci_dev *dev, unsigned int nr);
|Any allocated resources should be freed before removing the device using
|the following function:
| void pci_free_irq_vectors(struct pci_dev *dev);
|If a device supports both MSI-X and MSI capabilities, this API will use the
|MSI-X facilities in preference to the MSI facilities. MSI-X supports any
|number of interrupts between 1 and 2048. In contrast, MSI is restricted to
|a maximum of 32 interrupts (and must be a power of two). In addition, the
|MSI interrupt vectors must be allocated consecutively, so the system might
|not be able to allocate as many vectors for MSI as it could for MSI-X. On
|some platforms, MSI interrupts must all be targeted at the same set of CPUs
|whereas MSI-X interrupts can all be targeted at different CPUs.
|If a device supports neither MSI-X or MSI it will fall back to a single
|legacy IRQ vector.
|The typical usage of MSI or MSI-X interrupts is to allocate as many vectors
|as possible, likely up to the limit supported by the device. If nvec is
|larger than the number supported by the device it will automatically be
|capped to the supported limit, so there is no need to query the number of
|vectors supported beforehand:
| nvec = pci_alloc_irq_vectors(pdev, 1, nvec, PCI_IRQ_ALL_TYPES)
| if (nvec < 0)
| goto out_err;
|If a driver is unable or unwilling to deal with a variable number of MSI
|interrupts it can request a particular number of interrupts by passing that
|number to pci_alloc_irq_vectors() function as both 'min_vecs' and
| ret = pci_alloc_irq_vectors(pdev, nvec, nvec, PCI_IRQ_ALL_TYPES);
| if (ret < 0)
| goto out_err;
|The most notorious example of the request type described above is enabling
|the single MSI mode for a device. It could be done by passing two 1s as
|'min_vecs' and 'max_vecs':
| ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_ALL_TYPES);
| if (ret < 0)
| goto out_err;
|Some devices might not support using legacy line interrupts, in which case
|the driver can specify that only MSI or MSI-X is acceptable:
| nvec = pci_alloc_irq_vectors(pdev, 1, nvec, PCI_IRQ_MSI | PCI_IRQ_MSIX);
| if (nvec < 0)
| goto out_err;
|4.3 Legacy APIs
|The following old APIs to enable and disable MSI or MSI-X interrupts should
|not be used in new code:
| pci_enable_msi() /* deprecated */
| pci_disable_msi() /* deprecated */
| pci_enable_msix_range() /* deprecated */
| pci_enable_msix_exact() /* deprecated */
| pci_disable_msix() /* deprecated */
|Additionally there are APIs to provide the number of supported MSI or MSI-X
|vectors: pci_msi_vec_count() and pci_msix_vec_count(). In general these
|should be avoided in favor of letting pci_alloc_irq_vectors() cap the
|number of vectors. If you have a legitimate special use case for the count
|of vectors we might have to revisit that decision and add a
|pci_nr_irq_vectors() helper that handles MSI and MSI-X transparently.
|4.4 Considerations when using MSIs
|Most device drivers have a per-device spinlock which is taken in the
|interrupt handler. With pin-based interrupts or a single MSI, it is not
|necessary to disable interrupts (Linux guarantees the same interrupt will
|not be re-entered). If a device uses multiple interrupts, the driver
|must disable interrupts while the lock is held. If the device sends
|a different interrupt, the driver will deadlock trying to recursively
|acquire the spinlock. Such deadlocks can be avoided by using
|spin_lock_irqsave() or spin_lock_irq() which disable local interrupts
|and acquire the lock (see Documentation/kernel-hacking/locking.rst).
|4.5 How to tell whether MSI/MSI-X is enabled on a device
|Using 'lspci -v' (as root) may show some devices with "MSI", "Message
|Signalled Interrupts" or "MSI-X" capabilities. Each of these capabilities
|has an 'Enable' flag which is followed with either "+" (enabled)
|or "-" (disabled).
|5. MSI quirks
|Several PCI chipsets or devices are known not to support MSIs.
|The PCI stack provides three ways to disable MSIs:
|2. on all devices behind a specific bridge
|3. on a single device
|5.1. Disabling MSIs globally
|Some host chipsets simply don't support MSIs properly. If we're
|lucky, the manufacturer knows this and has indicated it in the ACPI
|FADT table. In this case, Linux automatically disables MSIs.
|Some boards don't include this information in the table and so we have
|to detect them ourselves. The complete list of these is found near the
|quirk_disable_all_msi() function in drivers/pci/quirks.c.
|If you have a board which has problems with MSIs, you can pass pci=nomsi
|on the kernel command line to disable MSIs on all devices. It would be
|in your best interests to report the problem to firstname.lastname@example.org
|including a full 'lspci -v' so we can add the quirks to the kernel.
|5.2. Disabling MSIs below a bridge
|Some PCI bridges are not able to route MSIs between busses properly.
|In this case, MSIs must be disabled on all devices behind the bridge.
|Some bridges allow you to enable MSIs by changing some bits in their
|PCI configuration space (especially the Hypertransport chipsets such
|as the nVidia nForce and Serverworks HT2000). As with host chipsets,
|Linux mostly knows about them and automatically enables MSIs if it can.
|If you have a bridge unknown to Linux, you can enable
|MSIs in configuration space using whatever method you know works, then
|enable MSIs on that bridge by doing:
| echo 1 > /sys/bus/pci/devices/$bridge/msi_bus
|where $bridge is the PCI address of the bridge you've enabled (eg
|To disable MSIs, echo 0 instead of 1. Changing this value should be
|done with caution as it could break interrupt handling for all devices
|below this bridge.
|Again, please notify email@example.com of any bridges that need
|5.3. Disabling MSIs on a single device
|Some devices are known to have faulty MSI implementations. Usually this
|is handled in the individual device driver, but occasionally it's necessary
|to handle this with a quirk. Some drivers have an option to disable use
|of MSI. While this is a convenient workaround for the driver author,
|it is not good practice, and should not be emulated.
|5.4. Finding why MSIs are disabled on a device
|From the above three sections, you can see that there are many reasons
|why MSIs may not be enabled for a given device. Your first step should
|be to examine your dmesg carefully to determine whether MSIs are enabled
|for your machine. You should also check your .config to be sure you
|have enabled CONFIG_PCI_MSI.
|Then, 'lspci -t' gives the list of bridges above a device. Reading
|/sys/bus/pci/devices/*/msi_bus will tell you whether MSIs are enabled (1)
|or disabled (0). If 0 is found in any of the msi_bus files belonging
|to bridges between the PCI root and the device, MSIs are disabled.
|It is also worth checking the device driver to see whether it supports MSIs.
|For example, it may contain calls to pci_irq_alloc_vectors() with the
|PCI_IRQ_MSI or PCI_IRQ_MSIX flags.