Intel 82599

Intel 82599-based X520 card, with two transceivers installed

This page or section is a work in progress and may thus be incomplete. Its content may be changed in the near future.

The Intel 82599 Ethernet controller is a 10 gigabit ethernet controller chipset, notably used on the X520 family of NICs. These NICs are end of life, but for this reason can be acquired on a very tight budget, in comparison to newer models like the X540 or X550. Unlike similar budgets cards the author investigated, documentation is available.

Hardware

Unlike most gigabit products you might encounter, 10 GbE products often have a choice of PHY. This is possible on gigabit using SFP, but far more 10 GbE cards have SFP+ ports. These take a PHY transceiver, rendering the card medium independent.

RJ45 connectors for these are often known to run hot. Other options for PHY interface are Fibre Optics, for which a plethora of options exist (single mode, multimode, wavelength, range...). A common modern choice might be 10GBASE-LR over OS2 fibre, which has a range of up to 10 km. Finally, the simplest is DAC, Direct Attach Copper, which works well for short runs, in theory up to 15 metres. This is especially suited to in-rack connections.

The 82599 controller is a PCIe 2.0 x8 device, so has a maximum throughput of 4 GB/s. This isn't actually quite enough to saturate the 2 PHYs it supports, full duplex, aggregating 40 Gbps.

Virtualization

This chipset is designed to work with SR-IOV. See also IOMMU.

This means that there is one Physical Function (PF), and a number of Virtual Functions (VFs). Largely, virtual functions mirror the PF, with some differences.

Operation Modes

The controller has an double-VLAN capability where it expects at least one (outer) VLAN identifier on received packets.

Operation

Initialization

The 82599 is initialised with the following sequence:

1. Disable Interrupts - this can be achieved through PCI by turning off enable on MSI and MSI-X, and also the legacy pin interrupt enable.
2. Issue a global reset by writing RST in the CTRL register of the PF.
3. Wait at least 10 ms
4. [Optional] Enable flow control
5. Wait for EEPROM auto-read to complete
6. Wait for DMA initialization, RDRXCTL.DMAIDONE.
7. Set Up the PHY and Link
   • This step depends on the PHY used.
8. Enable counters
9. Initialize Receive
10. Initialize Transmit
11. Enable interrupts

PHY Initialization

As mentioned before, the X520 uses SFP+ pluggable modules. However, SFP+ itself uses an interface call XGMII, but is backwards compatible with Gigabit SFP using SGMII, and 100 MBit SFP using MII.

There are then further differences based on the physical medium, but with so much commonailty too, the result is two initialization modes:

SGMII

This supports BX PHYs, which is a fibre standard for 100 MBit ethernet, and SGMII, Serial Gigabit Media Independent Interface.

So, this is the 1/0.1 gigabit mode.

1. Electrical link setup. The spec indicates that this is handled by the EEPROM.
2. Configure AUTOC.LMS to select the link mode.
3. Configure SERDESC register as necessary.
4. Verify correct status (link up and speed) in LINKS.

XAUI

XAUI, or 10 Gigabit Attatchment Unit Interface, is an extension of XGMII. This mode also supports BX4, CX4, KX, KX4, KR, and SFI backplane standards.

• Electrical link setup. The spec indicates that this is handled by the EEPROM.
• Configure AUTOC.LMS to select the link mode, set 10G_PMA_PMD_PARALLEL and AUTOC2.10G_PMA_PMD_Serial to the appropriate settings.
• Configure SERDESC register as necessary.
  • If using KX/KX4/KR mode with autonegotiation, configure the KX_Support field of AUTOC.
• Restart the link by restarting autonegotiation using AUTOC.
• Verify correct status (link up and speed) in LINKS.

Statistics

Statistics registers (bytes/packets Rx/Tx through specified queues) can be initialized here.

Interrupt Initialization

Initializing the interrupts is similar between legacy/MSI and MSI-X, with MSI-X being the most modern system.

• Interrupt causes are mapped to vectors by programming the IVAR[n] registers.
• IVAR_MISC supports a couple of additional sources in MSI-X mode.
• Program SRRCTL[n].RDMTS (Split Receive control) for the minimum threshold interrupt.
• Set the EIAC[n] register for interrupt auto-clear Tx and Rx (MSI-X only). This is a performance optimization.
• Set the auto-mask in EIAM and EIAM[n].
• Set interrupt throttling in EITR[n] and GPIE.
• Enable interrupts through EIMS[n]

Receive Initialization

The 82599 supports 128 Rx and Tx queues.

Registers

The 82599s registers are accessed through MMIO at PCI BAR0. An alternative I/O space access mechanism optionally exists, through an ADDR/DATA pair at BAR2.

CTRL

MMIO+0x0000, mirrored at +0x0004 for compatibility.

Register

31:27	26	25:4	3	2	1:0
RSVD	RST	RSVD	LRST	PEMD	RSVD

Detail

Field	Description
RST	Device Reset
LRST	Link Reset
PEMD	PCIe Master Disable

STATUS

MMIO+0x0008

Register

31:20	19	18	17:10	9:8	7	6:4	3:2	1:0
RSVD	PMS	IOVA	nVFs	RSVD	LU	RSVD	LID	RSVD

Detail

Field	Description
PMS	PCIe Master Enable Status
IOVA	IOV Active, reflects VFE in IOV capability
nVFs	Number of Virtual Functions
LU	Link Up
LID	LAN ID

CTRL_EXT

MMIO+0x0018

Register

31:29	28	27	26	25:18	17	16	15	14	13:0
RSVD	DRV_LOAD	RSVD	xVLAN	RSVD	RO_DIS	NS_DIS	RSVD	PFRSTD	RSVD

Detail

Field	Description
DRV_LOAD	Driver Loaded
xVLAN	Extended VLAN - expects tagged packets
RO_DIS	Relaxed Ordering DIsable
NS_DIS	No Snoop Disable
PFRSTD	PF Reset Done

EXVET

MMIO+0x5078

Extended VLAN Ether Type.

Register

31:16	15:0
VET EXT	RSVD

VET EXT - Outer-VLAN EtherType. This is the VLAN tag protocol identifier. Default value is 0x8100 for 802.1Q.

AUTOC

MMIO+0x42A0

Autonegotiation control register

AUTOC

16	15-13	12	11	10	9	8-7	6-2	1	0
KRS	LMS	RAN	RATD	D10GMP	1GPP	10GPPP	ANSF	ANACK2	FLU

References

The full manual can be found here - Intel® 82599 10 Gigabit Ethernet Controller: Datasheet