Accelerating Internet growth has challenged operators of cloud services facilities and web-scale data centers to search for more cost-efficient servers — no easy task given intensive design optimizations with existing technologies over the past few years. A new design door has to open to achieve significant cost-per-bit improvements and industry interest has zeroed in on the server-to-switch interface.
In particular, more cost-effective designs can be realized with subsystem components – boards, connectors, cables, tools and switch ASICs – capable of handling single-lane 25 Gb/s Ethernet (25GbE) traffic. Fortunately, ongoing efforts by participants in the IEEE 802.3 working group during the past eight years has put a complete 25GbE solution within easy reach in a relatively short time.
The primary reason for the almost immediate availability of 25GbE technologies is that 100GbE uses four 25 Gb/s lanes to achieve its data rate. This makes it highly likely that certain connectors, switch ASICs and other technologies will be appropriate for implementations of the proposed standard, which envisions a single 25 Gb/s lane. From this perspective, although introduced earlier, 100GbE could be considered high-density 25GbE.
Much of the architectural foundation work for 25GbE is based on three standard-making efforts that have either been ratified or are well on their way to ratification:
- IEEE 802.3ba™-2010, which was initiated in 2006 and among other things establishes a standard for 100 Gb/s Ethernet using four lanes of 25 Gb/s data traffic over single-mode fiber (100GBASE-LR4 and 100GBASE-ER4);
- IEEE 802.3bj™-2014, which was initiated in 2010 specifies 100 Gb/s Ethernet across an electrical backplane with four lanes of 25 Gb/s data traffic using either NRZ signaling (100GBASE-KR4) or PAM-4 signaling (100GBASE-KP4) and copper cable (100GBASE-CR4) with four lanes of 25 Gb/s data traffic with NRZ electrical signaling;
- IEEE P802.3bm, which specifies a standard for 100 Gb/s Ethernet for chip-to-chip and chip-to-module electrical interfaces (CAUI-4) and 100 Gb/s Ethernet using four lanes of 25 Gb/s data traffic over multi-mode fiber (100GBASE-SR4), and is now in the Sponsor Ballot stage of the process.
In addition to the years of effort contributed by IEEE 802.3 groups and individuals, other industry efforts have also been making it easier to fast-track 25GbE into the industry. A summary of applicable specifications and technologies is shown in Table 1.
On the road to a standard
A collection of appropriate technologies embodied in previous standards work is not a standard itself. Additional work will be necessary to properly specify 25Gb/s Ethernet operation. But when the work initiated by the IEEE 802.3 25GbE Study Group is finished the result will be a complete Ethernet solution, which has particular significance for designers.
The task force of objectives (yet to be ratified by the Working Group) include:
- Support a MAC data rate of 25 Gb/s
- Support full-duplex operation only
- Preserve the Ethernet frame format utilizing the Ethernet MAC
- Preserve minimum and maximum FrameSize of current IEEE 802.3 standard
- Support a BER of better than or equal to 10-12 at the MAC/PLS service interface (or the frame loss ratio equivalent)
- Support optional Energy-Efficient Ethernet operation
- Define a single-lane 25 Gb/s PHY for operation over a printed circuit board backplane consistent with channels specified in IEEE Std 802.3bj-2014 Clause 93
- Define a single-lane 25 Gb/s PHY for operation over links consistent with copper twin axial cables, with lengths up to at least 3m
- Define a single-lane 25 Gb/s PHY for operation over links consistent with copper twin axial cables, with lengths up to at least 5m
- Define a single-lane 25 Gb/s PHY for operation over MMF consistent with IEEE P802.3bm Clause 95
- Provide appropriate support for OTN
Added to the tool kit of server and server subsystem designers will be a Physical Coding Sublayer (PCS) encoding specification, Forward Error Correction (FEC) sublayer specification, a specification for Energy Efficient Ethernet (EEE) operation, and other opportunities offered by the Ethernet ecosystem.
The introduction of 25GbE will enable the same types of breakout solutions for 25GbE from 100GbE that have become so popular with 10GbE and 40GbE. In the 10GbE/40GbE solutions, for example, each QSFP28 port can be used with a passive electrical or optical breakout cable to create four discrete 10 Gb/s interfaces to be used as 10GbE ports, which is being used today to enable high density ToR (Top of Rack) configurations.
Some aspects of the proposed 25GbE standard are going to be newly specified. The individuals involved in its development have to consider– at the very least – a diverse set of server architectures, data center architectures and server interconnects in their deliberations. This rich diversity has implications for the standard in addressing backplane, cabling and MMF optical interconnect options for end-of-row applications now and in the future.
Inside the framework established by these goals, there are still implementation refinements to be considered by the 25GbE Study Group. A topic of interest to design engineers involves a decision on how to allocate the channel budget between the server, cable and switch. As stated in the objectives, one option envisions a cable length of 3 meters, which would provide a greater portion of the channel budget to the server and/or switch cards but would also limit cabling runs to intra-rack implementations. The 5-meter option enables inter-rack communication in addition to intra-rack communication. Its inclusion in the standard – along with a consistent channel budget allocation – does not prevent the use of shorter cable, of course, and could allow the use of 3-meter cable without forward error correction (FEC).
Including support for a copper backplane largely assumed to be based upon 100GBASE-KR4 and optical interconnect specification largely assumed to be based upon 100GBASE-SR4 are other goals that need refinement by the study group. They arise from the previously mentioned wide variety of server-to-switch architectures that could benefit from a 25GbE standard.
Conclusion
The work of the IEEE 802.3 25GbE Study Group holds the promise of creating a quickly realizable standard that can be developed quickly with high confidence. 25GbE will enable the design of low cost-per-bit technologies that can benefit Web-scale data centers and cloud services networking facilities in the near term, as well as enterprise data centers in the future.
Matt Brown is a member of the Ethernet Alliance, IEEE 25G Study Group Architecture Ad Hoc Chair, and Principal Systems Engineer at AppliedMicro. He is responsible for architecture and specifications definition for a range of products including mixed signal and digital transceivers, Ethernet PHYs, packet processors, and OTN framers. He has over 30 years of experience developing system and device products addressing data communications and telecom applications. Matt has been a steady participant and contributor in various physical layer standards including 802.3ap, 802.3ba, 802.3az, 802.3bm, and SFP+, and was Editor-In-Chief for the 802.3bj standard.
