PCIe Gen2, 5.0GT/s 48-l ne, 12-port PCIe Switch a
Features Features
PEX 8648 Vitals PEX 8648 Vitals
o 48-lane, 12-port PCIe Gen2 switch o 48-lane, 12-port PCIe Gen2 switch
- Integrated 5.0 GT/s SerDes - Integrated 5.0 GT/s SerDes
o 27 x 27mm2, 676-pin FCBGA package o 27 x 27mm2, 676-pin FCBGA package
o Typical Power: 3.7 Watts o Typical Power: 3.7 Watts
PEX 8648 Key Features PEX 8648 Key Features
o Standards Compliant o Standards Compliant
- PCI Express Base Specification, r2.0
(backwards compatible w/ PCIe r1.0a/1.1)
- PCI Express Base Specification, r2.0
(backwards compatible w/ PCIe r1.0a/1.1)
- PCI Power Management Spec, r1.2 - PCI Power Management Spec, r1.2
- Microsoft Vista Compliant - Microsoft Vista Compliant
- Supports Access Control Services - Supports Access Control Services
- Dynamic link-width control - Dynamic link-width control
- Dynamic SerDes speed control - Dynamic SerDes speed control
o High Performance o High Performance
- Non-blocking switch fabric - Non-blocking switch fabric
- Full line rate on all ports - Full line rate on all ports
- Packet Cut-Thru with 140ns max packet
latency (x16 to x16)
- Packet Cut-Thru with 140ns max packet
latency (x16 to x16)
- 2KB Max Payload Size - 2KB Max Payload Size
- Read Pacing (bandwidth throttling) - Read Pacing (bandwidth throttling)
- Dual-Cast - Dual-Cast
o Flexible Configuration o Flexible Configuration
- Ports configurable as x1, x2, x4, x8, x16 - Ports configurable as x1, x2, x4, x8, x16
- Registers configurable with strapping
pins, EEPROM, I2C, or host software
- Registers configurable with strapping
pins, EEPROM, I2C, or host software
- Lane and polarity reversal - Lane and polarity reversal
- Compatible with PCIe 1.0a PM - Compatible with PCIe 1.0a PM
o Dual-Host & Fail-Over Support o Dual-Host & Fail-Over Support
- Configurable Non-Transparent port - Configurable Non-Transparent port
- Moveable upstream port - Moveable upstream port
- Crosslink port capability - Crosslink port capability
o Quality of Service (QoS) o Quality of Service (QoS)
- Eight traffic classes per port - Eight traffic classes per port
- Weighted round-robin source
port arbitration
- Weighted round-robin source
port arbitration
o Reliability, Availability, Serviceability o Reliability, Availability, Serviceability
- 3 Hot Plug Ports with native HP Signals - 3 Hot Plug Ports with native HP Signals
- All ports hot plug capable thru I2C
(Hot Plug Controller on every port)
- All ports hot plug capable thru I2C
(Hot Plug Controller on every port)
- ECRC and Poison bit support - ECRC and Poison bit support
- Data Path parity - Data Path parity
- Memory (RAM) Error Correction - Memory (RAM) Error Correction
- INTA# and FATAL_ERR# signals - INTA# and FATAL_ERR# signals
- Advanced Error Reporting - Advanced Error Reporting
- Port Status bits and GPIO available - Port Status bits and GPIO available
- Per port error diagnostics - Per port error diagnostics
- Performance Monitoring - Performance Monitoring
Per port payload & header counters Per port payload & header counters
Version 1.0 2009
PEX 8648
The ExpressLaneTM PEX 8648 device offers PCI Express switching
capability enabling users to add scalable high bandwidth, non-blocking
interconnection to a wide variety of applications including servers,
storage systems, and communications platforms. The PEX 8648 is
well suited for fan-out, aggregation, and peer-to-peer applications.
High Performance & Low Packet Latency
The PEX 8648 architecture supports packet cut-thru with a maximum
latency of 140ns (x16 to x16). This, combined with large packet memory
and non-blocking internal switch architecture, provides full line rate on all
ports for performance-hungry applications such as servers and switch
fabrics. The low latency enables applications to achieve high throughput and
performance. In addition to low latency, the device supports a packet payload
size of up to 2048 bytes, enabling the user to achieve even higher throughput.
Data Integrity
The PEX 8648 provides end-to-end CRC (ECRC) protection and Poison bit
support to enable designs that require end-to-end data integrity. PLX also
supports data path parity and memory (RAM) error correction as packets
pass through the switch.
Flexible Register & Port Configuration
The PEX 8648’s 12 ports can be configured to lane widths of x1, x2, x4, x8,
or x16. Flexible buffer allocation, along with the device's flexible packet
flow control, maximizes throughput for applications where more traffic
flows in the downstream, rather than upstream, direction. Any port can be
designated as the upstream port, which can be changed dynamically. The
PEX 8648 also provides
several ways to
configure its registers.
The device can be
configured through
strapping pins, I2C
interface, host
software, or an optional
serial EEPROM. This
allows for easy debug
during the development
phase, performance
monitoring during the
operation phase, and
driver or software
upgrade. Figure 1
shows some of the PEX
8648’s common port
configurations.
Figure 1. Common Port Configurations
PEX 8648
PEX 8648
x4
PEX 8648
PEX 8648
x8
PEX 8648
PEX 8648
x8
PEX 8648
PEX 8648
x8
11 x4 2 x44 x8
10 x4 6x42 x8
Figure 1. Common Port Configurations
PEX 8648
PEX 8648
PEX 8648
PEX 8648
x4
PEX 8648
PEX 8648
PEX 8648
PEX 8648
x8
PEX 8648
PEX 8648
PEX 8648
PEX 8648
x8
PEX 8648
PEX 8648
PEX 8648
PEX 8648
x8
11 x4 2 x44 x8
10 x4 6x42 x8
Dual-Host & Failover Support
The PEX 8648 product supports a Non-Transparent
(NT) Port, which enables the implementation of multi-
host systems in communications, storage, and blade
server applications. The NT port allows systems to
isolate host memory domains by presenting the
processor subsystem as an endpoint rather than another
memory system. Base address registers are used to
translate addresses; doorbell registers are used to send
interrupts between the address domains; and scratchpad
registers (accessible by both CPUs) allow inter-
processor communication (see Figure 2).
CPU
CPU
Prim a ry Ho st
Primary Host
Root
Complex
Root
Complex
End
Point
End
Point End
Point
End
Point End
Point
End
Point
PEX 8648
PEX 8648
NT
CPU
CPU
Secondary Host
Secondary Host
Non-Transparent
Port
Figure 2. Non-Transparent Port
CPU
CPU
Prim a ry Ho st
Primary Host
CPU
CPU
Prim a ry Ho st
Primary Host
Root
Complex
Root
Complex
Root
Complex
Root
Complex
End
Point
End
Point End
Point
End
Point End
Point
End
Point
End
Point
End
Point
End
Point
End
Point End
Point
End
Point
End
Point
End
Point End
Point
End
Point
End
Point
End
Point
PEX 8648
PEX 8648
NT
CPU
CPU
Secondary Host
Secondary Host
CPU
CPU
Secondary Host
Secondary Host
Non-Transparent
Port
Figure 2. Non-Transparent Port
Dual Cast
The PEX 8648 supports Dual Cast, a feature which
allows for the copying of data (e.g. packets) from one
ingress port to two egress ports allowing for higher
performance in dual-graphics, storage, security, and
redundant applications.
Read Pacing
The Read Pacing feature allows users to throttle the
amount of read requests being made by downstream
devices. When a downstream device requests several
long reads back-to-back, the Root Complex gets tied up
in serving this downstream port. If this port has a narrow
link and is therefore slow in receiving these read packets
from the Root Complex, then other downstream ports
may become starved – thus, impacting performance. The
Read Pacing feature enhances performances by allowing
for the adequate servicing of all downstream devices.
Hot Plug for High Availability
Hot plug capability allows users to replace hardware
modules and perform maintenance without powering
down the system. The PEX 8648 hot plug capability
feature makes it suitable for High Availability (HA)
applications. Three downstream ports include a
Standard Hot Plug Controller. If the PEX 8648 is used in
an application where one or more of its downstream
ports connect to PCI Express slots, each port’s Hot Plug
Controller can be used to manage the hot-plug event of
its associated slot. Every port on the PEX 8648 is
equipped with a hot-plug control/status register to
support hot-plug capability through external logic via the
I2C interface.
SerDes Power and Signal Management
The PEX 8648 supports software control of the SerDes
outputs to allow optimization of power and signal
strength in a system. The PLX SerDes implementation
supports four levels of power – off, low, typical, and
high. The SerDes block also supports loop-back modes
and advanced reporting of error conditions, which
enables efficient management of the entire system.
Interoperability
The PEX 8648 is designed to be fully compliant with the
PCI Express Base Specification r2.0, and is backwards
compatible to PCI Express Base Specification r1.1 and
r1.0a. Additionally, it supports auto-negotiation, lane
reversal, and polarity reversal. Furthermore, the PEX
8648 is tested for Microsoft Vista compliance. All PLX
switches undergo thorough interoperability testing in
PLX’s Interoperability Lab and compliance testing at
the PCI-SIG plug-fest.
Applications
Suitable for host-centric as well as peer-to-peer traffic
patterns, the PEX 8648 can be configured for a wide
variety of form factors and applications.
Host Centric Fan-out
The PEX 8648, with its symmetric or asymmetric lane
configuration capability, allows user-specific tuning to a
variety of host-centric applications. Figure 3 shows a
typical server design where the root complex provides a
PCI Express link that needs to be expanded to a larger
number of smaller ports for a variety of I/O functions. In
this example, the PEX 8648 has a 16-lane upstream port,
and five downstream ports using x8 and x4 links.
The PEX 8648 can also be used to create PCIe Gen1 (2.5
Gbps) ports. The PEX 8648 is backwards compatible
with PCIe Gen1 devices. Therefore, the PEX 8648
enables a Gen 2 native Chip Set to fan-out to Gen 1
endpoints. In Figure 3, the PCIe slots connected to the
PEX 8648’s downstream ports can be populated with
either PCIe Gen1 or PCIe Gen 2 devices. Conversely,
the PEX 8648 can also be used to create Gen 2 ports on
a Gen 1 native Chip Set in the same fashion.
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
x16
x4
Memory
Memory
Memory
Memory
x8
x8
x16
x4
Endpoint
Endpoint
Endpoint
Endpoint
Chipset
Chipset
Chipset
Chipset
Endpoint
Endpoint
Endpoint
Endpoint
PCIe Gen1 or PCIe Gen2 slots
x8
PEX 8648
PEX 8648
PEX 8648
PEX 8648
Figure 3. Fan-in/out Usage
Communications Systems
The PEX 8648 can also be utilized in communications
applications. Figure 4 shows a PEX 8648 being used in a
router to connect multiple endpoints in a line card to the
control module using PowerPC. The link-widths for each
module can be configured as required. The peer-to-peer
communication feature of the PEX 8648 allows the
endpoints to communicate with each other without any
intervention or management by the host.
Memory
Memory
Memory
Memory
PowerPC
PowerPC
PowerPC
PowerPC
End
Point
End
Point
End
Point
End
Point End
Point
End
Point
End
Point
End
Point
Control
Module
Line Card
End
Point
End
Point
End
Point
End
Point
End
Point
End
Point
End
Point
End
Point
End
Point
End
Point
End
Point
End
Point
End
Point
End
Point
End
Point
End
Point
PEX 8648
PEX 8648
PEX 8648
PEX 8648
x8 or x4
Figure 4. Router Usage
Backplane Communication
With 48 lanes and 12 ports, the PEX 8648 is well suited
for backplane applications requiring high connectivity (a
large number of ports). Figure 5 represents an ATCA or
MicroTCA backplane application with two switch fabric
blades and multiple AMC blades. In this example, two
PEX 8648s provide peer-to-peer data exchange for up to
20 AMC blades connecting to the switch fabric. The
PEX 8648 utilizes its NT port to isolate the hosts on the
switch fabrics and the remaining 10 downstream ports
on each switch are used to fan-out to the 20 AMC blades
(or 10 AMC Carrier Modules).
10-20 AMC or 10 AMC
Carrier Modules Fabric
To
Backplane
To other
Fabric
PEX 8648
PEX 8648
PEX 8648
PEX 8648
PEX 8648
PEX 8648
PEX 8648
PEX 8648
NT
CPU
Figure 5. ATCA/MicroTCA Backplane Usage
Fail-Over Storage Systems with Dual Cast
The PEX 8648’s Dual Cast feature proves to be very
useful in storage systems. In Figure 6, the Dual Cast
feature enables the PEX 8648 to copy data going to its
four downstream ports to the backup system and vice
versa (see yellow traffic patterns) in one transaction as
opposed to having to execute two separate transactions
to send data to the redundant chassis. By offloading the
task of backing up data onto the secondary system,
processor and system performance is enhanced. Non-
Transparent (NT) ports are used to isolate the host
domains of the backup system from the primary system.
PEX 8648
PEX 8648
8 Disk Chassis
FC
FC
x4 x4
FC
FC
8 Disk Chassis
FC
FC
x4 x4
FC
FC
8 Disk Chassis
FC
FC
x4 x4
FC
FC
8 Disk Chassis
FC
FC
x4 x4
FC
FC
x8
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
Memory
Memory
Memory
Memory
Chipset
Chipset
Chipset
Chipset
CPU
CPU
CPU
CPU
CPU CPU
CPU CPU
x8
Memory
Memory
Chipset
Chipset
Memory
Memory
Chipset
Chipset
x8
Backup
System
PEX
8648
PEX
8648
NT
PEX 8616
PEX 8616
NT
PEX 8616
PEX 8616
PEX 8616
PEX 8616
NT
PEX 8616
PEX 8616
NT
PEX 8616
PEX 8616
PEX 8616
PEX 8616
NT
PEX 8616
PEX 8616
NT
PEX 8616
PEX 8616
PEX 8616
PEX 8616
NT NT
PEX 8616
PEX 8616
PEX 8616
PEX 8616
NT
All
x8s All
x8s
PEX 8616
PEX 8616
Figure 6. Dual Cast in Storage Systems
Software Usage Model Development Tools
From a system model viewpoint, each PCI Express port
is a virtual PCI to PCI bridge device and has its own set
of PCI Express configuration registers. It is through the
upstream port that the BIOS or host can configure the
other ports using standard PCI enumeration. The virtual
PCI to PCI bridges within the PEX 8648 are compliant
to the PCI and PCI Express system models. The
Configuration Space Registers (CSRs) in a virtual
primary/secondary PCI to PCI bridge are accessible by
type 0 configuration cycles through the virtual primary
bus interface (matching bus number, device number, and
function number).
PLX offers hardware and software tools to enable rapid
customer design activity. These tools consist of a
hardware module (PEX 8648RDK), hardware
documentation (available at www.plxtech.com), and a
Software Development Kit (also available at
www.plxtech.com).
ExpressLane PEX 8648RDK
The PEX 8648RDK is a hardware module containing the
PEX 8648 which plugs right into your system. The
PEX 8648RDK can be used to test and validate customer
software, or used as an evaluation vehicle for PEX 8648
features and benefits. The PEX 8648RDK provides
everything that a user needs to get their hardware and
software development started.
Interrupt Sources/Events
The PEX 8648 switch supports the INTx interrupt
message type (compatible with PCI 2.3 Interrupt signals)
or Message Signaled Interrupts (MSI) when enabled.
Interrupts/messages are generated by PEX 8648 for hot
plug events, doorbell interrupts, baseline error reporting,
and advanced error reporting.
Software Development Kit (SDK)
PLX’s Software Development Kit is available for
download at www.plxtech.com/sdk. The software
development kit includes drivers, source code, and GUI
interfaces to aid in configuring and debugging the
PEX 8648.
Product Ordering Information
Part Number Description
PEX8648-BB50BC 48-Lane, 12-Port PCI Express Switch (27x27mm2)
PEX8648-BB50BC F 48-Lane, 12-Port PC I Ex press Switch, Pb-F ree (27x 2 7m m2)
PLX Technology, Inc.
870 Maude Ave.
PEX8648-BB RDK PEX 8648 Rapid Development Kit
Please visit the PLX Web site at http://www.plxtech.com or contact PLX sales at 408-774-9060 for sampling.
Sunnyvale, CA 94085 USA
info@plxtech.com
www.plxtech.com
© 2009 PLX Technology, Inc. All rights reserved. PLX and the PLX logo are registered trademarks of PLX Technology, Inc. ExpressLane is a trademark of PLX Technology,
Inc., which may be registered in some jurisdiction. All other product names that appear in this material are for identification purposes only and are acknowledged to be
trademarks or registered trademarks of their respective companies. Information supplied by PLX is believed to be accurate and reliable, but PLX Technology, Inc. assumes no
responsibility for any errors that may appear in this material. PLX Technology, Inc. reserves the right, without notice, to make changes in product design or specification.
PEX8648-SIL-PB-1.0 04/09