CORD: Central Office Re-Architected as a Datacenter

Larry Peterson, Open Networking Lab

 

Landscape

Network operators are in the throes of unprecedented challenges. The proliferation of video traffic, mobile devices, and OTT services has pushed operator networks into uncharted territory. According to Krish Prabhu, CTO of AT&T Labs, for example, data traffic has increased 100,000 percent in the last eight years, and looking forward, plans are now underway to roll out ultra-fast fiber and access to 100 cities across the US.

Network operators want to make their networks efficient, programmable, elastic and agile to meet the challenges of user bandwidth demands, as well as to create new revenue streams with innovative services. They want to benefit from both the economies of scale (infrastructure constructed from a few commodity building blocks) and the agility (the ability to rapidly deploy and elastically scale services) that cloud providers enjoy today.

These cloud-inspired benefits are especially needed at the edge of operator network: in the Telco Central Office (CO). These facilities contain a diverse collection of purpose-built devices, assembled over 50 years, (standards-based architecture with vendor proprietary). This makes them both a source of significant CAPEX and OPEX and a potential barrier to rapid innovation. Moreover, large service providers operate thousands of such COs, each serving tens of thousands of residential, mobile, and enterprise customers.

In response to these challenges, network operators are re-inventing their networks to better leverage best practices in cloud elasticity and agility, Software Defined Networking (SDN), and Network Function Virtualization (NFV). This paper introduces one such effort—CORD—a collaborative effort between AT&T and the Open Networking Lab.

Introducing CORD

CORD re-architects the Central Office as a datacenter. The basic approach centers on unifying the following three related but distinct threads:

• The first is SDN, which is about separating the network’s control and data planes. This makes the control plane open and programmable, and that can lead to increased innovation. It also allows for simplification of forwarding devices that can be built using merchant silicon, resulting in less expensive white-box switches.
• The second is NFV, which is about moving the data plane from hardware appliances to virtual machines. This reduces CAPEX costs (through server consolidation and replacing high-margin devices with commodity hardware) and OPEX costs (through software-based orchestration). It also has the potential to improve operator agility and increases the opportunity for innovation.
• The third is the Cloud, which defines the state-of-the-art in building scalable services—leveraging software-based solutions, micro-service architecture, virtualized commodity platforms, elastic scaling, and service composition, to enable network operators to rapidly innovate.

The goal of CORD is not only to replace today’s purpose-built hardware devices with their more agile software-based counterparts, but also to make the CO an integral part of every Telco’s larger cloud strategy and to enable them to support more attractive and meaningful networking services.

Commodity Hardware

The target hardware for CORD consists of a collection of commodity servers and storage, interconnected by a leaf-spine fabric constructed from white-box switches. An illustrative example is shown in Figure 1.

Figure 1. Target hardware, constructed from commodity servers, storage, and switches.

Although similar in design (albeit on a smaller scale) to a conventional datacenter, there are two unique aspects to this hardware configuration:

• The switching fabric—organized as a leaf-spine topology—is optimized for traffic flowing east-to-west, between the access network that connects customers to the CO and the upstream links that connects the CO to the operator’s backbone. There is no north-south traffic in the conventional sense.
• The racks of GPON OLT MACS commoditize connectivity to the access network. They replace proprietary and closed hardware that connects millions of subscribers to the Internet with an open, software-defined solution. (GPON is the example access technology in the reference implementation of CORD, but the same argument applies to other access technologies as well.)

Software Building Blocks

With respect to software, our reference implementation of CORD exploits three open source projects:

• OpenStack is the cluster management platform. It provides the core IaaS capability, and is responsible for creating and provisioning virtual machines (VMs) and virtual networks (VNs).
• ONOS is the network operating system that manages the underlying white-box switching fabric. It also hosts a collection of control applications that implement services on behalf of Telco subscribers. ONOS is also responsible for embedding virtual networks in the underlying fabric, which is in turn accessed via OpenStack’s Neutron API.
• XOS is a service abstraction layer that unifies infrastructure services (provided by OpenStack), control plane services (provided by ONOS. It provides explicit support for multi-tenant services, making it possible to create, name, operationalize, manage and compose services as first-class operations.

Virtualizing Legacy Devices

The first step is to virtualize the existing hardware devices, transforming each device into its software service counterpart and running it on commodity hardware. In the process, functionality is likely disaggregated and re-packaged in new ways. Figure 2 shows the devices that CORD virtualizes. They include Optical Line Termination (OLT), Customer Premises Equipment (CPE), and Broadband Network Gateways (BNG).

Figure 2. Legacy CO, including three physical devices to be virtualized.

Due to space limitations, this overview focuses on GPON technology and virtualizing the OLT to produce vOLT; there are also virtualized counterparts to the CPE (vCPE) and BNG (vBNG).

The first challenge is to create a commodity I/O Blade. AT&T has developed an open specification for a GPON MAC 1RU “pizza box,” as shown in Figure 3. This board includes the essential GPON Media Access Control (MAC) chip under control of a remote control program via OpenFlow. This replaces an existing closed and proprietary OLT chassis (not shown) that integrates this GPON MAC chip with GPON protocol management, 802.1ad-compliant VLAN bridge, and Ethernet MAC functions.

Figure 3. GPON OLT IO Blade.

The end result is to bring the access network interface under the same SDN-based control paradigm as the white-box based switching fabric. In other words, virtual OLT (vOLT), is implemented as a combination of: (1) Merchant Silicon (i.e., commodity GPON interface boards) and (2) an SDN control function that sets up and manages control plane functions of an OLT (e.g., 802.1X, IGMP Snooping, and OAM).

The vOLT control application running on top of ONOS facilitates attachment and authentication (AAA), establishes and manages VLANs connecting consumer devices (see next section) and the CO switching fabric on a per-subscriber basis, and manages other control plane functions of the OLT.

Service Orchestration

Replacing hardware devices with software running in virtual machines is a necessary first step, but is not by itself sufficient. Just as all the devices in a hardware-based CO must be wired together in a meaningful way, their software counterparts must also be managed as a collective. This process is often called service orchestration, but if network operators are to enjoy the same agility as cloud providers, the abstractions that underlie the orchestration framework must fully embrace (1) the elastic scale-out of the resulting virtualized functionality, and (2) the composition of the resulting disaggregated (unbundled) functionality. A model that simply “chains” VMs together as though it is operating on their hardware-based counterparts will not achieve either goal.

Our approach is to adopt Everything-as-a-Service (XaaS) as a unifying principle. This brings the disparate functionality introduced by virtualizing the hardware devices under a single coherent model. The control functions run as scalable services (these functions run on top of ONOS), the data plane functions run as scalable services (these functions scale across a set of VMs), the commodity infrastructure is itself managed as a service (commonly known by the generic name IaaS), and various other global cloud services running in the CO are also managed as scalable services. No matter the role it plays in the overall CORD architecture, each service is structured in exactly the same way: it supports a logically centralized interface, called a service controller; it is elastically scale across a set of service instances (corresponding to VMs and OpenFlow switches); and it is multi-tenant with an associated tenant abstraction.

While vOLT, vCPE, and vBNG refer to the virtualized counterpart of the three physical devices, they are not complete and “pluggable” elements in CORD until they are packaged as services—each includes a multi-tenant service controller, and each scales independently across a set of service instances. Because, the new architecture no longer requires functionality to be bundled along the same boundaries as before, it is more intuitive to think of the virtualization process as resulting in three generic, multi-tenant services:

• Access-as-a-Service (ACCaaS): Implemented by a vOLT control application running on ONOS, where each tenant corresponds to a Subscriber VLAN.
• Subscriber-as-a-Service (SUBaaS): Implemented by a vCPE data plane function scaled across a set of containers, where each tenant corresponds to a Subscriber Bundle.
• Internet-as-a-Service (INTaaS): Implemented by a vBNG control application running on ONOS, where each tenant corresponds to a Routable Subnet.

If a Content Distribution Network (CDN)—itself a scalable cloud service deployed throughout the operator’s network, including caches in the CO—is added to these three new services, we have an example of the three kinds of services outlined in the Introduction: a cloud service (CDN), a data plane service (Subscriber-as-a-Service), and two control plane services (Access-as-a-Service, Internet-as-a-Service). This results in the legacy CO depicted in Figure 2 being re-architected into the datacenter version shown in Figure 4.

Figure 4. Four scalable services running in a CO, with a bare-metal switch located on the customer premises.

Path to Real-World Deployment

CORD is a significant milestone in bringing cost effectiveness and agility to Telco Central Offices. The plan is to build a fully functional reference implementation, followed by validating the architecture through lab trials and eventually field trials, and hardening the system along the way based on trial data. Taken as a whole, the goal is to bring CORD close to readiness for commercial deployments in operator networks.

 

---

 

Larry Peterson is Chief Architect at the Open Networking Laboratory, Director of the PlanetLab Consortium, and the Robert E. Kahn Emeritus Professor of Computer Science at Princeton University. He currently splits his time between Silicon Valley, Princeton, and Tucson, where he has an appointment at the University of Arizona. Peterson is also co-author of the best selling networking textbook Computer Networks: A Systems Approach. In 2007 he co-founded CoBlitz LLC to CDN technology developed on PlanetLab. CoBlitz was acquired by Verivue in 2010, and subsequently by Akamai in 2012. Peterson is a former Editor-in-Chief of the ACM Transactions on Computer Systems, and served as program chair for SOSP, NSDI, and HotNets. He is a member of the National Academy of Engineering, a Fellow of the ACM and the IEEE, and a past recipient of the IEEE Kobayashi Award and the ACM SIGCOMM Award. He received his Ph.D. from Purdue University in 1985.

 

Editor:

Jose M. Verger is a networking industry veteran who has worked in new product development, engineering and product management for Cisco Systems, 3COM, Bell Communications Research (Bellcore), AT&T and multiple successful start-ups such as Sentient Networks, Point Red and Wavezero. Currently Jose is at Verizon focusing on mobile public networks architecture and planning for enterprise services including the virtualization efforts.