NTT DOCOMO’s 5G Experimentations and Trials on Network Slicing

Ashiq Khan (khan@nttdocomo.com), Takuya Shimojo (takuya.shimojou.gt@nttdocomo.com), Anass Benjebbour (benjebbour@nttdocomo.com), and Shigeru Iwashina (iwashina@nttdocomo.com), Research Laboratories, NTT DOCOMO, INC.

IEEE Softwarization, December 2017

 

1. Introduction

With the availability of component technologies and the progress made in standardization, DOCOMO is progressing well towards its target of 5G commercial roll out in 2020, coinciding with the Tokyo Olympics. DOCOMO itself has enormously contributed to the development of the component technologies and standards for 5G. Many such innovations and consequent specifications come from the extensive experimentation and trials carried out by DOCOMO R&D. Where the Radio Access Network (RAN) trials [1]-[4] are focusing on meeting stringent 5G requirements, the core network and its operation is focusing on network slicing. This article presents an overview of DOCOMO’s 5G-related experiments and trials, focusing on 5G mobile core network, and the automation of network operation in the 5G era and beyond.

2. 5G Mobile Core Network Experiments

DOCOMO is continuing experimentation on a slice-based network infrastructure suitable for the mobile core network in the 5G era. Such experimentation, in collaboration with Ericsson, is focusing on suitable physical implementation of slices in the context of a mobile core network, control plane (C-plane) design, simultaneous multi-slice connection from a single UE, and a variety of other technical improvements. Our immediate future objective is to validate and optimize the 3GPP Release 15 mobile core (Figure 1) [5]. However, as this is still in the standardization process, our approach is to emulate the Release 15 core by using similar concepts and solutions from the already completed Release 14 specifications.

Figure 1: Applying non-roaming 5G System architecture for multiple PDU session in reference point representation (dotted lines with colour are not part of the original figure).

In our Release 15 based experimental mobile core network system for 5G, multiple user planes (U-planes) can be set up as different slices. This has been shown in Figure 2 as User Plane Functions (UPFs). U-plane is the user/session data transfer plane in mobile cellular networks. In order to utilize existing implementation as much as possible, a Control and User Plane Separation (CUPS) [6] principle was used from the Evolved Packet Core (EPC) functions. The UPFs in our experimental system thus have been implemented by using the U-plane functions of the Serving Gateway (S-GW) and Packet Data Network Gateway (P-GW) from the EPC [7].

Figure 2: 5G Core (5GC) network experimental system in DOCOMO.

Each of the UPFs has a dedicated Session Management Function (SMF) (see Figure 1), responsible for session management in that U-plane. In our experimental system, the SMFs are implemented by combining the legacy session management functions from the Mobility Management Entity (MME), S-GW and P-GW [7].

The Access and Mobility Management Function (AMF), responsible for admission control and mobility management has been implemented by using the respective functionalities from a legacy MME. The AMF remains common for all UPFs. At present, in our experimental system, the common control plane (Figure 1) in Release 15 is emulated only by the AMF we have developed (Figure 2) for simplicity.

Similar to present Release 15 specification, one User Equipment (UE), that is a mobile terminal, can connect to a single AMF in our prototype. The AMF manages multiple SMFs, thus multiple UPFs. One UE can connect to multiple UPFs, which enables it to connect to multiple slices simultaneously. Slice-specific resources are allocated to the pairs of SMFs-UPFs. By comparing the green and blue dotted rectangles in Figure 1 with the elements of the same color in Figure 2, the relevance of our prototype with the Release 15 mobile core can be understood clearly. Functions highlighted by red dotted rectangles in Figure 1 remain common over slices.

Using the prototype explained above, we have already performed successful trials on simultaneous multi-slice connections from a single UE. In one such trial, as shown in Figure 2, a robot is programmed to be an UE. The robot has a head-mounted camera for a security application. Real-time video is sent from the camera to a remote surveillance center e.g. to a security company.

The robot itself stands and balances on two wheels, where the wheels are controlled from a remote server. Sensor data on the robot’s physical position is transmitted to a remote robot control server. The server, after analyzing the status of the robot’s position, sends instruction back to the robot’s wheels. The wheels, moving accordingly, ensure that the robot continues to stand.

The robot’s balance-related session emulates an Ultra-Reliable Low Latency Communication (URLLC) service, where the Round Trip Time (RTT) needs to be kept below 40 ms. If the RTT raises beyond that, the robot tumbles to the floor.

The video transmission from the robot’s camera emulates an enhanced Mobile BroadBand (eMBB) session. Here, transmission latency is not of high concern and is above 50 ms. For this, high bandwidth and necessary resources are allocated.

Using the above scenario, we create two different slices, one for the URLLC session, and the other for the eMBB session. When the URLLC session is transported through the eMBB slice, the robot falls. We then create a URLLC slice which has an RTT below 40 ms and dedicate it to the URLLC session. This keeps the robot standing and the surveillance application running. However, the RAN part is not sliced and is emulated by WiFi for simplicity. In future, we would integrate this sliceable Release 15-based 5G Core (5GC) system with 5G New RAT (NR) in order to create end-to-end slices. The objective of this experimentation is to closely follow the Release 15 progress, and give feedback into standardization based on implementation and experiment results.

3. Automation of Network Operation in the 5G Era

Virtualization, networking slicing, softwarization etc. bring added complexity to a network. In legacy non-virtualized networks, as an example, an Element Manager (EM) performed all Fault, Configuration, Accounting, Performance, and Security (FCAPS) management of a network function and its hardware in an integrated manner. Virtualization, which decouples the software of a network function from the hardware, necessitates root-cause analysis and distributed event-handling mechanisms. Added dynamicity by Software Define Networks (SDNs) also requires an enhanced operation system to be sufficiently handled.

Network slicing in the 5G era, which encompasses Network Functions Virtualization (NFV) and SDNs to realize multiple end-to-end network slices, will come with different layers of network management and operation, such as:

- Operation of physical network infrastructure
- Operation of Infrastructure/Network/Platform/Software-as-a-Service (XaaS) domains
- Operation of network slices
- Operation of services in network slices
- Management of customers of services in network slices

Network slices, services and customers can also span over multiple operators networks. Conventional operation techniques where manual intervention is the norm will not be able to handle such complexity and scale. It is an absolute necessity now to automate the operations at all these layers to ensure the economical as well as technical viability of network slicing.

DOCOMO is continuing R&D on automation of operation with our partner Ericsson. Assuming network slicing over distributed cloud infrastructure over Wide Area Network (WAN) protocols, we are focusing first on automating all Life-Cycle Management (LCM) operations of a slice. Slice LCM includes but is not limited to instantiation, topological re-configuration, scaling, failure recovery, and termination of a network slice. Different levels of isolation between slices are also being investigated.

Figure 3: Network slice operation system on distributed cloud infrastructure

We have already successfully automated instantiation, topology reconfiguration and scaling of end-to-end slices in a setup as shown in Figure 3. The inter-cloud WAN is emulated on Multiprotocol label switching (MPLS) [8]. The WAN is managed by a WAN Orchestrator (WAN-O) which sets up per-tenant VPNs with bandwidth guarantees over MPLS. WAN-O, based on the instruction from the Automated Slice Operation (ASO) System, executes this task automatically by coordinating with the OpenDaylight (ODL) SDN controllers [9] and datacenter gateways (IP Edges). Border Gateway Protocol (BGP) and Path Computation Element (PCE) Communication Protocol (PCEP) are used for this purpose (Figure 3).

Per-tenant network functions are instantiated in cloud datacenters in the form of Virtual Machines (VMs). In our experiments, we are using OpenStack [10] as the cloud management system. Kernel-based Virtual Machine (KVM) is used as the hypervisor. Neutron, the network controller in OpenStack, does not currently support per-tenant bandwidth isolation. In our experiments, this function is carried out at the ASO layer.

Cloud resources are also isolated on a per-tenant basis. Resources of one tenant can only host the VMs from that tenant. VMs of one tenant are also interconnected by tenant-specific bandwidth-guaranteed transport plane Virtual Private Networks (VPNs), connecting other VMs of the same tenant in different clouds.

An IP packet destined to a remote VM but connected to the same logical neutron network is labelled by the Open vSwitch [11] in a cloud with a MPLS label and encapsulated in a Generic Routing Encapsulation  (GRE) [12] packet. The MPLS label associates the packet with the respective Virtual Routing and Forwarding (VRF) instance on the datacenter gateway. The datacenter ODL instance is responsible for programming the soft-switch accordingly and exchanging the label with the datacenter gateway by means of BGP [13]. In the remote datacenter, the reverse process takes place, resulting in packet delivery at the correct soft-switch and hence attached VM.

Such interconnected per-tenant VMs create an end-to-end slice, dedicated to a tenant. A tenant could be an enterprise customer, a customer-facing service, or some other similar entity.

Policy rules: In a multi-tenant, multi-slice end-to-end hosting and networking scenario, closed-loop automation requires both per-tenant policies, as well as the network operator’s own. In our experimental system, per-tenant policies can be set to limit compute, storage and network resource usage, block the execution of unauthorized operations, trigger actions including scaling, healing, and topology reconfiguration to meet the Service-Level Agreement (SLA) with a tenant. We use Topology and Orchestration Specification for Cloud Applications (TOSCA) [14] for modeling and setting policy rules.

Automated operation: At present, we have successfully automated the instantiation, scaling and topology reconfiguration of slices in distributed cloud scenarios. The operation system is fed with operation models and policy required for each model. Based on these models, multiple VMs are instantiated in different clouds, and then connected over the WAN with bandwidth guarantees, in a fully automated way without any manual intervention. Scaling and topology reconfiguration (e.g. inter-cloud VM migration) can also be performed in a model-driven automated way, governed by policy.

We are also experimenting with cases where a tenant can perform some of the operation by itself (Customers in Figure 3). This requires strict inter-slice isolation, and a robust policy definition to avoid any effects on other tenants. We’ll address this in future.

4. Conclusions

DOCOMO plans to commercially roll-out 5G in 2020. As 5G wireless technologies are more advanced, the first commercial 5G deployment is expected to start from the RAN. In consequent years, the 5G Core, network slicing and automated operation systems will see commercial roll out as the 5G services provided by DOCOMO grow larger. Therefore, a migration strategy will become vital for not only DOCOMO, but for all operators. In the future, we will continue to contribute to the development, experimentation and standardization of cutting edge technologies for 5G and beyond in cooperation with our partners and the global community.

 

References

[1] A. Harada, Y. Inoue, D. Kurita, T. Obara, “5G Trials with Major Global Vendors,” NTT DOCOMO Technical Journal, Vol. 17, No. 4, pp. 60-69, Apr. 2016.
[2] Y. Kishiyama, A. Benjebbour, S. Nagata, Y. Okumura, T. Nakamura, “NTT DOCOMO 5G Activities –Towards 2020 Launch of 5G Services,” NTT DOCOMO Technical Journal, Vol. 17, No. 4, pp. 4-15, Apr. 2016.
[3] T. Nakamura, A. Benjebbour, Y. Kishiyama, S. Suyama, T. Imai, “5G Radio Access: Requirements, Concept and Experimental Trials,” IEICE Transactions on Communication, Vol. E98-B, No. 8, pp. 1397-1406, Aug. 2015.
[4] 5G Trial Sites, NTT DOCOMO, INC., https://www.nttdocomo.co.jp/english/corporate/technology/rd/docomo5g/trial_site/index.html
[5] 3GPP TS 23.501 V1.3.0, System Architecture for the 5G System; Stage 2, Release 15, 2017.
[6] 3GPP TS 23.214, Architecture enhancements for control and user plane separation of EPC nodes, Release 14, 2016.
[7] 3GPP TS 23.401, General Packet Radio Service (GPRS) enhancements for Evolved Universal Terrestrial Radio Access Network (E-UTRAN) access, Release 8, 2015.
[8] E. Rosen, A. Viswanathan, R. Callon, Multiprotocol Label Switching Architecture, RFC 3031, IETF, 2001.
[9] OpenDaylight, The Linux Foundation, https://www.opendaylight.org/
[10] OpenStack, The OpenStack Foundation, https://www.openstack.org/
[11] Open vSwitch, The Linux Foundation, http://openvswitch.org/
[12] D. Farinacci, T. Li, S. Hanks, D. Meyer, P. Traina, Generic Routing Encapsulation (GRE), RFC 2784, IETF, 2000.
[13] Y. Rekhter, T. Li, S. Hares, A Border Gateway Protocol 4 (BGP-4), RFC 4271, IETF, 2006.
[14] TOSCA Simple Profile for Network Functions Virtualization (NFV) Version 1.0, OASIS, 2017.

 

---

 

Ashiq Khan is an Assistant Manager at NTT DOCOMO, INC. where his present work focus is on the research and development of the 5G mobile network for 2020. He has 10 years of work experience in network virtualization, and was involved in ETSI NFV standardization, NFV-based mobile network development, and the consequent commercial roll-out of the world’s first multi-vendor virtualized Evolved Packet Core (vEPC) mobile system by DOCOMO in March 2016. He served in the OPNFV TSC as a founding member, and initiated the Doctor and Promise projects, leading to their success implementation in OpenStack.

Dr. Khan is now the chair of the OPNFV Tokyo User Group, and an OPNFV Ambassador. He holds a PhD in computer science from the University of Tsukuba.

 

Takuya Shimojo is Researcher at NTT DOCOMO, INC. His current research interests include Network Slicing, 5G Core System, and Software Defined Network. He is also responsible for initiating and leading collaboration projects with external parties to realize the world’s first 5G mobile network, including a 1000-times-higher data rate and 1 millisecond ultra-low latency.

He holds a master's degree in electrical engineering from the University of Tokyo.

 

Anass Benjebbour obtained his Ph.D. and M.Sc. degrees in Telecommunications in 2004 and 2001, respectively, and his B.Sc. diploma degree in Electrical Engineering in 1999, all from Kyoto University, Japan. In 2004, he joined NTT DOCOMO, INC.

Since 2010, he has been a leading member of its 5G team. Dr. Benjebbour served as 3GPP standardization delegation from DOCOMO, Japan delegation to ITU-R WP5D, secretary of the IEICE RCS conference from 2012 to 2014, associate editor for the IEICE Communications Magazine from 2010 to 2014, and associate editor for the IEICE Transactions on Communications from 2014 to 2018. He is an author or a coauthor of 100+ technical publications, 4 book chapters and is an inventor of 50+ patent applications. He is a senior member of IEEE and IEICE.

 

Shigeru Iwashina is an Executive Research Engineer of the Smart ICT Platform Research Group in NTT DOCOMO, INC.

He received his B.E. and M.E. degrees from Tokyo Institute of Technology in 1993 and 1995, respectively. Since joining DOCOMO in 1995, he has been engaged in various research and development activities, including the PDC cellular systems, the core network of the IMT2000, and the core network of the LTE system. His current research interests are NFV-based mobile core network virtualization and 5G network architecture.

 

Editor:

Chris Hrivnak is a Sr. Member of the IEEE and The Photonics Society.  He is also a member of the IEEE Life Sciences Community, the IEEE Software Defined Networks (SDN) Community and the IEEE Internet Technology Policy Community and has participated in the IEEE Experts in Technology and Policy (ETAP) Forum..  He graduated from Baldwin Wallace University and completed varied coursework at Cleveland State, Case Western Reserve, Gould Management Education Center, McKinsey, Hughes R&D Productivity, UT-Dallas, Tulane Law and Pepperdine Graziadio.  Broad range of interests including but not limited to augmented intelligence, autonomous systems, additive manufacturing, information & communications technologies (ICT), life sciences, dark physics, etc.