Can utilities support high bandwidth for IT applications but ensure low latency for mission critical OT traffic?
The utility industry is undergoing a comprehensive transformation as it modernizes to a more efficient and smarter grid. In this transformation, utilities are faced with a dilemma. Their networks have been built into two separate silos. The operational technology silo (or OT) is mandated with carrying a wide variety of mission-critical data from substations, generation facilities, and other locations back to their control centers to ensure reliable energy delivery.
The Information Technology (or IT) silo is mandated with managing many of the new applications associated with a smarter grid such as Smart Meter data and video surveillance. Utilities can no longer justify the cost of keeping these as siloed networks and desire convergence. Yet the requirements of both networks are contradictory.
OT’s requirement is to provide the determinism and guaranteed performance required by protection applications to keep the lights on. IT requires scalable networks to meet the bandwidth intensive applications emerging to deliver a smarter grid.
Figure 1: OT/IT Utility Requirements
The substation WAN is the key infrastructure to transport the relay protection multiplexed channels, which are an integral and necessary part of modern power network OT protection systems. However, utility WANs need to address growth in other substation IT services such as video surveillance and the collection of smart meter data, in addition to traditional voice, control and automation, email, and corporate LAN traffic. So the overall challenge is how to deliver a converged OT/ IT network while the meeting the diverse requirements of each organization.
Figure 2: Top bandwidth intensive applications for utilities
How much data does a Utility generate?
The main business of Utilities is to generate and distribute power, but in fact utilities are generating a significant amount of data across both OT and IT networks. Let’s look at 3 key applications:
- Synchrophasers (OT): These were introduced after the major power outage in the Northeast United States in 2003 to provide greater details on the state of the power grid by taking samples of voltage, frequency and phase up to 60 times per second. Prior to synchrophasers, the grid would collect data only once every 4 seconds. Each Performance Measurement Unit (PMU) can generate up to 15 GB/day and a utility may deploy 100s of PMUs throughout their power grid;
- Smart Meter data (IT): Over 50% of US households have a smart meter used to measure power consumption at intervals up to 15 minutes. These are used to measure in near real-time the overall power consumption throughout the grid, and they allow utilities to reallocate power resources based on peak demand. For a utility with 1M households, up to 1TB/day of smart data can be generated and needs to backhauled to the utility central and backup datacenters; and
- Surveillance Cameras (IT): As cybersecurity becomes a close second in priority for utilities to reliability of the smart grid, video surveillance is becoming a key tool to enhance security especially at unmanned substations. The emerging trend is to backhaul security video to a central site where it can be viewed and analyzed using facial recognition to enhance security. Each security camera can generate up to 9 GB/day and there may be 5-10 cameras at each substations based on the site demands.
Bottom line, utilities generate significant data
If we sum up these 3 applications, this can generate between 6 – 8 TB of data per day. As OT and IT networks converge, the network needs to meet the needs of these bandwidth intensive applications yet still meet the high performance required by mission critical applications on the OT side.
Utilities can no longer justify the cost of keeping these as siloed networks and desire convergence. Yet the requirements of both networks are contradictory.
To address the challenge of growing bandwidth demand from IT applications with low-latency performance for mission critical OT applications, Ciena has validated a best-in-class solution: the Ultra-Low Latency Packet WAN.
The Ultra-Low Latency Packet WAN Solution integrates Ciena’s 3930 Service Delivery Switch and 5142 Service Aggregation Switch to handle IT applications with the SEL-ICON substation multiplexer to aggregate and transport utility Teleprotection Relay (TPR), Supervisory Control and Data Acquisition (SCADA), Generic Object-Oriented Substation Events (GOOSE) and DS0 services while delivering low latency transport of mission critical OT traffic. By combining TDM and Ethernet transport options with a comprehensive range of data interfaces, this solution can enable utilities to migrate legacy mission critical services to converge their IT and OT networks. The solution also enables OT and IT groups to maintain their own network elements while part of an end to end solution.
Putting the Solution to the Test
As part of the Ultra-Low Latency Packet WAN Solution, Ciena’s Solution Validation team performed over 200 hours of testing of the solution. One of the key tests was to build a network combining mission critical teleprotection traffic and higher bandwidth services such as DS-1 and Video traffic which could be used for video surveillance. The teleprotection and DS-1 traffic were fed from SEL 411L and ICON multiplexers into the Ciena Packet WAN (edge and aggregation packet rings). The test bed is shown in Figure 3.
Quality of Service (QoS) in Ciena’s Packet WAN is key to providing guarantees on the performance of mission critical teleprotection traffic over a multi-service network and deliver the required deterministic latency performance when congestion from other lower priority traffic such as video is present. Under congestion situations, QoS architecture prioritizes mission-critical traffic over lower priority traffic. From the Ciena Packet WAN perspective, all SEL ICON traffic is regarded as mission critical and transported as high priority traffic. The scheduler for SEL ICON traffic was set to Strict Queue which enables the scheduler to process all SEL ICON traffic prior to all lower priority traffic, which ensures similar latency and jitter even during network congestion.
Congestion is then introduced on the 10G network NNI ports using lower priority traffic and measurements taken again. Congestion traffic is shown in red highlight in the diagrams below.
To ensure the highest priority of mission critical traffic, Ciena’ Packet WAN can also prioritize teleprotection traffic to be set to the highest priority (priority 7), which is typically reserved for network management traffic. Most utilities would agree that teleprotection traffic should be given the highest priority to maintain the reliability of the grid at the highest levels and keep the lights on.
During our tests, the Packet WAN was flooded with up to 10G video traffic on top of the existing teleprotection and DS1 traffic. Test measurements demonstrated that there are no significant impairments to latency on teleprotection traffic or on other high priority traffic under congestion versus non-congestion conditions. Teleprotection ultra-low latency performance was maintained even when the substation WAN was flooded with lower priority traffic.
Figure 3: Ultra-Low Latency Packet WAN Test Bed
It’s impossible to predict what new bandwidth intensive applications utilities will be required to handle over their substation WANs in the coming years, especially when it comes to the internet of things. However, creating a reliable, available and secure network that is ready to cope with unpredictable data demand over OT and IT networks is non-negotiable.
Check out this application note, if you want to learn more about the Ultra-Low Latency Packet WAN solution.
Also join us at UTC Regions 8 Fall Technical Training Meeting; Sheraton Denver Tech Center, Denver, CO October 18, 2018 where I will be presenting “How to Prioritize Teleprotection Traffic Over A Multi-Service Substation WAN.”