High-fidelity integrated simulation and HIL testing require real resources with simulations in the loop. Commercial inverters, solar emulators, grid simulators, programmable loads, PMU, smart meters, electric vehicles, and protection relays are integrated for realistic experimental real-time studies for advanced PHIL and CHIL analysis with a focus on RES integration and validation of energy management and control systems. For example, ANGLE can utilize a plethora of hardware devices such as Cinergia Regenerative AC/DC converter, Cinergia Regenerative AC Grid simulator, Chroma Programmable AC/DC load, Regatron DC source-sink, relays, Opal-RT/Typhoon-HIL/RTDS simulators, Regatron 4Q amplifier, etc. to link different configuration setups for real-time grid analysis while integrating RES in various grid scales while adjusting grid and network parameters, contingency scenarios, etc.

WAMS provides a real-time dynamic view of power systems based on real-time data streamed from PMUs, allowing traditional supervisory control and data acquisition systems (SCADA) to be combined with energy management systems (EMS) for fast and proactive grid stability and situational awareness. ANGLE supports real-time synchrophasor-based measurement and monitoring of WAMS applications based on devices such as Arbiter Sentinel PMU, GPS antennas, OpenPDC, RTDS/Opal-RT, SEL3505 controllers, etc. This configuration can replicate the real-time dynamics of the energy system and represent realistic grid operations for different relevant studies. 

The growing impetus towards green transportation is achieved using EV integration in the electric grid. The mobility-to-grid modernization at ANGLE will be performed using a HIL setup with Growatt EV Chargers and RTDS/Opal-RT. Experiments will preview how vehicle charging and its dynamics affect grid operations regarding stability and efficiency.

IEDs form an integral part of substation automation systems, and different vendors manufacture them. Interoperability among multi-vendor devices must be ensured to reduce repair and maintenance times and allow plug-and-play system expansion. The digital real-time simulators, power amplifiers such as Doble F6350e, and multi-vendor IEDs at ANGLE can assist in carrying out interoperability tests. These results could help the user visualize the smart substation architecture and its operation before initial deployment.

The research and development process for analyzing and designing communication networks, devices, protocols, and applications can be expensive and time-consuming. A network simulator can simulate/emulate communication networks spanning over hundreds of miles and provide performance evaluation results in terms of latency, throughput, jitter etc. The cyber-system layer is modelled at ANGLE using communication network emulation/simulation platforms that support co-simulation capabilities (EXataCPS, ns-3, CORE). The EXataCPS, for example, is a network emulator which can model a network digital twin to represent cyber-physical system communication networks comprised of thousands of nodes, various protocol layers, application layers, physical layers, and devices, supporting standard-based controller and network implementation (e.g., IEC 61850/61499, SunSpec) as well as interoperability and communication testing. EXataCPS and HYPERSIM (Opal-RT) are integrated into a co-simulation platform within the same hardware and connected via virtual Ethernet links, but the data are still transmitted as standard communication protocols. This allows integration of the emulated network with equipment and physical system dynamics simulation, packet-level emulation to predict system behavior under attacks, scalability to represent the entire network while respecting timing constraints, fast communication at Layer 2, ensuring that the timing constraints of the overall system are met at real-time speeds with real-world high fidelity.

Adverse cyber-physical events, including natural disasters, hardware and software faults, cyber-attacks, etc., make resiliency critical in the context of cyber-physical energy systems.  In the area of cybersecurity, the introduction of information and communication technologies expanded the threat landscape of the power grid, evident by the increasing number of cyber-attacks on critical grid infrastructure.  For example, DER integration and cyber-physical microgrids regulate onsite energy generation through support functions enabled by smart inverters. Smart inverters, being consumer electronic firmware-based devices, are susceptible to increasing security threats. If inverters are maliciously controlled, they can significantly disrupt MG operation and electricity delivery and impact grid stability. ANGLE testbed supports capabilities of CHIL cybersecurity testing of such devices (a variety of inverters are present in the lab) and tests associated vulnerabilities in a lab environment to assess and visualize the respective attack impact. ANGLE capabilities can test studies towards achieving a secure and resilient 100% renewable energy grid with the development of multi-layer control and system methods, ranging from system stability controls to secure co-design of hardware/software/ physical systems.

DERs, as distribution-level resources, can leverage renewable or non-renewable resources for power generation (e.g., distributed rooftop solar PV, wind (micro)turbines, diesel generators, microgrids, controllable loads and demand-response, energy storage, etc.). DERs grant users and aggregators control over the power they produce and consume. DERs are interconnected, interoperable, and support remotely controllable features, thus, their security and control features should be thoroughly considered. The HIL setup at KAUST employs, among others, a Chroma PV emulator, Regatron DC Source/Sink, Cinergia Regenerative AC/DC converter, Cinergia Regenerative AC Grid simulator, Chroma Programmable AC/DC load, integrated with the real-time simulators. The consequences of the compromised security of DER on its operation can be assessed. Also, integration of DER standards can be validated while the quality and performance of DER components (e.g., maximum power point tracking (MPPT) efficiency, de-rating) can be evaluated. Furthermore, mitigation methodologies and design practices to prevent adversaries and safeguard DER systems can be evaluated before their actual deployment.

The experimental setup considers several multi-timescale factors, as shown in Figure 1. The simulation layer implements a high-fidelity model of the power grid model in digital real-time simulation (e.g., RTDS/RSCAD, RT-LAB/Opal-RT, TyphoonHIL) to test the transient and dynamic behavior of such systems in real time. The digital real-time simulators will soon be integrated via real-time communication interfaces with high-performance compute nodes located physically at ANGLE lab at KAUST to enable faster-than-real-time co-simulation. The high-performance computing cluster will run the algorithms to monitor (quantification of metrics) and control (setpoint and containment control) the power system under study. In addition, real-time simulators will be integrated via high-speed fiber connections to the KAUST high-performance computing system (Shaheen II/III), comprised of 197,568 cores and 790TB memory. At the control layer, local control functions will be hosted on actual controller devices (e.g., inverters) in a HIL fashion. Also in this layer is the protocol server which will implement the selected operation technology protocol (e.g., OPC UA). The supervisory layer consists of the global control functionality and the necessary HMI inputs.