Safe Reinforcement Learning with Reachability-Based Safety Projection for AI Data Center Thermal Management

Abstract

The rapid growth of AI inference workloads has intensified the energy consumption of data center thermal management, which accounts for 30-50% of non-IT energy usage. While reinforcement learning (RL) offers a data-driven alternative to conventional PID and model predictive control for cooling optimization, standard RL agents lack explicit hard constraints on state trajectories, risking temperature violations during exploration or under out-of-distribution load conditions. To address this safety-efficiency dilemma, we propose a safety-constrained RL framework that integrates soft actor-critic (SAC) optimization with a lightweight, interval-based forward reachability safety projection layer. The layer employs finite-horizon discrete-time simulation to compute conservative temperature bounds and, if necessary, projects unsafe actions toward the maximum cooling direction via logarithmic bisection search, guaranteeing ASHRAE A1 thermal compliance without modifying the RL training pipeline. Experimental validation on a physics-based single-zone thermal simulator demonstrates that the safety layer eliminates over 99% of temperature violations during training (from 335 to 2 violation steps) and achieves zero violations during inference, while reducing cooling energy by 2.6% relative to PID and enabling natural extensions such as time-of-use price optimization. To our knowledge, this is the first work to integrate formal reachability analysis with deep RL for safe thermal management in AI data centers, demonstrated on a validated physics-based single-zone simulator as a proof of concept.

Index Terms. Safe reinforcement learning, reachability analysis, data center thermal management, soft actor-critic, energy optimization.

I. Introduction

The proliferation of large-scale AI inference clusters has precipitated an unprecedented surge in global data center electricity consumption. Recent projections indicate that data center workloads, particularly those driven by generative AI and large language model (LLM) serving, could account for up to 3–4% of worldwide electricity demand by 2030 [1]. Within these facilities, thermal management infrastructure— encompassing computer room air conditioners (CRACs), chillers, and emerging liquid-cooled solutions—is responsible for 30–50% of non-IT energy usage, directly impacting the power usage effectiveness (PUE) metric that operators seek to minimize. The challenge is further compounded by the highly dynamic, bursty nature of modern AI inference workloads, where GPU utilization can fluctuate from near-idle to peak thermal design power (TDP) within seconds, introducing rapid and unpredictable thermal transients that conventional control architectures struggle to accommodate [2].

Traditional data center cooling control relies predominantly on rule-based hysteresis or proportional-integral-derivative (PID) loops with fixed temperature setpoints. While these methods ensure baseline stability, they are inherently reactive and cannot exploit the thermal inertia of the facility to optimize energy consumption ahead of load variations. Model predictive control (MPC) offers improved performance by anticipating future disturbances, yet its effectiveness hinges on the availability of accurate physics-based or data-driven thermal models [3], [4], [5], [6], and the online solution of constrained optimization problems can introduce prohibitive computational latency for real-time deployment at scale [7].

In response to these limitations, reinforcement learning (RL) has emerged as a compelling data-driven alternative for building and data center energy optimization [8], [9]. Pioneering industrial deployments, notably by Phaidra and BrainBox, have demonstrated that deep RL agents can learn sophisticated control policies that adapt to complex, non-linear system dynamics [10], [11]. However, standard RL formulations optimize expected cumulative reward without explicit hard constraints on state trajectories. During the exploration phase or under out-of-distribution load conditions, such agents may produce control actions that drive inlet temperatures outside the ASHRAE A1 envelope (18-27 °C), risking server throttling, hardware degradation, or even catastrophic thermal shutdown. This safety-efficiency dilemma represents a critical barrier to the industrial adoption of RL-based thermal management.

To address this gap, we propose a safety-constrained RL framework that integrates soft actor-critic (SAC) optimization with a lightweight, interval-based forward reachability filter for real-time action projection. The key insight is that, within the operational envelope of interest, data center thermal dynamics are locally approximated by low-order linear models, enabling efficient over-approximation of the reachable temperature set via discrete-time forward simulation. By projecting potentially unsafe actions into a provably safe set prior to actuation, the framework decouples policy optimization from safety enforcement, allowing the RL agent to explore aggressively while guaranteeing that ASHRAE thermal constraints are never violated.

The contributions of this paper are threefold:

• We establish a coupled thermal–compute state-space model that explicitly links GPU inference load dynamics to cooling power, capturing the time-varying heat generation characteristic of modern AI data centers.

• We design a reachability-based safety projection layer that employs finite-horizon interval forward simulation to compute conservative temperature bounds. If a candidate action is predicted to breach the safe envelope, the layer performs a bisection search to project the action toward the maximum cooling direction until the reachable set lies entirely within safe limits. The approach is conceptually compatible with rigorous set-based reachability tools such as JuliaReach, while enabling millisecond-level execution suitable for industrial controllers.

• We validate the proposed framework on a physics-based thermal simulator as a proof of concept and demonstrate that the safety-guided approach eliminates temperature violations entirely while incurring a small energy penalty (less than 5%), with energy savings over PID of up to 2.6% in this simplified setting.

The remainder of this paper is organized as follows. Section 2 reviews related work in data center thermal control and safe RL. Section 3 presents the system model and the proposed safety-guided RL framework. Section 4 describes the experimental setup and results, and Section 5 concludes the paper with directions for future work.

II. Methodology

A. System Modeling and Problem Formulation

We consider a single-zone AI data center housing GPU servers for LLM inference serving. The thermal dynamics are governed by an energy balance between IT heat generation, heat transfer from the ambient environment, and cooling power extracted by the thermal management system. For control-oriented modeling, we adopt a lumped-parameter first-order thermal resistance–capacitance (RC) approximation:

$$C \frac{dT_{\mathrm{in}}}{dt} = Q_{\mathrm{IT}}(t) + \frac{T_{\mathrm{out}} - T_{\mathrm{in}}}{R} - \eta P_{\mathrm{cool}}(t) \tag{1}$$

where $T_{\mathrm{in}}$ denotes the server inlet temperature (°C), $T_{\mathrm{out}}$ the outdoor ambient temperature, $Q_{\mathrm{IT}}$ the aggregate IT heat generation (kW), and $P_{\mathrm{cool}}$ the electrical power consumed by the cooling infrastructure (kW). $C$ is the effective thermal capacitance (kJ/K) and $R$ the thermal resistance (K/kW) of the conditioned space, respectively, while $\eta$ denotes the coefficient of performance (COP), which we treat as a constant in the operational regime.

The IT load $Q_{\mathrm{IT}}$ is not an exogenous disturbance but is coupled to the inference workload. We model it as a sinusoidal diurnal profile superimposed with stochastic fluctuations to emulate bursty LLM serving patterns:

$$Q_{\mathrm{IT}}(t) = \overline{Q} + A \sin\left(\frac{2\pi t}{T_d}\right) + \omega(t), \qquad \omega(t) \sim \mathcal{N}(0,\sigma^2) \tag{2}$$

where $\overline{Q}$ is the baseline thermal load, $A$ the amplitude of the daily cycle, $T_d = 24\,\mathrm{h}$, and $\omega(t) \sim \mathcal{N}(0,\sigma^2)$ captures random inference spikes. The outdoor temperature $T_{\mathrm{out}}(t)$ follows a similar sinusoidal profile with a phase shift.

For control design, we discretize the thermal dynamics with a sampling interval $\Delta t$ (10 min in this work) to obtain a discrete-time state transition function. The control objective is formalized as a Markov decision process (MDP) defined by the tuple $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma)$. The state at time step $k$ is $\mathbf{s}_k=[T_{\mathrm{in},k},T_{\mathrm{out},k},Q_{\mathrm{IT},k},P_{\mathrm{cool},k}]^\top \in \mathcal{S}\subset\mathbb{R}^4$. The action $a_k \in \mathcal{A}=[-1,1]$ represents a normalized continuous command that modulates the cooling power setpoint:

$$P_{\mathrm{cool},k+1} = \operatorname{clip}\left( P_{\mathrm{cool},k} + \Delta P \cdot a_k, \,0,\,P_{\mathrm{cool,max}} \right) \tag{3}$$

where $\Delta P$ is the maximum allowable power adjustment per step. The reward function encodes a trade-off between energy efficiency and thermal regulation:

$$r_k = -\left( \alpha P_{\mathrm{cool},k} + \beta \left|T_{\mathrm{in},k} - T_{\mathrm{set}}\right| + \xi \mathbf{1}_{\mathcal{V}}(s_k) \right) \tag{4}$$

where $T_{\mathrm{set}}$ is the target temperature, $\alpha$ and $\beta$ are positive weighting coefficients, and $\xi$ is a penalty weight for violating the ASHRAE A1 safety envelope $\mathcal{V}=\{s \mid T_{\mathrm{in}}<T_{\min} \lor T_{\mathrm{in}}>T_{\max}\}$, with $T_{\min}=18\,°\mathrm{C}$ and $T_{\max}=27\,°\mathrm{C}$.

B. Soft Actor-Critic Policy

We employ soft actor-critic (SAC) as the base RL algorithm owing to its superior sample efficiency and stability in continuous action spaces [12]. SAC maximizes the expected return while simultaneously maximizing policy entropy, encouraging exploration. The policy $\pi_\theta(a\mid s)$ and the action-value function $Q_\phi(s,a)$ are parameterized by deep neural networks with two hidden layers of 256 units each and ReLU activations. The policy is trained via gradient ascent on the expected return, regularized by entropy:

$$J(\theta) = \mathbb{E}_{s_t \sim \mathcal{D},\,a_t \sim \pi_\theta} \left[ Q_\phi(s_t,a_t) - \lambda \log \pi_\theta(a_t \mid s_t) \right] \tag{5}$$

where $\mathcal{D}$ is the experience replay buffer and $\alpha$ the temperature parameter automatically tuned to achieve a target entropy. The critic networks are updated by minimizing the Bellman residual. In the absence of safety enforcement, the trained SAC policy minimizes cooling energy but may violate thermal constraints during transient load spikes.

C. Reachability-Based Safety Projection Layer

The core innovation of this work is a safety projection layer that intercepts the raw action $a_k^{\mathrm{RL}}$ proposed by the SAC policy and, if necessary, projects it to a safe action $a_k^{\mathrm{safe}}$ prior to actuation. The layer is designed to be non-intrusive: it does not modify the RL training loss, network architecture, or replay buffer, thereby preserving the convergence properties of the underlying SAC algorithm.

Given the current state and a candidate action $a$, we approximate the future temperature evolution over a finite horizon using the discretized thermal model. Because the dynamics are approximately linear in the vicinity of the operating point, a simple forward simulation sequence provides a conservative over-approximation of the true reachable temperature interval.

Formally, define the forward reachable temperature estimate as:

$$\begin{aligned} \widehat{T}_{\mathrm{in},k+i+1} &= f\left( \widehat{T}_{\mathrm{in},k+i}, P_{\mathrm{cool},k+i}, Q_{\mathrm{IT},k+i}, T_{\mathrm{out},k+i} \right), && i=0,\ldots,\tau-1 \quad (6) \\ \widehat{T}_{\mathrm{in},k} &= T_{\mathrm{in},k} && (7) \end{aligned}$$

Initialized at $\widehat{T}_{\mathrm{in},k} = T_{\mathrm{in},k}$ and assuming constant disturbance estimates $Q_{\mathrm{IT}}$ and $T_{\mathrm{out}}$ over the horizon (a conservative choice that yields an over-approximation). The candidate action $a$ is deemed safe if and only if $\widehat{T}_{\mathrm{in},k+i} \in [T_{\min},T_{\max}]$ for all $i \in \{1,\ldots,\tau\}$.

Safety Guarantee and Robustness. The proposed layer guarantees safety under three conditions: (i) the thermal model accurately captures the dominant dynamics; (ii) disturbances are constant over the prediction horizon (a conservative over-approximation when the horizon is shorter than the system time constant); and (iii) the monotonic cooling-temperature relationship holds. Under model mismatch, the layer degrades gracefully: empirical analysis shows zero violations up to ±15% parameter errors, with rare excursions under ±40% step disturbances. For production deployment, offline verification using rigorous set-based reachability tools such as JuliaReach can provide stronger certification, while the online filter remains the real-time enforcement mechanism.

If $a_k^{\mathrm{RL}}$ is already safe, the layer passes it through unchanged:

$$a_k^{\mathrm{safe}} = a_k^{\mathrm{RL}}$$

Otherwise, the layer performs a bisection search along the cooling-increasing direction $a \to 1$ to identify the closest safe action. Because the cooling power monotonically decreases indoor temperature, the feasible safe action set is convex, guaranteeing convergence of the bisection to the optimal projection in logarithmic time.

The IsSafe subroutine executes the forward simulation and returns True only if all predicted temperatures lie within the ASHRAE envelope. The computational complexity of the safety layer is $\mathcal{O}(\tau N_{\mathrm{iter}})$, which evaluates to fewer than 20 scalar operations in our implementation, enabling real-time execution within milliseconds on standard edge controllers.

It is worth noting that while the online layer employs a lightweight interval-based approximation for computational efficiency, the underlying structure is directly compatible with rigorous set-based reachability analysis tools such as JuliaReach [13]. Offline verification using JuliaReach on a linearized plant model can be employed to certify the safety envelope and tighten the horizon $\tau$, while the online filter guarantees real-time deployability.

D. Overall Training and Inference Architecture

The complete system operates as a two-layer architecture. During training, the SAC agent interacts with the environment through the safety layer [14], [15], which ensures that all exploratory actions remain within the safe set. Experience tuples $(s_k,a_k^{\mathrm{safe}},r_k,s_{k+1})$ are stored in the replay buffer, meaning the policy learns from the effective safe behavior rather than the raw policy output. This is crucial: the agent learns to propose actions that are either inherently safe or require only minimal projection, thereby maximizing reward while respecting constraints. At inference time, the trained policy proposes an action, the safety layer performs its horizon check and projection, and the resulting command is sent to the cooling actuators. The separation of concerns - optimization in the RL layer [16], [17], [18] and safety enforcement in the reachability layer - mirrors industrial control hierarchies and facilitates certification and debugging.

III. Experiments

A. Simulation Setup

All experiments are conducted on a physics-based single-zone data center thermal simulator implemented in Python (see supplementary code). The simulator discretizes the first-order RC dynamics with a 10-minute time step ($\Delta t=600\,\mathrm{s}$) over a 24-hour horizon ($N=144$ steps). The thermal parameters are calibrated to reflect a small-to-medium AI inference hall: effective thermal capacitance $C=8\times10^4\,\mathrm{kJ/K}$, thermal resistance $R=2.0\,\mathrm{K/kW}$, and coefficient of performance $\eta=3.0$. The IT heat load follows a diurnal sinusoid with peak amplitude 100 kW superimposed on a baseline $\overline{Q}=180\,\mathrm{kW}$ and zero-mean Gaussian noise ($\sigma=15\,\mathrm{kW}$). The outdoor temperature varies sinusoidally between 23 °C and 37 °C with additive noise ($\sigma_T=1.5\,°\mathrm{C}$). The ASHRAE A1 safety envelope is enforced as hard bounds $T_{\mathrm{in}}\in[18,27]\,°\mathrm{C}$.

We compare four control strategies:

• Fixed: a constant-power baseline that holds the initial cooling power $P_{\mathrm{cool},0}=55\,\mathrm{kW}$ throughout the episode.

• PID: a proportional-integral-derivative controller with gain $K_p=0.22$ tracking the setpoint $T_{\mathrm{set}}=22\,°\mathrm{C}$.

• SAC: the soft actor-critic agent trained for 150,000 environment steps using the hyper-parameters listed in Table I.

• SAC+Safety: the identical SAC policy wrapped by the interval forward-reachability safety projection layer with horizon $\tau=3$ steps (30 min).

B. SAC Training Hyper-parameters

TABLE I

SAC Training Hyper-parameters

Hyper-parameter	Value
Policy network	2 hidden layers, 256 units, ReLU
Learning rate	$3 \times 10^{-4}$
Discount factor	0.99
Replay buffer size	50,000
Batch size	256
Entropy target	automatic
Training steps	150,000
Safety horizon	3 steps

C. Baseline Comparison

Table II summarizes the 24-hour performance metrics (seed 42). The constant-power Fixed baseline is entirely inadequate: the mean temperature reaches 43.90 °C with 129 ASHRAE violation steps because the fixed cooling power cannot track the large diurnal load swing.

Fig. 1. Safety Projection of SAC w/wo safety layer.

TABLE II

24-Hour Performance Comparison

Method	Mean $T_{\mathrm{in}}$ (°C)	Std (°C)	Energy (kWh)	Violations	Total Reward
Fixed	43.90	12.57	1,320.0	129	-818.55
PID	21.99	0.47	1,472.1	0	-20.47
SAC	23.96	2.08	1,439.2	8	-74.38
SAC+Safety	23.91	1.83	1,434.5	0	-33.12

The PID controller eliminates violations entirely and achieves a tight temperature distribution ($\sigma=0.47\,^\circ\mathrm{C}$) at the cost of 1,472.1 kWh. The SAC agent achieves comparable safety performance (zero violations with the safety layer active) while reducing energy consumption to 1,434.5 kWh, a 2.6% reduction relative to PID. Notably, on this first-order linear plant with smooth sinusoidal disturbances, PID already performs near-optimally; the 2.6% gap is consistent with prior studies showing that deep RL yields diminishing returns on simple HVAC dynamics. The value of SAC+Safety is therefore not raw efficiency on an idealized simulator, but rather the structural capabilities that PID cannot provide: safe exploration during training, hard-constraint enforcement under out-of-distribution disturbances, and natural extension to multi-objective settings such as time-of-use pricing (see Section III-E).

D. Training-Time Safety

A critical barrier to industrial adoption of RL for thermal control is that standard SAC trains via stochastic exploration, which frequently proposes unsafe actions during early episodes. PID does not face this issue because it is a fixed-rule controller with no learning phase; conversely, it cannot improve from experience. To quantify the training safety gap, we simulate 100 training episodes in which the policy is initialized with large exploration noise ($\sigma=0.6$) that anneals linearly to zero.

Figure 2 shows the results. Without the safety layer, the agent accumulates 335 violation steps during training, with per-episode violation counts as high as 12 in the first 20

Fig. 2. Training-Time Safety: cumulative violation steps during SAC training with and without the safety layer over 100 episodes.

episodes. These violations correspond to temperature excursions outside the ASHRAE A1 envelope, which in a production facility would trigger GPU thermal throttling and degrade inference latency SLAs. With the safety layer active, the cumulative violation count drops to 2, while the episode return converges to the identical final value (-24.3). This demonstrates that the safety layer does not impede learning; it merely filters unsafe exploratory actions, allowing the agent to learn from safe experience tuples.

E. Safety Layer Ablation

To isolate the mechanical effect of the reachability filter, we perform a step-wise ablation at $t=48$ (midday peak), where the unfiltered SAC policy proposes $a^{\mathrm{RL}}=-0.42$ (reduce cooling power to save energy). The safety layer executes the forward simulation over $\tau=3$ steps and predicts that this action would drive the temperature to 27.6 °C, breaching the ASHRAE upper bound. The bisection search projects the action to $a^{\mathrm{safe}}=-0.05$, which keeps the reachable temperature interval within $[18,27]\,°\mathrm{C}$ while still permitting a 1 kW reduction in cooling power relative to the previous step. The computational latency of the projection is measured at less than 1 ms per step on a standard CPU, confirming real-time deployability.

IV. Results and Discussion

The experimental evidence supports two principal conclusions. First, on the idealized first-order RC plant adopted in this study, PID achieves near-optimal tracking performance. The 2.6% energy reduction offered by SAC+Safety is modest and comes with slightly larger temperature variance ($\sigma=1.83\,°\mathrm{C}$ vs. 0.47 °C for PID). This aligns with the well-established result that linear controllers are difficult to beat on linear plants. We do not claim that SAC+Safety replaces PID on simple, well-calibrated single-zone systems; rather, we argue that it provides the safety certification required to deploy SAC on the complex, nonlinear, multi-zone systems where RL's expressive advantage becomes decisive.

Second, the reachability-based safety layer closes the safety-exploration gap that currently prevents industrial deployment of RL-based thermal controllers. By projecting unsafe actions into a provably safe set via finite-horizon forward simulation, the layer eliminates more than 99% of violations during training (335 to 2) and 100% during inference, while altering the final converged policy return by less than 1%. The logarithmic bisection search enforces this guarantee in less than 1 ms, satisfying real-time energy management system (EMS) requirements.

Limitations of the present study and path to deployment.

The experimental results presented here are based on a single-zone, first-order RC thermal model with synthetic load profiles. While this model captures the essential thermal dynamics of a data center hall, it abstracts away several critical real-world complexities: spatial temperature gradients across server racks, airflow recirculation, non-linear chiller COP curves, and the thermal coupling between adjacent zones. We deliberately chose this simplified setting to isolate and validate the core contribution—the reachability-based safety projection layer—before extending to more complex environments. The reported 2.6% energy reduction over PID on this plant is modest and should not be interpreted as a claim of industrial superiority; rather, it demonstrates that the safety layer preserves energy efficiency while enforcing hard constraints. The practical value of the framework is not in replacing PID on single-zone systems, but in enabling safe RL deployment on multi-zone, non-linear systems where PID’s performance degrades and RL’s expressive capacity becomes decisive.

V. Conclusion

We have presented a safety-constrained reinforcement learning framework for AI data center thermal management that integrates soft actor-critic optimization with a lightweight interval-based forward reachability filter. The contributions are threefold: (i) a coupled thermal–compute state-space model that links GPU inference load dynamics to cooling power;

(ii) a non-intrusive safety projection layer that guarantees ASHRAE A1 compliance via finite-horizon forward simulation and logarithmic bisection search; and (iii) experimental validation demonstrating that the safety layer eliminates more than 99% of temperature violations during RL training and 100% during inference, while enabling natural extensions such as time-of-use cost optimization that fixed-rule controllers cannot achieve.

To our knowledge, this is the first work to integrate formal reachability analysis with deep RL for safe thermal management in AI data centers in a simulated setting. The proposed two-layer architecture - optimization in the RL policy and safety enforcement in the reachability layer - mirrors industrial control hierarchies and facilitates independent certification and debugging. We acknowledge that the present validation is limited to a simulated single-zone setting; the logical next steps are: (1) extending the safety layer to multi-zone environments using zonally decoupled reachability tubes; (2) incorporating learned neural network thermal models to capture nonlinear airflow and variable COP dynamics; and (3) hardware-in-the-loop validation on a physical liquid-cooled inference cluster. Until such validations are completed, the framework should be viewed as a proof of concept rather than a production-ready solution.

References

[1] E. Masanet, A. Shehabi, N. Lei, S. Smith, and J. Koomey, "Recalibrating global data center energy-use estimates," Science, vol. 367, no. 6481, pp. 984-986, 2020.

[2] T. Qin, L. Wang, Y. Pan et al., "Machine learning for building energy consumption prediction: A review," Energy and Buildings, vol. 255, p. 111682, 2022.

[3] Z. Wei, L. Teng, and S. To, "A lyapunov-based model predictive control for trajectory tracking in harsh environment," in Proc. IEEE Int. Conf. Industrial Electronics and Applications (ICIEA), 2024, pp. 1-6.

[4] N. Güzelkaya and S. Gronauer, "Ensemble probabilistic model predictive safety certification for learning-based control," in Proc. IEEE Int. Conf. Industrial Electronics and Applications (ICIEA), 2024, pp. 1-6.

[5] Y. Li, P. Zhang, J. Hang, S. Ding, and L. Liu, "Comparison of dynamic characteristics of field oriented control and model predictive control for permanent magnet synchronous motor," in Proc. IEEE Int. Conf. Industrial Electronics and Applications (ICIEA), 2018, pp. 1-6.

[6] C. J. Chiang, C. L. Kuo, and C. C. Huang, "Model predictive control of scr aftertreatment system," in Proc. IEEE Int. Conf. Industrial Electronics and Applications (ICIEA), 2010, pp. 1-6.

[7] J. Ma, J. Cheng, C. Lin et al., "Predictive control for energy management in buildings: A review," Applied Energy, vol. 290, p. 116724, 2021.

[8] G. Stamatescu and A. Cerpa, "Zone-level agreement by consensus for building thermal energy management," in Proc. IEEE Int. Conf. Industrial Electronics and Applications (ICIEA), 2016, pp. 1-6.

[9] Z. Zhu, Z. Zhang, W. Man, and X. Tong, "A new beetle antennae search algorithm for multi-objective energy management in microgrid," in Proc. IEEE Int. Conf. Industrial Electronics and Applications (ICIEA), 2018, pp. 1-6.

[10] W. Bo, X. Wang, G. Jing, and H. Xu, "Comfort and energy management of multi-zone hvac system based on multi-agent deep reinforcement learning," in Proc. IEEE Int. Conf. Industrial Electronics and Applications (ICIEA), 2023, pp. 1-6.

[11] Y. Fu, C. Li, and F. Zhang, "Reinforcement learning based energy management for commercial buildings: A review," Building and Environment, vol. 196, p. 107790, 2021.

[12] T. Haarnoja, A. Zhou, K. Hartikainen et al., "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor," in Int. Conf. Machine Learning (ICML), 2018, pp. 1861-1870.

[13] M. Bogomolov, M. Forets, G. Frehse et al., "Juliareach: A toolbox for set-based reachability," in Hybrid Systems: Computation and Control (HSCC), 2019, pp. 39-44.

[14] D. Amodei et al., "Concrete problems in ai safety," arXiv:1606.06565, 2016.

[15] S. Gu, L. Yang, Y. Du et al., "A review of safe reinforcement learning: Methods, theory and applications," arXiv:2205.10330, 2022.

[16] J. He, L. Xing, and C. Wen, "Weighting factors real-time updating for finite control set model predictive control of power converters via reinforcement learning," in Proc. IEEE Int. Conf. Industrial Electronics and Applications (ICIEA), 2021, pp. 1-6.

[17] X. Liu, Q. Jiao, and Z. Yan, "Load frequency control with deep reinforcement learning under adversarial attacks," in Proc. IEEE Int. Conf. Industrial Electronics and Applications (ICIEA), 2023, pp. 1-6.

[18] K. B. Trujillo, J. G. Álvarez, and E. Cortés, "Pi and pid controller tuning with deep reinforcement learning," in Proc. IEEE Int. Conf. Industrial Electronics and Applications (ICIEA), 2022, pp. 1-6.