Research on Reinforcement Learning Based Regulation Scheme for Renewable Energy System in Green Buildings

Reinforcement learning is actually the process by which an intelligent body learns to make optimal decisions while interacting with its environment. The rules on how to choose actions based on states and rewards are called strategies π. The optimization goal of reinforcement learning strategy optimization is given in the following equation: (11) maxmize∑t∞rt\[\max mize\sum\limits_{t}^{\infty }{{{r}_{t}}}\]

Markov Decision Process (MDP) is mainly based on Markov Process and Dynamic Programming Theory, which provides a mathematical framework for sequential decision making to represent and deal with decision problems with uncertainty and delayed feedback [28]. In general, a Markov decision process consists of the following five tuples.

An intelligent body interacts with its environment (MDP) at discrete time steps by executing strategy π : S → P(A). Strategy π is defined as a probability distribution over the action space conditional on each state: S × A → [0,1]. The goal of the intelligent body in learning a strategy is the total reward that can be obtained from a given state given the strategy. The formula is as follows: (12) J(π)=Eρ0,π,T[ ∑t=0∞γtrt ]\[J(\pi )={{\text{E}}_{{{\rho }_{0}},\pi ,T}}\left[ \sum\limits_{t=0}^{\infty }{{{\gamma }^{t}}}{{r}_{t}} \right]\] where the state of the environment is initialized at the beginning of training as s₀, sampled from the initial state distribution: s₀ ~ ρ₀(s₀), and at each time step, depending on the state of the environment: s_t+1 ~ T(·|s_t,a_t), the intelligent body chooses the corresponding action a_t, where a_t ~ π(s_t),s_t+1, the state value function V^π(s) and the action value function Q^π(s,a) are defined as the expectation of the long-term gain obtained by following the strategy π, described as follows: (13) Vπ(s)=Eπ[ ∑t=0∞γtrt+1|s0=s ]∀s∈S\[{{V}^{\pi }}(s)={{\text{E}}_{\pi }}\left[ \sum\limits_{t=0}^{\infty }{{{\gamma }^{t}}}{{r}_{t+1}}|{{s}_{0}}=s \right]\forall s\in S\] (14) Qπ(s,a)=Eπ[ ∑t=0∞γtrt+1|s0=s,a0=a ]∀s∈S,a∈A\[{{Q}^{\pi }}(s,a)={{\text{E}}_{\pi }}\left[ \sum\limits_{t=0}^{\infty }{{{\gamma }^{t}}}{{r}_{t+1}}|{{s}_{0}}=s,{{a}_{0}}=a \right]\forall s\in S,a\in A\]

The learning objective of reinforcement learning in a Markov decision process is to find the optimal policy π of maximizing the value function, i.e., π_* = argmax_πV^π(s) = argmax_πQ^π(s,a).

In a Markov decision process, each action lasts for one discrete time step, whereas in a semi-Markov decision process, each time-abstracted action lasts for a number of finite time steps. Equipped with a set of options in a Markov decision process becomes a semi-Markov decision process, while the optimal option value function on the included options is used to select the optimal option to execute the option internal policy. The formal definition of the option value function Qoπ(s,o)$Q_{o}^{\pi }(s,o)$ is as follows: (15) Qoπ(s,o)=Eπ[ ∑t=0∞γtrt+1|s0=s,o0=o ]∀s,o∈S×O\[Q_{o}^{\pi }(s,o)={{\text{E}}_{\pi }}\left[ \sum\limits_{t=0}^{\infty }{{{\gamma }^{t}}}{{r}_{t+1}}|{{s}_{0}}=s,{{o}_{0}}=o \right]\forall s,o\in S\times O\]

The above equation represents an estimate of the long-term cumulative return obtained by following strategy π, in state s option o. Where π is an option-based strategy. The option learning framework uses a call-and-return execution mechanism, specifically, the intelligent body selects an option o according to its option value function Qaπ(s,o)$Q_{a}^{\pi }(s,o)$ and follows the option internal policy π_o to select the original action, the environment returns to the next state, at which time the intelligent body decides whether to terminate the option according to the termination function of the option o, and if it does so, it selects a new option according to the option value function Qoπ(s,o)$Q_{o}^{\pi }(s,o)$ and repeats the process. An option and repeat the process.

Q-Learning is a value-based algorithm in reinforcement learning algorithms, Q-Learning refers to the expected gain that can be obtained by taking an action in a state at a certain moment in time, and the environment will feed back the corresponding reward value according to the action of the intelligent body, the main idea of the algorithm is to construct the gain Q of the State and the Action into a Q-table to be stored, and then according to the Q-value to select the action that can get the biggest gain. In the process of updating the Q-function, Q-Learning usually adopts ò–greedy strategy for exploration. The principle of the ò–greedy-exploration strategy is to greedily select the best current action while maintaining a certain amount of exploration in anticipation of selecting a better action. Specifically, the ò–greedy-exploration strategy selects the action corresponding to the maximum value of the function with probability 1–ò, and randomly selects an action in the current state with probability ò. After selecting an action, Q-Learning updates the Q-function using the Bellman equation, i.e.,: (16) Q(s,a)=r+γ(maxa′Q(s′,a′))\[Q(s,a)=r+\gamma ({{\max }_{{{a}^{\prime }}}}Q({{s}^{\prime }},{{a}^{\prime }}))\]

The Deep Q-Network (DQN) algorithm is an approach that incorporates Deep Neural Networks (DNN) and Q-Leaming. The target network mechanism refers to the use of two neural networks with the same structure but different parameters in DQN, where the Q-eval network has the most recent parameters, while the Qtarget network uses parameters that are some time old [30]. The Q-function of the deep neural network parameterization is denoted as Q_(s,a;θ), where parameter θ is updated by minimizing the time difference error: (17) LDQN(θ)=Est,at,rt,st+1'~D[ y−Q(st,at;θ) ]2\[{{\text{L}}^{DQN}}(\theta )={{\text{E}}_{{{s}_{t}},{{a}_{t}},{{r}_{t}},s_{t+1}^{'}\tilde{\ }D}}{{\left[ y-Q({{s}_{t}},{{a}_{t}};\theta ) \right]}^{2}}\]

Outputting actions directly based on the current state leads to another very important algorithm in reinforcement learning, namely the policy gradient and parameterizing π_ϕ, and optimizing policy parameter ϕ with the objective of maximizing the long term cumulative gain J(π_ϕ) = J(ϕ), which utilizes rewards to directly augment and diminish the likelihood of choosing an action. Following the policy gradient theorem, policy parameter ϕ is updated in the direction of the increasing gradient of the curated objective function, as in the following equation: (18) ∇ϕJ(ϕ)=Eπϕ[ ∇ϕlogπϕ(at|st)qπϕ(st,at) ]\[{{\nabla }_{\phi }}J(\phi )={{\text{E}}_{{{\pi }_{\phi }}}}\left[ {{\nabla }_{\phi }}\log {{\pi }_{\phi }}({{a}_{t}}|{{s}_{t}}){{q}^{{{\pi }_{\phi }}}}({{s}_{t}},{{a}_{t}}) \right]\] where ∇_ϕlogπ_ϕ(a_t|s_t) is the gradient of strategy π_ϕ with respect to parameter ϕ, and q^π0(s_t,a_t) is the expectation of the true cumulative return, which can be estimated in different ways.

The strategy constraints of the trust domain strategy optimization algorithm are defective in the way of implementation, in order to further improve the efficiency of strategy update, the new algorithm Proximal Policy Optimization Algorithm (PPO) proposes a new way of constraining the difference between the old and the new strategies in the process of strategy updating, and eliminates the effect of the drastic change of the strategies by cropping the strategy updating amplitude, and its strategy gradient updating method is as follows: (19) LPPO(ϕ)=Eπϕ[min(ρtA^(st,at),clip(ρt,1−∫,1+∫)A^(st,at))]\[{{L}^{PPO}}(\phi )={{\text{E}}_{{{\pi }_{\phi }}}}[\min ({{\rho }_{t}}\hat{A}({{s}_{t}},{{a}_{t}}),clip({{\rho }_{t}},1-\int{,}1+\int{)\hat{A}({{s}_{t}},{{a}_{t}}))}]\]

Where A^(st,at)$\hat{A}({{s}_{t}},{{a}_{t}})$ is the old strategy π_ϕ− the estimation of the dominance function: Aπϕ−(st,at)=qπϕ−(st,at)−vπϕ−(st)${{A}^{{{\pi }_{{{\phi }^{-}}}}}}({{s}_{t}},{{a}_{t}})={{q}^{{{\pi }_{{{\phi }^{-}}}}}}({{s}_{t}},{{a}_{t}})-{{v}^{{{\pi }_{\phi }}-}}({{s}_{t}})$, ρ_t is the importance sampling weights: ρt=πϕ(at,st)πϕ−(at,st)\[{{\rho }_{t}}=\frac{{{\pi }_{\phi }}({{a}_{t}},{{s}_{t}})}{{{\pi }_{\phi }}-({{a}_{t}},{{s}_{t}})}\].

Actor-Critic based reinforcement learning algorithms, as another class of policy-based reinforcement learning algorithms, integrate the value function based reinforcement learning methods and policy-based reinforcement learning methods by using Actor to directly optimize the parameters of the policy while also using Critic estimation of the value function v(s) to evaluate the performance of the policy q^πϕ(s,α).

Unlike the stochastic policy gradient algorithm which outputs policies as probability distributions over the action space, the deterministic policy gradient algorithm (DPG) optimizes the deterministic policy μ_ϕ with the objective of maximizing the Q function. For example, in the Deep Deterministic Policy Gradient Algorithm (DDPG), the policy update formula is as follows: (21) ∇ϕJ(ϕ)=Eμϕ[ ∇ϕμϕ(s)∇aQμϕ(s,a)|a−μϕ(s) ]\[{{\nabla }_{\phi }}J(\phi )={{\text{E}}_{{{\mu }_{\phi }}}}\left[ {{\nabla }_{\phi }}{{\mu }_{\phi }}(s){{\nabla }_{a}}{{Q}^{{{\mu }_{\phi }}}}(s,a){{|}_{a-{{\mu }_{\phi }}(s)}} \right]\]

The output of the deterministic policy gradient algorithm is a deterministic action, so it performs better on problems in the continuous action space, and is well able to solve continuous control problems such as robotics.

The DDPG algorithm is able to deal with continuous state space problems, therefore, only the state space S constituents need to be specified. There is no need to make specific constraints on each constituent element, and according to the realistic physical constraints, it can be seen that the PV power, customer load, SOC and real-time tariffs satisfy Eqs. (26)-(28), respectively: (26) 0≤Ppv(t)≤∞\[0\le {{P}_{pv}}(t)\le \infty \] (27) 0≤Pload(t)≤∞\[0\le {{P}_{load}}(t)\le \infty \] (28) −∞≤Rgrid(t)≤∞\[-\infty \le {{R}_{grid}}(t)\le \infty \]

In the DDPG model, the current state s is input to the actor network in the intelligent body, and the actor network gives the probability of successive actions, the probability of successive actions presents a normal distribution curve, and the actor network outputs a determined action a based on the probability, which may not correspond to the maximum probability value of action a. Since the intelligent body can deal with the continuous action control problem, here it is sufficient to constrain only the maximum and minimum values for the action-battery charging and discharging power.

The purpose of this paper is to examine the benefits and drawbacks of various reinforcement learning algorithms for ZEH energy system management, ultimately leading to a more efficient and appealing regulation strategy. The setting of the reward function in the DDPG model mainly takes into account the user’s comprehensive energy cost and the operation of the battery, which is consistent with that of the reward function in the DQN model and the Q-learning model, so that it is easy to compare and evaluate the regulation effect in a specific In the DDPG model, the reward function setting mainly considers the user’s comprehensive energy consumption cost and battery operation, and is consistent with the reward function setting in the DQN model and Q-learning model, which facilitates the comparison and evaluation of the regulation effect in the specific case analysis.