Recursive neural network-based design of unmanned aircraft swarm collaborative mission execution and autonomous navigation system

Recurrent neural network-based target detection methods can be divided into two main parts: two-stage target detection algorithms represented by FastR-CNN and single-stage target detection algorithms represented by Yolo series. The two-stage algorithm approach is: first generate the candidate region, and then through the convolutional neural network for classification or regression, usually has a higher accuracy but slower detection speed. The single-stage algorithm is: the target detection is regarded as a regression problem of the bounding box, and the image features are extracted directly to predict the object category and location, which usually has faster detection speed and is more capable of learning the generalized features of the object. For the real-time requirements of UAV control, this paper selects the single-stage detection algorithm to meet the needs of real-time control.Yolo series has launched nine versions since its introduction, of which YoloV5 has a greater advantage in ecology, accuracy and real-time, especially in the case of small samples more stable and robust, and in the lower performance of the GPU embedded platforms with better real-time performance. Therefore, in this paper, the Yolov5 target detection model is chosen as the network model for visual reconstruction to take into account the real-time control of UAVs and the convenience of future deployment of edge computing devices.

The basic principle of the YOLO family of algorithms is to directly predict and localize multiple targets in an image by employing a single neural network model with a single forward propagation. This class of algorithms usually divides the input image into S × S grid, each grid is responsible for predicting whether or not a target is contained within that grid as well as its location and category, and anchor frames obtained by clustering based on the data characteristics are introduced to assist in the prediction of the target, with multiple anchor frames predicted for each grid, with the anchor frames’ sizes and dimensions being correlated to the size and dimensions of the target. When the center point of an object falls into a grid cell, that grid is responsible for predicting the object, and each grid predicts the location information of B bounding boxes and compares it with the real labeled box, which is calculated as (2): (2)Cij=Pi,j(object)×IOUpredtruth$$C_i^j = {P_{i,j}}(object) \times IOU_{pred}^{truth}$$

Eq:

Cij$$C_i^j$$ - confidence score for the jrd bounding box in the ind grid;

Pi,j(object)$${P_{i,j}}\left( {object} \right)$$ - the function associated with the category;

IOUpredtruth$$IOU_{pred}^{truth}$$ - the intersection and concurrency ratio of the predicted box with the real labeled box.

Convolutional neural networks with multiple layers and branches and different convolutional kernel sizes are used to extract feature maps at different scales and then fused, and supervised learning is used to complete network training using samples containing Ground Truth. The loss function usually consists of localization error (including center coordinates, quadratic cost function of width and height), confidence error (using cross-entropy loss), and category error (binary cross-entropy or quadratic cost function). Subsequent elimination of overlapping bounding boxes and reduction of false detection rate are carried out by means of post-processing such as non-maximal value suppression, so that information such as the localization and category of each object in the input image can be directly inferred from the training results. Its category error is calculated as (3): (3)E1=∑i=0S2∑j=0BWijobj[C^ijlog(Cij)−(1−C^ij)log(1−Cij)]$${E_1} = \sum\limits_{i = 0}^{{S^2}} {\sum\limits_{j = 0}^B {W_{ij}^{obj}} } \left[ {\hat C_i^j\log \left( {C_i^j} \right) - \left( {1 - \hat C_i^j} \right)\log \left( {1 - C_i^j} \right)} \right]$$

In Eq:

S² - the number of image division grid cells;

B - number of bounding boxes;

Cij$$C_i^j$$ - prediction frame confidence score;

C^ij$$\hat C_i^j$$ - true frame confidence score;

Wijobj$$W_{ij}^{obj}$$ - positive or negative sample sign.

The position prediction of the bounding box is calculated as follows: (4)bx=(scalexy⋅σ(tx)−scalexy−12)+cx$${b_x} = \left( {scal{e_{xy}} \cdot \sigma \left( {{t_x}} \right) - \frac{{scal{e_{xy}} - 1}}{2}} \right) + {c_x}$$ (5)by=(scalexy⋅σ(ty)−scalexy−12)+cy$${b_y} = \left( {scal{e_{xy}} \cdot \sigma \left( {{t_y}} \right) - \frac{{scal{e_{xy}} - 1}}{2}} \right) + {c_y}$$ (6)bw=pwetw$${b_w} = {p_w}{e^{{t_w}}}$$ (7)bh=pheth$${b_h} = {p_h}{e^{{t_h}}}$$

In Eq:

(bx,by)$$\left( {{b_x},{b_y}} \right)$$ - pixel coordinates of the center point of the prediction box;

(cx,cy)$$\left( {{c_x},{c_y}} \right)$$ - coordinates of the upper left corner of the grid cell;

(tx,ty)$$\left( {{t_x},{t_y}} \right)$$ - the offset of the center point;

scale_xy - a scaling factor greater than 1 to ensure that the prediction frame will reach the extreme values;

σ(⋅)$$\sigma \left( \cdot \right)$$ - the sigmoid function;

(bw,bh)$$\left( {{b_w},{b_h}} \right)$$ - the width and height of the prediction box;

(pw,ph)$$\left( {{p_w},{p_h}} \right)$$ - the width and height of the a priori bounding box obtained by clustering;

(tw,th)$$\left( {{t_w},{t_h}} \right)$$ - the scaling of the a priori bounding box.

The localization box error is calculated using the squared loss function: (8)E2 = ∑i=0s2∑j=0BWijobj[(σ(tx)ij−σ(t^x)ij)2+(σ(ty)ij−σ(t^y)ij)2] + ∑i=0s2∑j=0BWijobj[(σ(tw)ij−σ(t^w)ij)2+(σ(th)ij−σ(t^h)ij)2]$$\begin{array}{rcl} {E_2} &=& \sum\limits_{i = 0}^{{s^2}} {\sum\limits_{j = 0}^B {W_{ij}^{obj}} } \left[ {{{\left( {\sigma \left( {{t_x}} \right)_i^j - \sigma \left( {{{\hat t}_x}} \right)_i^j} \right)}^2} + {{\left( {\sigma \left( {{t_y}} \right)_i^j - \sigma \left( {{{\hat t}_y}} \right)_i^j} \right)}^2}} \right] \\ &+& \sum\limits_{i = 0}^{{s^2}} {\sum\limits_{j = 0}^B {W_{ij}^{obj}} } \left[ {{{\left( {\sigma \left( {{t_w}} \right)_i^j - \sigma \left( {{{\hat t}_w}} \right)_i^j} \right)}^2} + {{\left( {\sigma \left( {{t_h}} \right)_i^j - \sigma \left( {{{\hat t}_h}} \right)_i^j} \right)}^2}} \right] \\ \end{array}$$

Eq:

(t^x,t^y)$$\left( {{{\hat t}_x},{{\hat t}_y}} \right)$$ - the center coordinates of the real markup box;

(t^wt^h)$$\left( {{{\hat t}_w}{{\hat t}_h}} \right)$$ - the width and height of the real labeling box.

The loss function is usually the sum of (3) and (8), which is used to build the dataset by collecting and labeling the data, and the neural network is trained using the Adam optimizer until the error is less than a set threshold.

Since UAVs can only capture a limited range of information due to the limitation of the viewing angle when they only utilize the front RGB camera on board to collect environmental information, the limitation of the viewing angle may lead to collisions between UAVs that cannot sense each other in a cooperative UAV navigation task. The advantages of introducing the distance between UAVs into the sensing data are as follows: 1)

When the distance between UAVs is too close, the decision-making part can adjust the strategy in time, such as changing the direction, as a way to avoid collision.

For the autonomous navigation task of cellular-connected UAVs in highly dynamic environments, the algorithms in this study are designed with the idea of firstly satisfying the two basic requirements of UAV navigation, i.e., safety and efficiency. Therefore, the complex decision-making process of UAV navigation in this context is transformed into solving three more tractable subproblems, namely: avoiding obstacles, approaching the destination, and choosing a specific action from the solution schemes of the first two subproblems to ensure the UAV’s air-ground communication connectivity. Therefore, this study designs an RSAC-based sub-network architecture for the first two sub-problems, which consists of an Evade Network and an Approach Network, each of which generates actions that match the sub-tasks, where the Evade Network is used to generate actions that are applicable for obstacle avoidance, and the Approach Network is used to generate actions applicable to approaching the destination.

The elaboration of the structure of the sub-networks of this study begins with the composition of their input vectors. Since this input relates to the subproblem of obstacle avoidance and it should contain information related to the historical motion characteristics of the dynamic obstacles, the input of this sub-network structure, as shown in (9), still adopts real-time updated historical information as the input vector, but the difference is that this historical information contains the subactions generated by the sub-network based on the previous historical information: (9)ht=(s0,a0,s1,a1,s2,…,at−1,st)$${h_t} = \left( {{s_0},{a_0},{s_1},{a_1},{s_2}, \ldots ,{a_{t - 1}},{s_t}} \right)$$

h_t is the historical information that is updated in real time with the state transfer of the Markov decision process, and it plays an important role in the subsequent UAV decision iterations, which are updated as shown in (10): (10)ht+1←ht∪(at,st+1)$${h_{t + 1}} \leftarrow {h_t} \cup \left( {{a_t},{s_{t + 1}}} \right)$$

Once the input vectors were determined, the design of the sub-network architecture was accomplished in conjunction with the classical LSTM network structure. The hierarchical sub-network designed in this study utilizes an Actor-Critic architecture similar to the classical SAC algorithm, with each sub-network consisting of an Actor π_θ, and two Critics Q_φ1 and Q_φ2. The introduction of two Value Networks avoids overestimation of the state-action pair Q values, and each Value Network is based on the same Eval-Target Network structure, which consists of an Eval Network with a parameter vector of φ, and a Target Network with a parameter vector of φ′. Parameters φ′ and φ are distinguished from each other by the fact that they are updated much less frequently. And it is worth mentioning that, unlike the input action information of the Value Network, the input action information of the Policy Network starts from a₋₁, which is assigned a default value of 0, aiming to maintain the uniform structure of the network.

Then, based on the overall algorithm flow, the action selection mode of the sub-network and the iterative update of its parameters are expounded, and in the interaction process between each cellular-connected UAV and the simulated complex dynamic urban environment, the strategy network of the sub-network Evade Network and Approach Network outputs the estimated probability distribution πθ(ωt|ht)$${\pi _\theta }\left( {{\omega _t}\left| {{h_t}} \right.} \right)$$ about angular velocity ω_t and the estimated probability distribution πo(vt|ht)$${\pi _o}\left( {{v_t}\left| {{h_t}} \right.} \right)$$ of linear velocity v_t based on the current historical trajectory h_t, and then the two sub-networks can generate actions respectively through equation (11). aevadet$$a_{evade}^t$$ and aapproacht$$a_{approach}^t$$: (11)a1=[ω1,vl]=[fo1(εi;hi),fθ2(εi;ht)]$${a_1} = \left[ {{\omega _1},{v_l}} \right] = \left[ {f_o^1\left( {{\varepsilon _i};{h_i}} \right),f_\theta ^2\left( {{\varepsilon _i};{h_t}} \right)} \right]$$

where fθ1(εi;hl)$$f_\theta ^1\left( {{\varepsilon _i};{h_l}} \right)$$ and fθ1(εi;hl)$$f_\theta ^1\left( {{\varepsilon _i};{h_l}} \right)$$ represent the results of sampling the distribution of the output of the first and second dimensions of the strategy network after adding noise, respectively. The specific architecture of the designed RSAC-based policy value network is shown in Fig. 1.

Figure 1.

The specific structure of the designed RSAC-based policy-value network

The Integrated Network then selects one of the two sub-actions as aIntegratedt$$a_{Integrated}^t$$. When the drone finishes executing the current action aIntegratedt$$a_{Integrated}^t$$ in the environment, it will be rewarded with rcvadet $$r_{cvade}^t$$ and rapproacht$$r_{approach}^{}t$$ for approaching an obstacle or a destination, and the history trajectory will be updated from h_t to h_t+1. At the same time, the sequences seqcvadet:(ht,aevadet,ht+1,revadet)$$seq_{cvade}^t:\left( {{h_t},a_{evade}^t,{h_{t + 1}},r_{evade}^t} \right)$$: and seqapproacht:(ht,aapproacht,ht+1,rapproacht)$$seq_{approach}^t:\left( {{h_t},a_{approach}^t,{h_{t + 1}},r_{approach}^t} \right)$$ are stored in the experience playback pool D$$\mathcal{D}$$, after which K trajectories are drawn from D$$\mathcal{D}$$ to iteratively update the neural network parameters φ & θ included in the RSAC algorithm.

As mentioned before, the value networks of the sub-networks are constructed i.e., to better train their strategy networks, while the estimates of their output Q values Qφ(ht,at)$${Q_\varphi }\left( {{h_t},{a_t}} \right)$$ serve as an evaluation of the decision to perform an action a, after observing the current historical trajectory h₂. For each sub-network, the objective function of the iterative update Q_φ is transformed into (12) with some fine-tuning: (12)JQ(φ)=E(ht,at)~D[12(Qφ(ht,at)−(r(hl,at)+γEht+1−p(ht,Qt)[Vφ′⋅(ht+1)]))2]$$J_Q(\varphi ) = \mathop {\mathbb{E}}\limits_{\left( {{h_t},{a_t}} \right) - \mathcal{D}} \left[ {\frac{1}{2}{{\left( {{Q_\varphi }\left( {{h_t},{a_t}} \right) - \left( {r\left( {{h_l},{a_t}} \right) + \gamma \mathop {\mathbb{E}}\limits_{{h_{t + 1}} - p\left( {{h_t},{Q_t}} \right)} \left[ {{V_{\varphi '}} \cdot \left( {{h_{t + 1}}} \right)} \right]} \right)} \right)}^2}} \right]$$

where Vφ′(hl+1)$${V_{\varphi '}}\left( {{h_{l + 1}}} \right)$$ is computed by Monte Carlo estimation of Eq. (13) by another target network. (13)Vφ(st)=EQi~π[Qn(hi,at)−αlog(π(ai|ht))]$${V_\varphi }\left( {{s_t}} \right) = \mathop {\mathbb{E}}\limits_{{Q_i}\sim \pi } \left[ {{Q_n}\left( {{h_i},{a_t}} \right) - \alpha \log \left( {\pi \left( {{a_i}|{h_t}} \right)} \right)} \right]$$

The process of iterative updating of parameters φ and φ′ will be repeated for several epochs (Epochs) until the value of Qφ(hi,at)$${Q_\varphi }\left( {{h_i},{a_t}} \right)$$ stabilizes and converges.

Similarly, small changes can be made to the objective function of the training strategy network as shown in (14): (14)JFi(θ)=Ehi−DEai−πθ[αlog(πθ(at|hi))−Qp(hi|at)]$${J_{{F_i}}}(\theta ) = \mathop {\mathbb{E}}\limits_{{h_i} - \mathcal{D}} \mathop {\mathbb{E}}\limits_{{a_i} - {\pi _\theta }} \left[ {\alpha \log \left( {{\pi _\theta }\left( {{a_t}|{h_i}} \right)} \right) - {Q_p}\left( {{h_i}|{a_t}} \right)} \right]$$

After this, a stochastic gradient ascent should be performed to maximize (14). Finally, the loss function of the temperature coefficient is now calculated by Eq. (15). (15)J(α)=Eai−πt[−α(log(πt(at|ht))+H¯)]$$J(\alpha ) = \mathop {\mathbb{E}}\limits_{{a_i} - {\pi _t}} \left[ { - \alpha \left( {\log \left( {{\pi _t}\left( {{a_t}|{h_t}} \right)} \right) + \overline {\mathcal{H}} } \right)} \right]$$

From the above introduction, it can be seen that the integrated complex strategy for autonomous UAV navigation in this algorithm evolves from three simpler strategies: avoidance, proximity and selection. The algorithm is named Layered-RSAC because it is based on the LSTM-enhanced SAC algorithm and it uses a layered neural network framework for solving complex optimization problems and applies a layer-by-layer optimization approach.

In order to explore the application effect of the proposed autonomous UAV navigation system, this paper carries out the construction of the system. Set the UAV autonomous navigation occupying space as a quadrilateral of 4km², and select four vertices as the localization points. The location of localization point 1 is the origin of the local coordinate system, and the direction of localization point 1 pointing to localization point 2 is the x-axis direction, and the direction of localization point 1 pointing to localization point 3 is the y-axis direction. Nine positions in the space were randomly selected to obtain their positioning data, and the positioning errors in the three directions are shown in Fig. 2 compared with the real position data measured by using the laser distance measuring equipment.

Figure 2.

System positioning error

Analysis of the data shows that the average localization error of the autonomous navigation system is 9.35 cm in the x-direction, 9.23 cm in the y-direction. While the average localization error in the Z-axis is 17.59 cm.

In order to improve the stability during takeoff and landing, ultrasonic altimetry data was fused in the near-ground stage for UAV altitude control. The landing point set during the experiment was (8, 8, 0). The UAV was unlocked from any position and took off, rose to a height of 3m, then flew to the pre-landing position (8, 8, -3), and then entered the landing mode.

A total of 18 flight tests were conducted, each time the UAV took off from a random position, and the actual landing position of the UAV is shown in Table 1. The average deviation of the landing in x-direction was 12.26cm, and the average deviation of the landing in y-direction was 9.72cm.

The metrics of the UAV’s operation during autonomous navigation are shown in Figure 3. Figure 3(a) demonstrates the variation of the UAV’s memory pool utilization versus the exploration completion rate; the memory pool was never full, and it is clear that the memory pool utilization was effectively controlled and grew rapidly at certain moments when it flew into a large amount of unknown space. The exploration completion rate explored more than 85% of the space.

Figure 3.

Navigation health indicators

Octree is a tree-based data structure for describing three-dimensional spaces, which can effectively avoid the problems of wasted storage space and excessive computational complexity. Figure 3(b) shows the change of the number of known nodes of the octree during the autonomous navigation of the UAV. According to the characteristics of the environmental obstacles, it can be seen that the idle nodes will trigger pruning as the exploration continues, and the number of idle nodes will become less in some periods of time, with a smoother overall growth trend. The occupied nodes, on the other hand, are less likely to trigger pruning because they can only explore to their surfaces, and the number keeps growing continuously. It can be seen that at t=150s, the change in the number of idle nodes and occupied nodes tends to be flat, indicating that most of the space has been explored at this time, and most of the corresponding nodes are known.