Research on the fault diagnosis method of offshore oil and gas field equipment combined with deep reinforcement learning

Offshore oil and gas field equipment due to its geographic location is more special, but by virtue of the personnel’s own experience to determine the failure of the equipment can not be found in a timely manner, which may lead to the emergence of offshore oil and gas field safety problems. Therefore, the acquisition of data for offshore oil and gas field equipment is particularly important. The main work of multi-source data acquisition of offshore oil and gas field equipment is to utilize various sensors for data acquisition and convert the collected physical signals into current or voltage signals for computer recognition through A/D converter.

Multi-source data acquisition of offshore oil and gas field equipment is the basic part to carry out equipment fault diagnosis, and the storage of the collected characteristic parameters can provide the data basis for data display, data analysis and fault diagnosis. The specific data acquisition process is as follows: 1)

Select various types of sensors according to the types of system monitoring parameters, and after the arrangement is completed, the current, displacement, pressure and other types of sensors can be utilized to obtain the physical signals of motor current, brake oil pressure, vibration acceleration, gate spacing and gate force and other monitoring parameters of offshore oil and gas field equipment.

After the above data acquisition process, the data acquisition system of this paper is basically established, in addition to the key monitoring parameters mentioned above, the overall monitoring parameters of the system include data analog and switching quantities as well as alarm information, etc., and the collected data will be stored in a database for offshore oil and gas field equipment monitoring and fault diagnosis.

After acquiring the multi-source sensor data of offshore oil and gas field equipment, different types of data have different fault characteristics, so they need to be effectively fused so that the machine model can better learn its data characteristics and provide a complete data structure for realizing data analysis, fault diagnosis, etc [18].

The constructed multi-source heterogeneous data model based on granular computing is shown in Figure 1. The granularity structure is defined according to the data types and characteristics, and the granularity of different integrated levels in the granularity structure is calculated to give a formalized description of the multi-structured granularity. Based on multi-scale sampling to achieve the fusion of data, a rule table is generated, and the rules in the table are used to perform comprehensive calculation on the basic data to form views with different granularity sizes and complete data modeling.

Figure 1.

Multisource heterogeneous data model

There are some data values in the actual data that deviate from the expected situation, if these data values and normal data values are put together for statistics, it will greatly affect the correctness of the data processing, however, if these data values are simply eliminated, important characteristic information may be overlooked. In this paper, an outlier is defined as a group of measured values whose deviation from the mean is more than two times the standard deviation, and a measured value whose deviation from the mean is more than three times the standard deviation is referred to as a highly abnormal outlier. When processing the data, highly abnormal outliers should be eliminated.

Based on the multi-source data of offshore oil and gas field equipment, mining the fault features in the data is crucial for diagnosing equipment faults. In this regard, this paper proposes a non-local mean (NLM) denoising algorithm optimized based on the PSO algorithm for denoising and extracting fault features of offshore oil and gas field equipment.

The NLM algorithm can effectively retain the detail information and sharp edges of the signal, which can be used to deal with one-dimensional vibration signals. The algorithm extracts the periodic shock components in the signal based on the local similarity between the neighborhoods of the sample points, and achieves filtering and noise reduction by averaging the similar sample points.

The filtering and noise reduction effect of the NLM algorithm is strongly influenced by the radius of the search box M, the bandwidth parameter λ and the radius of the similarity box P. The larger M is, the better the noise reduction effect is, but too large increases the computation time. λ is too small to cause noise fluctuations, and too large to make the signal too smooth, losing important information. The value of P is too large so that the signal loses key information, and too small so that the similar block of information is not representative and noisy. If the combination of parameters is set artificially, it may lead to the loss of some fault characteristics in the filtered signal. Therefore, in this paper, we optimize the [M, λ, P]-parameter combination of NLM with the help of improved PSO algorithm to improve the performance of noise reduction and extraction of fault features.

When searching for optimization with the help of PSO algorithm, the algorithm explores the optimal solution according to the size of the value of the fitness function, and thus the construction of the fitness function directly affects the merit of the optimization result. Cliff index can highlight the shock characteristics of the noise reduction signal, but its stability is poor. The energy entropy index can evaluate the uniformity and complexity of the energy distribution in the signal, and its stability is good and noise resistance is strong.

Therefore, this paper combines the advantages of the crag and energy entropy indexes to construct the minimum energy entropy-crag ratio index (MHK)$$\left( {{M_{HK}}} \right)$$, whose expression is: (1)MHK=min(HEn/∑i=1nK)$${M_{HK}} = \min \left( {{H_{En}}\Bigg /\mathop \sum \limits_{i = 1}^n K} \right)$$ (2)HEn=−lg(k)−1∑i=1kpilgpi$${H_{En}} = - \lg {(k)^{ - 1}}\sum\limits_{i = 1}^k {{p_i}} \lg {p_i}$$

Where H_En is the energy entropy of the reconstructed signal after noise reduction, ∑i=1nK$$\sum\limits_{i = 1}^n K$$ is the sum of the cliff values of each IMF component in the reconstructed signal after noise reduction, and pi=ε=EMF(i)E$${p_i} = \varepsilon = \frac{{{E_{MF}}(i)}}{E}$$, k are the modal numbers.

The smaller the value of the NLM-filtered signal M_HK is, the more obvious the fault impact characteristics in the signal at this time, and the less noise interference it contains. In this paper, M_HK is selected as the fitness function of PSO algorithm, then the steps of PSO algorithm to optimize the combination of NLM parameters are as follows: 1)

Population initialization. Based on the existing research, set the PSO parameters, i.e., learning factor C₁ = C₂ = 2, population size of 50, iteration number K = 50, weights ω_max = 0.85 and ω_min = 0.35. Considering the noise reduction performance and computational efficiency, set the optimization range of NLM algorithm M to [500,1500]$$\left[ {500,1500} \right]$$, λ to [0.3,3] $$\left[ {0.3,3} \right]$$, and P to [10, 30].

After noise reduction and filtering extraction of offshore oil and gas field equipment features using the PSO-NLM algorithm, it may lead to the existence of some missing values of data, which, if not effectively and reasonably processed will result in the construction of erroneous data leading to the legacy of fault diagnosis results. Therefore, for the processing of incomplete data, this paper proposes to utilize the cubic spline interpolation (CSI) algorithm for filling.

The CSI algorithm is a high-precision interpolation algorithm [20]. The algorithm has the following two characteristics: 1)

When there is N + 1 data point, N intervals will be spaced out, requiring the generation of N curvelets, each of which is a cubic polynomial.

Each curve must satisfy the following four conditions: 1)

The endpoints of the curve pass through the data points, viz: (4)gi(xi)=yi,gi(xi+1)=yi+1$${g_i}\left( {{x_i}} \right) = {y_i},{g_i}\left( {{x_{i + 1}}} \right) = {y_{i + 1}}$$

Let Si=gi″(xi)$${S_i} = {g''_i}\left( {{x_i}} \right)$$, step h_i = x_i+1 − x_i. The values of a_i, b_i, c_i, d_i are denoted by S_i, h_i, y_i. It can be found by the above conditions: (8){ di=yi bi=Si2 ai=Si+1−Si2 ci=yi+1−yihi−2hiSi+hiSi+16$$\left\{ {\begin{array}{l} {{d_i} = {y_i}} \\ {{b_i} = \frac{{{S_i}}}{2}} \\ {{a_i} = \frac{{{S_{i + 1}} - {S_i}}}{2}} \\ {{c_i} = \frac{{{y_{i + 1}} - {y_i}}}{{{h_i}}} - \frac{{2{h_i}{S_i} + {h_i}{S_{i + 1}}}}{6}} \end{array}} \right.$$

As: (9){ gi′(xi)=ci gi−1′(xi)=3ai−1hi−12+2bi−1hi−1+ci−1$$\left\{ {\begin{array}{*{20}{l}} {g_i^\prime \left( {{x_i}} \right) = {c_i}} \\ {g_{i - 1}^\prime \left( {{x_i}} \right) = 3{a_{i - 1}}h_{i - 1}^2 + 2{b_{i - 1}}{h_{i - 1}} + {c_{i - 1}}} \end{array}} \right.$$

Bringing the expressions for a_i, b_i, c_i, and d_i into Eq. (9) is given by gi′(xi)=gi−1′(xi)$${g'_i}\left( {{x_i}} \right) = {g'_{i - 1}}\left( {{x_i}} \right)$$: (10)hi−1Si−1+2(hi−1+hi)Si−1+hiSi+1=6(f[xi,xi+1]−f[xi−1,xi])$${h_{i - 1}}{S_{i - 1}} + 2\left( {{h_{i - 1}} + {h_i}} \right){S_{i - 1}} + {h_i}{S_{i + 1}} = 6\left( {f\left[ {{x_i},{x_{i + 1}}} \right] - f\left[ {{x_{i - 1}},{x_i}} \right]} \right)$$

Eq. f[xi,xi+1]=(yi+1−yi)/hi$$f\left[ {{x_i},{x_{i + 1}}} \right] = \left( {{y_{i + 1}} - {y_i}} \right)/{h_i}$$.

The commonly used boundary conditions are: 1)

Natural boundary condition g0″(x0)=gn−1″(xn)=0$${g''_0}\left( {{x_0}} \right) = {g''_{n - 1}}\left( {{x_n}} \right) = 0$$, i.e. S₀ = S_n = 0.

If natural boundary conditions are used, Eq. (10) can be expressed in matrix form, i.e.: (11)[ 2(h1+h2) h2 0 ⋯ 0 0 h2 2(h2+h3) h3 ⋯ 0 0 0 h3 2(h3+h4) ⋯ hn−2 0 ⋮ ⋮ ⋮ ⋮ ⋮ 0 0 0 ⋯ hn−2 2(hn−2+hn−1)] [ S1 S2 ⋮ Sn−1]=6[ f[x2,x3]−f[x1,x2] f[x3,x4]−f[x2,x3] ⋮ f[xn−1,xn]−f[xn−2,xn−1]]$$\begin{array}{c} \left[ {\begin{array}{c} {2\left( {{h_1} + {h_2}} \right)}&{{h_2}}&0& \cdots &0&0 \\ {{h_2}}&{2\left( {{h_2} + {h_3}} \right)}&{{h_3}}& \cdots &0&0 \\ 0&{{h_3}}&{2\left( {{h_3} + {h_4}} \right)}& \cdots &{{h_{n - 2}}}&0 \\ \vdots & \vdots & \vdots &{}& \vdots & \vdots \\ 0&0&0& \cdots &{{h_{n - 2}}}&{2\left( {{h_{n - 2}} + {h_{n - 1}}} \right)} \end{array}} \right] \\ \left[ {\begin{array}{c} {{S_1}} \\ {{S_2}} \\ \vdots \\ {{S_{n - 1}}} \end{array}} \right] = 6\left[ {\begin{array}{c} {f\left[ {{x_2},{x_3}} \right] - f\left[ {{x_1},{x_2}} \right]} \\ {f\left[ {{x_3},{x_4}} \right] - f\left[ {{x_2},{x_3}} \right]} \\ \vdots \\ {f\left[ {{x_{n - 1}},{x_n}} \right] - f\left[ {{x_{n - 2}},{x_{n - 1}}} \right]} \end{array}} \right] \\ \end{array}$$

Due to the high accuracy of the CSI algorithm, when sampling the offshore oil and gas field equipment data, the sampling interval can be appropriately made larger than the minimum sampling interval required by the sampling theorem, and the near-field undersampled data are well recovered by the CSI algorithm, which can be realized by linear interpolation on other occasions.

DDPG is a deep reinforcement learning algorithm suitable for dealing with continuous action control problems. The algorithm combines the advantages of deep Q-networks and Actor-Critic algorithms with offline policy and model-free features [21]. In DDPG, the Actor policy network is responsible for exploring in the environment and outputting action decisions. This is critical for dealing with continuous action control in real-world problems such as rolling bearing life prediction, which typically have highly complex action spaces. The Critic evaluation network evaluates the strengths and weaknesses of each action and provides feedback for gradient updates, which guides the training direction of the Actor policy network.

In the DDPG algorithm, two neural networks of the same structure, Actor and Critic, are the online network and the target network, respectively. When the algorithm runs, the parameters θ_Q and θ_μ of the online Q network and the online strategy network are randomly initialized, and then the corresponding target networks Q′ and μ′ are initialized using the parameters θQ′$${\theta '_Q} $$ and θμ′$${\theta '_\mu }$$, and the expression of the initialization process is: (12)θQ′←θQ,θμ′←θμ$${\theta '_Q} \leftarrow {\theta _Q},{\theta '_\mu } \leftarrow {\theta _\mu }$$

The intelligent body selects action a_t according to the current policy, and the goal of the Actor network is to output the optimal action, this process is accomplished by the online policy network μ(s|θμ)$$\mu \left( {s|{\theta _\mu }} \right)$$ with the expression: (13)at=μ(st|θμ)$${a_t} = \mu \left( {{s_t}|{\theta _\mu }} \right)$$

where s_t is the current state and a_t is the action outputted by the Actor network, which will be used to interact with the environment and obtain a reward.

In the decision making process of the DDPG algorithm, the action a_t is executed, the feedback reward value r_t and the new observed state s_t−1 are obtained, and the interaction experience (st,at,rt,st+1)$$\left( {{s_t},{a_t},{r_t},{s_{t + 1}}} \right)$$ is deposited into the experience pool. When the interaction experience reaches a certain number, N interaction experience is randomly selected from it for training and updating the network parameters.The expression for calculating the objective Q value of the Critic network is: (14)yt=rt+γQ(st+1,μ(st+1;θμ);θQ_)$${y_t} = {r_t} + \gamma Q\left( {{s_{t + 1}},\mu \left( {{s_{t + 1}};{\theta _\mu }} \right);{\theta _{\underset{\raise0.3em\hbox{$\smash{\scriptscriptstyle-}$}}{Q} }}} \right)$$

The Critic network is updated by minimizing the TD bias, which is the difference between the current estimate and the target value and represents the error of the Critic network on the value of the action, and the update expression is: (15)L(θQ)=1N∑i=1N(Q(si,ai|θQ)−yi)2$$\mathcal{L}\left( {{\theta _Q}} \right) = \frac{1}{N}\sum\limits_{i = 1}^N {{{\left( {Q\left( {{s_i},{a_i}{\theta _Q}} \right) - {y_i}} \right)}^2}}$$

This process enhances the Critic network’s accurate estimation of action values, which provides more reliable gradient information for the Actor network and promotes the optimization and improvement of strategies. It is through the Actor-Critic structure that the entire DDPG algorithm iteratively optimizes the policy and value functions to achieve efficient learning in continuous action control problems.

The Actor policy network updates the network parameters based on the policy gradient with the goal of maximizing the long-term cumulative rewards, and the update expression is: (16)∇θzJ≈1N∑i=1N∇aQ(si,a|θQ)∇θjμ(si|θμ)$${\nabla _{{\theta _z}}}J \approx \frac{1}{N}\sum\limits_{i = 1}^N {{\nabla _a}} Q\left( {{s_i},a|{\theta _Q}} \right){\nabla _{{\theta _j}}}\mu \left( {{s_i}|{\theta _\mu }} \right)$$

where N is the number of samples drawn, ∇aQ(si,a|θQ)$${\nabla _a}Q\left( {{s_i},a|{\theta _Q}} \right)$$ is the gradient of the Critic network with respect to the action, and ∇θμμ(si|θμ)$${\nabla _{{\theta _\mu }}}\mu \left( {{s_i}|{\theta _\mu }} \right)$$ is the gradient of the Actor network with respect to the policy parameters.

The Critic’s expression for updating the target network is: (17)θQ′←τθQ+(1−τ)θQ′$${\theta _{Q'}} \leftarrow \tau {\theta _Q} + (1 - \tau ){\theta _{Q'}}$$

where θQ′$$\theta_{Q'}$$ is the parameter of the target Critic network, θ_Q is the parameter of the online Critic network, and τ is the hyperparameter of the soft update.

Actor’s target network update expression is: (18)θμ′←τθμ+(1−τ)θμ′$${\theta _{\mu '}} \leftarrow \tau {\theta _\mu } + (1 - \tau ){\theta _{\mu '}}$$

where θμ′$${\theta _{\mu '}}$$ is the parameter of the target Actor network and θ_μ is the parameter of the online Actor network.

DDPG adopts a soft update method, so that the parameters of the target network maintain a certain degree of smoothness when updating, avoiding too drastic changes, and the whole DDPG algorithm realizes the joint learning of the action value and the strategy by means of this update method of the target network.

Sparse Noise Abatement Self-Encoder (SDAE) consists of two parts: encoder and decoder. SDAE firstly adds noise to the original input data to get the damaged data samples, the encoder performs feature extraction on the damaged data samples, and the decoder reconstructs the extracted higher-order features into the undamaged data, and this training process greatly improves the model’s noise immunity. The higher-order features extracted by the SDAE are very robust The high order features extracted by SDAE are extremely robust, and are able to express the high-dimensional original features in an abstract way without being disturbed by noise, so the SDAE model has a very good noise immunity [22].

The SDAE first “destroys and contaminates” the original input data sample x ∈ R^m to obtain the damaged data sample x^∈Rm$$\hat x \in {R^m}$$, and then the encoder maps x^$$\hat x$$ to the hidden layer vector z ∈ R^h by the following formula. i.e: (19)z=fφ(x^)=11+e−Wx^+B$$z = {f_\varphi }(\hat x) = \frac{1}{{1 + {e^{ - W\hat x + B}}}}$$

Where W ∈ R^{h × m} is the weight matrix connecting the input and hidden layers and B ∈ R^{h × 1} is the bias vector matrix connecting the input and hidden layers.

The decoder maps the hidden layer vector z to an output layer of the same dimension as the input layer by the following equation to obtain the reconstructed original input feature x′. i.e: (20)x′=qα(z)=11+e−W′z+B′$$x\prime = {q_\alpha }(z) = \frac{1}{{1 + {e^{ - W^\prime z + B^\prime }}}}$$

where W′ ∈ R^{m × h} is the weight matrix connecting the hidden and output layers and B′ ∈ R^{m × 1} is the bias vector matrix connecting the hidden and output layers.

The goal of SDAE training is to reconstruct the damaged samples x^$$\hat x$$ into the undamaged samples x′, which is achieved by minimizing the following equation, which is used to measure the error between the damaged and undamaged samples. Namely: (21)Lreconstnction=1m∑i=1m(xi′−xi)2$${L_{reconstnction}} = \frac{1}{m}\sum\limits_{i = 1}^m {{{\left( {{{x_i'}} - {x_i}} \right)}^2}}$$

where xi′$${{x_i'}}$$ is the ind dimensional feature quantity of the reconstructed input features and x_i is the ith dimensional feature quantity of the undamaged original input features.

The correlation between information such as fault characteristics of offshore oil and gas field equipment leads to a large number of redundant features in the high-dimensional original input information. Adding sparsification constraints to the SDAE model can force the SDAE model to automatically remove redundant features during the training process and extract features with high differentiation for equipment fault diagnosis, thus improving the performance of equipment fault diagnosis.

In this paper, we construct a deep mapping network F based on the DDPG reinforcement learning framework combined with a sparse noise-reducing self-encoder to reduce the domain distribution differences and enhance the cross-domain invariance of features. By adjusting the hyperparameter set {L, N} of the deep domain adaptive network, the source and target domains can be synchronously projected into different deep hidden layers (common space) and produce different domain adaptive effects. Where L denotes the number of hidden layer layers of the deep domain adaptive network, and N denotes the additive noise rate (i.e., random zero probability).

In order to better realize the fault diagnosis of offshore oil and gas field equipment, this paper proposes a deep domain adaptive fault diagnosis model based on DDPG-SDAE, which evaluates the cross-domain adaptability of the deep mapping network through a comprehensive evaluation function, which consists of a domain difference penalty term and a class distance penalty term.

First, the MMD metric is chosen to measure and minimize the distributional variability between two domains, i.e: (22)MMD(Xs,Xt)=‖1ns∑i=1nsF(xsi)−1ns∑j=1nsF(xtj)‖$$MMD\left( {{X_s},{X_t}} \right) = \left\| {\frac{1}{{{n_s}}}\sum\limits_{i = 1}^{{n_s}} F \left( {x_s^i} \right) - \frac{1}{{{n_s}}}\sum\limits_{j = 1}^{{n_s}} F \left( {x_t^j} \right)} \right\|$$

where MMD(Xs,Xt)$$MMD\left( {{X_s},{X_t}} \right)$$ is the edge distribution difference between the source domain feature set X_s and the target domain feature set Xt(P(Xs)≠P(Xt))$${X_t}\left( {P\left( {{X_s}} \right) \ne P\left( {{X_t}} \right)} \right)$$, n_s and n_s denote the number of samples in the source and target domains, respectively, and F denotes the mapping matrix, which in this paper refers specifically to a t-layer SDAE network, i.e: (23)F=ht=tanh(Wth)t≥1$$F = {h_t} = \tanh \left( {{W^t}h} \right)t \geq 1$$

From the above equation, it is clear that the difference between the distributions of two domains can be assessed empirically in terms of the distance between the centers of their samples. Obviously, the larger the MMD, the greater the difference between the two distributions, and conversely, the closer the MMD is to 0, the more similar the two distributions are. Therefore, the domain difference penalty term is defined as: (24)JMMD=argminMMD(Xs,Xt)$${J_{MMD}} = \arg \min MMD\left( {{X_s},{X_t}} \right)$$

Then, a class distance metric is introduced in the source domain samples to ensure that the recognizability information of DDPG reinforcement learning is maximally preserved in the feature mapping. Assuming a n classification problem, x^k denotes the input features belonging to class k faults, and ht(xk)$${h_t}\left( {{x^k}} \right)$$ denotes the features after mapping to the common space (the output of SDAE layer t). The expectation and variance of ht(xk)$${h_t}\left( {{x^k}} \right)$$ are denoted as E[ht(xk)]$$E\left[ {{h_t}\left( {{x^k}} \right)} \right]$$ and Var[ht(xk)]$$Var\left[ {{h_t}\left( {{x^k}} \right)} \right]$$, respectively.Thus, the intra-class distance, i.e., the distance between samples of the same class, can be computed as: (25)Dintra=∑i=1nVar[ht(xk)]$${D_{\operatorname{int} ra}} = \sum\limits_{i = 1}^n {\sqrt {Var\left[ {{h_t}\left( {{x^k}} \right)} \right]} }$$

The class spacing, i.e. the distance between the samples of class i and class j, can be calculated as: (26)Dinter = ‖E[ht(xi)]−E[ht(xj)]‖2 −Var[ht(xi)]−Var[ht(xj)]$$\begin{array}{rcl} {D_{\operatorname{int} er}} &=& {\left\| {E\left[ {{h_t}\left( {{x^i}} \right)} \right] - E\left[ {{h_t}\left( {{x^j}} \right)} \right]} \right\|_2} \\ &&- \sqrt {Var\left[ {{h_t}\left( {{x^i}} \right)} \right]} - \sqrt {Var\left[ {{h_t}\left( {{x^j}} \right)} \right]} \\ \end{array}$$

where ‖⋅‖2$${\left\| \cdot \right\|_2}$$ denotes the Euclidean distance, 1 ≤ i ≤ j ≤ n. D_intra and D_inter are used to denote the intraclass compactness and interclass separateness of the samples in the source domain, respectively. To facilitate categorization, samples within the same class should be as close as possible, while samples within different classes should be as far away as possible. Therefore, the class distance penalty term minimizing D_intra while maximizing D_inter can be calculated as: (27)Jclass=argmin(λDintra−Dinter)$${J_{class}} = \arg \min \left( {\lambda {D_{\operatorname{int} ra}} - {D_{\operatorname{int} er}}} \right)$$

Finally, Eqs. (24) and (27) are integrated to obtain the final evaluation function as: (28)Jfinal = argmin(μJMMD+Jclass) = argmin(μJMMD+λDintra−Dinter)$$\begin{array}{rcl} {J_{final}} &=& \arg \min \left( {\mu {J_{MMD}} + {J_{class}}} \right) \\ &=& \arg \min \left( {\mu {J_{MMD}} + \lambda {D_{\operatorname{int} ra}} - {D_{\operatorname{int} er}}} \right) \\ \end{array}$$

where μ and λ are non-negative scaling coefficients, J_MMD is used to correct the overall displacement of the feature space, and J_class is used to retain more distinguishable clustering regions. The evaluation function can quantitatively measure the effect of domain adaptation, so it can be used to search for optimization in the preset parameter space by the grid search method, and select the optimal parameter set {L,N}$$\left\{ {L,N} \right\}$$ to obtain the best cross-domain learning effect.

The flowchart of offshore oil and gas field equipment fault diagnosis based on DDPG-SDAE algorithm is shown in Fig. 2, and its specific realization steps are as follows:

Figure 2.

Fault diagnosis process based on DDPG-SDAE

Step1 Collect multi-source heterogeneous data of offshore oil and gas field equipment, make samples, and add labels to the samples.

Step2 Preprocess the samples and divide them into training set and test set in the ratio of 7:3.

Step3 Set the parameters in the network. For example, the number of neurons in each layer, the number of iterations, the sparse penalty term parameter, the edge restriction parameter and so on.

Step4 Take the training set as the input of the first layer network and train the first MSDA network.

Step5 Take the output of the hidden layer of the DDPG front network as the input of the next layer network and train this layer network. Iterate the operation in this way until the last layer is trained.

Step6 Take the output of the last hidden layer of the SDAE network as the input of the SoftMax classifier and use the raw data labels to train the network parameters of SoftMax.

Step7 Use the obtained initialized network parameter values as the initial values of each parameter in the model, and then fine-tune the whole network by back-propagation algorithm to complete the training.

Step8 The test samples are input into the fine-tuned DDPG-SDAE network, and then the extracted features are input into the trained SVM classifier to calculate the fault diagnosis rate of the algorithm.

In order to evaluate the performance of the proposed PSO-NLM algorithm in performing fault feature extraction for offshore oil and gas field equipment, a periodic Gaussian pulse signal at 90Hz with a bandpass of 50% and a repetition frequency of 15Hz with random noise is used for analysis in this paper. The sampling frequency of the signal is 120 kHz, and the width of the pulse train is 2 s. The composite signal and the Gaussian pulse train are shown in Fig. 3, which shows that the composite signal is dominated by noise, and the Gaussian pulse train is almost completely submerged in it.

Figure 3.

Nose-free pulse signal

The PSO-NLM algorithm proposed in this study is used for filtering and noise reduction of the composite signals.In order to verify the effectiveness and superiority of PSO-NLM, two commonly used fast filtering algorithms, i.e., median filtering (MF) and wavelet thresholding noise reduction (WTD), are also used to compare with the proposed PSO-NLM method, taking into account that the PSO-NLM is an improvement on the NLM method. Therefore, NLM is also used to analyze the original signal and compare it with the proposed method. In the noise reduction process, the unique parameter window length of median filtering is selected as 35, the optimal predictor variable threshold selection is used in wavelet threshold filtering, soft threshold noise reduction method, sym3 wavelet basis function is selected, the number of decomposition layers is set to 4, the structural block half-width of 90 is selected for NLM and PSO-NLM, the search region half-width of 6000, and the filter parameter is set to 1. The denoising effect of different algorithms is shown in Fig. 4. Where Fig. 4(a)~(d) shows the feature extraction effect of MF, WTD, NLM and PSO-NLM, respectively. As can be seen from the figure, all four methods can filter out a large amount of noise and highlight the Gaussian pulse sequence. The median filtering denoises the most remaining noise, while wavelet thresholding denoises the waveform with serious distortion. NLM and PSO-NLM denoise the Gaussian pulse sequence more purely and with less distortion.

Figure 4.

The noise extraction effect of different algorithms

In order to quantitatively compare the denoising performance of the four filtering methods, signal-to-noise ratio (SNR) and mean absolute error (MAE) are introduced as signal denoising evaluation indexes. White noise from -10dB to 20dB SNR is sequentially added to the original Gaussian pulse sequence, and MAE is used as a quantitative index to examine the denoising performance of median filtering, wavelet threshold filtering, NLM and PSO-NLM in different noise environments. Figure 5 shows the feature extraction effect of different algorithms under different signal-to-noise ratios.

Figure 5.

MAE of various algorithms under SNR

It can be seen that the PSO-NLM algorithm has high filtering accuracy under various signal-to-noise ratios, especially at low signal-to-noise ratios, and still has good robustness. When the signal-to-noise ratio is increased from -10dB to 20dB, the feature extraction MAE of PSO-NLM algorithm is reduced from 0.574 to 0.081. In addition, the running time of the different algorithms is counted in this paper, and the four algorithms have a running time of less than 0.8s, which can be used as a feature extraction method for fault diagnosis in offshore oil and gas fields. Among them, the running time of PSO-NLM algorithm is only 0.051s, which is about 81.48% higher than that of NLM algorithm, and has the optimal running efficiency. Therefore, the PSO-NLM algorithm is applied to extract fault features from offshore oil and gas field equipment, which has better extraction efficiency.

In order to ensure the integrity of multi-source heterogeneous data of offshore oil and gas field equipment, this paper introduces the CSI algorithm to interpolate and fill them. In order to further understand the effect of this algorithm for interpolation filling, this paper carries out simulation verification of the nearest neighbor interpolation, linear interpolation, cubic interpolation and CSI algorithm with corresponding simulation signals.

The given simulation signal S(t) is superimposed by two signals S₁ and S₂, where S₁ is a sinusoidal signal with a frequency of 20 Hz, and S₂ is a cosine FM signal with a frequency range of 0 to 20 Hz. Fig. 6 shows the corresponding time-domain waveform result of the simulation signal S(t), which has a period of 2 s and an amplitude range of ±3 m/s².

Figure 6.

Simulation signal time domain waveform

Based on the simulated signal, the nearest neighbor interpolation (CI), linear interpolation (LI), cubic interpolation (CUI), and CSI algorithms are used to obtain the corresponding upper and lower envelope effects of the simulated signal as shown in Fig. 7, in which Figs. 7(a)~(d) show the interpolation results of CI, LI, CUI, and CSI, respectively. According to the simulation effect diagrams obtained for several interpolation methods, their effects are analyzed and evaluated in terms of smoothness and similarity. 1)

Smoothness. From the theoretical analysis, the so-called “smoothness” is also known as the degree of smoothness, which is quantitatively described mathematically as the existence and continuity of the k-order derivatives of a function or a curve, which is called the curve with k-order smoothness. The higher the order, the better the smoothness. Closest Interpolation, Linear Interpolation are all zero order smoothness, i.e. not smooth, Cubic Interpolation and CSI algorithm are interpolation methods that achieve more off-order smoothness by utilizing polynomials of lower number of times. Therefore, theoretically speaking, the order of the above four interpolation methods according to the degree of smoothness from low to high should be CI<LI<CUI<CSI, according to the simulation verification of the effect of comparing the degree of smoothness of the upper and lower envelopes in the figure, it can be seen that the CSI algorithm obtains the relative best smoothness of the envelope, cubic interpolation is second to the closest, linear interpolation method is the worst. The simulation results are in good agreement with the theoretical analysis results.

Figure 7.

The upper and lower envelope of different algorithms

Rolling bearings, the most common component in offshore oil and gas field equipment, are used as an example to analyze the effectiveness of this paper’s method. The rolling bearing data collected from the Electrical Engineering Laboratory of CWRU University was selected to be the identification sample. The localized damage of the bearings was created with an electric spark machine to obtain no fault (Q1), outer ring faulty bearings (Q2), inner ring faulty bearings (Q3) and rolling element faulty bearings (Q4), respectively. The signals of the four fault states are filled by feature selection and interpolation using the PSO-NLM algorithm and CSI algorithm given in the previous section, and a total of 470 groups of no-fault bearing data are obtained, and 320, 180 and 120 groups of rolling body fault, inner ring fault and outer ring fault data are obtained respectively. Among them, the first 370 groups of fault-free data, the first 200 groups of rolling body fault data, the first 100 groups of inner ring fault data and the first 60 groups of outer ring fault data are used to form the training data, and the remaining data are used to test the recognition rate of the model. The G-DBM and PSO-G-DBM models are selected as the comparison models, and the fault diagnosis results of different models are obtained as shown in Fig. 8, where Figs. 8(a)~(c) show the fault diagnosis results of the G-DBM, PSO-G-DBM and DDPG-SDAE models, respectively.

Figure 8.

Fault diagnosis of different models

Based on the fault diagnosis confusion matrix of different models, it can be seen that after noise reduction and interpolation to fill in the signal, different types of equipment fault recognition is better, the three models for the normal state of the bearing signal recognition rate is higher than 90%, and the diagnostic accuracy of the DDPG-SDAE model designed in this paper can reach 98.95%. For the two categories of outer ring faults and rolling body faults, the diagnostic accuracy of G-DBM and PSO-G-DBM models are lower than 90%, and the accuracy of the fault diagnostic model designed in this paper reaches 92.64% and 94.84%, respectively. This fully demonstrates that the offshore oil and gas field equipment fault diagnosis model proposed in this paper after combining deep reinforcement learning has a better diagnostic effect, which can better help the offshore oil and gas field equipment overhaul and guarantee the safety of offshore oil and gas field equipment.

The DDPG-SDAE model designed in this paper has cross-domain and cross-operating condition fault diagnosis capability, in order to validate the fault diagnosis migration effect of the model, this paper takes three different types of equipment fault datasets of an offshore oil and gas field platform as an example, and designs a total of nine source-domain and target-domain migration tasks, as shown in Table 1, by considering the single-sensor migration task and the multi-sensor information fusion migration task. In the test, in order to construct the source and target domain tasks, any one sensor data is taken as the source domain data and the other two sensor data as the target domain data when performing the single source domain migration task. A total of six single-sensor migration tasks were obtained due to the differences in distribution between the target domain test data and the source domain training data. In multi-source domain migration, in order to fully utilize the acquired historical data from multiple source domains, any two of the three sensors’ acquired sensor data are treated as the source domain data while the remaining one is treated as the target domain dataset in order to perform the multi-sensor fusion migration task, thus a total of three multi-sensor migration tasks are obtained.

In order to verify the effectiveness and superiority of the proposed method, the proposed method is tested on single-sensor migration diagnosis and multi-sensor migration tasks to compare with CNN and shallow migration DACNN models. Fig. 9 shows the fault diagnosis migration results of different models, where Fig. 9(a)~(b) shows the results of single-sensor and multi-sensor migration tasks, respectively.

Figure 9.

Fault diagnosis migration results of different models

On the single-sensor migration task, for CNN, the classification performance of different sensors shows large fluctuations, such as the migration task 1→3 from sensor 1 to sensor 3 and 2→3 from sensor 2 to sensor 3 whose diagnostic accuracies are 86.96% and 97.82%. While in 2→1 and 3→1, its diagnostic accuracy is only 73.07% and 64.92% respectively. Therefore, the fault occurs inside the planetary gear, and the signal is transmitted to the case surface through gear meshing and shaft vibration, which is picked up by the sensor. Affected by the vibration signal transmission path, the information obtained by the sensor from different locations is obviously different, which leads to a large distribution difference in the data obtained by different sensors. It can be seen that the diagnostic accuracy of sensor 2 to sensor 3 is significantly higher than that of sensor 1 to sensor 3, while the diagnostic accuracy of sensor 2 to sensor 1 is also significantly higher than that of sensor 3 to sensor 1. It shows that the closer the installation distance of two sensors is, the more conducive to obtaining more similar diagnostic information, and the difference between the two data distributions is relatively smaller, which in turn is conducive to obtaining better generalization ability. In addition, it can be found that both the DACNN and the proposed method using the migration learning technique have significantly higher classification accuracy than the CNN. Comparing several other migration tasks, the lowest diagnostic accuracy of the proposed DDPG-SDAE model can reach 96.77% (3→1), while the highest diagnostic diagnostic accuracy can reach 99.05% (3→2). Compared with the shallow migration CNN, its fault diagnosis accuracy is improved by about 10~32 percentage points. It shows that the model in this paper can significantly improve the migration capability under different sensor data by utilizing unlabeled datasets in the target domain and minimizing the classifier output error to obtain more stable and higher diagnostic accuracy. In addition, on the multi-sensor migration task, the classification accuracy of this paper’s model is more competitive compared to the single-sensor results, and the highest classification accuracy is obtained by combining deep reinforcement learning, domain adaptation techniques, and minimizing the classifier output.