Research on big data visualization in the talent cultivation guarantee system of social sports instruction and management oriented to the concept of OBE

Scrapy is a powerful Python-based open source web crawler framework designed to help users efficiently and quickly crawl data from web pages. The framework provides a suite of tools and features that enable users to easily build, manage, and extend web crawlers. Scrapy’s flexibility and customizability have made it the tool of choice for many data acquisition projects. The framework provides crawling data functions and rich expansion interface, can meet this paper to build talent profiling system for talent such as papers, patents and other information acquisition needs. The framework can be customized and extended according to your own needs.

Scrapy framework consists of the following modules: 1)

Engine: the engine is used to handle the data flow processing of the whole system, trigger transactions, and is the core of the whole framework.

K Mean clustering algorithm is very commonly used algorithm. Its algorithmic ideas have evolved with the times, but the most basic ideas have not changed much [25]. The following is a simple description of the most basic K-means method using its computational process. There are mainly several steps as follows:

Step 1: We select K samples in the data space as the initial center, where each object represents a clustering center;

Step 2: For the data in the sample set, their belonging is determined by calculating their distances from these clustering centers. We assign them to the class corresponding to the clustering center closest to them according to the closest distance criterion;

Step 3: Update the clustering centers by taking the mean value corresponding to all the objects in each category as the clustering center for that category, calculating the value of the objective function, and then continuing to put in the samples to calculate their distances to continue to classify them according to the closest distance and recalculate the clustering centers;

Step 4: Repeat step 2 and then for each iteration to determine whether the clustering center and the value of the objective function has changed, know that the clustering center and the value of the objective function no longer change, and then output the results.

K-means algorithm is often used because of the relatively simple operation method, the principle is easy compared with other clustering methods, the operation speed is fast and other advantages, so this clustering algorithm is often used by us. However, the K-means algorithm has some disadvantages and shortcomings while having these advantages. For example, the most primitive calculation method has a great deal of randomness in the selection of the initial center, and the K-means method is highly dependent on the selection of the initial center. The selection of the initial clustering center has a great influence on the results of the final cluster analysis. When our initial clustering center is not chosen appropriately, it may cause the algorithm results to be unstable, so that the final clustering results are likely to be incorrect, and it is highly likely that it will converge to the local optimum of the clusters instead of the global optimum. Similarly, the choice of K-value is often based on the user’s personal experience, and the choice of K-value has a significant impact on the final clustering result. When different values of K are chosen for the same sample dataset, the clustering results may be very different, which will have a great influence on the final result, and the selection of K values will largely determine the clustering results. Therefore, most of the improved K-means algorithms make relevant improvements in these two directions to improve the K-means algorithm by choosing more appropriate initial centers and more appropriate K values, and then get better results.

The K-means algorithm with improved initial clustering centers is a related improvement of the initial clustering centers used to reduce the uncertainty generated by the random selection of clustering centers. In the following, we first introduce the relevant parameters for calculating the initial clustering centers for the improved K-means method. For dataset D = {x₁, x₂, x₃, …, x_n}, the first thing we do is to calculate the mean of the distance between the overall samples, denoted as AveDist, where n is the number of samples [26]. This is calculated as follows: (2)AveDist=2n(n−1)∑i=1n−1∑j=i+1nd(xi,xj)

where dist(x_i, x_j) is the Euclidean distance between two points, and both x_i and y_i are m-dimensional data. The calculation method is as follows: (3)dist(xi,xj)=||xi−xj||=(xi1−xj1)2+......+(xim−xjm)2

where m is the dimension of the sample point in question.

The number of sample points contained within each sample neighborhood is defined as follows: in a m-dimensional space data set D = {x₁, x₂, x₃, …, x_n} has been given. For each data, the determination of the number of points within its associated neighborhood is determined by the following equation: (4)ρ(xi,ε)=∑1≤i≤n,1≤j≤n,i≠jsgn(ε−dist(xi,xj))

where the sgn function is: (5)sgn(x) = 1,x≥0 sgn(x) = 0,x<0

The result of ρ(x_i, ε) is the number of samples in the sample x_i neighborhood is also called the density parameter. The ε is the domain radius, whose value is generally the mean of the distance between the overall samples, and can also be taken ε = α × AveDist, 0 < α ≤ 1 when the direct use of the mean of the distance does not yield a large enough clustering K-value, of which is in the case of obvious classification boundaries, α usually take the value of 0.3, 0.4, 0.5. The specific value is chosen experimentally by the desired K value.

In the above calculation we can get the density parameters for each sample. Through these density parameters we can make relevant improvements to the K-means method. Its method of selecting the initial clustering center is through the relevant density parameters calculated above. Here we sort the ρ(x_i, ε) calculated for each sample x_i from largest to smallest, and first select the x_i with the largest sample density as the first initial clustering center. Then remove x_i and all the samples within its neighborhood and find the remaining samples corresponding to the maximum density parameter of x_i and remove all the points within its neighborhood, and continue to search downward. Continue down the list until we find the cluster centers we need. When the final number of clustering centers is less than the required number of clustering centers, we can appropriately reduce the value of α and then select.

Based on the above description here I provide a systematic description of how the improved K-means method performs the selection of initial clustering centers:

Step 1: Calculate the distance dist(x_i, x_j) between two pairs of data in the dataset and then calculate the associated average distance by using the AveDist formula;

Step 2: Calculate the ρ(x_i, ε) of each sample after choosing the appropriate ε parameters for i = 1, 2, 3, …, n. Place it in dataset S={ρ(xi,ε),i=1,2,3,...,n} ;

Step 3: Select the largest ρ(x_i, ε) corresponding to x_i from dataset S as the first initial clustering center. Then eliminate all samples within the ε neighborhood of x_i. Then continue to select the largest ρ(x_i, ε) corresponding samples from the remaining sample points as the second clustering center;

Step 4: Keep repeating step (3) to know that all elements in S are eliminated or the number of initial clustering centers required for clustering is reached;

Step 5: The selected initial clustering center is noted as V = {v₁, v₂, v₃, …, v_K} where K is the number of classes of the required clusters. Then use the selected V as the initial clustering center to calculate the distance between each element in the sample set and the initial clustering center. The one with the smallest distance is selected as grouped into one class;

Finally then update the clustering centers vj=1|Cj|∑x∈Cjx repeating step (5) knowing that each v_j is not changing. The stabilized result is the clustering result obtained based on the improved K-means method.

The data of this experiment came from the social sports college of a university, containing five samples of social sports guidance and management talents since graduation to date of the various indicators of data, as shown in Table 1. Cluster analysis was carried out by using the parallelized K-Means algorithm according to the weight table of each index determined in Chapter II.

The first step of the K-Means algorithm is firstly the selection of the number k of class clusters, which has a great influence on the later clustering process and the final clustering results, and it can be said that whether the selection of the k value is appropriate or not directly determines whether the final clustering results are of practical significance. In this experiment, the choice of k-value determines whether the studied results have real application value for the development of social sports guidance and management talents, optimizing education and teaching, and improving the quality of talent training.

As far as the current domestic and international research results in K-Means clustering are concerned, there is no recognized and authoritative k-value selection strategy proposed. When the number of hypothetical class clusters is equal to or higher than the real number of class clusters, the indicator will rise slowly, and once trying to get less than the real number of class clusters, the indicator will rise sharply. That is, the critical point k value is the optimal number of class clusters. In this experiment, this method is used for the determination of k. The chosen metric is the sum of the center-of-mass distances of the k class clusters.

In this paper, the cases where k is selected for each value from 1 to 10 are compared separately, and the results are shown in Fig. 3.

Figure 3.

K value selection comparison

It can be seen that when k is taken as 10, 9, 8, 7 and 6 in turn, the bar graph shows a rather slow increase, and the change of the slope of the corresponding value of k in the curve graph is not obvious, so it is considered that the real suitable for the experimental data should be taken as k is less than 6. Again, when k is taken as 5 and 4, the increase and the slope in the two graphs have a more obvious change. When k is 3, 2 and 1, the increase in the bar graph rises sharply and the slope in the curve graph changes significantly, therefore, the value of k should be greater than 3. From this, we can draw the result that it is more appropriate to choose the value of k as 4 or 5 in this experiment.

First of all, from the overall clustering results, the comprehensive quality of all the social sports guidance and management talents of the college ranges from 63 to 85 points, and the number of social sports guidance and management talents with more than 80 points is relatively small, so it can be seen that the overall quality level of the social sports guidance and management talents is in the middle of the range, and there are no social sports guidance and management talents with especially outstanding comprehensive ability to be evaluated by the new index system at the present time. The new index system shows that there are no social sports guidance and management talents with outstanding comprehensive ability. Combined with the actual situation of social sports guidance and management talents, it is also more reasonable to see that most of the school’s professional courses are concentrated in the second semester of the sophomore year and the first semester of the junior year, and the professional specialties of the general social sports guidance and management talents are only gradually highlighted after the three samples.

From the results of the four categories of clustering a, b, c, d, the five first-level quality indicators of the practical innovation quality of the biggest float, the humanistic quality of the smallest float. It shows that the ability of social sports guidance and management talents in practice and innovation (social practice ability, scientific and technological innovation ability, academic research ability, community work ability) is relatively large differences, some social sports guidance and management talents in this aspect of the weighted score almost reached the full score, but there is a part of the social sports guidance and management talents scores do not reach the pass mark, the grade gap is large. This shows that the school still pays more attention to the quality of practice and innovation of social sports guidance and management talents, has carried out the corresponding promotion work, and has achieved certain positive results, or most of the social sports guidance and management talents have realized the importance of this quality, and some of them have already strengthened this aspect of the exercise through specific practice and scientific and technological competitions, etc., and achieved good results. And achieved good results. The effectiveness of this quality is gradually showing, and it is necessary to continue to insist on and improve the cultivation program of practical innovation quality.

The following is a description of the results of each category of the chart to show the general range of scores and total scores of the various qualities of various types of social sports guidance and management personnel, and to make a specific analysis. 1)

Class a

Table 2 shows the clustering results of category a. Category a has the largest number of social sports guidance and management talents, with more than 90 people, accounting for about one-third of the total number of social sports guidance and management talents, and the overall quality level is also 60 to 84 points, encompassing the largest range of scores, which shows that the scores basically meet the standard, but the scores are mostly fluctuating between 60 to 100 points, with no particularly outstanding performance, and a very small number of A very small number of students were below the passing score. It can be said that this part of the social sports guidance and management talents can basically serve as a generalization of the quality of social sports guidance and management talents in the sample, presenting basically an overall educational effect and characteristics of the sample.

Firstly, the clustering results of equity-based social sports guidance and management talents this section mainly focuses on the sample data that has been data standardization, based on the principal component analysis method to carry out a comprehensive evaluation, firstly, according to the correlation matrix to test whether the data is suitable for principal component analysis, as shown in Figure 4.

Figure 4.

Variable correlation analysis

According to the correlation analysis matrix it can be seen that there is a phase relationship between the 5 variables, which is suitable for principal component extraction.

The feasibility of its principal component analysis was further analyzed by KMO and Bartlett’s test, as shown in Table 6:

According to the KMO and Bartlett’s test, the KMO value, if higher than 0.8, indicates that it is very suitable for analysis; between 0.7 and 0.8, it indicates that it is more suitable for analysis; if this value is between 0.6 and 0.7, it indicates that it can be analyzed; and if this value is less than 0.6, it indicates that it is not suitable for principal component analysis. Bartlett’s test of sphericity corresponds to a p-value of less than 0.05 passes the test, which also indicates that it is suitable for principal component analysis.

In this case, KMO value = 0.861 and Bartlett’s test corresponding to p-value < 0.05 both indicate that the case data is suitable for principal components.

Principal component analysis is calculated based on the correlation coefficient matrix or covariance matrix, and the eigenvalues or eigenroots are important concepts of the matrix. According to the eigenroot, the proportion of variance contribution of each principal component can be calculated (or called the variance explanation rate, the same below), the variance contribution rate refers to the proportion of the variance explained by the principal component to the total variance; the larger the value, the stronger the ability of the principal component to synthesize the information of the original variables, this paper is based on the total variance explanation of the data of the talent of the social sports instruction and management is specific as shown in Table 7.

This paper selects the principal components with eigenvalues greater than 1 to carry out principal component analysis, as can be seen from the above table: the principal component analysis will construct 3 principal components (the first 3 components to do the cumulative variance calculation), the eigenroot value is greater than 1, in order of 4.305, 2.976, 2.238. The variance of the 3 principal components of the variance explained rate is 47.156%, 15.641%, 14.584%, respectively, and the cumulative variance Explanation rate is 77.381%, the cumulative variance explanation rate is close to 80%, the information of the original index is lost less, the effect of principal component analysis is more ideal, and it has research significance.

Based on the matrix of component score coefficients it is possible to analyze the information composition of the three principal components that have been identified, as shown in Table 8.

Based on the results of the composite score, the following frequencies of social sports instruction and management talents’ scores were derived as shown in Table 9. It can be seen that the number of social sports guidance and management talents with a comprehensive rating of one to five stars are 11, 68, 121, 74 and 26 respectively.