Research on Data Compression and Efficient Transmission Technology in the Framework of Big Data Processing

This paper mainly focuses on the problem of compression ratio caused by dictionary storage structure and dictionary updating method, and improves LZW. Then use the gray correlation to analyze the physical attributes between big data, select the data with strong correlation, use BP neural network to train the model, and put the trained model into the Internet terminal, so as to realize the efficient transmission for data fusion. After simulation experiments and algorithm efficiency tests, it can be seen that the compression time is reduced relative to the LZSS and LZW algorithms, and the improved LSW algorithm’s static data compression rate is compression rate of 50.75%, and the compression rate of triggering class data is 9.07%, and it is also very helpful in saving network bandwidth. With the increase of nodes, the fusion algorithm has good performance in terms of delay and network life cycle. When the nodes are 500, its life cycle reaches 1.8 × 10⁷. Therefore, the algorithm in this paper is suitable for the application scenario of data compression and efficient transmission of big data.