Research on the Integration and Optimization Strategy of Multimodal Learning Resources under the Online-Offline Fusion Teaching Mode
Publié en ligne: 26 sept. 2025
Reçu: 01 févr. 2025
Accepté: 11 mai 2025
DOI: https://doi.org/10.2478/amns-2025-1082
Mots clés
© 2025 Ruxuan Li, published by Sciendo
This work is licensed under the Creative Commons Attribution 4.0 International License.
With the rapid development of information technology, people’s demand for education is also changing. Traditional classroom teaching can no longer meet the needs of students, and more and more schools have begun to implement online and offline integrated teaching mode. The online-offline integrated teaching mode is a new teaching mode combining traditional teaching mode and modern information technology [1-4]. This teaching mode makes full use of the advantages of information technology and teaching resources, integrates traditional teaching with online teaching, and forms an educational mode with wider coverage and more diversified teaching methods [5-6]. In this teaching mode, the integration and optimization strategy of multimodal learning resources is of great significance.
With the rapid development of Internet technology, the access to learning resources has become more convenient and abundant. However, the diversity and complexity of learning resources also bring certain troubles to learners. In this context, the optimization and integration of learning resources become particularly important [7-10]. The integration of learning resources is to better meet the needs of teachers and students and improve the effectiveness of education and teaching. Traditional classroom teaching often use some textbooks, teaching PPT and other teaching resources, but the limitations of this teaching method is relatively large, we should take advantage of modern science and technology, the integration of information technology and education teaching, integration of learning resources, to create a personalized, diversified, out-of-control education and teaching atmosphere [11-14]. The optimization of learning resources is to improve the effectiveness of teaching resources. The optimization of resources can make the teaching resources closer and closer to the students’ concepts, so as to maximize the teaching effect [15-17].
In order to realize the efficient integration of multimodal learning resources, this paper constructs an intelligent platform for multimodal learning resources, and completes the functional construction of the intelligent platform from the aspects of multimodal learning knowledge graph, knowledge tracking and learning path recommendation. In terms of multimodal learning knowledge graph, Dict, a dictionary resource, is integrated on BBLC model to enhance the accuracy of knowledge graph partitioning, and Bert model is used to train the discipline-related text data of learning resource database and calculate the CRF layer objective function. The overall framework of the knowledge tracking model is formulated from the embedding representation module, encoder, decoder, and prediction result representation module. Positive cosine function position encoding is used as the position encoding of the knowledge tracking model, the gate mechanism is used to optimize the computational effect under the embedding representation as well as the dot product attention mechanism, and the scaled dot product computational attention is used to optimize the multi-head attention module, ReLU is selected as the activation function, and the final knowledge tracking prediction result is finally outputted with the Sigmoid compression function. The knowledge graph-based search space optimization (SSO) algorithm is used to generate a candidate set of learning paths, measure the degree of knowledge in the candidate set of learning paths using the constraint rules of comprehensibility, reasonableness, and validity, and form dynamic learning paths, and ultimately find the optimal paths to enable learners to reach their learning goals. Relying on the multimodal learning resource intelligence platform constructed, the experiments of knowledge tracking and learning path recommendation are carried out respectively to test the effectiveness of the knowledge tracking model and learning path recommendation model applied by the platform.
Multimodal learning resources are categorized into various forms such as text resources, video resources, sound resources and picture resources. The intelligent platform has the characteristics of modularization, visualization and integration, which can reflect the visualization of learning differences, provide structured high-quality content, and also assist teachers in conducting activity-based cooperative inquiry, providing technical support for the integration of multimodal learning resources.
Multimodal resources refer to learning resources in multiple media forms that contain different perceptual channels (e.g., visual, auditory, tactile, etc.) [18]. These resources aim to provide a richer and more comprehensive learning experience through text, images, audio, video and other forms. Multimodal resources are characterized by their diversity, multi-channel and multi-sensory forms, and have the ability to present information in a more comprehensive way. The function of multimodal resources is to satisfy the perception and learning needs of different learners, and to improve the expression and delivery of information.
The integration of multimodal resources can help task group teaching to build a structured and deep learning experience. The integration and learning of modal learning resources requires the use of a variety of resources to support students to complete the task, these resources can include text, images, audio, video and other forms. Multimodal resource integration can organically integrate information and materials from different media and forms to support task implementation and deep learning.
On the whole, multimodal resource integration provides teachers with more flexible and innovative teaching tools, and lays the foundation for the improvement of teaching quality and the cultivation of students’ overall literacy.
First, teachers can rely on smart platforms to collect learning data and track learning activities. A significant feature is the learning process data acquisition and intervention design of the smart platform. Learning process data includes the acquisition of each student’s cognitive level inside and outside the classroom, and smart platforms can enter the classroom by virtue of their portability and simple interactivity.
Secondly, teachers can refer to the innovative interaction methods of smart platforms to construct group collaborative knowledge. Intelligent platforms support group collaborative knowledge construction, whether in face-to-face classrooms or virtual learning spaces.
Thirdly, students can actively explore and learn through the intelligent cognitive tools of the smart platform. By utilizing the e-cognitive tools provided by the Smart Platform, students are able to conduct independent inquiry activities or collaborative explorations with their group partners. This not only helps to cultivate students’ self-learning ability, but also significantly enhances their innovative thinking.
Fourth, teachers and students can utilize the real-time social functions of the smart platform to enhance community communication. The operation of smart platforms on smart mobile terminals facilitates real-time communication among teachers, students and parents.
Intelligent platform is a learning platform that integrates multimodal resources and intelligent technologies. It provides students with personalized, interactive and intelligent learning experiences by integrating multimodal resources such as text, images, audio and video, combined with intelligent algorithms and technologies. Intelligent platform has the characteristics of modularization of learning tasks, visualization of learning process, and holistic learning activities, and at the same time, it enjoys the functional characteristics of reflecting visual learning differences, providing structured quality content, and assisting activity-based cooperative inquiry.
The intelligent platform has obvious advantages in the integration of multimodal learning resources. First, the intelligent platform can support multimodal resource generation. The process of producing video teaching resources is no longer limited to computer operation, and various intelligent platforms provide convenient production tools. Secondly, relying on intelligent platforms, multimodal resources can be efficiently managed, and the use of intelligent platforms enables the content of courseware to be adjusted and changed at any time.
The multimodal learning resource platform constructed in this paper mainly includes three major functional modules.
Multimodal learning knowledge graph. Through the construction of this knowledge graph, the integration of multimodal learning resources is completed. Knowledge tracking based on multimodal learning knowledge map. Through knowledge tracking to simulate the learning process of students in the multimodal learning resource platform, predict the future learning performance of students, and provide a reference basis for the adjustment of teaching content. Learning path generation and recommendation based on multimodal knowledge graph. Based on the knowledge structure of learning resources in the knowledge graph, the learning path recommendation method is optimized to realize the planning of students’ personalized learning path.
To learn any course, the first task is to master the knowledge points themselves and the interrelationships among them, and secondly, the educational knowledge graph constructed in this paper consists of ternary groups reflecting the attributes of entities and the interrelationships among entities, so the main work in this section is to obtain the data required for multimodal learning resources to constitute the graph, and the specific tasks are named entity identification, attribute extraction, and relationship extraction [19].
Knowledge point entities are the cornerstone of constituting educational knowledge graphs, how to efficiently and accurately obtain subject knowledge point entities from text is the key to constituting educational knowledge graphs. Named Entity Recognition (NER) is an important task in a class of NLP domains and a subfield of information extraction that helps to convert textual data of multiple structural types into structured data that can be read by computers. The purpose of this task is to be able to accurately identify the information present in the text, such as location names, organization names, meaningful dates, etc., which can also be referred to as entity extraction. In this section, the purpose of NER is to extract knowledge point entities from the textual data of the subject domain of the database, in particular, the term “term” is used in this paper to denote a knowledge point entity.
The task of named entity recognition can be solved using a variety of technological approaches, which are usually categorized into the following groups: dictionary-based, statistical-based, rule-based, deep learning, and hybrid techniques. Among them, the use of a combination of rule-based and dictionary-based techniques refers to the manual establishment of recognition rules corresponding to the text data by experts in the field to achieve the task of entity recognition, which will have a high accuracy rate and fast recognition, but the disadvantage is that the method will consume a lot of human and material resources, poor portability, and has greater limitations. The current deep learning method based on large-scale data has gradually become the mainstream of the times, this method has a better portability, low artificial involvement and high efficiency.
In this section, we mainly design the NER model with the best effect for the knowledge text data in the database subject area, and at the same time, we analyze and compare it with some commonly used models of NER, so as to prove that the proposed model can have a good effect in the entity extraction task in the database subject area.
First of all, the construction of the database subject domain knowledge graph will involve a large number of specialized terms, such as “concurrency control”, “multi-granularity tree”, etc., and the traditional lexical tools do not contain these terms in the dictionary, which often leads to unsatisfactory lexical results. This often leads to unsatisfactory results. Therefore, this section incorporates the dictionary resource Dict of the database discipline on the BBLC model, i.e., this section defines a lexicon of proper nouns in the database domain containing 971 words to be added into the custom thesaurus of jicba, and implements the correction of words to ensure the accuracy of the partitioning results, while the partitioning in this paper only considers the Chinese text. Constructing a customized lexicon is conducive to reducing the difficulty of extracting knowledge point entities in massive text. Improve the accuracy of named entity recognition, in order to ensure the high accuracy of NER, this paper as far as possible to build a custom dictionary covering the terminology of all database subject areas, the encoding format is UTF-8.
Secondly because the BILSTM-CRF deep learning model needs to implement encoding on the given text data, this paper adopts the Bert model to train the text data related to the database learning resource disciplines, from which the word vector representation of the text data to be recognized can be derived. Meanwhile, because the vocabulary involved in this paper is highly specialized, the generalized dataset can not meet the requirements of specialization, so in this section, the completed dataset processed in Section 2 is used as the original input of the Bert model, which is processed and processed to obtain the word vector representation.
Then the word vectors containing rich features pre-trained by the Bert model are used as inputs to the BILSTM network. That is, two LSTM structures are utilized to automatically extract features and model them.
Finally, the global optimal tag sequence is obtained for scoring by combining the neighboring stemming between the tags and participating in the computation with the transfer matrix of the CRF layer, which ultimately predicts the probability that the target vocabulary is an entity and then completes the NER task. Specifically, the CRF layer objective function is shown in Equation (1):
Where
Meanwhile the CRF layer utilizes the Viterbi method for the optimal solution path to predict the optimal solution path as shown in Eq. (2):
The labeled completed dataset was used for training and testing of the Dict-BBLC model. Randomly 4/5 of this dataset is used for training.
In the knowledge tracking task of the knowledge tracking model, the data material available to the researcher is mainly the practice records of students under a specific course provided by online learning platforms [20]. The datasets provided by online education platforms oriented to the knowledge tracking task vary in terms of field attributes, but many common parts can still be extracted as a whole.
The input form of the model proposed in this chapter is an embedded representation. Embedded representations can also include teacher IDs as well as school IDs or even context vectors extracted from the text, while considering that some datasets do not contain these fields. Therefore, although there is potential to introduce embedded representations for multiple fields, only the exercise ID and skill ID are currently used in the model.
In this paper, we propose that the output of the model can be viewed as the probability that a particular student will answer correctly given the exercise ID. The input to the model is the student’s historical practice record, which consists of the exercise ID, the answer result, and the skill ID to which the exercise belongs; the output of the model is the predicted odds of answering the particular exercise.
The model extracts information from the questions answered at the beginning of the course to Question
The model proposed in this chapter of the thesis is divided into four main parts.
One, the embedded representation module for the input of the exercise record;
Two, the encoder based on the self-attention mechanism;
Three, decoder based on self-attention mechanism;
Four, the prediction result representation module.
The focus of the model is on the encoder-decoder part, which are mainly implemented by a combination of multi-head attention computing module, feed-forward neural network, and layer normalization layer.
The specific process of generating learning path recommendation candidate set using knowledge graph based search space optimization algorithm (SSO) is as follows.
First, the prerequisite relationships between the knowledge points are read based on the given prerequisite map
Problem definition Since learning is a long process consisting of many different stages, the recommendation of learning paths focuses more on the achievable learning outcomes. In this paper, we use the degree of improvement of the learner’s knowledge level to measure the effectiveness of the learning path with the following expression:
Where Multiple constraint rules Learning path recommendation pushes knowledge sequences to learners that match their learning ability level by studying and analyzing the fit between learners’ personalized features and learning resource attribute features. Therefore, personalized learning path recommendation can be regarded as either a planning problem modeling, where the goal of the model is to solve the formulaic expression of the degree of matching between learner characteristics and learning resource characteristics20; or an objective optimization problem, where a variety of constraint rules are used to measure the degree of superiority of the knowledge points in the candidate set of learning paths, and ultimately to find the optimal path for the learner to achieve the learning goal. The learner repeats a certain knowledge point in the learning process, and still cannot achieve good learning results, at this time, it is necessary to shift to a new knowledge point. The importance of the remaining three conditions rises step by step, this paper is based on this, respectively, from the comprehensibility, reasonableness, effectiveness of the three aspects of the setting of the recommendation rules, in each time step from the candidate set of screening out the highest degree of matching problems recommended to the learner, until the learning goals, and finally each moment of the problem connected to the formation of a dynamic learning path.
In this chapter, a knowledge tracking experiment will be conducted on a real dataset to verify the validity and interpretability of the knowledge tracking model adopted in the functional construction of the multimodal learning resource intelligence platform constructed in this paper. The dataset used in this experiment is ASSISTments2021. The dataset, provided by the ASSISTment online tutoring platform and widely used for KT tasks, is the largest version of the auxiliary dataset consisting of data collected from September 2021 through October 2022 by preprocessing the data.
The comparison models selected for knowledge tracking experiments in this paper are DKT, SAKT, DKVMN, DKT+Forget, EERNN, and EKT. In this section, the performance will be compared using this paper using the area under the curve (AUC), and of course, the larger the value of the AUC represents the better the knowledge tracking prediction performance of the model. The AUC values of all models are specifically shown in Table 1. It is easy to see that there is a large performance performance gap between different models, while compared with other models, the AUC value of this paper’s model reaches 0.809, which is higher than the other comparison models, and is able to explicitly capture the relationship between the student-based learning data and the textual content of multimodal learning resources.
AUC
| Model | AUC |
|---|---|
| DKT | 0.715 |
| DKT+Forget | 0.726 |
| SAKT | 0.745 |
| DKVMN | 0.716 |
| EERNN | 0.774 |
| EKT | 0.765 |
| Model of this article | 0.809 |
In order to further verify that the model in this paper is robust to the sparsity of the dataset, this paper conducted experiments on groups of students with different number of question-answering sessions on the multimodal learning resource intelligence platform, respectively. Four groups of students were selected based on the number of times each user answered questions, thus constituting control groups with less than 10, 100, 1000 and 10000 interactions. The performance of all models is shown in Figure 1. It can be seen that the model in this paper outperforms the other comparative models in all conditions, and the AUC value is always higher than the level of 0.8. In addition, for the group of students who answered fewer questions, the performance enhancement of this paper’s model is more significant. Therefore, it can be concluded that even if the number of user interactions with the multimodal learning resource platform is low, the model in this paper is still able to effectively learn the students’ knowledge state.
Performance comparison of different student groups
Students’ knowledge states are visualized as explicit proficiency vectors, where each element reflects their knowledge of the relevant concepts of the multimodal learning resources, ensuring the interpretability of the prediction results. However, in the real world, students may practice very few topics compared to the huge exercise space, and if a student practices only a few times at a time, it is difficult for the knowledge tracking model to track his/her knowledge level and predict his/her performance. To alleviate this problem and improve the prediction performance of the knowledge tracking model, the knowledge relationship between exercises is further considered. A student was randomized to show his/her mastery level of six knowledge concepts over three time periods. The knowledge concepts in this time are exemplified by equations, and the knowledge points of equations won by K1~K6 bets are as follows.
K1: Multiplying and dividing integers K1: Gragh’s linear equations K1: Proportions K1: Exponents K1: Equations solved in two or fewer steps K1: Venn diagrams
The student’s level of mastery of the above six knowledge points is specifically shown in Table and Figure 2. As can be seen from the figure, the student continued to make progress in skill K3, and there may have been an element of learning as she practiced more and more practice topics related to K3 over the time period. On the contrary, his level of skill K4 decreased over time (from 0.65 to 0.36). Because he attempted only a few relevant practice problems at a time, he may have forgotten his K4 knowledge. These observations suggest that she needs to review the K4 knowledge points in time. Based on the visualization of knowledge mastery ability, the intelligent guided learning system can provide him with more personalized training services in practice.
Student knowledge state map
In this paper, we visualize the knowledge concept vectors of the questions to demonstrate their relationships in the model through more intuitive observation. In this paper, we select the five most common knowledge concepts and their corresponding exercises in the dataset (for better illustration, we only focus on some of the knowledge concepts and categorize the other knowledge concepts into the “other” category), and use different colors to mark the knowledge concepts for each exercise. The clustering results are shown in Figure 3. It can be found that exercises with the same knowledge concepts are easier to group because they are closer in the knowledge space. Therefore, the model in this paper can naturally follow the knowledge relationship between exercises in the modeling process.
Knowledge concept clustering
This chapter will explore the effectiveness of the learning path recommendation model proposed in this paper in building the multimodal learning resources intelligent platform function through the learning path recommendation experiment. The learning path recommendation experiment in this chapter is mainly divided into two parts, one is the exercise recommendation experiment based on multimodal learning resources, and the other is the students’ satisfaction test experiment in the face of the learning path recommendation function.
The experimental design of this paper is based on the learner’s knowledge point mastery map, which shows the learner’s mastery of knowledge points. The data of learners’ questions come from an online tutoring organization, which contains 223 learners’ answers to 20 exercises in junior high school mathematics, which examines 8 knowledge points related to functions, and the distribution of knowledge points involved in each exercise is shown in Figure 4.
Exercises to check the knowledge points
When using great posterior probability to determine learners’ mastery of a knowledge point, the prior distribution probability needs to be considered because different prior distributions will have different impacts on the miss rate and guessing rate of the model parameters, which ultimately affect the accuracy of the cognitive level. Four different distributions are mainly included, including uniform distribution, polynomial distribution, and improved dynamic prior distribution. Negative log-likelihood (NLL) and Akaike Information Criterion (AIC) are used to measure the fit of the model under the four different prior distributions. The two metrics, NLL and AIC, are described below.
NLL NLL is a common indicator to measure the prediction effect of the model, which evaluates the prediction accuracy of the model by calculating the difference between the predicted and actual results of the model, and the smaller the value of NLL, the better the prediction effect of the model. NLL Is an index to assess the degree of model fitting and complexity. It weighs the model’s fitting ability and overfitting risk by considering both the model’s goodness-of-fit and the number of parameters to find an optimal model. The smaller the AIC, the better the model. The NLL and AIC of the models in this paper under different distributions are specifically shown in Fig. 5. According to the results shown in the figure, under both NLL and AIC, the improved dynamically generated prior distribution model fits better than the other three prior distributions, with NLL and AIC reaching 5193 and 10843, so the model can be used to obtain the learners’ mastery of the knowledge points.
NLL and AIC under various distributions
In order to verify the effectiveness of the learning path recommendation model in this paper, certain indicators need to be used to evaluate the system, and learner satisfaction as an evaluation indicator is a common method, which is especially suitable for the evaluation of learning path recommendation.
This chapter adopts students’ satisfaction as the evaluation index of the experiment, and 50 subjects who tried out the multimodal learning resources intelligent platform of this paper were selected from S university students by random sampling, and a questionnaire was distributed to them. The survey adopted the standard academic random sampling method to ensure the representativeness of the subjects and the credibility of the questionnaire results, and the questions were designed from the three dimensions of learning effect, learning attitude, and overall evaluation of the platform. A total of 50 questionnaires were distributed and 50 valid questionnaires were returned.
This section aims to assess learners’ learning outcomes, which are quantitatively analyzed in terms of deeper understanding of course knowledge and improved learning efficiency through the application of the platform in this paper. The learning outcomes of the learners are specifically shown in Figure 6. As can be seen from the figure, 21 students chose “strongly agree” and “approve” on the topic of “enhancing knowledge understanding”, and 18 students preferred “improving learning efficiency”, indicating that the system has a certain effect on learners’ learning.
Learning effect analysis diagram
This section analyzes the application effect of this paper’s learning path recommendation model from the aspects of learning initiative and learning interest, and the specific results are shown in Figure 7. Only 5 and 4 people chose “disapprove” and “strongly approve” in the aspects of improving initiative and interest in learning respectively, which indicates that the platform of this paper is able to promote learners’ learning attitude to a certain extent, so that learners can invest more time and energy in course learning. This indicates that the platform can promote learners’ learning attitude to a certain extent, so that learners can devote more time and energy to course learning.
Learning attitude analysis diagram
The purpose of this section is to provide an overall evaluation of the learners’ perception of the system, the results of which are shown in Figure 8. Twenty-two and 20 respondents chose “favorable” and “very favorable” in terms of good operability and advantages of the platform respectively, which is more than half of the respondents. This indicates that the multimodal learning resources intelligent platform constructed in this paper has been recognized by students in terms of use, has certain advantages compared with other platforms, conforms to the aesthetics and operating habits of contemporary college students, and provides them with a good learning experience.
Overall evaluation of the system analysis diagram
In this paper, the multimodal learning resources intelligent platform is used as a strategy to optimize the integration of multimodal learning resources, from the determination of the multimodal learning knowledge graph and the knowledge tracking and learning path recommendation on the basis of the knowledge graph, to complete the construction of the functions of the intelligent platform and to improve the efficiency of the integration of multimodal learning resources.
In order to test the effectiveness of the knowledge tracking and learning path recommendation functions of the multimodal learning resources intelligent platform, knowledge tracking experiments and learning path recommendation experiments are carried out on the basis of the application of the platform in this paper.
In the knowledge tracking experiment, the AUC value of this paper’s knowledge tracking model reaches 0.809, and the AUC value in the test experiments with different number of times of answering the questions of the student group can always maintain the level of 0.8 or more, which is always higher than that of the comparative models such as DKT, SAKT, DKVMN, DKT+Forget, EERNN, EKT, and so on. Taking a student’s mastery on six knowledge points as an example, the clustering results show that the exercises of the same knowledge concepts are closer in the knowledge space, and the knowledge tracking model in this paper can naturally follow the knowledge relationship between exercises in the modeling process.
The learning path recommendation experiment mainly includes two parts: the exercise recommendation experiment and the student satisfaction experiment. In the exercise recommendation experiment, the values of NLL and AIC under the improved dynamic prior distribution of this paper’s learning path recommendation model are the highest, reaching 5193 and 10843, respectively, which proves the feasibility of this paper’s model in acquiring learners’ knowledge mastery. In the student satisfaction experiment, 21 students “strongly agreed” and “agreed” that the learning path recommendation function of the multimodal learning resource intelligence platform could “enhance knowledge understanding”, and 18 students agreed to “improve learning efficiency”. In terms of learning attitude, only 5 people and 4 people chose “disapprove” and “strongly agree” for “improving initiative” and “improving learning interest”, respectively. In terms of the overall evaluation of the platform, 22 respondents and 20 respondents chose “yes” and “strongly approve” in the face of “good operability” and “platform advantages” respectively.
Overall, the multimodal learning resources intelligent platform constructed in this paper has better performance in knowledge tracking and learning prediction, learning path recommendation and learning interest enhancement, and promotes the efficient integration of multimodal learning resources.