After the new medical multimedia documents are generated, they will be saved into the medical multimedia database. The fourth step is retrieval process. In this step, user can upload an annotated medical multimedia document Figure 1 d1 with arbitrarily arbitrary format to execute the medical multimedia retrieval Figure 1 d2. In this case, the engine will return the result by matching the annotations of uploaded multimedia document with medical multimedia documents in database Figure 1 d3.
In the returned result, the user will be asked to give additional annotations Figure 1 d4 to the medical multimedia he selected to make the annotations more abundant and accurate. In our approach, all the annotations will be described by text. The physical location information of m i is saved in rational database linked to the corresponding real file.
Semantic annotations will provide meaningful text reflecting users' personal understanding to m i.
Semantic Models for Multimedia Database Searching and Browsing
However, not all the annotations can accurately represent the semantic information of m i. During the retrieval process, w i for every annotation could not be constant. Obviously, more frequently used annotations during the retrieval process can better express semantic information, and they should be assigned a greater weight. We design an adjustment schema as follows:. The initial weight assignment and the adjustment process need to check all the medical multimedia documents in the database, and this work will cost many computational resources. To solve this problem, we can execute this process only once when the search engine is built.
In addition, the adjustment process can be performed as background thread. We use ontology technology to describe the medical semantic information. In the ontology representation, each node describes one certain semantic concept and the ontology representation satisfies a recursive and hierarchical structure. Our approach adopts composite pattern [ 17 ] as the data structure to represent the relation of annotations. In our approach, ontology semantic information will be merged with medical multimedia by two ways.
We utilize an optimal data hiding-based strategy for medical multimedia document storage. Our approach supports user feedback, and it may cause the modification of the semantic content, so we will design an effective approach to search and modify the semantic data in the medical multimedia. In our approach, we do not use some popular and security approaches such as neural network and wavelet technology and directly save the semantic information in the head of the medical multimedia.
During every retrieval process, we cannot directly read and write medical multimedia documents in hard disk because this will cost lots of communication and computation time. To solve this problem, we adopt a cache-based approach. When the search engine is initialized, the semantic information in medical multimedia documents will be extracted to the rational database e. This work will be executed in background thread. Client users visit the rational database through the annotation and retrieval interface, and then the medical server will find the real file of the medical multimedia.
Figure 2 shows the structure of the server which saved medical multimedia documents in the database. The cardinality of A mi will be more and more. In A mi , wrong or less frequently used annotations inevitably exist, which will waste much retrieving resource and storage space. In order to solve this problem, we define an optimization approach to eliminate the annotations which may be useless.
This process is called annotation refinement.
The purpose is to retain most of the high frequency annotations and eliminate the annotations with less use. Check A mi and remove the i th row when a i satisfies. Because this operation needs too much computation resource, we will execute the annotation refinement every long time interval and during the time of less retrieval requirements or system maintenance.
After retrieval, the engine will return some medical multimedia documents. Our approach supports user feedback, so for a particular returned multimedia document, the user can add additional annotations to enrich the semantic information. In summary, during the retrieval progress, the annotations will be more and more abundant. But rarely used annotations will also be removed.
Table of contents
There will be some new annotations added into the annotation matrix A mi because of the user feedback. Therefore, our approach is a dynamic framework, which is used for the longer time and the more accurate results we can obtain. In this section, performance evaluation model will be designed to measure the performance of our approach.
These models are based on the following five criteria: recall ratio, precision ratio, background process time cost, retrieval time cost, and additional storage cost. The recall and precision ratios are the most common measurements for evaluating the retrieval performance. Now we use them to evaluate the performance of our approach. We can get different recall and precision ratios in different retrieval processes.
The recall ratio is computed by the proportion of retrieved relevant medical multimedia documents in total relevant multimedia documents, and the precision ratio is computed by the proportion of retrieved relevant medical multimedia documents in total retrieval multimedia documents.
Get this edition
Therefore, the recall ratio R and precision ratio P can be defined as follows:. It is an important issue to determine whether a returned multimedia document is relevant. In this paper, because the database is not very large, we will perform the judgment based on the users' understanding. In our approach, several background processes will cost time. We define the background process time as follows:. T ref represents the annotation refinement time eliminate the redundant or error annotations and T cac represents the time cost of cache of the semantic information into the rational database.
We define T r as the time cost for a special retrieval as follows:. Because the file size after merging will increase, the additional storage cost will be taken into consideration. The increase rate for storage P s is defined as follows:.
How to construct the dataset is an important problem in the experiment. Some general databases have been proposed.https://it.oqomurofyh.tk
PepeSearch: Semantic Data for the Masses
However, these databases can only perform the experiments aiming to one particular medical type e. Cross-type medical multimedia retrieval requires a wide variety of files such as images, videos, and audios, so these databases are not suitable to perform the experiments. We have constructed a medical database containing various medical types including images, videos, and audios. This medical database contains 10, medical multimedia documents, including 8, images, 1, videos, and 1, audios. All the annotations of the medical multimedia documents were provided through users manually annotating.
In this paper, we developed some software modules to verify the effectiveness of our approach. In the server, background process will be executed every 24 hours. Table 1 shows the software tools and the running environment profiles in the experiments. In the experiment, we choose a medical multimedia document called sample document and upload it to the search engine. The server will return the result by matching the annotations of uploaded multimedia document with medical multimedia documents in database.
Before the search, some users will be asked to give the annotations to the sample multimedia document in the annotation interface. After uploading the file in retrieval interface, the system will search all the semantic medical multimedia documents whose semantic information is similar with the sample document. To measure the performance, we use the images, videos, and audios as the sample files to execute the retrieval.
In order to demonstrate the performance of the cross-type retrieval, we specially record the recall and precision ratios of using one type to search the other two types e. To every multimedia type, we perform 10 different retrieval processes using 10 different sample documents and calculate the average recall and precision ratios to other multimedia types.
The average recall and precision ratios are illustrated in Figure 3. Figure 3 indicates that in the retrieval process between different medical types, the recall and precision ratios are good. This is because our approach completely abandons the physical feature extraction and executes the retrieval processes based on semantic annotations which are described as text. In order to carry out the retrieval process, we have to perform several background processes whose time cost includes T mer , T ref , and T cac defined in Section 3.