I.R.I.S. Working Group

Interoperable Multimedia Retrieval in Distributed Systems

A.I.R. - Architecture for Interoperable Retrieval on Distributed and Heterogeneous Multimedia Repositories

Click here to watch a video presentation about the A.I.R. multimedia middleware framework (Use Case: Interoperable Image Search).

Nowadays multimedia data is produced and consumed at an ever increasing rate. Similarly to this trend, diverse storage approaches for multimedia data have been introduced. These observations lead to the fact that distributed and heterogeneous multimedia repositories exist whereas an unified and easy access to the stored multimedia data is not given.

This project presents an architecture, named AIR, that offers the aformentioned retrieval possibilites. To ensure interoperability, AIR makes use of recently issued standardisation efforts, in names the MPEG Query Format (MPQF) (multimedia query language) and the JPSearch transformation rules (metadata interoperability).

AIR can be operated in many different facets within a distributed and heterogeneous multimedia search and retrieval framework. In general, the tasks of every internal component of the AIR architecture highly depends on the registered databases and use cases. In this context, two main query processing strategies have to be distinguished, as illustrated next.

(a) local query processing: Heterogeneous systems provide their local metadata format (e.g., Dublin Core, MPEG-7, etc.) and a local / autonomous data set. A query transmitted to such systems is understood as a whole and the items of the result set are the outcome of an execution of the whole request. Of course, transformation of the used metadata format (e.g., from Dublin Core to MPEG-7) may be needed for some systems. In addition, depending on the degree of overlap among the data sets (e.g., the same image is annotated in all databases), the individual result sets may contain duplicates. However, a result aggregation process only needs to perform an overall ranking of the result items of the involved retrieval systems. Here, duplication elimination algorithms may be applied as well. (b) distributed processing: Heterogeneous systems may depend on different data representation (e.g., ontology based semantic annotations and XML based low level features) and query interfaces (e.g., SPARQL and XQuery) but describe a common (linked) global data set. In this context, a query transmitted to AIR needs to be evaluated and optimized, which results into a specific query execution plan. In series, parts of the query are forwarded to the respective engines and executed. Now, the result aggregation has to deal with a correct consolidation of the partial result sets. In this context, AIR needs to be extended to a federated multimedia database management system.

The architecture of AIR is subdivided in several components:


Peripheral Communication is provided by two peripheral communication layers: the Backend Communication Layer (management of backends, query distrbution) and the Client Communication Layer (MPQF query generation and execution). From a technical point of view, both communication layers provide a set of methods, e.g. exposed by a Webservice or a CORBA interfaces.

The Backend Management Layer main functionalities are the (de-) registration of services offered by a backend and the distribution of MPQF queries. A backend can register their retrieval services with their supported service capabilities. These descriptions are standardized in MPQF, allowing the specification of the retrieval characteristics of registered backends.

The main purpose of the MPQF Factory Layer
is the generation and validation of MPQF queries. This layer also encapsulates interfaces for inserting preprocessing plug-ins (e.g., metadata extraction or file conversion).

The MPQF Management Layer organizes the registration of MPQF queries and their distribution to the applicable services. After the registration with a unique identifier of the entire query, the distribution of the query depends on the underlying search concept. For the local processing scenario, the whole query can be transmitted to the services. In contrast to that, in a distributed processing scenario, the query will be automatically divided in segments by analyzing the query types used in the condition tree.

The MPQF Interpreter is located at the backend and is supposed to act as a mediator between AIR and a particular retrieval service. Its main purpose is to accept the MPQF query and to prepare it for retrieval (e.g., metadata transformation).

The Backend Benchmarking Layer should provide informations about the search quality of a backend in order to base the re-ranking of the result set on it. This topic is currently under development.

The Response Layer
performs in general the result aggregation (on the basis of the information provided by the Backend Benchmarking Layer) and returns the aggregated result set.