Key research questions

The project’s socio-economic key research question: How can key developments relevant to an organization be identified, monitored, interpreted and aggregated to gain actionable insights?

This translates into the following key scientific research question: How can methods for analyzing text and multi-media information deal with semantically rich multimedia information, in which dynamically changing factual and subjective aspects are intertwined key ingredients, and how can this be scaled to very large data volumes, be executed in near real-time, with outcomes aggregated to produce usable overviews?

Scientific/technological innovation

Entities and events are seen as the most salient ingredients of textual and audiovisual content; they represent valuable knowledge ingredients that are heavily searched for, and conveniently serve as the linking pin between heterogeneous content sets. Semantic search is structured around entities, their properties, relations, developments as well as pertinent attitudes and emotions. The project identifies three main thematic research lines around semantic search, defined in terms of the types of information being dealt with:

  1. textual,
  2. speech and
  3. multi-media.

For 1. the most important innovations are dynamic stakeholder modeling, automated methods for identifying frames in public information, a novelty-based approach to information extraction, and the development of text-to-text generation techniques for Dutch language summaries as applied to highly dynamic data. For 2. the most important innovation is semantic analysis in the output of automatic speech recognizers. For 3., the most important innovations are in the use of pictorial cues for image/video search, in fusing metadata with pictorial cues in a mobile setting, and in the use of body signals instead of traditional annotations to recognize emotions in video material.

Scientific/technological goals

The project takes on the challenge of building semantic search technology, with a special focus on issue management algorithms and tools, i.e., for identifying entities, issues, naming stakeholders, capturing attitudes and emotions, and monitoring relevant events and changes in highly dynamic environments. The project’s overall goal is three-fold:

  1. To create semantic search and discovery tools for monitoring news and social media mining purposes;
  2. To create semantic analysis and discovery tools for the output of automatic speech recognizers;
  3. To provide a generic content-based system for accessing images and video data that can be specialized to analyze body signals.

Knowledge to be developed

To realize the scientific/technological goals, knowledge needs to be developed on the dynamics of entity-topic associations, on flexible and adaptive entity recognition and search, on discovering entities in web-based resources, on the relation between automated content analysis and public opinion data collected with pollsters, on combining data-driven patterns and top-down knowledge, on retrieval and text mining technologies for user generated content, on semantic interpretation of spoken materials, on the constraints of mobile phones on image processing and image/display quality and data transfer, on recognizing emotions in measurements of body signals.

Scientific approach

The project adopts the following strategy: use cases are defined and baselines are set up using technologies available from the partners; multiple cycles of

  1. model/algorithm building at the cutting edge of applied research,
  2. experimental evaluation,
  3. prototype module release and
  4. integration of prototypes in open source, open standards-based web services or profit partner’s solutions.

Applied requirements

The requirements are robust, scalable and adaptive tooling. Scalability concerns both the number and types of information sources being monitored, and the number and types of entities, topics, stakeholders, events. The system’s results should be interpretable, meaning that links should be provided to background knowledge and that automatically measured perspectives correlate well with manually determined ones. Moreover, its outputs should be available in near real-time. Additional requirements for the envisaged multimedia work are robustness against imaging perturbations in a mobile environment and against uncertainty in measurements of body signals.