Crescent - Spoken multimodal dialogue-based search for mobile and web-based applications

Important results from the project

Findwise and Talkamatic have productified their initial dialogue-based search application by developing multiplatform grammar driven search service and an adaptable multi modal dialogue based search solution. The solution uses Precision Search (a grammar aware search solution inspired by Graph search). The multimodality of the solution with voice, textual and touchscreen interaction enables search in many environments: be they screen-free, hands-free or for search `on-the-go.´

Expected long term effects

A productified, multimodal solution has been developed that is more robust, flexible and scalable. The solution is adaptable to various information and search domains, contexts and environments. Both Findwise and Talkamatic feel confident in being able to take advantage of the increased mobility of data and the likely new screen- and hands-free and `on-the-go´ search applications that will result.

Approach and implementation

An integrated mobile product was successfully developed that included multimodal (spoken, textual and touch) interaction with a search interface. The decision to embed a web solution inside a smartphone app, providing the voice functionality on top of a web page, was a good move; it allowed the search functionality to be developed in parallel with the speech interface. The app was tested by a major Swedish manufacturer with their data. A multiplatform approach for the speech modality was examined but put on hold at this time due to the current limitations in speech technology products.