ACT-CV - Computer Vision for Cognitive Modeling

This package is intended to create an easy to use connection between arbitrary graphical user interfaces and a cognitive modeling tool.

Overview

Five types of connection targets have been implemented

cross platform interactive screen grabber
video file player using OpenCV
camera grabber using OpenCV
HTML5 renderer using QtWebKit
HTML5 browser plugin using the Gecko engine (Firefox / XUL) - see ACT-CV-XUL sub-project
grayscale random image generator

The screen images/videos are analyzed with OpenCV. The following demos are currently available:

object matching demo that uses cross-correlations to find the KDE/MS Windows folder symbol on the screen and moves the mouse to one of them.
line finder using the function cvHoughLines2 from OpenCV
DOM inspector that returns textual elements from a web page
optical flow demo that computes the flow using consecutive images. The demo is intended for car driving scenarios. The flow can be used to detect whether the current egocentric direction (indicated by a focus on the screen) matches the heading of the virtual camera.

The package consists of four elements:

act-cv is a standalone application running the demos
ACT-CV.dll / libAct-CV.so is the server library that provides the computer vision to cognitive models implemented somewhere else. A lisp interface for the usage with ACT-R can be created by using make interface
act-cv-testclient is a simpe standalone application that uses the ACT-CV library
an optional client/server version of ACT-CV using ICE

Documentation

Questions about installation, usage, and so on are covered in the online documentation. This is part of the source code release as well.

Download

Get ACT-CV here.

Other Tools

ACT-CV-XUL: A XUL-based browser that allows to control HTML5 applications from ACT-R. Includes a POMDP-based mechanism for creating finite automata from HTML5 applications that are suitable for fast-time simulation.
Source code only: Download
Lean embedded Lisp: Small framework that shows how to start a Lisp child process from C++, load ACT-R, run a model, and exchange data between parent process and embedded cognitive architecture.
Source code only: Download

References

Halbrügge, M. (2015). Fast-Time User Simulation for Dynamic HTML-based Interfaces. In Taatgen, N. A., van Vugt, M. K., Borst, J. P. & Mehlhorn, K. (Eds.), Proceedings of the 13th International Conference on Cognitive Modeling, pages 51-52, Groningen: University of Groningen

Halbrügge, M. (2013). ACT-CV: Bridging the Gap between Cognitive Models and the Outer World. In Brandenburg, E., Doria, L., Gross, A., Güntzler, T., and Smieszek, H. (Eds.), Grundlagen und Anwendungen der Mensch-Technik-Interaktion. 10. Berliner Werkstatt Mensch-Maschine-Systeme, pages 205-210, Berlin: Universitätsverlag der TU Berlin

Halbrügge, M., Deml, B., Färber, B. A. & Bardins, S. (2007). ACT-CV - Die Erweiterung von ACT-R um Bildverarbeitungsalgorithmen erlaubt die schnelle Erzeugung mächtiger Benutzermodelle. In In: Grandt. M. & Bauch, A. (Eds.), Simulationsgestützte Systemgestaltung - DGLR-Bericht 2007-04, pages 313-331, Bonn: DGLR e.V.