This package is intended to create an easy to use connection between arbitrary graphical user interfaces and a cognitive modeling tool.

Overview

Four types of connection targets have been implemented

• a cross platform interactive screen grabber
• a video file player using OpenCV
• a camera grabber using OpenCV
• a HTML5 renderer using QtWebKit
• a HTML5 browser plugin using the Gecko engine (Firefox / XUL) - see ACT-CV-XUL sub-project
• a grayscale random image generator

The screen images/videos are analyzed with OpenCV. The following demos are currently available:

• an object matching demo that uses cross-correlations to find the KDE/MS Windows folder symbol on the screen and moves the mouse to one of them.
• a line finder using the function cvHoughLines2 from OpenCV
• a DOM inspector that returns textual elements from a web page
• an optical flow demo that computes the flow using consecutive images. The demo is intended for car driving scenarios. The flow can be used to detect whether the current egocentric direction (indicated by a focus on the screen) matches the heading of the virtual camera.

The package consists of four elements:

• act-cv is a standalone application running the demos
• ACT-CV.dll / libAct-CV.so is the server library that provides the computer vision to cognitive models implemented somewhere else. A lisp interface for the usage with ACT-R can be created by using make interface
• act-cv-testclient is a simpe standalone application that uses the ACT-CV library
• an optional client/server version of ACT-CV using ICE

Usage

Find Lines in Images

This algorithm works quite straightforward. Use this one to check whether ACT-CV works well.

Find Textual Elements on Web Sites

ACT-CV currently returns the content of the following HTML tags:

• A - link anchor
• DIV - generic divisions
• TD - table cells
• INPUT - input elements, e.g. buttons
• OPTION - entries in a choice group

The returned position is the rectangle used by the complete element, not just the text within. The relative positioning of the text (top, bottom, center, ...) is currently not taken into account.

Find visual Objects in Images

You must provide

• image files of the objects you would like to find in the video / on the screen
• a text file that defines these objects

You can find sample files in the imgs directory of the source code distribution.

The definition file has one line per match target. Each of these lines consists of three entries:

1. Object type. This will go into the VALUE slot of the visual buffer chunk that represents this object when you use the ACT-CV device for ACT-R.
2. Image file name. ACT-CV will read this file and search for it in every frame.
   Important: The file name must not contain spaces!
3. Cross correlation threshold. ACT-CV uses cross-correlations to determine whether the object is somewhere in the video (or on the screen). If you want exact matches and have a source without much noise, like from the computer screen, set this one near to 1.

Usage with ACT-R

ACT-R is an elaborate architecture for the generation of cognitive models. ACT-CV has been designed for the use with ACT-R.

ACT-CV depends on the CFFI package, which is available for all major Lisps. This way, ACT-CV is not bound to a single flavor of Lisp. After having loaded CFFI, you can load the file "act-cv-dev.lisp" from the actr-device directory of the distribution. There is a sample ACT-R model in "act-cv-test.lisp" in the same directory that demonstrates how ACT-CV works.

How to obtain

ACT-CV is distributed under the terms of the GNU General Public License. See the file LICENSE in the root directory of the source code distribution.

Download the latest version of ACT-CV from http://sourceforge.net/project/showfiles.php?group_id=214599

Building ACT-CV

Prerequisites

For building ACT-CV, you must have the following software packages

• OpenCV http://sourceforge.net/projects/opencvlibrary/
• Java SDK with headers and libs, i.e. http://java.sun.com/javase/downloads

If you want to use ACT-CV with Lisp, you need the CFFI package for Lisp: http://common-lisp.net/project/cffi/

The Lisp interface of ACT-CV has been generated using SWIG (http://www.swig.org/), but you will only need this if you want to make changes to the interface.

ACT-CV comes with a client/server version that allows to run image processing and the software system that uses the results of the processing (i.e. the cognitive model) on different machines. Client and server communicate via ICE (http://www.zeroc.com/ice.html).

On Linux

aclocal ; libtoolize ; autoconf ; automake -a -c
./configure [CXXFLAGS="-I /path/to/Ice/include -L /path/to/Ice/lib"]
make
make install

configure should find both your Java and OpenCV installation automatically. If not, please direct it to them by providing additional directories to c++ (as shown above).

On Windows

You need cmake (cross platform make: http://www.cmake.org) for generating MS Visual Studio files. During the cmake call you have to point out where your Java and OpenCV headers and libraries are located.

Acknowledgements

Parts of this work have been supported within the DFG funded project "Automatische Usability-Evaluierung modellbasierter Interaktionssysteme für Ambient Assisted Living", see also http://www.tu-berlin.de/?id=135088

Parts of this work have been supported within the DFG excellence initiative research cluster "Cognition for Technical Systems - CoTeSys", see also http://www.cotesys.org

References

Halbrügge, M. (2015). Fast-Time User Simulation for Dynamic HTML-based Interfaces. In Taatgen, N. A., van Vugt, M. K., Borst, J. P. & Mehlhorn, K. (Eds.), Proceedings of the 13th International Conference on Cognitive Modeling, pages 51-52, Groningen: University of Groningen

Halbrügge, M. (2013). ACT-CV: Bridging the Gap between Cognitive Models and the Outer World. In Brandenburg, E., Doria, L., Gross, A., Güntzler, T., and Smieszek, H. (Eds.), Grundlagen und Anwendungen der Mensch-Technik-Interaktion. 10. Berliner Werkstatt Mensch-Maschine-Systeme, pages 205-210, Berlin: Universitätsverlag der TU Berlin

Halbrügge, M., Deml, B., Färber, B. A. & Bardins, S. (2007). ACT-CV - Die Erweiterung von ACT-R um Bildverarbeitungsalgorithmen erlaubt die schnelle Erzeugung mächtiger Benutzermodelle. In In: Grandt. M. & Bauch, A. (Eds.), Simulationsgestützte Systemgestaltung - DGLR-Bericht 2007-04, pages 313-331, Bonn: DGLR e.V.