ACT-CV - Machine Vision for Cognitive Modeling
ACT-CV - Machine Vision for Cognitive Modeling Documentation

This package is intended to create an easy to use connection between arbitrary graphical user interfaces and a cognitive modeling tool.


Four types of connection targets have been implemented

The screen images/videos are analyzed with OpenCV. The following demos are currently available:

The package consists of four elements:


Find Lines in Images

This algorithm works quite straightforward. Use this one to check whether ACT-CV works well.

Find Textual Elements on Web Sites

ACT-CV currently returns the content of the following HTML tags:

The returned position is the rectangle used by the complete element, not just the text within. The relative positioning of the text (top, bottom, center, ...) is currently not taken into account.

Find visual Objects in Images

You must provide

You can find sample files in the imgs directory of the source code distribution.

The definition file has one line per match target. Each of these lines consists of three entries:

  1. Object type. This will go into the VALUE slot of the visual buffer chunk that represents this object when you use the ACT-CV device for ACT-R.
  2. Image file name. ACT-CV will read this file and search for it in every frame.
    Important: The file name must not contain spaces!
  3. Cross correlation threshold. ACT-CV uses cross-correlations to determine whether the object is somewhere in the video (or on the screen). If you want exact matches and have a source without much noise, like from the computer screen, set this one near to 1.

Usage with ACT-R

ACT-R is an elaborate architecture for the generation of cognitive models. ACT-CV has been designed for the use with ACT-R.

ACT-CV depends on the CFFI package, which is available for all major Lisps. This way, ACT-CV is not bound to a single flavor of Lisp. After having loaded CFFI, you can load the file "act-cv-dev.lisp" from the actr-device directory of the distribution. There is a sample ACT-R model in "act-cv-test.lisp" in the same directory that demonstrates how ACT-CV works.

How to obtain

ACT-CV is distributed under the terms of the GNU General Public License. See the file LICENSE in the root directory of the source code distribution.

Download the latest version of ACT-CV from

Building ACT-CV


For building ACT-CV, you must have the following software packages

If you want to use ACT-CV with Lisp, you need the CFFI package for Lisp:

The Lisp interface of ACT-CV has been generated using SWIG (, but you will only need this if you want to make changes to the interface.

ACT-CV comes with a client/server version that allows to run image processing and the software system that uses the results of the processing (i.e. the cognitive model) on different machines. Client and server communicate via ICE (

On Linux

aclocal ; libtoolize ; autoconf ; automake -a -c
./configure [CXXFLAGS="-I /path/to/Ice/include -L /path/to/Ice/lib"]
make install

configure should find both your Java and OpenCV installation automatically. If not, please direct it to them by providing additional directories to c++ (as shown above).

On Windows

You need cmake (cross platform make: for generating MS Visual Studio files. During the cmake call you have to point out where your Java and OpenCV headers and libraries are located.


Parts of this work have been supported within the DFG funded project "Automatische Usability-Evaluierung modellbasierter Interaktionssysteme für Ambient Assisted Living", see also

Parts of this work have been supported within the DFG excellence initiative research cluster "Cognition for Technical Systems - CoTeSys", see also


Halbrügge, M. (2015). Fast-Time User Simulation for Dynamic HTML-based Interfaces. In Taatgen, N. A., van Vugt, M. K., Borst, J. P. & Mehlhorn, K. (Eds.), Proceedings of the 13th International Conference on Cognitive Modeling, pages 51-52, Groningen: University of Groningen

Halbrügge, M. (2013). ACT-CV: Bridging the Gap between Cognitive Models and the Outer World. In Brandenburg, E., Doria, L., Gross, A., Güntzler, T., and Smieszek, H. (Eds.), Grundlagen und Anwendungen der Mensch-Technik-Interaktion. 10. Berliner Werkstatt Mensch-Maschine-Systeme, pages 205-210, Berlin: Universitätsverlag der TU Berlin

Halbrügge, M., Deml, B., Färber, B. A. & Bardins, S. (2007). ACT-CV - Die Erweiterung von ACT-R um Bildverarbeitungsalgorithmen erlaubt die schnelle Erzeugung mächtiger Benutzermodelle. In In: Grandt. M. & Bauch, A. (Eds.), Simulationsgestützte Systemgestaltung - DGLR-Bericht 2007-04, pages 313-331, Bonn: DGLR e.V.

ACT-CV - Machine Vision for Cognitive Modeling
© 2015 Marc Halbruegge (