ANNOtator Challenge – Develop a game to gamify the annotating
Background
Deep learning has enabled reliable image classification and object detection. In our reseach, we focus on the transferability of deep learning techniques to the art domain. However, deep learning technologies are based on deep neural networks consisting of millions of parameters that need to be learned. To train such networks, also hundreds of thousands annotations are needed. Thus, finding ways to acquire large amount of labelled data is necessary.
The Challenge
In all areas, computer vision relies on training data. These are images with framed objects to give the computer classification examples to learn. Large databases are operated around the world for this purpose, mostly based on photographs. But pictures from history, like artworks and graphics, stem from their own visual culture which deviates from the ‘realistic’ photo and shows big regional and epochal differences. Therefore, we need a huge amount of annotations to understand the content of the image (theme/ iconography), to capture historical objects (real-life facts) and to sort the artwork according to criteria (interpictoriality).
It’s about a bigger accessibility to our past and a deeper development of the cultural heritage, for science as well as for the public. Getting the necessary annotations in good quality is very resource-intensive and in many cases it cannot be done by the humanities. A huge problem of the crowdsourcing of annotations is the control of quality. In this case, it’s guaranteed by the game itself: as a buyer, players control the goods on offer themselves.
Therefore the task of the challenge is to develop a game to gamify the annotating – so it’s fun!
Your starting point: code and image data
Regarding an interface for image annotation to integrate in your game development, you may use „Exact“ (developed by Christian Marzahl). This is a collaborative online tool for labeling image data and you can retreive it from GitHub: https://github.com/ChristianMarzahl/Exact
Please use only pictures from early modernity and European Art (about 1400 – 1800) or a closer frame fitting to your game design. Useful online repositories that provide you with open access material are:
Art Institute Chicago https://www.artic.edu/image-licensing
British Museum https://www.britishmuseum.org/terms-use/copyright-and-permissions
Cleveland Museum of Art https://www.clevelandart.org/open-access
J. Paul Getty Center https://search.getty.edu/gateway/landing
Metropolitan Museum, New York https://www.metmuseum.org/about-the-met/policies-and-documents/open-access
National Gallery Washington https://images.nga.gov/en/page/show_home_page.html
Rijksmuseum, Amsterdam https://www.rijksmuseum.nl/en/search?ii=0&p=1
Städel Museum, Frankfurt a. M. https://sammlung.staedelmuseum.de/en/concept#cc
If you even want to dig deeper, check this list of open access in GLAM:
https://docs.google.com/spreadsheets/d/1WPS-KJptUJ-o8SXtg00llcxq0IKJu8eO6Ege_GrLaNc/edit#gid=1216556120
And if you would like to use any other data or have questions, feel free to ask us.
Your aim
The most annotation games are kept very simple and directly target the annotation, as for example ARTigo, sometimes with the consequence that the games become boring or even unfair. The annotation should only fill in a small part of the game and quality control should also be included.
The idea of the ANNOtator is that the annotation itself is the source material and capital of the player: ANNOtator is a classical trading simulation that takes place in the Middle Ages and the Early Modern Period, inspired by games like Patrician, Hanse or Anno 1602. You trade with things, literally taken out of the images. This could be pitchers, fruits, furniture, tulips, arms and many other popular goods at that time. At the places the players travel to, they are able to work through different regionally adapted image data sets and secure their goods which they sell elsewhere. In short: the market regulates the annotations.
But of course scientists can also intervene in the game by displaying new demands or saturations.
What to submit
Develop an initial game concept with pen and paper. How can the game be fun, which types of players does it appeal to? How can annotation be gamified? Be inspired by the works of art. How can you help shape the style and visual core of the game? Aim for a game design with good integration of the annotations and unite game graphics and an intuitive annotation interface. Finally, a promising marketing concept for the game will accompany your submission.
Your award
The team that submitted the best solution for our challenge will be invited for a week-long stay here at FAU in Erlangen, Bavaria, Germany, in early summer 2021. We will cover your travel expenses and offer you an interesting program where you can learn more about our research, but also about the region. This will include a visit of the medieval city of Nürnberg as well as a hiking trip to the unique natural preserve of the Franconian Alb.
As a special price, we will take you on a tour through the Germanisches Nationalmuseum, which is the largest museum of cultural history in the German-speaking region. The curator will take you behind the scenes and show you a great variety of artwork that is just waiting for some annotation work.
In case that travelling will not be possible due to COVID-19 restrictions, your team will be awarded with 2,500 €, but of course, we hope that it will be possible to welcome to here at FAU soon.
Source: FAU Open Research Challenge