The Game of Video Game Object: A Minimal Theory of when we see Pixels as Objects rather than Pictures

While looking to the future, we have overlooked what is right before us. With new technology, haptics, rendering, virtual reality, we have spent much energy discussing immersion and presence, thinking sometimes about current technology, but often about hypothetical perfect experiences or future perfect technology.

In this, we have forgotten something rather fundamental: How do we in the first place decide to see a group of pixels on a screen as an object to which we have access, rather than as a picture of an object? This paper explores this question through a playable essay. At first, we may think that we will identify anything interactive as an object, but the playable essay demonstrates that this is much more complex and pragmatic, and that this identification has three steps – identifying pixels as an object rather than a picture, reasoning about the object as a specific type of object (such as a ball), and identifying it as a real instance of a type of object (such as a calculator).

I conclude that we identify objects not with a general list of properties (like “being interactive” or “physical”), but on implicit rules that we use depending on the type of object we are considering, and on what we are trying to do at a given time. I identify nine such tentative rules. Finally, there are many kinds of video game worlds, from the default 3D worlds of many game engines to social worlds. Examining the Unity3D engine used to create the game, I argue that game worlds are fundamentally not designed as bottom-up simulations of a world, but are deliberately implemented in human categories, and that we understand them as such. Within that frame, our relation to video game objects is pragmatic, and we will accept pixels as objects when it is helpful to our goals and plans.