Agre’s Surveillance and Capture, 26 Years later

This is an edited text based on a talk Lisa Conrad, PhD and me gave at the EASST 2020 in the panel on classic Science and Technology Studies (STS) papers. I think the text is also interesting for computer scientists, design researchers and privacy researchers, so I post it here.

Any page numbers that appear in the text as (p.###) refer to the page numbers in Agre, Philip E. “Surveillance and Capture: Two Models of Privacy.” The Information Society 10, no. 2 (April 1, 1994): 101–27.

I added links and edited out some abbreviations and such, but mostly kept the text as we read it in our talk. Additional comments and explanations are in blocks like the one this very paragraph is in.

“Surveillance and capture: two models of privacy” 1 is a 26-page article published in the Journal “The Information Society” in 1994. In this article, Philip Agre (from UCSD’s department of communication) contrasts the models of surveillance and capture, two ways of gaining knowledge about people and things. Surveillance “is built upon visual metaphors and derives from historical experiences of secret police surveillance.” (p.101) whereas capture “…is built upon linguistic metaphors and takes as its prototype the deliberate reorganization of industrial work activities to allow computers to track them in real time.” (p. 101)

Capture is achieved by creating “grammars of action”: activities are divided into basic units; these units are then put into a certain sequence. Transforming activities into sequences of basic units makes them representable within computers. Agre explores the social, political and economic implications of capture. In this talk, we would like to draw out 5 aspects of the paper that – we think – turns it into a classic Science and Technology Studies-paper. By doing so, we also propose certain qualities that distinguish an Science and Technology Studies-paper.

Middle-range theoretical concepts

The text’s key concept, capture, is a concept that aligns with other middle-range theoretical concepts from Science and Technology Studies-research, such as Star’s boundary object, Jasanoff’s sociotechnical imaginary or Latour’s blackboxing. This means, it is an abstraction from empirical observations that lends itself to being applied to other empirical settings. It also means that the concept is propositional. It does not claim to be in any way a final word.

What does capture mean? Agre suggests the concept of capture to describe a specific way of applying computers to the world. It is “a way of looking at things… a set of metaphors” (p.105) most evident in the “professional practice of computer people” (p.122) and the systems they build. Simply put, capture stands for the practice of re-organizing certain activities so that they can be represented by computers in real time (p.109).

Agre says, his paper “introduces a reasonably substantive theory of capture as part of the historical dynamic of a market economy” (p.122) But he sees his conceptualization of capture as a “hypothesis seeking validation” (p.121): “It is far too early to make any final assessment of capture as a social phenomenon.” (p.115)

Subversion of Dualities

This ”dualisms” discussed here is a recurring topic in science and technology studies. But also non-academics will be familiar with it. You get in contact with a culture/technology dualism when people say that a technology (e.g. a gun) is neutral, that it is the people using it causing the problem (Attribution only to ‘the culture’. Or when kids consuming a lot of internet videos, this is attributed to “the algorithm” of YouTube (attribution to ‘the technological’). And subverting the dualism is assuming that there it is not either technology or culture but both combined, a hybrid 2

Firstly, the concept of capture subverts the dualism of the social vs. technology. Instead, it explores the fundamental entanglement of social and technological issues. Secondly, Agre’s model of capture subverts dualist assumptions on the effects of technology: that it is either utopian,helping to unravel oppressive structures or anti-utopian, helping to enforce these structures.

In contrast, the concept of capture aims at “…a more complicated appreciation…” (p.116) of these issues of power and change. It draws attention to “complex patterns of negotiation and control that operate in all directions, not just from the top down” (p.115).

Grounded in careful empirical accounts of actual technologies

The analysis is grounded in qualitative empirical research. Agre develops the concept of capture to condense observations he makes in industrialized work settings. He portrays the use of computerized tracking systems in the context of package delivery, accounting and bookkeeping, just-in-time manufacturing and state bureaucracy (p.102-104). He gives detailed and comprehensible descriptions of technological devices and arrangements (i.e. data collection via bar code readers at NCR, location-sensitive bracelets worn by people convicted of crimes, tracking trucks via GPS, computer systems measuring and comparing store performances).

a scanning device for tracking packages CC-BY-SA 2 By: IAEA 3. Source, Title: „Package tracking barcode scanner 3“

He shares with other scholars in Science and Technology Studies that he writes of technology in a detailed and accessible style. This style testifies to a great appreciation of technology and that it is not seen as secondary to ‘social’ issues. Agre (just like other Science and Technology Studies scholars) deals with specific technologies, not just 'tech' in a generic sense. At the same time, his writing is aware of the amount of technological details a lay reader can reasonably process. He does not get carried away by his excitement and inside knowledge.

A work- and organization-based perspective on representation/intervention

This section talks about representation. Typical forms of representation in the natural sciences might be graphs, tables or formulas – typical topics for Science and Technology researchers. But interestingly, “capture” is not a science concept…

Agre discusses a topic well known in Science and Technology Studies: The creation of formal and abstract structures that are said to “represent”. Instead of looking at scientific fields of physics or biology, Agre discusses representation in the context of increasing work efficiency and computerization in industry jobs, a project that is not driven by scientists and their experimentation but by computing professionals and their philosophical assumptions about work practices and representability.

a person in from of a computer, the screens show code on a black background.

developer at work – CC-BY-SA 3 by Matthew (WMF) Source, Title: „Coding Shots Annual Plan high res-5“

It might seem strange to talk about philosophical assumptions of programmers here, but this is the interesting point. The way that their discipline and culture sees the worlds is “capturable”: Almost anything in the world can be described for computers and made computable. Agre explores this topic in other publications, too, like “Formalization as a Social Project”6, “The Soul Gained and Lost”7 and “Writing and Representation” 8. In these publications, Agre argues that computing professionals and the disciplines of AI and computer science assume that the world and activities in it are representable in language-like, logical patterns and also reshape the world to work with this assumption.

For capture to function, one needs to “…make human activities and physical processes trackable”. For this, activities need to be “…reconstructed through assimilation to a transcendent (‘virtual’) order of mathematical formalism" (p.107)”. As the model of capture is based on linguistic metaphors, Agre uses the term “Grammar of Action” for the formalized and representable activity.

Some of my other posts here on the blog are about user research. And one thing I do as user researcher is creating/describe grammars of action: I describe what people do and often put it in a flow diagram for product managers and developers to work with.

a flowchart, formalizing the editing process on the collaborative semantic ‘knowledge base’ Wikidata

CC-BY 4: by Jan Dittrich (WMDE) Source

According to Agre, this formalization is accompanied by the myth that it “…has been ‘discovered.’” The grammars of action, their basic units and the sequencing are assumed to be a clean and clear language-like structure which ‘underlies’ human activities, but is buried beneath the ongoing flow of daily work – a platonistic idea 3. “The activity in question, in other words, is said to have already been organized according to the grammar” (p.110).

However, rather than being a transparent representation, capturing an activity actually has profound effects on the activity itself: Capture claims to represent, but it actually intervenes. Grammars of action are applied. They involve a re-organization (p.110) of the activities to make them compatible with computer technologies. The redesigned activity is then encouraged by “social and technical means” (p.110), becoming ingrained in the work so that people “… out of necessity, orient their activities toward the capture machinery and its institutional consequences”

This is why I said above that user researchers create/describe grammars of action: Although the creation/description is based on empirical study, the grammars of action are an abstraction (which always could be otherwise) and new products and features and build upon the described grammar of action inevitably change and shape the activities in the grammar of action.

Your workplace’s time tracking system influences what gets counted and recognized as work. Assuming work can be measured by time and that it is best measured by a punch clock changes how employees see work, although it seems that the system just neutrally tracks the concepts “start of work” and “end of work”, which according, to the grammar of action, have been there all the time and stay as they are after the punch clock is introduced.

Web platforms like LinkedIn take an assumed abstract activity like “maintaining business contacts” and they split them up in what “underlies” them: Exchanging messages, expressing appreciation by “liking” etc. But e.g. “liking” did not exist before, it is a new thing, but it is said that it “represents” the “underlying” concept of appreciating something.

Similarly, a web forum like discourse models conversations based on the idea that underlying a conversation is the exchange of information, that this exchange happens in taking turns, that in each turn parties react to previous information and that conversations are triggered by an initial message that kicks the conversation off.

Longevity of the concepts

A 1994 IBM ThinkPad 360 This is how a really modern laptop looked like at the time Agre wrote his paper – this model had up to 50MHz single core processing power, in 2020 we are at 4400MHz on 8 cores, if you buy a high end model (Image source)

“Surveillance and Capture” strongly draws upon the developments at the time of writing: Agre’s examples refer to very early instances of ubiquitous computing like tracking of goods and people. It was written when the world wide web was just getting more popular and only briefly after mosaic was developed, the first web browser to integrate images and text.

Yet, its mid-range theoretical concepts (capture and grammars of action) can be used to analyze current developments: Grammars of actions can be seen in “many (if not all)” (p. 109) user interfaces and thus in modern web services – provided by google, facebook or amazon. Their automatic, real-time analysis of the activities of their users allows them to exploit user actions for their “behavioral surplus” (Zuboff) which is used to generate huge wealth of internet companies 4.

Agre also suggests that capture can reduce transaction costs. He hypothesises that capturing and computerization will lead to actors “[transforming] productive activities along a trajectory toward an increasingly detailed reliance upon (or subjection to) market relations” and continues that “The result is a generalized acceleration of economic activity whose social benefits in terms of productive efficiency are clear enough, but whose social costs ought to be a matter of concern.” This, in hindsight, reads like a foreshadowing of the tightly tracked and market-driven gig- and click- economies of uber and mechanical turk.

Even more spot on is this quote: “The ultimate use of such measurements is the establishment of bidding for services in real-time markets, whereby the control function previously provided by bureaucracy is transferred to the inherent discipline of the market.” (p.117) Which reads very much like gig- and click-platforms 5

Thus, we find the paper has aged very well and continues to provide a useful framework for researching work, organization and its intimate entanglements with computer technology.


We think paper is a good candidate for a classic Science and Technology Studies text:

  1. Its core concepts, capture and the related grammars of action, are similar to other mid-range theoretical concepts from Science and Technology Studies research
  2. It is nuanced in its non-dualism of social and technological aspects. They are treated as intertwined in “sociotechnical activity systems” (p.105). Also, capture subverts dualist assumptions on utopian or dystopian effects of technology and instead focuses on “complex patterns of negotiation”
  3. It is grounded in empirical observations and concrete descriptions of specific technologies.
  4. It discusses the myth of transparent representation, a typical Science and Technology Studies topic, but it does so in the context of computerization in organizations and industry.
  5. While written 26 years ago based on examples taken from the fast-moving field of computing, its theoretical models, capture and grammars of action are still useful to analyze developments that happened long after the paper’s writing.

We hope to have motivated the audience to (re) read the paper – also to discover other interesting points that we could not discuss in the limited time.

We think these 5 points are also what distinguishes Science and Technology Studies research and we are curious to hear what you think.

(By Lisa and Jan , edited for this blog by Jan)

  1. I guess this is aiming at Foucault’s Surveiller et punir, (“Surveil and Punish”) although Foucault’s book is published as “Disciplin and Punish” in English. 

  2. The hybrid of gun and person e.g. is described in Latour, Bruno. “On Technical Mediation.” Common Knowledge 3, no. 2 (1994): 29–64. 

  3. Platon argued that what we perceive is not really real; what we perceive are merely shadows of what is really real. Universal forms create the mere shadows. See Wikipedia’s Theory of Forms article. 

  4. Zuboffs book “The Age of Surveillance Capitalism” introduces the mentioned term, “behavioral surplus” for anything that can be a tracked about users and subsequently capitalized. See this review for a summary 

  5. So why does it need capture create such marketplaces? The challenge is that market mechanisms have their costs, and this is why firms are internally not organized like a market themselves, as Coase argues in his classic “The nature of the firm”. Since capture lowers the costs of these market mechanisms, gig- and click-work becomes possible. Coase, R. H. “The Nature of the Firm.” Economica 4, no. 16 (1937): 386–405. 

  6. Agre, Philip E. “Formalization as a Social Project.” Quarterly Newsletter of the Laboratory of Comparative Human Cognition 1, no. 14 (1992): 25–27. 

  7. Agre, Philip E. “The Soul Gained and Lost: Artificial Intelligence as a Philosophical Project.” Stanford Humanities Review 4, no. 2 (1995): 1–19. 

  8. Agre, Philip. “Writing and Representation.” Narrative Intelligence 46 (2003): 281.