Assignments

Reading Annotations

Each week a selection of course readings will be posted as a single assignment on Perusall - a system for students to collaboratively annotate course readings. To earn this credit, you will be expected to post 3 quality annotations per assignment to Perusall. A quality annotation is one in which you synthesize concepts, ask thought-provoking questions, or connect ideas to external issues. I have found that students get the most out of Perusall when they respond to each other’s annotations. Annotations must be completed before class to receive credit.

Note

A single assignment can have multiple readings, but you need only submit 3 annotations total. You do not need to annotate every reading in the assignment.

Field Notes

Ethnographers keep field journals to thickly document what they observe in social environments. Typically field notes will describe a physical environment and the participants in that environment (including appearances, roles, behaviors, moods, interactions, conversations, etc.). Field notes will also interpret what was observed, considering how the empirical material affirms, contests, or transforms how we understand cultural phenomena. In data ethnography, the social environment under consideration can be anywhere data is produced, applied, presented, or talked about. This assignment is about paying attention to the numbers (and other forms of data) that inhabit daily life and practicing cultural interpretation.

Learning Objectives

  • Recall, define, explain, apply, and synthesize course concepts and ideas
  • Engage the genre and practice of thick description
  • Develop observational habits in data-saturated environments
  • Develop skill in interpreting cultural meaning

Instructions

Consider a statistic, dataset, data standard, data infrastructure, or data collection environment that you encountered in the past two weeks. Write 300-400 words, first, detailing the environment and, second, responding to one of the following prompts.

  • What cultural assumptions about what knowledge is and how knowledge is produced can you glean from this data environment?
  • How do binary oppositions structure discourse about data in this environment? With what consequences?
  • What metaphors are used to characterize data in this environment, and what assumptions about data are conveyed through these metaphors?
  • What is one master narrative architecturally designed into this data infrastructure? How does that narrative manifest both semiotically and materially through the data infrastructure?
  • What invisible forms of labor sustain this data infrastructure or data environment? How do the individuals that engage in that labor approach and care for their work? What social forces render that work invisible?
  • How is data science defined in this space? Through what kinds of boundary work do social groups advocate for what should count as legitimate data science? What are the social consequences of this definitional work?
  • What counts as credible data expertise in this data environment? How do these ideas mediate who gets a voice in the data environment?
  • In what ways are data practices embodied in this environment? How does attention to embodied practice change the ways we understand the data?
  • What rituals do data collectors engage in this data environment? What shared beliefs motivate these rituals, and how do the rituals manifest in the “raw” data that gets produced?
  • What stakes do individuals have in how numbers in this data environment get reported? How do social or financial incentives to have the numbers meet or beat certain thresholds animate how individuals behave and the choices they make in relation to this data collection practice, data analysis, or data reporting? Why do those incentives exist?
  • How is the meaning of this statistic or data value socially debated, negotiated, or settled upon? At what thresholds do the data values become actionable, and how do social groups determine levels of actionability? How and under what conditions do these decisions get made?
  • How are data socially mobilized in this space? What strategies do individuals, institutions, or social groups engage to render the data persuasive?

Sample Work

  • Previous Student Field Notes here
  • Here’s an example of some very short ““thick description”” write-ups of two data environments from my own research.

Criteria for completion

  • Student writes 300-400 words about a statistic, dataset, data standard, or data collection environment.
  • Student writes about a data situation encountered in the past two weeks.
  • Student responds to a prompt, engaging course concepts to interpret the cultural meaning of the data situation analyzed.

Mini-Projects + Substantive Revision

The course mini-projects will provide you with opportunities to practice ethnographic research and cultural analysis. Mini-projects are the predominant way that you will practice the Data Ethnography Methods learning dimension in this course.

Note

Please note that you may complete the mini-projects independently or in a pair. If you choose to complete the project as a pair, both students must participate in data collection, analysis, and writing. Be prepared to submit a short paragraph describing the breakdown of labor, along with your write-up.

Learning Objectives

  • Identify, implement, and critique data ethnography research methods
  • Develop observational habits in data-saturated environments
  • Develop skill in analyzing social forces and systems
  • Plan and execute a substantive writing revision, while evaluating feedback

Instructions

For this assignment, you will select a standard classification system that supports a dataset or system. You will then investigate that information infrastructure through a socio-cultural lens, aiming to respond to the question: What is a cultural assumption embedded in this classification system that goes on to shape the social outcomes of the data it supports? You may briefly research the history of changes to the infrastructure; however, this assignment is not a literature review, and you should limit your use of secondary sources to support your arguments. This paper should represent your cultural analysis of this dataset, not that of others. To do so, you should closely examine the structure of the classification system, using the Infrastructural Analysis Worksheet.

Selecting a data infrastructure

Before you select an infrastructure, you should identify a public dataset that organizes information into a series categories

Examples of classification systems that I’ve seen referenced in public datasets include:

  • Standard Industrial Classification System (SIC) or the North American Industrial Classification System (NAICS)
  • Carnegie Classification of Institutions of Higher Education
  • BISAC Subject Headings List
  • Diagnostic and Statistical Manual of Mental Disorders (DSM)
  • Uniform Crime Reporting crime classifications
  • Association of Computing Machinery (ACM) Computing Classification System
  • Candid’s Philanthropy Classification System (PCS)

Analyzing the data infrastructure

To analyze this infrastructure, you should complete the Infrastructural Analysis Worksheet, which you will submit along with the final write-up. You should aim to answer most of the questions in this worksheet, but you do not need to answer all of them. To answer these questions, you can rely on a diverse range of resources. First and foremost, you should browse the classification system itself, noting how concepts are organized. However, to answer the questions in the worksheet, you may also need to review the websites of the organizations that created/steward the classification, review changelogs that identify changes to the classification, and conduct research to find journal articles, news articles, blog posts, or other grey literature, discussing the classification.Finally, you should consider how everything you learned shaped the way values are presented in the dataset where you originally identified this infrastructure.

Write-up

Following your analysis, you will write-up a 750-1000 word memo presenting an an answer to the question: What is a cultural assumption embedded in this classification system that goes on to shape the social outcomes of the data it supports? I encourage you to use this memo template for outlining your write-up. You should engage thick description as you detail the structure and organization of the infrastructure and interpret its symbolic meaning.

Important

I want to emphasize that this is not a literature review, and you should not approach this assignment simply by regurgitating what you learn in sources you find. Instead, you should focus on analyzing the ways in which the infrastructure both enables and delimits knowledge production by closely studying how its component parts are organized and how they impact your dataset, along with which stakeholder groups advocated for certain changes. The sources you find should serve to provide context for the points that you make. Your write-up should have introductory and concluding paragraphs that clearly indicate the central point you are making in the write-up and that point should be carried through all of the examples and arguments presented in the write-up.

Criteria for completion

  • Student submits a mostly complete Infrastructural Analysis Worksheet.
  • Student writes a 750-1000 word memo describing and interpreting the data infrastructure. All sources are properly cited in the write-up.
  • Student synthesizes evidence into one argument, presented at the start of the write-up and carried through each paragraph.

For this assignment, you have two options: you can 1) watch 1 video (at least 45 minutes long) of a talk about data delivered by a data scientist, or 2) three TED talks (at least 15 minutes each) about data. Please confirm your selection with me ahead of time by DM’ing me on Slack.

Analyzing the talks

Following the framework in the Discourse Analysis Worksheet, you should examine how data is talked about in the video(s). Consider the epistemologies vocalized, the binary oppositions that structure the language, the rhetorical strategies used, and what this tells us about data culture. You should take detailed notes by responding to several sections within the worksheet. As “data” to back up claims regarding what you notice about the discourse, you should transcribe a series of quotes from the talks. You can add these quotes directly to the worksheet. You will submit the worksheet with your final write-up.

Write-up

Following your analysis, you will write-up a 750-1000 word memo detailing an argument about the data discourse conveyed in the talk(s). What cultural assumptions about (choose one): 1) what data is, 2) data’s relationship with truth, or 3) what makes data good vs. bad … are reflected in discourse about data? I encourage you to use this memo template for outlining your write-up. Direct quotations should be referenced throughout (and cited) to back up your claims, but do not count towards the overall word count. Your write-up should have introductory and concluding paragraphs that clearly indicate the central point you are making in the write-up and that point should be carried through all of the examples and arguments presented in the write-up.

Warning

If you choose to watch three TED Talks, please note that you should still only present one argument, synthesizing what you learn across all three videos. This means that you should compare discourse across multiple talks in each paragraph. Assignments that present three distinct discourse analyses/do not offer in-paragraph synthesis and comparison will not receive full credit.

Criteria for completion

  • Student submits the Discourse Analysis Worksheet with several sections completed.
  • Student writes a 750-1000 word memo detailing the discourse of the talk(s). Direct quotations are included as evidence, and not included in the overall word count.
  • Student synthesizes evidence into one argument, presented at the start of the write-up and carried through each paragraph. If analyzing multiple talks, student offers comparison and synthesis in each paragraph.

For this assignment you will conduct a 30-minute Zoom interview with an individual for whom data figures centrally in their work. The goal of your interview will be to deepen understanding of the cultures of data work.

Preparing for the interview

Your interview should address the question: How does the interviewee define the boundaries of data science? How do they define/what do they consider legitimate data science expertise? Write up 10 open-ended questions in preparation for the interview that aim to unpack this theme. You should submit these 10 interview questions along with your memo in your Google Drive.

During the interview

Ask the interviewee if they are comfortable with you recording the conversation. You may record with their express permission. If they do not agree to recording, you should be prepared to take notes during the interview.

Analyzing the interview

You may wish to use this worksheet to help you with your analysis. However, you are not required to submit this worksheet with your memo.

Write-up

Following your interview, you will write-up a 750-1000 word memo describing and interpreting what was discussed throughout the conversation. Your write-up should address the prompt: How does the interviewee define the boundaries of data science? How do they define/what do they consider legitimate data science expertise? I encourage you to use this memo template for outlining your write-up. Your write-up should have introductory and concluding paragraphs that clearly indicate the central point you are making in the write-up and that point should be carried through all of the examples and arguments presented in the write-up.

Criteria for completion

  • Student writes/submits 10 relevant interview questions developed in preparation for the interview.
  • Student writes/submits a 750-1000 word memo describing and interpreting what was discussed in the interview.
  • Student synthesizes evidence into one argument, presented at the start of the write-up and carried through each paragraph.

For this assignment, you will observe a setting in which data figures centrally. You may choose to observe a class engaging in some form of data collection, a hack-a-thon, or even a friend working on a coding assignment for a data science course. The only data setting that I discourage for this assignment is talks that are being presented on-line. These are better suited for the discourse analysis mini-project. As you observe, you will be aiming to gather data that addresses one of the following two questions: 1) How are data practices embodied in this data environment, and with what outcomes/consequences for the data? OR 2) What rituals are engaged in this data, and with what outcomes/consequences for the data?

Before the Event

Prior to your observation, you need to obtain permission to observe. If you are observing a class, you should get permission from the instructor. If you are observing an event, you should get permission from the organizer. …if a lab, get permission from the lab manager or faculty director, etc. Be sure to cc me when you email to obtain permission. You may use this email template when obtaining permission.

During the Observation

Throughout the event you should take extensive field notes (at least 6 double-spaced, typed pages) about the physical environment of the event, how people behave and interact, how people talk about and data, the rituals they perform, the infrastructures they engage, and more generally how data figures into the social setting. You may talk to individuals in the environment to ask clarifying question, but if you do, you should disclose that you are an ethnographer studying the space.

Analyzing the Observation

You may wish to use this worksheet to help you with your analysis. However, you are not required to submit this worksheet with your memo.

Write-up

Following your observation, you will write-up a 750-1000 word memo describing the event and offering a qualitative interpretive analysis responding to either 1) How are data practices embodied in this data environment, and with what outcomes/consequences for the data? OR 2) What rituals are engaged in this data, and with what outcomes/consequences for the data?. I encourage you to use this memo template for outlining your write-up. Your write-up should have introductory and concluding paragraphs that clearly indicate the central point you are making in the write-up and that point should be carried through all of the examples and arguments presented in the write-up.

Criteria for completion

  • Student submits the equivalent of about 6 double-spaced typed pages of field notes taken during the event.
  • Student writes/submits a 750-1000 word memo describing and interpreting the event.
  • Student synthesizes evidence into one argument, presented at the start of the write-up and carried through each paragraph.

Revision

After receiving feedback from me on both submissions, you will have an opportunity to revise one of your submissions. This assignment has three steps:

  • Read this document on creating a revision plan.
  • Complete the Revision Plan worksheet.
  • Revise your submission in the same document you had previously been working in based on what you outlined in your revision plan.

Remember that grammar, spelling, and punctuation concerns should not be included in a revision plan. Those concerns are addressed in a proofreading stage, not a revising stage.

Resources

  • Memo template is here
  • Example Mini-Project 1 are here
Note

Due to IRB restrictions, I cannot provide examples of past Mini-Project 2 assignments. If you are struggling to figure out how to compose the write-up, please plan to visit office hours.

Final Project

In May 2009, Data.gov - a web portal for accessing US government datasets - was launched by then federal Chief Information Officer Vivek Kundra. Following this, in December 2009, then US President Barack Obama signed the Open Government Data Directive, requiring that all federal agencies post at least 3 high value datasets on data.gov within 45 day. A few years later in May 2013, Pres. Obama signed an Executive Order to: “Mak[e] Open and Machine Readable the New Default for Government Information.”

The Order required that the US Office of Management and Budgeting, in collaboration with the CIO and CTO, put out and oversee an Open Data Policy. This policy required the following:

  • Data needs to be published in machine-readable formats
  • Data needs to be licensed openly
  • Data needs to be described with metadata.

Prompt

For this assignment, I would like you to imagine that the Data.gov Program Management Office in US General Services Administration Technology Transformation Services, the Office of Government and Information Services (OGIS), and the Office of Management and Budget (OMB) are jointly championing a new initiative to improve the accessibility of open government metadata. Administrative metadata describing how datasets are managed or stewarded gets published with most open government datasets. Similarly, descriptive metadata gets published in the form of data dictionaries that provide official definitions of observations and variables present in the dataset. However, user-friendly descriptions of how the dataset was produced, how standard definitions were chosen, how categories were divided, how measurements get taken, what assumptions and judgments are built into the data, why certain information may be missing, and where and how the data gets referenced is often harder to come by and rarely published in a succinct format accessible to diverse users.

To help advance this initiative, imagine the offices will be contracting with a number of teams of consultants to develop a series of dataset user guides. Unlike many existing metadata formats, dataset user guides will be narrative documents. In the process of advancing a similar municipal level initiative, the NYC Open Data Team (with Julia Marden, Tiny Panther Consulting, Sharon Lintz and Jo Polanco) defined dataset user guides as describing “the content of a dataset, how it was created, the agency who maintains it, and how users can begin to use the data.”

This semester your project group will develop a user guide for a federal dataset. Each week you will work through exercises that help apply course concepts from that week to the study of this dataset. Note that you will study the dataset ethnographically and strive to effectively communicate how the dataset came to be, how it has disseminated, how it can be used, and what some of its limitations are.

Your final user guide may be presented in any format you wish, and I encourage you to be creative. However, the format you choose should be tailored to an audience that you identify in advance.

Learning Objectives

  • Apply the course concepts in studying the provenance of a dataset
  • Develop skill in analyzing social forces and systems
  • Produce effective and descriptive data documentation that puts data in context
  • Plan and execute a substantive writing revision, while evaluating feedback

Exercises

Assignment Expectations

  • The exercises I provide are not graded and not reviewed by me; they are meant to frame your research and help you generate material for the project. I will only be formally reviewing your final user guide submission.
  • Because each group is working on a different dataset, there will likely be components of each exercise that are not relevant to your group; I only expect you to complete about 70% of each exercise. Skip the parts that aren’t relevant to you.
  • You should be meeting with your groups at least one hour each week outside of class to coordinate your work and plan to work an additional hour independently to make progress towards your research.
  • The point of this project is not to perform a statistical analysis on this dataset. Instead, you will study the dataset ethnographically and strive to effectively communicate how the dataset came to be, how it has disseminated, how it can be used, and what some of its limitations are.
  • By November, at least part of your group work time should transition from researching the dataset to composing a user guide. Your final submission should not be a compilation of text from the exercises you completed. That text was written for you, not for an outside audience. Part of the goal of this project is to be able to translate your research into information that is accessible to an audience.

Project Resources

Criteria for Completion

  • Student team submits a completed group contract, first draft, and final project draft on time.
  • Student team generates a user guide that demonstrates having completed substantial research across all five project exercises.
  • Student team generates a user guide with language, visuals, and formats tailored towards a particular audience.

Community Labor

Ethnography is at its best when it involves collaborative inquiry and interpretation. Because of this, I want to encourage us to foster a cooperative community in our classroom. Ethnographers know that building and sustaining strong communities are important and often invisible forms of labor. In an effort to foreground and reward that labor, I’ve built opportunities to contribute to the course community into our grading contract. There are four opportunities for earning community labor points in this course. It will be your responsibility to keep track of your community labor via our course labor log.

Contributing on Slack

The first opportunity for earning community labor points is through posting in the #sds-237-discussions Slack channel. For every conversation that you initiate in this channel, you will earn 1 community labor point. For every conversation that you respond to in this channel with substantive summary, critique, or reflection, you will earn 1 community labor point. Finally, for every question that you ask or answer in the #sds-237-questions channel, you will earn 1 community labor point.

Completing Group Evaluations

Mid-way through the semester you will have an opportunity to evaluate your own contributions to your group work, along with that of your peers. Completing group evaluations will count for 1 community labor point.

Participating in Peer Review Workshops

Throughout the semester, I will set up workshops where students may submit their written work for peer review in preparation for assignment submissions. Each peer review workshop that you participate in will count for 2 community labor points.

Contributing Class Notes

Following a class period, up to two students may type up a 1-page outline of what was covered in that class period and post a link to that outline in #sds-237-class-notes. You can sign-up to serve as the notetaker for a certain class here. These notes should be a full, single-space page, and should make sense to someone that was not present in class. Each class note outline that you contribute will count for 2 community labor points.

Note that throughout the semester, I may offer additional opportunities to earn community labor points as unexpected forms of course labor arise.

Advanced: Lead a Class Reading Discussion

Leading a class reading discussion is one way to demonstrate advanced level work. Reading discussions should be 15 minutes maximum. Because we have other activities planned, I will need to cut off discussions if they go too long. Students leading a discussion should start by taking 2-3 minutes to summarize the main arguments of the reading, dicussing how the author supports those arguments. After this, it is up to the student(s) presenting how they wish to structure the remaining time. I’ve had students pose discussion questions to the whole class. I’ve also had students pose questions for small groups to discuss first (note that if you take this option, there is really only time to discuss at most 2 questions; trust me, the time goes by fast). I’ve also had students do more experimental things like create Kahoots or present case studies for the class to think through. It’s up to you! Just get us reflecting on the reading! Often students create slides to help structure the discussion, but this is not necessarily a requirement.