Note from the Editor, Tricia Wang: Kicking off our Co-designing with machines edition is Alicia Dudek (@aliciadudek), Innovation Insight Lead & Design Ethnographer at Deloitte Digital Australia. Using design thinking, ethnography, and other deep contextual customer research methods, she designs, conducts, and trains others in the world of customer empathy. Her contribution to this edition is the first science fiction to explore robots conducting ethnographic work. She uses a fictional story with Cee Cee, the robo-ethnographer, to examine what aspects of fieldwork can be conducted by a robot. I first met Alicia Dudek at an EPIC conference in London, where I became a fan of her work and promptly interviewed her for our edition that featured the best from EPIC. Read the interview, Play nice – design ethnographer meets management consultant, and find more of her writings on ethnography at her site.
Increasingly we are seeing more conversations about ‘what does it look like when the robots take your job.’ Once upon a time we believed this was some remote future where we’d finally invented the technology that could replace our bio-body’s ingenious functions. Now we are coming into a time where our technology has grown so advanced that the replacement of ourselves with robots is not only imagined, but plausible and even possible. An example of this shift is the imagining of white collar jobs ‘going robo’ that was recently covered by Quartz.
Writing this piece I wanted to have a little fun imagining a wonderful world where we can work hand in hand with robot peers. It is exciting to imagine the day when artificial intelligence is on par with that of our human research team members. Ethnographic technology is sometimes slow to progress due to the art and science nature of our work, but if we had the magic wand to unite all the drones, phones, data smarts, and humanly arts, we might have robo-colleagues as a part of our team one day soon. Friendly humans and friendly robots conducting ethnography together are a powerful combination. Also thank you to Elizabeth Dubois for writing this piece about trace interviews, which has some cool ideas on how we might conduct interviews.
– Alicia Dudek
Lou muses over her tea as she prepares for fieldwork with Cee Cee.
Lou mused over the steam rising from her cup of tea. She gathered her thoughts around what she’d be looking for in the field next week. She and her team were going to shadow young families and understand how they managed their finances. Field work was always one of the most exciting and exhausting parts of the data collection in her ethnography projects. What would be the right focus area for a trip into this family’s everyday life? She knew she’d have to cover the basics of bank accounts, credit cards, laptop / tablet / phone usage, calendar keeping, overall scheduling, family diaries, but what else might be valuable? What else could help to point the team in the direction of the golden nuggets of insight? All these years of traipsing in and out of the field and analysing scores of transcripts, videos, audios had left her always questioning, what’s next? What were the mental parameters that led her to the deep and meaningful insights from field observations? What was that ineffable thing that clients kept hiring her for again and again? How does an ethnographer see differently to find the golden nuggets?
Lou was jostled out of this reverie as Cee Cee energetically buzzed into the office and landed on Louise’s desk with a plop. “Louise I’m here for my briefing for the field work to be conducted.” Lou looked up from her imagined fieldwork and focused on Cee Cee’s entry into her office. In the past Lou had had dozens of assistants, grad students, and junior ethnographers to help with her work. None of them was quite like Cee Cee, who was rather innovative and definitely pushed Lou’s ways of working to new places. “Alright Cee Cee let’s get going on the briefing and I’ll tell you what we’re looking for and how to behave when you get out there.” Lou readjusted her posture and swung around to meet Cee Cee head on and get into the briefing.
“Listen carefully Cee Cee, we’re going out to shadow young families and learn about how they manage their finances. It’s going to be quite a list of things we’re looking to understand. We’re trying to observe these people’s lives and how they make decisions. This means we’ll be shadowing them through their entire day for at least 30 days – at home, at work, at school, at sports practice, everywhere reasonable. We’re looking for anything tied to money, value, decision-making, and trade-offs. We want to know how they decide what to do with their resources and how they plan for their future. We’re looking for any artefacts or systems they use to help with this, including calendars, diaries, spreadsheets, notepads, notice boards, any documentation of time or money earned or spent. Make sure you grab a peek and record any of that, and ask them to explain how they use that stuff.
We’ll want to mostly observe and fade into the background, but we can ask a few questions for the family to talk us through how they do things. Start by asking the family about themselves and their story. Then later on you can progress to ask about how their money beliefs were formed, especially about who influenced them. A good indicator is also what was their family like and what has their life been like overall. Make sure you snap images of any family photos or wall art.” Lou leaned back and sipped the tea for a breather.
“Hmm, what else do you need to know Cee Cee… We’re also looking for how the parents teach the kids about money. So any lessons, piggy banks, chore lists, you know those sorts of things are of interest. What we’d really like to know is how they feel about any of the time or money decisions they need to make. We’re looking for their facial expressions, body language, and how they react to different stimuli and conversation topics. You can even check their temperature to see if any particular topics get them heated or bothered.” Lou paused and ran through her standard list of briefing points one more time.
“Make sure you scan the barcodes you see in their environment and log the items they reference. I want to see a full 3D mapping of the spaces they live in and all the objects that are in those places. We’ll then be able to calculate the net value of the environment. Ooh and make sure you pay careful attention to the usage or wear patterns on any technology, flooring etc. We want to know what they are handling, touching, and using. What do you think Cee, is that a clear enough brief? Do you have any questions?”
Cee Cee was visibly processing the long list of instructions Lou had just downloaded. “Just a couple questions, Lou.”
“Ok go Cee Cee, what do you want to know?”
“Will I be flying home for the night?”
“No we’ll have you installed in the house with a charging cradle in an out of the way place. I am sure they’ll forget you’re even there in no time, as long as the kids don’t catch you and try to take you apart to see how you work, like that time when you were doing field work for me in Kansas. So zoom on fast and go be the fly on their walls.”
“Got it boss. I’ll go get started on preparations. First I’ll pop into the lab and get the mechbots to install the thermal to my camera array and double check all my rotors and systems. Then I’ll charge for the night and ensure the delivery bots get me to the first subjects’ house tomorrow with the charger cradle. Duration of observation per subject is due for 30 days during which I will livestream data into the holoserver room for your perusal.”
“Thanks Cee Cee, I look forward to reviewing what you collect and Loz’s initial pattern analysis.” With that Lou dismissed Cee Cee and turned back to the cup of tea that was still slightly warm in her hands and swigged the last gulps. As she swallowed the tea she grinned thinking, my how things have changed!
30 days later…
As she meanders down the corridor towards the holoserver, Lou scrolls through the list of notifications from Cee Cee in the field over the last few days. Cee Cee has indicated a very normal distribution of traffic except for a few shifts in the patterns. Lou double taps and zooms into one of the meta analyses of human traffic throughout the Robertsons’ family home. One of Cee Cee’s notations comments on what a large percentage of time is spent in the garage by the father and the two older kids. She can see that Loz has selected some representative video VR scenes to illustrate what that time usually looks like. Lou decides to take a quick preview through the VR sensors in her regular glasses.
She takes a stroll through the 3D visualisation of the Robertsons’ garage and sure enough both older kids and the dad are all crowded around a tool bench tinkering with robo parts. “A hobby for the family, little human bots building more bots,” she chuckles under her breath, and approaches the holoserver projecting Loz’s bearded face on the frosted glass door.
She walks through the automatic door into the holoserver cube and carefully pulls on the VR sense suit. “Morning Loz, let’s get Cee Cee’s last data running up so I can see the entire highlight reel.” Securing her gloves and popping the goggles over her eyes, she is immersed in flooding sunlight as she stands in front of the Robertsons’ front door.
In the VR goggles, Lou catches her breath as her eyeballs reel at the 360 degree visual immersion. Calibrating, Lou focuses on the data she wanted to see, “overview please Loz.” Loz’s immersive voice replied promptly, “Sure, Lou, no worries.” The virtual walls in her goggles quickly show her an overall data visualisation of all the sights and sounds that Cee Cee was able to record in her 30 day observation.
“Loz, your preliminary analysis indicates insight probability around body temperature spike correlations? Please show me the sub data for this.” Lou starts scrolling through the instances of spiked temperatures. She’s noticing that in the elevated temperature readings, Loz has pulled the physical activity-related spikes into a separate analysis. “Hmm so it isn’t exercise, perhaps a wee bit more emotional?” Suspending the video, Lou can see that the majority of these moments are found in the garage. “Let’s see the transcript analysis for these moments, key quotes, trace interview analysis, and play me the top three; oh but first give me the financial insight overview.”
Loz begins a rundown of the financial insights, “Lou we know from their banking history that both adults came from ‘squandered inheritance’ backgrounds so they’ve both picked up rather conservative risk radars. This has led to their early choices to hand over the financial budgeting and planning to AIbankers and roboinvestors. Since the family was set-up this way there is very little financial deviation from the preset path. The only unconstrained part of the finances is the spending on education and educational materials for the 3 children.”
“Ok that’s not bad. Let’s go back to those garage moments; let’s play the video. I’d like to start with the video correlated to the big heat spike and the root cause in the trace interview analysis.” Lou watches and listens as daddy Robertson calmly takes the two kids through some complex robotics wiring to make the robot tail on their contraption open and close smoothly. He hands the tools over to the elder child and lets them have a go at putting the delicate robot parts right. Lou can see his temperature rising as the kid starts to fumble and force the parts into place, but his face is placid. The temperature differential hits its zenith when the kid smashes the motor and parts roll across the garage floor. “Ok, let’s have a highlight reel of parallel moments throughout the 30 days.”
Loz queues up all the similar moments from the month with the Robertsons, and Lou is stunned to see there were over a dozen examples of the delicate robot parts getting ruined during the kids’ garage sessions with dad. As Lou listens to the transcripts and views the conversations, she started to form a feeling in her gut that these robot parts have a very deep emotional trigger for daddy Robertson. As she approaches the 8th example of the robot parts scene, she can see the trace analysis is very brightly correlated and robustly related to lots of other text threads from daddy Robertson across the month.
From experience Lou knows that particular gut feeling of an approaching golden nugget. She can feel her so-called spidey sense tingling as she dives into the part of the story that Loz had indicated as possible root cause. Lou watches daddy Robertson gingerly holding some of the broken robot parts as he tells the kids about how in his childhood he never had the chance to build things because of how his parents had hovered over him in fear that he would break expensive parts. He continued on about how he wanted the kids to have respect, but never fear breaking the parts as long as they were learning.
“Loz, how many other participants displayed themes of something like ‘acceptable loss in pursuit of learning’ or ‘fear of failure’? I want to see any decisions made where the loss was accounted for as a learning expense.” Loz displays 14 other participant families that had shown strong verbal and personality correlations to the theme. Lou peers through the 3D quote cloud that represents all the statements the family members had made that filtered into that theme. “What an odd vein we’ve stumbled into Loz! I think we will have to mark this down as a possible meta-theme for further exploration in the prototyping phase. A possible golden nugget we’ve got here! Prepare a pre-report on this theme throughout all 36 households and have it projected in the design cube, and call in the design team. Have the teabot deliver my cuppa and I’ll be down there in a short while. Very exciting, good work team.”
Loz projects a beaming bearded holoface at her holding up a peace sign as Lou pulls off the gloves and suit and swishes out of the room.
“Groovy work Lou. l’ve sent the pre-report to your design cube and booted up the design team so they’ll be ready for analysis and ideation when you arrive. I’ve also sent you the overall meta-theme report with the top 10 candidates for meta-themes highlighted and networked with their supporting data points. I will run the other meta-themes through the designAI bootcamp algorithms in case any great new innovations pop-up out of the mix.”
*Notes on a brief history of Cee Cee & Loz, a pair of transformative design researchers:
Computer AI-aided electronic ethnographers (CEE) had their first recorded mention in an Ethnography Matters post way back in 2016. In the following years, advancements in facial and body recognition technology, the shrinking of battery & drone tech, and further leaps in cognitive computing and AI allowed ethnotechnologists to cobble together the first Cee Cee. This effort was underpinned by the group Ethnotechwork and their crowdfunding success on Kickstarter, where they raised over 17 million dollars in under 93 hours.
The success came as a huge surprise as their original goal had only been for around 15,200 USD, to buy a drone and combine Noldus FaceReader technology, advanced Dragon dictation software, and their own home brew AI (a close cousin of IBM’s Watson). You could say Cee Cee’s lineage began with Watson, Dragon, & FaceReader. Yet to give Cee Cee life, Ethnotechwork had their work cut out for them and assembled a crack team, who were later dubbed the first ethnotechnologists. They set out to combine the home brew AI smarts with all the technological appendages that would be useful for Cee Cee to best interact and collect data in the field.
Cee Cee model 1 was a simple cobbling together of a Hexo+ drone with an additional array of 3D spatial mapping technology, but it could record holistic data of an individual’s interactions with their environment. Further advances in natural language processing as well the addition of AI-driven thermal imaging allowed Cee Cee to actually ‘see’ emotions in observed subjects and note those moments in the data to help speed up the pattern finding algorithms. Cee Cee model 2 was a fully equipped self-flying people-o-meter, with custom designed indoor drone tech that allowed easy manoeuvring in very tight interior spaces. At this stage the underlying infrastructure required to collect comprehensive field data was solid, but the ethnographic part of Cee Cee’s home brew cognitive computing brain needed feeding.
Ethnotechwork started putting together a plan to ‘feed’ Cee Cee’s ‘brain’ all of the world’s ethnographies, countless theses, blog posts, articles, published studies, transcripts, videos and books. Ethnotechnologists mapped a wide swath of subject matter that Cee Cee would need, from linguistics, to behavioural economics, to basics of home and car repair. It had all seemed very straightforward to the crew of ethnotechnologists to feed the brain everything they could get their hands on and help the cognitive machinery get itself into useful order.
When the public got wind of Cee Cee’s next iteration, in which all the knowledge needed to deeply analyse humanity was going to be fed into the mind of an autonomous AI, and furthermore, it was an AI that could fly and spy, things got a bit heated. Protests against Ethnotechwork started up across the EU and US initially and spread quite rapidly. People were livid at the thought that the rise of the machines was imminent and believed that work feeding Cee Cee’s brain had to be halted immediately. Protesters demanded, how could Cee Cee be ethical? Transparent? Or anything other than a bloody nuisance? Round and round the debates started to go. Ethnotechwork started getting threatened with violence and attacks. The labs were even evacuated a number of times as the security risks were rising.
It was starting to look like Cee Cee model 3 would never happen. The ethnotechnologists were dismayed and helplessness started to set in across the team, but they decided to host a last ditch hack-a-thon to see what they could come up with to assuage the public opinion disaster outside their doors. Ethnotechwork invited in dozens of their staunchest advocates, prominent futurists, and everyone who wanted to help bring Cee Cee model 3 to the world. They hacked all night and all day looking for a way to make the whole endeavour possible.
In the quiet wee hours of the morning, Jo, one of the UX designers, burst out into a bit of a hysterical fit of laughter. The folks still groggily moving around her snapped to attention thinking she’d lost her composure, not unlike a few who already had cracked in the marathon effort. Jo ran to the whiteboard and grabbed one of the green markers and feverishly started sketching a huge green button. She frantically started tapping the whiteboard with the pen,
“This is it crew. It is the answer. It has been in our own professional history since the beginning of human centred design. All the way back since the PARC crew at Xerox did that famous early observational research that led to the green button!”
The sleepy crowd was growing around her whiteboard as she went on excitedly explaining, “You see to make Cee Cee model 3 palatable, it’s not about a green button.”
Jo grabbed the fattest red marker she could find and started retracing the lines of the green button in huge red swaths of colour. “It’s about a HUGE red button. A giant stop button of control, agency, and empowerment for humanity. They just want to be able to stop Cee Cee model 3 whenever they see fit. To assert their position as subject. It’s the consent form opt-out at any time clause made physical!”
“It’s too simple, it can’t possibly be that simple.” cried out one of the voices in the crowd. Then silence fell over the room again. Slowly murmurs started and the crowd was visibly nodding along and the idea began to gain momentum. As the elegance of the solution dawned, all around people began to grab paper and pens and started sketching out different options, features, and permutations of the giant red stop button. Voice command versions came first. Then facial recognition based ‘stop’ buttons, and on and on into the middle of the day.
Eventually Ethnotechwork finessed the ‘opt-out at any time clause made physical’ to such a degree that public opinion subsided (and a Kardashian got married so the public was quickly distracted). That’s how the way was paved for Cee Cee model 3 to be released as the first full-fledged computer AI-aided electronic ethnographer with an impressive set of brains to match the technology on the outside. To this day Cee Cee’s exterior cladding still carries the inscription “You may ask me to stop at any time” with a crude drawing of the original red button concept.
This article is part the Co-designing with Machines Edition. Read other articles in this edition.
Like what you’re reading? Ethnography Matters is a volunteer run site with no advertising. We’re happy to keep it that way, but we need your help. We don’t need your donations, we just want you to spread the word. Our reward for running this site is watching people spread our articles. Join us on twitter, tweet about articles you like, share them with your colleagues, or become a contributor. Also join our Slack to have deeper discussions about the readings and/or to connect with others who use applied ethnography; you don’t have to have any training in ethnography to join, we welcome all human-centric people from designers to engineers.
