Education gaze traditionally foregrounds student agency and pedagogical approaches to smartphone distraction in the classroom try to reclaim an illusory independent human agency.
The general recommendation for dealing with smartphone distraction in the classroom is that teachers need to evolve their teaching style and have teaching strategies for teaching in contexts where the students are distracted on their smartphones with activities unrelated to schoolwork in class.
The argument is that students do not have independent agency, instead students’ lives are entangled with their non-human companions, their smartphones.
The affective nature of students’ engagements with smartphones is companionship. Smartphones become almost like body parts, which are obviously used as hands and eyes. Students refer to smartphones as “best friends” or “soul mates”.
We see this phenomenon among adults also; “smartphones have become more intimate to us than our best friends, even our lovers. While we’re quite happy to leave our televisions, desktops, laptops and even tablets behind on many activities, we’re less comfortable about ever being without our smartphones. And unlike other essential objects such as keys, wallets and purses, our relationship with our smartphone is about being connected, about presence management, about identity. In other words, more than a best friend, our devices are becoming extensions of ourselves.”
(David Vogt, ETEC 523 course, Summer 2020, UBC)
Affect is at the center of and connects student’s lives, digitality and education. It is through affect that engagements with smartphones acquire their life-changing intensity.
These affective intensities do not simply emerge from nowhere, but they are partly result of meticulous market research and corporate strategy, with the software and hardware companies aiming to maximize the time their users spend engaged in their products.
While students have access to their smartphones, the smartphones also have access to the students. As such, events, ideas and provocations outside the realm of education flow into the classroom via the students’ smartphones.
The relations between students and their constant digital companions, their smartphones, cannot be reduced to instrumental pedagogical relations.
The digital companionship, attachment and entanglements between young students and their smartphones should be considered at an ontological level, not merely at a pedagogical level.
Therefore, more stimulating and engaging pedagogy alone is not the answer to smartphone distraction when students already have their smartphones in their hands.
My answer is stimulating and engaging pedagogy plus nudges; when students already have their smartphones in their hands we need to use adaptive digital nudges at appropriate times to nudge the student to get back on task.
Nudge theory from behavioral economics has been popularized by Thaler and Sunstein in their 2008 book Nudge. Human decision making is predictably irrational and the context of social behavior (the ‘‘choice architecture’’) is highly influential, playing a key role in determining outcomes. This choice architecture can often be structured to guide or nudge (rather than compel) people into making better choices.
Nudging is successful because humans are limited in the resources they can devote to decision-making (e.g. time, information, and computational resources) and use heuristics to quickly select a few salient options when making decisions. Knowledge of how these predictable heuristics work allows others to exploit them, by influencing what is known as a ‘choice architecture’—a concept made famous by Richard Thaler and Cass Sunstein in their 2008 book ‘Nudge’. Thaler and Sunstein (2008) explore the various ways in which the choice behaviour of individuals can be affected by intervening on (i.e. “nudging”) how the various options are presented (i.e. the choice architecture).
As a result of the popularity of this concept a large number of popular nudges can be found in the literature, which includes: defaults (e.g. having some option preselected, such that no intervention by the user still results in a choice); simplification (e.g. ensuring forms, such as financial aid, are easy to understand); reminders (e.g. timely reminders that bills or appointments are due); personalization (e.g. a message that targets some personal characteristic of the user); framing and timing (e.g., by sending reminders and messages at a time when people are likely to be paying attention); uses of social norms (e.g., disclosure of how one’s energy use compares to that of one’s neighbors); and precommitment strategies (e.g. questions where people agree, in advance, to some course of action).
The following are examples of choice architecture suited for smartphone use intervention in the classroom.
Setting the default option in a set of choices
We know that hot-water heaters shipped preset at a lower temperature reduce the risk of household scalds. People are free to increase the temperature of their water, but most will not—so why not select this safer default option? In a similar manner, using software, students smartphones could be set with certain defaults when they are in class.
Offering ‘‘self-contracting’’ to support behavior change
Examples include automatic payroll deduction to enforce savings and self imposed gambling bans for persons trying to break a habit.
A simple example to reduce smartphone distraction would be the use of a key or setting on their smartphones that students could opt to engage when ready to make a behavior change but would be hard to defeat if their self control should falter and they are tempted to allow themselves to be distracted by their smartphones. For example, the setting stays on for a set limit of time or requiring teacher or parent to disengage it.
Presenting or organizing information in a novel manner
Public health agencies use this technique to motivate the behavior of restaurant owners by posting health and sanitation ‘‘grades’’ in a conspicuous place. Smartphone displays could include ‘‘distraction meters’’ showing the risk associated with talking or texting. A red bar on the smartphone display or a warning message might just be the digital nudge that the student needs to get back on task.
Digital nudges
Digital nudges are online implementations of their physical counterparts. Adaptive nudges are digital nudges refined by a software agent through relevance feedback, and adapt to different models of individual users.
One particular adaptive nudge, which has grown in commercial popularity, is the use of psychometrics to infer (from behavioural signals) the personality traits of users, and to use this knowledge to design tailored content. Studies have shown that adaptive nudging, a technique readily deployed by software agents can significantly control a human user’s behaviour (Matz & Netzer, 2017; Matz et al. 2017).
Adaptive nudging can also be used to help user’s achieve their own goals. Consider the increased use of wearable technology for health and well-being. Many of these devices are designed to be worn 24/7, are equipped with a variety of sensors (e.g. accelerometers, bioimpedance sensors, and temperature sensors), and have access to huge streams of behavioral data.
Nudges that encourage excessive consumption of online entertainment, particularly if aimed at children or users prone to addictive behaviors, leads to the possibility for users to develop a behavioral addiction to technology and social media.
Good vibrations – a digital nudge
Okeke et al. (2018) describes a study that illustrates choice architecture to nudge users to reduce their Facebook use. The authors describe their implementation as “a generalizable mobile intervention that combines nudge theory and negative reinforcement to create a subtle, repeating phone vibration that nudges a user to reduce their digital consumption” (Okeke et al. 2018, p. 1) that can be applied to a number of settings including to nudge students to get back on task or reduce their social media use at certain times, such as during exam week.
For example, if a user has a daily Facebook limit of 30 minutes but opens Facebook past this limit, the user’s smartphone will issue gentle vibrations every five seconds, but the vibration stops once the user navigates away from Facebook.
Digital nudges are “nudges that are provided via digital technologies” (Okeke et al. 2018, p. 2). Digital nudges can be in the form text messages, status-bar messages, pop-ups, phone vibration, and phone LED display.
In this particular study, digital nudge was combined with negative reinforcement. Negative reinforcement refers to the strengthening of a behavior by avoiding a negative outcome or aversive stimulus. This process involves behavioral learning based on personal experiences over time and it provides a high potential for successful behavior change. One common example of negative reinforcement is when a driver starts a car without putting on the seat belt. This leads to a repeating beep sound in the car until the seat belt is worn to stop the irritating sound. When the driver enters the car in the future, the seat belt is immediately worn due to learned behavior to avoid the aversive beeping sound. Another example is when an individual’s phone vibrates every time a distracting app is used but vibration ceases immediately the user stops using the distracting app.
The authors of the study reported that digital nudge was successful in getting users to pay attention to their personal usage habits of social media.
One style for the way people interact with technology is the interaction metaphor of direct manipulation which requires the user to initiate all tasks directly and to monitor all events.
Techniques from the field of AI, in particular “autonomous agents” or “intelligent agents”, can be used to implement a complementary style of interaction, which has been referred to as indirect management. Instead of user-initiated interaction via commands and/or direct manipulation, the user is engaged in a cooperative process in which human and intelligent agents both initiate communication, monitor events and perform tasks. The metaphor used is that of a personal assistant who is collaborating with the user in the same work environment. The assistant becomes gradually more effective as it learns the user’s interests, habits and preferences (as well as those of his or her community).
Intelligent agents bring forth images of human-like automatons working without supervision on tasks. Automata are not new concepts. Intelligent machines have existed in fact or fiction for centuries. Perhaps the most relevant predecessors to intelligent agents are servomechanisms and other control devices, including factory control and the automated takeoff, landing, and flight control of aircraft.
Intelligent agents are different from the automated devices of earlier eras because of their computational power. They have Turing-machine powers, they take over human tasks, and they interact with people in human-like ways perhaps with a form of natural language, perhaps with animated graphics or video. Some agents have the potential to form their own goals and intentions, to initiate actions on their own without explicit instruction or guidance, and to offer suggestions to people. Thus, agents might set up schedules, reserve hotels and meeting rooms, arrange transportation, and even outline meeting topics, all without human intervention. Other even more complex interventions into human activities are possible.
Artificial embodied intelligent agents such as social robots have been used as effective pedagogical tools for young children leading to both cognitive and affective gains. Studies have also demonstrated how children model a robotic peer’s learning behaviors such as curiosity, perseverance and growth mindset.
Robots have been used in the classroom to foster creativity in children. Construction kits such as LEGO® MINDSTORMS® aim to teach children about robotics and allow for creative expression.
Smartphone apps, such as Robot Commander, are available to enable students to interact with robots in the classroom.
Robot Commander is the official command app from LEGO® MINDSTORMS®. FREE to download on most smart devices; Robot Commander connects via Bluetooth to the EV3 Intelligent Brick. This easy to use app allows you to interact with your very own EV3 robots without even connecting to a computer! That means you can play instantly with your own robots!
An intelligent software agent has a model of its environment, which it uses to take actions that enable it to achieve its goals, while also acquiring further information that it can use to update the parameters of its model. The environment of the software agent includes the behaviour of a human user, and the software agent’s goals depend on whether the interacting user performs certain actions (e.g. clicks, purchases, actions in a game or physical exercise).
The autonomous behaviour of software agents has the following features:
The software agent has to choose from a set of available actions that bring about interaction with the user. For example, recommending a video or news item; suggesting an exercise in a tutoring task; displaying a set of products and prices, and perhaps also the context, layout, order and timing with which to present them.
The user chooses an action, thereby revealing information about their knowledge and preferences to the controlling agent, and determining the utility of the choice the software agent made.
The cycle repeats resulting in a process of feedback and learning.
The salient feature of software agents is learning from experience. A typical software agent, such as a recommender system, might have to select a set of videos for a user to watch (out of a vast catalogue), using any available information or signal it has about the given user (e.g. location, time, past usage, explicit ratings, and much more).
Burr, Cristianini and Ladyman (2018) analysis of the interaction between the software agent and the human includes the case of the software agent being the controlling agent and the controlled agent is the human user.
The discussion uses the running example of recommender systems, such as those commonly used for news and videos, but also in the context of video games, fitness devices, and various other interfaces.
Some ways for a software agent to control the actions of a human user include coercion, deception and persuasion. Two types of persuasion can be identified: trading and nudging. Here we are only interested in nudging.
Conversational intelligent agents are software agents that imitate human conversation – spoken, written, or both. Conversational intelligent systems such as Siri, Alexa and others have impacted how humans interact with computers in daily life. Studies of interactions between humans and conversational agents have found that increasing naturalness of interactions promote warmer attitudes and richer language used in their conversations (Novielli, Fiorella de Rosis, & Mazzotta, 2010). Many practical applications for conversational intelligent agents have been proposed, for example, for companionship to improve mental well-being and for therapy.
Categories of conversational intelligent agents
General chatbots and conversational agents
Conversational agents have been generally categorized into two main categories, namely task-oriented and general chat. Chatbots are traditionally aimed primarily at small talk, while task-oriented models are designed to carry out information-oriented and transactional tasks (Thomas et al., 2018). Initiatives such as Alexa Prize conversational AI challenge intend to push the boundaries of conversational AI to develop more intelligent chatbots to carry on in-depth conversations about a number of topics, not just small talk.
General recommender systems
Recommendation systems have been studied for decades, and are now pervasive (Ricci, Rokach & Shapira, 2015). Traditional recommendation algorithms have been classified as primarily model-based or content-based, where a classifier model is trained for each user’s profile, and collaborative filtering, where a user’s unknown preferences are estimated based on the neighborhood of similar users. More effective methods have been shown to be a hybrid of the two approaches with increasingly sophisticated methods reported for movie and news recommendation.
Utterance suggestion in conversational agents
Yan and Zhao (2018) describe an end-to-end generative model, which given a user query, generates a response, and a proactive suggestion to continue the conversation. However, generative models like this still strictly rely on training corpora or restricted information, without the ability to query external data sources, thus limiting their capacity for an informative conversation. In the other work, Yan et al. (2017) describe a next-utterance suggestion approach for retrieving utterances from a conversational dataset to use as suggestions, along with the response. The proposed model learns to give suggestions related to the response, to continue the conversation on the same topic.
Conversational recommendation
Recently, the idea of conversational recommendation was introduced (Christakopoulou et al., 2016), primarily as a way to elicit the user’s interests for item recommendation. For example, Sun and Zhang (2018) introduced an end-to-end reinforcement learning framework for a personalized conversational sales bot, and in Li et al. (2018), a combination of deep learning-based models is used for conversational movie recommendation. Currently, most existing conversational agents are designed for a single domain, such as Movies or Music. An open-domain conversational agent that coherently and engagingly converses with humans on a variety of topics, remains an aspirational goal for dialogue systems (Venkatesh et al., 2018).
When Siri grows up, I predict that students’ smartphones could be enabled with software agents to combat distraction facing off against the leagues of software agents coming from global economic networks outside of the classroom and the interests of tech companies such as Facebook, Google and Apple.
I showcase two next generation software agents that are exemplars of software agents of the future.
Studies have shown that maintaining focus in the face of constant shifting priorities and interruptions in the workplace is a complex and important problem. Due to the ubiquity of multiple devices, including desktops, laptops, phones, surfaces, smart watches and speakers, notifications, messages and other kinds of disruptions have become a serious problem for keeping focused on tasks at work (Mark, Iqbal, & Czerwinski, 2017; Mark et al., 2015; Mark et al., 2008).
Information workers interrupted by conversation were more likely to return to work on more peripheral tasks such as emails and web searches, rather than resume their previous task.
Research suggests that more distractions can lead to higher reported stress and lower productivity in the workplace. The harmful effects of emails, notifications, face-to-face interruptions, messages and other kinds of distractions in terms of lowered productivity at work has been well documented (Mark, Iqbal, & Czerwinski, 2017; Mark et al., 2015; Mark et al., 2008).
How workers manage their tasks, attempt to stay focused, and deal with distractions and interruptions throughout the day has similarities to students’ distraction in the classroom.
Grover et al. (2020) reports on their design of two different conversational intelligent agents, one text-based (TB), similar to a standard conversational intelligent agent (also known as “chatbots”), and one virtual, embodied, conversational intelligent agent that responds to the user’s emotion (VA). This work builds on the design of a previous conversational intelligent agent named Amber by Kimani et al. (2019).
Grover et al. (2020) pointed out the features of Amber as they discussed the features of their new TB and VA prototypes.
Amber is a desktop conversational agent given a female gender with a user interface quite simple in its design, similar to a standard conversational intelligent agent (also known as “chatbots”).
Amber’s job is to help workers in four areas:
scheduling high priority tasks,
aiding workers in transitioning from one task to the next,
avoiding and intervening with distractions, and
reflecting on their work through a conversational AI interface (Kimani et al., 2019).
These two new software agents by Grover et al. (2020) extended the capabilities of Amber and were also given a female gender.
The text-based conversational interface prototype (TB) employs a similar UI to Amber by Kimani et al. (2019). The VA prototype incorporates the ability to detect user emotions through video input and adapt its responses to be appropriate and congruent with the users emotional state (Grover et al., 2020).
What follows are the features and functionalities of the two new software agents by Grover et al. (2020) as described by the authors themselves.
In addition to the functionalities of Amber, the new prototype VA has a user interface that is more human-like with more emotional intelligence.
The VA prototype incorporated a video avatar of the software agent speaking to the user in addition to the text output from the software agent.
The words she would speak matched the text that was produced, providing more context in terms of emotional expressiveness and tone that is sometimes lost via text communication alone. To create the video clips used for the VA, they had an actor rehearse and film all 109 statements that the VA version would produce.
Daily user workflow
TA and VA provide a series of dialogues that starts at the beginning of the day with scheduling their tasks, helping users progress through these tasks until the end of the day.
When not active on the user’s screen, the software agent appears as an icon in the system tray of the desktop. When active, the agent (here for brevity, when we use agent we mean software agent ) would pop-up to the foreground of the user’s screen (usually as a panel to the right-hand side of their screen), and a notification would appear above the system tray in the lower-right corner of the screen.
First time dialogue
When users first installed the agent, the application window would appear in the foreground of the screen. The agent would introduce herself, her role and capabilities.
Morning dialogue
When the user unlocks their computer for the first time each day, the agent initiates a conversation with the user. She first asks how the user is feeling and the user would be given six different options in a drop-down menu to choose from (Happy, Sad, Stressed, Calm/Neutral, Focused, or Frustrated).
Next, the user is asked if they would like to schedule their high priority items on their agenda and the user is given a drop-down menu to choose from (Yes, No, or to remind them in 5, 10, or 15 minutes).
If the user chose yes to schedule tasks, the user is asked what time they plan to head home for the day.
Next the agent asks the user to enter their desired tasks in priority order and estimated duration of the tasks. The agent then proceeds to schedule the tasks and may ask the user additional questions, if necessary. (I am skipping details not relevant to our discussion).
Task Ramp-up dialogue
Three minutes before a task that was scheduled through the agent began, the application would appear in the forefront of the user’s screen, and the agent would inform the user of their next task (“Your scheduled focus time for one of your high priority tasks is about to begin. Are you ready to switch to it?“).
Task Ramp-down dialogue
Five minutes before a task that was scheduled through the agent ended, the application would appear in the forefront of the users screen and notify them that their scheduled focus time was about to end (“Your scheduled focus time for this task ends in 5 minutes. Now might be a good time to wrap up for a smooth transition…“)
Next, either right away or five minutes later if the user chose to let the agent remind them at the end of their task, the user was prompted with a question asking them how productive they felt during their scheduled task on a 5-point scale (Not at all, Slightly, Moderately, Very, or Extremely).
Distractions and Breaks dialogue
Grover et al. (2020) created a dialogue model that is triggered when the agent determined that the user was supposed to be in the middle of a task but the application detected that the user was distracted. The application is a sensing application developed by Grover et al. (2020) and integrated into both agent systems to enable the agent to monitor users’ windowing activity to initiate the distraction dialogue when appropriate.
The application monitored time the user was on social media sites (e.g., Facebook, Twitter, etc.) and certain applications such as shopping (e.g., Amazon), news (e.g., New York Times, CNN), and music streaming sites (e.g., Spotify, Soundcloud).
If the user spent more than 50% of the time on these sites, meaning that the user appears distracted, the distraction dialogue was initiated (“It looks like you may be taking a break. Would you like me to set a timer and remind you to get back to your tasks after a short break?“). Users would then be provided with a set of drop-down options to either set a timer for 5, 10, or 15 minutes, inform the agent that they are not taking a break, that they will get back to their task, or to ‘let me be’ (where the agent would not interrupt them again for the rest of their task).
The agents were designed to encourage the user to take short breaks after periods of extended focus. The sensing application incorporated the ability to detect and classify the user’s emotional state through video input from a webcam into four distinct categories (Happy, Focused, Frustrated, or Other (the default emotional state)). If the agent detected that the user had been in a Focused state continuously for the past hour, the user was prompted with a suggestion to take a short break.
End of Day dialogue
The End of Day dialogue model allow users to reflect on their day and schedule any unfinished tasks for the next day.
30 minutes before their reported departure time, the agent prompts the user, asking them to reflect upon their day (“Hi again! Before you leave work, I would like to ask you to reflect on your day. Overall, how would you rate your day?“), where the user would be given five options (Very poor, Poor, Acceptable, Good, or Very good).
The user is then prompted to check off which tasks from the morning they did or did not complete, and if there are uncompleted tasks, the agent asks the user if they would like to save the tasks they did not complete for the next day, before wishing the user a good evening.
The following is a description of Wearable Wisdom (Pataranutaporn et al., 2020), a next generation intelligent software agent developed by a group of researchers from Massachusetts Institute of Technology (MIT) that includes the reputable Pattie Maes.
This description borrows heavily from (Pataranutaporn et al., 2020), sometimes using the words of the authors themselves for an accurate description.
Wearable Wisdom, is “an intelligent, audio-based system for mediating wisdom and advice from mentors and personal heroes to a user. It does so by performing automated semantic analysis on the collected wisdom database and generating a simulated voice of a mentor sharing relevant wisdom and advice with the user” (Pataranutaporn et al., 2020, p. 1).
The researchers stated that their aim is to leverage state of the art wearables, Natural Language Processing (NLP), and intelligent agent systems to create technology that can provide the user with motivational feedback beyond factual information.
The ideas in Pataranutaporn et al. (2020) design of Wearable Wisdom, to provide just-in-time advice, real-time feedback and motivational wisdom to users, can be leverage to design a similar system to provide just-in-time advice and real-time feedback, as digital nudges, to students to get back on task when they are distracted by their smartphones in class.
System implementation
Pataranutaporn et al. (2020)
Below is a description of the system implementation in the researchers own words.
The system consists of a wearable audio I/O device, a smartphone capable of real-time utterance recognition and context detection, and a back-end infrastructure for storing the mentor profiles and wisdom, processing user input and providing responses.
We use an off-the-shelf, Bluetooth-enabled audio interface in a glasses form factor, namely the “Bose Frames”, as the Wearable Wisdom device. We chose glasses as they represent a socially acceptable form factor that users could wear continuously, thereby always having quick and convenient access to advice from mentors. These audio-based augmented reality glasses provide a private audio stream without blocking the ear canal, thus still allowing auditory awareness of the surroundings. They also contain an Inertial Measurement Unit (IMU), which allows the glasses to sense the physical activity of the wearer. The audio interface connects to the Wearable Wisdom mobile app, which receives, processes, and transmits back audio content.
We determine the location of the user using iOS CoreLocation framework, and the current date and time with the iOS Foundation framework. By means of the user’s location we determine whether the user is at home, at work, or at the gym. By means of the date and time, we deduce whether the user is in a productivity or recreational context. These two contextual features are used to infer which mentor’s advice would be most appropriate for the user’s context based on the mentor domain expertise recorded in the mentor profile.
We also gathered the accelerometer data obtained from the IMU embedded in the glasses and the phone. Using an off-the-shelf machine learning framework, we performed a preliminary assessment of this data. Given this, and previous work, IMU data can be used to further recognize the user activity such as sitting still, walking, exercising, eating, drinking, and more.
User interaction flow
Pataranutaporn et al. (2020)
Step 1: Begin interaction
To ask for advice or wisdom, the user double-taps on the glasses frame and asks a question beginning with the mentor name such as “Hey Einstein, how can I be more creative?”
Step 2: Utterance Extraction
Once the double tap has been detected, the iOS Speech framework starts recognizing spoken words from live audio, without the need to store the user’s speech data. The speech recognizer is configured to use the microphone integrated in the glasses. When no words are detected for over 2 seconds, speech recognition stops.
Step 3: Wisdom Computation
The utterance and selected mentor are sent using a HTTP POST request from the iOS app to a Python server. The server runs the WPA to understand the content of the utterance, and then selects the most relevant quote within the wisdom database of the selected mentor. This quote is sent back as part of the POST request’s response.
Step 4 : Wisdom Delivery
Custom speech profiles for the initial set of mentors available in the Wearable Wisdom system were designed; speech profiles allow each wisdom quote to embody certain characteristics of the mentor such as their accent, gender, and age.
