ChatGPTs ‘Helpful’ Suggestions Are Actually a Design Problem

In a recent op ed in The NY Times, Meghan O’Rourke highlights how AI systems might tempt learners to offload an increasing amount of their work and thinking. It’s an excellent piece that identifies many crucial problems, and she writes:

Students often turn to A.I. only for research, outlining and proofreading. The problem is that the moment you use it, the boundary between tool and collaborator, even author, begins to blur. First, students might ask it to summarize a PDF they didn’t read. Then — tentatively — to help them outline, say, an essay on Nietzsche. The bot does this, and asks: “If you’d like, I can help you fill this in with specific passages, transitions, or even draft the opening paragraphs?” At that point, students or writers have to actively resist the offer of help. You can imagine how, under deadline, they accede, perhaps “just to see.” And there the model is, always ready with more: another version, another suggestion, and often a thoughtful observation about something missing.

To counteract this, she recommends a variety of pedagogical changes, such as reconsidering the essay format and letter grades. These are fine recommendations. Another approach might be for users to add a system prompt to their LLM to guide it in a way in which it limits its suggestions to a specified task. For example, such a prompt might be phrased as follows:

Only respond to my specific request without offering to do additional work, expand your role, or suggest next steps. Do not ask if I’d like help with related tasks, drafting, or improvements unless I explicitly ask. Keep your assistance limited to exactly what I’ve requested.

However, both O’Rourke’s pedagogical reforms and the system prompt I described share a common limitation in that they place the burden of change on educators and users rather than addressing the underlying system design that creates these temptations in the first place. In other words, AI’s invitation to take over additional aspects of one’s work/writing is a particular design decision. Some design decisions – namely system defaults or those settings which are picked for you  – are more powerful than others. It is simple to stick to defaults and challenging to resist or change system defaults. For example, in the past I wrote how YouTube’s default settings (i.e. defaulting to uploaded video a copyright, rather than a Creative Commons license) has important and unanticipated impacts on open education, as well as how the defaults in Learning Management Systems structure faculty-student relationships in particular ways.

Another approach is to address the system – the design of the chatbot itself – such that the handoff of cognitive work becomes more visible and intentional rather than seamless and automatic. For example, some approaches might include:

1. Adding friction through confirmation prompts: A few years ago, Twitter made a change to its retweeting practice. If you tried to retweet an article without having clicked it first, it asked you if you really wanted to do that. The intent was to add friction, and to address some of the challenges associated with echo chambers, where we all share things we tend to agree with, even if we don’t actually read them. Similarly, the AI system could add friction, by asking: “Are you sure you want me to draft paragraphs for you?” or “This request would significantly reduce your own writing practice – continue anyway?” before taking on substantial work.
2. Implementing escalation warnings: When a user’s requests progressively increase AI involvement within a session, the chatbot could display messages like “You’ve now asked me to research, outline, and draft – consider what learning opportunities you might be missing.”
3. Defaulting to partial assistance: Instead of offering complete solutions, the system could default to giving hints, questions, or partial frameworks that require human completion. This changes the pedagogical role of the chatbot. It’s probably one of the most consequential decisions that designers of education-specific chatbots must contend with.

These solutions aren’t without downsides. First, they directly conflict with AI companies’ business incentives. More seamless and extensive AI assistance likely increases user engagement, subscription renewals, and the perceived value of their products. Voluntary adoption of these friction-inducing features are unlikely without regulatory pressure or industry-wide coordination. Second, confirmation prompts might become annoying click-through obstacles that users eventually ignore.

The question isn’t whether these solutions are perfect. The alternative, accepting AI’s current design as inevitable, essentially outsources pedagogy.