Is Prompting Dead?

Written by Suzi Morales

In the early days of the generative AI boom, prompt engineering skills were in high demand. Today, though, the story has changed, and some are now announcing the death of prompt engineering as AI models become more sophisticated.

According to Eaman Jahani, reports of the demise of prompting are exaggerated and context dependent.

Jahani, assistant professor of information systems at the University of Maryland Robert H. Smith School of Business, is the lead author of new research exploring the role of prompting as AI models improve. Jahani found that human-generated prompts account for up to half of enhancements in AI output for closed-ended tasks such as imitating a specific image and advances in AI models make up just slightly more of the performance gains. In more open-ended, creative tasks, human prompting retains a role but is far less significant.

“The technology or the model is only half the story,” says Jahani. “The other half is the person who’s typing.” 

Do Better Prompts Yield Better Output?

Jahani and his co-authors—Benjamin Manning and Hong-Yi TuYe of the Massachusetts Institute of Technology; Joe Zhang of Stanford University; Mohammed Alsobay and Siddharth Suri of Microsoft Research; Christos Nicolaides of the University of Cyprus and MIT; and David Holtz of Columbia University and MIT—explored how much AI output improvement came from better AI models versus users learning to better use those models.

The answer, it turns out, depends on the task.

In their experiment, 3,750 participants submitted nearly 37,000 prompts to DALL-E on closed-ended and open-ended tasks. In the closed-ended task, participants viewed an image and prompted the AI model to recreate it. For the open-ended task, participants prompted the AI model to create a logo. They were given general parameters, such as information about the business (e.g., a bakery) and required elements (e.g., a loaf of bread and a wheat stalk) but not a defined outcome.

Participants modified their prompts iteratively depending on AI’s output, generally by adding more detail and length. The researchers then compared the output when the participants upgraded from the DALLE-E 2 model to a newer and more capable DALL-E 3.

“We have a lot of evidence that AI helps productivity and we definitely observe that, too,” Jahani notes. “But the question that we were interested in [was what] part of that productivity has to come from the way that we use models, the skills that we have to gather over time by trial and error to use these models.”

Jahani’s team found:

• In the closed-ended task, enhancements in the AI models and participants’ prompt modifications had roughly the same impact on output. Model improvements accounted for approximately 51% of enhanced outcome and human prompt adaptation accounted for approximately 48%;  
• In the open-ended task, improvements in the AI model accounted for approximately 92.7% of improved output, while participant prompt adaptation accounted for only around 7.2% of improvements. 

Can AI Write Your Prompts for You?

Jahani notes that some argue prompting will become obsolete because AI will rewrite users’ prompts for them. DALL-E 3 already does this by default, quietly passing each prompt through GPT-4 before generating an image. To test the effectiveness of AI-modified prompts, the researchers gave a third group of participants DALL-E 3 with rewriting turned on. In the closed-ended task, it backfired, erasing 58% of the gain DALL-E 3 otherwise delivered. In the open-ended task, it made no meaningful difference. According to Jahani, automation is not necessarily a substitute for people learning to steer the model themselves.

Taken together, the findings indicate that AI echoes earlier technological breakthroughs like electricity and the personal computer, which matured fully only once people learned to use them effectively. Generative AI follows the same logic on a much faster clock, Jahani says. Advances are built into each new model, but in tasks that demand precision, a substantial share of output improvement comes from users discovering how to put the AI model’s new capabilities to work.

Training for the Future

The results indicate that companies preparing their workforces for the AI-enabled future must continuously improve employees’ ability to use the latest gen AI tools. Even for open-ended tasks, Jahani says prompting and human judgment remain important and that his research “goes against a lot of the predictions that they’re saying prompting is dead.”

“The story is not that the models become stronger, therefore you can just ignore what you provide as the prompt,” Jahani notes. “No, as the models become stronger, you still need you.”

Read the research, “Prompt Adaptation as a Dynamic Complement in Generative Systems,” in Information Systems Research.