This blog post discusses research that is currently in its preprint stage on SSRN. A preprint means the study has not yet undergone formal peer review, and its findings should be interpreted with caution until further validation.
Artificial intelligence (AI) models, particularly large language models (LLMs), are increasingly used in business, education, and policymaking. However, understanding how to interact with these models effectively remains a challenge. A recent preprint study by Lennart Meincke, Ethan Mollick, et al. from the Wharton School of Business on SSRN explores the nuances of AI prompting, shedding light on how different approaches can yield vastly different results.
Key Findings: Prompting Isn’t One-Size-Fits-All
The study tested AI performance using different prompting strategies, including polite prompts (e.g., “Please answer the following question”), commanding prompts (e.g., “I order you to answer”), and structured prompts. The results were surprising:
- Politeness Doesn’t Always Help: The study found that saying “please” did not consistently improve AI performance. In some cases, it even decreased accuracy.
- Benchmarking Standards Matter: AI performance varies based on how correctness is measured. The study used rigorous methods, testing each question 100 times to account for variability.
- Formatting is Crucial: Providing a structured prompt with clear instructions generally improved AI accuracy, reinforcing previous research that suggests formatting plays a vital role in model performance.
- Performance Variability: Even under controlled conditions, AI responses fluctuated significantly, emphasizing that AI reliability is still an evolving challenge.
Why This Matters
For professionals leveraging AI for decision-making, these findings highlight the need for careful prompt engineering. Unlike human communication, where politeness can foster cooperation, AI systems respond based on their training data and inherent biases. Therefore, rather than relying on politeness, users should prioritize clarity and structure in their prompts.
Takeaways for AI Users
- Use Explicit Formatting: Structure your prompts clearly to improve AI responses.
- Test for Consistency: If using AI for critical tasks, test prompts multiple times to gauge reliability.
- Be Skeptical of Universal Tricks: No single prompt strategy works for all cases; adapt based on context.
This study serves as a reminder that AI is a powerful but imperfect tool. Understanding its limitations and best practices for interaction will be key to leveraging its full potential.
Since this research is a preprint, further peer-reviewed studies will be necessary to confirm these findings.
This blog was human-authored with the assistance of OpenAI ChatGPT-4.5.