ASO Patents

Document type	Academic paper, not a patent
Title	Large Language Models Can Accurately Predict Searcher Preferences
Authors	Paul Thomas, Seth Spielman, Nick Craswell, Bhaskar Mitra
Affiliation	Microsoft, with deployment at Bing
Venue	ACM SIGIR 2024, pages 1930 to 1940. Preprint arXiv:2309.10621, September 2023
Platform	Web search, not an app store
Source	arxiv.org/abs/2309.10621

What This Paper Covers and Why It Matters for ASO

Scope note. This is a Microsoft paper about web search. It says nothing about apps. It is on this site for one reason: Apple’s App Store ranking paper cites it, and the method Apple applied to App Store relevance labels is the method this paper established.

To understand why App Store search changed in 2026, you have to understand what changed in information retrieval in 2023.

Commercial search engines have always needed relevance labels: a human looks at a query and a result and grades how well they match. Those labels train and evaluate the ranker. The problem is that they are slow and expensive, so every search company has far fewer of them than it wants.

The authors point out a second problem that is easy to miss. The humans doing the labeling are third party annotators, not the people who ran the query. They are guessing at someone else’s intent, and they often guess wrong.

The finding

Using large language models with prompts calibrated against real user feedback, the authors report that:

• LLM generated labels were as accurate as human labelers.
• Measured against gold labels taken from actual searchers, LLM labels were better than third party crowd workers, at a fraction of the cost.
• The models identified difficult queries in much the same way humans did.
• Prompt design mattered. Both systematic changes and simple paraphrases moved accuracy.

The work was deployed at Bing and validated against TREC data, so this is a production result rather than a benchmark exercise.

Why this should change how you think about metadata

ASO has always been written as though search were string matching with weights. This paper, and Apple’s application of it, describes something different.

Relevance is a judgment. A reader looks at the query and at what your app says about itself and forms an opinion on a graded scale. For years that reader was a contractor. Now it is a model trained on those contractors. Either way, your metadata is being read and assessed, not scanned for tokens.

Three practical consequences follow:

• Metadata that reads as incoherent to a person reads as incoherent to the judge. Keyword strings assembled for an algorithm are being evaluated by something that understands sentences.
• Because labels can now be generated at scale, the textual relevance signal gets stronger rather than weaker over time. The historical advantage of behavioral signals was partly that there were simply more of them.
• Prompt sensitivity means the judgment criteria can shift without any announcement. A rewording of an internal rubric can move rankings, and nothing external would show why.

Paper Summary

Search systems rely on relevance labels, traditionally produced by third party annotators who may misunderstand what the original searcher actually wanted. The authors propose generating those labels with large language models, using prompts calibrated against real user feedback rather than against annotator guidelines alone.

They deployed the approach at Bing and validated it using TREC data. The models matched human labeler accuracy, flagged difficult queries in similar ways, and when compared against high quality gold labels derived from actual users, produced better labels than third party workers at substantially lower cost. Prompt design, including minor paraphrasing, measurably affected accuracy.

The conclusion the field drew, and that Apple later acted on for the App Store, is that LLM relevance judgments are a viable substitute for human assessors when calibrated against verified user preferences.

A detailed breakdown of the prompt design and the agreement analysis will follow in a later update to this article.