ASO Patents

What This Paper Covers and Why It Matters for ASO

Every patent on this site is an inference. Apple designed something, claimed it, and we read the claim and reason about whether it runs.

This is not that. This is Apple’s own engineers describing the production App Store ranker, in a peer reviewed venue, with measured numbers from a worldwide A/B test. It is the most direct public evidence about App Store ranking that exists.

The ranker is a multi-objective learning to rank model trained on two separate label sources. Not a weighted field formula.

• Behavioral relevance: the likelihood a user clicks or downloads a result. Labels come from downloads and clicks in aggregated App Store search logs.
• Textual relevance: a result’s semantic fit to the query, as judged by human experts on a five point scale.

The two are combined by scalarization, meaning a weighted mix of the training data rather than a blended score. The same query and app pair, with an identical feature vector, appears in the training data more than once, once carrying a behavioral label and once carrying a textual label, with strict separation so each objective’s gradients are computed independently. The mixing ratio is a tunable hyperparameter, and the paper names 90-10, 70-30 and 50-50 as example mixes.

Sit with that. The weight of textual relevance in App Store ranking is a dial Apple can turn, and turning it would change what every ASO tool in the market measures, silently.

The numbers

NDCG is graded relevance discounted by position, where 1.0 is a perfect ordering. The figure to remember is behavioral NDCG@3 of 0.479. On its own behavioral measure, App Store search puts the app users actually want into the top three about half as well as a perfect ranking would. That gap is the space the entire ASO industry works inside, and this is the first time anyone has published its size.

The tail finding

The worldwide A/B test produced a statistically significant +0.24% increase in conversion rate, defined as the proportion of search sessions with at least one app download, and the augmented model beat production in 89% of storefronts.

The average hides the real result. Broken out by query frequency, the conversion lift is roughly 3.7% in the rarest query bucket, about 1.3% in the next, and falls to approximately zero for head queries. The paper’s explanation is that tail queries lack the traffic to produce reliable behavioral signals, so textual relevance carries the ranking there.

This is the 2012 navigational versus functional split from US9405832B2, confirmed with production data fourteen years later. On high volume branded queries your metadata is close to irrelevant. On long tail descriptive queries it is most of what you have.

The judgment layer nobody in ASO talks about

Human judges score query and app pairs on a five point scale using the app’s metadata. Those labels were scarce and expensive, leaving the textual objective, in the paper’s words, under-powered. So Apple fine tuned an in-house LLM on the existing human judgments and generated millions of new labels across multiple storefronts and languages.

A fine tuned 3 billion parameter model more than doubled the F1 of a pretrained model ten times its size. The paper also notes that few shot prompts using string labels outperformed numeric ones.

Two clarifications that matter for how this gets quoted. The LLM is an offline label generator, not a live re-ranker, and the ranker’s internal architecture is not modified. And the labels are pointwise, meaning each query and app pair is scored alone rather than compared against another.

The practical consequence is the part to internalize: your metadata is read as a document and graded, by a person or by a model trained on people. It is not matched as a string.

What the paper does not say

• The ranker’s architecture. Only “production ranker” and “feature vector” appear. No model family, no layers.
• Any feature names.
• Which metadata fields the judges see. The prompt template shows three placeholders, app metadata 1, 2 and 3, and never names them. So there is still no published basis for any claim about title versus subtitle versus keyword weighting.
• The definitions of the five relevance grades, which are redacted in the prompt figure.
• The candidate retrieval stage. The paper covers ranking only.
• Personalization, position bias, and how localization is handled beyond the labels being multilingual.

Paper Summary

Large scale commercial search systems optimize relevance along two complementary objectives: behavioral relevance, meaning results users tend to click or download, and textual relevance, meaning a result’s semantic fit to the query as judged by experts. Behavioral labels are abundant. Expert textual labels are scarce and expensive, which leaves the textual objective weak in multi-objective training.

The authors evaluate several LLM configurations for producing textual relevance labels and find that a specialized fine tuned model significantly outperforms a much larger pretrained one. They use it to generate millions of labels, then augment the production ranker’s training data with them.

The result is an outward shift of the Pareto frontier: offline NDCG improves for behavioral relevance while simultaneously improving for textual relevance, rather than trading one against the other. A worldwide A/B test on the App Store ranker validated the offline gains, with the largest improvements on tail queries where behavioral signals are sparse.

Three of the six authors connect directly to the patent record on this site. Sean Suchter is a named inventor on Apple’s two pending search applications, and Evangelia Christakopoulou holds a granted Apple recommendation patent. The acknowledgments thank Don Dini, who is a named inventor on one of those same applications. The papers and the patents are the same team.

A detailed breakdown of the method, the training pipeline and the evaluation design will follow in a later update to this article.

Related reading: US9405832B2 on query classification.