Summary
App Store product page testing is not a comparison of two counts. It is an experimentation platform that assigns each user to one version of your page, remembers that assignment, records a single outcome per person, aggregates daily, and reports a conversion rate that is an estimate rather than a tally.
Apple documents part of this in App Store Connect help. Two patent filings describe the rest, including several things that appear nowhere in the documentation or in any industry guide we could find.
Methods used
| Stage | Method | Where it is stated |
|---|---|---|
| Assignment | Randomized selection or a predefined rule set, tied to a user or device identifier and stored so repeat visits stay consistent | Patent |
| Traffic allocation | Set by the developer, across up to three treatments plus the original | Documented |
| Outcome measurement | Binary download outcome recorded on the user’s first visit, stored as timestamped records in a facts database partitioned by date | Patent |
| Aggregation | Single pass daily pipeline across all running experiments, using per user statistics tables | Patent |
| Statistical approach | Bayesian methods incorporating prior information about your page’s performance, reported with a 90 percent credible interval | Documented |
| Statistical detail | Bayesian sequential testing, alongside chi squared tests, applied to daily aggregated conversion rates | Patent |
| Reporting | Unique impressions, estimated conversion rate, estimated relative lift, confidence level | Documented |
| Additional collection | Share rate, referral traffic from shared links, time on the information page, installation data, per treatment | Patent only |
| After install | The assigned version carries through, so the app is presented according to the same treatment on launch | Patent only |
What was not previously known
Before listing these, it is worth being precise about what Apple already publishes, because several claims circulating in ASO writing treat documented facts as discoveries.
Already documented by Apple: that the statistics are Bayesian, that results carry a 90 percent credible interval, that a variant needs at least 90 percent confidence to be marked as performing better or worse, that tests can run up to 90 days, that results update daily, that impressions are counted per unique device, and that the reported conversion rate incorporates existing data as well as test data.
Not documented anywhere, and found only in the patents:
- The testing is sequential, and chi squared tests are involved. Apple’s help says Bayesian. The patent says Bayesian sequential testing and names chi squared tests alongside it. The sequential part is what makes continuous monitoring legitimate rather than dangerous, and it has never been stated publicly.
- Apple collects engagement metrics per treatment that you are never shown. Share rate, referral traffic from shared links and time spent on the information page are all named in the patent as things the system collects for each version. None of them appear in the results table.
- The treatment continues into the app after install. This sits in the independent claim, not in a footnote: on launching the downloaded app, the app is presented in accordance with the version the user saw in the store.
- The outcome is one binary event per person, on their first visit. Unique impressions are documented, but the patent is specific that what gets recorded is whether a download followed that first visit.
- Assignment can follow a predefined rule set, not only randomization. The patent allows both.
- The infrastructure runs millions of concurrent experiments through a single daily pass over instrumentation data, with per user statistics tables and a facts database partitioned by date. This is why a number exists the day after launch.
How results are formed
The four numbers you are shown
| Metric | What it is |
|---|---|
| Unique impressions | The number of unique devices that viewed each variant |
| Estimated conversion rate | “The estimated percentage of people that downloaded or pre-ordered your app from a certain product page variant. This estimate incorporates the data observed during the test, as well as existing data.” |
| Estimated relative lift | “The estimated relative increase in conversion rate for a variant as compared to the selected baseline.” |
| Confidence level | “The probability that the data gathered in a test suggests that two variants are performing differently.” |
Your conversion rate is an estimate, not a count
This is the most misread part of the whole system, and Apple states it plainly: the estimate incorporates existing data as well as the data observed during the test.
That is a Bayesian prior. In plain terms, the system starts out believing your variant will perform roughly like your page already performs, and it only moves away from that belief as evidence accumulates.
Two consequences follow. Early in a test your variants will look suspiciously similar to your baseline, because the prior is doing most of the work. And the number you see is not the number you would get by dividing downloads by impressions yourself. If you reconcile the dashboard against your own arithmetic and they disagree, this is why.
The 90 percent bar and the 90 day ceiling
A variant is only marked as performing better or worse once it reaches at least 90 percent confidence. The credible interval shown is also a 90 percent interval, meaning there is a 90 percent probability that the true conversion rate or lift falls inside the displayed range.
Tests run for up to 90 days, and Apple will flag a test as likely to be inconclusive when there probably will not be enough data even after that.
Apple also makes the point that this site’s traffic table exists to illustrate, in its own words: it takes fewer weeks of data to determine significance on higher improvement values, such as 30 percent, than on lower ones, such as 5 percent. Small effects are expensive to prove. That is not a quirk of your app.
The engagement metrics you are not shown
Here is where the patent goes well past the documentation. Alongside download rate and installation data, the system is described as collecting, per treatment:
- Share rate: what proportion of page views produced a share action
- Referral traffic: the traffic arriving from links that users shared
- Time spent on the information page
None of these appear in the results table. Your test reports four numbers. The system is described as recording at least seven.
Be careful about what that does and does not prove. The patent says these signals are collected per treatment. It does not say a winner is chosen on them. Anyone claiming Apple ranks your variants on dwell time is going beyond the document.
What it does establish is that Apple measures your page on dimensions you cannot see, and companies do not usually enumerate signals in a patent that they have no intention of using. Three things follow that are worth thinking about:
- Dwell time cuts both ways. A variant that wins on downloads while losing badly on time spent is persuading faster, which is good, or persuading more superficially, which may reappear later as weaker retention. Your test will not tell you which, and Apple’s system has the data to know.
- Your product page has a distribution function, not just a conversion function. Share rate and referral traffic are being measured per version. A variant that people forward is doing something a variant that merely converts does not.
- Two variants can tie on conversion and differ on everything else. When a test comes back inconclusive, that is not the same as the two pages being equivalent. It means they were equivalent on the one dimension you were shown.
What actually decides the winner
Conversion rate, estimated with priors, compared against the baseline you selected, with a confidence level derived from Bayesian methods. The patent adds that the testing is sequential and that chi squared tests are part of the apparatus.
Because lift is defined relative to the baseline, the baseline’s own drift during the test is built into every number you read. If your default page’s performance moves for an unrelated reason, every variant’s reported lift moves with it.
How it works
Step 1: your app gets several versions
An app is associated with multiple presentation versions of its information page. The elements that can vary are the icon, the screenshots, other media items and the description text. You can run up to three treatments against the original.
The same machinery powers two different features. Product page optimization compares versions to find a winner. Custom product pages serve a deliberately different version to a known audience with no winner intended. Underneath, both are one app with several versions and a rule deciding who sees which.
Step 2: each user is assigned, and the assignment sticks
When someone accesses the store, the system detects that access along with a user identifier and selects one version for them, either at random or by a predefined rule set. The identifier and the chosen version are written into a data structure, so the same person sees the same version on later visits.
This means the unit of measurement is the person, not the visit. It rules out the most common source of noise in store experiments, where someone sees version A on Monday, version B on Thursday, installs, and the install lands on whichever page they saw last.
Step 3: the traffic arithmetic
You choose what share of visitors enters the test and how it splits across treatments. That freedom is the trap, because the split you pick decides how long you wait.
If your product page gets 3,000 views a day and you run three treatments across the whole audience, each version is seeing roughly 750 views a day. Put half the audience in the test instead and each version sees about 375. Here is what those numbers buy you, starting from a 25 percent conversion rate.
| If the variant is really better by | New conversion rate | People needed per variant | Days at 200 per day | Days at 750 per day |
|---|---|---|---|---|
| 5 percent relative | 26.3% | about 19,000 | about 96 | about 26 |
| 10 percent relative | 27.5% | about 4,900 | about 24 | about 7 |
| 20 percent relative | 30.0% | about 1,300 | about 6 | about 2 |
| 40 percent relative | 35.0% | about 330 | about 2 | under 1 |
The 90 day ceiling is doing real work here. A five percent improvement at 200 views per variant per day does not finish inside the window. Apple’s own guidance agrees with the direction: higher improvement values need fewer weeks.
Step 4: what gets counted
For each user’s first visit, the system records the visit date, the user identifier, and whether a download followed. One person, one yes or no. Those records are stored as timestamped conversion records in a facts database partitioned by date, and conversion rates are computed per version over the collection period.
Step 5: how a winner is decided
Statistical testing is applied to the daily aggregated conversion metrics. Apple documents the approach as Bayesian. The patent adds that it is sequential, and that chi squared tests are part of it.
The sequential part matters. Ordinary A/B testing advice says never look early, because every extra look is another chance to see a fake winner. That advice belongs to a fixed horizon test, where you set the sample size in advance and read the result once. A sequential method assumes continuous monitoring and accounts for it.
So looking daily is not the error the standard advice describes. Acting on day three still is, because no statistical method extracts certainty from a few hundred observations, and the 90 percent marking exists precisely to stop you.
Step 6: the treatment follows the user into the app
The least discussed part of the system sits in the independent claim. It covers detecting the download, then causing the app to be downloaded such that on launch, the app is presented in accordance with the same presentation version the user saw in the store.
Mechanically that requires the assignment to travel with the install. Strategically it reframes what a test is. A treatment is not a set of screenshots. It is a promise made in the store and kept in the app, and the pairing is what gets measured.
What this means for how you test
- Test big changes. The 90 day ceiling makes small differences unprovable for most apps. If you cannot describe the difference between your variants in one sentence, it is too small.
- Fewer treatments, more traffic each. Three treatments split your test audience three ways and can push a winnable test past the window.
- Check daily, decide slowly. The method tolerates watching. It does not tolerate concluding early.
- Inconclusive is not the same as equivalent. It means the two pages matched on the one metric you were shown.
- Do not change the listing mid test. Lift is measured against the baseline, so moving the baseline moves every number.
- Match the onboarding to the winning page. The system already assumes that continuity exists.
- Remember the icon is everywhere. An icon variant changes what people see in search results and on the home screen, not only on your page.
Sources and caveats
- Apple, Product Page Optimization, App Store Connect Analytics help. The documented metric definitions, the Bayesian approach, credible intervals, the 90 percent confidence marking and the 90 day window.
- US20230176843A1, App Store Information Page Customization. Assignment, the collected engagement metrics, and the post install continuity claim. Pending application.
- US12332765B2, Systems and methods for variant testing at scale. The first visit binary outcome, the daily single pass pipeline, sequential testing and chi squared tests. Granted June 2025, active.
- US12430660B2 and US12561221B2, for how results are later presented against comparable apps.
Caveats worth keeping. A patent describes what a company designed and claimed, which is strong evidence and not documentation, and one of these two is a pending application whose claims may still narrow. Where the documentation and a patent disagree, the documentation describes the shipped product and the patent describes the design. And the sample size table is standard statistics rather than Apple’s own method, included to show the scale of the constraint rather than to predict when your specific test will resolve.
Bir yanıt yazın