Skip to content
AppDues
Log in
Contact salesSign up

When a paywall test lifts trial starts but refunds erase the gain

A paywall change can look like a win on trial starts and still lose money. The useful question is not whether more people begin a trial, but whether the people who begin one stay, convert, and keep the charge. This guide turns a subscription-app case study into a practical checklist for judging paywall experiments, running user research, and choosing a pricing model that fits your audience.

The paywall trap: trial starts are not the scoreboard

Two paywall designs tempt almost every subscription team. The first removes the close control entirely, so the only way forward is to begin a trial. The second keeps the close control on the main paywall but answers a dismissal with a discounted offer that has no exit of its own. Each design can push the trial-start number up, and each can seem reasonable because the user has already spent time in onboarding.

The bill arrives later, as refunds. In the BoldVoice case, refunds climbed by more than the extra trial starts were worth. Users who felt cornered into a trial disputed the charge as soon as it appeared, and they left resentful. The takeaway is blunt: a tactic that forces or tricks someone into a trial usually gets settled at the refund stage.

A sturdier scoreboard is net revenue after refunds, counted per exposed user, on a matured cohort. That means waiting long enough for trials to convert and for refunds to settle before calling a winner. The discipline is slow, and the hardest part is patience: sitting on exciting early data for a month or more.

A decision checklist for paywall experiments

Use this checklist before shipping a paywall change to everyone.

  1. Define the exposed population. Count only users who actually saw the change, not the whole install base.
  2. Record trial starts and conversion separately. A trial start is an intermediate event, not revenue.
  3. Add refunds to the outcome. Net revenue after refunds is the number that matters.
  4. Wait for a matured cohort. Give trials time to convert and refunds time to settle before deciding.
  5. Compare per exposed user. Aggregate revenue can hide a worse result for the group that saw the test.
  6. Ask whether the tactic forces or tricks. If a user feels trapped, expect a refund and a grudge.
  7. Stop optimizing every word and button. Improve the paywall for a while, then move on to the product.

A simple worked example from the captured evidence: suppose a hard paywall adds trial starts, but refunds rise by more than that gain. On a net-revenue-after-refunds basis, the change is a loss even though the trial-start chart looks better. The correct action is to reject the change, not to celebrate the top of the funnel.

User research as a calendar rule, not a department

BoldVoice treats user research as a recurring calendar commitment rather than a separate function. Automated invites go to a deliberate mix of users: long-term subscribers, recent trial converts, and people who did not convert. The invite asks for a 15-minute call and books whichever team members have space. Everyone on the team, including engineers, joins at least two interviews a week.

Two additional layers support that cadence. A super-user group meets roughly weekly, and engineers show unshipped features to heavy users so those users can shape the product before release. Users also visit the office so the team can watch what they tap and where they linger. Talking to users verbally and watching them use the product is an underrated skill.

There is a known bias: happy users are more likely to book calls. To correct for it, use a short three-question survey that can be answered by replying to an email, and a brief flow at cancellation that asks what went wrong. Without that correction, a team can end up making happy users happier and building a narrower, more technical product instead of a wider one.

Paid from day one vs freemium: match the model to the audience

BoldVoice launched from Y Combinator's summer 2021 batch and charged from the start: $10 a month with a one-week free trial and no experimentation. Charging was a way to identify the ideal customer profile. Willingness to pay correlates with whether the problem resonates, and it kept the team from chasing feedback from people who would never become customers. The energy went into onboarding and the core product instead of the paywall.

Pricing later matured. BoldVoice leads with an annual plan at around $150 a year, offers monthly as a fallback, and has a $200-a-year tier with extra features such as chat. The philosophy did not move toward freemium. The reason is audience fit: Duolingo's freemium base works because it teaches the basics of every language to a huge audience of hobby learners, and monetizing a fraction is enough. BoldVoice intentionally ignores that audience. Its users are intermediate-to-advanced professionals whose alternative is expensive one-on-one accent coaching. That is a different ICP, a different value proposition, and a different willingness to pay.

The same logic ruled out a college-campus launch, which BoldVoice tried. Students had less money, less urgency, and had not yet sat in the meeting where a colleague got listened to instead of them. The general lesson is to avoid combining playbooks from different apps into a mishmash strategy. A narrow free taste can still work. Closing the paywall unlocks the speech assessment, the first daily plan, and a coach video; continuing on day two starts a seven-day trial. Sometimes called a reverse trial, this arrangement preserves the onboarding paywall for users who are already prepared to pay.

A viral tool is not a strategy, but it can carry levers

BoldVoice's in-house ML team, working with Hollywood accent coaches who help label training data, built a tool that guesses where you are from after about ten seconds of speech. They shipped it as an ungated web app with no download and no trial, called the Accent Oracle, and mentioned it in a monthly investor update. An angel investor posted it on Threads, and the next day about 100,000 people in Korea tried it.

The lesson is not to try to engineer virality. It is that you can add levers to something that already has pull. A city-level guess re-spiked the Oracle, and showing users a preview of fuller results waiting in the app lifted the number who made the jump. A TikTok trend of users crying at low scores gave the brand another unpaid spike, with hundreds of millions of views for one creator by the founder's estimate. The caveat is that humor brings reach, not purchase intent.

Small team, high revenue: what actually compounds

BoldVoice's explanation for roughly $1M of ARR per employee is unglamorous. Hire founder-type doers who do not need direction. Hold one company-wide meeting on Mondays and leave the rest of the week for building and talking to users. Default users to the annual plan so paid acquisition pays back within the trial window and every channel's return is legible the same week.

Everyone ships. The founder uses Claude and ships pull requests despite not being technical. The designer takes Figma work through to prototypes that are about 80% of the way to production before engineers touch them. The durable advantage sits in the work that took longest to build: the team's bet is that non-native English pronunciation is a niche too narrow for general-purpose models to pursue, yet a market far too large to ignore.

Frequently asked questions

Why did the hard paywall test fail even though trial starts rose?

Refunds rose by more than the gain in trial starts. People who felt forced into a trial asked for their money back as soon as the charge hit, so the change lost money on a net-revenue-after-refunds basis.

What metric should replace trial starts in a paywall test?

Net revenue after refunds, per exposed user, on a matured cohort. Wait at least an extra week for trials to convert and about two more weeks for refunds to settle before declaring a winner.

How does BoldVoice run user interviews across a small team?

Automated invites go to a mix of long-term subscribers, recent converts, and non-converters. The invite books whichever team members have space, and every team member, including engineers, joins at least two 15-minute interviews a week. A super-user group meets roughly weekly, and users also visit the office for in-person observation.

Why does BoldVoice charge from day one instead of using freemium?

Charging helps identify the ideal customer profile because willingness to pay correlates with whether the problem resonates. BoldVoice serves intermediate-to-advanced professionals whose alternative is expensive one-on-one accent coaching, which is a different audience and value proposition from a mass-market language app.

What is a reverse trial in this context?

A reverse trial leads with a small free sample instead of a full free tier. At BoldVoice, closing the paywall opens the speech assessment, the first daily plan, and a coach video; to keep going on day two, the user starts a seven-day trial. Buyers who were ready to pay at onboarding still meet the paywall first, so the free sample does not undercut that moment.