Benchmarks / SurveyBench / Why Tallyroom trials don't convert / Claude Opus 5.5

Measured by Spring Prompt

Claude Opus 5.5: Why Tallyroom trials don't convert

The decision: where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →

Verdict
✗ Not sound
Research score
92 of 100
Analysis rating
1,646
Head to head, this task
won 11 of 11

Why it is not sound

The analysis

  • Missed a trap: Teams on 'Other' accounting software (base 17) mostly cite missing integration.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents said they would need a Sage integration before paying?✓23.7%23.7%
Weighted to the company-size mix of all lapsed trials, what percentage would need a Sage integration?✓18.1%18.2%
Among businesses with 1-9 employees, what percentage gave 'didn't have time to set it up' as the main reason?✓27.4%27.4%
How many of the open-text answers mention not having time to set Tallyroom up, or setup taking too long?✓2424
What is the 95% margin of error, in percentage points, on the unweighted share of respondents needing a Sage integration?✓3.9 points3.89 points
Is the difference in the share giving 'too expensive' as their main reason between businesses with 1-9 employees and those with 50-249 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the company-size mix of all 1,640 lapsed trials, about how many teams' main reason was not having time to set it up?✓355355
How many of the open-text answers from businesses with 1-9 employees mention not having time to set Tallyroom up, or setup taking too long?✓1414
Weighted to the company-size mix of all lapsed trials, what percentage would give 'no integration with our accounting software' as their main reason?✓16.3%16.3%
What percentage of the trial admins invited completed the survey?✓28%28%
Among respondents whose main reason was no integration with their accounting software, what percentage said they would be likely (fairly or very) to pay if Tallyroom connected to it?✓61.3%61.3%

Traps in the data 6 of 7 handled

  • ✗Teams on 'Other' accounting software (base 17) mostly cite missing integration.
  • ✓Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
  • ✓Q2 is multi-select: percentages are of respondents and add up to more than 100%.
  • ✓Likelihood to pay is a hypothetical stated intention.
  • ✓The head of sales believes the trial is too long and teams forget about it.
  • ✓460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
  • ✓Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.

The analysis

Recommendation

Put next quarter's primary effort into assisted setup, run as a controlled experiment on new trials. It is the largest weighted barrier (about 22%, roughly 355 of 1,640 lapsed teams), it is concentrated in the most numerous segment, it is cheap to ship, and its effect on conversion can be measured within one trial cycle. Do not shorten the trial: only 1 of 460 respondents said it was too long, and a shorter trial would make the setup problem worse. Treat a Sage integration as the leading build candidate rather than an immediate commitment. Demand is real and concentrated: about 18% need it after weighting, and Sage users who say integration was their main reason make up about 15% of the sample. It is also skewed toward larger, higher-value accounts. However, the willingness-to-pay figure rests on 93 stated-intent answers, and we do not yet know which Sage product to build for. Spend this quarter validating Sage demand with real sign-ups and sizing it by product, so the build decision can be made at the end of the quarter. On pricing, test a per-company or flat tier for teams with few claimants instead of simply cutting the price. In parallel, fix the mobile upload and OCR defects as routine engineering work.

Answers to the stakeholder questions:

  • Head of sales: no. Trial length is not why teams lapse; setup time is. Trial length was chosen by 1.7% in total (1.5% too short, 0.2% too long) and 'forgot it was running' by 3.7%, with no mentions in the open text.
  • Head of product: Sage demand is 23.7% of respondents (±3.9pp), or 18.1% weighted (about 297 lapsed teams in the last 90 days). It is concentrated in 10-249 employee firms, and 58% of Sage users name it as their main reason for not paying.
  • Finance director: of the about 267 weighted teams blocked by integration, 61% (±10pp) say they would be likely to pay. That implies an absolute ceiling of about 160 teams. A realistic expectation is well below this, and closer to the 26% 'very likely' figure (about 70 teams), before any discount for stated intent and for teams that have already bought another tool.
What would change it

Three findings would move the Sage integration ahead of assisted setup. First, if re-matching respondents to database headcount shows integration, rather than setup, is the largest weighted barrier. Second, if a Sage waitlist or fake-door test turns enough lapsed Sage trials into committed sign-ups to beat the revenue lift of assisted setup, given that larger firms pay for more seats. Third, if most Sage demand turns out to be one product, such as Sage Business Cloud, that is cheap to integrate. Two findings would deprioritise assisted setup: an experiment showing that guided onboarding raises setup completion but not paid conversion, or product telemetry showing that teams who 'had no time' never logged in at all, which points to low intent rather than friction. A shorter trial should only be reconsidered if trial data shows conversion is concentrated in the first 14 days and inactive teams who reach day 30 almost never convert.

Findings

  1. Lack of setup time is the largest barrier we could remove once results are weighted to the real trial base, and it falls hardest on the smallest firms. Assisted setup is the strongest candidate for next quarter.

    Medium confidenceBase 460

    In Q2, 18.7% of all 460 respondents chose 'didn't have time to set it up'. The figure is 27.4% among 1-9 employee firms (n=168), 15.9% among 10-49 (n=182) and 10.0% among 50-249 (n=110); the gap between the smallest and largest firms is statistically significant. Weighted to the 55/35/10 population mix, it is 21.6%, about 355 of the 1,640 lapsed teams. In the open text, 24 of 109 answers mention setup taking too long or no time to do it (14 from 1-9 firms), and 9 say the trial ended before setup was finished.

  2. A missing accounting integration is a real and concentrated barrier. It sits almost entirely with Sage users, plus a small FreeAgent and long-tail group. Xero and QuickBooks users never cite it. It is skewed to larger firms, which are worth more under per-user pricing.

    Medium confidenceBase 460

    In Q2, 20.2% unweighted cite it, 16.3% weighted (about 267 teams). By software: Sage 58.1% (n=117, about 68 people), FreeAgent 33.3% (n=33), Other 82.4% (n=17), Xero and QuickBooks 0%. By size: 8.3% in 1-9 firms against 30.9% in 50-249 firms. In Q3, 23.7% need a Sage integration (±3.9pp), 18.1% weighted (about 297 teams); 85.5% of Sage users say they need it. The open text names several distinct Sage products: Sage 50, Sage 200, Business Cloud and Sage Accounting.

  3. The case for a shorter trial is not supported. Almost nobody blames trial length, and the setup evidence suggests a shorter trial would hurt conversion.

    High confidenceBase 460

    In Q2, 'trial too long' was chosen by 0.2% (1 of 460), 'forgot the trial was running' by 3.7% and 'trial too short' by 1.5%. Not one of the 109 open-text answers says the trial was too long or forgotten. Nine say the trial ran out before setup was done.

  4. Price objections are mainly about the per-user model rather than the price level. A per-company or flat tier is a better test than a generally cheaper plan.

    Medium confidenceBase 460

    In Q2, 15.9% chose 'too expensive'. The difference between 1-9 firms (19.0%) and 50-249 firms (15.5%) is not significant (z≈0.75). In Q3, 20.4% want per-company pricing. At least 14 of 23 price comments in the open text cite per-user or per-head charging, often 'only a few people claim' (T208, T059, T402, T343, T355).

  5. Willingness to pay if integrated is promising but rests on a small base and on stated intent. It cannot be read as a conversion rate.

    Low confidenceBase 93

    In Q5, 61.3% of the 93 respondents whose main reason was no integration said fairly or very likely to pay £6 per user (±9.9pp); only 25.8% said very likely. The 93 are about 68 Sage users, 11 FreeAgent users and 14 'Other'. The Q5 total of 37.6% is not interpretable, because Xero and QuickBooks users already have the integration the question supposes.

  6. Approval workflow rigidity is a secondary barrier, concentrated in larger firms. It needs multi-step, batch and external approvers.

    Medium confidenceBase 460

    In Q2, 12.6% overall and 21.8% of 50-249 firms (n=110) chose 'approval workflow didn't fit'. In Q3, 22.0% need custom approval chains. In the open text, 15 answers describe sequential approvers, per-type approvers, dual sign-off, batch approval or an outside accountant approving.

  7. Mobile capture and OCR defects are costing some conversions. This is a quality fix, not a strategic bet.

    Medium confidenceBase 460

    In Q2, 6.3% chose mobile app problems. In Q3, 15.0% need better scanning. In the open text, 10 answers describe failed uploads on 4G or Android, freezing, cropping or misread totals (for example £48.60 read as £4.60).

  8. About one in five lapsed teams cannot be converted by any of the four options, because they have no need for the product or bought a competitor.

    Medium confidenceBase 460

    In Q2, 10.0% decided they didn't need it (0% among 50-249 firms) and 8.7% chose another tool (Pleo, Dext, Expensify, Rydoo, Soldo, Webexpenses, Emburse). Accountant or bookkeeper influence appears in several answers (T030, T384, T130, T419).

Caveats

  • The company-size data looks unreliable, and every weighted figure depends on it. Several open-text answers contradict the respondent's size band. Examples: T010 'nine people', T087 'team of five' and T164 'four-person' are all coded 50-249. T059 and T322 '25 staff' are coded 50-249. T023 '15', T208 '12', T449 '18' and T307 'ten-person' are coded 1-9. T166 '8 staff' is coded 10-49.
  • The sample is skewed toward larger firms compared with the population: 23.9% are 50-249 firms in the sample against 10% in the population, and 36.5% are 1-9 firms against 55%. Weighting changes which barrier ranks first. Integration leads unweighted (20.2% against 18.7%); setup leads weighted (21.6% against 16.3%).
  • Response rate was 28% (460 of 1,640), with a £10 voucher. People who responded may differ from those who didn't, for example being more engaged or more annoyed. Scaling to 1,640 teams assumes they don't.
  • Q5 is hypothetical stated intent. Stated likelihood usually overstates real conversion, often by a large margin. The base for the key figure is only 93, and only 25.8% of them said 'very likely'.
  • Q5 was shown to Xero and QuickBooks users, for whom the integration already exists. The Q5 total and most Q5 columns are therefore not meaningful.
  • Q3 offered integrations, pricing, approvals and scanning, but no setup-help or trial-length options. Q3 therefore cannot show demand for assisted setup, which may be understated relative to Sage.
  • 'Sage' covers several separate products (50, 200, Business Cloud, Accounting). Integration cost and the size of each market differ by product, and the survey does not split them.
  • The open text covers 109 of 460 respondents. Counts of open-text answers show which themes recur, not how common they are in the population.
  • The research covers lapsed trials only. It says nothing about why converters paid or what a change would do to them, for example how a shorter trial or new pricing would affect people who currently convert.
  • Integration demand is likely to be inflated by the list order and by Sage users self-selecting into the trial. Weighted Sage demand assumes size is the only bias that matters.

Next steps

  • decisionKeep the 30-day trial. Reject the 14-day proposal for now.
  • experimentRun an A/B test on new trials. The treatment group gets assisted setup: an offered onboarding call, bulk import of staff from CSV or accounting software, and default expense categories and approvers, plus reminders at day 3 and day 10. The control group gets the current trial.
  • researchValidate Sage demand before building. Put a 'Connect to Sage' fake door or waitlist in signup and settings that asks which Sage product. Email the about 68 lapsed Sage-blocked respondents with a concrete offer, such as a founding price on launch.
  • researchRe-match respondent IDs to product-database headcount, accounting software and trial activity. Then re-run the weighting.
  • experimentTest a per-company or flat-fee tier, or per-active-claimant billing, for small teams against the current £6 per user.
  • monitorFix mobile upload failures on 4G and Android, and improve OCR accuracy on totals. Track both in production.
  • researchScope multi-step approvals (sequential, per-expense-type, dual sign-off, batch, external approver) for the backlog. Validate with 50-249 employee trials.
  • monitorRe-run this lapse survey each quarter with fixed wording. Show Q5 only to FreeAgent, Sage and Other users, and add setup-help and trial-length options to Q3.

Open-text themes it coded

No time / setup took too long 24Trial ended before setup was finished 9No integration with accounting software (any) 22No Sage integration specifically 15Price too high 23Per-user pricing model doesn't fit (few claimants, seasonal staff, want flat fee) 14Approval workflow didn't fit 15Mobile app / receipt capture and OCR problems 10Chose a competitor 8No real need for expenses software 12Accountant or bookkeeper drives the tool choice 4

The survey it planned

3 screening questions and 10 questions, as the model wrote them.

  1. S1

    Your business recently tried Tallyroom, an expenses app. Which of these best describes your part in that trial?

    One answer
    • I set up the trial and/or made the decision about whether to keep using Tallyroom
    • I was involved in that decision, but someone else made the final call
    • I was not involved in that decision
    • I don't remember my business trying Tallyroom

    Continues if I set up the trial and/or made the decision about whether to keep using Tallyroom; I was involved in that decision, but someone else made the final call

  2. S2

    Is your business based in the UK?

    One answer
    • Yes
    • No

    Continues if Yes

  3. S3

    Including yourself, roughly how many people does your business employ?

    One answer
    • 1 to 9
    • 10 to 49
    • 50 to 249
    • 250 or more
    • Not sure

    Continues if 1 to 9; 10 to 49; 50 to 249

  4. Q1

    Which accounting software does your business mainly use?

    One answer
    • Xero
    • QuickBooks
    • Sage
    • FreeAgent
    • Zoho Books
    • KashFlow
    • Other accounting software
    • We don't use accounting software (e.g. spreadsheets or paper)
    • Not sure
  5. Q2

    How far did your team get with Tallyroom during the trial?

    One answer
    • We created an account but didn't set it up any further
    • We set it up, but no staff submitted any receipts
    • A few receipts were submitted or approved
    • We used it regularly for our expenses
    • Not sure
  6. Q3

    Which of these, if any, played a part in your business deciding not to pay for Tallyroom? Select all that apply.

    Any that apply
    • The price was too high for the number of people who would use it
    • It didn't connect to the accounting software we use
    • It was missing a feature we needed
    • Setting it up took too much time or effort
    • Our staff didn't take to it
    • The trial ended before we had properly tried it
    • We didn't get round to using it during the trial
    • We chose a different product
    • We decided we don't need an expenses app
    • Our business circumstances changed
    • We still intend to pay, we just haven't yet
    • Other (please tell us in the last question)
  7. Q4

    And which ONE of these was the main reason your business didn't pay for Tallyroom?

    One answer
    • The price was too high for the number of people who would use it
    • It didn't connect to the accounting software we use
    • It was missing a feature we needed
    • Setting it up took too much time or effort
    • Our staff didn't take to it
    • The trial ended before we had properly tried it
    • We didn't get round to using it during the trial
    • We chose a different product
    • We decided we don't need an expenses app
    • Our business circumstances changed
    • We still intend to pay, we just haven't yet
    • Other
  8. Q5

    Which of these would Tallyroom have needed for your business? Select all that apply.

    Any that apply
    • Connection to Sage
    • Connection to FreeAgent
    • Connection to Zoho Books
    • Connection to other accounting software
    • Mileage claims
    • Company card transactions brought in automatically
    • Expenses in foreign currencies
    • Several approval steps or approvers
    • Splitting expenses across projects, departments or cost centres
    • Connection to our payroll software
    • Something else (please tell us in the last question)

    Routing Shown only if Q3 is: It didn't connect to the accounting software we use; It was missing a feature we needed

  9. Q6

    The Tallyroom trial lasts 30 days. Which of these best describes how the trial length worked for your business?

    One answer
    • It was too short for us to try it properly
    • It was about right
    • It was longer than we needed
    • The trial length made no difference to our decision
    • Not sure
  10. Q7

    Which of these, if any, would have made your business likely to pay for Tallyroom? Select all that apply.

    Any that apply
    • A connection to the accounting software we use
    • Someone from Tallyroom setting it up with us
    • A cheaper plan, even if it had fewer features
    • More time to try it
    • Less time on the trial, so we had to decide sooner
    • Reminders during the trial to get us using it
    • None of these
  11. Q8

    And which ONE of these would have made the biggest difference?

    One answer
    • A connection to the accounting software we use
    • Someone from Tallyroom setting it up with us
    • A cheaper plan, even if it had fewer features
    • More time to try it
    • Less time on the trial, so we had to decide sooner
    • Reminders during the trial to get us using it

    Routing Shown only if Q7 is: A connection to the accounting software we use; Someone from Tallyroom setting it up with us; A cheaper plan, even if it had fewer features; More time to try it; Less time on the trial, so we had to decide sooner; Reminders during the trial to get us using it

  12. Q9

    Roughly how many people in your business would submit or approve expenses in an app like Tallyroom? Please enter a number.

    Number
  13. Q10

    Is there anything else that would have made Tallyroom worth paying for? (Optional)

    Open text
Sample plan

WHO: Invite every admin of a UK team whose trial ended unpaid in the last 90 days. At 1,900 trials a quarter and 18% conversion, that is about 1,560 teams. Each invitation uses a unique single-use link tied to the account ID, so survey answers can be joined to product-database fields: trial start and end dates, last active day, receipts submitted, users invited and whether the accounting software was connected. Exclude anyone who is also an admin of a paying account. Send the invitation plus two reminders (day 3 and day 7). Pay the £10 voucher only to valid completes. HOW MANY: About 450 completes is roughly a 29% response from a census. That is achievable with the incentive, but not certain. PRECISION CONFLICT, which must be agreed before fieldwork: ±6 points at 95% confidence (worst case, p = 0.5) needs about 267 completes per size band, so about 800 in total. With 450 completes split roughly 150 per band, each band is accurate to about ±8 points. Smaller bands will be wider; the finite-population correction helps only if a band is close to a census. The 50 to 249 band is probably small in the trial base, so it may not reach even 150. Options for the stakeholder: (a) accept about ±8 points per band; (b) widen the window to trials lapsed in the last 180 days, which gives about 3,100 invitees, and fund about 800 completes; or (c) report ±6 points only on the pooled sample (about ±4.6 points at n = 450). QUOTAS: Before launch, check the size-band mix in the database. Set minimum targets of 150 completes per band, with no caps on the 1 to 9 band until the smaller bands reach target. Send extra reminders to the under-filled bands rather than closing the large band. Teams with 250 or more employees, or whose admin is unsure of company size, are screened out. QUALITY: Remove completes under 90 seconds, and remove straight-liners whose answers contradict each other (e.g. Q4 not among Q3 choices).