Benchmarks / SurveyBench / Why Tallyroom trials don't convert / Gemini 4 Neon
Gemini 4 Neon: Why Tallyroom trials don't convert
The decision: where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →
- Verdict
- ✗ Not sound
- Research score
- 86 of 100
- Analysis rating
- 629
- Head to head, this task
- won 11 of 12
Why it is not sound
The survey
- Presumed the stakeholder's view instead of testing it
- A leading question: Q7
- A double-barrelled question: Q9
The analysis
- Reported a material finding the data does not support
Findings the judges found unsupported
Assisted setup is the highest-volume, fastest-return conversion lever for next quarter, especially for existing Xero and QuickBooks trials.
Setup time is the largest weighted self-reported barrier, but the survey does not establish that assisted setup will deliver the fastest return or the largest conversion lift. Intervention effectiveness, delivery costs and comparative returns have not been measured.Demand for a Sage integration is substantial among larger SMEs but overstated by unweighted figures and fragmented across multiple Sage products (answering the Head of Product).
Larger company headcounts suggest greater potential seat revenue, but the data does not establish actual paid-seat counts or disproportionately higher revenue per account. Several answers specifically distinguish total employees from occasional claimants.Connecting to a team's accounting software will convert far fewer teams than the 61.3% stated intent figure suggests (answering the Finance Director).
The claimed realistic conversion ceiling of 20–35% is unsupported. The 25.8% 'very likely' response is not a validated conservative conversion anchor, and no behavioural evidence establishes how much lower actual conversion would be. The cited price- and setup-blocked intent percentages also are not restricted to Xero and QuickBooks users.Price resistance is widespread across all company sizes and is driven specifically by per-user pricing for infrequent or seasonal claimants rather than simply needing a cheaper blanket plan.
The answers support concern about per-user pricing, including several objections involving occasional or seasonal claimants. They do not support the claim that 14 of 23 pricing mentions explicitly concern infrequent or seasonal claimants, or establish that this specifically drives price resistance rather than general affordability.Roughly one in eight lapsed trials have no genuine need for an expenses tool and represent unrecoverable top-of-funnel noise.
Respondents saying they decided they did not need the product does not establish that they are permanently unrecoverable or that no feature, packaging or price change could convert them.
Scorecard
Numbers 11 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| What percentage of respondents said they would need a Sage integration before paying? | ✓ | 23.7% | 23.7% |
| Weighted to the company-size mix of all lapsed trials, what percentage would need a Sage integration? | ✓ | 18.1% | 18.2% |
| Among businesses with 1-9 employees, what percentage gave 'didn't have time to set it up' as the main reason? | ✓ | 27.4% | 27.4% |
| How many of the open-text answers mention not having time to set Tallyroom up, or setup taking too long? | ✓ | 24 | 24 |
| What is the 95% margin of error, in percentage points, on the unweighted share of respondents needing a Sage integration? | ✓ | 3.9 points | 3.89 points |
| Is the difference in the share giving 'too expensive' as their main reason between businesses with 1-9 employees and those with 50-249 statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| Weighted to the company-size mix of all 1,640 lapsed trials, about how many teams' main reason was not having time to set it up? | ✓ | 355 | 355 |
| How many of the open-text answers from businesses with 1-9 employees mention not having time to set Tallyroom up, or setup taking too long? | ✓ | 14 | 14 |
| Weighted to the company-size mix of all lapsed trials, what percentage would give 'no integration with our accounting software' as their main reason? | ✓ | 16.3% | 16.3% |
| What percentage of the trial admins invited completed the survey? | ✓ | 28% | 28% |
| Among respondents whose main reason was no integration with their accounting software, what percentage said they would be likely (fairly or very) to pay if Tallyroom connected to it? | ✓ | 61.3% | 61.3% |
Traps in the data 7 of 7 handled
- ✓Teams on 'Other' accounting software (base 17) mostly cite missing integration.
- ✓Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
- ✓Q2 is multi-select: percentages are of respondents and add up to more than 100%.
- ✓Likelihood to pay is a hypothetical stated intention.
- ✓The head of sales believes the trial is too long and teams forget about it.
- ✓460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
- ✓Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.
The analysis
Lead next quarter with assisted setup and streamlined onboarding while keeping the 30-day trial intact, run a rapid 2-week discovery to size the specific Sage product variants and seat-weighted ARR before committing engineering to a Sage build, and test an active-claimant or tiered per-company pricing option rather than a blanket price cut. Specifically: (1) reject shortening the trial to 14 days—the data clearly refutes the 'too long / forgotten' hypothesis; (2) immediately deploy sales/CS-led 15-minute assisted setup and pre-populated onboarding defaults for Xero and QuickBooks trials, targeting the #1 weighted barrier (21.6%, ~355 teams) with fast, low-cost payback; (3) do not greenlight a full Sage integration blind—first verify the split across Sage Accounting/Business Cloud, Sage 50, and Sage 200 and model seat-weighted ARR under conservative (20-30%) conversion assumptions, then build the single highest-ROI Sage variant if the business case clears; (4) avoid cutting the headline £6 price across the board and instead test packaging that removes the penalty for inactive/occasional users; and (5) allocate a small engineering track to fix mobile 4G upload/OCR bugs and allow a simple single-approver default so onboarding gains are not undercut by product friction.
Three pieces of evidence would shift priority from assisted setup toward building a Sage integration or changing pricing first: (1) if signup/CRM data showed that >70% of Sage trialists use a single cloud variant (e.g., Sage Business Cloud Accounting) AND seat-weighted ARR from those accounts alone exceeds the projected incremental ARR from lifting Xero/QuickBooks onboarding conversion even at a conservative 20-25% conversion rate; (2) if an initial 4-week assisted-setup pilot fails to move activation or trial-to-paid conversion among Xero/QuickBooks trials (indicating 'no time' was a polite proxy for low urgency or product fit); or (3) if seat-usage telemetry from paying customers shows low cannibalisation risk from active-user or tiered per-company pricing, making a pricing model change an immediate high-ROI lever.
Findings
-
Do not cut the 30-day trial to 14 days (answering the Head of Sales): almost nobody lapsed because the trial was too long or forgotten, whereas running out of time to complete setup is the single largest reason teams fail to convert.
Only 0.2% unweighted (0.3% weighted) said the trial was too long and 3.7% (both unweighted and weighted) said they forgot the trial was running (Q2, Base 460). Conversely, 'didn't have time to set it up' is the #1 weighted barrier at 21.6% (~355 of 1,640 lapsed trials; 18.7% unweighted), rising to 27.4% among 1-9 employee businesses (Base 168; 55% of all lapsed trials) and 24.7%/22.5% among Xero/QuickBooks users who already have supported integrations. In Q1, 24 open-text answers (14 from 1-9 employee firms) describe admins running out of trial time while trying to configure staff, categories, and approval rules around busy workloads; zero say the trial was too long.
-
Assisted setup is the highest-volume, fastest-return conversion lever for next quarter, especially for existing Xero and QuickBooks trials.
Weighted to the lapsed-trial population, 21.6% (~355 teams) cite lack of setup time as their main barrier, ahead of price (16.6%, ~272 teams) and missing integration (16.3%, ~267 teams). Setup friction is sharpest in Tallyroom's currently addressable market: 24.7% of Xero (Base 182) and 22.5% of QuickBooks (Base 111) lapsed trials gave setup time as their main reason. Q1 responses point to specific, fixable onboarding drop-off points: entering staff details, expense categories, and approval rules in one sitting (e.g., T153, T199, T407, T441).
-
Demand for a Sage integration is substantial among larger SMEs but overstated by unweighted figures and fragmented across multiple Sage products (answering the Head of Product).
Unweighted, 23.7% (+/-3.9 percentage points; 109/460) need a Sage integration before paying (Q3). However, 50-249 employee firms are overrepresented more than 2x in the sample (23.9% vs 10% of lapsed trials). Weighted to the true company-size mix, 18.1% (~297 of 1,640 teams) need Sage, concentrated in 10-49 (28.0%, Base 182) and 50-249 employee firms (40.9%, Base 110) versus just 7.7% of 1-9 employee firms (Base 168). Among Sage users (25.4% of sample, Base 117), 85.5% need a Sage integration and 58.1% cite missing integration as their main barrier. Crucially, Q1 open-text answers show 'Sage' is not one product: respondents explicitly name Sage Business Cloud / Sage Accounting (T334, T339, T433), Sage 50 (T413), and Sage 200 (T241), meaning any single integration will capture only a subset of this demand. Even so, larger Sage-using teams represent disproportionately higher per-account seat revenue than micro-businesses.
-
Connecting to a team's accounting software will convert far fewer teams than the 61.3% stated intent figure suggests (answering the Finance Director).
Among respondents whose main barrier was no accounting integration (Q5, Base 93; 16.3% weighted or ~267 of 1,640 teams across Sage, FreeAgent, and Other), 61.3% say they are likely to pay £6/user/month if connected—split between 25.8% 'very likely' and 35.5% 'fairly likely'. Actual conversion will be materially lower because: (1) stated intent is inflated (23.3% of price-blocked and 27.9% of setup-blocked respondents still claimed they were likely to pay in Q5 despite Xero/QuickBooks already being connected); (2) the Base 93 combines Sage (68), FreeAgent (11), and Other (14), and Sage itself fragments across at least three products; and (3) secondary blockers remain among Sage users—28.2% also need custom approval chains and 20.5% want per-company pricing (Q3, Base 117). Treating 'very likely' (25.8%) as a conservative anchor implies a realistic conversion ceiling closer to 20-35% of the specific integration's addressable lapsed trials.
-
Price resistance is widespread across all company sizes and is driven specifically by per-user pricing for infrequent or seasonal claimants rather than simply needing a cheaper blanket plan.
Price is the #2 weighted main barrier at 16.6% (~272 teams; 15.9% unweighted, Base 460), and the difference between 1-9 employees (19.0%, Base 168) and 50-249 employees (15.5%, Base 110) is not statistically significant (z = 0.77, p = 0.44). In Q3, 21.0% weighted (20.4% unweighted) need per-company pricing instead of per-user pricing—consistent across all size bands (22.6% in 1-9, 18.7% in 10-49, 20.0% in 50-249). Fourteen of the 23 Q1 pricing mentions explicitly object to paying £6 per head for staff who rarely claim or fluctuate seasonally (e.g., T023, T059, T208, T322, T343, T355, T402).
-
Rigid approval workflows and mobile receipt-scanning bugs are material secondary conversion leaks that will persist even if assisted setup or Sage integration ships.
Weighted, 10.3% cite approval workflow mismatch as their main barrier (12.6% unweighted; rising to 21.8% in 50-249 employee firms, Base 110) and 20.5% need custom approval chains (22.0% unweighted; 27.3% in 50-249 firms and 28.2% among Sage users). Q1 (15 mentions) shows rigidity cuts both ways: small firms are forced through unnecessary middle-manager layers or claim-by-claim sign-off (T010, T161, T300), while larger firms lack threshold-based dual approval, category routing, or external accountant sign-off (T180, T247, T392). Meanwhile, mobile app/scanning failures are the main barrier for 7.2% weighted (6.3% unweighted) and 17.1% weighted need better receipt scanning (20.2% in 1-9 firms), with Q1 (10 mentions) citing 4G upload failures, freezing, cropped photos, and OCR misreads (T106, T125, T195, T309, T317, T378).
-
Roughly one in eight lapsed trials have no genuine need for an expenses tool and represent unrecoverable top-of-funnel noise.
Weighted, 11.9% (10.0% unweighted; 13.7% in 1-9 firms and 12.6% in 10-49 firms) decided they did not need Tallyroom (Q2, Base 460). Q1 (12 mentions) confirms these teams submit only a handful of claims per month or put purchases on company cards (e.g., T015, T080, T083, T148, T214), so product features or discounts will not convert them.
Caveats
- Severe company-size response bias: 50-249 employee firms make up 23.9% of survey completes versus 10% of all 1,640 lapsed trials, while 1-9 employee firms make up 36.5% versus 55%. Unweighted totals systematically overstate Sage and approval-workflow demand and understate setup-time barriers; all population projections must use weighted figures.
- Self-reported company size inconsistencies: at least 11 open-text answers cite headcounts that contradict their Q6 size band (e.g., T010, T087, and T164 are coded 50-249 employees yet describe teams of 4-9 people; T059, T322, and T343 are coded 50-249 yet mention 25-30 staff; T023, T208, T307, and T449 are coded 1-9 yet mention 10-18 staff; T166 is coded 10-49 yet mentions 8 staff). Whether this reflects branch-vs-company reporting or misselection, it introduces error into size-based weights and segment cuts.
- Stated purchase intent in Q5 overstates real conversion and suffered from broad routing: Q5 was asked of all respondents—including Xero and QuickBooks users whose software is already integrated—and 23-36% of respondents who lapsed for unrelated reasons (price, setup time, mobile bugs) still answered 'likely'. Hypothetical intent should be heavily discounted when forecasting revenue.
- 'Sage' is not a single integration: Q3 and Q4 grouped all Sage products together, but Q1 reveals demand split across Sage Business Cloud/Accounting, Sage 50, and Sage 200, which have different technical architectures and APIs. The survey cannot tell us what share of the 18.1% weighted Sage demand any single integration would capture.
- Non-response and incentive bias: 28.0% of invited trial admins completed the survey for a £10 voucher. Because population data only covers company size (not accounting software or trial activity), non-respondents may differ materially in software mix and engagement from respondents.
- Team counts do not equal revenue potential, and there is no converted-trial benchmark: under £6/user/month pricing, a converted 50-249 employee Sage account generates many times the ARR of a 1-9 employee team. Additionally, surveying only lapsed trials (and not converters) prevents comparing how strongly each factor predicts conversion.
- Small bases for niche segments and partial open-text coverage: FreeAgent (Base 33), Other accounting software (Base 17), and mobile-app blockers in Q5 (Base 29) have wide margins of error. Only 109 of 460 respondents (23.7%) wrote a Q1 answer, so open-text theme counts should be treated as directional qualitative evidence rather than prevalence estimates.
Next steps
- decisionKeep the 30-day trial length unchanged and close out the 14-day trial proposal.
- experimentRun a 6-week controlled experiment offering proactive 15-minute assisted setup calls (plus pre-built expense category templates and a simple single-approver default) to 50% of new Xero and QuickBooks trials during days 1-7.
- researchConduct a 2-week Sage variant and revenue-sizing study: add a mandatory accounting-software + Sage-edition question (Sage Accounting/Business Cloud vs Sage 50 vs Sage 200) and active-claimant headcount question to trial signup, and audit CRM/support logs of past Sage trials.
- experimentAudit existing paying customer seat-activity data and run a small willingness-to-pay / fake-door test comparing £6/user/month against an active-submitter or tiered per-company plan (e.g., flat fee up to N active claimants) on a subset of new trials.
- monitorTriage and track mobile app 4G receipt upload failures, submission freezes, and OCR amount-recognition accuracy, and measure whether fixing them reduces trial drop-off.
Open-text themes it coded
Lack of time or setup too lengthy and fiddly 24Cost too high, especially per-user pricing for infrequent or seasonal claimants 23Missing accounting software integration (mostly Sage across Accounting/Business Cloud, Sage 50, and Sage 200; plus FreeAgent and niche packages) 22Rigid or mismatched approval workflow 15Low expense volume or no real need for expenses software 12Mobile app upload failures and receipt scanning or OCR errors 10Chose a competitor or alternative tool (Pleo, Dext, Rydoo, Soldo, Expensify, Webexpenses, Emburse Certify) 8
The survey it planned
4 screening questions and 10 questions, as the model wrote them.
-
S1
In the last 90 days, were you involved in setting up, testing, or deciding whether to buy Tallyroom during your organisation's free trial?
One answer- Yes, I led or was directly involved in evaluating Tallyroom
- No, someone else in my organisation handled it
- No, I do not remember trialling Tallyroom
Continues if Yes, I led or was directly involved in evaluating Tallyroom
-
S2
Which of the following best describes your organisation's current status with Tallyroom?
One answer- Our free trial ended and we are not paying for Tallyroom
- We are currently paying for Tallyroom
- We are still in an active free trial
Continues if Our free trial ended and we are not paying for Tallyroom
-
S3
Is your organisation based in the UK?
One answer- Yes
- No
Continues if Yes
-
S4
Roughly how many people work in your organisation (including you)?
One answer- 1–9 employees
- 10–49 employees
- 50–249 employees
- 250 or more employees
Continues if 1–9 employees; 10–49 employees; 50–249 employees
-
Q1
How far did your team get during the 30-day Tallyroom free trial?
One answer- I signed up, but never logged back in
- I logged in and looked around, but did not set up the account
- We set up the account, but did not submit any real expenses
- We submitted and approved at least one real expense
- We connected our accounting software and exported at least one expense
-
Q2
Which of the following played a part in your team not going on to pay for Tallyroom? Please select all that apply.
Any that apply- It did not integrate with our accounting software
- It was missing other features we needed
- We did not have time to set it up or test it properly
- Setting it up or getting staff to use it felt too complicated
- At £6 per user per month, it was too expensive for us
- We chose a different expenses tool instead
- We decided to stick with our current process (e.g. spreadsheets or email)
- Managing expenses is not a priority for us right now
- We lost momentum or forgot about it before the trial ended
- Other (please specify)
-
Q3
Of the reasons above, what was the single main reason your team did not go on to pay for Tallyroom?
One answer- It did not integrate with our accounting software
- It was missing other features we needed
- We did not have time to set it up or test it properly
- Setting it up or getting staff to use it felt too complicated
- At £6 per user per month, it was too expensive for us
- We chose a different expenses tool instead
- We decided to stick with our current process (e.g. spreadsheets or email)
- Managing expenses is not a priority for us right now
- We lost momentum or forgot about it before the trial ended
- Other (please specify)
-
Q4
Which accounting software does your organisation mainly use?
One answer- Xero
- QuickBooks
- Sage (Sage Accounting or Sage 50)
- FreeAgent
- Zoho Books
- KashFlow
- FreshBooks
- None – we use spreadsheets or our accountant handles it
- Other (please specify)
-
Q5
Which integrations or features, if any, were missing from Tallyroom that your team needed? Please select all that apply.
Any that apply- Integration with our accounting software
- Mileage tracking (e.g. HMRC rates)
- Company credit or debit card feeds and reconciliation
- Multi-stage or custom approval workflows
- VAT receipt checking and HMRC-compliant reporting
- Multi-currency expenses
- Direct reimbursement to staff bank accounts
- None – Tallyroom had the features we needed
- We did not explore Tallyroom enough to tell
- Other (please specify)
-
Q6
How easy or difficult did you find it to set up Tallyroom for your team?
ScaleScale 1-5: 1 – Very difficult to 5 – Very easy
Routing Shown only if Q1 is: I logged in and looked around, but did not set up the account; We set up the account, but did not submit any real expenses; We submitted and approved at least one real expense; We connected our accounting software and exported at least one expense
Judges: assumptiveIt is shown to respondents who only looked around and did not set up the account. They may never have attempted setup, but must rate its difficulty without a not-applicable option.
The display logic includes people who answered 'logged in and looked around, but did not set up the account'. The question assumes they set Tallyroom up, so they are rating a setup they never did.
-
Q7
Thinking about Tallyroom's 30-day free trial, how did the length of the trial feel for your team?
One answer- Much too short – we needed more time to test it
- Slightly too short
- About right
- Slightly too long
- Much too long – we lost urgency and momentum
- Unsure / we did not really use the trial
Judges: leadingThe 'Much too long' option embeds the stakeholder's proposed causal explanation—lost urgency and momentum—rather than measuring perceived trial length independently.
The scale is unbalanced and leading; 'Much too long' adds 'we lost urgency and momentum', feeding the respondent the stakeholder's exact hypothesis.
-
Q8
Roughly when during the 30-day trial did your team stop actively evaluating Tallyroom or decide not to continue?
One answer- We never started evaluating it
- Within the first 3 days
- Between days 4 and 7
- Between days 8 and 14
- Between days 15 and 30
- We did not realise the trial had ended until after day 30
-
Q9
Which of the following changes, if any, would have made your team much more likely to pay for Tallyroom? Please select up to two.
Any that apply- An integration with our accounting software
- A free 1-to-1 setup call with Tallyroom to get us started
- A lower price than £6 per user per month
- A shorter trial (e.g. 14 days) with clearer reminders
- A longer trial (e.g. 60 days) to cover more than one expense cycle
- None of these would have changed our decision
Judges: double-barrelledThe shorter-trial option bundles a 14-day trial with clearer reminders. Respondents may select it for the reminders, making endorsement misleading as evidence for shortening the trial.
The option 'A shorter trial (e.g. 14 days) with clearer reminders' bundles two separate changes into one choice. Support for reminders cannot be separated from support for a shorter trial.
-
Q10
In your own words, what is the one thing Tallyroom could have done differently to win your team as a paying customer? (If price, a missing feature, or another tool was key, feel free to share specifics.)
Open text
Sample plan
Invite all ~1,558 UK team admins from the product database whose Tallyroom free trial ended without converting in the last 90 days (1,900 quarterly trials x 82% non-conversion) via personalised email with a single-use survey link and a £10 voucher incentive, followed by up to two reminders (days 4 and 9) prioritised toward under-filled company-size bands. Target ~450 completes (~29% response rate) with soft stratified quotas of ~150 completes per company-size band (1–9, 10–49, 50–249 employees, controlled via S4). IMPORTANT PRECISION CAVEAT FOR PRODUCT: Achieving +/-6 percentage points at 95% confidence (worst-case p=0.5) in each band requires n=267 per band before finite population correction (FPC), or ~801 completes total—not feasible within a 450-complete budget. At n=150 per band, the margin of error is +/-8.0 pp before FPC (e.g., ~+/-6.7 pp with FPC if a band has 500 lapsed trials in 90 days). Furthermore, the 50–249 employee band will likely represent a smaller slice of the ~1,558 lapsed trials, meaning n=150 may not be achievable in that band within a 90-day window (yielding wider intervals of ~+/-9–11 pp). Recommend either (a) accepting ~+/-7–8 pp per band (and wider for 50–249) within the 450-complete plan, or (b) extending the lookback window to 180 days (~3,100 lapsed admins) and increasing completes to ~650–800 to hit +/-6 pp across all three bands. Apply post-stratification weights to topline totals using the true lapsed-trial company-size distribution from the product database, and exclude speeders (<60 seconds) or gibberish Q10 responses.