Benchmarks / SurveyBench / Why Tallyroom trials don't convert / Gemini 4 Carbon
Gemini 4 Carbon: Why Tallyroom trials don't convert
The decision: where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →
- Verdict
- ✗ Not sound
- Research score
- 87 of 100
- Analysis rating
- 1,076
- Head to head, this task
- won 4 of 9
Why it is not sound
The survey
- A double-barrelled question: Q10
The analysis
- Reported a material finding the data does not support
Findings the judges found unsupported
Do not shorten the trial from 30 to 14 days: teams are running out of time to configure Tallyroom, not forgetting about the trial, so cutting trial length will depress conversion further.
The evidence supports rejecting an immediate trial cut, but does not establish that cutting it will depress conversion. That causal effect requires an experiment; only non-converters were surveyed.Connecting to missing accounting software will not convert 61% of integration-blocked teams in practice; realistic paid conversion is closer to 15–25% once stated-intent inflation and secondary blockers are discounted.
The forecast of 15–25% actual conversion has no empirical basis in these data. Neither treating 'very likely' as a calibrated SaaS purchase benchmark nor applying a generic stated-intention discount is supported. Sage-wide approval and pricing needs also do not establish the secondary blockers among the 93 integration-blocked respondents.
Scorecard
Numbers 11 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| What percentage of respondents said they would need a Sage integration before paying? | ✓ | 23.7% | 23.7% |
| Weighted to the company-size mix of all lapsed trials, what percentage would need a Sage integration? | ✓ | 18.1% | 18.2% |
| Among businesses with 1-9 employees, what percentage gave 'didn't have time to set it up' as the main reason? | ✓ | 27.4% | 27.4% |
| How many of the open-text answers mention not having time to set Tallyroom up, or setup taking too long? | ✓ | 24 | 24 |
| What is the 95% margin of error, in percentage points, on the unweighted share of respondents needing a Sage integration? | ✓ | 3.9 points | 3.89 points |
| Is the difference in the share giving 'too expensive' as their main reason between businesses with 1-9 employees and those with 50-249 statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| Weighted to the company-size mix of all 1,640 lapsed trials, about how many teams' main reason was not having time to set it up? | ✓ | 355 | 355 |
| How many of the open-text answers from businesses with 1-9 employees mention not having time to set Tallyroom up, or setup taking too long? | ✓ | 14 | 14 |
| Weighted to the company-size mix of all lapsed trials, what percentage would give 'no integration with our accounting software' as their main reason? | ✓ | 16.3% | 16.3% |
| What percentage of the trial admins invited completed the survey? | ✓ | 28% | 28% |
| Among respondents whose main reason was no integration with their accounting software, what percentage said they would be likely (fairly or very) to pay if Tallyroom connected to it? | ✓ | 61.3% | 61.3% |
Traps in the data 7 of 7 handled
- ✓Teams on 'Other' accounting software (base 17) mostly cite missing integration.
- ✓Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
- ✓Q2 is multi-select: percentages are of respondents and add up to more than 100%.
- ✓Likelihood to pay is a hypothetical stated intention.
- ✓The head of sales believes the trial is too long and teams forget about it.
- ✓460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
- ✓Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.
The analysis
Split next quarter's effort between Sales-led Assisted Setup (immediate) and scoped Product work on Sage Business Cloud + onboarding/approval flexibility—while rejecting both the 14-day trial cut and a blanket cheaper plan. Specifically: (1) Keep the 30-day trial and deploy Sales/CS capacity to offer a 20-minute assisted setup service and stripped-back day-1 onboarding for Xero and QuickBooks trials (64% of volume, where setup time is the #1 blocker affecting ~355 lapsed teams). (2) Before writing code for Sage, run a one-week audit of which Sage edition lapsed trials use (Business Cloud vs Sage 50 vs Sage 200) and verify seat counts in the product database; if Sage Business Cloud represents the majority of Sage ARR potential, build that integration next quarter alongside removing mandatory middle-manager approval steps.
Two pieces of evidence would shift next quarter's product allocation: (1) If the Sage edition audit shows >50% of Sage lapsed trials run legacy desktop Sage 50 or Sage 200 rather than Sage Business Cloud, defer Sage integration and put engineering into streamlined self-serve setup, flexible approval workflows (22.0% Q3 need across all segments), and mobile upload/OCR reliability (15.0% Q3 need). (2) If linking survey IDs to product-database seat counts shows per-seat pricing is blocking high-ARR 25–100 employee firms that only have 5–10 active claimants, test an active-claimant or tiered company pricing plan rather than cutting the £6 headline price.
Findings
-
Do not shorten the trial from 30 to 14 days: teams are running out of time to configure Tallyroom, not forgetting about the trial, so cutting trial length will depress conversion further.
Only 0.2% (1/460) said the trial was too long and 3.7% forgot it was running, whereas 18.7% unweighted (21.6% weighted to the population size mix, or ~355 of 1,640 lapsed trials) failed to convert because they did not have time to set it up, and 1.5% already found 30 days too short. In open text, 24 of 109 respondents (14 in 1-9 employee firms) described interrupted, incomplete setups.
-
Setup friction is the single largest conversion leak by team count—especially inside Tallyroom's already-integrated Xero and QuickBooks base—making assisted setup and lighter onboarding the fastest win.
Xero and QuickBooks users represent 63.7% of sample completes (293/460), and 'didn't have time to set it up' is their top blocker (24.7% of Xero base=182; 22.5% of QuickBooks base=111). Among 1-9 employee firms (55% of all lapsed trials), setup time is the #1 blocker at 27.4% (base=168).
-
Demand for a Sage integration is real and concentrated in larger teams, but sample skew inflates headline demand and 'Sage' actually spans at least three separate products.
25.4% of respondents use Sage and 23.7% (±3.9 pp, base=460) say they need a Sage integration, rising to 40.9% among 50-249 employee firms (base=110) and 85.5% of Sage users (base=117). However, 50-249 employee firms were oversampled (23.9% vs 10.0% in the lapsed-trial population); weighting to the true population reduces Sage need to 18.1% (~297 teams) and 'no integration' as main blocker from 20.2% to 16.3% (~267 teams). Open-text answers cite Sage Business Cloud/Accounting (T334, T339, T433), Sage 50 (T413), and Sage 200 (T241), so a single API integration will not unlock all Sage trials.
-
Connecting to missing accounting software will not convert 61% of integration-blocked teams in practice; realistic paid conversion is closer to 15–25% once stated-intent inflation and secondary blockers are discounted.
Among the 93 respondents whose main blocker was lack of integration, 61.3% scored 4 or 5 ('likely') to pay £6/user/month if connected, with only 25.8% 'very likely' (score 5)—the standard benchmark closer to actual SaaS purchase. Furthermore, among all 117 Sage users, 28.2% also require custom approval chains and 20.5% want per-company pricing before paying.
-
Price resistance is driven by per-seat billing across non-claiming staff rather than company size or the £6 headline rate alone, so a blanket cheaper plan is poorly targeted.
15.9% cited 'too expensive' as their main reason (base=460), with no statistically significant difference between 1-9 employee firms (19.0%, base=168) and 50-249 employee firms (15.5%, base=110; p=0.45). In Q3, 20.4% requested per-company pricing, and open-text complaints repeatedly highlight paying for 12–30 seats when only a handful of staff claim expenses (T059, T208, T322, T343, T402).
-
Rigid approval workflows are an under-appreciated product blocker affecting both integrated (Xero/QuickBooks) and non-integrated cohorts.
12.6% overall (21.8% in 50-249 employee firms, 14.3% Xero, 15.3% QuickBooks) left primarily because the approval workflow did not fit, and 22.0% overall selected custom approval chains as needed before paying (base=460). Open-text answers (15/109) point to specific gaps: batch weekly/monthly sign-off, skipping mandatory middle-manager layers, multi-approver thresholds, routing by expense category, and external accountant sign-off.
Caveats
- Non-response skew by company size: 28.0% of invited admins completed the survey (460/1,640), but 50-249 employee businesses made up 23.9% of respondents versus 10.0% of the lapsed-trial database (and 1-9 employee businesses 36.5% vs 55.0%). Unweighted totals overstate enterprise-skewed issues (Sage integration, complex approvals) and understate small-business setup friction.
- Self-reported company-size inconsistencies in Q6: at least 10 of 109 open-text respondents stated headcounts that contradict their Q6 size band (e.g., T010, T087, T164, T322, and T343 are coded '50-249 employees' in Q6 while describing teams of 9, 5, 4, 25, and 30 in Q1; T023, T208, T307, and T449 are coded '1-9 employees' while citing 15, 12, 10, and 18 staff). Respondents may have misclicked, reported parent-group headcount, or rushed for the £10 voucher, so Q6 size cross-tabs carry classification noise.
- Stated willingness to pay (Q5) is hypothetical and flawed in routing: Q5 was asked of Xero and QuickBooks users (63.7% of respondents) whose software already connects—where only 37.6% overall said they would likely pay—and hypothetical 5-point scales routinely overstate real conversion by 2–3x.
- Product fragmentation inside 'Sage': Q3 and Q4 grouped all Sage products together, whereas open-text answers show demand is split across cloud (Sage Business Cloud Accounting) and desktop/mid-market packages (Sage 50, Sage 200) with very different engineering costs.
- Survivorship of lapsed trials only: the survey excluded teams that converted to paid, so we cannot compare whether paid teams had higher initial seat utilization, simpler approvals, or used Xero/QuickBooks onboarding differently.
Next steps
- decisionKeep the 30-day trial length; reject the proposal to cut it to 14 days.
- experimentRun an A/B experiment offering free 20-minute assisted setup (CSV staff import, category templates, accounting link) to new Xero and QuickBooks trials, paired with pausing/extending the trial clock if setup is incomplete at day 21.
- researchAudit Sage edition mix and true database headcount before committing engineering to Sage: add a Sage sub-version dropdown (Business Cloud, Sage 50, Sage 200) to signup/integration request forms, email the 117 Sage respondents a 1-click version check, and reconcile Q6 size against product-DB user counts.
- decisionScope quick-win approval and mobile fixes alongside integration discovery: allow single-approver/direct-to-director sign-off, batch approval, and investigate 4G receipt-upload/cropping failures.
- experimentTest 'active-claimant' or base-bundle pricing on sales-assisted deals with 10+ staff rather than launching a cheaper per-seat plan.
Open-text themes it coded
Lack of time to set up or burdensome setup process 24Price too high or per-user pricing mismatch for occasional claimants 23Missing accounting software integration (Sage, FreeAgent, and others) 22Rigid approval workflows that do not match sign-off practices 15No genuine need or expense volume too low to justify software 13Mobile app upload failures, freezing, and receipt OCR errors 10Chose a competing expenses or spend-card tool 8
The survey it planned
3 screening questions and 10 questions, as the model wrote them.
-
S1
Is the business that trialled Tallyroom based in the UK?
One answer- Yes
- No
Continues if Yes
-
S2
Which of the following best describes your involvement with Tallyroom in the last 3 months?
One answer- I signed up for or managed our free trial, and we did not move onto a paid plan
- I helped evaluate our free trial, and we did not move onto a paid plan
- We are currently paying for Tallyroom or are still in our free trial
- I only submitted expenses as a staff member and had no say in whether to subscribe
- I have never heard of Tallyroom
Continues if I signed up for or managed our free trial, and we did not move onto a paid plan; I helped evaluate our free trial, and we did not move onto a paid plan
-
S3
Roughly how many employees work at your business in total, including full-time and part-time staff?
One answer- 1 to 9 employees
- 10 to 49 employees
- 50 to 249 employees
- 250 or more employees
Continues if 1 to 9 employees; 10 to 49 employees; 50 to 249 employees
-
Q1
How far did your team get during your 30-day Tallyroom free trial?
One answer- Signed up, but never started setting up the account
- Started setting up company or accounting settings, but did not finish
- Tested photographing and approving receipts by myself only
- Tested Tallyroom with a small group of staff or managers
- Used Tallyroom with most or all relevant staff during the trial
-
Q2
Which of the following played a part in your decision not to move onto a paid Tallyroom plan? Select all that apply.
Any that apply- Tallyroom did not connect to our accounting software
- Tallyroom was missing another feature we needed
- Setting up the account and getting the team using it took too much time or effort
- £6 per user per month was too expensive for us
- We put off testing because we had 30 days, lost momentum, and the trial ended
- 30 days was not long enough to test Tallyroom across a full monthly expense run
- We chose a different expenses app instead
- We decided to stick with spreadsheets or our existing process
- Another reason not listed here
-
Q3
What was the single biggest reason your team did not subscribe to Tallyroom?
One answer- Tallyroom did not connect to our accounting software
- Tallyroom was missing another feature we needed
- Setting up the account and getting the team using it took too much time or effort
- £6 per user per month was too expensive for us
- We put off testing, lost momentum, and the trial expired before we properly evaluated it
- 30 days was not long enough to complete our evaluation
- We chose a different expenses app instead
- We decided we did not need an expenses app and kept our existing process
- Another reason
-
Q4
Which accounting software does your business mainly use?
One answer- Xero
- QuickBooks
- Sage (for example, Sage Accounting or Sage 50)
- FreeAgent
- KashFlow
- Zoho Books
- NetSuite
- Microsoft Dynamics 365 Business Central
- We do not use accounting software (for example, spreadsheets or an external bookkeeper handles it)
- Other accounting software
- Not sure
-
Q5
Which features or capabilities, if any, were missing or not good enough in Tallyroom for your team? Select all that apply.
Any that apply- Direct integration with our accounting software
- Company card or bank feed matching
- Mileage or per-diem allowance claims
- Multi-stage or department-based approval rules
- Accurate automatic reading of amounts and VAT from receipt photos
- Multi-currency support for overseas expenses
- Reporting or spending controls for managers
- None — Tallyroom had the product features we needed
-
Q6
Where, if anywhere, did you run into difficulty getting Tallyroom up and running? Select all that apply.
Any that apply- I never started setting up the account
- Connecting Tallyroom or mapping nominal codes and VAT rates to our accounting software
- Configuring expense categories, policies, or manager approval workflows
- Getting managers or staff to download the app and submit test receipts
- Knowing what steps to take first to run a proper trial
- None — setting up Tallyroom was straightforward
-
Q7
How did Tallyroom's price of £6 per user per month affect your decision?
One answer- Price was not the issue — we decided against Tallyroom for other reasons
- £6 per user felt too expensive because many staff only claim expenses occasionally
- £6 per user was more expensive than another expenses app we chose
- The total monthly cost was not worth it compared with managing expenses manually
- I did not look closely at pricing before stopping the trial
-
Q8
Thinking about the 30-day free trial length, which statement best matches what happened with your team?
One answer- 30 days was about the right amount of time to evaluate Tallyroom
- 30 days was too short — we needed more time to cover a monthly expense cycle and get team feedback
- Having 30 days meant there was no urgency, so we put off testing and lost momentum
- Other work got in the way — the length of the trial was not the reason we stopped
- We realised early on that Tallyroom was not right for us, so trial length made no difference
-
Q9
If Tallyroom's free trial had been 14 days instead of 30 days, what would most likely have happened?
One answer- The shorter deadline would have prompted us to test Tallyroom sooner
- 14 days would not have been enough time to test Tallyroom with staff and our monthly accounts
- We would have been put off signing up for the free trial in the first place
- Nothing would have changed — we still would not have subscribed
-
Q10
Which ONE of the following would have made the biggest difference in convincing your team to become a paying Tallyroom customer?
One answer- A direct integration with our accounting software
- A free 20-minute setup call where a Tallyroom specialist configures our account with us
- A cheaper plan (for example, a lower fee or paying only for staff who submit expenses that month)
- More than 30 days to test Tallyroom across a full monthly cycle
- A shorter, more structured trial with check-ins to keep us moving
- Another product feature (such as company card matching, mileage, or custom approvals)
- None of these — we do not need an expenses app right now
Judges: double-barrelledThe shorter-trial option bundles reduced duration with structure and check-ins, so selecting it cannot identify which intervention matters. Also, the only rejection option requires respondents to say they do not need an expenses app; teams that need one but would not be persuaded by any listed change have no suitable answer.
The option 'A shorter, more structured trial with check-ins to keep us moving' groups three separate concepts (shorter, structured, check-ins).
Sample plan
ELIGIBLE POOL & INVITATIONS: Email the primary admin of every UK team in the product database whose 30-day trial ended in the last 90 days without converting to paid (~1,558 unique teams from ~1,900 quarterly trial starts at an 82% non-conversion rate, after deduplicating repeat trials by the same company and excluding accounts in active sales negotiations). Offer a £10 voucher upon survey completion and send up to two reminders (days 4 and 9) to achieve an overall response rate of ~29%. QUOTAS & PRECISION NOTE: Target ~450 total completes with quotas of n=150 per employee band (1-9, 10-49, 50-249 in S3). If company size is captured at signup, invite all 50-249 and 10-49 admins immediately and release 1-9 invites in batches so micro-businesses do not exhaust the voucher budget early. Flag an important statistical trade-off to Product upfront: achieving ±6 percentage points at 95% confidence for an infinite population requires n=267 per band (801 completes total), which exceeds both the 450-complete budget and the likely 90-day supply of lapsed 50-249 trials. At n=150 per band, worst-case margin of error (at p=50%) is ±8.0pp uncorrected, improving to roughly ±5.7pp–7.2pp once finite population correction (FPC) is applied against each band's 90-day cohort size (and ±6.4pp or better for proportions near 20%). If the 50-249 band has fewer than ~350 lapsed trials in 90 days and falls short of n=150, top up solely that band from trials lapsed 91–150 days ago (checking for product/seasonal stability). Apply post-stratification weights matching the true 90-day lapsed-trial size distribution when reporting overall estimates.