Benchmarks / SurveyBench / Why Tallyroom trials don't convert / Gemini 4 Hydrogen
Gemini 4 Hydrogen: Why Tallyroom trials don't convert
The decision: where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →
- Verdict
- ✗ Not sound
- Research score
- 85 of 100
- Analysis rating
- 1,088
- Head to head, this task
- won 8 of 12
Why it is not sound
The survey
- Presumed the stakeholder's view instead of testing it
- A leading question: Q7
- A double-barrelled question: Q7
The analysis
- Missed a trap: Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.
Scorecard
Numbers 11 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| What percentage of respondents said they would need a Sage integration before paying? | ✓ | 23.7% | 23.7% |
| Weighted to the company-size mix of all lapsed trials, what percentage would need a Sage integration? | ✓ | 18.1% | 18.2% |
| Among businesses with 1-9 employees, what percentage gave 'didn't have time to set it up' as the main reason? | ✓ | 27.4% | 27.4% |
| How many of the open-text answers mention not having time to set Tallyroom up, or setup taking too long? | ✓ | 24 | 24 |
| What is the 95% margin of error, in percentage points, on the unweighted share of respondents needing a Sage integration? | ✓ | 3.9 points | 3.89 points |
| Is the difference in the share giving 'too expensive' as their main reason between businesses with 1-9 employees and those with 50-249 statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| Weighted to the company-size mix of all 1,640 lapsed trials, about how many teams' main reason was not having time to set it up? | ✓ | 355 | 355 |
| How many of the open-text answers from businesses with 1-9 employees mention not having time to set Tallyroom up, or setup taking too long? | ✓ | 14 | 14 |
| Weighted to the company-size mix of all lapsed trials, what percentage would give 'no integration with our accounting software' as their main reason? | ✓ | 16.3% | 16.3% |
| What percentage of the trial admins invited completed the survey? | ✓ | 28% | 28% |
| Among respondents whose main reason was no integration with their accounting software, what percentage said they would be likely (fairly or very) to pay if Tallyroom connected to it? | ✓ | 61.3% | 61.3% |
Traps in the data 6 of 7 handled
- ✓Teams on 'Other' accounting software (base 17) mostly cite missing integration.
- ✓Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
- ✓Q2 is multi-select: percentages are of respondents and add up to more than 100%.
- ✓Likelihood to pay is a hypothetical stated intention.
- ✓The head of sales believes the trial is too long and teams forget about it.
- ✓460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
- ✗Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.
The analysis
Reject cutting the trial to 14 days and reject a blanket price cut. Put next quarter's sales and onboarding effort immediately into Assisted Setup (plus guided self-serve templates) to capture the 21.6% weighted share—especially Xero and QuickBooks trials—stalling before configuration is finished. Put next quarter's product engineering effort into scoping and building a Sage integration (specifically targeting whichever Sage edition dominates our 10-249 employee trial pipeline, likely Sage Business Cloud / Sage 50) as the highest-LTV expansion lever, while pairing it with lightweight fixes to approval flexibility or active-user pricing tests rather than across-the-board discounting.
Three findings would alter this plan: (1) If trial-pipeline telemetry shows Sage trialists are fragmented across legacy desktop Sage 50 and Sage 200 rather than Sage Business Cloud, requiring multiple costly connectors next quarter—in which case product effort should shift to custom approval workflows and mobile OCR reliability for Xero/QB first; (2) If an assisted-setup experiment fails to lift Xero/QB paid conversion because 'no time to set it up' proves to be polite rationalisation for low underlying need; or (3) If a pricing experiment charging per active claimant (or flat company tiers) generates higher net revenue among Xero/QB teams than projected incremental Sage ARR.
Findings
-
Do not cut the 30-day trial to 14 days: trial length is virtually never cited as a problem, whereas running out of time to finish setup is the #1 weighted cause of non-conversion.
In Q2 (n=460), only 0.2% said 'The trial was too long' and 3.7% said they 'Forgot the trial was running' (vs 1.5% saying 'too short'). Conversely, 18.7% unweighted (21.6% weighted to the 1,640 lapsed-trial size mix, ~355 teams) gave 'Didn't have time to set it up' as their main reason, rising to 27.4% among 1-9 employee businesses (n=168). In Q1 open text (n=109), 24 respondents (14 in 1-9 employee firms) described setup getting interrupted or taking longer than the trial allowed.
-
Assisted setup offers the fastest, broadest conversion lift because it directly targets already-compatible Xero and QuickBooks trials that churn before completing onboarding.
Xero (39.6%) and QuickBooks (24.1%) make up 63.7% of all respondents (n=460), yet 24.7% of lapsed Xero trials (n=182) and 22.5% of QuickBooks trials (n=111) failed to convert primarily due to setup time. Across the full lapsed-trial population (55% 1-9 employees), setup time is the single largest weighted blocker at 21.6%, ahead of missing integrations (16.3%) and price (16.6%).
-
Demand for a Sage integration is substantial among mid-sized businesses (10-249 employees) and represents a high-value product opportunity, though weighted population demand is lower than unweighted totals suggest and is split across distinct Sage products.
Unweighted, 23.7% (±3.9 pp at 95% CI, n=460) need a Sage integration before paying, and 25.4% (n=117) of respondents use Sage (58.1% citing missing integration as main reason; 85.5% needing it before paying). However, Sage need rises sharply with firm size—7.7% in 1-9 (n=168), 28.0% in 10-49 (n=182), and 40.9% in 50-249 (n=110)—so weighting to the true lapsed-trial mix (55%/35%/10%) lowers overall Sage demand to 18.1%. Open-text answers also split across Sage Business Cloud (T334, T339), Sage 50 (T413), Sage 200 (T241), and Sage Accounting (T433).
-
Connecting to missing accounting software unlocks the strongest stated willingness to pay (61.3% likely), but actual paid conversion will be materially lower due to hypothetical-intent inflation and secondary product/pricing blockers.
In Q5, 61.3% (35.5% fairly likely, 25.8% very likely) of integration-blocked respondents (n=93) say they would pay £6/user/month if connected, vs 37.6% overall (n=460) and 23.3%–36.2% in other main-reason segments. However, only 10.3% of Sage users (n=117) and 15.2% of FreeAgent users (n=33) selected 'None of these' on Q3 prerequisites; 28.2% of Sage users also need custom approval chains and 20.5% want per-company pricing. Even among already-integrated Xero and QuickBooks trials, 18.1%–21.6% still balk at price and 14.3%–15.3% at approval workflows.
-
A blanket price cut is poorly targeted: price sensitivity does not differ significantly by company size, and objections focus on per-seat billing for infrequent claimants rather than the £6 unit rate alone.
In Q2, 15.9% overall cited 'Too expensive' as their main reason, with no statistically significant difference between 1-9 employees (19.0%, n=168) and 50-249 employees (15.5%, n=110; z=0.77, p=0.44). Only 23.3% of price-blocked teams (n=73) say they would pay £6/user/month even with integration. In Q3, 20.4% (n=460) requested per-company pricing (not per user), and Q1 comments (e.g., T059, T208, T322, T343, T355, T402) repeatedly object to paying for total headcount when only a few staff submit claims.
-
Inflexible approval workflows and mobile receipt-scanning failures are significant secondary churn drivers—especially among larger and field-based teams—and risk undermining retention even if setup or Sage integration are fixed.
In Q2, 'Approval workflow didn't fit' accounts for 12.6% overall and rises to 21.8% among 50-249 employee businesses (n=110), while 'Custom approval chains' is the #2 most requested requirement in Q3 (22.0% overall; 27.3% in 50-249; 28.2% among Sage users). Mobile app issues account for 6.3% overall (10.8% in QuickBooks, n=111) and 10 open-text answers cite upload failures on 4G, freezing, or misread OCR totals destroying trust (T195, T309, T317, T378). FreeAgent integration is a smaller niche (7.2% unweighted, 9.1% weighted, n=33 users).
Caveats
- Severe company-size non-response bias in unweighted tables: 50-249 employee businesses make up 23.9% of survey completes (n=110) vs 10% of all 1,640 lapsed trials, while 1-9 employee businesses are 36.5% vs 55%. Unweighted totals therefore overstate Sage integration and approval-workflow demand and understate setup-time barriers.
- Hypothetical willingness-to-pay bias in Q5: 61.3% 'likely' (35.5% 'fairly likely', 25.8% 'very likely') overstates real-world conversion, especially because 89.7% of Sage users also flagged additional prerequisites in Q3 (such as custom approval chains and per-company pricing).
- Unspecified Sage product editions in structured questions: Q3 and Q4 group all Sage users together, whereas Q1 open text reveals distinct, non-interchangeable products (Sage Business Cloud, Sage 50, Sage 200, and Sage Accounting). Building a single connector will not capture all 25.4% of Sage users.
- Non-response and recall limitations: 28.0% of invited admins completed the survey (with a £10 voucher incentive and up to 90 days post-trial recall), so teams least engaged during the trial—including those who forgot or lacked time—may be under-represented despite weighting.
- Internal inconsistencies between Q6 company size and Q1 descriptions: several respondents in the 50-249 bracket wrote of having 'five', 'nine', or '25' staff (e.g., T010, T059, T087, T164, T322), and some in 1-9 wrote of '15' or '18' staff (T023, T449), suggesting confusion between parent company size and local team size.
- Open-text answers cover 109 of 460 respondents and explain failure mechanisms rather than standalone population prevalence.
Next steps
- decisionKeep the 30-day trial length unchanged and cancel any plan to cut it to 14 days.
- experimentLaunch an Assisted Setup pilot (15-minute live onboarding call + simplified default templates and CSV user import) for new Xero and QuickBooks trials.
- researchRun a 1-week follow-up check on Sage trialists (short 2-question email poll or sales call log audit) to quantify exact edition split (Sage Business Cloud vs Sage 50 vs Sage 200) and average billable seat count.
- experimentPrototype and A/B test an 'active-user' or tiered per-company pricing option alongside £6/user/month rather than cutting the per-seat price.
- monitorAudit mobile receipt upload failures on 4G/Android and scope top 2 approval blockers (multi-stage/email approval and batch sign-off) into the post-Sage product backlog.
Open-text themes it coded
Missing accounting software integration (Sage, FreeAgent, or other packages) 22Insufficient time or overly burdensome setup during the trial 24Price too high or per-user billing misaligned with occasional claimants 23Approval workflow did not match team sign-off practices 15Low claim volume or no perceived need for expenses software 12Mobile app reliability and receipt scanning or OCR failures 10Chose a competitor or alternative tool 8
The survey it planned
3 screening questions and 10 questions, as the model wrote them.
-
S1
Were you involved in setting up, testing, or deciding whether your business would pay for Tallyroom during your recent free trial?
One answer- Yes, I was the main person responsible
- Yes, I was involved alongside someone else
- No, I was not involved
Continues if Yes, I was the main person responsible; Yes, I was involved alongside someone else
-
S2
Which of the following best describes your business's location and current status with Tallyroom?
One answer- We are a UK-based business and chose not to pay after our free trial ended
- We are a UK-based business and are currently paying for Tallyroom
- We are based outside the UK
- We have not trialled Tallyroom / I am not sure
Continues if We are a UK-based business and chose not to pay after our free trial ended
-
S3
Approximately how many people work for your business in total (including yourself)?
One answer- 1–9 employees
- 10–49 employees
- 50–249 employees
- 250 or more employees
Continues if 1–9 employees; 10–49 employees; 50–249 employees
-
Q1
How far did your team get with Tallyroom during your 30-day free trial?
One answer- We signed up but never really got started
- I explored it myself, but did not invite other staff
- We invited staff and tested some features, but did not connect our accounting software
- We connected our accounting software and ran real expenses through it
- Other
-
Q2
Which of the following reasons explain why your team did not subscribe to Tallyroom after the trial? Select all that apply.
Any that apply- It did not connect to the accounting software we use
- It was missing other features we needed
- Setting it up or getting staff onto it took too much time or effort
- The price (£6 per user per month) was too high for our budget
- We ran out of time before we could finish evaluating it
- The 30-day trial felt too long, and we lost momentum or forgot about it
- We decided to stick with our current process (e.g. spreadsheets, email, or paper receipts)
- We chose a different expenses app
- Our business needs changed and we no longer needed an expenses app
- Other
-
Q3
What was the single biggest reason your team did not subscribe to Tallyroom?
One answer- It did not connect to the accounting software we use
- It was missing other features we needed
- Setting it up or getting staff onto it took too much time or effort
- The price (£6 per user per month) was too high for our budget
- We ran out of time before we could finish evaluating it
- The 30-day trial felt too long, and we lost momentum or forgot about it
- We decided to stick with our current process (e.g. spreadsheets, email, or paper receipts)
- We chose a different expenses app
- Our business needs changed and we no longer needed an expenses app
- Other
-
Q4
Which accounting software does your business mainly use?
One answer- Xero
- QuickBooks
- Sage (e.g. Sage Business Cloud, Sage 50)
- FreeAgent
- KashFlow
- Clear Books
- Zoho Books
- Spreadsheets or manual bookkeeping (no accounting software)
- Another accounting package
- Not sure
-
Q5
Which, if any, missing integrations or features made Tallyroom less useful for your business? Select all that apply.
Any that apply- Integration with Sage
- Integration with FreeAgent
- Integration with another accounting or payroll system
- Multi-level or custom approval workflows
- Mileage tracking
- Company credit card or bank feed reconciliation
- VAT or HMRC compliance reporting features
- Multi-currency support
- Project or cost-centre tracking
- None — Tallyroom had all the features and integrations we needed
- Not sure — we did not explore the features enough
-
Q6
Thinking about the length of Tallyroom's 30-day free trial, how did 30 days feel for your team?
One answer- Too short — we needed more time to set it up and test it properly
- About right — 30 days was enough time to evaluate it
- Too long — we lost focus or forgot about it before the trial ended
- Trial length did not matter — we knew early on it was not right for us
- Not sure
-
Q7
Which of the following, if available during your trial, would have made your team more likely to become a paying Tallyroom customer? Select all that apply.
Any that apply- Direct integration with our accounting software
- A free 1-to-1 guided setup call to connect our accounting software and onboard staff
- A lower price per user (below £6 per user per month)
- A shorter, more focused trial (e.g. 14 days) with milestone reminders
- A longer trial (e.g. 60 days) to cover a full month-end cycle
- Specific features we needed that were missing
- None of these — we would not have subscribed regardless
Judges: leading, double-barrelledThe shorter-trial option describes it positively as 'more focused' and bundles shortening the trial with milestone reminders. Selecting it cannot establish whether the shorter duration, the reminders, or both would help.
The shorter-trial option is framed favourably ('more focused'). It also bundles 'milestone reminders' with the shorter length, so anyone who wanted reminders is counted as wanting a 14-day trial. The longer-trial option gets no equivalent sweetener.
-
Q8
If Tallyroom offered a 14-day free trial instead of a 30-day free trial, how would that have affected your team's likelihood of subscribing?
One answer- Much more likely to subscribe (a shorter deadline would have kept us focused)
- Slightly more likely to subscribe
- Would have made no difference
- Slightly less likely to subscribe (we needed more than 14 days)
- Much less likely to subscribe (14 days is not enough time to test and get sign-off)
- Not sure
-
Q9
To help us check survey quality, please select 'Monthly expenses' from the options below.
One answer- Travel receipts
- Monthly expenses
- Supplier invoices
- Annual accounts
-
Q10
In your own words, what is the one thing Tallyroom could have done differently during your trial that would have made your business pay for it?
Open text
Sample plan
Population and invitations: Send an email invitation (with reminders at Day 3 and Day 7) and a £10 voucher incentive to the admin of every UK team whose 30-day Tallyroom trial ended in the last 90 days without paying (~1,558 teams, based on ~1,900 quarterly trial starts at an 18% conversion rate). Completes and quotas: Target 450 valid completes (~29% response rate, excluding anyone failing S1–S3 or Q9 attention check), using stratified quotas of 150 completes per company-size band (1–9, 10–49, 50–249 employees) to support subgroup comparisons. Weight overall totals back to the true company-size distribution of lapsed trials in the product database. Precision note: Achieving +/-6 percentage points at 95% confidence (at p=0.5) requires n=267 per band without finite population correction (FPC), or roughly 175–200 per band with FPC if each band holds ~500 lapsed trials (~525–600 total completes). At the 450-complete cap (150 per band), the maximum 95% margin of error per band is +/-8.0 pp uncorrected (or roughly +/-6.7–7.0 pp with FPC, though tighter if a band such as 50–249 has a smaller database population and approaches a census). If strict +/-6 pp in all three bands is mandatory, expand the target to ~550–600 completes or confirm exact band populations in the database to leverage FPC.