Benchmarks / SurveyBench / Why Tallyroom trials don't convert / Gemini 3.5 Flash Lite
Gemini 3.5 Flash Lite: Why Tallyroom trials don't convert
The decision: where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →
- Verdict
- ✗ Not sound
- Research score
- 55 of 100
- Analysis rating
- 662
- Head to head, this task
- won 2 of 10
Why it is not sound
The survey
- Presumed the stakeholder's view instead of testing it
- A research objective no question serves
- Ignored the precision the brief asked for, which the sample cannot deliver
- A leading question: Q6
- A leading question: Q7
- A leading question: Q9
- A double-barrelled question: Q2
- A double-barrelled question: Q4
- A double-barrelled question: Q7
- A double-barrelled question: Q9
The analysis
- Got at least one set number wrong
- Reported a material finding the data does not support
- Missed a trap: Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
Findings the judges found unsupported
Lack of a Sage integration is the single biggest product barrier for larger businesses, accounting for over 40% of lapsed trials among 50-249 employee firms and representing 25.4% of all lapsed trials.
The 40.9% figure measures larger respondents selecting Sage integration as a requirement, not its share as the main barrier. The 25.4% figure is Sage usage among unweighted respondents, not the proportion of all lapsed trials blocked by Sage integration. Company-size weighting is needed before generalising demand to all lapsed trials.Connecting to missing accounting integrations (primarily Sage) unlocks strong purchase intent among currently lapsed prospects.
This presents a hypothetical stated intention to pay as if building integrations 'unlocks' purchases, and marks it high confidence. Stated intent overstates behaviour.Per-user pricing models alienate larger teams with low claim volume, representing a major secondary pricing barrier.
The data does not show that per-company pricing demand is concentrated in larger teams. In Q3 it is 22.6% among 1-9 employee firms vs 20.0% among 50-249 employee firms.
Scorecard
Numbers 6 of 11 right
| Question | Its answer | True | |
|---|---|---|---|
| What percentage of respondents said they would need a Sage integration before paying? | ✓ | 23.7% | 23.7% |
| Weighted to the company-size mix of all lapsed trials, what percentage would need a Sage integration? | ✗ | 26.5% | 18.2% |
| Among businesses with 1-9 employees, what percentage gave 'didn't have time to set it up' as the main reason? | ✓ | 27.4% | 27.4% |
| How many of the open-text answers mention not having time to set Tallyroom up, or setup taking too long? | ✗ | 18 | 24 |
| What is the 95% margin of error, in percentage points, on the unweighted share of respondents needing a Sage integration? | ✓ | 3.9 points | 3.89 points |
| Is the difference in the share giving 'too expensive' as their main reason between businesses with 1-9 employees and those with 50-249 statistically significant at the 95% level? (1 for yes, 0 for no) | ✓ | No | No |
| Weighted to the company-size mix of all 1,640 lapsed trials, about how many teams' main reason was not having time to set it up? | ✗ | 307 | 355 |
| How many of the open-text answers from businesses with 1-9 employees mention not having time to set Tallyroom up, or setup taking too long? | ✗ | 8 | 14 |
| Weighted to the company-size mix of all lapsed trials, what percentage would give 'no integration with our accounting software' as their main reason? | ✗ | 18.3% | 16.3% |
| What percentage of the trial admins invited completed the survey? | ✓ | 28% | 28% |
| Among respondents whose main reason was no integration with their accounting software, what percentage said they would be likely (fairly or very) to pay if Tallyroom connected to it? | ✓ | 61.3% | 61.3% |
Traps in the data 6 of 7 handled
- ✓Teams on 'Other' accounting software (base 17) mostly cite missing integration.
- ✗Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
- ✓Q2 is multi-select: percentages are of respondents and add up to more than 100%.
- ✓Likelihood to pay is a hypothetical stated intention.
- ✓The head of sales believes the trial is too long and teams forget about it.
- ✓460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
- ✓Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.
The analysis
Do not shorten the trial. Put next quarter's product effort into building a Sage integration, and sales/onboarding effort into assisted setup (such as onboarding templates or concierge setup) for smaller teams. Sage integration directly addresses the primary blocker for high-value mid-market prospects, while assisted setup resolves the main friction point for micro-businesses.
Definitive usage analytics showing that trials typically achieve full setup and engagement within the first 14 days, combined with a willingness-to-pay test demonstrating that prospects requesting Sage integrations will not churn on price once built.
Findings
-
Lack of a Sage integration is the single biggest product barrier for larger businesses, accounting for over 40% of lapsed trials among 50-249 employee firms and representing 25.4% of all lapsed trials.
25.4% of total respondents use Sage (Q4). 85.5% of Sage users say they need a Sage integration before paying (Q3), and 58.1% of Sage users gave no integration as their main reason for not paying (Q2).
-
Setup friction is a severe barrier specifically for micro-businesses (1-9 employees), where over a quarter cite it as the main reason for failing to convert.
27.4% of 1-9 employee businesses (Base: 168) gave 'Didn't have time to set it up' as their main reason (Q2), supported by numerous open-text mentions regarding lack of time.
-
The head of sales' hypothesis to shorten the trial from 30 to 14 days is unsupported and would likely worsen conversion.
Only 0.2% of respondents stated the trial was too long, while 18.7% stated they ran out of time to set it up. Open-text answers confirm that users struggled to find time within the existing 30 days.
-
Connecting to missing accounting integrations (primarily Sage) unlocks strong purchase intent among currently lapsed prospects.
61.3% of respondents whose main reason for not paying was 'No integration with our accounting software' stated they would be likely to pay £6/user/month if connected (Q5, Base: 93).
-
Per-user pricing models alienate larger teams with low claim volume, representing a major secondary pricing barrier.
20.4% overall (and up to 26.1% of QuickBooks users) requested per-company pricing in Q3, with qualitative feedback reinforcing that paying per head for non-active users makes the tool unviable.
Caveats
- Survey relies on self-reported reasons for churn after the fact, which may be subject to rationalisation (e.g., citing 'price' when setup or integration was the true blocker).
- Sample represents 460 completes out of 1,640 lapsed trials (28% response rate); non-response bias may exist if unresponsive trialists had different reasons for churning.
- Expressed purchase intent in Q5 (£6/user/month if integrated) typically overstates actual conversion behavior upon release.
Next steps
- decisionPrioritise engineering resources to build and launch the Sage accounting integration.
- decisionReject the proposal to shorten the free trial to 14 days.
- experimentDesign and test an assisted setup or automated import onboarding flow for 1-9 employee businesses.
- researchConduct pricing sensitivity research (e.g., Van Westendorp) on flat-fee or per-company pricing tiers.
Open-text themes it coded
Missing Sage integration 21Lack of time / setup friction 20Per-user pricing mismatch for low-volume claimants 16Rigid or mismatched approval workflows 13Realisation that expenses tool is unnecessary / low volume 10Mobile app or OCR technical failures 7Churned to a competitor 8
The survey it planned
2 screening questions and 9 questions, as the model wrote them.
-
S1
What is the total number of employees in your company, including yourself?
One answer- 1 to 9 employees
- 10 to 49 employees
- 50 to 249 employees
- 250 or more employees
Continues if 1 to 9 employees; 10 to 49 employees; 50 to 249 employees
-
S2
Were you involved in the decision to trial Tallyroom for your company's expense management?
One answer- Yes, I was the primary decision maker or setup user
- Yes, I was part of the evaluation team
- No, I was not involved
Continues if Yes, I was the primary decision maker or setup user; Yes, I was part of the evaluation team
-
Q1
What is your primary accounting software used for your business finances?
One answer- Xero
- QuickBooks Online
- Sage
- FreeAgent
- Xero and QuickBooks
- Other (please specify)
- We do not use cloud accounting software
-
Q2
Which of the following best describes why your team did not subscribe to a paid Tallyroom plan after your free trial ended? (Select all that apply)
Any that apply- It lacked an integration with our accounting software
- It lacked key features or functionality we needed
- It was too difficult to set up or configure
- The price (£6 per user/month) was too high for the value provided
- We forgot we were trialing it or ran out of time to evaluate it properly
- Our team decided not to use any digital expense tool
- Other
Judges: double-barrelledOne checkbox combines forgetting about the trial with running out of evaluation time. These are distinct explanations with different implications for shortening the trial.
'We forgot we were trialing it or ran out of time to evaluate it properly' joins two opposite reasons in one option. Forgetting suggests the trial was too long; running out of time suggests it was too short. This makes the option useless for O4 and for the trial-length decision.
-
Q3
You indicated that missing features or integrations influenced your decision. Which specific integrations or features were missing that you needed?
Open textRouting Shown only if Q2 is: It lacked an integration with our accounting software; It lacked key features or functionality we needed
-
Q4
How much did setup difficulty or lack of guidance contribute to your decision not to subscribe?
ScaleScale 1-5: Did not contribute at all to Major factor in not subscribing
Judges: double-barrelledA single rating combines setup difficulty and lack of guidance, so their separate contributions cannot be identified.
'Setup difficulty' and 'lack of guidance' are two separate things rated on one scale.
-
Q5
How do you feel about the current pricing of £6 per user per month for Tallyroom?
One answer- Too expensive for the value offered
- Fairly priced, but budget was not approved
- Fairly priced, but other factors stopped us
- Price was not a factor in our decision
-
Q6
How did you find the length of the 30-day free trial period?
One answer- Too long (we lost momentum or forgot about it)
- Just right
- Too short (we needed more time to test it thoroughly)
Judges: leadingThe 'Too long' option supplies the stakeholder's explanation—lost momentum or forgetting—rather than allowing respondents to judge trial length independently of that explanation.
The options attach specific, emotive reasons to the lengths (e.g. 'lost momentum or forgot'), heavily steering respondents towards the stakeholder's narrative.
-
Q7
If Tallyroom offered a shorter trial period (for example, 14 days) combined with guided onboarding support, how likely would you have been to complete your evaluation?
ScaleScale 1-5: Much less likely to Much more likely
Judges: leading, double-barrelledIt bundles a shorter trial with guided onboarding support. Support can make the package attractive without respondents favouring a shorter trial, steering and confounding the test of trial length.
It bundles a shorter trial with guided onboarding, an attractive add-on, so the onboarding steers answers towards apparent support for the shorter trial and the two effects cannot be separated. It also assumes respondents did not complete their evaluation, when many may have evaluated fully and decided against paying.
-
Q8
If you could wave a wand and have Tallyroom fix or add one thing to make you subscribe today, what would it be?
Open text -
Q9
Which of the following potential improvements would most likely have convinced your company to subscribe? (Rank your top 3)
Ranking- Integration with additional accounting software (e.g., Sage, FreeAgent)
- Assisted setup and dedicated onboarding help
- A lower per-user subscription price
- A shorter, more focused trial period with reminders
- Additional approval workflow features
Judges: leading, double-barrelled, assumptiveThe shorter-trial option is positively framed as 'more focused' and bundled with reminders, so its ranking cannot isolate preference for shortening. Requiring three ranked improvements also assumes at least three would have helped convince the company to subscribe.
The option 'A shorter, more focused trial period with reminders' is framed positively ('more focused') and bundles reminders with the shorter trial, which favours the sales hypothesis. The forced top-3 ranking offers no 'none of these' option, so it assumes some improvement would have convinced every respondent.
Sample plan
We will pull a database list of all UK company admins whose Tallyroom free trial ended in the last 90 days without converting to a paid subscription. Invitations will be sent via email with an incentive of a £10 voucher upon survey completion. To achieve a precision of +/- 6 percentage points at a 95% confidence level across three company size bands (1-9, 10-49, 50-249 employees), we will quota-sample to secure roughly 150 completes per band, totaling approximately 450 completes.