Benchmarks / SurveyBench / Why Tallyroom trials don't convert / Gemini 3.8 Flash

Measured by Spring Prompt

Gemini 3.8 Flash: Why Tallyroom trials don't convert

The decision: where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →

Verdict
✗ Not sound
Research score
72 of 100
Analysis rating
898
Head to head, this task
won 9 of 11

Why it is not sound

The survey

  • Presumed the stakeholder's view instead of testing it
  • Ignored the precision the brief asked for, which the sample cannot deliver
  • A leading question: Q5

The analysis

  • Reported a material finding the data does not support
  • Missed a trap: 460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
  • Missed a trap: Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.

Findings the judges found unsupported

  • Shortening the trial to 14 days will harm conversion; teams lapse because setup is too time-consuming, not because the trial is too long.The survey supports rejecting an immediate trial cut, but cannot establish that shortening the trial will harm conversion. That requires a controlled experiment; self-reported setup difficulties do not establish the effect of trial length.
  • Building a Sage integration unlocks a distinct, high-intent segment of lapsed trials, especially among mid-sized SMBs.Sage users are 25.4% of respondents, not an established 25.4% of all lapsed trials. Also, Q2 identifies the main reason for not paying, not the sole barrier. Describing this segment as unlocked by an integration treats a reported requirement as sufficient to produce conversion.
  • Integration alone converts roughly six in ten blocked prospects, but stated willingness to pay overestimates actual revenue.The 61.3% is hypothetical stated payment intent, not the proportion integration alone would convert. The caveat acknowledges this but contradicts the finding's headline. Differences between reason groups also do not prove that integration causes payment intent or solves conversion.
  • Rigid approval workflows and per-user pricing block larger SMBs with higher customer lifetime value.The data does not establish higher customer lifetime value for larger businesses. Approval problems are more common among larger respondents, but pricing objections are not: 15.5% of the largest group versus 19.0% of the smallest cite expense. The finding overextends the larger-business pattern to pricing and customer value.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents said they would need a Sage integration before paying?✓23.7%23.7%
Weighted to the company-size mix of all lapsed trials, what percentage would need a Sage integration?✓18.1%18.2%
Among businesses with 1-9 employees, what percentage gave 'didn't have time to set it up' as the main reason?✓27.4%27.4%
How many of the open-text answers mention not having time to set Tallyroom up, or setup taking too long?✓2424
What is the 95% margin of error, in percentage points, on the unweighted share of respondents needing a Sage integration?✓3.9 points3.89 points
Is the difference in the share giving 'too expensive' as their main reason between businesses with 1-9 employees and those with 50-249 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the company-size mix of all 1,640 lapsed trials, about how many teams' main reason was not having time to set it up?✓355355
How many of the open-text answers from businesses with 1-9 employees mention not having time to set Tallyroom up, or setup taking too long?✓1414
Weighted to the company-size mix of all lapsed trials, what percentage would give 'no integration with our accounting software' as their main reason?✓16.3%16.3%
What percentage of the trial admins invited completed the survey?✓28%28%
Among respondents whose main reason was no integration with their accounting software, what percentage said they would be likely (fairly or very) to pay if Tallyroom connected to it?✓61.3%61.3%

Traps in the data 5 of 7 handled

  • ✓Teams on 'Other' accounting software (base 17) mostly cite missing integration.
  • ✓Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
  • ✓Q2 is multi-select: percentages are of respondents and add up to more than 100%.
  • ✓Likelihood to pay is a hypothetical stated intention.
  • ✓The head of sales believes the trial is too long and teams forget about it.
  • ✗460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
  • ✗Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.

The analysis

Recommendation

Reject the proposal to shorten the trial to 14 days immediately. Split next quarter's effort across Product and Sales: 1) Product should build a Sage integration (targeting the 18.1% weighted demand, predominantly higher-seat businesses) and introduce lightweight assisted onboarding flows; 2) Sales and Success should introduce an assisted onboarding concierge to rescue the ~355 teams dropping off due to setup friction.

What would change it

If technical feasibility scoping reveals that integrating Sage (e.g. Sage 50 desktop vs Sage Business Cloud) requires more than one full quarter of engineering velocity, or if an assisted setup pilot fails to lift conversion among self-serve micro-businesses.

Findings

  1. Shortening the trial to 14 days will harm conversion; teams lapse because setup is too time-consuming, not because the trial is too long.

    High confidenceBase 460

    Only 0.2% (1 respondent) felt the trial was too long, and 3.7% forgot it was running. In contrast, 18.7% unweighted (and 21.6% population-weighted, ~355 lapsed teams) cited lack of time to set up as the primary reason they did not convert, rising to 27.4% among micro-businesses (1-9 employees). 24 open-text comments explicitly described setup running out of runway.

  2. Building a Sage integration unlocks a distinct, high-intent segment of lapsed trials, especially among mid-sized SMBs.

    High confidenceBase 117

    Sage users represent 25.4% of lapsed trials (n=117). Among them, 58.1% cited missing integration as their sole barrier to conversion, and 85.5% reported needing a Sage integration to pay. Demand scales strongly with company size (40.9% in 50-249 employees vs 7.7% in 1-9 employees).

  3. Integration alone converts roughly six in ten blocked prospects, but stated willingness to pay overestimates actual revenue.

    Medium confidenceBase 93

    61.3% of respondents who dropped out due to missing integrations state they are likely (35.5% fairly, 25.8% very) to pay £6/user/month if connected. However, for those citing setup or cost as blockers, intent drops to 27.9% and 23.3%, proving integration solves intent only for integration-blocked teams.

  4. Rigid approval workflows and per-user pricing block larger SMBs with higher customer lifetime value.

    High confidenceBase 460

    21.8% of 50-249 employee businesses dropped out because the approval workflow didn't fit (vs 7.1% for 1-9 employees), and 27.3% need custom approval chains. Additionally, 20.4% across sizes requested per-company pricing because infrequent submitters inflate per-seat bills.

Caveats

  • Survey sample over-represents larger businesses (23.9% in sample have 50-249 employees vs 10% in the lapsed trial population), skewing unweighted integration and workflow figures higher. Population-weighted numbers must be used for sizing.
  • Stated willingness to pay (61.3% likely among integration-blocked users) reflects hypothetical intent and will discount significantly upon actual checkout.
  • Survey excludes businesses that never signed up for a trial; it reflects barriers encountered during trial evaluation rather than top-of-funnel acquisition gaps.

Next steps

  • decisionMaintain the 30-day trial and decline the head of sales' proposal to reduce to 14 days
  • decisionScope and build Sage Business Cloud and Sage 50 integration
  • experimentPilot an 'Assisted Onboarding' program (in-app setup wizard plus proactive booking link for admins on day 2)
  • researchConduct qualitative discovery on approval workflows and active-user pricing models

Open-text themes it coded

Missing accounting integrations (Sage, FreeAgent, others) 22Setup friction, lack of administrative time, and unfinished onboarding 24Pricing friction: Per-seat model penalises companies with low-frequency submitters 23Inflexible approval hierarchies (no external accountant sign-off, multi-tier, or batching) 15No perceived need or low expense volume 12Mobile app unreliability and OCR receipt scanning inaccuracies 10Lost to direct competitors or corporate card alternatives 8

The survey it planned

2 screening questions and 10 questions, as the model wrote them.

  1. S1

    Did your business evaluate Tallyroom on a 30-day free trial within the last 90 days, and were you involved in testing, administering, or deciding on whether to purchase it?

    One answer
    • Yes, I was the primary administrator or key decision maker
    • Yes, I was involved as a tester or reviewer
    • No, I was not involved in this trial evaluation
    • No, our business never trialled Tallyroom

    Continues if Yes, I was the primary administrator or key decision maker; Yes, I was involved as a tester or reviewer

  2. S2

    How many people are currently employed by your company across the UK?

    One answer
    • 1 to 9 employees
    • 10 to 49 employees
    • 50 to 249 employees
    • 250 or more employees

    Continues if 1 to 9 employees; 10 to 49 employees; 50 to 249 employees

  3. Q1

    What was the single primary reason your team decided not to purchase Tallyroom after your trial ended?

    One answer
    • Missing a required accounting integration
    • Missing critical product features
    • Pricing was too expensive for our budget
    • Setup and configuration took too much effort or time
    • Ran out of time to evaluate properly
    • Lost momentum or forgot about the trial
    • Decided to stay with our existing process (e.g. spreadsheets, manual paper)
    • Selected a different expense management software
    • No ongoing business need for an expense tool
  4. Q2

    Which of the following factors contributed to your decision not to buy? (Select all that apply)

    Any that apply
    • Pricing structure (£6/user/month) was not viable
    • Did not connect with our specific accounting software
    • Lacked specific approval workflows or customisation
    • Receipt scanning / OCR was inaccurate or slow
    • Setting up users, categories, or integrations was difficult
    • Lack of hands-on onboarding or live support during trial
    • The 30-day trial was too short to get team members to submit receipts
    • The trial dragged on too long and we lost internal momentum
    • Chosen software offered better features or lower price
    • Expense volume was too low to justify paying
  5. Q3

    Which accounting software does your company currently use as its primary system?

    One answer
    • Xero
    • QuickBooks Online
    • Sage (Sage 50, Sage Business Cloud, or Sage Intacct)
    • FreeAgent
    • Microsoft Dynamics 365 / Business Central
    • Spreadsheets (Excel / Google Sheets)
    • Other
  6. Q4

    Which missing integration or product capability was the biggest barrier to adopting Tallyroom?

    One answer
    • Integration with Sage
    • Integration with FreeAgent
    • Integration with other ERP/accounting system
    • Corporate credit card live feeds
    • Multi-tiered approval workflows
    • Mileage tracking & UK vehicle compliance
    • Per-diem / subsistence allowances
    • None — missing features were not a barrier
  7. Q5

    How did you find the 30-day duration of the free trial?

    One answer
    • Far too short — we needed more time to test with employees
    • Slightly too short — we rushed at the end
    • Just the right amount of time to evaluate
    • Too long — we finished testing early and lost interest or momentum
    • Too long — we forgot we were trialling it until it expired
    Judges: leading

    Both 'too long' options require respondents to endorse lost interest, momentum or forgetting. This builds the stakeholder's proposed explanation into every answer indicating excessive trial length.

    The options attach specific reasons (e.g., 'forgot we were trialling it') to the duration sentiment. A respondent cannot simply say it was 'too long' without also agreeing to the specific reason provided.

  8. Q6

    At what stage in your 30-day trial did your team stop actively evaluating or testing Tallyroom?

    One answer
    • Never started testing after signing up
    • Within the first 7 days
    • Between day 8 and day 14
    • Between day 15 and day 21
    • Active until the final week (days 22–30)
    • Tested continuously until the trial expired
    Judges: ambiguous

    'Active until the final week (days 22–30)' overlaps with testing until expiry, so respondents active through day 30 could reasonably choose either of the final two options.

    The options 'Active until the final week (days 22–30)' and 'Tested continuously until the trial expired' overlap. A team that tested to the end could pick either, so the late-stage answers cannot be cleanly separated.

  9. Q7

    How would a shorter 14-day trial have affected your evaluation of Tallyroom?

    One answer
    • It would have created urgency and helped us decide faster
    • It would have made no difference to our decision
    • It would have been too short; we would have abandoned without evaluating
    • Not sure
  10. Q8

    How did you perceive Tallyroom's pricing (£6 per user per month)?

    Scale

    Scale 1-5: 1 - Extremely expensive for the value offered to 5 - Excellent value for money

  11. Q9

    If Tallyroom had provided a dedicated 20-minute assisted setup call (connecting accounting software, importing users, and testing one expense), how would that have impacted your likelihood to subscribe?

    One answer
    • Would have significantly increased my likelihood to subscribe
    • Would have slightly increased my likelihood to subscribe
    • No difference — setup was already simple
    • No difference — our blocker was unrelated to setup (e.g., price, missing integration)
  12. Q10

    What is the single most important change Tallyroom could make to win your business in the future?

    Open text
Sample plan

Email invitations will be distributed to all trial administrators from UK accounts that completed a 30-day trial in the last 90 days without subscribing (approx. 4,600 eligible contacts across 3 quarters/months). A £10 digital voucher incentive will drive a target completion rate of 10-15%. Total target sample: N=450 completes, stratified into three equal quotas of n=150 across company size bands: Micro (1-9 employees, n=150), Small (10-49 employees, n=150), and Medium (50-249 employees, n=150). With n=150 per band, the estimated margin of error per segment is ±8.0% at 95% confidence without FPC; with Finite Population Correction applied to the sub-cohort populations (~1,200 to ~1,800 per band), precision reaches approximately ±7.5%. Survey recruitment will run until all three quotas are met, closing over-represented segments dynamically.