Benchmarks / SurveyBench / Why Tallyroom trials don't convert / Mistral Medium 3.5

Measured by Spring Prompt

Mistral Medium 3.5: Why Tallyroom trials don't convert

The decision: where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →

Verdict
✗ Not sound
Research score
57 of 100
Analysis rating
540
Head to head, this task
won 2 of 13

Why it is not sound

The survey

  • Presumed the stakeholder's view instead of testing it
  • Ignored the precision the brief asked for, which the sample cannot deliver
  • A leading question: Q6
  • A double-barrelled question: Q1

The analysis

  • Got at least one set number wrong
  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Missed a trap: Teams on 'Other' accounting software (base 17) mostly cite missing integration.
  • Missed a trap: Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
  • Missed a trap: Likelihood to pay is a hypothetical stated intention.
  • Missed a trap: The head of sales believes the trial is too long and teams forget about it.

Findings the judges found unsupported

  • Sage integration is the single biggest unmet need among lapsed trialists, especially for larger teams (50-249 employees) and Sage users (85.5% of Sage users need it).Sage integration is the largest listed need only in the unweighted sample. Size-weighted demand is approximately 18.1%, below per-company pricing (21.3%) and custom approval chains (20.4%). The reported weighted figure of 18.9% is also incorrect.
  • Lack of accounting integration is the top reason for not paying among Sage, FreeAgent, and 'Other' software users (58.1%, 33.3%, and 82.4% respectively).The finding presents the result for 'Other' software users confidently without warning that its base is only 17. That small-base result should not be treated as an established segment finding.
  • 61.3% of respondents whose main reason was 'no integration' would be likely to pay if Tallyroom connected to their accounting software.The 61.3% is a hypothetical stated likelihood, not evidence that these respondents would actually pay. The analysis subsequently treats it as actual willingness to pay and evidence of conversion impact without qualification.
  • Setup time is a major barrier, especially for small teams (1-9 employees: 27.4% cited 'didn't have time to set it up' as the main reason).The supplied open-text answers do not support the claimed 28 setup-time mentions or 15 mentions from 1–9 employee teams. The analysis also gives a conflicting setup-theme count of 26 elsewhere. The table-based setup finding itself is supported.
  • Per-company pricing (not per-user) is a need for 20.4% of respondents, with higher demand among QuickBooks users (26.1%) and larger teams (50-249: 20.0%).Demand for per-company pricing is not higher among larger teams: 20.0% for 50–249 employees is below both the overall 20.4% and the 22.6% for 1–9 employees.
  • Weighted to the population, ~15.2% of lapsed trials cited 'no integration' as the main reason, and ~18.9% would need Sage integration before paying.The weighted calculations are incorrect. Using the supplied population mix gives approximately 16.3% for missing integration and 18.1% for Sage integration demand, not 15.2% and 18.9%. Weighted setup time is approximately 21.6%, making it the leading main reason.

Scorecard

Numbers 10 of 11 right

QuestionIts answerTrue
What percentage of respondents said they would need a Sage integration before paying?✓23.7%23.7%
Weighted to the company-size mix of all lapsed trials, what percentage would need a Sage integration?✓18.9%18.2%
Among businesses with 1-9 employees, what percentage gave 'didn't have time to set it up' as the main reason?✓27.4%27.4%
How many of the open-text answers mention not having time to set Tallyroom up, or setup taking too long?✓2824
What is the 95% margin of error, in percentage points, on the unweighted share of respondents needing a Sage integration?✓3.9 points3.89 points
Is the difference in the share giving 'too expensive' as their main reason between businesses with 1-9 employees and those with 50-249 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the company-size mix of all 1,640 lapsed trials, about how many teams' main reason was not having time to set it up?✗69355
How many of the open-text answers from businesses with 1-9 employees mention not having time to set Tallyroom up, or setup taking too long?✓1514
Weighted to the company-size mix of all lapsed trials, what percentage would give 'no integration with our accounting software' as their main reason?✓15.2%16.3%
What percentage of the trial admins invited completed the survey?✓28.05%28%
Among respondents whose main reason was no integration with their accounting software, what percentage said they would be likely (fairly or very) to pay if Tallyroom connected to it?✓61.3%61.3%

Traps in the data 3 of 7 handled

  • ✗Teams on 'Other' accounting software (base 17) mostly cite missing integration.
  • ✗Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
  • ✓Q2 is multi-select: percentages are of respondents and add up to more than 100%.
  • ✗Likelihood to pay is a hypothetical stated intention.
  • ✗The head of sales believes the trial is too long and teams forget about it.
  • ✓460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
  • ✓Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.

The analysis

Recommendation

Prioritise Sage integration as the next quarter's product effort. The data shows it is the single biggest unmet need among lapsed trialists, especially for larger teams and Sage users, and 61.3% of those who cited 'no integration' as their main reason would pay if it were available. This addresses the highest-impact barrier to conversion. Assisted setup should be the secondary focus, as setup time is a major barrier for small teams (27.4% of 1-9 employee teams). A cheaper plan or shorter trial are lower priorities, as price is not the primary barrier (only 15.9% cited it as the main reason), and trial length is rarely mentioned (1.5% too short, 0.2% too long).

What would change it

Evidence that Sage integration would not drive a significant uplift in conversions (e.g., low actual adoption of Sage integration among paying customers, or low willingness to pay post-integration). Alternatively, evidence that assisted setup or per-company pricing would have a larger impact on conversion rates than Sage integration.

Findings

  1. Sage integration is the single biggest unmet need among lapsed trialists, especially for larger teams (50-249 employees) and Sage users (85.5% of Sage users need it).

    High confidenceBase 460

    23.7% of all respondents need Sage integration (Q3). Among Sage users, 85.5% need it. Among 50-249 employee teams, 40.9% need it. Weighted to population, 18.9% need it.

  2. Lack of accounting integration is the top reason for not paying among Sage, FreeAgent, and 'Other' software users (58.1%, 33.3%, and 82.4% respectively).

    High confidenceBase 460

    Q2: 20.2% of all respondents cited 'no integration' as the main reason. For Sage users, this jumps to 58.1%. For FreeAgent, 33.3%. For 'Other', 82.4%.

  3. 61.3% of respondents whose main reason was 'no integration' would be likely to pay if Tallyroom connected to their accounting software.

    High confidenceBase 93

    Q5: Among the 93 respondents who cited 'no integration' as their main reason, 61.3% (net likely) said they would pay £6/user/month if integration existed.

  4. Setup time is a major barrier, especially for small teams (1-9 employees: 27.4% cited 'didn't have time to set it up' as the main reason).

    High confidenceBase 460

    Q2: 18.7% of all respondents cited setup time as the main reason. For 1-9 employee teams, this is 27.4%. Open-text answers show 28 mentions of setup time issues, with 15 from 1-9 employee teams.

  5. Price is a significant barrier, but not the primary one. 15.9% of all respondents cited 'too expensive' as the main reason, with no statistically significant difference between small (1-9) and larger (50-249) teams.

    Medium confidenceBase 460

    Q2: 15.9% overall. 1-9 employees: 19.0%, 50-249 employees: 15.5%. Margin of error for 1-9 employees: ~6.5%, for 50-249: ~8.5%. Overlap in confidence intervals means no significant difference.

  6. Per-company pricing (not per-user) is a need for 20.4% of respondents, with higher demand among QuickBooks users (26.1%) and larger teams (50-249: 20.0%).

    Medium confidenceBase 460

    Q3: 20.4% overall need per-company pricing. QuickBooks users: 26.1%. 50-249 employees: 20.0%.

  7. Only 1.5% of respondents cited 'the trial was too short' as the main reason for not paying, and 0.2% cited 'the trial was too long'.

    High confidenceBase 460

    Q2: 1.5% too short, 0.2% too long. Open-text answers show no mentions of trial length as a barrier.

  8. Weighted to the population, ~15.2% of lapsed trials cited 'no integration' as the main reason, and ~18.9% would need Sage integration before paying.

    High confidenceBase 460

    Weighted to company-size mix (55% 1-9, 35% 10-49, 10% 50-249): 'no integration' = 15.2%, Sage need = 18.9%.

  9. Among Sage users, 58.1% cited 'no integration' as the main reason, and 85.5% would need Sage integration before paying.

    High confidenceBase 117

    Q2 and Q3 for Sage users (base=117): 58.1% main reason = no integration, 85.5% need Sage integration.

Caveats

  • The survey only includes lapsed trialists (those who did not convert), so it does not capture reasons for conversion or satisfaction among paying customers.
  • Open-text answers are from a subset of respondents (109 out of 460), so themes may not be fully representative.
  • The survey was incentivised (£10 voucher), which may have attracted respondents with stronger opinions or more time to complete it.
  • The population of lapsed trials is skewed toward smaller businesses (55% 1-9 employees), which may limit generalisability to larger teams.
  • The margin of error for smaller segments (e.g., Sage users, FreeAgent users) is higher, reducing confidence in precise percentages for those groups.
  • The 'no integration' reason is conflated with 'no need' in some open-text answers (e.g., teams that don't need an expenses tool may also lack integration).
  • The trial length question (Q2) had very low responses for 'too short' (1.5%) and 'too long' (0.2%), suggesting it is not a primary barrier.

Next steps

  • decisionBuild and launch Sage integration as the top priority for next quarter.
  • experimentRun an experiment with assisted setup (e.g., onboarding calls, pre-configured templates) for a subset of new trialists.
  • researchConduct follow-up interviews with 20-30 Sage users who lapsed due to 'no integration' to validate their willingness to pay post-integration.
  • monitorMonitor the impact of Sage integration on conversion rates and gather feedback from new Sage users.
  • researchExplore per-company pricing as a future option, but deprioritise for next quarter.

Open-text themes it coded

No integration with accounting software 23Setup time or complexity 26Price or per-user pricing 25Approval workflow mismatch 17Mobile app or receipt scanning issues 11No need for an expenses tool 19Chose another tool 7

The survey it planned

2 screening questions and 9 questions, as the model wrote them.

  1. S1

    Are you the admin of a UK-based team that started a Tallyroom trial in the last 90 days but did not go on to pay for the service?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    How many employees does your company have?

    One answer
    • 1-9
    • 10-49
    • 50-249
    • 250 or more

    Continues if 1-9; 10-49; 50-249

  3. Q1

    What was the main reason your team did not continue with Tallyroom after the trial?

    One answer
    • It was too expensive
    • We didn’t need all the features
    • It didn’t integrate with our accounting software
    • It was too complicated to set up or use
    • We didn’t have time to evaluate it properly
    • We forgot about the trial
    • We found a better alternative
    • Other (please specify)
    Judges: double-barrelled

    The setup-or-use option combines onboarding difficulty and ongoing usability in one answer, so these distinct barriers cannot be distinguished.

    'It was too complicated to set up or use' asks about two different stages (setup vs. ongoing use).

  4. Q2

    Which of the following accounting software do you currently use? (Select all that apply)

    Any that apply
    • Xero
    • QuickBooks
    • Sage
    • FreeAgent
    • FreshBooks
    • Zoho Books
    • Other (please specify)
    • None
  5. Q3

    If Tallyroom integrated with your accounting software, how likely would you be to try it again?

    Scale

    Scale 1-5: Not at all likely to Very likely

    Routing Shown only if Q2 is: Sage; FreeAgent; FreshBooks; Zoho Books; Other (please specify); None

    Judges: assumptive

    It is shown to respondents selecting 'None' at Q2, but assumes they have accounting software with which Tallyroom could integrate.

    It is shown to respondents who chose 'None' or 'Other', yet it refers to 'your accounting software', assuming they have software that could be integrated.

  6. Q4

    What additional features would make Tallyroom more valuable to your team? (Select up to 3)

    Any that apply
    • Assisted setup (e.g., guided onboarding)
    • A cheaper plan (e.g., fewer features or users)
    • More accounting integrations
    • Mobile app improvements
    • Better reporting
    • Automated expense categorisation
    • Other (please specify)
  7. Q5

    How satisfied were you with the ease of setting up Tallyroom during the trial?

    Scale

    Scale 1-5: Very dissatisfied to Very satisfied

  8. Q6

    Would a shorter trial period (e.g., 14 days instead of 30) have made you more likely to evaluate Tallyroom sooner?

    One answer
    • Yes, much more likely
    • Yes, somewhat more likely
    • No, it wouldn’t have made a difference
    • No, I would have been less likely to evaluate it
    Judges: leading

    The options distinguish two degrees of increased likelihood but only one degree of decreased likelihood, giving the proposed shorter trial an unbalanced positive response scale.

    It frames a shorter trial only as a way to 'evaluate sooner', which steers towards the head of sales' view. It also assumes the respondent delayed their evaluation.

  9. Q7

    How many people in your team used Tallyroom during the trial?

    Number
  10. Q8

    What is your role in the company?

    One answer
    • Owner/Founder
    • Finance/Accounting Manager
    • Office Manager
    • HR Manager
    • Other (please specify)
  11. Q9

    To ensure we’re getting accurate feedback, please select 'Strongly agree' for this question.

    Scale

    Scale 1-5: Strongly disagree to Strongly agree

Sample plan

Invite the admin of each UK team whose Tallyroom trial ended in the last 90 days without paying (approx. 1,900 teams per quarter). Aim for 450 completes, with quotas to ensure proportional representation across company-size bands (1-9, 10-49, 50-249 employees). The £10 voucher incentive will be offered to boost response rates.