Benchmarks / SurveyBench / Why Tallyroom trials don't convert / Mercury 3

Measured by Spring Prompt

Mercury 3: Why Tallyroom trials don't convert

The decision: where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →

Verdict
✗ Not sound
Research score
51 of 100
Analysis rating
752
Head to head, this task
won 4 of 13

Why it is not sound

The survey

  • Presumed the stakeholder's view instead of testing it
  • A double-barrelled question: Q6

The analysis

  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Missed a trap: Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
  • Missed a trap: The head of sales believes the trial is too long and teams forget about it.
  • Missed a trap: 460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
  • Missed a trap: Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.

Findings the judges found unsupported

  • Sage integration is the strongest product-led opportunity: it is the most frequently selected missing feature overall and dominates among Sage users.Sage leads the unweighted feature selections, but calling it the strongest product-led opportunity across lapsed trials is not established. After company-size weighting, custom approval chains are selected by about 20.4%, versus 18.1% for Sage integration. Feature requests alone also do not establish which investment would lift conversion most.
  • Integration has a credible conversion payoff, but it will not convert everyone: the integration-blocked group is fairly likely to pay if the connection exists.The 61.3% answered a hypothetical question about connection to their own accounting software, not specifically Sage. The 93 integration-blocked respondents include FreeAgent and Other software users, so this percentage cannot be attributed to a Sage integration.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents said they would need a Sage integration before paying?✓23.7%23.7%
Weighted to the company-size mix of all lapsed trials, what percentage would need a Sage integration?✓18.1%18.2%
Among businesses with 1-9 employees, what percentage gave 'didn't have time to set it up' as the main reason?✓27.4%27.4%
How many of the open-text answers mention not having time to set Tallyroom up, or setup taking too long?✓2424
What is the 95% margin of error, in percentage points, on the unweighted share of respondents needing a Sage integration?✓3.9 points3.89 points
Is the difference in the share giving 'too expensive' as their main reason between businesses with 1-9 employees and those with 50-249 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the company-size mix of all 1,640 lapsed trials, about how many teams' main reason was not having time to set it up?✓355355
How many of the open-text answers from businesses with 1-9 employees mention not having time to set Tallyroom up, or setup taking too long?✓1414
Weighted to the company-size mix of all lapsed trials, what percentage would give 'no integration with our accounting software' as their main reason?✓16.3%16.3%
What percentage of the trial admins invited completed the survey?✓28%28%
Among respondents whose main reason was no integration with their accounting software, what percentage said they would be likely (fairly or very) to pay if Tallyroom connected to it?✓61.3%61.3%

Traps in the data 3 of 7 handled

  • ✓Teams on 'Other' accounting software (base 17) mostly cite missing integration.
  • ✗Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
  • ✓Q2 is multi-select: percentages are of respondents and add up to more than 100%.
  • ✓Likelihood to pay is a hypothetical stated intention.
  • ✗The head of sales believes the trial is too long and teams forget about it.
  • ✗460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
  • ✗Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.

The analysis

Recommendation

Findings

  1. Sage integration is the strongest product-led opportunity: it is the most frequently selected missing feature overall and dominates among Sage users.

    High confidenceBase 460

    Q3: 23.7% (109/460) of lapsed trials said they would need a Sage integration before paying; among the 117 Sage users, 85.5% selected it. Q2: 58.1% of Sage users cited no accounting-software integration as the main reason for not paying. Weighted to the lapsed-trial company-size mix, 18.1% (~297/1,640) would need Sage integration.

  2. Integration has a credible conversion payoff, but it will not convert everyone: the integration-blocked group is fairly likely to pay if the connection exists.

    Medium confidenceBase 93

    Among the 93 respondents whose main reason was no integration, 61.3% said they would be fairly or very likely to pay if the Sage integration existed.

The survey it planned

3 screening questions and 10 questions, as the model wrote them.

  1. S1

    Are you the person responsible for managing your company's Tallyroom account or trial?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    How many employees does your company have in the UK?

    One answer
    • 1-9
    • 10-49
    • 50-249
    • 250 or more

    Continues if 1-9; 10-49; 50-249

  3. S3

    Did your company's Tallyroom free trial end within the last 90 days without moving to a paid plan?

    One answer
    • Yes
    • No

    Continues if Yes

  4. Q1

    Which accounting software does your company currently use?

    One answer
    • Xero
    • QuickBooks
    • Sage
    • Other (please specify)
    • We don't use accounting software
  5. Q2

    What was the main reason your team did not continue with Tallyroom after the free trial?

    One answer
    • It cost more than we expected or wanted to pay
    • It did not integrate with our accounting software
    • Setting it up took too long or was too complicated
    • Our team did not use the app enough
    • Managers found approving expenses too time-consuming
    • Key features were missing
    • We chose a different expenses product
    • Expenses were not a priority right now
    • The trial length did not suit us
    • Other (please specify)
  6. Q3

    Which of the following missing features or integrations, if any, contributed to your decision not to subscribe? Select all that apply.

    Any that apply
    • Integration with Sage
    • Integration with another accounting package
    • Automatic receipt reading/OCR
    • Multi-currency expenses
    • Department or project codes
    • Expense report generation
    • Payroll or finance export
    • A better mobile app
    • More flexible approval rules
    • Better reports/analytics
    • None - missing features/integrations were not an issue
    • Other (please specify)
  7. Q4

    How easy or difficult was it to set up Tallyroom?

    Scale

    Scale 1-5: Very difficult to Very easy

    Judges: assumptive

    Assumes respondents experienced setup sufficiently to rate it. Eligible admins may never have attempted setup, and no not-applicable response is available.

    It assumes the respondent actually attempted to set up Tallyroom, which may not be true for all lapsed trials.

  8. Q5

    During the 30-day trial, how often did people in your team use Tallyroom?

    One answer
    • Daily
    • Several times a week
    • About once a week
    • Once or twice in total
    • Never/almost never
  9. Q6

    Which statement best describes your view of the 30-day trial length?

    One answer
    • It was too long; our team lost interest before the end
    • It was about right
    • It was too short; we needed more time to evaluate it
    • The length did not matter; we just did not use it enough
    Judges: double-barrelled

    Options bundle trial-length judgments with explanations, particularly 'too long' with losing interest. Respondents cannot endorse the length judgment independently of that explanation. The wording embeds the stakeholder's proposed mechanism, while the length-did-not-matter option is restricted to insufficient use.

    Each option ties a view of the trial length to a cause. 'Too long' comes bundled with 'our team lost interest', 'too short' with 'we needed more time to evaluate', and 'did not matter' with 'we just did not use it enough'. Someone who agrees with only one half of an option cannot answer accurately.

  10. Q7

    How does the price of £6 per user per month compare with what you expected to pay for an expenses app?

    One answer
    • Much lower than expected
    • Somewhat lower than expected
    • About what I expected
    • Somewhat higher than expected
    • Much higher than expected
    Judges: assumptive

    Assumes respondents had an expected price for an expenses app; those without a prior expectation cannot give a valid answer using the available options.

    Assumes the respondent had a prior expectation of what they would pay for an expenses app.

  11. Q8

    If Tallyroom had offered free assisted setup (with a Tallyroom specialist configuring your account), would your team have been more likely to subscribe?

    One answer
    • Yes, definitely
    • Probably yes
    • No more or less likely
    • Probably not
    • Definitely not
  12. Q9

    Would a cheaper plan (for example, a lower price with fewer features) have made your team more likely to subscribe?

    One answer
    • Yes, definitely
    • Probably yes
    • No more or less likely
    • Probably not
    • Definitely not
  13. Q10

    What is the one change that would have made your team most likely to pay for Tallyroom?

    Open text
Sample plan

Invite the admin contact in the product database for every UK team whose Tallyroom trial lapsed in the last 90 days. Send email invitations with the £10 voucher incentive. Stratify by company-size band using S2 and continue inviting until quotas are complete: 267 for 1-9 employees, 267 for 10-49 employees, and 267 for 50-249 employees (801 total). This is required to achieve the specified ±6 percentage point margin of error at 95% confidence within each band. The stated target of about 450 completes is not sufficient for that per-band precision; 150 per band would give roughly ±8 percentage points. If the 450-complete cap is unavoidable, the results should be treated as directional or the precision requirement should be relaxed.