Benchmarks / SurveyBench / Why Tallyroom trials don't convert / DeepSeek V4.1 Flash 1017

Measured by Spring Prompt

DeepSeek V4.1 Flash 1017: Why Tallyroom trials don't convert

The decision: where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →

Verdict
✗ Not sound
Research score
78 of 100
Analysis rating
1,221
Head to head, this task
won 6 of 14

Why it is not sound

The survey

  • A research objective no question serves

The analysis

  • Reported a material finding the data does not support
  • Made a recommendation that does not follow from the data
  • Missed a trap: Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.

Findings the judges found unsupported

  • Sage integration is the clearest product gap and should lead next quarter's product effort.Sage is a clear product gap, but the evidence does not establish that it should lead next quarter's effort. Size-weighted main reasons put setup first at about 21.6%, versus 16.3% for missing integration. Weighted Sage demand supports further validation, not a confident conclusion that building it is the best conversion investment.
  • Integration-blocked prospects are the most likely to pay, so fixing the right integration is the best conversion lever.Integration-blocked respondents have the highest stated purchase intention among the groups shown, but this does not establish that integration is the best conversion lever. Hypothetical intentions do not measure the incremental conversion lift from competing interventions.
  • Assisted setup is the broad second lever, especially for small firms, but it has weaker stated willingness to pay than integration.Assisted setup is incorrectly ranked second despite setup being the most common size-weighted main reason. It also misdescribes Q5: the 27.9% answered a hypothetical about accounting integration, not whether they would pay if they got past setup. That percentage cannot estimate willingness to pay following assisted setup.
  • Cheaper or per-company pricing is a real but lower-yield lever; do not lead with it.The survey establishes a pricing barrier, but not that changing pricing is a lower-yield intervention. Price-blocked respondents' willingness to pay the unchanged £6 price after an integration does not measure their response to a cheaper or per-company plan.

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents said they would need a Sage integration before paying?✓23.7%23.7%
Weighted to the company-size mix of all lapsed trials, what percentage would need a Sage integration?✓18.1%18.2%
Among businesses with 1-9 employees, what percentage gave 'didn't have time to set it up' as the main reason?✓27.4%27.4%
How many of the open-text answers mention not having time to set Tallyroom up, or setup taking too long?✓2424
What is the 95% margin of error, in percentage points, on the unweighted share of respondents needing a Sage integration?✓3.9 points3.89 points
Is the difference in the share giving 'too expensive' as their main reason between businesses with 1-9 employees and those with 50-249 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the company-size mix of all 1,640 lapsed trials, about how many teams' main reason was not having time to set it up?✓355355
How many of the open-text answers from businesses with 1-9 employees mention not having time to set Tallyroom up, or setup taking too long?✓1414
Weighted to the company-size mix of all lapsed trials, what percentage would give 'no integration with our accounting software' as their main reason?✓16.3%16.3%
What percentage of the trial admins invited completed the survey?✓28%28%
Among respondents whose main reason was no integration with their accounting software, what percentage said they would be likely (fairly or very) to pay if Tallyroom connected to it?✓61.3%61.3%

Traps in the data 6 of 7 handled

  • ✓Teams on 'Other' accounting software (base 17) mostly cite missing integration.
  • ✗Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
  • ✓Q2 is multi-select: percentages are of respondents and add up to more than 100%.
  • ✓Likelihood to pay is a hypothetical stated intention.
  • ✓The head of sales believes the trial is too long and teams forget about it.
  • ✓460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
  • ✓Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.

The analysis

Recommendation

Put next quarter's primary product effort into a Sage integration, because Sage users are a large segment (25.4%), 85.5% of them need the integration, 58.1% give missing integration as the main reason, and integration-blocked prospects show the highest stated willingness to pay (61.3% likely). Pair this with a secondary assisted-setup programme for small firms: setup time is the second-largest barrier overall and the largest among 1-9 employee firms. Do not cut the trial to 14 days; the evidence does not support it and it risks worsening setup completion. Do not lead with a cheaper plan; test per-company pricing later because it is a real but lower-yield lever. Sales should target Sage, FreeAgent and unsupported-accounting users with the integration roadmap, while qualifying out teams with very low expense volume.

What would change it

A powered conversion experiment showing assisted setup produces a larger trial-to-paid lift than Sage integration would change the priority. So would evidence that the Sage segment is much smaller in the weighted population than the unweighted tables imply, that Sage users' actual willingness to pay collapses at £6/user/month, or that building the Sage integration is disproportionately expensive. A strong per-company pricing test that converts the price-blocked segment at high rates would also move pricing up the list.

Findings

  1. Do not cut the trial to 14 days: the data point to setup time, not trial length, as the barrier.

    High confidenceBase 460

    Only 0.2% chose 'trial was too long', 3.7% forgot the trial was running and 1.5% said it was too short. Meanwhile 18.7% overall and 27.4% of 1-9 employee firms chose 'didn't have time to set it up'; 24 of 109 open-text answers mention setup time or setup taking too long.

  2. Sage integration is the clearest product gap and should lead next quarter's product effort.

    High confidenceBase 460

    25.4% of respondents use Sage; 85.5% of Sage users say they would need a Sage integration before paying, and 58.1% of Sage users give 'no integration with our accounting software' as the main reason they did not pay. Weighted to the lapsed-trial size mix, 18.1% would need Sage integration.

  3. Integration-blocked prospects are the most likely to pay, so fixing the right integration is the best conversion lever.

    Medium confidenceBase 93

    Among respondents whose main reason was no integration, 61.3% said they would be fairly or very likely to pay £6/user/month if Tallyroom connected to their accounting software (35.5% fairly likely + 25.8% very likely), versus 37.6% overall and 23.3% among the price-blocked.

  4. Assisted setup is the broad second lever, especially for small firms, but it has weaker stated willingness to pay than integration.

    Medium confidenceBase 460

    18.7% overall give 'didn't have time to set it up' as the main reason; weighted to the 1,640 lapsed trials this is 21.6%, about 355 teams. It is 27.4% among 1-9 employee firms. Open text has 24 setup-time mentions, 14 from 1-9 employee firms. But only 27.9% of setup-blocked respondents are likely to pay if they got past setup, versus 37.6% overall.

  5. Cheaper or per-company pricing is a real but lower-yield lever; do not lead with it.

    High confidenceBase 460

    15.9% give 'too expensive' as the main reason and 20.4% say per-company pricing is needed before paying. But only 23.3% of the price-blocked are likely to pay £6/user/month. The 'too expensive' difference between 1-9 employees (19.0%) and 50-249 employees (15.5%) is not statistically significant at 95%.

  6. Existing Xero and QuickBooks integrations are not the problem; missing integrations are concentrated in Sage, FreeAgent and other systems.

    High confidenceBase 460

    0.0% of Xero users and 0.0% of QuickBooks users cite no integration as the main reason. By contrast, 58.1% of Sage users, 33.3% of FreeAgent users and 82.4% of Other-software users do.

  7. Approval workflow is a meaningful larger-firm blocker and a credible secondary product opportunity.

    Medium confidenceBase 460

    12.6% overall give 'approval workflow didn't fit' as the main reason, rising to 21.8% among 50-249 employee firms. 22.0% overall and 27.3% of 50-249 employee firms say they would need custom approval chains before paying.

  8. A substantial group simply does not need an expenses app; sales qualification and onboarding must filter for real expense volume.

    High confidenceBase 460

    10.0% say they decided they didn't need it, 36.5% select none of the Q3 needs, and 12 open-text answers describe no need, overkill, low claim volume or using a company card instead.

  9. Mobile receipt capture and OCR are a minority but trust-damaging blocker and should be fixed alongside integration work.

    Medium confidenceBase 460

    6.3% give mobile app problems as the main reason; 15.0% say better receipt scanning is needed before paying; 10 open-text answers describe upload failures, blurry/cropped photos, OCR misreads or app freezing.

  10. Unweighted totals over-represent larger firms, so decision-makers should use weighted estimates for population-level conclusions.

    High confidenceBase 460

    Survey completes are 36.5% 1-9 employees, 39.6% 10-49 and 23.9% 50-249, versus the lapsed-trial population of 55%, 35% and 10%. Completion was 460 of 1,640, or 28.0%.

Caveats

  • This survey covers only lapsed trials, so it explains non-conversion but cannot show what would have converted or how many current trialists would convert under a changed product.
  • Completion is 28.0% (460/1,640); non-response bias may be substantial, and the £10 voucher may have attracted a particular type of admin.
  • The sample over-represents larger firms: 50-249 employees are 23.9% of completes but 10% of lapsed trials; 1-9 employees are 36.5% of completes but 55% of the population. Unweighted totals should not be treated as population estimates.
  • Q5 is stated purchase intent for a hypothetical £6/user/month price after integration, not observed payment. It should be treated as directional, especially for small subgroups such as the 93 respondents whose main reason was no integration.
  • Q2 forces a single main reason, but barriers overlap: price, setup, integration and approval workflow are not mutually exclusive. Q3 shows substantial overlap in what teams say they need.
  • Open-text answers are available for only 109 of 460 respondents; counts are illustrative of themes, not representative prevalence estimates.
  • Subgroup bases are small for FreeAgent (33) and Other accounting software (17), and Q5 subclass bases are modest; subgroup percentages should be read with wide uncertainty.
  • The 95% margin of error on the unweighted 23.7% Sage-integration need is about 3.9 percentage points under a simple-random-sample assumption; with finite-population correction it is about 3.3 points, but non-response bias is likely more important than sampling error.

Next steps

  • decisionApprove a Sage integration discovery and build plan for next quarter.
  • experimentRun an assisted-setup experiment for small firms inside the 30-day trial: guided import, done-with-you setup, and an onboarding checklist.
  • experimentKeep the 30-day trial and test any shorter-trial variant only as a small controlled experiment, not as a rollout.
  • experimentTest per-company pricing or a hybrid seat model with a controlled price test before changing the plan.
  • researchInterview Sage, FreeAgent and unsupported-software lapsed admins, plus their accountants, to validate integration requirements and the accountant referral route.
  • monitorBuild a monthly trial-cohort dashboard by accounting software, company size, setup completion and conversion.

Open-text themes it coded

Missing accounting integration - Sage specifically 15Missing accounting integration - other systems / FreeAgent / KashFlow / Clear Books / unsupported 7Setup time / trial too short to complete setup 24Price / per-user cost too high 24Approval workflow did not fit 15No need / overkill / low expense volume 12Mobile app / receipt scanning / OCR reliability 10Chose competitor / accountant recommendation 8

The survey it planned

5 screening questions and 10 questions, as the model wrote them.

  1. S1

    Are you the person who set up or administers Tallyroom for your organisation?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  2. S2

    Where is your organisation based?

    One answer
    • UK-based or primarily UK-based
    • Outside the UK
    • Not sure

    Continues if UK-based or primarily UK-based

  3. S3

    When did your organisation's Tallyroom trial end?

    One answer
    • In the last 30 days
    • 31–90 days ago
    • 91–180 days ago
    • More than 180 days ago
    • We are still in a trial
    • We never had a Tallyroom trial
    • Not sure

    Continues if In the last 30 days; 31–90 days ago

  4. S4

    Did your organisation start paying for Tallyroom after the trial?

    One answer
    • Yes, we pay for Tallyroom now
    • No, we did not start paying
    • Not sure

    Continues if No, we did not start paying

  5. S5

    How many employees does your organisation have in total?

    One answer
    • 1–9
    • 10–49
    • 50–249
    • 250 or more
    • Not sure

    Continues if 1–9; 10–49; 50–249

  6. Q1

    Thinking about why your organisation did not start paying for Tallyroom after the trial, which ONE was the main reason?

    One answer
    • The price was too high for the value we got
    • It did not connect to the accounting software we use
    • Setting up or importing data took too much time or support
    • Staff did not use it enough / low adoption
    • The trial was too short / we ran out of time
    • The trial was too long / we lost momentum
    • We chose another product
    • Our needs changed / we no longer needed it
    • It was missing a feature other than accounting integration
    • Another reason
    • Not sure / I was not involved enough in the decision
  7. Q2

    Which of these, if any, were reasons your organisation did not start paying? Select all that apply. Do not select 'None of these' with other reasons.

    Any that apply
    • The price was too high for the value we got
    • It did not connect to the accounting software we use
    • Setting up or importing data took too much time or support
    • Staff did not use it enough / low adoption
    • The trial was too short / we ran out of time
    • The trial was too long / we lost momentum
    • We chose another product
    • Our needs changed / we no longer needed it
    • It was missing a feature other than accounting integration
    • Another reason
    • None of these
    • Not sure / I was not involved enough in the decision
  8. Q3

    Please rank these possible changes from 1 (most likely to have made your organisation start paying) to 5 (least likely).

    Ranking
    • A lower price than £6 per user per month
    • Assisted setup and data import
    • A connection to our accounting software
    • A shorter trial (for example, 14 days instead of 30)
    • A longer trial than 30 days
  9. Q4

    Which accounting software does your organisation mainly use for bookkeeping or expenses?

    One answer
    • Xero
    • QuickBooks (including QuickBooks Online)
    • Sage
    • FreeAgent
    • Microsoft Dynamics 365 Business Central
    • Oracle NetSuite
    • SAP Business One
    • Excel / spreadsheets
    • Another accounting software
    • None
    • Not sure
  10. Q5

    Apart from accounting software connections, which Tallyroom features, if any, were missing or not good enough for your organisation? Select all that apply.

    Any that apply
    • Mobile receipt capture
    • Manager approval workflow
    • VAT or tax handling
    • Company card or bank feed reconciliation
    • Project or client coding
    • Multi-currency
    • User roles and permissions
    • Reporting and exports
    • Something else
    • Nothing was missing or not good enough
    • Not sure
  11. Q6

    How difficult or easy was it for your organisation to set up Tallyroom during the trial?

    Scale

    Scale 1-5: Very difficult to Very easy

    Judges: assumptive

    Assumes the organisation attempted setup and the respondent can assess its difficulty. The screeners do not establish this, and the scale has no not-attempted or don't-know response.

    It assumes the organisation attempted to set up Tallyroom during the trial. Many lapsed trials may never have started setup, and there is no 'did not set up' or not-applicable option.

  12. Q7

    How much do you agree or disagree with this statement: 'The 30-day trial gave us enough time to evaluate Tallyroom properly.'

    Scale

    Scale 1-5: Strongly disagree to Strongly agree

  13. Q8

    When, if at all, did your organisation decide not to pay for Tallyroom?

    One answer
    • Before starting the trial
    • During the first week of the trial
    • In weeks 2–3 of the trial
    • In week 4 or later, but before the trial ended
    • After the trial ended
    • We never made a formal decision
    • Not sure
  14. Q9

    If Tallyroom had cost less per user per month, what is the highest monthly price per user your organisation would have been willing to pay after the trial?

    One answer
    • Less than £3
    • £3 to £4.99
    • £5 to £5.99
    • £6 (the price at the time)
    • More than £6
    • No price would have made us pay
    • Not sure
  15. Q10

    What is the one thing Tallyroom could have done differently that would most likely have led your organisation to pay after the trial?

    Open text
Sample plan

Invite every admin in the product database for UK teams whose Tallyroom trial ended 0–90 days ago and did not convert, after de-duplicating teams. Send an email invitation to the admin, link to the online questionnaire, offer a £10 voucher on completion, and send two reminders. The requested ~450 completes is not sufficient for ±6 percentage points at 95% confidence within each of the three employee bands: that requires about 267 completes per band, i.e. about 801 completes in total. Set quotas of 267 completes in each of 1–9, 10–49 and 50–249 employees; exclude 250+ and not sure. Based on 1,900 trials per quarter and 18% conversion, about 1,558 lapsed trials are available per quarter, so 801 completes requires roughly a 51% response rate; invite all eligible admins and monitor response by band, with extra reminders or phone follow-up if a band lags. If the 450-complete cap is fixed and cannot be changed, the maximum feasible quota is about 150 per band, which gives ±8 percentage points per band at 95% confidence, not ±6; product stakeholders must either accept that relaxation or fund the larger sample.