Benchmarks / SurveyBench / Why Tallyroom trials don't convert / Muse Ember

Measured by Spring Prompt

Muse Ember: Why Tallyroom trials don't convert

The decision: where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →

Verdict
✗ Not sound
Research score
93 of 100
Analysis rating
1,392
Head to head, this task
won 6 of 13

Why it is not sound

The survey

  • A double-barrelled question: Q10

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents said they would need a Sage integration before paying?✓23.7%23.7%
Weighted to the company-size mix of all lapsed trials, what percentage would need a Sage integration?✓18.1%18.2%
Among businesses with 1-9 employees, what percentage gave 'didn't have time to set it up' as the main reason?✓27.4%27.4%
How many of the open-text answers mention not having time to set Tallyroom up, or setup taking too long?✓2424
What is the 95% margin of error, in percentage points, on the unweighted share of respondents needing a Sage integration?✓3.9 points3.89 points
Is the difference in the share giving 'too expensive' as their main reason between businesses with 1-9 employees and those with 50-249 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the company-size mix of all 1,640 lapsed trials, about how many teams' main reason was not having time to set it up?✓355355
How many of the open-text answers from businesses with 1-9 employees mention not having time to set Tallyroom up, or setup taking too long?✓1514
Weighted to the company-size mix of all lapsed trials, what percentage would give 'no integration with our accounting software' as their main reason?✓16.3%16.3%
What percentage of the trial admins invited completed the survey?✓28%28%
Among respondents whose main reason was no integration with their accounting software, what percentage said they would be likely (fairly or very) to pay if Tallyroom connected to it?✓61.3%61.3%

Traps in the data 7 of 7 handled

  • ✓Teams on 'Other' accounting software (base 17) mostly cite missing integration.
  • ✓Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
  • ✓Q2 is multi-select: percentages are of respondents and add up to more than 100%.
  • ✓Likelihood to pay is a hypothetical stated intention.
  • ✓The head of sales believes the trial is too long and teams forget about it.
  • ✓460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
  • ✓Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.

The analysis

Recommendation

For next quarter put main product and sales effort into assisted setup for new trials - done-for-you configuration of users, expense categories and approval templates within the first days, targeted at 1-9 and 10-49 firms - and keep the 30-day trial, adding early human nudges instead of cutting it to 14 days. Do not make a full Sage build the main quarter bet: Sage need is strong in Sage users (85.5% of 117) but only 18.1% weighted overall, integration is 16.3% weighted as main reason, and only 61.3% of that group say they would pay £6. Treat Sage as a validated second-track discovery/roadmap item, and test per-company pricing for occasional users and approval-workflow templates (MD-only, batch, multi-approver, external accountant) inside the assisted-setup pilot, because price (16.6% weighted) and approval fit (12.6% main reason, 22.0% need custom chains) are close behind setup and are not solved by integration or a shorter trial.

What would change it

Evidence that incoming (not lapsed) trials are much more Sage-heavy than this sample, that Sage-blocked deals are losing paid pipeline at high rate in CRM/accountant interviews, or that an assisted-setup pilot does not improve setup completion and trial-to-paid, would shift priority to building Sage first. Evidence from a price-structure test that flat per-company pricing lifts conversion enough to offset per-user revenue loss, without setup help, would elevate the cheaper-plan option. Conversely, a pilot showing setup help mainly helps teams who then cite price or no-need would argue for combining setup with pricing simplification.

Findings

  1. Assisted setup, not a shorter trial, is the largest overall lever: not having time to set up is the top weighted main reason for non-payment and is concentrated in the small businesses that dominate the lapsed population.

    High confidenceBase 460

    Unweighted, 18.7% cited didn't have time to set it up. Weighted to population size mix (55% 1-9, 35% 10-49, 10% 50-249) this is 21.6% (0.55x27.4 + 0.35x15.9 + 0.10x10.0), about 355 of 1,640 lapsed teams. Among 1-9 firms it is 27.4% (base 168) vs 10.0% for 50-249. 24 of 109 open-text answers mention setup time, 15 of them from 1-9 firms.

  2. Cutting the trial from 30 to 14 days would very likely reduce conversion, not lift it. There is essentially no evidence that trial length or forgetting caused non-payment.

    High confidenceBase 460

    Q2: trial too long 0.2% (1 respondent), trial too short 1.5%, forgot trial was running 3.7% (base 460). Open texts repeatedly say the trial ended before setup was finished, e.g. setup took longer than the trial allowed, ran out of trial before finishing setup. Shortening would remove the time respondents say they lacked.

  3. Head of product question - Sage demand is real and strong inside the Sage segment, but narrower overall than the unweighted headline suggests because larger businesses are over-represented in the sample.

    Medium confidenceBase 460

    23.7% of all respondents say they need Sage integration before paying (base 460, 95% margin of error about 3.9pp). Weighted to population size mix it falls to 18.1% (0.55x7.7 + 0.35x28.0 + 0.10x40.9). Inside Sage users (base 117), 85.5% need Sage and 58.1% cite no integration as main reason, vs 3.3% of Xero and 0.9% of QuickBooks users needing Sage. FreeAgent shows the same pattern on a smaller base (72.7% of 33 users, 7.2% overall).

  4. Finance director question - connecting to their software would not convert most integration-blocked teams at £6 per user per month on stated intent, before any intention-behaviour gap.

    Medium confidenceBase 93

    Among the 93 respondents whose main reason was no integration, 61.3% said fairly/very likely to pay £6 if connected (Q5 net likely). That is 38.7% unlikely/neutral. Applied to the weighted 16.3% citing no integration as main reason, stated likely payers are only about 10% of all lapsed teams. Q5 was hypothetical, asked of everyone using accounting software including Xero/QuickBooks users already connected, and anchored at £6.

  5. Price matters, mainly as per-user structure for occasional users rather than a proven size-based price gap, and a blanket cheaper plan is not evidenced as the fix.

    Medium confidenceBase 460

    Too expensive is main reason for 15.9% unweighted, 16.6% weighted (base 460). 20.4% say they need per-company pricing in Q3, and many open texts say paying for every seat when only a few claim was the blocker. The 1-9 (19.0%, base 168) vs 50-249 (15.5%, base 110) too-expensive gap is 3.5pp and not statistically significant at 95% (z about 0.75). Among too-expensive citers only 23.3% said likely to pay £6 if integrated (base 73), indicating integration alone does not resolve price objection.

  6. Approval workflow is a material secondary barrier that assisted setup should address, especially for larger teams, and it is not solved by any of the four headline options alone.

    Medium confidenceBase 460

    12.6% cite approval workflow didn't fit as main reason (21.8% for 50-249 vs 7.1% for 1-9), 22.0% say they need custom approval chains in Q3, and 15 open texts describe MD-only, verbal/batch, paper-first, multi-approver and external accountant approval mismatches.

  7. A large minority are not convertible by integration, setup or price fixes, setting a ceiling on any single lever.

    Medium confidenceBase 460

    Q3: 36.5% need none of the listed items (51.6% of Xero users). Q2: 10.0% decided they didn't need it, 8.7% chose another tool, 6.3% cite mobile app problems, with 12, 8 and 10 open-text mentions respectively. These teams need a different value proposition or will not convert.

  8. Unweighted totals overstate integration/Sage demand and understate setup demand; all population projections must use size weighting.

    High confidenceBase 460

    Sample size mix is 36.5% 1-9, 39.6% 10-49, 23.9% 50-249 vs population 55%, 35%, 10%. Integration main reason falls from 20.2% unweighted to 16.3% weighted; Sage need falls from 23.7% to 18.1%; setup rises from 18.7% to 21.6%. Ranking changes after weighting, with setup first, price 16.6% and integration 16.3% next.

Caveats

  • Only 460 of 1,640 invited trial admins completed (28.0%), with a £10 voucher; non-respondents may differ systematically by reason for leaving, software and size, and only trial admins (not managers, bookkeepers or accountants who feature in open texts) were surveyed.
  • Weighting adjusts only for company size using the product-database mix; it assumes respondents are representative within size bands on software and reasons, so weighted Sage/integration estimates retain extra uncertainty beyond the simple 3.9pp margin, which also does not apply to small subgroups (e.g. FreeAgent base 33, Other base 17).
  • Q2 forces a single main reason, understating teams with multiple barriers shown in open text (e.g. integration plus price); Q3 needs and Q5 likelihood are hypothetical stated intent, not observed payment, and Q5 was asked even of Xero/QuickBooks users already integrated.
  • Open text is 109 of 460 respondents (23.7%), so mention counts corroborate closed questions but are not population prevalence estimates; coding of borderline answers (e.g. configuration difficulty vs lack of time) involves judgement.
  • No causal evidence in this survey that assisted setup, a cheaper plan, a shorter trial or a Sage build would change behaviour; £6 is anchored in Q5, price elasticity and build cost/effort are not measured, and 0% reporting no accounting software limits generalisation beyond current users.
  • Trials ended in the last 90 days, so answers are retrospective and may rationalise decisions made for other reasons.

Next steps

  • experimentPilot assisted setup for new trials (concierge configuration within 48 hours, approval and category templates) vs control, prioritising 1-9 and 10-49 firms
  • decisionKeep 30-day trial and test day-3/day-7 human setup nudges; do not cut to 14 days
  • researchAudit incoming trials, enquiries and lost deals for Sage/FreeAgent share and talk to bookkeepers/accountants who insist on Sage
  • experimentTest per-company flat pricing vs per-user for teams with occasional claimants
  • experimentAdd approval templates for MD-only, batch weekly sign-off, two-director and external accountant approvers, included in assisted setup
  • monitorMonitor paid conversion, activation and loss reasons by size and accounting software after changes, and re-survey non-respondents

Open-text themes it coded

No integration with accounting software 22Didn't have time to set up / setup took too long 24Too expensive / per-user pricing 23Approval workflow didn't fit 15Decided they didn't need it 12Mobile app / receipt scanning problems 10Chose another tool 8

The survey it planned

2 screening questions and 10 questions, as the model wrote them.

  1. S1

    Are you the admin of a UK-based team whose Tallyroom trial ended in the last 90 days and did not convert to a paid plan?

    One answer
    • Yes
    • No

    Continues if Yes

  2. S2

    How many employees does your company have?

    One answer
    • 1-9
    • 10-49
    • 50-249
    • 250 or more

    Continues if 1-9; 10-49; 50-249

  3. Q1

    Which accounting software does your company currently use most for its finances?

    One answer
    • Xero
    • QuickBooks Online
    • Sage
    • FreeAgent
    • Kashoo
    • Spreadsheets, such as Excel or Google Sheets
    • No accounting software
    • Another accounting software
  4. Q2

    How much did you or your team use Tallyroom during the trial?

    Scale

    Scale 1-5: Not at all to A great deal

  5. Q3

    How easy or difficult was it to set up Tallyroom for your team?

    Scale

    Scale 1-5: Very difficult to Very easy

    Judges: assumptive

    Assumes the team attempted setup. Respondents who never attempted it still see this question and have no applicable response.

    It assumes the respondent tried to set up Tallyroom. Q2 allows 'Not at all' use, and there is no 'did not set it up' option, so non-users are forced to rate ease.

  6. Q4

    Which of the following were reasons your team did not pay for Tallyroom? Please select all that apply.

    Any that apply
    • The price of £6 per user per month was too high
    • It did not integrate with our accounting software
    • It was missing a feature we needed
    • Setting it up took too much time or was too difficult
    • We did not have enough time during the trial to evaluate it properly
    • The trial was too long and we lost interest or forgot about it
    • We chose a different product
    • We no longer needed an expenses tool
    • We did not get the budget or approval needed to pay
    • We had technical problems during the trial
    • Another reason
  7. Q5

    Which one of these was the main reason your team did not pay for Tallyroom?

    One answer
    • The price of £6 per user per month was too high
    • It did not integrate with our accounting software
    • It was missing a feature we needed
    • Setting it up took too much time or was too difficult
    • We did not have enough time during the trial to evaluate it properly
    • The trial was too long and we lost interest or forgot about it
    • We chose a different product
    • We no longer needed an expenses tool
    • We did not get the budget or approval needed to pay
    • We had technical problems during the trial
    • Another reason
  8. Q6

    Which of the following, if available during your trial, would have been important to your team? Please select all that apply.

    Any that apply
    • Integration with my current accounting software
    • A bank or credit card feed
    • Mileage and travel claims
    • Per diem and allowance claims
    • Automatic extraction of receipt details
    • Custom approval workflows
    • Support for multiple currencies
    • Integration with payroll or human resources software
    • Another integration or feature
    • None of these
  9. Q7

    If one of the following had been different during your trial, which would have been most likely to lead your team to pay?

    One answer
    • Integration with our accounting software
    • Assisted setup and onboarding
    • A lower price per user per month
    • A shorter 14-day trial
    • I would not have paid in any case
    • Something else
  10. Q8

    Which statement best describes the length of the trial for your team?

    One answer
    • We did not have enough time to evaluate Tallyroom properly
    • We lost interest or forgot about the trial before it ended
    • We had enough time to make a decision
    • We did not use the trial enough to judge whether to pay
    • The length of the trial did not affect our decision
  11. Q9

    To help us check that responses are being read carefully, please select 'I am reading carefully' below.

    One answer
    • I am reading carefully
    • I am not reading carefully
    • I do not know
  12. Q10

    In your own words, what was the single most important reason your team did not pay for Tallyroom, and what, if anything, could Tallyroom have done differently?

    Open text
    Judges: double-barrelled

    It asks for two things, the main reason for not paying and what Tallyroom could have done differently, in a single open response.

    Asks for both the single most important reason they did not pay AND what could have been done differently in one open question.

Sample plan

Invite the full census of approximately 1,558 admins (about 1,900 trial starts per quarter x 82% not converting) whose UK trial ended in the last 90 days without paying, one unique email link per team from the product database, with reminders until quotas fill; the 10-question survey is designed to take under 5 minutes. Set equal interlocking quotas by screener company-size band: 267 completes each for 1-9, 10-49 and 50-249 employees, for 801 completes in total. This uses the worst-case proportion calculation 1.96^2 x 0.25 / 0.06^2 = 267 per band to achieve +/-6 percentage points at 95% confidence. About 450 completes cannot meet that requirement (150 per band would give about +/-8 percentage points before weighting and non-response), so either increase completes to at least 801 (about 51% of the 90-day lapsed frame, likely requiring stronger reminders/incentive or a longer eligibility window), formally relax precision, and first verify each band has at least 267 people in the frame. Weight pooled estimates back to the frame's company-size distribution and replace attention-check failures to maintain quotas.