Benchmarks / SurveyBench / Why Tallyroom trials don't convert / GPT-6.2 Sol

Measured by Spring Prompt

GPT-6.2 Sol: Why Tallyroom trials don't convert

The decision: where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →

Verdict
✓ Sound research
Research score
94 of 100
Analysis rating
1,354
Head to head, this task
won 6 of 13

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents said they would need a Sage integration before paying?✓23.7%23.7%
Weighted to the company-size mix of all lapsed trials, what percentage would need a Sage integration?✓18.1%18.2%
Among businesses with 1-9 employees, what percentage gave 'didn't have time to set it up' as the main reason?✓27.4%27.4%
How many of the open-text answers mention not having time to set Tallyroom up, or setup taking too long?✓2424
What is the 95% margin of error, in percentage points, on the unweighted share of respondents needing a Sage integration?✓3.9 points3.89 points
Is the difference in the share giving 'too expensive' as their main reason between businesses with 1-9 employees and those with 50-249 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the company-size mix of all 1,640 lapsed trials, about how many teams' main reason was not having time to set it up?✓355355
How many of the open-text answers from businesses with 1-9 employees mention not having time to set Tallyroom up, or setup taking too long?✓1314
Weighted to the company-size mix of all lapsed trials, what percentage would give 'no integration with our accounting software' as their main reason?✓16.3%16.3%
What percentage of the trial admins invited completed the survey?✓28%28%
Among respondents whose main reason was no integration with their accounting software, what percentage said they would be likely (fairly or very) to pay if Tallyroom connected to it?✓61.3%61.3%

Traps in the data 7 of 7 handled

  • ✓Teams on 'Other' accounting software (base 17) mostly cite missing integration.
  • ✓Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
  • ✓Q2 is multi-select: percentages are of respondents and add up to more than 100%.
  • ✓Likelihood to pay is a hypothetical stated intention.
  • ✓The head of sales believes the trial is too long and teams forget about it.
  • ✓460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
  • ✓Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.

The analysis

Recommendation

Make assisted setup the primary sales and product experiment next quarter, starting with small Xero and QuickBooks teams. Fund Sage discovery and a narrow pilot in parallel, but defer a full build until demand for a specific Sage product translates into paid use. Keep the 30-day trial and current list price while testing earlier outreach and alternative packaging separately.

What would change it

Move effort from assisted setup to Sage if onboarding support fails to generate enough incremental paid accounts to cover delivery costs and one Sage product shows enough reachable demand, paid pilot uptake, and retention to justify build and maintenance. Prioritise packaging if it produces more contribution per trial than setup help, rather than simply more sign-ups. Shorten the trial only if a controlled test shows better paid conversion at the same follow-up horizon without hurting activation or early retention.

Findings

  1. Put the first conversion experiment into assisted setup, especially for small teams already on Xero or QuickBooks.

    Medium confidenceBase 460

    Lack of setup time was the main reason for 18.7% of 460 respondents. After adjusting for company size, it rises to 21.6%, ahead of missing integration at 16.3% and price at 16.6%. That projects to about 355 of the 1,640 lapsed teams, not 355 recoverable sales. Setup was the main reason for 27.4% of 168 businesses with 1–9 employees and for 70 of 293 Xero or QuickBooks users (23.9%). The open text points to interrupted configuration and teams never getting invited: 24 answers mention setup time, 13 from respondents classified as having 1–9 employees.

  2. Sage is the strongest new integration candidate, but scope it by Sage product before committing to a build.

    Medium confidenceBase 460

    109 of 460 respondents (23.7%) selected Sage as something they would need before paying, versus 7.2% for FreeAgent. The Sage figure falls to 18.1% after size weighting. Among 117 Sage users, 85.5% selected the integration as a need and 58.1% named missing integration as their main blocker. Comments name Sage Business Cloud or Sage Accounting, Sage 50, and Sage 200, so the overall demand figure does not size demand for any one connector.

  3. Do not forecast paid conversions from the integration question. Use it to recruit a pilot, not to book revenue.

    Low confidenceBase 93

    Among the 93 respondents who named missing integration as their main blocker, 57 (61.3%) said they were fairly or very likely to pay £6 per user per month if connected; 24 (25.8%) said very likely. This is hypothetical intent across accounting packages, not a Sage-specific result or an observed conversion rate. Text answers also show that missing integration can coexist with price objections.

  4. Keep the 30-day trial. Test earlier outreach and setup help rather than using a 14-day deadline to create urgency.

    Medium confidenceBase 460

    Only 1 of 460 respondents (0.2%) named the trial being too long as the main reason, versus 17 (3.7%) who forgot it was running and 7 (1.5%) who said it was too short. Forgetting does not show that shortening would help, and setup comments describe teams already running out of time before they could use the product.

  5. Do not roll out a cheaper plan on this evidence. Test whether a different billing model solves the problem before cutting price across the board.

    Medium confidenceBase 460

    Price was the main blocker for 15.9% of 460 respondents, or 16.6% after size weighting. The difference between 1–9 employees (19.0%, n=168) and 50–249 (15.5%, n=110) is not significant in an unadjusted two-proportion test (p≈0.44). Separately, 20.4% selected per-company pricing as something they would need. Comments distinguish high total cost from objections to paying for occasional or seasonal users.

Caveats

  • Only 460 of 1,640 invited admins completed the survey. The sample underrepresents 1–9 employee firms (36.5% versus 55% in the database) and overrepresents 50–249 employee firms (23.9% versus 10%). Weighting corrects that mix but cannot correct for nonresponse within size bands or differences by accounting package, engagement, or voucher motivation.
  • The 3.9 percentage-point margin is the conventional unweighted 95% binomial margin assuming independent random sampling, without a finite-population correction. It does not capture nonresponse or measurement bias. Likewise, the price comparison does not establish that the size groups are equivalent.
  • The survey covers recent non-buyers and records admins' retrospective accounts. It cannot establish the causal effect of any change on conversion among new trials, and the admin may not be the person who authorises payment.
  • Q2 forces a single main reason; Q3 allows multiple prerequisites and does not show that any one is sufficient. Q5 is hypothetical and was asked even of Xero and QuickBooks users whose software is already supported. The supplied tables do not cross-tabulate Q5 by accounting package.
  • The 109 open-text answers explain how blockers arise but should not be used to estimate prevalence. Some descriptions of headcount conflict with the supplied size labels; the small-business text count uses those labels as given.

Next steps

  • decisionKeep the 30-day trial next quarter; use outreach to bring stalled teams back rather than moving the expiry date.
  • experimentRandomise eligible new Xero and QuickBooks trials to the current experience or an early setup offer that helps configure approvals, invite staff, and export a first expense.
  • researchIdentify the exact Sage products and versions in the trial pipeline, then recruit buyers for a narrowly scoped paid pilot on the best-supported candidate.
  • experimentTest a company-level or active-user offer against current pricing on supported accounts, separately from the setup experiment.

Open-text themes it coded

Not enough time for setup, or setup took too long 24Missing accounting integration, often leaving manual re-entry or import 22Price, value for money, or per-user billing objection 25Approval workflow did not match how the team signs off expenses 15Mobile receipt capture, recognition, or upload was unreliable 10Little need for a dedicated tool, often because expense activity was low or existing methods sufficed 13Chose another tool or an alternative card-based solution 8

The survey it planned

3 screening questions and 9 questions, as the model wrote them.

  1. Q1

    Were you one of the people responsible for managing the Tallyroom trial for the team named in your invitation?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  2. Q2

    Was the business that started this trial based in the UK?

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  3. Q3

    Has this team ever paid for Tallyroom for use after this trial?

    One answer
    • Yes
    • No
    • Not sure

    Continues if No; Not sure

  4. Q4

    When you started the trial, how many employees did your business have across all locations? Count both full-time and part-time employees.

    One answer
    • 1–9
    • 10–49
    • 50–249
    • 250 or more
    • No employees
    • Not sure
  5. Q5

    How far did your team get with Tallyroom during the trial? Choose the furthest step anyone on your team reached.

    One answer
    • No one signed in
    • Someone signed in, but no one submitted an expense
    • Someone submitted an expense, but none were approved
    • Someone approved an expense, but none were sent to our accounting software
    • Someone sent an approved expense to our accounting software
    • Not sure
  6. Q6

    Why did your team not pay for Tallyroom after the trial? Select all that apply. If you choose ‘Not sure’ or ‘None of these’, select it on its own.

    Any that apply
    • Tallyroom did not connect to the accounting software we needed
    • The connection to our accounting software did not work as we needed
    • A feature other than an accounting connection was missing or did not meet our needs
    • Setting up Tallyroom took too much time or effort
    • Tallyroom was difficult to use once set up
    • The price of £6 per user per month was too high for us
    • The trial ended before we had enough time to evaluate Tallyroom
    • We lost track of the trial
    • We did not get the internal approval we needed
    • We preferred another tool or our existing process
    • We no longer needed an expenses app or the project was put on hold
    • Another reason
    • None of these
    • Not sure
  7. Q7

    Of the reasons you selected, which mattered most to your team’s decision not to pay?

    One answer
    • Tallyroom did not connect to the accounting software we needed
    • The connection to our accounting software did not work as we needed
    • A feature other than an accounting connection was missing or did not meet our needs
    • Setting up Tallyroom took too much time or effort
    • Tallyroom was difficult to use once set up
    • The price of £6 per user per month was too high for us
    • The trial ended before we had enough time to evaluate Tallyroom
    • We lost track of the trial
    • We did not get the internal approval we needed
    • We preferred another tool or our existing process
    • We no longer needed an expenses app or the project was put on hold
    • Another reason
    • No single reason mattered most
    • Not sure

    Routing Shown only if Q6 is: Tallyroom did not connect to the accounting software we needed; The connection to our accounting software did not work as we needed; A feature other than an accounting connection was missing or did not meet our needs; Setting up Tallyroom took too much time or effort; Tallyroom was difficult to use once set up; The price of £6 per user per month was too high for us; The trial ended before we had enough time to evaluate Tallyroom; We lost track of the trial; We did not get the internal approval we needed; We preferred another tool or our existing process; We no longer needed an expenses app or the project was put on hold; Another reason

  8. Q8

    During the trial, what accounting software, if any, did your team need Tallyroom to connect to? Enter the name, or write ‘None’ or ‘Not sure’.

    Open text
  9. Q9

    What was missing or did not work as your team needed? Please name the accounting connection or feature and briefly describe what you needed it to do.

    Open text

    Routing Shown only if Q6 is: Tallyroom did not connect to the accounting software we needed; The connection to our accounting software did not work as we needed; A feature other than an accounting connection was missing or did not meet our needs

  10. Q10

    How did the 30-day trial period compare with the time your team needed to decide whether to pay for Tallyroom?

    One answer
    • It was shorter than we needed
    • It was about right
    • It was longer than we needed
    • We did not get far enough to judge
    • Not sure
  11. Q11

    Thinking back to your trial, which one change, if any, would have made your team most likely to pay for Tallyroom? Assume only that one thing changed.

    One answer
    • Add a connection to accounting software we needed that Tallyroom did not support
    • Offer a free 30-minute call to help us set up Tallyroom
    • Offer the same features at £4 per user per month instead of £6
    • Make the free trial 14 days instead of 30 days
    • None of these would have made us more likely to pay
    • Not sure
  12. Q12

    If your free trial had lasted 14 days instead of 30, with no other changes, would your team have been more or less likely to pay for Tallyroom?

    One answer
    • Much more likely
    • Somewhat more likely
    • About as likely
    • Somewhat less likely
    • Much less likely
    • Not sure
Sample plan

Email one named trial admin per UK team whose 30-day trial ended in the 90 days before launch without converting. Use the product database to verify eligibility, deduplicate teams and suppress those that have since paid. Identify the team and trial in the invitation and offer a £10 voucher for a completed eligible survey. Invite the full eligible list and send up to two reminders to nonrespondents, prioritising underrepresented size bands if size is known. Aim for about 450 completes total, with soft targets of roughly 150 each for 1–9, 10–49 and 50–249 employees. Retain and report responses outside those bands separately; do not use hard caps that turn away eligible admins. Under a simple random sample and before any finite-population correction, ±6 percentage points at 95% confidence requires about 267 completes per band at the worst-case proportion of 50%, or about 801 across the three bands. At 150 per band the corresponding margin is about ±8 points. Check eligible population counts by band before fielding: a finite-population correction can reduce the sampling margin under an appropriate sampling model, but cannot fix nonresponse bias. Do not promise ±6 points with 450 completes; if it is non-negotiable and cannot be supported by the available frame, agree a larger sample and, if needed, a longer recruitment window before launch. On mobile, pilot the longest route to confirm it takes no more than five minutes. In Q7, show only the substantive reasons selected in Q6 plus the final two options. Randomise the substantive options in Q6 and the first four options in Q11, keeping the remaining options at the end.