Benchmarks / SurveyBench / Why Tallyroom trials don't convert / GPT-6.1 Terra

Measured by Spring Prompt

GPT-6.1 Terra: Why Tallyroom trials don't convert

The decision: where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →

Verdict
✗ Not sound
Research score
88 of 100
Analysis rating
1,273
Head to head, this task
won 7 of 9

Why it is not sound

The survey

  • A research objective no question serves

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents said they would need a Sage integration before paying?✓23.7%23.7%
Weighted to the company-size mix of all lapsed trials, what percentage would need a Sage integration?✓18.2%18.2%
Among businesses with 1-9 employees, what percentage gave 'didn't have time to set it up' as the main reason?✓27.4%27.4%
How many of the open-text answers mention not having time to set Tallyroom up, or setup taking too long?✓2424
What is the 95% margin of error, in percentage points, on the unweighted share of respondents needing a Sage integration?✓3.9 points3.89 points
Is the difference in the share giving 'too expensive' as their main reason between businesses with 1-9 employees and those with 50-249 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the company-size mix of all 1,640 lapsed trials, about how many teams' main reason was not having time to set it up?✓355355
How many of the open-text answers from businesses with 1-9 employees mention not having time to set Tallyroom up, or setup taking too long?✓1414
Weighted to the company-size mix of all lapsed trials, what percentage would give 'no integration with our accounting software' as their main reason?✓16.3%16.3%
What percentage of the trial admins invited completed the survey?✓28%28%
Among respondents whose main reason was no integration with their accounting software, what percentage said they would be likely (fairly or very) to pay if Tallyroom connected to it?✓61.3%61.3%

Traps in the data 7 of 7 handled

  • ✓Teams on 'Other' accounting software (base 17) mostly cite missing integration.
  • ✓Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
  • ✓Q2 is multi-select: percentages are of respondents and add up to more than 100%.
  • ✓Likelihood to pay is a hypothetical stated intention.
  • ✓The head of sales believes the trial is too long and teams forget about it.
  • ✓460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
  • ✓Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.

The analysis

Recommendation

Put next quarter's primary effort into a controlled assisted-setup pilot for new trials, while sales and product validate Sage demand and integration scope. Retain the 30-day trial. Do not yet commit to a broad price cut or a Sage build; use actual paid conversion, revenue and delivery cost to decide which to scale.

What would change it

Prioritise a Sage build if validated demand for a specific Sage product, actual purchase commitments or a credible pilot forecast show greater incremental paid revenue after build and support costs than assisted setup. Shift towards pricing if a controlled offer produces better incremental revenue. Reconsider trial length only if a controlled test shows that 14 days improves paid conversion without damaging activation or revenue.

Findings

  1. Prioritise testing assisted setup, especially for the smallest businesses. It addresses an actionable barrier in the largest population size segment, but its effect on conversion is not yet established.

    Medium confidenceBase 460

    Setup was the main reason for 18.7% of respondents (86/460) and 27.4% of businesses with 1–9 employees (46/168). Size-weighting gives approximately 21.6%, equivalent to about 355 of 1,640 lapsed teams. In the supplied open text, 24 answers mention setup time, including 14 tagged as 1–9 employees.

  2. Sage demand is substantial enough to validate now, but not yet enough to commit to building a particular Sage integration.

    Medium confidenceBase 460

    109/460 respondents (23.7%) said they needed Sage before paying; size-weighted, this is approximately 18.2%. Among Sage users, 100/117 (85.5%) said they needed it and 68/117 (58.1%) gave missing integration as their main reason. Open answers name Sage 50, Sage 200 and Sage Business Cloud, so the required product coverage needs checking.

  3. Do not interpret stated payment likelihood as the number of teams that will pay after an integration is built.

    Medium confidenceBase 93

    Among the 93 respondents citing missing integration as their main reason, 57 (61.3%) said they were fairly or very likely to pay £6 per user per month if connected. This is a hypothetical intention, not observed payment, and the table does not identify the Sage users within that group.

  4. Keep the 30-day trial for now; the survey does not support cutting it to 14 days.

    Medium confidenceBase 460

    One of 460 respondents (0.2%) chose 'trial too long' as their main reason, 17 (3.7%) chose forgetting it and seven (1.5%) chose 'trial too short'. Several open answers describe running out of time to set up. These responses do not measure the causal effect of shortening a trial.

  5. Investigate pricing structure before broadly lowering prices. A cheaper plan may help some teams, but the evidence does not show that a blanket reduction is the best investment.

    Medium confidenceBase 460

    73/460 (15.9%) named expense as their main reason; 94/460 (20.4%) selected per-company pricing as something needed before paying. The main-reason price shares were 19.0% among 1–9 employees and 15.5% among 50–249 employees; the difference is not statistically significant at 95%.

Caveats

  • The survey covers only admins of trials that ended without paying, not all trials or current prospects. It cannot directly estimate trial-to-paid conversion or the effect of an intervention on conversion.
  • 460 of 1,640 invited admins completed the survey. Self-selection and the voucher may affect results; size-weighting corrects the stated size mix only, not other nonresponse biases or the accounting-software mix.
  • The size-weighted estimates use the displayed segment counts and population proportions. The approximately 355 setup-blocked teams is an illustrative extrapolation, not an observed count.
  • The 3.9 percentage-point margin uses the conventional 95% simple-random-sample approximation without a finite-population correction. Applying that correction would give approximately 3.3 points, but neither margin accounts for self-selection or other survey bias.
  • Some open answers conflict with their supplied company-size tags: for example, T010 is tagged 50–249 but describes nine people, and T087 is tagged 50–249 but describes five. Validate segmentation before relying heavily on size-specific targeting or weighting.
  • Only 109 open answers are supplied. Theme counts describe these answers, not all 460 respondents; answers can express multiple themes.
  • Q3 is multi-select, and a stated need is not proof that fulfilling it alone will produce a purchase. Q5 is hypothetical and was also asked of Xero and QuickBooks users, whose software is already supported.
  • The price comparison is based on 168 small and 110 larger respondents. A non-significant difference is not proof that the groups have identical price sensitivity.

Next steps

  • decisionRetain the 30-day trial and allocate capacity to an assisted-setup pilot.
  • experimentRandomly offer setup assistance to eligible new trial admins, including help connecting software, inviting staff and configuring approvals.
  • researchInterview and qualify Sage-using prospects; verify which Sage products and workflows they need, then seek paid pilot commitments.
  • experimentTest a per-company or occasional-user pricing offer against existing pricing in a controlled trial.
  • monitorAudit company-size records and track activation and paid conversion by validated size and accounting software.

Open-text themes it coded

Insufficient time for setup or setup took too long 24Missing accounting integration or automatic sync 22Price, perceived value or per-user pricing 25Approval workflow did not fit 15Little or no need for an expenses app 12Chose another tool 8Receipt capture, scanning or mobile reliability problems 10

The survey it planned

2 screening questions and 10 questions, as the model wrote them.

  1. S1

    Are all of the following true? You are an admin for a UK-based team whose Tallyroom free trial ended within the last 90 days, and the team has not paid for Tallyroom since the trial ended.

    One answer
    • Yes
    • No
    • Not sure

    Continues if Yes

  2. S2

    Approximately how many people did your business employ when the Tallyroom trial ended? Include yourself and part-time employees.

    One answer
    • 1–9
    • 10–49
    • 50–249
    • 250 or more
    • Not sure

    Continues if 1–9; 10–49; 50–249

  3. Q1

    How far did your team get during the Tallyroom trial?

    One answer
    • We signed up but did not set anything up
    • We started setup but did not submit an expense
    • We submitted at least one expense but did not approve one
    • We approved at least one expense but did not send one to accounting software
    • We sent at least one approved expense to accounting software
    • Not sure
  4. Q2

    What contributed to your team not paying for Tallyroom after the trial? Select all that apply.

    Any that apply
    • The price was too high
    • We needed an accounting integration that was not available
    • We needed another feature that was missing
    • Setup or connecting our accounting software was difficult
    • We did not have enough time or resources to set it up or try it
    • We forgot about the trial or lost momentum
    • The trial ended before we could evaluate Tallyroom
    • We did not have enough expenses or users to justify it
    • We chose another solution
    • Our needs or priorities changed
    • We could not get approval to buy it
    • There was another reason
    • Not sure
  5. Q3

    Which one of the reasons you selected was the main reason your team did not pay?

    One answer
    • The price was too high
    • We needed an accounting integration that was not available
    • We needed another feature that was missing
    • Setup or connecting our accounting software was difficult
    • We did not have enough time or resources to set it up or try it
    • We forgot about the trial or lost momentum
    • The trial ended before we could evaluate Tallyroom
    • We did not have enough expenses or users to justify it
    • We chose another solution
    • Our needs or priorities changed
    • We could not get approval to buy it
    • There was another reason
    • Not sure
  6. Q4

    Which accounting software did you need Tallyroom to connect to? Select all that apply.

    Any that apply
    • Sage
    • FreeAgent
    • Xero
    • QuickBooks
    • Another accounting system
    • Not sure

    Routing Shown only if Q2 is: We needed an accounting integration that was not available

  7. Q5

    What missing integration or feature did you need? If you selected another accounting system, please name it. You may leave this blank.

    Open text

    Routing Shown only if Q2 is: We needed an accounting integration that was not available; We needed another feature that was missing

  8. Q6

    If someone from Tallyroom had offered to help your team set up and connect its accounting software during the trial, how much difference do you think that would have made to your decision to pay?

    One answer
    • It might have made the difference between paying and not paying
    • It would have helped, but probably would not have changed our decision
    • It would not have helped
    • We already received this kind of help
    • Not sure
  9. Q7

    Thinking about your team's experience, how did the 30-day trial length feel?

    One answer
    • Much too short
    • A little too short
    • About right
    • A little too long
    • Much too long
    • We did not use enough of the trial to judge
    • Not sure
  10. Q8

    If your trial had been 14 days instead of 30 days, what effect do you think that would have had on your team's ability to decide whether to pay?

    One answer
    • It would have made deciding easier
    • It would have made no difference
    • It would have made deciding harder
    • Not sure
  11. Q9

    Tallyroom's price is £6 per user per month. If the price had instead been £3 per user per month, with the same features, how do you think that would have affected your team's decision?

    One answer
    • We probably would have paid
    • We might have paid
    • We probably would not have paid
    • Not sure
  12. Q10

    Which one of the following, if available during your trial, would have been most likely to help your team decide to pay?

    One answer
    • A connection to an accounting system we needed
    • Help from Tallyroom with setup and connecting accounting software
    • A lower-priced plan
    • A 14-day trial instead of a 30-day trial
    • Something else
    • None of these would have helped
    • Not sure
Sample plan

Use the product database to invite every identifiable admin of a UK team whose trial ended in the last 90 days without paying. Send one invitation per team and a reminder to non-responders; offer the £10 voucher to eligible completers. Aim for about 450 completes, with soft targets of 150 in each eligible employee band. Monitor response by band and use targeted reminders where database size data are available; do not assume all bands can reach target. At 150 completes per band, the worst-case 95% margin of error is approximately ±8 percentage points before any finite-population correction, not ±6. Without that correction, ±6 requires approximately 267 completes per band, or 801 in total. Check the eligible population size in each band: a finite-population correction may reduce the required sample if respondents can be treated as a random sample of that band, but 450 does not guarantee the requested precision. Report achieved sample sizes, corrected margins where appropriate, and non-response limitations; seek a larger sample or relax the precision requirement if needed.