Benchmarks / SurveyBench / Why Tallyroom trials don't convert / GPT-6.2 Astra

Measured by Spring Prompt

GPT-6.2 Astra: Why Tallyroom trials don't convert

The decision: where to put next quarter's product and sales effort to lift trial-to-paid conversion: a new accounting integration, assisted setup, a cheaper plan, or a shorter trial. The brief and the data →

Verdict
✓ Sound research
Research score
97 of 100
Analysis rating
1,467
Head to head, this task
won 7 of 10

Scorecard

Numbers 11 of 11 right

QuestionIts answerTrue
What percentage of respondents said they would need a Sage integration before paying?✓23.7%23.7%
Weighted to the company-size mix of all lapsed trials, what percentage would need a Sage integration?✓18.1%18.2%
Among businesses with 1-9 employees, what percentage gave 'didn't have time to set it up' as the main reason?✓27.4%27.4%
How many of the open-text answers mention not having time to set Tallyroom up, or setup taking too long?✓2424
What is the 95% margin of error, in percentage points, on the unweighted share of respondents needing a Sage integration?✓3.9 points3.89 points
Is the difference in the share giving 'too expensive' as their main reason between businesses with 1-9 employees and those with 50-249 statistically significant at the 95% level? (1 for yes, 0 for no)✓NoNo
Weighted to the company-size mix of all 1,640 lapsed trials, about how many teams' main reason was not having time to set it up?✓355355
How many of the open-text answers from businesses with 1-9 employees mention not having time to set Tallyroom up, or setup taking too long?✓1414
Weighted to the company-size mix of all lapsed trials, what percentage would give 'no integration with our accounting software' as their main reason?✓16.3%16.3%
What percentage of the trial admins invited completed the survey?✓28%28%
Among respondents whose main reason was no integration with their accounting software, what percentage said they would be likely (fairly or very) to pay if Tallyroom connected to it?✓61.3%61.3%

Traps in the data 7 of 7 handled

  • ✓Teams on 'Other' accounting software (base 17) mostly cite missing integration.
  • ✓Teams of 50-249 people are 22% of respondents but 10% of lapsed trials. They cite missing integration and use Sage far more, so unweighted figures overstate integration and Sage demand and understate setup time: unweighted, missing integration is the top reason; weighted, setup time is.
  • ✓Q2 is multi-select: percentages are of respondents and add up to more than 100%.
  • ✓Likelihood to pay is a hypothetical stated intention.
  • ✓The head of sales believes the trial is too long and teams forget about it.
  • ✓460 of 1,640 invited admins completed (about 28%), with a £10 voucher; those who respond may differ from those who do not.
  • ✓Only teams that did not pay were surveyed, so the survey cannot show what distinguishes them from teams that did; paying teams may also have struggled with setup.

The analysis

Recommendation

Keep the 30-day trial and current headline price. Next quarter, have sales test hands-on setup where an integration already exists, and have product identify and validate the most commercially promising Sage edition with real buyers. Fund a connector build only once there is evidence of payable demand for that specific workflow. Compare options on incremental paying teams and contribution after delivery costs, not on the number who ticked a box.

What would change it

Prioritise the Sage build if enough buyers on one feasible edition commit at the actual seat price and a working pilot produces better expected contribution than assisted setup after build and support costs. Do not scale setup help if it improves activation without enough additional retained payers to cover staff time. Revisit price or trial length if controlled offers to new trials improve retained contribution per trial start, rather than merely discounting existing buyers or moving payment forward.

Findings

  1. Put sales effort into testing assisted setup first. Once the size mix is corrected, lack of setup time overtakes missing integration as the most common main reason. Teams describe starting setup and being interrupted, not just forgetting to start.

    Medium confidenceBase 460

    Setup accounts for 86/460 (18.7%) unweighted, against 93/460 (20.2%) for integration. Weighting the three size bands to 55%/35%/10% gives 21.6% for setup versus 16.3% for integration. That projects to about 355 of 1,640 lapsed teams citing setup, not 355 recoverable sales. Setup was cited by 46/168 microbusinesses (27.4%), and by 24.7% of Xero users (n=182) and 22.5% of QuickBooks users (n=111), whose software already connects. Of 109 written answers, 24 mention setup time, including 14 labelled 1–9 employees.

  2. Sage is the strongest candidate for a new integration, but scope the edition and posting requirements before committing engineering time. Demand for 'Sage' is not necessarily demand for the same connector.

    Medium confidenceBase 460

    109/460 (23.7%) selected Sage as a need, versus 7.2% for FreeAgent. The Sage estimate falls to 18.1% at the population size mix: selection rises from 7.7% of 168 microbusinesses to 28.0% of 182 businesses with 10–49 employees and 40.9% of 110 with 50–249. Among 117 Sage users, 100 (85.5%) selected Sage and 68 (58.1%) named missing integration as their main blocker. The written answers name Sage 50, Sage 200, Business Cloud and Sage Accounting, and describe avoiding re-keying approved expenses.

  3. Finance should not forecast a number of payers from this survey. Integration-blocked teams are promising sales prospects; their stated intent is not a conversion rate, nor is it Sage-specific.

    High confidenceBase 93

    Of 93 respondents whose main reason was missing integration, 57 (61.3%) said they were likely to pay £6 per user per month if connected: 35.5% fairly likely and 25.8% very likely. Across all 460, 37.6% were likely, but 293 already used supported software. There is no Sage-only intent table, and neither an actual purchase nor the incremental effect of a connector was measured.

  4. Do not cut the default trial to 14 days on this evidence. Forgetting is different from finding the trial too long, and a shorter deadline could hurt teams still trying to finish setup.

    Medium confidenceBase 460

    Only 1/460 (0.2%) chose 'too long'; 17 (3.7%) forgot the trial was running and 7 (1.5%) chose 'too short'. By comparison, 86 cited setup time. One admin started on day two but never found time to resume (T263). No trial lengths were compared.

  5. Test pricing structure before cutting the price for everyone. Some buyers object to paying for occasional claimants, not necessarily to the rate for an active user.

    Medium confidenceBase 460

    73/460 (15.9%) named price as the main reason; 20.4% selected per-company pricing as a need on a separate multi-select question. T208 describes only two regular submitters; T355 describes seasonal headcount. The main-reason difference between 1–9 employees (32/168, 19.0%) and 50–249 (17/110, 15.5%) is not statistically significant in a two-sided two-proportion test at 5%. That is not evidence the groups are identical.

Caveats

  • 460 of 1,640 invited admins completed (28.0%). This was an incentivised voluntary response from recent non-payers, not a random sample of all trial starts. It does not explain why similar teams paid or estimate lift against a control.
  • Microbusinesses are underrepresented (36.5% versus 55% of lapsed trials); businesses with 50–249 employees are overrepresented (23.9% versus 10%). Size weighting corrects those shares only. Population projections assume respondents resemble nonrespondents within each size band.
  • The ±3.9 percentage-point margin is the nominal, unweighted 95% binomial normal-approximation margin for 109/460, without a finite-population correction. It does not bound nonresponse or measurement bias. The price significance test likewise assumes independent observations.
  • Q2 forces one main reason; Q3 allows several prerequisites. Neither a selected need nor removal of the main blocker guarantees payment. For example, T322 cites both Sage and price, and T329 cites both setup time and upload failures.
  • Only 109 respondents supplied text. Theme counts include overlapping mentions and use the supplied segment labels; they explain mechanisms rather than estimate prevalence. Some labels conflict with the prose (e.g. T010 is labelled 50–249 but says nine people). Verify size against the database before targeting accounts.
  • The available cross-tabs do not support an edition-specific Sage forecast, size-weighted payment intent or a comparison of the four options' net returns. FreeAgent (n=33) and Other (n=17) are particularly small software subgroups.

Next steps

  • experimentRandomise new supported-software trial accounts to usual onboarding or an early assisted session that connects accounts, invites staff and takes a receipt through approval and export. Keep trial length and price fixed.
  • researchQualify Sage leads by edition, workflow, seat count and buying authority; demonstrate the required posting flow and seek paid pilot commitments at £6 per user per month.
  • experimentTest a capped or per-company offer against current pricing in a separate cohort, rather than rolling out a blanket discount.
  • monitorKeep the default at 30 days. Review activation and payment timing across both successful and lapsed trials before considering a randomised 14-versus-30-day test.

Open-text themes it coded

Insufficient time or too much work to finish setup 24Missing Sage connection 15Missing FreeAgent connection 3Other missing accounting connection 4Price, including the cost or unpredictability of paying by headcount 23Approval rules, sequence or channel do not match existing sign-off 15Mobile capture, recognition or upload failures 10Too little need for a dedicated expenses tool 12Selected a competing tool or card-based alternative 8

The survey it planned

1 screening question and 10 questions, as the model wrote them.

  1. S1

    Which statement describes your role in the Tallyroom trial for the team named in your invitation?

    One answer
    • I was the trial admin, or shared that responsibility.
    • I took part but was not a trial admin.
    • I did not take part.
    • I don’t recognise this trial.

    Continues if I was the trial admin, or shared that responsibility.

  2. Q1

    What was the main reason your team didn’t start paying for Tallyroom at the end of the trial? A few words are enough.

    Open text
  3. Q2

    How far did your team get during the trial? Choose the furthest step anyone on your team completed.

    One answer
    • No one tried it after signing up
    • Looked around or worked on setup, but didn’t submit an expense
    • Submitted an expense, but didn’t approve one
    • Approved an expense, but didn’t send one to accounting software
    • Sent an approved expense to accounting software
    • Not sure
  4. Q3

    Which of these, if any, contributed to your team not starting a paid subscription? Select all that apply.

    Any that apply
    • We didn’t need a new way to manage expenses
    • Another product suited us better
    • Tallyroom didn’t connect to the accounting software we needed
    • A feature we needed was missing
    • Something didn’t work reliably
    • We found setup difficult
    • We couldn’t get enough colleagues to try it
    • The price was too high
    • We couldn’t get spending approval
    • We needed more time to evaluate it
    • The trial slipped off our radar
    • Other work took priority
    • Another reason
    • None of these
    • Not sure
  5. Q4

    During the trial, which accounting software was most important for Tallyroom to connect to for your team?

    One answer
    • Xero
    • QuickBooks
    • Sage Accounting (Sage Business Cloud Accounting)
    • Sage 50 Accounts
    • FreeAgent
    • IRIS KashFlow
    • Clear Books
    • Another product
    • We didn’t need an accounting connection
    • Not sure
  6. Q5

    What was the name of that accounting product? Please include the edition if you know it.

    Open text

    Routing Shown only if Q4 is: Another product

  7. Q6

    What was the most important thing your team needed to do that the missing feature would have let you do?

    Open text

    Routing Shown only if Q3 is: A feature we needed was missing

  8. Q7

    By the end of the 14th day of your trial, which best described your team?

    One answer
    • We had decided to subscribe
    • We had decided not to subscribe
    • We were still evaluating Tallyroom
    • We had paused our evaluation without a decision
    • We hadn’t started evaluating Tallyroom
    • I don’t remember or don’t know
  9. Q8

    Imagine your team had been offered a 14-day trial instead of a 30-day trial, with everything else unchanged. How do you think that would have affected the chance of your team starting a paid subscription?

    One answer
    • Much less likely to subscribe
    • A little less likely
    • No difference
    • A little more likely
    • Much more likely to subscribe
    • Can’t say
  10. Q9

    Thinking back to your team’s trial, which ONE change, if any, would have done the most to make your team more likely to pay?

    One answer
    • A new connection to accounting software we needed
    • A free setup session with someone from Tallyroom
    • A plan costing less than £6 per user per month
    • A 14-day trial instead of 30 days
    • A different change
    • None of these changes would have helped
    • Can’t say
  11. Q10

    At the time of the trial, roughly how many people worked for your whole business, including working owners? Please count people across the business, not just Tallyroom users.

    One answer
    • 1–9
    • 10–49
    • 50–249
    • 250 or more
    • Not sure
Sample plan

Freeze the product-database frame on the invitation date: UK teams whose trial ended in the preceding 90 days without conversion. Deduplicate teams and send one personalised, single-use survey link per team to its recorded admin. Invite all eligible admins, offer a £10 voucher per eligible completion and remind nonrespondents, prioritising underfilled size bands. At steady volumes the frame is roughly 1,558 teams (1,900 × 82%); 450 completes would require about 29% participation. Aim for about 150 completes per requested size band, monitoring database size where available and verifying with Q10; retain other/unknown sizes separately. Soft targets, not a promise of equal yield. Before launch, count eligible teams per band and agree a precision trade-off: at a worst-case 50% proportion, 150 random observations give about ±8 percentage points at 95% confidence without finite-population correction; ±6 requires about 267 per band (801 total). For inference only to this fixed frame, check each known band size N and required n = ceiling(N × 266.78 / (N − 1 + 266.78)); do not assume band sizes or that voluntary response is random. If required yields are infeasible, increase the completes target where possible or report wider intervals; no quota guarantees accuracy against nonresponse bias. Pilot for a five-minute maximum, including the longest branch; allow open questions to be skipped. Shuffle substantive reasons in Q3 and the four proposed changes in Q9, keep residual answers last, and make ‘None of these’ and ‘Not sure’ exclusive in Q3.