Behavioural Insights Units Are Producing Evidence That Procurement Panels Cannot Use

Australia's investment in behavioural insights capability inside government has, by any reasonable measure, produced a credible body of experimental evidence. The Behavioural Economics Team of the Australian Government (BETA), operating within the Department of the Prime Minister and Cabinet, has published methodologically sound randomised controlled trials on subjects ranging from ATO debt recovery letter framing (2017) to COVID-19 health compliance messaging (2021). These are not discussion papers or literature reviews. They are controlled experiments with randomised treatment assignment, defined outcome metrics, and published effect sizes.

The ATO debt recovery trial, for instance, tested variation in social norm framing, loss aversion framing, and identity-based appeals across a sample of overdue accounts. The finding — that certain norm-based frames produced meaningfully higher payment rates than the control — was published and is publicly available. The COVID-19 compliance messaging trials similarly tested framing variants relevant to mask-wearing and physical distancing behaviour at a moment when the Australian government was spending substantially on paid media campaigns directed at exactly those behaviours.

Neither body of findings was integrated into the concurrent paid campaigns delivered through SSMG-era panel arrangements. The creative running in market during those campaigns was developed under brief-writing assumptions that predated — or ran parallel to — the experimental evidence without any formal mechanism for convergence.

BX NSW, the behavioural insights function embedded within NSW Customer Service, published its 2022–23 Annual Report documenting more than 40 completed behavioural trials across a range of policy domains. The quality of the experimental work is not in question. What the report does not contain — and this absence is itself diagnostic — is any documented pathway, formal or informal, by which trial outputs inform the creative deliverables produced under NSW coordinated procurement arrangements for paid media.

The gap is not a matter of awareness or institutional indifference. Communications directors in major federal agencies are frequently briefed on BETA outputs. They read the trial results. In some cases, the same executives oversee both the behavioural insights function and the communications directorate. The constraint is not attitudinal. It is that the panel deed under which they procure creative services does not provide a compliant mechanism for substituting experimentally validated messaging for the assumption-based creative agreed at brief-writing stage.

Government is funding rigorous experiments whose results have no contractual pathway into the campaigns those experiments were designed to improve.

This is a systemic architecture problem, and it has a specific contractual address.

The Framework for the Coordinated Procurement of Advertising and Related Services (FRAEW) and its predecessor coordinated arrangements were designed to serve a particular model of government campaign development. That model is linear and sequential: strategic objectives are defined, a brief is written, creative is developed under a head agreement with a panel supplier, media is purchased against a locked creative specification, and evaluation occurs post-campaign against metrics agreed at deed execution.

This architecture was a reasonable response to the governance failures that preceded coordinated procurement. The ANAO and successive parliamentary scrutiny exercises had identified problems with sole-source creative engagements, inconsistent value-for-money disciplines, and inadequate evaluation. Coordinated panels addressed those problems by standardising the procurement pathway and anchoring evaluation to pre-agreed metrics.

What the architecture did not anticipate — because behavioural insights units at the scale now operating in Australian government did not exist when the deed templates were developed — is the possibility that credible experimental evidence about messaging effectiveness would be generated during campaign execution, rather than before brief-writing. The deed structures embed an epistemological assumption that the best available evidence about what messaging will work is captured at brief-writing stage and does not materially change during execution.

That assumption was reasonable in 2010. It is not reasonable in 2025, when BETA, BX NSW, and Service Victoria's CX function are running experiments on four-to-sixteen-week cycles and producing effect-size data that is directly relevant to in-market creative decisions.

The value-for-money obligations under the Commonwealth Procurement Rules, as operationalised through panel deed variation clauses, compound the problem. Any mid-cycle change to creative deliverables must be documented as a deed variation, and that variation must be supported by a value-for-money case. The current guidelines do not specify how to construct a value-for-money case when the basis for change is experimental evidence rather than changed campaign objectives or budget. The result is an effective compliance cost for evidence-triggered creative changes that is high enough to function as a de facto prohibition — regardless of the quality of the evidence, and regardless of the communications director's professional judgement about its relevance.

The mismatch in operating cadence between behavioural insights units and panel deed structures is not marginal. Behavioural insights teams operate on experiment-and-iterate cycles that typically run four to sixteen weeks from design to result. Panel deed terms typically run three to five years, with creative refresh provisions anchored to annual planning cycles. The ratio is roughly one order of magnitude. An agency that commissions a BETA messaging trial in March and receives results in June is operating under a deed whose next formal creative refresh opportunity may not arise until the following financial year's planning cycle — assuming the refresh provision is triggered at all.

Even where agencies have established informal relationships between their behavioural insights units and their communications directorates — and several have — the absence of a formal evidence hand-off mechanism in deed schedules means that integrating trial results requires one of two things: a deed variation, with its associated probity exposure and documentation burden, or a new procurement, which resets the timeline entirely and typically takes longer than the remaining campaign period.

The practical consequence is that validated behavioural findings accumulate in published reports and internal briefing documents while the campaigns those findings were designed to improve continue running creative that was developed against assumptions the experiments have since qualified or contradicted. The 12–24 month figure is not a ceiling on the lag; it is a reasonable estimate for the minimum time between a trial result and the next compliant opportunity to act on it through a coordinated panel arrangement.

The ANAO's 2021 performance audit of government advertising identified deficiencies in pre-campaign testing rigour as a finding. It did not name the procurement structure as a contributing cause. This omission is not a criticism of the ANAO's work — the audit was scoped to advertising practice, not to deed architecture — but it has had a consequential effect on the reform agenda. Because the structural driver was not named by the oversight body with the clearest mandate to surface it, the systemic problem has been addressed through capability investment (more BI staff, more trial capacity) rather than through contractual reform. Additional trial capacity that cannot reach market through panel arrangements does not reduce the lag; it increases the volume of stranded evidence.

The ANAO identified deficient pre-campaign testing without ever naming the procurement structure as the mechanism producing that deficiency.

The UK Government Communication Service's OASIS (Objectives, Audience insight, Strategy, Implementation, Scoring) framework is sometimes cited in Australian government communications contexts as a best-practice planning model. What is less frequently observed is that OASIS was designed to operate in conjunction with GDS's A/B testing infrastructure — and that the combination required specific contractual provisions that the framework alone does not supply.

GDS documented its approach to in-flight creative iteration across a sequence of blog posts published between 2018 and 2023. The core architectural decision was to treat creative variants not as fixed deliverables against a brief but as versioned artefacts subject to continuous evidence review. A creative execution that entered market was treated as a hypothesis. If A/B or multivariate testing against a pre-specified confidence threshold produced evidence that an alternative execution performed materially better against the campaign's primary metric, the alternative could be substituted without triggering the full variation process that would otherwise apply to a change in creative deliverables.

GDS achieved this within existing supplier panel arrangements — not by removing panels or circumventing procurement rules — by inserting 'test and learn' schedules into supplier agreements. These schedules defined the evidence standard that would constitute a compliant basis for creative substitution: minimum sample size, randomisation method, confidence interval, and the specific metrics against which performance would be assessed. By transferring the probity risk from the variation decision itself to the evidence standard, GDS resolved the compliance dilemma that currently confronts Australian communications directors. A creative change was compliant if and only if it met the defined evidence standard. The probity question became auditable rather than judgement-dependent.

The key contractual innovation — one that Australian panel deed templates do not currently reflect — is the separation of the creative brief from the creative execution. The brief remains fixed for procurement purposes: it defines the campaign objective, target audience, strategic direction, and mandatory compliance requirements. The execution is treated as a hypothesis subject to in-flight revision where evidence meets the threshold. Australian deed templates currently treat both as a single fixed deliverable, which is why the variation clause is the only available instrument for any mid-cycle change, regardless of its basis.

The GDS model does not require abandoning coordinated procurement, reducing supplier accountability, or creating special arrangements for individual agencies. It requires a schedule amendment to existing deed templates — a mechanism that Australian procurement teams have used before and have the institutional capacity to execute.

The United States Digital Service's 2019 working paper on Delivery-Driven Policy articulates a contract design condition that is directly relevant to the Australian federal context. The central argument is that policy — including campaign policy — should be governed by delivery evidence rather than by assumptions fixed at policy-setting stage. The contracting implication is specific: the contracting authority must pre-authorise a defined class of evidence as sufficient to trigger a change order without full re-procurement.

In the USDS model, this pre-authorisation takes the form of a methodological schedule appended to the contract. The schedule defines what counts as admissible evidence for the purposes of a compliant change order: the experimental design requirements (randomisation, control group specification), the minimum sample size, the confidence interval, and the outcome metrics against which the evidence will be assessed. A change order supported by evidence meeting the schedule standard is treated as compliant with the value-for-money obligation by definition — because the contracting authority has already made the value-for-money determination by establishing the evidence standard.

Applied to the Australian federal context, this approach would require the Digital Transformation Agency, in coordination with the Department of Finance's Whole of Australian Government communications function, to develop what might be termed a Methodological Evidence Schedule — analogous in form to an evaluation framework schedule, but specifying the admissibility standard for behavioural evidence rather than the post-campaign evaluation criteria. BETA's published trial protocols, which already specify randomisation method, sample size rationale, and confidence intervals, would provide a ready-made template for the schedule's technical specifications.

Service Victoria's customer experience team has operated closer to this model than any other Australian government entity, applying iterative evidence review to service delivery design and, in some cases, to communications assets. The practice exists without the formal deed schedule instrument that would make it reproducible, auditable, and transferable to other agencies. The absence of the instrument means that Service Victoria's approach depends on the continued presence of individuals with the judgement to navigate the compliance boundary informally — a fragile foundation for what should be a durable institutional capability.

The structural gap between behavioural insights outputs and paid media creative can be traced to three specific instruments in coordinated panel deed structures. Identifying them precisely matters because reform directed at the wrong mechanism will not resolve the problem, and because the reform that is required is considerably less disruptive than the scope of the problem might suggest.

First: Creative refresh provisions. As currently drafted in coordinated panel arrangements, creative refresh provisions define refresh triggers in terms of campaign phases and calendar milestones. A refresh is initiated when the campaign moves from awareness to conversion phase, or when the annual planning cycle produces a new brief, or when a defined budget threshold is reached. What the provisions do not specify — and what they would need to specify for the evidence pipeline to function — is an evidence threshold. There is no procedural basis in current deed language for a refresh triggered by the availability of validated behavioural trial results, even where the communications director, the BI unit, and the panel supplier are in full agreement that the results warrant a creative change. The procedural gap eliminates the option regardless of the substantive case for it.

Second: Evaluation framework schedules. These schedules, agreed at deed execution, specify the metrics against which campaign performance will be assessed and the evidence sources that will be used in that assessment. They govern not only post-campaign reporting but, more significantly, the utilisation decisions that determine which panel suppliers are engaged for subsequent campaigns. Behavioural trial results produced by BETA or BX NSW during a campaign period have no formal standing in these schedules as currently drafted. They are not named as admissible evidence sources. Their absence from the schedule means that a communications director who informally acts on BI trial results has no mechanism for documenting that decision in the formal evaluation record — which means the evidence does not enter the institutional memory that governs future procurement decisions. The pipeline is blocked not only at the creative change stage but at the evaluation stage that would otherwise create incentives for future integration.

Third: Deed variation clauses. These are the only existing mechanism for any mid-cycle change to creative deliverables that falls outside the defined refresh provisions. A deed variation requires a documented value-for-money case. The Commonwealth Procurement Rules and Finance guidance documents specify what constitutes a value-for-money case for variations arising from changed scope, changed budget, changed campaign objectives, or market testing. They do not specify how to construct a value-for-money case when the basis for change is experimental evidence — a trial result showing that a different message frame produces a higher compliance rate among the target population than the current in-market creative.

The absence of that specification is not a trivial gap. It means that two communications directors in different agencies, facing identical evidence of the same quality, may reach different conclusions about whether a variation is compliant — and both may be correct, because the guidance is genuinely ambiguous. It creates inconsistent treatment across agencies, significant probity uncertainty for individual decision-makers, and a rational incentive to avoid the variation process entirely. The de facto outcome is that validated evidence is set aside not because anyone has decided it lacks merit, but because the compliance pathway for acting on it has not been specified.

The reform required is contractual and procedural, not cultural or capability-based. It does not require new institutions, additional investment in behavioural insights capacity, or changes to the Commonwealth Procurement Rules at the legislative level. It requires three specific instruments, each of which has a clear institutional home and a demonstrated precedent for execution.

A Behavioural Evidence Schedule appended to coordinated panel deeds. This schedule would establish a defined evidence standard — specifying minimum sample size, randomisation method, blinding requirements, confidence interval threshold, and the outcome metrics against which behavioural evidence will be assessed — under which a validated BI trial result constitutes a compliant basis for creative variation without requiring a full deed variation process. The schedule would be developed by DTA in consultation with BETA and, at the state level, by Finance NSW in consultation with BX NSW and by the Victorian Department of Premier and Cabinet in consultation with Service Victoria's CX function.

The schedule would specify that a creative variation supported by trial results meeting the defined evidence standard satisfies the value-for-money obligation under the Commonwealth Procurement Rules — resolving the probity uncertainty at its source rather than leaving it to individual communications directors to navigate case by case. The evidence standard itself becomes the value-for-money determination, consistent with the USDS Delivery-Driven Policy model.

Updated Commonwealth Procurement Rules guidance from Finance. The schedule instrument requires a corresponding guidance document clarifying that evidence-triggered creative variations meeting the Behavioural Evidence Schedule standard satisfy the value-for-money obligation. The legislative hook for this guidance already exists: Finance has previously issued guidance clarifying the interaction between value-for-money obligations and accessibility compliance requirements under WCAG, establishing that accessibility uplift variations meeting defined technical standards are compliant without requiring a full variation process. The same mechanism applies here. The guidance does not require a change to the Rules themselves; it requires Finance to exercise the interpretive function it has already exercised in the accessibility context.

Amended evaluation framework schedules to include BI trial results as admissible evidence. The evaluation framework schedules attached to coordinated panel deeds would be amended to name behavioural insights trial results — meeting the Behavioural Evidence Schedule standard — as admissible evidence sources for the purposes of mid-campaign review and post-campaign evaluation. This amendment would create the formal record-keeping pathway that currently does not exist, enabling communications directors to document evidence-based creative decisions in the evaluation record and creating institutional memory that influences future procurement decisions.

At the state level, NSW and Victoria represent the natural pilots. BX NSW's 2022–23 Annual Report data set — more than 40 completed trials, many with direct relevance to communications campaigns — would immediately support a pilot Behavioural Evidence Schedule without requiring additional trial investment. The question is not whether the evidence exists to populate the schedule; it is whether Finance NSW and the Victorian Department of Premier and Cabinet will prioritise the deed amendment process required to give that evidence a formal entry point into coordinated procurement.

The argument that deed schedules are too rigid to accommodate a new evidence standard, or that the Commonwealth Procurement Rules create an insuperable barrier to evidence-triggered creative variations, is not supported by the recent history of coordinated procurement amendments. WCAG 2.2 compliance requirements were incorporated into paid media creative obligations through evaluation framework schedule amendments without full re-procurement of panel arrangements.

The mechanism used was a schedule amendment published by the coordinating authority — DTA, in coordination with Finance — with a defined transition period and compliance guidance issued to both agencies and panel suppliers. Panel suppliers were required to demonstrate WCAG compliance capability as a condition of continued utilisation. The amendment did not require agencies to re-run competitive processes for creative services already under deed. It required the coordinating authority to exercise its administrative function of maintaining and updating schedule instruments.

That mechanism is directly transferable to a Behavioural Evidence Schedule. The institutional memory for executing it exists within DTA and Finance procurement teams. The legal basis — the coordinating authority's power to issue and amend schedule instruments — is the same. The stakeholder engagement process required would involve the same set of panel suppliers and agency communications directorates already engaged in the WCAG transition.

The accessibility parallel is useful for a second reason. WCAG amendments were accepted by panel suppliers because they defined a technical standard that suppliers could meet and demonstrate — the compliance obligation was specific, auditable, and applied uniformly. A Behavioural Evidence Schedule would have the same properties: the evidence standard is specific (defined sample size, randomisation method, confidence interval), auditable (trial results are published or can be made available to the coordinating authority), and applies uniformly to any agency seeking to invoke the evidence-triggered variation pathway.

What the accessibility parallel also demonstrates is the importance of the coordinating authority's role as the instrument-issuing body. Individual agencies cannot unilaterally amend deed schedules. The reform requires DTA and Finance to act in their coordinating capacity — a governance point that is sometimes obscured by the framing of this as an agency-level communications problem rather than a whole-of-government procurement architecture problem.

The position of a communications director in a major federal agency who has read the relevant BETA trial results is, under current arrangements, genuinely untenable. Proceeding with assumption-based creative when validated experimental evidence is available and relevant cannot comfortably be described as meeting a reasonable standard of evidence rigour — particularly as the ANAO has signalled, through its 2021 performance audit, increasing scrutiny of pre-campaign testing practice. The audit did not specify what constitutes adequate pre-campaign testing, but the direction of travel in ANAO scrutiny of government advertising is clear.

At the same time, the panel deed under which that communications director procures creative services does not provide a compliant pathway for acting on the available evidence. The deed variation process exists but its application to evidence-triggered changes is unspecified. The creative refresh provision exists but its triggers do not include evidence thresholds. The evaluation framework schedule exists but does not admit BI trial results as evidence sources. The communications director is in compliance difficulty whether they act on the evidence or ignore it.

The political and reputational risk of a poorly performing public health campaign, a compliance messaging campaign that fails to shift behaviour, or a public information campaign built on messaging assumptions that a contemporaneous randomised trial has qualified — that risk is borne by the minister and the agency. It is not borne by the panel supplier, whose deed does not require evidence integration and who has met all contractual obligations by delivering creative against the agreed brief. This asymmetry of accountability is a design feature of the current governance architecture, not a failure of individual judgement on the part of suppliers or agency staff.

The accountability asymmetry creates a predictable institutional behaviour: agencies invest in behavioural insights capability to demonstrate evidence rigour, publish the trial results to demonstrate transparency, and then commission paid media campaigns under deed arrangements that cannot ingest those results — at which point the evidence investment and the media investment proceed on parallel tracks that do not intersect. The citizen receives a campaign whose creative was developed without the benefit of the evidence the government paid to generate.

Resolving this requires naming the problem precisely. It is a contractual and governance problem. The three binding constraints — creative refresh provisions without evidence thresholds, evaluation framework schedules that exclude BI trial results, and deed variation clauses with no specified pathway for evidence-triggered changes — are specific instruments that can be amended through mechanisms that already exist and have been used. The reform does not require new legislation, new institutions, or new capability investment. It requires the coordinating authority to exercise its existing instrument-issuing function with a clear understanding of what the instruments need to say.

Directing reform effort at capability uplift, cultural change programmes, or additional investment in behavioural insights trial capacity — without addressing the contractual architecture — will increase the volume of stranded evidence without reducing the lag between evidence generation and citizen-facing creative. The pipeline is not blocked by a shortage of evidence. It is blocked by three specific deed mechanisms that can be identified, drafted against, and amended.

SoudCoh works with government communications teams to map the specific deed provisions constraining evidence integration and draft the schedule language required to resolve them within existing paid media and social advertising procurement frameworks. For teams seeking a structured review of their current panel deed constraints and a pathway to evidence-integrated campaign delivery, the starting point is a deed provision audit against the three mechanisms identified in this piece.