The Hawkfry data platform toolkit is how we build a new data platform for a client. It packages the way we set up the cloud, load the data, model it and test it into steps that an AI carries out and an engineer approves. Every client gets the same well-built platform, and their own team can run and extend it once we have handed over.
Why Building a Data Platform Was Slow
Before the toolkit, setting up a client's data platform was manual work that lived in our consultants' heads.
Every engagement repeated the same groundwork from memory: the cloud projects, the permissions, the data warehouse, the conventions for naming and testing. It took days to get to the first working piece, and each platform came out slightly different from the last.
That is a poor use of an experienced engineer's time, and it is a problem for the client too. A platform that only makes sense to the person who built it is hard to hand over.
What the Data Platform Toolkit Is
The toolkit holds what we know about building a platform in three forms.
- A ready-made project to copy. The folder layout, the automated checks and the operating guides are stamped into the client's own repository on day one, so every platform starts from the same shape.
- Written methods for an AI to follow. For work that needs judgement, such as investigating a source system or modelling a table, the toolkit gives an AI coding agent our method step by step.
- Rules that are enforced. Some things are not left to memory or to goodwill. Formatting, testing standards and security defaults are checked automatically every time something changes.
The video at the top of this page shows every file in the toolkit, grouped by what it does. Of the 272 files, 77 are for the AI: instructions on how to work and examples of what good looks like. The other 195 build the platform, keep it safe, and explain it to the people who run it.
An engineer runs the build with the AI doing most of the typing. The engineer's job is to make the decisions and to say yes or no at each checkpoint.
The Five Stages of a Data Platform Build
A platform built in one unbroken run hides which step actually went wrong. So the toolkit builds in five stages, and each one ends with a check that proves the stage before it worked.
Building the platform
- 1
Cloud foundations
Separate development, test and production environments, all created from code.
Check: Every environment builds cleanly.
- 2
Prove the connections
A one-row example is run end to end, through every permission it needs.
Check: The example runs.
- Infrastructure up and working
- 3
Load the raw data
One loader per source system. Its data is reviewed before it reaches the warehouse.
Check: Every source has data in the warehouse.
- 4
Clean and check each source
The AI works out what one row really represents, writes the cleaned table and validates it against the source.
Check: Every cleaned table passes its tests.
- All sources cleaned and tested
- 5
Make it run by itself
Daily schedules, alerts when a load fails, and automated checks on every future change.
Check: The checks pass and someone else could take it over.
- Platform running on its own
The checks take minutes. Skipping one is a false economy, because a problem missed at one stage tends to surface two stages later, in a place that looks unrelated.
Building Reports One Business Question at a Time
Once the platform is running, we turn the data into the numbers the business relies on.
We do this one business question at a time, and nothing is built until the success criteria are written down and agreed. That means what one row represents, which existing figures the result has to match and how closely, and what is out of scope.
If those cannot be written down, the question is not ready, and the right move is another conversation with the business. Once they are agreed, the AI builds the model, checks its output against the existing report, and a person from the business signs it off in the test environment before it goes live.
A business question comes in
- What they need to know
- What one row means
- Figures to match
- What is out of scope
Phase 01
Prep
Agree what a correct answer looks like
- What one row of the result represents
- Which existing figures it must match, and how closely
- The edge cases that must hold
- What is out of scope
The business agrees
Do these criteria capture the question? Nothing is built until the answer is yes.
Phase 02
Build
One table at a time
- Lay out the tables the question needs
- Choose how each one is stored
- Write the model
- Add the tests
Automatic check
Every model builds cleanly.
Phase 03
Validate
Prove it against the criteria
- Audit the finished table for gaps, broken links and personal data
- Reconcile it against the existing report
- Match every criterion to a check
Automatic check
Every success criterion has a check that passes.
Phase 04
Ship
Into the business's hands
- Review the change
- Put it in the test environment
- The business checks it against their own numbers
The business signs off
Signed off by the business, then it goes live.
Where the AI Helps and Where People Decide
The AI does a great deal of the work, but it is set up to behave differently from an engineer following a checklist.
Asks first
Questions before code
It does not work silently. It checks its inputs and stops at every decision that belongs to the business.
Some records point to a customer who does not exist. Should they be grouped as unknown, or left out?
- Group them as unknown (recommended)
- Leave them out entirely
Leaves a record
Real files, not a chat
Every investigation, model and test is saved in the client's repository, where it can be reviewed like any other change.
Checks itself
Tests its own work at every level
From the cloud setup to the final report, nothing it builds goes untested. That rule holds even when it is working on its own.
Gets better
The toolkit grows from the work
When a build turns up a new kind of check, it is added to the toolkit for the next client. Nothing specific to a client ever goes back in.
What the Client Ends Up With
We want a client to be able to run and extend their platform without us, so we keep it deliberately ordinary.
- An ordinary Google Cloud platform in their own account, with BigQuery as the warehouse.
- Everything defined in code in their own repository. Nothing is created by hand or hidden in a console.
- Separate development, test and production environments, set up so that a change still under review cannot touch production data.
- No long-lived keys for deployments. Automated deployments use short-lived credentials, and each data loader can see only its own credentials and its own data.
- Runbooks and ways of working written for their team.
- The AI instructions that built it, so their own people can keep building the same way.
The toolkit deliberately stops at clean, tested, modelled data. Dashboards and reporting tools are a separate decision that depends on the client, and data policies such as ownership and retention remain theirs to set.
The First Build
The first platform we built this way was for a client that had nothing on Google Cloud when we started.
The build took about six weeks to handover. Before we handed over, figures from the new platform were checked against the client's old reporting.
After watching us work in the first workshop, their team asked to adopt the same AI-assisted way of building. We left the toolkit in their repository so they could.
We have since used the toolkit for a client in a different industry, and each build has improved it.
Data Platform Toolkit FAQs
What is a data platform toolkit?
It is a packaged, repeatable way of building a data platform: a starting project, a set of methods and a set of enforced rules. Ours is built so that an AI can do most of the building while an engineer approves each stage.
How long does it take to build a data platform this way?
Our first build took about six weeks from start to handover. The time depends mostly on how many source systems there are and how quickly the business can agree what its numbers mean.
Does the AI build the platform unsupervised?
No. The AI stops at the end of every stage and waits for an engineer to confirm the result. Business decisions are put to a person, and a stakeholder signs off each report before it goes live.
Who owns the platform when it is finished?
The client does. It runs in their cloud account, the code is in their repository, and it uses ordinary Google Cloud services, so any competent data team can take it over.
Can our own team keep building after handover?
Yes. The runbooks, the ways of working and the AI instructions that built the platform all stay in your repository, so your team can add sources and models the same way we did. At our first build, the client's team asked to work this way after seeing it in the first workshop.
Which cloud does it work on?
Today it builds on Google Cloud with BigQuery as the warehouse.
Does it include dashboards?
No. It delivers clean, tested, modelled data. Which reporting tool sits on top is a separate choice we help clients make.
Need a Data Platform You Can Run Yourself?
If you are starting from scratch, or replacing something that only one person understands, we can build it with you and hand it over properly.
Tell us about your data, or read how we help clients build, operate and own their platforms.
