The Digital Finance Cooperative Research Centre (DFCRC) is a 10 year $180 million research programme funded by the Australian Government and industry. Their role is to research and trial next-generation financial infrastructure (e.g. asset tokenisation, CBDC, and market design). We were engaged to consult on data strategy and AI-augmented analytics for novel financial use cases drawing on large datasets.
Objectives & outcomes
The DFCRC engaged us over the span of a year to design and deliver the data architecture for financial decision makers of the future. The brief: work with PHD researchers and industry partners to integrate data so that AI can sit on top of it to safely enhance decision making.
Key outcomes targeted:
- •Single, governed data foundation to integrate internal, partner and public datasets.
- •AI systems sitting on top of the data foundation capable of safely enhancing decision making.
- •The ability to build reports from financial data in an automated way.
Context, constraints & success criteria
The programme spanned multiple institutions with different data formats, and research priorities. Data included internal decision making criteria, external financial data, macroeconomic data, and live market data.
It became apparent that the project would have to deal with a rapidly changing landscape for AI integration, and so we needed the data to be designed in a way that would allow for interchangeable integration of whatever new models and techniques were developed. We experimented with a variety of approaches from training models, to RAG, and novel ways of structuring the input data.
The primary constraint was the need to work with stakeholders across multiple institutions, with different data formats, and research priorities. We handles this by building for scalability from the start. This enabled us to quickly iterate, test new ideas and research different approaches with very little re-engineering of the data.
Approach
Engagement model
Discovery + Consultation with researchers and industry partners.
We started by understanding the landscape and needs of the decision makers, in this case investment committees. After understanding the data that would be needed, we found two benefits to a best-practice, modern data architecture approach.
- •The ability to automate much of the process from data to insights. This allows for much faster insights and less labour hours to be spent on manually finding and converting data.
- •The ability to search for insights semantically, i.e. with AI assistance. This allows for extremely fast discovery of supporting information and had the ability to drastically improve decision making.
Build & phases
Three concurrent tracks produced quick wins while laying a scalable foundation.
Discovery
Stakeholder mapping, data inventory, governance & risk review, success metrics.
- Catalogue sources (internal, partner, public)
- Agree evaluation criteria (accuracy, freshness, explainability, cost, latency).
- Baseline current state: lineage, data quality, reporting gaps and duplication.
Design
Architecture, standards and operating model for repeatable research pilots and iterations.
- Target architecture: cloud data platform, fact/dimension tables, analytics and AI layer.
- Governance: RBAC, testing, lineage and cost guardrails.
- AI patterns: RAG, anomaly detection, structured JSON outputs.
Delivery
Building the “thin slice”, then scale by template.
- Quickly standing up the priority datasets and low hanging fruit.
- Meeting initial model accuracy benchmarks.
Iteration
Scaling out to additional datasets and use cases.
- Idempotency and retries for data ingestion and model training.
- Evaluation, backtesting and human-in-the-loop review of model accuracy.
- Documentation of the data and model for transparency and auditability.
Selected results
- Successfully showed that integrated data and AI can be used to enhance decision making for investment committees.
- Integrated data from internal decision making criteria, external financial data, macroeconomic data, and live market data into a single, governed data foundation.
- Successfully demonstrated the ability to search for insights semantically, i.e. with AI assistance.
- Successfully demonstrated the ability to build reports from financial data in an automated way.
Solution overview
Data platform. Cloud warehouse as the system of analysis; ingestion via ELT; modeling with dbt; automated tests; lineage capture; cost/freshness monitors. A governed semantic layer exposes certified metrics to BI tools and notebooks.
AI-enhanced analysis. Retrieval-augmented generation and data documentation; anomaly/outlier detection for time-series; templated executive summaries that include links to underlying models, assumptions and sources.
Custom dashboard. Role-based access control (RBAC) to ensure each user only sees the data and actions appropriate to their role. Built with Next.js, the dashboard is fast, secure, and includes custom chatbots for guided analysis.
Notable benefits
What this means for financial decision-makers:
- Trustworthy numbers: consistent definitions with tests and lineage embedded.
- Faster insight: analysts spend time evaluating hypotheses, not wrangling data.
- Explainable AI: every AI-assisted answer includes citations and guardrails.
- Comparability: pilots use the same metrics, enabling apples-to-apples assessment.
- Scalability: new datasets and partners plug into contracts and standards.
- Security by design: RBAC ensures least-privilege access to sensitive data.
- Lower total cost: reusable assets and automated quality checks reduce run costs.
AI is an accelerant, not a substitute. In this context it helps domain experts see more, sooner.
The Kali Software team