> For the complete documentation index, see [llms.txt](https://experience-digital-transform.gitbook.io/experience-digital-transformation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://experience-digital-transform.gitbook.io/experience-digital-transformation/analytics/technical-specifications/11.-internal-workflows-and-data-lifecycle-appendix.md).

# 11. Internal Workflows and Data Lifecycle (Appendix)

Appendix — the end-to-end data lifecycle: ingestion and harmonisation, reconciliation and sovereign decision, and aggregation and proactive orchestration.

{% hint style="info" %}
🔄 **Three stages, ingestion to insight:** harmonise, reconcile, orchestrate.
{% endhint %}

This section outlines the end-to-end data lifecycle, illustrating how the Analytics BB interacts with adjacent GovStack components to reconcile data and orchestrate proactive workflows securely.

## 11.1 Ingestion and Harmonisation Pipeline

* **Event capture.** Change Data Capture (CDC) mechanisms or scheduled/file feeds detect new or changed data in upstream systems.
* **Pseudonymisation (where applicable).** Before data reaches the analytical perimeter, the ingestion/registry layer and the Information Mediator apply irreversible cryptographic hashing to any PII. For climate datasets, which are typically non-personal, this step handles sensitive or licensed-source restrictions instead.
* **Silver-layer harmonisation.** The Analytics BB receives the payload via the API intake interface, validates and cleans it, and merges it into the Golden Record and estimate tables, updating the historical state.

## 11.2 Reconciliation and Sovereign Decision

* **Multi-source assembly.** For each analytical unit, every available source is stored as a separate estimate, each with its own value, confidence and known limitations.
* **Gap analysis.** The Building Block computes the discrepancy per unit against the agreed threshold; units that agree are surfaced as low-risk, units that diverge are queued for adjudication.
* **Governed human review.** A steward or officer designates the authoritative source (or a documented blend) and records a rationale; the decision is written append-only with identity and timestamp, and is reversible.

## 11.3 Aggregation and Proactive Orchestration

* **Gold-layer modelling.** The orchestrator runs batch ELT pipelines to transform validated Silver records into Gold-layer data marts (e.g. a validated inventory or indicator mart).
* **Scoring and monitoring.** MLOps microservices evaluate Gold tables against models or policy matrices; reconciliation monitors re-flag units whose figures shift.
* **Dispatch and notify.** When a condition is met, the proactive-trigger module dispatches a standardised event to the Workflow BB, or a webhook to consuming systems when data is published or promoted.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://experience-digital-transform.gitbook.io/experience-digital-transformation/analytics/technical-specifications/11.-internal-workflows-and-data-lifecycle-appendix.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.