> For the complete documentation index, see [llms.txt](https://experience-digital-transform.gitbook.io/experience-digital-transformation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://experience-digital-transform.gitbook.io/experience-digital-transformation/analytics/technical-specifications/4.-key-digital-functionalities.md). # 4. Key Digital Functionalities The Key Digital Functionalities (KDFs) that define the minimum viable capabilities of the Analytics Building Block. {% hint style="success" %} 🧩 **Twelve functionalities, one Building Block** — from ingestion to AI-assisted query, each independently testable. {% endhint %} The Analytics Building Block must be accessible through standard web and API interfaces. It should support integration with third-party Business Intelligence tools, open-data portals, and automated workflow systems. The following Key Digital Functionalities (KDFs) define the minimum viable capabilities of the Analytics BB. ## 4.1 Data Ingestion Interface The Analytics BB must expose a standardised intake interface to receive harmonised data from upstream ingestion and registry Building Blocks. It shall not perform raw extraction or PII processing; these are delegated to adjacent Building Blocks. The intake interface must support batch and micro-batch delivery modes and enforce schema validation at the point of ingestion into the Silver layer. It must also support file-based delivery (including spreadsheets) for early-maturity deployments where automated integration is not yet available. ## 4.2 Data Lakehouse Storage and Governance The Building Block must define and maintain the conceptual and logical education data models alongside their physical realisations across three logical storage layers — Bronze (immutable archive), Silver (harmonised golden records), and Gold (aggregated data marts) — using open table formats with ACID (Atomicity, Consistency, Isolation, Durability) compliance and schema versioning. A unified governance layer must enforce metadata cataloguing, management of shared code sets and education domain terminologies, schema evolution policies, cross-layer lineage tracking, and configurable retention, backup and archival policies per dataset and layer, as well as fine-grained access control (including Role-Based Access Control \[RBAC], row- and column-level security, and dataset-level permission management across all layers). ## 4.3 Transformation and Orchestration The Building Block must support scheduled and event-driven batch transformation pipelines that promote data from Bronze to Silver and from Silver to Gold. As data moves across the layers, pipelines must apply data validation (schema, completeness, range and referential checks), cleaning (standardisation and de-duplication), and reconciliation (matching, survivorship and golden-record construction). Pipeline execution must be managed by an enterprise orchestrator, with full lineage capture and automated alerting on failure. ## 4.4 Multi-Source Reconciliation and Data Validation The Building Block must be able to hold, for a given analytical unit, several estimates of the same quantity — one per source — each retaining its own value, confidence and known limitations. It must reconcile these sources by quantifying and explaining the gap between them against an agreed threshold, rather than silently averaging them. Where sources diverge beyond threshold, the unit must be flagged for governed human review, and the authoritative value must be recorded as an explicit, append-only decision (selected source or documented blend) with a rationale, a responsible actor and a timestamp. Reconciliation rules (matching, thresholds and survivorship) must be transparent, explainable and configurable by data stewards, and the resulting designation of "official versus supplementary" evidence must be preserved. This capability underpins climate use cases in which national inventories and Earth-observation-derived estimates must be reconciled cell by cell. {% hint style="info" %} 🧑‍⚖️ **Governed human review.** Where sources diverge beyond the agreed threshold, the unit is flagged for review, and an officer designates the authoritative source (or a documented blend) as "official versus supplementary." The decision is append-only, reversible, and always a logged human judgement — never a silent average. {% endhint %} ## 4.5 Analytical Query Engine The Building Block must provide a multi-engine query capability: (a) a federated query engine for ad-hoc exploratory analysis across the Silver and Gold layers supporting MPP (Massively Parallel Processing) execution, (b) a high-performance columnar OLAP engine for sub-second dashboard and reporting queries against materialised Gold marts, and (c) a dedicated, high-concurrency OLTP serving engine for productionalization, designed to store and serve pre-aggregated results and lightweight Gold data marts to data visualization platforms with sub-second response times. All query and serving engines must share a unified access-control enforcement layer. The query layer must support geospatial and Earth-observation data types (vector and raster) for spatial analysis in climate and environmental use cases. ## 4.6 AI-Assisted Query Synthesis and Recommendations The Building Block must include a natural-language to SQL module that allows authenticated public officials to retrieve data-driven insights using plain-language prompts without SQL knowledge. The module must be governed by mandatory AST-level security enforcement blocking all DML/DDL operations and dynamically injecting RBAC/ABAC policies before execution. The module may additionally generate advisory recommendations — data-quality flags, reconciliation and mapping proposals, and anomaly detection — grounded in the data catalogue; all such AI outputs are advisory only and must be validated by a human before any official data model, registry record or reconciled value is approved. {% hint style="warning" %} 🤖 **AI is advisory, never authoritative.** Every AI-generated recommendation — data-quality flag, reconciliation proposal, mapping, anomaly detection — requires explicit human validation before any official value is approved. The system never lets a model decide on its own. {% endhint %} ## 4.7 Predictive Analytics and MLOps The Building Block must provide isolated multi-tenant sandbox environments for data-science workloads, supporting the full MLOps lifecycle including feature engineering, distributed model training, accuracy validation, model versioning, and deployment as containerised microservices for real-time scoring. ## 4.8 Proactive Workflow Trigger The Building Block must be capable of automatically dispatching execution signals to downstream external systems when predefined thresholds or event conditions are detected in scoring or reconciliation outputs — for example, a household vulnerability index falling below a policy threshold, or a reconciliation discrepancy exceeding the agreed threshold for a greenhouse-gas inventory cell (re-flagging it for review). This transforms service delivery and monitoring from reactive to proactive. ## 4.9 Identity and Access Management Integration The Building Block must not maintain local user credentials. It must fully delegate authentication to an external Identity / Access Management Building Block using federated SSO standards (OpenID Connect / SAML 2.0). Within the analytical perimeter, a hybrid RBAC + ABAC + PBAC model must enforce fine-grained, attribute-aware access at row and column level, with a documented role model (e.g. Data Steward, Analyst, Software Developer, Administrator) and segregation of duties. ## 4.10 Audit and Immutable Logging Every access to confidential or restricted data, and every reconciliation or source-selection decision, must be automatically logged. All audit logs must be cryptographically chained, making retroactive modification detectable and infeasible even for super-administrative accounts. Logs must capture the record identifier, fields accessed, subject/record token (where personal data applies), user identity and role, timestamp, and stated justification. {% hint style="danger" %} 🔐 **Audit logs are immutable.** Logs are cryptographically chained. No account — including super-administrative accounts — can modify or delete a log entry. {% endhint %} ## 4.11 Interoperability and Event Notification The Building Block must expose documented REST/OpenAPI interfaces with managed API versioning, authentication and rate limiting, and must integrate with the Information Mediator Building Block for inter-BB exchange. It must support event-driven notification (webhooks) so that consuming systems are informed when data is published, promoted or updated, reducing polling and enabling loosely coupled integration. ## 4.12 Resilient Cloud Infrastructure The Building Block must be deployed on cloud-native infrastructure distributed across a minimum of three independent Availability Zones with synchronised cross-zone storage replication, and should support geo-replication across regions to ensure high availability for national education services and regional administrative units. All infrastructure must be managed through GitOps and IaC practices. Ephemeral compute nodes must be automatically provisioned for heavy batch workloads and terminated on completion. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://experience-digital-transform.gitbook.io/experience-digital-transformation/analytics/technical-specifications/4.-key-digital-functionalities.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.