> For the complete documentation index, see [llms.txt](https://experience-digital-transform.gitbook.io/experience-digital-transformation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://experience-digital-transform.gitbook.io/experience-digital-transformation/analytics/technical-specifications/6.-functional-requirements.md). # 6. Functional Requirements Detailed functional requirements derived from the Key Digital Functionalities, each tagged REQUIRED or RECOMMENDED. {% hint style="success" %} ✅ **Your conformance checklist.** Each requirement traces back to a Key Digital Functionality in Section 4 — REQUIRED items are non-negotiable for a conformant deployment. {% endhint %} The following functional requirements are derived from the Key Digital Functionalities defined in Chapter 4. Each requirement is tagged REQUIRED or RECOMMENDED. Implementers must satisfy all REQUIRED items for a conformant deployment. ## 6.1 Data Ingestion Interface * The Analytics BB must expose a standardised API endpoint for receiving harmonised datasets from upstream ingestion and registry Building Blocks. **REQUIRED** * The intake interface must perform schema validation against the active data-catalogue schema at the point of ingestion and reject non-conformant payloads with structured error responses. **REQUIRED** * The intake interface must support batch file delivery (including spreadsheet uploads), event-driven micro-batch streaming, and API-based delivery. **REQUIRED** * The interface must generate a unique ingestion receipt token for each accepted payload, enabling end-to-end lineage tracing. **REQUIRED** * The intake interface must support idempotent re-delivery, ensuring duplicate payloads do not create duplicate records in the Silver layer. **RECOMMENDED** ## 6.2 Data Lakehouse Storage and Governance * The Building Block must maintain logically separated Bronze, Silver, and Gold storage layers using open table formats with ACID compliance (e.g., Apache Iceberg or Delta Lake). **REQUIRED** * The Silver layer must implement a Golden Records concept, enforcing deduplication and master-data-management rules across all ingested entity types. **REQUIRED** * A unified metadata catalogue must be maintained covering table schemas, field semantics, shared code sets and domain terminologies, cross-layer data lineage, data-quality scores, and access classification. **REQUIRED** * Schema evolution must be managed through versioned migration scripts; backward-incompatible changes must be validated against downstream mart dependencies before deployment. **REQUIRED** * The unified governance layer must enforce fine-grained access control across all storage layers, including Role-Based Access Control (RBAC), row- and column-level security, and dataset-level permission management. **REQUIRED** * Data retention, backup and archival policies must be configurable per dataset and per logical layer, with automated archival to cold storage upon policy expiry. **RECOMMENDED** ## 6.3 Transformation and Orchestration * The Building Block must support declarative pipeline definitions specifying source tables, transformation logic, target marts, and scheduling parameters. **REQUIRED** * Pipelines must apply data validation, cleaning and reconciliation as data is promoted from Bronze to Silver and from Silver to Gold, recording validation outcomes and rejected records. **REQUIRED** * All pipeline executions must be logged with start time, end time, records processed, records rejected, and final status. **REQUIRED** * Failed pipeline runs must trigger automated alerting to designated operations contacts with sufficient diagnostic context for root-cause analysis. **REQUIRED** * The orchestration layer must support event-driven pipeline triggering in addition to time-based scheduling. **RECOMMENDED** * The Building Block should support lineage capture from Silver source tables through all transformation steps to Gold mart targets. **REQUIRED** ## 6.4 Multi-Source Reconciliation and Validation * The Building Block must store multiple estimates for the same analytical unit — one row per source — each retaining its own value, confidence and known limitations, without merging them at ingestion. **REQUIRED** * The Building Block must compute, per unit, the discrepancy between sources against a configurable threshold and flag units that diverge beyond it for adjudication. **REQUIRED** * Reconciliation rules (matching, thresholds, survivorship and blend rules) must be externally configurable and explainable to non-technical data stewards. **REQUIRED** * The authoritative value for a unit must be recorded as an append-only decision (selected source or documented blend) with rationale, responsible actor and timestamp; reversal must create a new record rather than overwrite. **REQUIRED** * The Building Block must preserve the designation of "official versus supplementary" for each reconciled unit and expose it to downstream analytics. **REQUIRED** * Where automated reconciliation is applied, the Building Block must support governed human review, deferring the final designation to a designated officer where required. **REQUIRED** ## 6.5 Analytical Query Engine * The Building Block must provide a federated query engine capable of executing MPP SQL queries across distributed Silver and Gold layer storage without data movement. **REQUIRED** * The Building Block must provide a columnar OLAP engine delivering sub-second query responses against pre-materialised Gold mart datasets. **REQUIRED** * All query and serving engines (Federated, Columnar OLAP, and OLTP Serving) must share a unified access-control enforcement layer applying RBAC, ABAC, and PBAC policies at execution time. **REQUIRED** * The query layer must support geospatial and Earth-observation data types (vector and raster) and spatial predicates for climate and environmental analysis. **REQUIRED** * The query layer must support spatial data types and administrative boundaries (e.g., regional administrative units, school district geofences) to enable geographic and spatial analysis of education metrics. **REQUIRED** * Heavy analytical workloads on the federated engine must not degrade SLA-bound dashboard queries on the OLAP engine. **REQUIRED** * The Building Block must provide a dedicated, high-concurrency OLTP serving engine (e.g., PostgreSQL) designed to store and serve pre-aggregated results and lightweight Gold data marts to data visualization platforms with sub-second response times. **REQUIRED** * Query results exceeding configurable size thresholds must be paginated or delivered asynchronously via a secure download endpoint. **RECOMMENDED** ## 6.6 AI-Assisted Query Synthesis and Recommendations * The Building Block must provide a natural-language query interface that accepts free-text prompts and returns structured SQL directed at the Silver or Gold layers. **REQUIRED** * The module must use semantic metadata retrieval (RAG) to ground SQL generation in the active data catalogue, preventing hallucinated table or column references. **REQUIRED** * Every generated SQL query must be passed through an AST-level security enforcement layer before execution. Any DML or DDL operation must be blocked unconditionally. **REQUIRED** * ABAC/RBAC row-level and column-level security policies must be dynamically injected into the query AST before execution, ensuring no user can retrieve data beyond their authorised scope. **REQUIRED** * Any AI-generated data-quality, reconciliation, mapping or anomaly recommendation must be presented as advisory and require explicit human validation before an official value is approved. **REQUIRED** * The module must present the generated SQL to the user before execution, with an option to review and reject, and results should be formatted as interactive tables, charts or maps. **RECOMMENDED** ## 6.7 Predictive Analytics and MLOps * The Building Block must provision isolated, multi-tenant sandbox environments for data-science workloads with configurable resource quotas. **REQUIRED** * Sandboxes must have read access to Silver and Gold layer datasets without write access to production storage. **REQUIRED** * The MLOps pipeline must support automated feature selection, distributed model training, accuracy validation, and model cataloguing. **REQUIRED** * Trained models must be deployable as containerised microservices exposing a standardised scoring API. **REQUIRED** * Model performance metrics and drift indicators must be monitored continuously post-deployment, with automated alerting on degradation. **RECOMMENDED** ## 6.8 Proactive Workflow Trigger and Event Notification * The Building Block must allow definition of threshold rules and event conditions against Gold-layer scoring or reconciliation outputs that, when met, dispatch execution signals to downstream systems. **REQUIRED** * Trigger and publication signals must be dispatched via authenticated, audited API calls or webhooks to registered downstream endpoints when data is published, promoted or updated. **REQUIRED** * Each dispatched signal must include: trigger identifier, triggering condition, subject/unit identifier, timestamp, and proposed action type. **REQUIRED** * Trigger rules must be version-controlled and require authorised approval before activation. **RECOMMENDED** ## 6.9 Identity and Access Management Integration * The Building Block must not store local user credentials. All authentication must be delegated to the GovStack Identity / Access Management Building Block using OpenID Connect or SAML 2.0. **REQUIRED** * User sessions must carry signed, verifiable tokens that the Analytics BB validates locally without calling the identity provider on each request. **REQUIRED** * Role assignments (Data Steward, Analyst, Software Developer, Administrator) must be managed through the Identity / Access Management BB and reflected in the Analytics BB policy engine. **REQUIRED** ## 6.10 Audit and Immutable Logging * Every query execution and every reconciliation/source-selection decision against Silver or Gold data must generate an audit-log entry containing: record identifiers accessed or decided, fields read, user identity, role, timestamp, and stated justification. **REQUIRED** * Audit logs must use cryptographic record chaining so that any tampering or deletion of a log entry is detectable. **REQUIRED** * Super-administrative accounts must be technically incapable of deleting or modifying audit-log entries. **REQUIRED** * Audit logs must be exportable in a machine-readable format for regulatory inspection. **RECOMMENDED** ## 6.11 Resilient Cloud Infrastructure * All infrastructure components must be defined as version-controlled Infrastructure as Code configurations. **REQUIRED** * The platform must support deployment across a minimum of three independent Availability Zones with synchronised cross-zone storage replication and automated failover, and should support geo-replication across regions to ensure high availability for national education services and regional administrative units. **REQUIRED** * Ephemeral compute capacity for batch ETL/ELT jobs must be automatically provisioned on job start and released on job completion. **REQUIRED** * A unified monitoring and alerting toolset must track platform health, performance telemetry, and security events in real time. **REQUIRED** * Orphaned or underutilised compute resources must be automatically detected and reclaimed. **RECOMMENDED** --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://experience-digital-transform.gitbook.io/experience-digital-transformation/analytics/technical-specifications/6.-functional-requirements.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.