Data Mesh Architecture on Google Cloud: Principles and Implementation

The centralized data platform has a scaling problem. As organizations grow, the central data team becomes a bottleneck: every data pipeline request goes through them, domain teams wait weeks for their data to reach analysts, and the central team loses context about what the data actually means in each domain.

Data mesh proposes a different model: treat data as a product, distribute ownership to the domains that produce it, and federate governance so that each domain team manages its own data while adhering to organization-wide standards.

This isn't just an organizational change — it has concrete technical implications for how you design your GCP data infrastructure. This guide covers the practical implementation: BigQuery dataset organization by domain, Dataplex for federated governance, authorized views for cross-domain access, and Analytics Hub for data product publishing.

Data Mesh Principles and GCP Mapping

The four principles of data mesh, mapped to GCP implementation:

1. Domain-oriented decentralized data ownership Each business domain (Orders, Customers, Products, Logistics) owns its own data. On GCP, each domain gets its own GCP project (or at minimum, its own BigQuery dataset) with IAM policies controlled by the domain team.

2. Data as a product Domain data is treated as a product with a clear interface: defined schema, SLAs on freshness and quality, documentation, and ownership. On GCP, this maps to curated BigQuery tables or authorized views published through Analytics Hub.

3. Self-serve data infrastructure platform A central platform team provides the tools and patterns for domains to build data products, without the central team being involved in each domain's data work. On GCP, this is Dataplex for governance, Cloud Composer for pipeline orchestration templates, and Terraform modules for consistent project setup.

4. Federated computational governance Global policies (data retention, PII classification, access audit) are enforced consistently but without a central bottleneck. On GCP, this is organization policies, Dataplex data quality rules, and Security Command Center.

Project and Dataset Structure

Organization
├── platform-project (central platform team)
│   ├── Dataplex governance configurations
│   ├── Analytics Hub exchange
│   └── Shared infrastructure (VPC, DNS, monitoring)
│
├── domain-orders-project
│   ├── BigQuery: orders_raw (append-only ingestion)
│   ├── BigQuery: orders_curated (domain-owned, governed)
│   └── BigQuery: orders_products (published data products)
│
├── domain-customers-project
│   ├── BigQuery: customers_raw
│   ├── BigQuery: customers_curated
│   └── BigQuery: customers_products
│
└── domain-logistics-project
    ├── BigQuery: logistics_raw
    ├── BigQuery: logistics_curated
    └── BigQuery: logistics_products

# Provision a new domain project with Terraform
# Each domain team gets their own project with standard structure

# Create the domain project
gcloud projects create domain-orders   --folder=DOMAIN_FOLDER_ID   --name="Orders Domain"

# Enable necessary APIs
gcloud services enable bigquery.googleapis.com dataflow.googleapis.com   dataplex.googleapis.com --project=domain-orders

# Create BigQuery datasets following naming convention
bq mk --dataset --location=europe-west4   --description="Append-only raw ingestion from Orders systems"   domain-orders:orders_raw

bq mk --dataset --location=europe-west4   --description="Curated and validated orders data"   domain-orders:orders_curated

bq mk --dataset --location=europe-west4   --description="Published data products for cross-domain consumption"   domain-orders:orders_products

Dataplex: Federated Governance

Dataplex is GCP's data governance and data management service. It organizes your data assets across BigQuery, Cloud Storage, and other services into logical lakes, zones, and assets, and applies governance policies consistently.

# Create a Dataplex lake for the Orders domain
gcloud dataplex lakes create orders-lake   --location=europe-west4   --display-name="Orders Domain Lake"   --description="Data lake for the Orders business domain"   --project=domain-orders

# Create zones within the lake
# Raw zone: no format enforcement, accept any data
gcloud dataplex zones create orders-raw-zone   --lake=orders-lake   --location=europe-west4   --type=RAW   --resource-spec-type=MANAGED   --display-name="Orders Raw Zone"   --project=domain-orders

# Curated zone: enforced format (Parquet/Avro/ORC) and schemas
gcloud dataplex zones create orders-curated-zone   --lake=orders-lake   --location=europe-west4   --type=CURATED   --resource-spec-type=MANAGED   --display-name="Orders Curated Zone"   --project=domain-orders

# Register BigQuery datasets as Dataplex assets
gcloud dataplex assets create orders-raw-bq   --lake=orders-lake   --zone=orders-raw-zone   --location=europe-west4   --resource-spec-type=BIGQUERY_DATASET   --resource-spec-name=projects/domain-orders/datasets/orders_raw   --display-name="Orders Raw BigQuery"   --project=domain-orders

Data Quality Rules

# Create a data quality scan for the orders curated dataset
gcloud dataplex datascans create data-quality orders-dq-scan   --location=europe-west4   --data-source-entity=projects/domain-orders/locations/europe-west4/lakes/orders-lake/zones/orders-curated-zone/entities/orders_curated   --project=domain-orders

# Define quality rules in YAML
cat > quality_rules.yaml << 'EOF'
rules:
- columnMap:
    checks:
    - column: order_id
      rule:
        nonNullExpectation: {}
    - column: order_date
      rule:
        nonNullExpectation: {}
    - column: total_amount
      rule:
        rangeExpectation:
          minValue: "0"
          maxValue: "1000000"
    - column: customer_id
      rule:
        nonNullExpectation: {}
  samplingPercent: 10.0
  rowFilter: "order_date >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)"
EOF

Data Products: Authorized Views

Data products are the interfaces other domains use to access your data. Use authorized views to give cross-domain access without exposing raw tables:

-- Create a data product view in orders_products dataset
CREATE OR REPLACE VIEW `domain-orders.orders_products.order_summary_v1`
AS
SELECT
  DATE_TRUNC(order_date, DAY) AS order_date,
  order_status,
  delivery_region,
  COUNT(*) AS order_count,
  SUM(total_amount) AS total_revenue,
  AVG(total_amount) AS avg_order_value,
  COUNT(DISTINCT customer_id) AS unique_customers
FROM `domain-orders.orders_curated.orders`
GROUP BY 1, 2, 3;

# Authorize the view to access the underlying table
# This allows the orders_products dataset to read from orders_curated
bq update   --source_dataset_project_id=domain-orders   --source_dataset=orders_curated   domain-orders:orders_curated

# Grant cross-domain access to the data product
# The logistics team can access the order summary
bq add-iam-policy-binding   --member=serviceAccount:logistics-analyst-sa@domain-logistics.iam.gserviceaccount.com   --role=roles/bigquery.dataViewer   domain-orders:orders_products

Analytics Hub: Publishing Data Products

Analytics Hub provides a marketplace for data products within your organization (and optionally externally):

# Create an Analytics Hub exchange
gcloud bigquery analytics-hub exchanges create orders-exchange   --location=europe-west4   --display-name="Orders Domain Data Exchange"   --description="Published data products from the Orders domain"   --project=platform-project

# Create a listing for a data product
gcloud bigquery analytics-hub listings create order-summary-listing   --exchange=orders-exchange   --location=europe-west4   --display-name="Order Summary (Daily)"   --description="Daily aggregated order metrics including count, revenue, and unique customers"   --categories=BUSINESS_INTELLIGENCE   --dataset=domain-orders:orders_products   --project=platform-project

# Consumers subscribe to the listing — no data movement, just access
gcloud bigquery analytics-hub listings subscribe projects/domain-logistics/locations/europe-west4/dataExchanges/orders-exchange/listings/order-summary-listing   --destination-dataset=domain-logistics:linked_orders_products   --project=domain-logistics

With Analytics Hub, consumers see the data in their own BigQuery project through a linked dataset — no ETL, no data copying, and the producer's data quality SLAs apply.

Central Metadata Catalog with Data Catalog

Dataplex automatically integrates with Data Catalog for metadata management:

from google.cloud import datacatalog_v1

client = datacatalog_v1.DataCatalogClient()

# Tag a table with domain ownership and data classification
tag_template = client.get_tag_template(
    name="projects/platform-project/locations/europe-west4/tagTemplates/domain_metadata"
)

tag = datacatalog_v1.Tag()
tag.template = tag_template.name
tag.fields["domain_name"] = datacatalog_v1.TagField(string_value="Orders")
tag.fields["data_owner"] = datacatalog_v1.TagField(string_value="orders-team@company.com")
tag.fields["data_classification"] = datacatalog_v1.TagField(enum_value=datacatalog_v1.TagField.EnumValue(display_name="CONFIDENTIAL"))
tag.fields["sla_freshness_hours"] = datacatalog_v1.TagField(double_value=24.0)

entry_name = datacatalog_v1.DataCatalogClient.entry_path(
    "domain-orders", "europe-west4", "bigquery_dataset", "orders", "orders_curated", "orders"
)

client.create_tag(parent=entry_name, tag=tag)

The data mesh pattern works best as an incremental migration: start with one domain, prove the model, and expand. Trying to reorganize all data at once almost always fails. Pick a domain with clear ownership and business value, build the data product, and let other teams subscribe via Analytics Hub.

For BigQuery optimization within each domain, see our BigQuery cost optimization guide. For the governance layer, see our Security Command Center guide for audit monitoring across all domain projects.