TradeStream: Financial Services Platform on Infrastream

End-to-End Architecture & Deployment Guide

1. Executive Summary

Field	Detail
Project	TradeStream EU
Client	FinCorp Global (Tier-1 Investment Bank)
Partner	Stellar Technologies
Platform	Infrastream
Objective	Deploy a cloud-native, high-frequency trading analysis platform compliant with European financial regulations (GDPR/MiFID II). The system must ingest real-time market data, detect fraud using ML, and provide a secure dashboard for traders.
Compute	Google Cloud Run (Microservices), Compute Engine (Redpanda Cluster)
Data	AlloyDB (Ledger), Redpanda (Streaming), Cloud Storage (Reports)
Security	Identity-Aware Proxy (IAP), VPC Service Controls, Secret Manager
CI/CD	Infrastream Build Definitions & Release Tracks
Agentic	Native MiFID II trade-surveillance Agent on Vertex AI Agent Engine, governed by Infrastream (least-privilege `accessControl`, managed identity, 3-legged consent)

2. Architecture Overview

This diagram illustrates the flow of data and the relationship between Infrastream manifests and the deployed infrastructure.

Architecture Diagram

3. Organization & Environment Setup

We begin by defining the organizational structure to ensure all resources are correctly tagged, billed, and governed. A specific unit for "Financial Services" is created to enforce stricter compliance policies.

OrganizationalUnit Manifest

# FILE: ./financial-services.yaml
apiVersion: lowops.manifests.v1
kind: OrganizationalUnit
metadata:
  name: financial-services
  organization: stellar-tech
spec:
  description: |
    Unit dedicated to banking, trading, and insurance clients. 
    Enforces strict compliance, audit logging, and data residency 
    requirements (GDPR/MiFID II).
  permissions:
    administrators:
      groups:
      - fin-compliance-leads
    contributors:
      groups:
      - trade-engineers

Environment Manifest

Next, we define the Production environment where live trading data will flow, setting a high logging frequency for auditability.

# FILE: ./environment/production/environment.yaml
apiVersion: lowops.manifests.v1
kind: Environment
metadata:
  name: production
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  description: |
    Production environment for TradeStream EU. High availability, 
    multi-zone, strict egress controls.
  permissions:
    administrators:
      groups:
        - sre-leads
    viewers:
      groups:
        - auditors
        - l1-support

4. The Project & Security Boundary

This manifest defines the Google Cloud Project. Crucially, the allowedEgress field implements a "Zero Trust Egress" policy. The application can only communicate with the London Stock Exchange and our email provider.

# FILE: ./environment/production/project/tradestream-eu/tradestream-eu.yaml
apiVersion: lowops.manifests.v1
kind: Project
metadata:
  name: tradestream-eu
  environment: production
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  description: |
    TradeStream Europe. Real-time market analysis and fraud detection.
    Region: europe-west3 (Frankfurt) for data residency.
  region: europe-west3
  allowedEgress:
    - api.london-stock-exchange.com
    - api.sendgrid.com
  hibernation:
    hibernate: false # Financial systems run 24/7
  maintenance:
    start: 2025-02-08T02:00:00Z
    end: 2025-02-08T04:00:00Z
    exclusions: []

5. Source Code & Build Pipeline

5.1 GitHub Repositories

We define the repositories. The AI Assistant is enabled for the ingestor repo to help review complex C++/Go code.

Ingestor Repository:

# FILE: ./github-repository/ts-ingestor-repo.yaml
apiVersion: lowops.manifests.v1
kind: GithubRepository
metadata:
  name: ts-ingestor-repo
  github-connection: default
  organization: stellar-tech
spec:
  description: High-performance market data ingestor (Go/C++).
  strategy: GITHUB_FLOW
  aiAssistant:
    pullRequest:
      enabled: true # AI helps review complex concurrency code
  configuration:
    wiki: true
    projects: true
    issues: true
  codeOwners:
    - path: "*"
      owners:
        - trade-engineers
  permissions:
    administrators:
      groups:
        - sre-leads
    contributors:
      groups:
        - trade-engineers

Trader UI Repository:

# FILE: ./github-repository/ts-trader-ui-repo.yaml
apiVersion: lowops.manifests.v1
kind: GithubRepository
metadata:
  name: ts-trader-ui-repo
  github-connection: default
  organization: stellar-tech
spec:
  description: React-based Trader Dashboard.
  strategy: GITHUB_FLOW
  pullRequest:
    template: default
  permissions:
    administrators:
      groups:
        - front-end-leads
    contributors:
      groups:
        - front-end-developers

5.2 Build Definitions

These manifests define the CI pipeline. The ingestor uses a Go pipeline with strict security scanning, while the UI uses a standard React/Node pipeline.

Ingestor Build:

# FILE: ./build-definition/ts-ingestor-build.yaml
apiVersion: lowops.manifests.v1
kind: BuildDefinition
metadata:
  name: ts-ingestor-build
  github-repository: ts-ingestor-repo
  github-connection: default
  organization: stellar-tech
spec:
  type: GOLANG # Optimized for high throughput
  stages:
    - analysis
    - test
    - security-scan # Critical for financial software
    - build
    - containerize
  golang:
    buildType: bin
    target: "./cmd/ingestor"
    versions:
      - "1.23"
    platforms:
      goos:
        - linux
      goarch:
        - amd64
  containerize:
    - name: ts-ingestor
      platforms:
        - linux/amd64
      registries:
        - docker-private # Internal secure registry

Trader UI Build:

# FILE: ./build-definition/ts-trader-ui-build.yaml
apiVersion: lowops.manifests.v1
kind: BuildDefinition
metadata:
  name: ts-trader-ui-build
  github-repository: ts-trader-ui-repo
  github-connection: default
  organization: stellar-tech
spec:
  type: GOLANG # Frontend built via containerized pipeline
  stages:
    - test
    - containerize
  containerize:
    - name: ts-trader-ui
      dockerfile: Dockerfile
      platforms:
        - linux/amd64
      registries:
        - docker-private

6. Data Layer (Stateful Resources)

The platform uses AlloyDB for the transactional ledger (ACID compliant) and a Redpanda cluster on VMs for the high-volume stream of market ticks.

AlloyDB Database

# FILE: ./environment/production/project/tradestream-eu/database/trade-ledger.yaml
apiVersion: lowops.manifests.v1
kind: Database
metadata:
  name: trade-ledger
  project: tradestream-eu
  environment: production
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  description: Primary AlloyDB cluster for trade records and user auth.
  alloydb:
    cpuCount: 8
    clusterSize: 2 # High Availability
    configuration:
      maxConnections: 5000
    backupConfig:
      quantityBasedRetention: 30 # 30 Days retention

Redpanda Cluster (Virtual Machine)

# FILE: ./environment/production/project/tradestream-eu/virtual-machine/redpanda-cluster.yaml
apiVersion: lowops.manifests.v1
kind: VirtualMachine
metadata:
  name: redpanda-cluster
  project: tradestream-eu
  environment: production
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  description: |
    Stateful Redpanda (Kafka) cluster for market data ingestion (3 nodes).
  meshStrategy: SIDECAR
  stateful:
    enabled: true
    clusterSize: 3
    allowTcp:
      - 9092 # Kafka API
      - 9644 # Admin API
    health:
      port: 9644
      protocol: http
      path: /v1/status/ready
      checkIntervalSec: 30
      unhealthyThreshold: 3
      healthyThreshold: 1
      timeoutSec: 5
  ports:
    9644: http
  configuration:
    source: redpanda-v23
    volumeMounts:
      /var/lib/redpanda/data:
        diskConfig:
          sizeGb: 500
          type: pd-ssd
        mountOptions:
          - rw
          - noatime
          - xfs
  machineType: c2-standard-4 # Compute optimized for streaming

7. Storage & Messaging

We define storage for regulatory reports and a Pub/Sub topic to decouple the fraud detection and alerting systems.

Cloud Storage Bucket

# FILE: ./environment/production/project/tradestream-eu/bucket/regulatory-reports.yaml
apiVersion: lowops.manifests.v1
kind: Bucket
metadata:
  organization: stellar-tech
  organizational-unit: financial-services
  environment: production
  project: tradestream-eu
  name: regulatory-reports
spec:
  storageClass: NEARLINE
  public: false
  description: Encrypted bucket for MiFID II compliance reports.
  processors:
    obscenityScanner: false

Pub/Sub Topic

# FILE: ./environment/production/project/tradestream-eu/pub-sub/fraud-alerts.yaml
apiVersion: lowops.manifests.v1
kind: PubSub
metadata:
  organization: stellar-tech
  organizational-unit: financial-services
  environment: production
  project: tradestream-eu
  name: fraud-alerts
spec:
  messageRetentionDurationSeconds: 604800
  notifications:
    - deploymentConfig: ts-notification-svc
      path: webhook/fraud-alert

8. Secrets Management

The ingestor's API key for the London Stock Exchange is stored securely in Secret Manager.

# FILE: ./environment/production/project/tradestream-eu/secret/lse-api-key.yaml
apiVersion: lowops.manifests.v1
kind: Secret
metadata:
  name: lse-api-key
  project: tradestream-eu
  environment: production
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  description: API Key for London Stock Exchange data feed.

9. Microservices (Deployment Configs)

We define the runtime configuration for the services: the ts-ingestor and the ts-trader-ui frontend dashboard.

Market Data Ingestor

# FILE: ./environment/production/project/tradestream-eu/deployment-config/ts-ingestor.yaml
apiVersion: lowops.manifests.v1
kind: DeploymentConfig
metadata:
  name: ts-ingestor
  organization: stellar-tech
  organizational-unit: financial-services
  environment: production
  project: tradestream-eu
spec:
  description: |
    Connects to Stock Exchange API, normalizes data, pushes to Redpanda.
  version: "1.0.0"
  container:
    env:
      - name: KAFKA_BROKERS
        value: redpanda-cluster.tradestream-eu.production.financial-services.stellar-tech.internal:9092
      - name: LOG_LEVEL
        value: warn
  health:
    liveness:
      port: 8080
      protocol: http
      path: /health
  scaling:
    min: 2
    max: 10
  ports:
    8080: http

Trader Dashboard UI

# FILE: ./environment/production/project/tradestream-eu/deployment-config/ts-trader-ui.yaml
apiVersion: lowops.manifests.v1
kind: DeploymentConfig
metadata:
  name: ts-trader-ui
  organization: stellar-tech
  organizational-unit: financial-services
  environment: production
  project: tradestream-eu
spec:
  description: React Frontend for Traders.
  version: "1.0.0"
  health:
    liveness:
      port: 80
      protocol: http
      path: /
  scaling:
    min: 2
  volumes:
    /usr/share/nginx/html/config:
      files:
        config.json:
          content: |
            {
              "apiUrl": "https://api.tradestream.stellar-tech.com",
              "refreshRate": 500,
              "currency": "EUR"
            }
  ports:
    80: http

10. Native Agentic — Trade Surveillance Agent

TradeStream is natively agentic. Beyond ML fraud scoring on the ingest path, a governed AI Agent performs MiFID II market-abuse surveillance: it consumes the fraud-alerts topic, correlates each alert against the trade ledger, and assembles a regulator-ready case file for the compliance desk. It runs on Vertex AI Agent Engine and is governed by Infrastream exactly like every other resource — capabilities declared in a typed manifest, reviewed in a PR, enforced by the engine. See The Agentic Development Platform and Agentic Governance.

Note: The agentic layer is early and under active development (ADK-only runtime; re-test per environment before production). Treat this section as a reference pattern.

Agent Manifest

# FILE: ./environment/production/project/tradestream-eu/agent/trade-surveillance.yaml
apiVersion: lowops.manifests.v1
kind: Agent
metadata:
  name: trade-surveillance
  project: tradestream-eu
  environment: production
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  description: >
    MiFID II market-abuse surveillance. Consumes fraud alerts,
    correlates them against the trade ledger, and assembles a
    regulator-ready case file for the compliance desk.
  target: AGENT_ENGINE
  project: tradestream-eu
  buildDefinition: trade-surveillance-build   # BuildDefinition (type: AGENT) — see below
  accessControl:
    database:
      secretSource:
        envVar: DATABASE_TRADE_LEDGER_CONNECTION_STRING
      schema: tradestream-eu
      name: trade-ledger
      privileges: [SELECT]        # Read-only — least privilege
    pubsub:
      subscribeTo:
        - fraud-alerts

Every permission is visible in this diff: the agent has read-only ledger access and subscribe-only access to fraud-alerts — it cannot publish, cannot write trades, and cannot reach the lse-api-key or any endpoint outside the project's egress allow-list. The engine automatically provisions its runtime identity (scoped IAM grants, refreshed tokens). When a compliance officer asks the agent to escalate a case into an external regulator portal, it acts under that officer's 3-legged consent, scoped to that one action — never a shared credential.

Agent Build Definition & Registry

buildDefinition names a BuildDefinition manifest with type: AGENT, publishing to a dedicated Python-type ArtifactRegistry:

# FILE: ./artifact-registry/agent-packages.yaml
apiVersion: lowops.manifests.v1
kind: ArtifactRegistry
metadata:
  name: agent-packages
  organization: stellar-tech
spec:
  type: AGENT
  public: false

# FILE: ./build-definition/trade-surveillance-build.yaml
apiVersion: lowops.manifests.v1
kind: BuildDefinition
metadata:
  name: trade-surveillance-build
  github-repository: ts-surveillance-repo
  github-connection: default
  organization: stellar-tech
spec:
  type: AGENT
  agent:
    registry: agent-packages
    entryModule: ts_surveillance.agents.trade_surveillance.agent
    packageName: ts_surveillance
    agentDirectory: src

Current limitation: entryModule, packageName, and agentDirectory are schema-required whenever type: AGENT, but only agent.registry is actually consumed by the engine today (to grant the CI service account artifact-push IAM) — the other three are declared but not yet read by any consumer. type: AGENT also has no CI workflow builder yet, so automated build generation will fail. Build and publish the agent wheel to the registry manually until this lands.

Agent Deployment Config

# FILE: ./environment/production/project/tradestream-eu/agent/trade-surveillance-deployment.yaml
apiVersion: lowops.manifests.v1
kind: AgentDeploymentConfig
metadata:
  name: trade-surveillance
  project: tradestream-eu
  environment: production
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  version: "1.0.0"

11. Observability & Monitoring

The Monitoring and Alerting manifests declare intended dashboards and alert routing.

Preview

The Monitoring and Alerting manifest kinds are defined in the schema but are not yet provisioned by the engine — declaring them today records intent but does not create dashboards or alert channels. In the meantime, use Google Cloud Monitoring directly. Track field-level status in the Monitoring and Alerting references.

Monitoring Dashboard

# FILE: ./environment/production/project/tradestream-eu/monitoring/trade-ops.yaml
apiVersion: lowops.manifests.v1
kind: Monitoring
metadata:
  name: trade-ops
  project: tradestream-eu
  environment: production
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  enabled: true
  description: "Main operations dashboard for TradeStream EU"

Alerting Configuration

# FILE: ./environment/production/project/tradestream-eu/monitoring/alerting/critical-alerts.yaml
apiVersion: lowops.manifests.v1
kind: Alerting
metadata:
  name: critical-alerts
  project: tradestream-eu
  environment: production
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  enabled: true
  description: "Critical alerts for the TradeStream trading path"
  notifications:
    slack:
      channel: "#fin-ops-critical"

12. Load Balancing & Authentication (IAP)

The Trader UI is securely exposed to the internet via a Global Load Balancer, enforcing Identity-Aware Proxy (IAP) for zero-trust access.

PublicIngress: Creates the Global Load Balancer and its iapPermissions allowlist (which Workspace groups may reach it at all)
IdentityProvider: Configures the login screen (Microsoft SSO) that authenticates each trader
HttpRoute: Routes traffic to the UI service — critically, its rule must name the IdentityProvider in its own authentication block, or the route serves traffic unauthenticated regardless of the other two manifests

The iapPermissions block in the Ingress manifest gates which Workspace groups may reach the application at all; the HttpRoute rule's authentication block is what actually puts each request through IAP + the trader-login GCIP tenant. Both are required together.

Public Ingress (Load Balancer)

# FILE: ./environment/production/project/tradestream-eu/public-ingress/trade-ingress.yaml
apiVersion: lowops.manifests.v1
kind: PublicIngress
metadata:
  name: trade-ingress
  project: tradestream-eu
  environment: production
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  description: |
    Global Load Balancer for TradeStream.
    Enforces Identity-Aware Proxy (IAP) for zero-trust access.
  domain:
    name: tradestream.stellar-tech.com
  iapPermissions:
    groups:
      - authorized-traders
      - compliance-officers
      - sre-leads

Identity Provider (SSO)

# FILE: ./environment/production/project/tradestream-eu/public-ingress/trader-login.yaml
apiVersion: lowops.manifests.v1
kind: IdentityProvider
metadata:
  name: trader-login
  public-ingress: trade-ingress
  project: tradestream-eu
  environment: production
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  description: "Corporate SSO for FinCorp Traders"
  displayName: "FinCorp Secure Login"
  logoUrl: "https://fincorp.com/logo.png"
  mode: REDIRECT
  providers:
    microsoft: true # Corporate Active Directory login
    google: false
    password: false

HTTP Route

# FILE: ./environment/production/project/tradestream-eu/public-ingress/http-route/ui-route.yaml
apiVersion: lowops.manifests.v1
kind: HttpRoute
metadata:
  organization: stellar-tech
  organizational-unit: financial-services
  project: tradestream-eu
  environment: production
  public-ingress: trade-ingress
  name: ui-route
spec:
  description: Route for Trader UI
  rules:
    - matches:
        - prefixMatch: /
      authentication:
        - type: IDENTITY_PROVIDER
          tenants:
            - trader-login   # the IdentityProvider manifest above — omitting this leaves the route unauthenticated
      action:
        destinations:
          - deploymentConfig: ts-trader-ui
            port: 80

13. Release Management

Finally, we tie everything together with a Release Track. This defines how code moves from the repository to the production environment.

Release Track

# FILE: ./release-track/fin-track.yaml
apiVersion: lowops.manifests.v1
kind: ReleaseTrack
metadata:
  name: fin-track
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  description: Strict release track for financial applications.
  releaseStages:
    - environments:
        - production
      approval_policy:
        stakeholder:
          group_email: fin-compliance-leads

Application Set

# FILE: ./release-track/application-set/core-trading.yaml
apiVersion: lowops.manifests.v1
kind: ApplicationSet
metadata:
  name: core-trading
  release-track: fin-track
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  description: Core high-frequency trading components.

Application: Ingestor

# FILE: ./release-track/application-set/application/ts-ingestor.yaml
apiVersion: lowops.manifests.v1
kind: Application
metadata:
  name: ts-ingestor
  application-set: core-trading
  release-track: fin-track
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  description: TradeStream Ingestor Application
  project: tradestream-eu
  meshStrategy: SIDECAR
  target: CLOUD_RUN
  source: ts-ingestor-build # Links to the BuildDefinition
  container: ts-ingestor
  accessControl:
    secrets:
      lse-api-key:
        envVar: LSE_API_KEY

Application: Trader UI

# FILE: ./release-track/application-set/application/ts-trader-ui.yaml
apiVersion: lowops.manifests.v1
kind: Application
metadata:
  name: ts-trader-ui
  application-set: core-trading
  release-track: fin-track
  organizational-unit: financial-services
  organization: stellar-tech
spec:
  description: Trader Dashboard Frontend
  project: tradestream-eu
  target: CLOUD_RUN
  source: ts-trader-ui-build # Links to the BuildDefinition
  container: ts-trader-ui

Key Infrastream Features Demonstrated

This use case showcases several critical Infrastream capabilities for financial services:

1. Regulatory Compliance by Design

GDPR/MiFID II compliance through data residency controls (europe-west3)
Complete audit trail via Git history
Automated backup retention (30 days)

2. Zero Trust Security

Egress controls limiting external communication
Identity-Aware Proxy (IAP) for authenticated access
Secret Manager integration for API keys
Service mesh with sidecar proxy pattern

3. High Availability & Performance

Multi-zone AlloyDB cluster (clusterSize: 2)
Autoscaling Cloud Run services (min: 2, max: 10)
Dedicated Redpanda cluster for high-throughput streaming
Health checks and monitoring at every layer

4. GitOps-Driven Deployment

Infrastructure as Code for all components
Automated CI/CD via BuildDefinitions
Controlled release process via Release Tracks
Version-controlled configuration management

5. Cost & Access Governance

Organizational Unit segregation for financial services
Group-based access control
Resource tagging and cost allocation
Maintenance window scheduling

6. Governed Agentic AI

Native surveillance Agent on Vertex AI Agent Engine, declared as a typed manifest
Least-privilege accessControl: read-only ledger, subscribe-only alerts
Managed agent identity (scoped IAM + refreshed tokens) provisioned automatically
3-legged consent when acting on behalf of a specific compliance officer

Business Value

Metric	Impact
Time to Production	Deploy complete trading platform in hours instead of months
Compliance	GDPR/MiFID II compliance built-in, not bolted on
Security Posture	Zero Trust architecture with provable audit trails
Governed Agentic AI	Native surveillance Agent with least-privilege, fully auditable access — autonomy without unbounded authority
Operational Overhead	Platform manages 90% of infrastructure complexity
Disaster Recovery	30-day retention with automated backups
Cost Transparency	Complete visibility into infrastructure costs by organizational unit

End-to-End Architecture & Deployment Guide​

1. Executive Summary​

2. Architecture Overview​

3. Organization & Environment Setup​

OrganizationalUnit Manifest​

Environment Manifest​

4. The Project & Security Boundary​

5. Source Code & Build Pipeline​

5.1 GitHub Repositories​

5.2 Build Definitions​

6. Data Layer (Stateful Resources)​

AlloyDB Database​

Redpanda Cluster (Virtual Machine)​

7. Storage & Messaging​

Cloud Storage Bucket​

Pub/Sub Topic​

8. Secrets Management​

9. Microservices (Deployment Configs)​

Market Data Ingestor​

Trader Dashboard UI​

10. Native Agentic — Trade Surveillance Agent​

Agent Manifest​

Agent Build Definition & Registry​

Agent Deployment Config​

11. Observability & Monitoring​

Monitoring Dashboard​

Alerting Configuration​

12. Load Balancing & Authentication (IAP)​

Public Ingress (Load Balancer)​

Identity Provider (SSO)​

HTTP Route​

13. Release Management​

Release Track​

Application Set​

Application: Ingestor​

Application: Trader UI​

Key Infrastream Features Demonstrated​

1. Regulatory Compliance by Design​

2. Zero Trust Security​

3. High Availability & Performance​

4. GitOps-Driven Deployment​

5. Cost & Access Governance​

6. Governed Agentic AI​

Business Value​