Airline Metrics — Modernised Architecture on UpCloud

Section 00

00

Migration Path

The PHP portal is not replaced on day one. A strangler fig migration moves modules across incrementally while keeping the portal live throughout.

The current CodeIgniter 3 portal is not a separate frontend — it is the entire application. It owns the database connections, the business logic, the job scheduling, the report generation, and the HTML rendering in one monolithic codebase. A big-bang rewrite is high risk and unnecessary. The strangler fig pattern lets you migrate one module at a time, with the old PHP portal and the new Quarkus API running in parallel behind Kong until PHP handles nothing and can be switched off.

The strangler fig in one sentence

Kong routes new API paths to Quarkus and all other traffic to the PHP portal. Each module that moves to Quarkus disappears from PHP. When PHP handles zero routes, it is decommissioned. No big bang. No dark period. Users notice only that the portal gets faster.

Routing split at Kong — how it looks

Phase 1 — today

PHP portal handles all routes

Direct MySQL queries

Live commission calc on load

Jobs.php scheduling

HOT processing on same server

Phase 2 — strangler active

/portal/* → PHP (read-only views)

/api/v2/ingestion/* → Quarkus

/api/v2/contracts/* → Quarkus

/api/v2/tripcube/* → Quarkus

PHP reads contract_payout_results

Phase 3 — PHP retired

All routes → Quarkus REST API

Qute/HTMX portal serves UI

Payout = stored SELECT

Temporal replaces Jobs.php

PHP decommissioned

Phase-by-phase module migration

Phase 1

Separate processing from portal — PHP portal untouched

Weeks 1–6

What moves

AMHotProcessor JAR → Quarkus container chain on UpCloud Kubernetes
HOT file intake → UpCloud Object Storage + RabbitMQ queue
Jobs.php HOT trigger → Temporal workflow (contract actuals + payout)
contract_payout_results table created — payout now stored

PHP portal during this phase

Continues reading from the same MySQL database
Flown Income tab now reads pre-computed payout — live calc removed
No UI changes visible to users
Processing runs faster and in parallel — users notice HOT files completing sooner

Phase 2

API layer + data tier — PHP portal reads from new API

Weeks 7–16

What moves

Quarkus REST API deployed — /api/v2/* routes live behind Kong
UpCloud Managed Databases provisioned — per-tenant schemas
UUID PK migration — dual-write old int PKs + new UUIDs during transition
Nightly ETL begins archiving data older than 24m to Iceberg
Trino cluster deployed — Tripcube queries routed through Trino
HashiCorp Vault replaces plaintext config files

PHP portal during this phase

PHP models refactored to call /api/v2/* endpoints instead of direct MySQL
Contract IQ Excel generation moves to Quarkus async worker
Tripcube queries route through Trino — primary DB no longer hit directly
PHP renders HTML — Quarkus provides the data — clean separation

Phase 3

Qute/HTMX portal replaces PHP — CodeIgniter decommissioned

Weeks 17–28

What moves

Quarkus Qute templates replace CodeIgniter views one module at a time
HTMX replaces jQuery AJAX calls — no JavaScript framework
Contract wizard migrated — Qute partials + HTMX step navigation
Report views migrated — Flown Target, Flown Income, Contract IQ
Tripcube UI migrated — filter form + async job status via HTMX poll
Kong stops routing to PHP — PHP server decommissioned

What the team gains

Single Quarkus process — API + portal in one deployable unit
No Node.js in production — no npm, no webpack, no build pipeline for JS
Qute templates feel like Blade/Twig — low learning curve from PHP
HTMX handles all dynamic UI with HTML attributes — no JS state management
Full CI/CD coverage — same Helm chart deploys everything

⚠

The dual-write UUID migration in Phase 2 is the highest-risk step. The current auto-increment PKs (TransHeadId, TicketHeadId, OffHeadId) are referenced by foreign keys across many tables. A shadow UUID column is added first, populated for all existing rows, then the PHP portal and Quarkus API dual-write to both until all consumers have migrated. The old integer PK columns are dropped only after Phase 3 completes.

Section 01

01

Architecture Overview

Full stack on UpCloud infrastructure — no AWS dependency.

The modernised architecture replaces the current single-server deployment with a seven-layer design: client intake, async queue-driven processing, a hot relational store, a cold Iceberg archive tier, a federated query engine, an API reporting layer, and cross-cutting security and observability planes. Every component runs on UpCloud infrastructure.

Key infrastructure decision — UpCloud throughout

UpCloud Object Storage (S3-compatible API) replaces AWS S3 and MinIO. UpCloud Managed Databases replaces Aurora. UpCloud Managed Kubernetes with KEDA replaces AWS Lambda for serverless container scaling. The only self-hosted services are RabbitMQ (queue), Kong (API gateway), Temporal (workflow), Trino (query federation), HashiCorp Vault (secrets), and the Grafana observability stack.

① Client intake — per-client UpCloud sub-account HOT file drop · thin container · GDPR region-locked

Client A (EU)Object Storage bucket · eu-de region

→

Intake container~50 lines · no IP · mTLS

→

Client B (APAC)ap-sg region

→

Client N…any region

↓

② API gateway + queue Kong on UpCloud Kubernetes · RabbitMQ fan-out · DLQ on failure

Kong gatewaymTLS · JWT · rate limit

→

RabbitMQfan-out exchange · DLQ

↓

③ Quarkus containers on UpCloud Kubernetes + KEDA Scale to zero · GraalVM native · one pod per message · Temporal orchestration

ParserDISH decode

→

ValidatorUUID assign · BigDecimal

→

Enrichmentmileage · FX · Valkey cache

→

PersistAurora write · tenant schema

→

Contractactuals · payout stored

Temporal (self-hosted on UpCloud VM) — full execution history · per-step retry · workflow_run_id audit trail

↓

④ Hot store — UpCloud Managed Databases Per-tenant schema isolation · UUID v4 PKs · DECIMAL financials · read replica · Valkey cache

Tenant A schemaEU region · dish_transactions · contract_payout_results

Tenant B schemaAPAC region

Tenant N schema…

Valkey (Redis)ref data <1ms · 1hr TTL

↓ nightly ETL >24 months

⑤ Cold archive — UpCloud Object Storage + Iceberg + Trino Parquet columnar · time-travel · partition evolution · GDPR row-delete · EMR replaced by self-hosted Trino

UpCloud Object StorageS3-compatible · region-locked per tenant

+

Apache Icebergtable format · time-travel · row-level delete

+

Parquet filescolumnar · snappy · partitioned by tenant+month

+

Trino clusterUpCloud Kubernetes · federates hot + cold

↓

⑥ Reporting API + portal Quarkus REST · JWT per tenant · payout = stored SELECT · Quarkus Qute + HTMX portal · server-rendered · no JS framework

Quarkus REST APItenant-scoped · query routing

→

Valkey (<1ms)reference data

/

DB read replicalast 24 months

/

Trinohistorical >24m

→

Qute/HTMX portalServer-rendered · UpCloud K8s

⑦ Cross-cutting: HashiCorp Vault (secrets) Grafana + Loki + Tempo (observability) WORM Object Storage (audit log) Region-locked tenants (GDPR) Per-client JWT kill switch

Section 02

02

Async Ingestion Pipeline

Replacing AMHotProcessor JAR + TCP socket + Jobs.php with a parallel, observable, self-healing pipeline.

The current AMHotProcessor listens on a single TCP socket and processes one HOT file at a time in a single JVM thread. If processing crashes, there is no automatic retry. If the server is busy processing one file when another arrives, it waits. The new design inverts all of this.

Current

Single TCP socket port 59300 — one file at a time
Java 8 JAR — OOM risk on large HOT files
No retry on failure — manual re-run required
No execution visibility — check server logs
HOT file + reporting on same server instance
Auto-increment PKs block cross-instance migration
IEEE 754 double for all financial arithmetic

Modernised

RabbitMQ fan-out — unlimited parallel files
Quarkus GraalVM native — sub-100ms cold start, ~50MB RSS
DLQ + automatic retry with exponential backoff
Temporal dashboard — full step-by-step execution history
KEDA scales containers to zero between HOT file runs
UUID v4 PKs — safe cross-instance and multi-region
Java BigDecimal throughout — BSP-grade precision

Temporal workflow steps per HOT file

1

Intake container receives S3 event

Client's UpCloud Object Storage fires event on HOT file upload. Container signs payload with client JWT and posts to Kong gateway.

2

Kong validates JWT + injects tenant_id

mTLS handshake. JWT validated. Per-client rate limit enforced. tenant_id injected into message header. Published to RabbitMQ HOT file queue.

3

Parser container — DISH fixed-width decode

Reads HOT file from Object Storage. Decodes all DISH record types (BFH01–BKP84) into typed Java records. GraalVM native binary — IP non-decompilable.

4

Validator container — UUID assignment

DISH §6.5.4 rule check. Duplicate detection. UUID v4 PKs assigned to all entities. Validation failures written to validation_errors table — nothing passes silently.

5

Enrichment container — mileage · NUC · FX

Reference data from Valkey cache (sub-1ms). TPM mileage proration per BKI63 segment. NUC conversion. FX to USD. All arithmetic in BigDecimal.

6

Persist container — tenant schema write

Aurora write via connection pool. IAM auth — no credentials in code. Full transaction: dish_transactions + mileage_fares commit together or roll back.

7

Contract container — actuals + payout store

Event-driven incremental update for affected contracts. Commission computed and written to contract_payout_results with workflow_run_id. No more manual re-population.

Section 03

03

Application Modules

The four core application modules — Contract Engine, Tripcube, Contract IQ, and the AI prediction layer — are each modernised independently. All use the same underlying Quarkus container runtime on UpCloud Kubernetes and share the Temporal workflow engine for orchestration visibility.

Contract engine

Temporal-orchestrated contract pipeline

Contract rules move from the base64-encoded localStorageData blob to proper relational tables. The 4-phase populate pipeline becomes a Temporal workflow with full step history and automatic retry. Payout is stored in contract_payout_results — never calculated live on page load. Overlap detection runs on every contract save.

Tripcube

Serverless query service via Trino

Tripcube queries move from hitting the primary database directly to querying Trino, which routes to the read replica for recent data or Iceberg for historical. The self-service query builder posts a job to RabbitMQ. The Tripcube container executes and writes output to UpCloud Object Storage as CSV or Parquet. User receives a signed URL download link.

Contract IQ

Parallel Temporal fan-out per contract

Each Contract IQ run spawns one Temporal child workflow per contract in the group. All contracts analyse in parallel — a 20-contract group takes the same time as a single contract analysis. The Excel workbook is generated on completion and stored in UpCloud Object Storage. Status is visible in the Temporal dashboard in real time.

AI prediction

Real ML model replacing rule-based projection

The current Deep_Dive_Model multiplies prior-year data by a manually-entered growth rate. The modernised layer trains a time-series forecasting model (e.g. Prophet or LightGBM) on historical Iceberg data per tenant. A prediction container serves inference results — projected tier, revenue shortfall, O&D mix — feeding into Contract IQ recommendation sheets.

Workflow engine — why Temporal

Temporal is chosen over Activiti/Camunda because the primary need is processing observability (HOT file pipeline visibility, contract populate step tracking) not BPMN process design. Temporal provides a dashboard showing every workflow run, every activity step, every failure and retry, with the workflow_run_id linking each payout figure back to the exact HOT file that produced it. This is the audit trail the current system completely lacks.

Section 04

04

Data Tier — Hot + Cold

UpCloud Managed Databases for recent data. UpCloud Object Storage + Iceberg + Trino for the archive. One SQL surface for both.

Hot store — UpCloud Managed Databases

Stores the last 24 months of transaction data. Per-tenant schema isolation — a bug in one tenant's query cannot reach another tenant's data. UUID v4 PKs throughout. DECIMAL(20,6) for all financial columns. utf8mb4. A new table — contract_payout_results — stores every commission calculation with timestamp and workflow_run_id, eliminating the live commission calculation on every page load.

Cold archive — UpCloud Object Storage + Apache Iceberg

A nightly Glue/Spark job moves transactions older than 24 months from the database to Parquet files on UpCloud Object Storage, wrapped in Apache Iceberg table format. Iceberg is specifically the right choice for BSP data for three reasons:

Time-travel

When a refund arrives in month 6 for a ticket from month 2, you can query the state of the transaction table as it was before the correction — then apply the update and prove the change. Replaces the current cross-hotfile UPDATE with no audit trail.

Partition evolution

Add a carrier partition next year without rewriting existing data. Old data keeps its old layout, new data uses the new layout, Trino queries work transparently across both.

GDPR delete

An Iceberg DELETE marks rows as deleted in the manifest immediately and physically removes them in the next compaction cycle. Right-to-erasure is provable via the manifest history. MySQL has no equivalent for archived data.

Trino — transparent query federation

Trino is self-hosted on UpCloud Kubernetes. It presents a single SQL surface across the Aurora connector (hot, last 24 months) and the Iceberg connector (cold, older). A query for "all transactions between Jan 2020 and Dec 2024" crosses the boundary — Trino executes both sub-queries in parallel and returns a merged result set. The REST API issues one SQL query and never needs to know which tier holds the data.

↻

At 25× current volume — approximately 105M transactions per year — the Parquet archive tier will hold around 2–3 billion rows. A Trino query against that volume with date and tenant partition pruning completes in seconds. An equivalent MySQL full-table scan would be measured in hours.

Section 05

05

Security, GDPR and Observability

Secrets

HashiCorp Vault

Replaces plaintext config files. All credentials stored encrypted at rest. Containers authenticate via Kubernetes service account tokens — no password in code or environment variables. Dynamic database credentials rotate automatically every 30 days with no redeployment.

GDPR

Region-locked tenants

EU client data stays in eu-de. APAC client data stays in ap-sg. Per-tenant UpCloud Object Storage buckets are region-locked at bucket policy level. Iceberg row-level delete satisfies right-to-erasure for historical data with a provable manifest trail. PII masked at the Enrichment container before Aurora or Parquet write.

Observability

Grafana + Loki + Tempo + Temporal

Structured JSON logs from every container emitted to Loki (workflow_run_id, tenant_id, duration_ms). Tempo distributed traces connect all containers in a single HOT file run. Grafana dashboards surface DLQ depth, processing latency, per-tenant throughput. Temporal dashboard shows step-by-step contract workflow history.

Audit

Immutable audit trail

Every container invocation, RabbitMQ message, and database write is logged to UpCloud Object Storage with WORM object lock policy — cannot be modified or deleted even by an account administrator. The workflow_run_id in contract_payout_results links every commission figure back to the exact HOT file, processing timestamp, and container chain that produced it.

Kill switch

Revoking a client's JWT in Vault and rotating their Kong usage plan key severs all access in seconds — no redeployment, no firewall change, no server restart. This replaces a multi-hour manual offboarding process with a 30-second operation.

Section 05b

05b

Frontend — Quarkus Qute + HTMX

Server-rendered templates, no JavaScript framework, no Node.js in production.

The portal is a B2B analytics dashboard used by authenticated airline and agency users. It is not a public-facing site and has no SEO requirements. The interaction patterns are form-heavy: filter configuration, contract wizards, report viewers, and file downloads. A JavaScript framework adds significant complexity without solving any real problem this portal has.

Next.js / React SPA

Server-side rendering for SEO — not needed. Full-stack API routes — redundant (Quarkus owns the API). Node.js server to maintain in production. npm ecosystem and bundler to manage. Complex client-side state management for what are essentially forms and tables.

Quarkus Qute + HTMX recommended

HTML rendered server-side in the same Quarkus JVM as the REST API. HTMX replaces AJAX with HTML attribute annotations — no JavaScript state machine. No Node.js, no npm, no webpack in production. Qute template syntax is close to Blade and Twig — low learning curve from CodeIgniter views. Single deployable container for API + portal.

Vite + React SPA

Reasonable fallback if the team has strong JavaScript skills. Static build served from UpCloud Object Storage. Fully decoupled from Quarkus. Higher initial complexity than Qute but no SSR overhead. Suitable if rich client-side interactivity (live charts, drag-and-drop) becomes a priority later.

How Qute + HTMX maps to the existing portal modules

Module

Current PHP approach

Qute + HTMX equivalent

Contract list

PHP foreach loop over CI model result

Qute {#for} loop · data from Quarkus service method

Contract wizard

Full page POST on each of 15 steps

HTMX hx-post · partial template swap · no page reload

Flown Target tab

CI model queries tp_trans_amounts directly

Qute template · data from /api/v2/contracts/{id}/actuals

Flown Income tab

Live commission calc in PHP model on load

Simple SELECT on contract_payout_results · Qute render

Tripcube builder

PHP form → synchronous query → download

HTMX form post → job queued → HTMX poll for status → signed URL

Contract IQ run

PHP fires send_performance_analyser job

HTMX trigger → Temporal workflow → HTMX progress poll

AI recommendations

Deep_Dive_Model PHP rules on request

Quarkus inference call → Qute partial renders results

HTMX in practice for the contract wizard

The 15-step contract wizard currently posts the full form and re-renders the entire PHP page on each step. With HTMX, each wizard step is a Qute partial template. Clicking "Next" sends an HTMX request to /contracts/wizard/step/3, which returns only the HTML for step 3. That fragment replaces the current step in place — no full page reload, no JavaScript state machine, no React component tree. The URL updates via HTMX history API. The whole interaction is driven by HTML attributes on the form buttons.

Template structure in Quarkus

src/main/resources/templates/
  base.html            ← topbar, nav, footer (replaces CodeIgniter layout)
  contracts/
    list.html          ← target list view
    wizard/step{"{n}"}.html  ← 15 wizard step partials (HTMX-swapped)
    flown-target.html  ← revenue report view
    flown-income.html  ← payout view (simple SELECT on contract_payout_results)
  tripcube/
    builder.html      ← query builder form
    status.html      ← job status (HTMX poll every 3s until complete)
  contract-iq/
    analyser.html     ← group selector + run button
    progress.html    ← Temporal workflow status (HTMX poll)

Section 12

12

Development & Setup Timeline

28-week end-to-end plan across three phases. Each phase is independently deployable and delivers value without requiring the next phase to be complete.

The timeline is structured around the three migration phases defined in Section 00. Phase 1 delivers the most immediate operational relief — processing separated from the portal, payout stored, DLQ visibility. Phase 2 brings the full data tier and API layer. Phase 3 completes the portal migration and decommissions PHP.

Recommended team composition

2

Backend / Quarkus engineers

All phases

1

DevOps / infrastructure engineer

All phases

1

Frontend / Qute engineer

Phase 3 primarily

1

QA / test automation

Phase 2 onwards

Phase 1

Processing separation

Wks 1–6

Weeks 1–2 · Infrastructure

UpCloud Kubernetes cluster provisioned via Terraform

RabbitMQ deployed on K8s via Helm

Temporal server + PostgreSQL backend deployed

HashiCorp Vault deployed + unsealed

UpCloud Container Registry configured

GitHub Actions self-hosted runner on UpCloud VM

Weeks 3–4 · Quarkus containers

Parser container — DISH fixed-width decode port from Java JAR

Validator container — DISH rule checks + UUID assignment

Enrichment container — mileage proration + BigDecimal FX

Persist container — Aurora write via connection pool

Unit tests >90% coverage on parser and validator

Risk: DISH edge cases in parser — allocate buffer time

Weeks 5–6 · Contract + payout

Contract container — incremental actuals update

contract_payout_results table created and populated

Temporal workflow wiring all 5 containers

KEDA autoscaler configured against RabbitMQ queue depth

PHP portal reads payout from new table — live calc removed

Integration test: synthetic HOT file → assert payout row

Phase 1 complete — HOT processing on Kubernetes, payout stored, portal unmodified and faster

Phase 2

API layer + data tier

Wks 7–16

Weeks 7–9 · Database migration

UpCloud Managed Databases provisioned per region

Schema migration scripts (Flyway) — all tables

UUID shadow columns added alongside int PKs — dual-write begins

Valkey cache deployed — reference data loaded

Read replica provisioned — Kong routes portal reads to replica

Weeks 10–12 · Quarkus REST API

Quarkus REST API skeleton — JWT auth, tenant scoping, health endpoints

Contract endpoints — /api/v2/contracts/* (list, detail, actuals, payout)

Tripcube endpoints — job submit, status poll, signed URL delivery

Contract IQ endpoints — group run, Temporal status, Excel download

Kong routing — /api/v2/* → Quarkus, /* → PHP portal

PHP models refactored to call API endpoints

Weeks 13–16 · Data archive tier

UpCloud Object Storage buckets provisioned per tenant region

Apache Iceberg catalog initialised per tenant namespace

Nightly ETL job — Aurora → Iceberg for data >24 months

Trino cluster deployed on K8s — Aurora + Iceberg connectors

Tripcube queries re-routed through Trino

Selenium functional test suite running on staging

k6 stress tests passing — P95 portal API <500ms

Phase 2 complete — full API layer live, data tier split, PHP portal reads from Quarkus API

Phase 3

Portal migration + PHP retirement

Wks 17–28

Weeks 17–19 · Qute foundation

Base Qute layout template — topbar, nav, footer

Authentication flow — login, session, JWT refresh

HTMX wired for common patterns — table refresh, modal, form post

Dashboard / home view migrated

Target list and contract detail views migrated

Weeks 20–23 · Core modules

Contract wizard — all 15 steps as Qute partials + HTMX step swap

Flown Target tab — Qute table render from API

Flown Income tab — payout SELECT rendered in Qute

Tripcube builder — form + HTMX job status poll

ACM contract configuration — complex filter UI, allow extra time

Weeks 24–26 · Remaining modules + UUID cutover

Contract IQ — Temporal progress polling via HTMX

AI recommendations — inference results in Qute partial

Admin views — user management, client onboarding

UUID dual-write ended — int PK columns dropped

Full Selenium suite passes against Qute portal

Weeks 27–28 · PHP decommission

Kong routes updated — all traffic to Quarkus

PHP portal server decommissioned

Multi-region geo provisioning — APAC and Americas namespaces live

Post-cutover smoke test — all modules verified

WORM audit log confirmed for all Lambda invocations

Weeks 17–28 · UI/UX retention — minimum retraining

Pixel-accurate topbar, sidebar nav, and section layout reproduced in Qute base template — users see no structural change

Existing colour scheme, typography, table density, and button placement carried over — no visual redesign during migration

All menu paths and URLs preserved — bookmarks and muscle memory unaffected

Contract wizard step labels, field names, and order kept identical to current PHP wizard

Flown Target and Flown Income tab names, column headers, and row grouping preserved exactly

Tripcube filter panel layout, field order, and output column names unchanged

Contract IQ group selector and results sheet layout matched to current Excel output format

HTMX interactions (step transitions, async job status) feel faster than current full-page reloads — improvement users notice positively without retraining

User acceptance testing (UAT) session with 2–3 power users per module before PHP cutover — sign-off required before Kong route switch

Any UI improvement requests deferred to a post-migration backlog — zero scope creep during Phase 3

Phase 3 complete — PHP retired, full Quarkus Qute portal, all three regions live

Critical path risk items

Three items have the highest risk of schedule slip and should be spiked early: (1) DISH parser edge cases — the current Java parser has undocumented handling of malformed records accumulated over years of production use; allocate a 2-week buffer. (2) UUID dual-write migration — the most complex database operation in the project; run it on a staging data clone first. (3) ACM contract wizard — the localStorageData blob decoding into relational tables has many edge cases; prototype this in week 1 of Phase 2 before committing to the timeline.

Section 13

13

Monthly Maintenance Activities

Ongoing operational responsibilities across managed services, self-hosted services, security, and data hygiene.

Once the architecture is fully operational, the monthly maintenance burden is largely automated — the managed services (UpCloud Databases, Kubernetes, Object Storage) handle patching and backups. The non-trivial ongoing work is concentrated in the self-hosted services, security rotation, and data tier management. The table below lists every recurring activity, its typical effort, frequency, and responsible party.

Self-hosted service operational responsibilities

These are the services your team owns end-to-end. UpCloud keeps the Kubernetes nodes running; everything below is your responsibility.

Service	What you own	Failure impact	Mitigation
RabbitMQ	Version upgrades · persistent volume backup · queue policy config · DLQ monitoring	HOT file processing stops — messages queue in client buckets until restored	Kubernetes restarts pod on crash. Persistent volume survives pod restart. DLQ alert fires within 60s of first failure.
Temporal server	PostgreSQL backend maintenance · version upgrades (migration scripts between versions) · worker count tuning · history retention policy	Contract actuals workflows and Contract IQ jobs cannot start new runs — in-flight workflows pause	Temporal is stateless compute; its state is in PostgreSQL. Back up PostgreSQL daily. Temporal version upgrades require running tctl schema update — document and test in staging first.
Trino cluster	Coordinator + worker pod health · memory tuning per query load · Iceberg catalog refresh · connector version alignment with Iceberg spec	Historical queries fail — portal falls back to Aurora read replica for recent data only. Tripcube jobs covering >24m data fail.	KEDA can scale Trino workers to zero when idle. Coordinator should always have at least 1 replica. Alerting on coordinator pod restarts.
Kong gateway	Route config updates for new tenants · plugin version upgrades · JWT key rotation · rate limit policy tuning	All portal and API access fails — total outage	Run 2 Kong replicas minimum. Kong is stateless in DB-less mode (config from Git). A pod restart picks up config in seconds.
HashiCorp Vault	Unseal on restart · root token rotation · audit log review · lease renewal monitoring · backup of encrypted secrets	All containers fail to start or rotate credentials — secrets unavailable until Vault is unsealed	Auto-unseal via UpCloud KMS or a Vault-native transit unseal. Never rely on manual unseal in production. Backup Vault snapshot to Object Storage daily.
Grafana + Loki + Tempo	Dashboard maintenance · alert rule updates · log retention policy · Tempo trace retention	Observability only — no impact on processing or portal	Lowest priority self-hosted service. Loki log retention set to 90 days. Tempo trace retention 30 days. Grafana config in Git.

Full monthly maintenance schedule

Activity	Frequency	Effort	Owner	Notes
Automated — no manual action required
UpCloud Managed Databases backup	Daily (automated)	Auto	UpCloud	Point-in-time recovery up to 7 days. Verify restore works quarterly.
UpCloud Kubernetes node patching	Managed rolling update	Auto	UpCloud	UpCloud patches worker nodes. Pod disruption budget ensures zero downtime during rolling update.
Vault dynamic database credential rotation	Every 30 days (automated)	Auto	Vault	Quarkus containers pick up new credentials automatically via Vault agent sidecar.
Nightly ETL — Aurora → Iceberg	Nightly (automated)	Auto	Spark/Glue	Monitor row count reconciliation alert. If ETL fails, Aurora grows — alert fires after 48h gap.
Iceberg compaction + snapshot expiry	Weekly (automated)	Auto	Spark job	Runs `CALL system.rewrite_data_files()` and `expire_snapshots()`. Keeps Parquet files at optimal size and physically removes GDPR-deleted rows.
DLQ monitoring + Grafana alerts	Continuous (automated)	Auto	Grafana	Alert fires within 60s of DLQ message. On-call engineer investigates. Most failures self-resolve on retry.
Temporal workflow history cleanup	Daily (automated)	Auto	Temporal	Retention policy set to 30 days for completed workflows, indefinite for failed workflows requiring investigation.
Weekly — low effort
DLQ review — failed HOT file messages	Weekly	30 min	DevOps	Review any messages in DLQ not auto-resolved. Check Temporal dashboard for stuck workflows. Re-queue or escalate.
Grafana dashboard review	Weekly	20 min	DevOps	Review processing latency trends, DB connection pool utilisation, Trino query times, KEDA scaling events.
WORM audit log integrity check	Weekly	15 min	DevOps	Verify Object Storage WORM policy still active on audit bucket. Check log ingestion rate has not dropped unexpectedly.
Monthly — moderate effort
Kubernetes Helm chart upgrades	Monthly	2–3 hrs	DevOps	Review Helm chart releases for RabbitMQ, Kong, Temporal, Grafana stack, Trino. Test upgrades in staging before production rollout.
Container image base layer updates	Monthly	1–2 hrs	DevOps + Backend	Rebuild all Quarkus container images against latest UBI base. Run full CI/CD pipeline. Catches OS-level CVEs in base images.
Vault snapshot backup verification	Monthly	30 min	DevOps	Restore Vault snapshot to test instance and verify all secrets are accessible. Confirms backup is valid, not just present.
UpCloud Managed Databases restore test	Monthly	1 hr	DevOps	Restore prior day backup to isolated test instance. Run schema validation and row count checks. Confirm RTO is within SLA.
Iceberg partition statistics refresh	Monthly	20 min	DevOps	Run `ANALYZE TABLE` equivalent via Trino to refresh partition statistics. Keeps Trino query planning accurate as data grows.
Tripcube output file cleanup	Monthly	Auto + 10 min	DevOps	Object Storage lifecycle policy expires signed-URL report files after 30 days. Verify policy is active and bucket size is stable.
AI model performance review	Monthly	2 hrs	Backend	Compare prediction inference results against actual contract outcomes for the prior month. Trigger model retrain if MAPE >15% on tier prediction.
Security — dependency vulnerability scan	Monthly	1–2 hrs	Backend	Run `mvn dependency-check:check` across all Quarkus services. Triage CVSS >7.0 CVEs. Patch and redeploy within 7 days for critical findings.
JWT key rotation — per-tenant	Monthly (or on request)	30 min	DevOps	Rotate JWT signing keys in Vault. Kong picks up new key on next config sync. Old key remains valid for 1hr overlap window to drain in-flight requests.
Quarterly — higher effort
Temporal server version upgrade	Quarterly	4–6 hrs	DevOps + Backend	Most operationally complex upgrade. Temporal requires sequential version upgrades (no version skipping). Run schema migration via tctl. Test all workflow types in staging before production. Keep previous version image available for 48hr rollback window.
Quarkus + GraalVM version upgrade	Quarterly	3–5 hrs	Backend	GraalVM native compilation is sensitive to library version changes. Test native compilation in staging. Some reflection config may need updating. Run full test suite including Testcontainers integration tests.
Full DR (disaster recovery) drill	Quarterly	4 hrs	DevOps	Simulate loss of primary DB instance. Measure failover time to read replica promotion. Simulate loss of Vault — test auto-unseal. Document RTO and compare against SLA. Fix any gaps found.
GDPR data audit	Quarterly	2–3 hrs	Backend + DevOps	Verify tenant data residency — confirm no EU data in APAC buckets or vice versa. Process any outstanding right-to-erasure requests via Iceberg row-level delete. Confirm compaction has physically removed deleted rows from prior quarter.
AI model retrain	Quarterly (or on trigger)	3 hrs	Backend	Retrain prediction model on latest 3 years of Iceberg data. Evaluate against holdout set. Deploy new model version if validation MAPE improves. Keep prior version for 30-day rollback.

Monthly effort summary

Routine weekly activities: approximately 1 hour per week. Monthly tasks: approximately 10–14 hours per month total across the team. Quarterly tasks (amortised): approximately 5–6 hours per month equivalent. Total ongoing platform maintenance estimate: 15–20 engineer-hours per month once fully operational — the majority of which is Helm upgrades, security scanning, and the Temporal version upgrade cadence.

Section 06

06

Monthly Cost Estimate

Estimated at current scale (12 airline clients, 4.2M transactions/year). Phase 2 target scale noted separately.

Basis

Prices based on UpCloud published rates (July 2025). UpCloud Object Storage at €0.0118/GB/month. Managed Kubernetes node at €19–€55/month per worker node depending on spec. Managed Databases from €11/month. Self-hosted services (Temporal, Trino, Kong, Vault, Grafana) are software costs only — billed as Kubernetes compute. All estimates in EUR/month.

Component	Service type	Spec / sizing	Est. monthly (EUR)
Kubernetes cluster — all containers
UpCloud Managed Kubernetes — worker nodes (processing)	Managed	3× 4 vCPU / 16 GB RAM nodes (KEDA scales pods per HOT file burst)	€165
UpCloud Managed Kubernetes — worker nodes (API + services)	Managed	2× 4 vCPU / 8 GB RAM nodes (Kong, REST API, Grafana, Loki)	€90
Load balancer (UpCloud)	Managed	1× standard load balancer	€12
Kubernetes subtotal			€267
Database tier
UpCloud Managed Databases — primary (write)	Managed	4 vCPU / 16 GB RAM / 200 GB SSD — supports all tenant schemas	€95
UpCloud Managed Databases — read replica	Managed	4 vCPU / 8 GB RAM — portal report queries only	€55
Valkey (Redis) — self-hosted in Kubernetes	Self-hosted	Runs in existing Kubernetes cluster (1 pod ~512MB)	€0 (within K8s)
Database subtotal			€150
Object Storage
UpCloud Object Storage — HOT file archive (raw files)	Managed	~100 GB (all HOT files ever received, compressed)	€1.20
UpCloud Object Storage — Iceberg Parquet archive	Managed	~300 GB (pre-24m transactions in Parquet, snappy compressed)	€3.54
UpCloud Object Storage — audit WORM log	Managed	~50 GB structured JSON logs with WORM policy	€0.59
UpCloud Object Storage — Tripcube output files	Managed	~20 GB (generated reports, 30-day TTL)	€0.24
Object Storage subtotal			€5.57
Self-hosted services (Kubernetes compute, already in K8s cost above)
RabbitMQ	Self-hosted	1 pod · 1 vCPU / 1 GB RAM — runs in K8s worker	€0 (within K8s)
Temporal server	Self-hosted	1 UpCloud VM — 2 vCPU / 4 GB RAM	€19
Trino cluster	Self-hosted	2 nodes × 4 vCPU / 16 GB RAM in K8s (scales down when idle)	€55
HashiCorp Vault	Self-hosted	1 UpCloud VM — 1 vCPU / 2 GB RAM	€8
Kong gateway	Self-hosted	In K8s — 1 pod, minimal resource	€0 (within K8s)
Grafana + Loki + Tempo	Self-hosted	In K8s — shared worker node	€0 (within K8s)
Self-hosted services subtotal			€82
Network + misc
UpCloud network egress	Managed	~1 TB/month estimated egress (portals, API, report downloads)	€0 (UpCloud public transfer is free to/from UpCloud services)
Private SDN networking between nodes	Managed	UpCloud utility network — free within a region	€0
Estimated monthly total (current scale)	12 clients · 4.2M transactions/year · 850 GB live DB		~€505/month

Phase 2 scale estimate — 25× growth

At 25× current volume (~105M transactions/year, ~21 TB Parquet archive, ~12 TB live DB), the primary cost driver shifts from compute to storage. Kubernetes node count scales linearly with burst processing load but the KEDA scale-to-zero means idle periods cost nothing.

Component	At 25× scale	Est. monthly (EUR)
Kubernetes workers (processing burst)	8–12 nodes auto-scaled by KEDA (avg billing ~6)	€270
Kubernetes workers (API + services)	4 nodes (higher API load)	€180
UpCloud Managed Databases (primary)	8 vCPU / 32 GB RAM / 2 TB SSD	€280
UpCloud Managed Databases (read replica × 2)	2 replicas for portal load distribution	€190
UpCloud Object Storage — Iceberg archive	~21 TB Parquet + manifests	€248
UpCloud Object Storage — HOT file archive	~2 TB raw HOT files	€24
Trino cluster	6 nodes × 8 vCPU / 32 GB RAM	€330
Temporal + Vault + misc VMs	Larger Temporal VM, HA Vault	€80
Estimated monthly total (25× scale)	~105M transactions/year · 21 TB archive · 30 clients est.	~€1,600/month

≈

At 25× scale the equivalent AWS stack (Lambda + Aurora Serverless + S3 + EMR Serverless + API Gateway + CloudWatch) would cost approximately €4,500–€6,500/month — roughly 3–4× higher. The primary saving comes from UpCloud's lower compute pricing and the absence of per-invocation Lambda charges at high volume.

Section 07

07

CI/CD Pipeline

GitHub Actions driven. Container builds, automated tests, and staged rollout on every merge to main.

Every code change to any of the Quarkus containers, the REST API, or the Qute/HTMX portal follows a consistent pipeline. The pipeline is defined in code alongside the application (GitOps). No manual deployments. A failed test at any stage blocks the merge.

Pipeline stages

01 · TRIGGER

PR / push to main

GitHub webhook → Actions runner on UpCloud self-hosted runner

02 · BUILD

Quarkus native compile

GraalVM native-image build · Docker multi-stage · image pushed to UpCloud Container Registry

03 · UNIT

Unit + integration tests

JUnit 5 · Mockito · Testcontainers (ephemeral DB + RabbitMQ)

04 · STAGING

Deploy to staging

Helm upgrade to staging namespace on UpCloud Kubernetes

05 · QA

Functional + stress

Selenium E2E · k6 load test · contract pipeline smoke test against staging data

06 · PROMOTE

Production rollout

Helm upgrade to production namespace · rolling update · Temporal health check

07 · VERIFY

Post-deploy smoke

API healthcheck · RabbitMQ queue depth · Temporal worker count · Grafana alert suppression window

Environment structure

Dev (local)

Docker Compose: Quarkus dev mode · RabbitMQ · MySQL · Vault dev · Temporal dev server. Hot reload. No UpCloud dependency.

Staging (UpCloud)

Full stack on UpCloud Kubernetes, eu-de region. Anonymised production data snapshot. All automated tests run here. Mirrors production topology exactly.

Production (UpCloud)

Multi-region UpCloud Kubernetes. Rolling deploys — zero downtime. Rollback via Helm revision in under 60 seconds. Feature flags via Vault dynamic config.

Infrastructure as code

All UpCloud infrastructure is defined in Terraform (UpCloud Terraform provider). Kubernetes manifests managed via Helm charts. A new tenant region is provisioned by adding a values override file and running the pipeline — no manual console clicks.

Section 08

08

QA and Automated Testing

Four-tier test pyramid covering unit, integration, functional (Selenium), and stress (k6).

Testing is automated end-to-end. No manual regression testing. Every PR must pass unit and integration tests. Staging deployments additionally run Selenium functional tests and a k6 stress test suite. The contract pipeline has a dedicated smoke test that processes a synthetic HOT file and asserts the correct payout result within a tolerance of ±0.01.

Unit tests

JUnit 5 + Mockito
DISH parser correctness — all record types
BigDecimal arithmetic precision assertions
UUID assignment determinism
Contract rule filter logic — every condition branch
Payout calculation edge cases (tier boundary, zero revenue, full refund)
Target: >90% line coverage on business logic

Integration tests

Testcontainers — ephemeral MySQL, RabbitMQ, Redis
Full container chain: Parser → Validator → Enrichment → Persist
Cross-tenant isolation: write tenant A, assert tenant B cannot read
Temporal workflow replay — verify step sequence and retries
Trino federation — query spanning Aurora + Iceberg fixture data
Nightly ETL: verify row counts match before and after Aurora→Iceberg move

Functional (Selenium)

Selenium 4 + Java WebDriver against staging portal
Login and tenant switching
Contract creation wizard — all 15 wizard steps
Flown Target tab — revenue grid renders correctly
Flown Income tab — payout figures match DB assertion
Tripcube report: configure → submit → download CSV
Contract IQ: group → run → verify Excel download
Run on every staging deploy via GitHub Actions matrix

Stress tests (k6)

k6 scripts run against staging after every deploy
HOT file ingestion: 50 simultaneous HOT file uploads
API load: 500 concurrent portal report requests
Tripcube: 20 simultaneous report generation jobs
RabbitMQ saturation: 1,000 messages burst — verify DLQ stays empty
DB connection pool: 200 concurrent persist operations via RDS Proxy equivalent
SLA targets: P95 portal API <500ms · contract populate <90s per contract

Contract pipeline smoke test

A dedicated integration test runs on every staging deploy:

1. Upload synthetic HOT file (50 tickets, known fares, 2 IATA numbers)
2. Assert: Parser container completes within 10s
3. Assert: All 50 rows in dish_transactions with correct UUID PKs
4. Assert: Temporal workflow reaches COMPLETED state
5. Assert: contract_actuals.AdjustedFare sum matches expected ±0.01
6. Assert: contract_payout_results row written with correct tier and payout
7. Assert: Flown Income API endpoint returns payout matching DB row
8. Upload refund HOT file → assert Ref_* fields updated, payout recalculated

Section 09

09

Multi-Geography Tenant Isolation

Per-tenant data residency enforced at infrastructure level — not just application logic.

Each airline tenant is provisioned in the UpCloud region that satisfies their data residency requirement. EU tenants (GDPR Article 46) never have their transaction data leave EU infrastructure. APAC and US tenants are similarly isolated. The shared control plane (Temporal, Kong, Vault, Grafana) runs in a neutral management region and holds no transaction data.

EU — Frankfurt (eu-de)

UpCloud Managed Databases — tenant schemas
UpCloud Object Storage — HOT file archive + Iceberg Parquet
Quarkus container workers (KEDA)
UpCloud Managed Kubernetes namespace
Valkey cache — reference data
All EU airline clients
GDPR Article 46 — data never leaves EU

APAC — Singapore (ap-sg)

UpCloud Managed Databases — tenant schemas
UpCloud Object Storage — HOT file archive + Iceberg Parquet
Quarkus container workers (KEDA)
UpCloud Managed Kubernetes namespace
Valkey cache — reference data
All APAC airline clients
PDPA (Thailand) / PDPC (Singapore) compliant

Americas — Chicago (us-chi)

UpCloud Managed Databases — tenant schemas
UpCloud Object Storage — HOT file archive + Iceberg Parquet
Quarkus container workers (KEDA)
UpCloud Managed Kubernetes namespace
Valkey cache — reference data
All Americas airline clients
Aligns with existing Chicago server environment

Shared management plane — neutral region (no transaction data)

Temporal serverWorkflow orchestration

Kong gatewayJWT · rate limiting · mTLS

HashiCorp VaultSecrets · cert rotation

Grafana + LokiMetrics · logs (no PII)

UpCloud Container RegistryAll container images

GitHub Actions runnersCI/CD pipeline

The management plane sees only workflow metadata, log aggregates, and container images — never raw transaction data or tenant PII. Tenant transaction data is strictly region-local.

How a new tenant region is provisioned

01

Add values file

New Helm values override for target region (e.g. us-chi)

02

Terraform apply

UpCloud provider creates: Kubernetes namespace · DB instance · Object Storage bucket

03

Kong route

New JWT issued in Vault · per-tenant rate limit added to Kong

04

Schema init

Flyway migrations run against new DB instance · tenant schema created

05

Iceberg catalog

New Trino catalog namespace registered · bucket policy locked to region

06

Live

Client receives Object Storage endpoint + JWT · first HOT file drop triggers ingestion pipeline

Section 10

Full System — Block Layout Diagram

Every component and its zone. Zones are separated by deployment boundary — client sub-account, regional data plane, shared management plane, and developer tools. Arrows show data flow direction.

Per-client UpCloud sub-account (one per airline tenant)

Object Storage bucket

HOT file drop · region-locked

→

Intake container

~50 lines · signs JWT · no IP

→

mTLS / HTTPS

to management plane Kong

Shared management plane (no transaction data)

Kong API gateway

mTLS · JWT validate · rate limit · tenant_id inject

→

RabbitMQ

fan-out exchange · DLQ · per-tenant routing key

→

HashiCorp Vault

secrets · dynamic DB creds · cert rotation

·

Temporal server

workflow orchestration · execution history

·

Container Registry

Quarkus images · immutable tags

EU region — Frankfurt

Quarkus workers (K8s)

Parser → Validator → Enrichment → Persist → Contract

↓

UpCloud Managed DB

EU tenant schemas · dish_transactions · contract_payout_results

↓ nightly ETL

Object Storage + Iceberg

Parquet archive · time-travel · GDPR delete

Valkey cache

Carriers · airports · IATA · <1ms

APAC region — Singapore

Quarkus workers (K8s)

KEDA scale-to-zero · GraalVM native

↓

UpCloud Managed DB

APAC tenant schemas · read replica

↓ nightly ETL

Object Storage + Iceberg

Region-locked · PDPC compliant

Valkey cache

Local ref data · TTL 1hr

Americas — Chicago

Quarkus workers (K8s)

Mirrors EU/APAC topology exactly

↓

UpCloud Managed DB

Americas tenant schemas

↓ nightly ETL

Object Storage + Iceberg

Parquet archive · Trino catalog

Valkey cache

Local ref data

Query federation + reporting layer

Trino cluster

UpCloud K8s · federates all 3 region DBs + Iceberg catalogs

→

Quarkus REST API

JWT scoped · routes Valkey / DB replica / Trino

→

Qute/HTMX portal

Server-rendered · same Quarkus process · no Node.js

→

Airline + agency users

Browser · HTTPS

CI/CD + observability (cross-cutting)

GitHub + Actions

Source · PR checks · pipeline trigger

→

Build + test runner

Quarkus native compile · JUnit · Testcontainers

→

Selenium + k6

Functional E2E · stress test on staging

→

Helm deploy

Rolling update to all 3 regional namespaces

·

Grafana + Loki + Tempo

Dashboards · alerts · distributed traces

·

WORM audit log

Object Storage · immutable · every invocation

Transaction data never crosses regional boundaries · management plane holds no PII · all secrets in Vault · kill switch = revoke JWT

Serverless, event-driven,built for 25× growth

Routing split at Kong — how it looks

Phase-by-phase module migration

What moves

PHP portal during this phase

What moves

PHP portal during this phase

What moves

What the team gains

Current

Modernised

Temporal workflow steps per HOT file

Temporal-orchestrated contract pipeline

Serverless query service via Trino

Parallel Temporal fan-out per contract

Real ML model replacing rule-based projection

Hot store — UpCloud Managed Databases

Cold archive — UpCloud Object Storage + Apache Iceberg

Trino — transparent query federation

HashiCorp Vault

Region-locked tenants

Grafana + Loki + Tempo + Temporal

Immutable audit trail

Next.js / React SPA

Quarkus Qute + HTMX recommended

Vite + React SPA

How Qute + HTMX maps to the existing portal modules

Template structure in Quarkus

Recommended team composition

Self-hosted service operational responsibilities

Full monthly maintenance schedule

Phase 2 scale estimate — 25× growth

Pipeline stages

Environment structure

Unit tests

Integration tests

Functional (Selenium)

Stress tests (k6)

Contract pipeline smoke test

EU — Frankfurt (eu-de)

APAC — Singapore (ap-sg)

Americas — Chicago (us-chi)

How a new tenant region is provisioned

Serverless, event-driven,
built for 25× growth