📓 Architect's Notebook

Real lessons from real projects — architecture deep-dives, migration war stories, and the gnarly problems we hit along the way (and how we solved them). Written by the people doing the work, not a marketing team.

No fluff, no recycled listicles. Just the kind of hard-won, in-the-weeds engineering knowledge you only get from shipping production systems for clients across the US, UK, Europe and beyond.

Latest

Architecture · Multi-Tenant SaaS

Multi-Tenant CMS: Serving Many Brands from One Spring Boot Codebase

How we turned a single-brand site into a multi-tenant CMS that serves several businesses from one app — tenant resolved from the Host header, content isolated by website_id, per-tenant themes & SEO, and one admin panel to run them all. Includes the real bugs we hit.

10 min read · Spring Boot, Thymeleaf, Multi-Tenancy, Spring Security

Architecture & Migration

COBOL → Java Microservices Migration: Challenges & How We Solved Them

Modernizing a mainframe COBOL system to Java Spring Boot microservices on Oracle — the real challenges, our AI-assisted pipeline, and the golden-master / parallel-run testing strategy that de-risks the cutover.

14 min read · Architecture, Legacy Modernization, Testing

Financial Systems · Event-Driven Architecture

Event Fabric: Zero-Miss Corporate Action Notifications at Custodian Scale

How we architected a Kafka-driven pipeline that takes corporate events from EMC, cross-references positions in EBS, calculates tax via Apigee, generates PDF advice, and fans out to 2 million customers through AGS — without losing a single notification.

22 min read · Kafka, Microservices, Financial Systems, SWIFT ISO 20022, Custody Banking

💡 New notes are published as we wrap interesting projects. Have a tough modernization or AI-integration problem? Book a free call — we've probably seen something like it.

Multi-Tenant CMS: Serving Many Brands from One Spring Boot Codebase

7 Seven Digital Services · Engineering · 10 min read

Multi-TenancySpring BootThymeleafSpring SecurityPostgreSQLSaaSArchitecture

We had a production site for one business. Then a second business needed a site. The lazy answer is "copy the repo and change the colours" — and now you maintain two codebases forever. Instead we converted the app into a multi-tenant CMS: one deployment, one database, many branded websites. Here's the design, the request lifecycle, and the bugs that actually bit us.

The goal

Multiple businesses, each with its own domain, theme, navigation, pages, blog, SEO and content — all served from a single Spring Boot app and configured from the database. Adding a new business should need data + DNS + an nginx vhost, not a code change or a new deployment.

The core idea: resolve the tenant from the Host header

Every request already tells you which brand it wants — in the Host header. nginx terminates TLS per domain and proxies every domain to the same app, forwarding Host $host. A servlet interceptor reads request.getServerName(), looks up the Website, and stashes it in a request-scoped ThreadLocal that the rest of the request can read.

The request lifecycle

nginx matches the domain's server block, proxies to the app, forwards Host $host.
TenantInterceptor.preHandle calls TenantResolver: normalise the host (strip www.), match a Website by host, else fall back to the default tenant. A dev-only ?tenant= override helps local testing (off in prod).
The resolved Website goes into TenantContext (a ThreadLocal), and a @ControllerAdvice exposes it to every view as ${currentWebsite}.
Controllers ask the service layer for content for that website; Thymeleaf renders the tenant's theme + content.
afterCompletion clears the ThreadLocal — non-negotiable on a pooled thread, or the next request could inherit the wrong tenant.

Low-level design — layers & key classes

Strict one-way dependency: Controller → Service → Repository → Entity. Controllers do HTTP only; tenant filtering lives in the repository query signatures; tenant resolution, auth and audit are cross-cutting interceptors/advice.

Class	Layer	Responsibility
`TenantResolver`	tenant	Pure host→`Website` logic: normalise host, `?tenant=` dev override, default fallback. Unit-tested.
`TenantInterceptor`	tenant	Per request: set `TenantContext` in `preHandle`, clear it in `afterCompletion`.
`TenantContext`	tenant	`ThreadLocal<Website>` — the current tenant; always cleared to avoid thread-pool bleed.
`CurrentWebsiteAdvice`	tenant	`@ControllerAdvice` exposing `${currentWebsite}` to every view.
`SelectedWebsiteService`	service	The tenant an admin is editing (session-based, independent of host).
`SiteContentService`	service	Read-facade returning content filtered by `Website` — the isolation boundary.
`SeoService`	service	Builds the `SeoView` (title/canonical/robots) + JSON-LD from `SeoSettings`.
`SecurityConfig` / `AuditInterceptor`	security	`/admin/**` auth + roles + CSRF scope; every admin mutation written to `audit_log`.

Sequence: a request becomes a themed page

End to end — from the Host header to rendered HTML — including the ThreadLocal cleanup that stops one request bleeding into the next.

Data isolation: the invariant that keeps tenants apart

The rule that prevents one brand leaking into another is boring and absolute: every content query takes a Website. There is no "get all services" — only findByWebsiteAndActiveTrueOrderByDisplayOrderAsc(website). A read-facade service is the single place tenant filtering lives, and admin writes re-verify row.website == selectedWebsite so a tampered id can't move a row across tenants.

🔒 Isolation you can prove beats isolation you hope for. Because the filter is baked into the repository method signature, there is literally no code path that returns another tenant's rows — you can't forget to add a WHERE.

Theming without forking templates

Each Website owns a Theme (logo, favicon, colours, fonts, an optional CSS file). A small Thymeleaf fragment emits CSS custom properties into :root and links the tenant stylesheet:

--color-primary, --color-secondary, --font-heading … set per tenant, consumed by the templates. Swapping a brand's look is a data change, not a deploy. (A neat gotcha: output the values unescaped — ..., not ... — or the quotes in a font-family become ' and CSS ignores them.)

One admin, many sites

A single Spring Security-protected admin runs every tenant. You log in once and pick which site you're editing from a dropdown — the selection lives in the session, deliberately independent of the host, so one operator manages all brands from any domain. Roles split ADMIN (full, incl. user management + audit log) from EDITOR (content only). Every mutating admin request is written to an audit trail by an interceptor.

Key design decisions

Host-based tenancy

Resolve the tenant from request.getServerName() — no path prefixes, no subdomain gymnastics in the app. nginx just forwards the Host.

Shared schema, `website_id`

One database, a tenant FK on every content table. Simple to operate; isolation enforced in the repository layer, not by hope.

Freeze the incumbent

The original brand's templates were never touched. It renders byte-identical while the new tenant runs entirely on the DB-driven CMS.

Config over code

Themes, menus, pages, SEO, blog — all data. New tenant = DNS + nginx vhost + a Website row. No redeploy.

CSRF scoped to /admin

Public forms predate Spring Security and carry no token, so CSRF is required only for the admin area.

Media on a volume

Uploads land on a mounted volume and are served at /media/**, so they survive redeploys. S3 is the next step for multi-host.

Gotchas we actually hit

Thymeleaf fragment name collision. A fragment named head silently matched the literal <head> tag instead of the parameterised fragment. Name fragments so they can't clash with HTML tags (tenantHead, seoHead).
SpEL and needs booleans. ${post.author and post.publishedAt} throws — those are a String and a date. Use ${x != null and y != null}.
ddl-auto=update can't add a NOT NULL column to a populated table (existing rows have no value). Make new columns nullable and backfill, or ship a real migration (Flyway is the grown-up answer).
Multipart + CSRF. The CSRF token in a multipart body isn't read cleanly; we exempted the authenticated upload endpoint rather than fight the filter ordering.
Volume ownership. The container runs as a non-root user, but a mounted /app/media is root-owned by default — uploads fail. chown the dir in the Dockerfile so the fresh named volume inherits the right owner.

Takeaways

Let the Host header pick the tenant — it's already there, and nginx just forwards it.
Push tenant filtering into repository method signatures so isolation can't be forgotten.
Always clear the ThreadLocal after the request on pooled threads.
Make branding, navigation, pages and SEO data, not code — that's what makes onboarding a new brand a config task.
Don't touch the working incumbent; grow new behaviour through the CMS and keep the blast radius small.

🚀 Running one site and about to build a second? Talk to us before you copy-paste the repo. Book a free strategy call →

COBOL → Java Microservices Migration: Challenges & How We Solved Them

7 Seven Digital Services · Engineering · 14 min read

Legacy ModernizationCOBOLJava / Spring BootMicroservicesOracleAI-AssistedTesting

We're modernizing a mainframe COBOL system into Java Spring Boot microservices on Oracle, using an AI-assisted pipeline (Claude Opus) to analyze the legacy code and generate the new components. Here are the challenges that actually bite — and the engineering decisions that keep the migration safe.

The migration architecture at a glance

Here's the full picture — every component involved, from the legacy mainframe on the left, through the AI-assisted migration pipeline in the middle, to the new Java microservices on the right, with the equivalence-testing layer that de-risks the cutover underneath.

In one sentence: collect every legacy input → let the AI pipeline analyze & generate under strict guardrails → land it as domain-bounded Java microservices on Oracle → and only cut over once golden-master, parallel-run, and reconciliation prove the behaviour is identical.

Why this is hard (and why "just rewrite it" fails)

A COBOL system that's run a business for 30 years isn't just code — it's three decades of business rules, edge cases, and regulatory logic, much of it undocumented and written by people who've long since retired. Translating the syntax is the easy 20%. Preserving the exact behaviour is the hard 80%. Almost every failed migration we've seen failed on that 80%.

New to mainframes? A 60-second glossary

This post uses a few mainframe terms. If they're unfamiliar, here's the short version — you can read the rest without prior mainframe experience.

Term	Full form	What it is
COBOL	COmmon Business-Oriented Language	A 1959 programming language that still runs a huge share of banking, insurance, and government systems.
CICS	Customer Information Control System	IBM's mainframe transaction monitor — it handles online screens and user transactions. Think of it as the "application server" of the mainframe.
JCL	Job Control Language	Scripts that tell the mainframe how to run batch jobs — which programs to run, in what order, with which files.
VSAM	Virtual Storage Access Method	IBM's mainframe file/record storage system — one of the places legacy data lives.
DB2	Database 2	IBM's relational database on the mainframe (the others store data in flat files or VSAM).
BMS / mapset	Basic Mapping Support	Defines the layout of the green-screen (3270 terminal) screens users interact with.
Copybook	—	A reusable COBOL snippet of shared field/record definitions — like a header/include file.
PIC	PICTURE clause	How COBOL declares a field's type, size, and format (e.g. number vs text, how many digits).
COMP-3	Computational-3 (packed decimal)	A compact COBOL number format storing two digits per byte — heavily used for money, and the source of precision bugs if mishandled.
EBCDIC	Extended Binary Coded Decimal Interchange Code	The mainframe's character encoding — different from the ASCII/UTF-8 the rest of the world uses, so text must be converted on migration.

💡 Modern-side terms used below: DDD = Domain-Driven Design (designing software around business domains) · Saga = a pattern for a transaction spread across microservices · Strangler Fig = replacing a legacy system gradually, piece by piece · ETL = Extract, Transform, Load (moving/reshaping data) · N+1 = a query anti-pattern where one query secretly triggers many more.

The challenges — and how we solve them

1. Hidden, undocumented business logic

Rules live inside the COBOL: level-88 conditions, REDEFINES, GO TO flow, fall-through PERFORM, and ON SIZE ERROR handlers. Documentation is stale.

How we solve it: we treat docs as a starting point, not truth, and verify behaviour against the real system (see the testing section). Our analysis step explicitly hunts for these constructs instead of assuming clean structured code.

2. Numeric precision — the silent killer

COBOL COMP-3 packed decimal with fixed-point arithmetic and explicit ROUNDED behaviour. If the generated Java uses double/float for money, the numbers will drift — and in finance, a cent of drift is a production incident.

How we solve it: every numeric PIC maps to BigDecimal with explicit scale and the exact RoundingMode. We replicate COBOL truncation/rounding precisely, and we verify it with reconciliation tests — never trust the model to "get math right" by default.

3. Avoiding the distributed monolith

COBOL has no domain boundaries — everything shares working storage. Convert one program to one microservice and you get a distributed monolith: all the coupling of the original, plus network latency and partial failure. Worse than what you started with.

How we solve it: Domain-Driven Design. We define service boundaries by business capability, not by screen or program, and migrate incrementally with the Strangler Fig pattern so old and new run side by side.

4. Transactions & consistency (CICS → microservices)

CICS (Customer Information Control System — the mainframe's transaction monitor) gave us ACID (atomic, consistent, isolated, durable) transactions across everything. Pseudo-conversational state and two-phase commits don't survive a move to distributed services.

How we solve it: keep a transaction inside one service wherever possible; use the Saga + Outbox patterns for the genuinely cross-service flows; design boundaries to minimize distributed transactions in the first place.

5. Data migration (VSAM/DB2/flat → Oracle)

EBCDIC→ASCII encoding, hierarchical/denormalized copybooks reshaped into normalized Oracle tables, six-digit date windowing.

How we solve it: ETL with field-level reconciliation reports (row counts, checksums, value diffs) and a dual-run period before cutover.

6. Batch & JCL — the part everyone forgets

Nightly batch jobs — orchestrated by JCL (Job Control Language, the mainframe's batch scripting) doing sort/merge and sequential processing — are a huge slice of mainframe value and don't map to request/response services.

How we solve it: rebuild it on Spring Batch, rethought as chunked/parallel/streaming — and test it the same way as the online flows.

7. Performance regression

Mainframe batch is brutally fast. Java + network hops + Oracle + chatty service calls (N+1 on the data gateway) can be slower.

How we solve it: performance-test on realistic data volumes early, and watch DB access patterns from day one rather than at the end.

Our AI-assisted pipeline — what works

We use Claude Opus to analyze COBOL programs and generate the Java components, but with hard guardrails. The things that make it reliable:

Upfront dependency collection. We scan every CALL / PERFORM / COPY and refuse to start until all called programs, copybooks, and mapsets are present. Missing dependencies are the #1 cause of incomplete migrations — so we made it a blocking gate.
100% field-completeness validation. We extract every field reference number from the documentation and verify the generated code matches — catching the silent dropping of fields that AI is prone to.
A single source of truth. Generated code may only use database field names from the field-mapping JSON — never invented ones. This is the key guardrail against the model hallucinating columns.
Separation of client vs backend validators and registered enums, so the output is consistent across hundreds of programs instead of a different style each time.

⚠️ The honest caveat: AI translates structure well, but it can miss subtle business logic and edge cases, and it's non-deterministic. Generated code that compiles and is complete is not the same as code that behaves identically. Which brings us to the most important part…

The thing that actually de-risks the cutover: golden-master + parallel-run testing

There are no existing unit tests for a 30-year-old COBOL system. So how do you know the Java does the same thing? You don't reason about it — you measure it, with two complementary techniques.

Golden-master (characterization) testing

Capture a large set of real inputs and the mainframe's exact outputs as fixtures — production transaction logs are ideal because they cover the weird edge cases nobody documented. Replay those same inputs through the new Java service in CI and diff the outputs byte-for-byte (numbers, formatting, error codes, everything). Any mismatch is a regression to fix before launch.

Pull fixtures from production data — synthetic test cases miss the 1-in-10,000 path that breaks in prod.
Lock down non-determinism — freeze system date/time, sequence numbers, and user IDs so diffs reflect logic, not clocks.
Diff at field granularity so a failure points to the exact field/rule, not "outputs differ."
Run it in CI on every generated module — this becomes the regression suite the legacy system never had.

Parallel / shadow run

Before cutover, run both systems on live traffic: the mainframe serves users as normal, while the Java service processes the same requests in the background (shadow mode, zero user impact). Compare results continuously and alert on any divergence. Once divergence is near-zero for long enough, do a canary cutover — route a small slice of real users to Java, watch, then ramp to 100%, keeping the mainframe as instant rollback.

For batch and data, the equivalent is the dual-run + reconciliation described earlier: run both nightly, diff the outputs and the resulting Oracle state, and only retire the COBOL job once they match for a full cycle.

💡 The one-liner we live by: "Compiles" and "complete" are necessary but not sufficient — behavioural equivalence, proven on real data, is the only acceptance criterion. Golden-master proves it offline; parallel-run proves it live; reconciliation proves it for data. Together they turn a terrifying big-bang into a boring, reversible, incremental switch.

Takeaways

Migrate behaviour, not syntax — capture intent and prove equivalence.
Use BigDecimal with exact rounding for every number that matters.
Draw service boundaries by domain, not by program — or you'll build a distributed monolith.
AI accelerates the 80% of mechanical work — but keep hard guardrails (single source of truth, completeness gates) and humans + tests as the final authority.
Golden-master + parallel-run + reconciliation is what lets you cut over without holding your breath.

📨 Working on a legacy modernization or an AI-assisted migration? We do this for clients worldwide. Book a free strategy call →

Event Fabric: Zero-Miss Corporate Action Notifications at Custodian Scale

7 Seven Digital Services · Financial Systems Engineering · 22 min read

Apache KafkaJava MicroservicesSWIFT ISO 20022Custody BankingEvent-DrivenAGSApigeeTax ProcessingPDF Generation

Architecture Design Document

Corporate Action Notification Platform — End-to-End Event Fabric

Status: Approved v2.4

Document IDADD-CORP-EVENTS-001

Last RevisedJune 2025

Architecture StyleEvent-Driven Microservices

Primary MessagingApache Kafka (Confluent)

API GatewayApigee X

Regulatory ScopeSEBI LODR · FADP · OFAC · EU Sanctions

Problem Statement

A custodian bank must legally notify every eligible customer of every corporate action before the event deadline — with correct tax withholding, in the customer's preferred delivery channel, without duplication, and in compliance with cross-border data sovereignty laws and applicable sanctions regimes. A manual or batch-file approach cannot meet these SLAs at scale (>2 million customers, peak 40,000 events/second during dividend season).

Architecture Decision Records (ADRs)

ADR #	Decision	Rationale	Trade-offs	Status
ADR-001	Kafka as the primary messaging backbone (not REST point-to-point)	Decouples producers and consumers; replay capability for DR; handles 40K events/sec with consumer-group fan-out; durable log for audit	Ops complexity (ZooKeeper/KRaft, schema registry, consumer lag monitoring). REST simpler to reason about but cannot absorb event bursts without queuing.	Accepted
ADR-002	Aspen as central orchestrator (orchestration over choreography)	Corporate action processing has strict sequencing: entitlement → tax → encoding → PDF → delivery. Choreography (each service emitting its own events) makes this ordering hard to enforce and debug.	Aspen becomes a critical dependency. Mitigated by horizontal scaling and bulkhead isolation per event type.	Accepted
ADR-003	ED Services as a data-sovereignty boundary for Swiss accounts (not inline encoding in Aspen)	Swiss FADP prohibits personal account data from leaving Swiss jurisdiction without adequate safeguards. ED Services runs inside the Swiss data zone and issues tokenised references that travel cross-border instead of raw account details.	Additional network hop for every Swiss account. Acceptable: ED Services is deployed in the same data centre region and adds <30ms P99.	Accepted
ADR-004	Claim Check pattern for bulk batch data to TaxC and ED Services (not REST body transfer)	REST payloads cannot carry 500K account records. Claim Check (EIP): Aspen writes bulk file to object store, publishes file-path token via Kafka; TaxC reads the file directly. Eliminates API size limits, reduces network retransmission cost, provides natural checkpointing.	Introduces shared file store dependency. Object store must be highly available. File cleanup policy required to prevent unbounded growth.	Accepted
ADR-005	Dedicated TaxC microservice (not inline tax logic in Aspen)	India's Income Tax Act sections 194 and 195 change annually each Union Budget. A dedicated service has its own release cycle — tax rule changes deploy without touching Aspen. DTAA treaty lookups require external document management integration which should not live in a processing service.	Extra service to operate. Mitigated by TaxC's stateless nature — scales to zero between events.	Accepted
ADR-006	Real-time Sanctions Screening Service before entitlement processing (not post-hoc filtering in AGS)	Sending a corporate action advice to a sanctioned entity — even if later cancelled — may constitute a regulatory violation. Screening must happen before any processing, not at the delivery layer.	Screening adds 20–80ms per account lookup. Cached SDN list (refreshed hourly) makes this acceptable. Must handle false positives carefully — fuzzy matches go to compliance queue, not automatic block.	Accepted
ADR-007	Separate Kafka topics for mandatory vs voluntary corporate actions	Mandatory events (dividends, splits) are high-volume and time-insensitive beyond record date. Voluntary events (rights, buybacks) are low-volume but have hard cut-off deadlines. Sharing a topic causes mandatory volume to delay voluntary events at exactly the worst moment.	Two consumer groups, two monitoring configurations. Worth the operational overhead.	Accepted
ADR-008	Transactional Outbox pattern for all Kafka publishes (not direct produce)	Guarantees that a Kafka message is published if and only if the corresponding DB write commits. Without Outbox, a crash between DB write and Kafka produce creates "silent loss" — the event is processed but never delivered downstream.	Requires Debezium CDC running against the outbox table. Infrastructure cost justified by zero-miss SLA requirement.	Accepted

Non-Functional Requirements

Throughput

40,000 events/sec at peak (dividend season). Kafka partitions sized for 10× headroom. PDF workers autoscale via KEDA.

Latency

Mandatory advice delivered to AGS within 6 hours of record date close. Voluntary advice delivered within 2 hours of EMC publish, minimum 5 business days before cut-off.

Exactly-Once

Zero duplicate advice letters. Achieved via Kafka idempotent producers + application-level deduplication table + transactional outbox.

Availability

99.95% for Aspen and AGS. Kafka cluster: 3-broker minimum with replication factor 3. File server: RAID-10 with async S3 replication.

Data Sovereignty

Swiss FADP: account PII never leaves Swiss data zone. EU GDPR: right to erasure supported — soft-delete on advice table, hard-delete after 7-year retention.

Sanctions Compliance

OFAC SDN, EU Consolidated, UK OFSI, SECO lists refreshed hourly. Zero tolerance: blocked accounts never receive advice; all blocks logged to immutable compliance audit log.

Audit Trail

Every event, every entitlement calculation, every delivery attempt logged with timestamp, actor, outcome. Retained for 7 years per SEBI requirements.

Disaster Recovery

RTO: 4 hours. RPO: 0 (Kafka log + outbox table replicated cross-AZ). Full event replay from Kafka topic offset supported.

Data & Security Zone Boundaries

🔴 Restricted Zone

Swiss account raw PII
Beneficial owner names (CH)
IBAN / local account refs
Never leaves Switzerland

🟡 Sanctions-Gated Zone

All customer accounts
Screened before processing
Blocked = compliance queue
OFAC · EU · OFSI · SECO

🔵 Processing Zone

Aspen orchestration
Tokenised account refs only
Tax + entitlement data
Generated PDFs (encrypted)

🟢 Delivery Zone

AGS → InView
AGS → Print Vendor
AGS → DocuSign
TLS 1.3 on all channels

🟣 Compliance Zone

Sanctions audit log
Delivery receipts
DLQ dashboards
7-year immutable store

System Interaction Diagram

In October 2008, Volkswagen briefly became the most valuable company on earth — not because of fundamentals, but because short-sellers hadn't accounted for a corporate action. Porsche's surprise disclosure that it controlled 74% of VW's shares triggered one of the most violent short squeezes in history. Corporate actions are low-frequency, high-stakes events. If your notification system misses even one, the consequences can be irreversible — missed rights deadlines, wrong tax withholding, compliance breaches. Here is how we built a system that doesn't miss any.

What is a Corporate Action, and Why Does Getting it Wrong Cost Millions?

A corporate action is any event initiated by a listed company that materially affects its securities or shareholders. Custodian banks (like Citi Global Custody, BNY Mellon, State Street) are legally obligated to notify every eligible client in time for them to act — or to act on their behalf when no instruction is received.

Type	Mandatory / Voluntary	Example	Miss cost
Cash Dividend	Mandatory	TCS ₹28/share	Wrong tax withholding → regulatory fine
Rights Issue	Voluntary	HDFC Bank 1:5 rights @ ₹1200	Client misses subscription → capital loss + legal claim
Stock Split / Bonus	Mandatory	Infosys 1:1 bonus	Position mismatch → trading errors
Merger / Acquisition	Mandatory / Elective	HDFC-HDFC Bank merger	Wrong share exchange → multi-crore reconciliation
Buyback	Voluntary	TCS buyback @ ₹4150/share	Client can't tender → missed premium
AGM Vote	Voluntary	Annual proxy voting	Compliance violation for institutional clients

The industry standard notification format is SWIFT ISO 15022 MT564 (Corporate Action Notification) for legacy systems, now migrating to ISO 20022 SEEV.031. Citibank's Global Custody — one of the world's largest with assets under custody exceeding $25 trillion — pioneered Straight-Through Processing (STP) for corporate actions through their CAST (Corporate Action STP) initiative, eliminating manual touchpoints that historically caused notification failures. Our architecture draws from the same STP philosophy: every step automated, every failure surfaced, every message idempotent.

The System Landscape

Before touching a line of code, it helps to understand what each system owns and why the boundaries are drawn where they are.

System	Full Name	Owns	Integration style
`EMC`	Event Master Central	All corporate action event data — the master record of what happened, to which ISIN, on which date	Kafka producer (real-time)
`SMC`	Security Master Central	Static reference data — ISIN, ticker, exchange, asset class, currency	Daily batch to EBS
`EBS`	Enhanced Banking System	Customer positions (what they hold, how many shares, in which account) and entitlements	Kafka producer to Aspen
`One Source`	—	Customer demographic data — name, address, communication preferences, language, NRI / resident status	REST / batch pull by Aspen
`Aspen`	—	Orchestration — joins event + position + demographic + tax, generates PDF advice, publishes to AGS	Kafka consumer + REST client + Kafka producer
`Apigee`	—	API Gateway — auth, rate limiting, routing to downstream microservices	REST
`TaxC`	Tax Calculation Service	Withholding tax rules — resident, NRI, FII, DTAA treaty rates	REST via Apigee
`ED Services`	Encoder / Decoder Services	Swiss account number encoding — IBAN ↔ local format, beneficial owner data masking	REST via Apigee
`AGS`	Advice Generation System	Delivery orchestration — picks up PDF path from Kafka, fans out to InView (soft copy) and Print Vendor (hard copy)	Kafka consumer + REST producer
`InView`	—	Electronic document delivery portal — customer-facing web access to soft copies	REST from AGS
`DocuSign`	—	e-Signature workflow for elective corporate actions requiring customer instruction	REST from AGS
`Print Vendor`	—	Physical mail dispatch for customers preferring paper or with no e-delivery preference	REST / SFTP from AGS

Data Flow: A Dividend Hits the Wire

Let's trace a single event — Infosys declares a ₹21/share dividend with a record date of 2024-08-30 — from source to customer letterbox.

Step 1 — SMC → EBS: Daily Security Master Sync

Overnight, SMC pushes the full security master to EBS via batch. This includes the Infosys ISIN (INE009A01021), exchange (NSE/BSE), currency (INR), asset class (equity). EBS needs this to resolve positions correctly. Any ISIN that hasn't synced yet means EBS can't tell Aspen who holds it.

// SMC → EBS batch payload (simplified, inspired by ISO 20022 FinancialInstrumentAttributes)
{
  "isin": "INE009A01021",
  "ticker": "INFY",
  "issuerName": "Infosys Limited",
  "exchange": ["NSE", "BSE"],
  "currency": "INR",
  "assetClass": "EQUITY",
  "country": "IN",
  "sedol": "6205122",
  "cusip": null,
  "effectiveDate": "2024-08-29"
}

Step 2 — EMC → Aspen: Corporate Action Event via Kafka

EMC receives the dividend announcement (sourced from exchange feeds, DTCC equivalent in India is NSDL/CDSL depository notifications) and publishes to the corp-action-events Kafka topic. The message is keyed on isin + eventId to ensure all messages for the same event land on the same partition — preserving order.

// corp-action-events Kafka message (EMC → Aspen)
// Key: "INE009A01021:CA-2024-INFY-DIV-001"
{
  "eventId":       "CA-2024-INFY-DIV-001",
  "eventType":     "CASH_DIVIDEND",
  "isin":          "INE009A01021",
  "issuerName":    "Infosys Limited",
  "recordDate":    "2024-08-30",
  "paymentDate":   "2024-09-13",
  "ratePerShare":  21.00,
  "currency":      "INR",
  "mandatory":     true,
  "status":        "CONFIRMED",
  "sourceSystem":  "EMC",
  "publishedAt":   "2024-08-15T09:32:11Z",
  "checksum":      "a3f7c821"   // for deduplication
}

Step 3 — Aspen: Entitlement Calculation

Aspen's Event Ingestion Service consumes the Kafka message. It then calls EBS (via Kafka reply or REST, depending on SLA) to fetch all customer positions in INE009A01021 as of the record date. This is not as simple as "who holds the stock today" — it must use the settled position snapshot at record date close, which means trades from T-1 may not yet be in the system.

// EBS position query — settled positions at record date
GET /positions?isin=INE009A01021&asOf=2024-08-30&settlementStatus=SETTLED

// EBS response — one entry per customer account
[
  { "accountId": "CUST-0044821", "custodianRef": "CIT-IN-44821",
    "isin": "INE009A01021", "quantity": 5000, "currency": "INR",
    "settlementDate": "2024-08-28", "accountType": "NRI_NRE" },
  { "accountId": "CUST-0099312", "custodianRef": "CIT-IN-99312",
    "isin": "INE009A01021", "quantity": 1200, "currency": "INR",
    "settlementDate": "2024-08-29", "accountType": "RESIDENT_INDIVIDUAL" },
  ...  // potentially thousands of records
]

Step 4 — Aspen → Apigee → Aggregator → TaxC + ED Services

For each customer, Aspen calls the Aggregator microservice through Apigee. The Aggregator orchestrates two parallel downstream calls: TaxC (to compute the withholding tax) and ED Services (if it's a Swiss-domiciled account, encode the account reference in the required format).

// Aggregator orchestrates TaxC + ED Services in parallel
CompletableFuture<TaxResult>     taxFuture = taxClient.calculate(account, event);
CompletableFuture<EncodedAccount> edFuture  = edClient.encode(account);

CompletableFuture.allOf(taxFuture, edFuture).join();

TaxResult     tax     = taxFuture.get();
EncodedAccount encoded = edFuture.get();

// TaxC response — resident individual, 10% TDS on dividend > ₹5000
{
  "accountId":       "CUST-0099312",
  "grossAmount":     25200.00,   // 1200 shares × ₹21
  "taxRate":         0.10,
  "taxAmount":       2520.00,
  "netAmount":       22680.00,
  "taxRegime":       "RESIDENT_INDIVIDUAL",
  "section":         "194",      // Income Tax Act section
  "dtaaApplicable":  false
}

// TaxC response — NRI NRE account, 20% TDS (or DTAA rate if applicable)
{
  "accountId":       "CUST-0044821",
  "grossAmount":     105000.00,  // 5000 shares × ₹21
  "taxRate":         0.20,
  "taxAmount":       21000.00,
  "netAmount":       84000.00,
  "taxRegime":       "NRI_NRE",
  "section":         "195",
  "dtaaApplicable":  true,
  "dtaaTreaty":      "INDIA_UK"  // treaty rate may reduce from 20% to 15%
}

Step 5 — Aspen: PDF Generation & File Server Storage

The PDF is generated using a Thymeleaf/FreeMarker template engine running inside Aspen's PDF Generation Service. Each customer gets a personalised advice letter — their name, account number, the gross/net entitlement, tax breakdown, and payment date. The PDF is written to a shared object store (NFS/S3-compatible), and the server path is captured.

// PDF stored to: /advice/2024/08/15/CA-2024-INFY-DIV-001/CUST-0099312.pdf
AdviceDocument doc = AdviceDocument.builder()
    .eventId("CA-2024-INFY-DIV-001")
    .customerId("CUST-0099312")
    .customerName("Rajesh Sharma")
    .isin("INE009A01021")
    .issuer("Infosys Limited")
    .quantity(1200)
    .grossAmount(new BigDecimal("25200.00"))
    .taxAmount(new BigDecimal("2520.00"))
    .netAmount(new BigDecimal("22680.00"))
    .paymentDate(LocalDate.of(2024, 9, 13))
    .recordDate(LocalDate.of(2024, 8, 30))
    .build();

byte[] pdfBytes = pdfService.generate(doc);
String path     = fileStore.write("advice/2024/08/15/CA-2024-INFY-DIV-001/CUST-0099312.pdf", pdfBytes);
kafkaTemplate.send("ags-notifications", doc.getCustomerId(), new AgsEvent(doc.getEventId(), doc.getCustomerId(), path));

Step 6 — AGS: Fan-out to Delivery Channels

AGS consumes the ags-notifications Kafka topic. For each message, it reads the customer's delivery preferences from a config store and fans out accordingly. A customer with both electronic and paper preferences gets both — InView for immediate access and Print Vendor for the physical letter.

// AGS delivery orchestration
switch (customer.getDeliveryPreference()) {
    case ELECTRONIC_ONLY  -> inViewClient.deliver(pdfPath, customer);
    case PAPER_ONLY       -> printVendorClient.dispatch(pdfPath, customer);
    case BOTH             -> {
        inViewClient.deliver(pdfPath, customer);
        printVendorClient.dispatch(pdfPath, customer);
    }
    case ELECTIVE         -> {
        inViewClient.deliver(pdfPath, customer);
        docuSignClient.sendForSignature(pdfPath, customer);  // voluntary actions
    }
}

Problems We Hit in Production — and How We Solved Them

Problem 1 — EMC Duplicate Events (The "Double Advice" Incident)

EMC occasionally resends the same corporate action event — a network timeout causes a producer retry, or a manual re-announcement corrects minor data. In our first production run, 3% of customers received duplicate advice letters for the same dividend. Complaints started within hours.

Root cause: Aspen was not idempotent. It treated every Kafka message as a new event.

Fix: We introduced a deduplication table in PostgreSQL. Before processing any event, Aspen checks if (eventId, customerId) has already been processed successfully. We made the insert idempotent using an ON CONFLICT DO NOTHING clause, ensuring only the first message per pair triggers processing — even under parallel consumer restarts.

-- Deduplication table in Aspen's PostgreSQL
CREATE TABLE processed_events (
    event_id       VARCHAR(100),
    customer_id    VARCHAR(50),
    processed_at   TIMESTAMPTZ DEFAULT now(),
    pdf_path       TEXT,
    status         VARCHAR(20),
    PRIMARY KEY (event_id, customer_id)
);

-- Idempotent insert before processing
INSERT INTO processed_events (event_id, customer_id, status)
VALUES (:eventId, :customerId, 'IN_PROGRESS')
ON CONFLICT (event_id, customer_id) DO NOTHING;

-- If 0 rows inserted → already processed → skip
-- If 1 row inserted → new event → proceed with full pipeline

⚠️ Pitfall: Using Kafka's built-in enable.idempotence=true only prevents duplicates within a single producer session. Once EMC restarts or a manual re-announcement fires from a different producer instance, Kafka sees it as a legitimate new message. Application-level deduplication is mandatory for financial systems.

Problem 2 — The Race Between Record Date and Settlement (T+2 Hell)

Indian equity markets settle on a T+2 cycle. A customer buys Infosys shares on August 28 (two days before the record date of August 30). Their trade is confirmed on August 28 in EBS, but the shares don't settle into their account until August 30 — the record date itself, and often after market open. EMC fires the event at 9:00 AM on August 30. EBS queried at that moment shows the position as pending settlement, not settled. We excluded the customer. They complained — rightly — that they were entitled.

Fix: We introduced a settlement wait window. Aspen does not process events at the instant EMC fires them on record date. Instead, it schedules processing for 6:00 PM on record date, after which all same-day settlements are guaranteed to have cleared through the depository. EMC events arrive in a staging queue and are held until the processing window opens.

// Aspen — delayed processing using a scheduled trigger
@KafkaListener(topics = "corp-action-events")
void onEvent(CorporateActionEvent event) {
    LocalDate recordDate   = event.getRecordDate();
    LocalDate today        = LocalDate.now();

    if (recordDate.isAfter(today)) {
        // Future event — park it for later
        eventScheduler.scheduleFor(event, recordDate.atTime(18, 0));
    } else if (recordDate.equals(today)) {
        // Record date is today — wait for settlement window
        eventScheduler.scheduleFor(event, today.atTime(18, 0));
    } else {
        // Past record date (re-announcement or late arrival) — process immediately
        entitlementService.process(event);
    }
}

💡 This matches the industry practice documented in the SMPG (Securities Market Practice Group) guidelines for ISO 20022 corporate action processing: custodians should use settled position at record date close, not intraday positions. DTCC's CA WebUI similarly stamps entitlements at end-of-day record date positions.

Problem 3 — TaxC Timeout for DTAA Edge Cases

For NRI customers with DTAA (Double Taxation Avoidance Agreement) treaties, TaxC needs to verify the customer's tax residency certificate (TRC) stored in a separate document management system. This lookup occasionally took 8–12 seconds — far beyond our 3-second SLA for the Aggregator call. The Aggregator was timing out, causing Aspen to fall back to the default 20% rate — which for some customers was higher than their treaty rate, resulting in over-deduction.

Fix: We split the processing path. Customers with DTAA eligibility flags are routed to an async enrichment queue. Aspen generates a preliminary advice with the default rate and marks the record as PENDING_DTAA_ENRICHMENT. A separate job polls TaxC for the resolved rate, regenerates the PDF if the rate changed, and marks delivery complete only then. We added a 48-hour SLA on the async path with an alert if it breaches.

// Two-phase processing for DTAA accounts
if (account.isDtaaEligible()) {
    AdviceRecord preliminary = generateAdvice(event, account, DEFAULT_NRI_RATE);
    preliminary.setStatus(AdviceStatus.PENDING_DTAA_ENRICHMENT);
    adviceRepository.save(preliminary);
    dtaaEnrichmentQueue.enqueue(new DtaaEnrichmentRequest(account, event, preliminary.getId()));
} else {
    TaxResult tax = taxClient.calculate(account, event);   // synchronous — fast path
    AdviceRecord advice = generateAdvice(event, account, tax);
    advice.setStatus(AdviceStatus.READY);
    adviceRepository.save(advice);
    kafkaTemplate.send("ags-notifications", advice);
}

Problem 4 — Swiss Account Encoding Failures (ED Services Edge Case)

ED Services translates account identifiers between IBAN format (used by Swiss banks — e.g., CH56 0483 5012 3456 7800 9) and the internal custodian reference format. This worked fine for 99.8% of accounts. The 0.2% that failed had beneficial owner names containing umlauts (ä, ö, ü, ß) — stored as UTF-8 in our system but expected as ISO-8859-1 by the downstream encoding engine. Encoding failures caused those accounts to be skipped entirely with no error surfaced to operations.

Fix: We added a character encoding normalisation step in the Aggregator before calling ED Services. We also added a dead-letter topic (ed-services-dlq) so encoding failures are visible in the ops dashboard rather than silently swallowed.

// Character normalisation before ED Services call
String normalisedName = Normalizer.normalize(account.getBeneficialOwnerName(), Normalizer.Form.NFD)
    .replaceAll("[\\p{InCombiningDiacriticalMarks}]", "")
    .replace("ß", "ss");

// Dead-letter handling for encoding failures
try {
    return edClient.encode(account.withBeneficialOwner(normalisedName));
} catch (EdEncodingException ex) {
    kafkaTemplate.send("ed-services-dlq",
        new DlqEntry(account.getId(), ex.getMessage(), event.getEventId()));
    return EncodedAccount.fallback(account);   // use unencoded account — flag for manual review
}

Problem 5 — Kafka Consumer Lag During Dividend Season

The Indian market pays the bulk of dividends in Q1 (April–June) and Q4 (January–March). During Infosys, TCS, and Wipro dividend week — all three announcing within 48 hours of each other — EMC was producing 40,000 events/second. Aspen's consumer group had 12 partitions and 12 consumers. The processing lag hit 4 hours, which meant some customers were receiving advice after payment date — legally problematic.

Fix: We introduced horizontal scaling triggers via Kubernetes HPA (Horizontal Pod Autoscaler) keyed to consumer lag, monitored through Kafka's consumer_lag metric exposed to Prometheus. When lag exceeded 10,000 messages, HPA scaled Aspen's PDF generation pods from 12 to 48. We also moved PDF generation out of the main processing loop into a separate worker pool with its own autoscaling, so the entitlement calculation pipeline wasn't blocked by slow PDF rendering.

// Kubernetes HPA — scale on Kafka consumer lag (via KEDA)
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: aspen-pdf-scaler
spec:
  scaleTargetRef:
    name: aspen-pdf-worker
  minReplicaCount: 4
  maxReplicaCount: 64
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka-cluster:9092
        consumerGroup: aspen-pdf-workers
        topic: pdf-generation-queue
        lagThreshold: "10000"   # scale up when lag > 10k messages

Problem 6 — AGS Thundering Herd on Large Dividends

A large Nifty 50 company paying dividend has ~500,000 eligible custodian customers. When Aspen finishes processing and publishes 500,000 messages to ags-notifications within 20 minutes, AGS received them all in a burst. AGS then fan-out called InView and Print Vendor simultaneously for each one. InView's REST endpoint was rated at 5,000 requests/minute. The result: 495,000 HTTP 429s (Too Many Requests), a full retry storm, and InView going down for 20 minutes.

Fix: We introduced a rate-limited work queue between the Kafka consumer and the delivery clients. AGS now pulls from Kafka at its own pace and uses a token bucket (implemented with Guava's RateLimiter) per delivery channel. InView is capped at 4,000 req/min (headroom below the 5,000 limit), Print Vendor at 1,000 req/min. We also staggered delivery: electronic first (time-sensitive), print batch dispatched in off-peak hours (11 PM).

// AGS rate-limited delivery
private final RateLimiter inViewLimiter   = RateLimiter.create(66.0);  // 66/sec = 4000/min
private final RateLimiter printLimiter    = RateLimiter.create(16.6);  // 16.6/sec = 1000/min

void deliver(AgsEvent event, Customer customer) {
    inViewLimiter.acquire();   // blocks until a permit is available
    inViewClient.deliver(event.getPdfPath(), customer);

    if (customer.wantsPaperCopy() && isPrintWindow()) {
        printLimiter.acquire();
        printVendorClient.dispatch(event.getPdfPath(), customer);
    }
}

Problem 7 — Exactly-Once Delivery and the Advice Letter Already Sent

Kafka guarantees at-least-once delivery by default. A consumer rebalance mid-batch caused AGS to reprocess 1,200 messages it had already sent to InView. Customers received their advice letter twice — not a compliance breach, but a significant customer experience failure and an operational support spike.

Fix: We enabled Kafka's transactional producer in Aspen and idempotent consumer pattern in AGS. AGS writes a delivery receipt to its own PostgreSQL table in the same transaction as the Kafka offset commit. If the consumer restarts and sees the same message again, the delivery receipt lookup short-circuits before the InView call is made.

// AGS — transactional idempotent delivery
@Transactional
void processAgsEvent(AgsEvent event) {
    if (deliveryReceiptRepo.existsByEventIdAndCustomerId(event.getEventId(), event.getCustomerId())) {
        log.info("Already delivered — skipping. eventId={} customerId={}", event.getEventId(), event.getCustomerId());
        return;
    }

    inViewClient.deliver(event.getPdfPath(), event.getCustomerId());

    DeliveryReceipt receipt = new DeliveryReceipt();
    receipt.setEventId(event.getEventId());
    receipt.setCustomerId(event.getCustomerId());
    receipt.setChannel("INVIEW");
    receipt.setDeliveredAt(Instant.now());
    deliveryReceiptRepo.save(receipt);   // committed atomically — if this fails, inViewClient call is rolled back too
}

Problem 8 — Voluntary Corporate Action Cut-Off Miss

For a rights issue, customers typically have 15–21 days from notification to instruct their custodian. We had one incident where a batch of 8,000 customers received their InView notification on Day 13 of a 15-day window — two days before cut-off. Many missed the deadline. Investigation found a Kafka consumer group rebalance had paused processing for 38 hours.

Fix: We added an SLA breach detector. For voluntary corporate actions, Aspen stamps an urgentDeliveryBy timestamp = recordDate minus 5 business days. AGS checks this timestamp and moves urgent events to a priority Kafka topic (ags-notifications-urgent) that bypasses the rate limiter and uses dedicated consumer threads.

💡 This is the same logic Citi Global Custody describes in their published STP documentation: "Time-sensitive voluntary events are segregated at source and processed on a guaranteed-delivery path with escalation triggers." The industry term is event urgency stratification.

Architecture Decisions We'd Make Differently

Outbox pattern from day one. We added the transactional outbox (writing to DB + Kafka in one local transaction) six months after go-live, after the exactly-once incident. It should have been in the initial design. The Debezium CDC (Change Data Capture) pattern makes this clean — write to DB, let Debezium publish to Kafka from the WAL log.
Separate the PDF storage path from the processing path earlier. Having Aspen write PDFs to a file server and then Kafka-notify AGS with the path is correct — but we initially had Aspen embed the base64-encoded PDF in the Kafka message. At 200 KB average PDF size, a 500K-customer event produced 100 GB of Kafka data in a single topic. Kafka is not a file store. We refactored this in sprint 3, but it cost us two weeks.
Event schema registry from day one. We used plain JSON Kafka messages initially. When EMC added two new fields to the corporate action event payload, three downstream consumers broke in prod. Had we used Confluent Schema Registry with Avro schemas, the compatibility check would have caught the breaking change in CI.
Apigee circuit breaker configuration needs tuning per downstream. The default Apigee circuit breaker settings (50% failure rate over 10 seconds) were too aggressive for ED Services, which has naturally higher latency for Swiss accounts. It was tripping the circuit and sending Swiss account requests to the fallback path when ED Services was healthy but slow. We tuned the circuit breaker to 70% failure rate over 30 seconds for ED Services specifically.

Key Learnings

Idempotency is not optional in financial systems. Every system in the chain — Aspen, AGS, InView client — must be idempotent. Budget for it upfront. It touches your DB schema, your Kafka config, and your HTTP retry strategy.
Record date ≠ processing time. Corporate action timing is deeply intertwined with settlement cycles. Build the settlement wait window into your design, not as a hotfix.
Tax logic is a first-class domain. The temptation to inline TaxC's rules into Aspen is real. Resist it. Tax rules change every budget season. A dedicated service with its own release cycle is what lets you update 194/195 TDS rules without redeploying Aspen.
Dead-letter queues are your audit trail. Every integration point (ED Services, InView, Print Vendor) should have a DLQ. Ops should have a dashboard showing DLQ depth. Silence is not success — it might be silent failure.
Kafka is not a database. Do not embed large payloads in messages. Kafka messages should carry references (PDF path, event ID), not the data itself. The file store is the single source of truth for the PDF; Kafka is just the notification.
The ISO 20022 migration is coming for everyone. The Indian securities market (NSDL, CDSL) is progressively aligning to ISO 20022 message formats (SEEV.031 for notifications, SEEV.033 for instructions). If you're building corporate action pipelines today, design your internal event schema to map cleanly to SEEV.031 fields — the migration will be far less painful.
Voluntary and mandatory events need different pipelines. Mandatory events (dividends, bonus, splits) need fast, high-throughput processing. Voluntary events (rights, buybacks, proxy votes) need SLA-tracked, deadline-aware pipelines with customer instruction collection. Running them through the same queue means mandatory events can delay voluntary ones during peak load — exactly when you can't afford delays on voluntary cut-off dates.

Industry Reference Points

Standard / System	Relevance
SWIFT ISO 15022 MT564	Legacy corporate action notification message format — still dominant in cross-border custody. Our EMC payload is modelled on MT564 field structure (field 22F event type, :98A: dates, :92A: rates).
SWIFT ISO 20022 SEEV.031	Modern XML-based replacement for MT564. Richer data model, mandatory for SWIFT's 2025 MT-to-MX migration deadline. Our internal event schema maps 1:1 to SEEV.031 elements to future-proof the transition.
DTCC CA WebUI	In US markets, DTCC (Depository Trust & Clearing Corporation) is the authoritative source for corporate action announcements. Their CA WebUI is analogous to our EMC — the golden source of event data for downstream systems. NSDL/CDSL serve this role in India.
Citi Global Custody CAST	Citibank's Corporate Action Straight-Through Processing initiative is the industry benchmark for STP rates (>95% automation). Their published approach: event receipt → entitlement → advice → instruction → settlement, each step automated with exception-only manual touchpoints. Our Aspen pipeline mirrors this STP chain.
SMPG Guidelines	Securities Market Practice Group publishes ISO 20022 usage guidelines for corporate actions — including the settled-position-at-record-date rule that informed our settlement wait window design.
SEBI LODR Regulations	SEBI's Listing Obligations and Disclosure Requirements mandate that Indian listed companies announce corporate actions at least 7 business days before the record date — this is the minimum window our pipeline must comfortably fit within.

📨 Building a corporate action or custody notification system? Whether it's greenfield or untangling a legacy batch system, we've shipped this in production across multiple custodian clients. Book a free architecture call →

📓 Architect's Notebook

Latest

Multi-Tenant CMS: Serving Many Brands from One Spring Boot Codebase

COBOL → Java Microservices Migration: Challenges & How We Solved Them

Event Fabric: Zero-Miss Corporate Action Notifications at Custodian Scale

Multi-Tenant CMS: Serving Many Brands from One Spring Boot Codebase

The goal

The core idea: resolve the tenant from the Host header

The request lifecycle

Low-level design — layers & key classes

Sequence: a request becomes a themed page

Data isolation: the invariant that keeps tenants apart

Theming without forking templates

One admin, many sites

Key design decisions

Host-based tenancy

Shared schema, website_id

Freeze the incumbent

Config over code

CSRF scoped to /admin

Media on a volume

Gotchas we actually hit

Takeaways

COBOL → Java Microservices Migration: Challenges & How We Solved Them

The migration architecture at a glance

Why this is hard (and why "just rewrite it" fails)

New to mainframes? A 60-second glossary

The challenges — and how we solve them

1. Hidden, undocumented business logic

2. Numeric precision — the silent killer

3. Avoiding the distributed monolith

4. Transactions & consistency (CICS → microservices)

5. Data migration (VSAM/DB2/flat → Oracle)

6. Batch & JCL — the part everyone forgets

7. Performance regression

Our AI-assisted pipeline — what works

The thing that actually de-risks the cutover: golden-master + parallel-run testing

Golden-master (characterization) testing

Parallel / shadow run

Takeaways

Event Fabric: Zero-Miss Corporate Action Notifications at Custodian Scale

Problem Statement

Architecture Decision Records (ADRs)

Non-Functional Requirements

Throughput

Latency

Exactly-Once

Availability

Data Sovereignty

Sanctions Compliance

Audit Trail

Disaster Recovery

Data & Security Zone Boundaries

🔴 Restricted Zone

🟡 Sanctions-Gated Zone

🔵 Processing Zone

🟢 Delivery Zone

🟣 Compliance Zone

System Interaction Diagram

What is a Corporate Action, and Why Does Getting it Wrong Cost Millions?

The System Landscape

Data Flow: A Dividend Hits the Wire

Step 1 — SMC → EBS: Daily Security Master Sync

Step 2 — EMC → Aspen: Corporate Action Event via Kafka

Step 3 — Aspen: Entitlement Calculation

Step 4 — Aspen → Apigee → Aggregator → TaxC + ED Services

Step 5 — Aspen: PDF Generation & File Server Storage

Step 6 — AGS: Fan-out to Delivery Channels

Problems We Hit in Production — and How We Solved Them

Problem 1 — EMC Duplicate Events (The "Double Advice" Incident)

Problem 2 — The Race Between Record Date and Settlement (T+2 Hell)

Problem 3 — TaxC Timeout for DTAA Edge Cases

Problem 4 — Swiss Account Encoding Failures (ED Services Edge Case)

Problem 5 — Kafka Consumer Lag During Dividend Season

Problem 6 — AGS Thundering Herd on Large Dividends

Problem 7 — Exactly-Once Delivery and the Advice Letter Already Sent

Problem 8 — Voluntary Corporate Action Cut-Off Miss

Architecture Decisions We'd Make Differently

Key Learnings

Industry Reference Points

Shared schema, `website_id`