Nguyen Hoang Phuc

Context

An enterprise tier for school chains, where each school is a tenant. I designed the infrastructure from scratch for the launch.

What I did

• Two tenancy models: a shared database with PostgreSQL Row-Level Security keyed on the tenant ID from the user's JWT, and dedicated database and servers for schools whose contracts require their own compliance and data residency.
• Identity on Amazon Cognito with Google and Microsoft SSO and tenant-aware auth.
• Perimeter of CloudFront, WAF and ALB in front of EKS, with WAF rules tuned from logs and metrics to remove false positives.
• Threat model at design time, then SAST, DAST, Aikido, Trivy and Gitleaks in CI/CD.
• Declined to host an SFTP server for a school's financial files; proposed pulling them from the school's own Azure Blob instead, avoiding a new internet-facing service.

Outcome

Launched for the first 3 school chains (~10,000 students) on schedule. Lesson I now apply everywhere: security is cheapest at design time, so it starts in the first design review.

Context

About 700 TB of photos and videos on S3, with high storage and CloudFront egress costs and weak lifecycle management.

What I did

• Studied the access pattern first: media is hot for about 30 days, then rarely viewed.
• Switched video transcoding from GPU to tuned CPU encoding with FFmpeg: slower, but scaled out, and about 80% smaller files at good quality, checked across many phone models.
• Re-encoded old videos still in Standard and Standard-IA; left colder classes alone where re-encoding cost more than it saved.
• Put Cloudflare Cache Reserve (R2-backed, 30-day TTL) in front of CloudFront so most views never pay CloudFront egress.

Outcome

Total media spend down 60%, measured in AWS Cost Explorer before vs after.

• Hired a mid-level engineer to share the load so I could focus on architecture, security and compliance.
• Two-way review: PR reviews for all Terraform, staging before production, runbooks and postmortems. My changes get reviewed too.
• Grew my teammate through ownership: they had never provisioned an AWS production environment; I gave them that work, and they now run production confidently.
• When ad hoc requests overloaded the team, I set up a ticket form and filtered the queue, which cut interruptions.
• Joined the ISO 27001:2022 working group for certification, then led the yearly surveillance audit.

• AI helps with debugging, IaC, scripts, architecture and learning, under strict rules: its own read-only access key with a few scoped writes, no access to secrets, and no change without my explicit approval.
• Designing the CloudFront, WAF and ALB setup with metrics, alerts and logging took about a day instead of days to a week, by my estimate.
• It is not always right: on an S3 cost spike it pointed at the largest buckets, but the real cause was continuous writes to a small one. I checked the data myself and enabled S3 request metrics. Lesson: AI only knows what it can see.

Internal tools sat behind a legacy VPN, which trusts anyone on the network. I replaced it with Cloudflare Zero Trust for internal access management, so access is checked per user and per application through the company's identity provider instead of network membership. Internal apps are published through Cloudflare Tunnel, so they need no public inbound ports.

A consumer bug (an infinite loop with an N+1 pattern) flooded our feature-flag server, which slowed every service that depended on it. With AI triaging a storm of alerts, I found the root cause, scaled the flag server as a hot fix in about 10 minutes, traced the traffic to the exact consumer and release, and rolled back through the ArgoCD repo. Every change was approved by me first.

The postmortem's systemic fix was a cache proxy per service so no single consumer can overload the flag server again. I led that rollout with the teams.

Launch day was fine. Days later, a large school chain's staff all used the same features every morning, and core pages slowed down. Scaling database CPU and memory did nothing, which pointed at the real cause: complex queries locking each other. My team traced and fixed queries and indexes until the database was stable, while the product team kept schools informed.

Afterwards I built k6 load tests for realistic peaks. Lesson: launch-day traffic is not real traffic.

RabbitMQ hit its ceiling at ~500K orders/day ahead of a sale forecast at 2M/day. I proposed Kafka, proved it on a non-critical internal flow, then helped migrate critical flows one at a time with idempotent consumers. Kafka ran self-hosted on VMs via Ansible (3 brokers, replication factor 3). We fixed rebalance stalls with static membership and hot partitions by hashing the order ID.

Outcome: 2M orders/day at the next real sale peak with no degradation. I won the other team leads over with incident data and a low-risk proof of concept.

Operations needed to view cameras at 1,000+ sorting stations for theft, damage and disputes. A third-party product was too expensive, so I designed and built our own: on-demand streaming where opening a camera assigns a Golang worker that repackages the NVR stream into HLS with FFmpeg, served to a ReactJS dashboard on Kubernetes in a Vietnam data centre.

Key choices: build over buy, on-demand over record-everything, pass-through over transcoding, and in-country hosting for latency. Delivered as one dashboard for every station, at about 1 Gbps peak.

Three sites, each for a reason: mission-critical order-to-PLC services run locally in each warehouse, user-facing services run on a local cloud in Vietnam, and cloud-native services run on GCP. Warehouses reach GCP over two Cloud Interconnect lines with a leased line as backup, failing over within ~30 seconds. I owned the Cloudflare load-balancing layer in front of the public APIs, with the local cloud as primary origin and GCP as secondary.

When both Interconnects degraded during a sale-night incident, warehouses kept sorting. I ruled out our stack, found the problem on the carrier's routing, and escalated with evidence until it was fixed in about an hour.

10+ GCP projects on Shared VPC, 10+ GKE clusters plus on-prem clusters through Anthos, and teams running up to ~200 services. I owned the GCP side of the platform: GitLab CI builds, tests and scans, ArgoCD syncs from Git to every cluster, and teams deploy and retire services themselves. Observability: Prometheus agents writing to VictoriaMetrics, Grafana, and ELK with hot/warm/cold tiers.

A lesson from production: 50 replicas packed on 3 spot nodes have the availability of 3 nodes. I fixed it with anti-affinity, topology spread and an on-demand base.

Moved off Atlas to a standard self-hosted design (~20 clusters on VMs, Ansible, GCE snapshots with journaling). When the largest ~3 TB cluster hit its vertical ceiling, we converted it in place to a sharded cluster keyed on tenant + order ID, matching the main query pattern. Honest hindsight: today I would default to a managed service unless there is a special need.