AWS Production Infrastructure
Ad-Hoc Cloud → Production-Grade Architecture
6 years of ad-hoc AWS. No architecture. Real security gaps.
Pickles Asia's AWS account had been provisioned in 2019 and expanded incrementally without a defined infrastructure architecture. By 2025, workloads were running in the default VPC with public and private resources mixed without a consistent subnet strategy. EC2 instances and ECS services sat behind security groups that had been widened over time to fix connectivity issues and never tightened. RDS was deployed as a single-AZ instance - a live auction platform with no database failover capability.
The Malaysia region launch created an opportunity and an obligation: stand up the new region properly, not as a clone of the existing ad-hoc setup. This required designing a proper infrastructure architecture that the team could deploy repeatably, understand fully, and operate with confidence.
- Workloads in the default VPC - no documented network topology, public and private traffic mixed
- Security groups with overly broad inbound rules - some with 0.0.0.0/0 on non-web ports
- RDS single-AZ - any instance failure meant manual restore from snapshot with 30–60 minute downtime
- Secrets hardcoded as Lambda/ECS environment variables in the AWS console - no rotation, no audit trail
- No WAF - web-facing ALBs exposed directly to the internet without OWASP layer protection
- No tagging policy - impossible to attribute costs by service, environment, or team
- Infrastructure deployed entirely via console - no IaC, no reproducibility, no CI/CD for infra changes
- No formal disaster recovery plan - no defined RPO/RTO targets, no runbooks for common failure scenarios
| Component | State |
| VPC topology | Default VPC |
| RDS | Single-AZ |
| Secrets management | Env vars in console |
| WAF | None |
| IaC | None |
| Disaster recovery | No plan |
| Cost attribution | No tags |
7 design pillars. Built to be operated by a team of 5.
Production-grade infrastructure. Reproducible. Operable. Secure.
Every critical workload distributed across 3 AZs. RDS Multi-AZ automatic failover confirmed <60 seconds RTO. ECS circuit-breaker deployment rollback auto-reverts any failed container deployment. No single AZ loss can take the platform down.
Every AWS resource defined in CDK TypeScript. Staging and production environments are identical reproductions - no configuration drift. New environments can be deployed from scratch with a single cdk deploy. Console-based infrastructure changes are blocked via SCPs.
All ECS tasks, Lambda functions, and database instances run in private subnets with no direct internet exposure. ALBs in public subnets are the only internet-facing layer. Security groups scoped to specific CIDR ranges. WAF blocks malicious requests at the CloudFront layer before they reach the application.
All services classified into 3 DR tiers with defined RPO/RTO targets. Runbooks written for AZ failure, region degradation, RDS failover, DDoS event, and accidental data deletion. The team knows exactly what to do and in what order when any infrastructure failure occurs.
Security designed in, not bolted on
All credentials (DB passwords, API keys, third-party tokens) stored in AWS Secrets Manager with automatic rotation via Lambda rotation functions for RDS. Environment variables in ECS task definitions reference Secrets Manager ARNs - no plaintext secrets in CDK code, CI/CD pipelines, or CloudWatch logs. Rotation events logged to CloudTrail. Zero console-set environment variable credentials in the new infrastructure.
Every ECS task and Lambda function has a dedicated IAM execution role scoped to exactly the AWS resources it needs - specific S3 bucket ARNs, specific DynamoDB tables, specific Secrets Manager secret ARNs. No wildcard ARNs. IAM roles reviewed and approved as part of the CDK PR process - any IAM change triggers a required security review label on the PR. Permission Boundaries applied to all application roles to prevent privilege escalation.
AWS Managed Rules covering OWASP Top 10 (SQLi, XSS, common exploits), AWS Bot Control (blocking scrapers and scanners), and Amazon IP Reputation list. Custom rate limiting rule: 500 requests per 5-minute window per IP before CAPTCHA challenge, 2,000 requests before block. Geo-restriction not applied - Pickles Asia operates internationally. WAF metrics exported to CloudWatch with per-rule dashboards. All blocked requests logged to S3 for post-incident analysis.
AWS Service Control Policies applied at the AWS Organization level to prevent manual console-based creation or modification of core infrastructure resources (VPCs, ECS services, RDS instances, Lambda functions). All changes must flow through the CDK CI/CD pipeline with PR review. This eliminates the pattern that created the original ad-hoc environment - console fixes that become permanent undocumented infrastructure.
Architectural choices, tradeoffs, and what nearly broke production
Multi-AZ active, not standby failover
Multi-AZ within a single region provides <60s automatic failover with zero configuration. It's built into RDS, ALB, and ECS. Cross-region failover adds operational complexity (DNS propagation, data replication lag). For an auction platform, <60s RDS recovery beats minutes-long cross-region cutover.
CDK over CloudFormation templates
CDK is a programming model on top of CloudFormation - it eliminates boilerplate and enables programmatic composition. You write TypeScript instead of maintaining 3000-line YAML. The CDK constructs encode AWS best practices (e.g., RDS defaults to encrypted, multi-AZ). Easier to version control and test.
What nearly failed: ElastiCache partition strategy
Initially, we designed Redis as a single node. First load test showed it bottlenecked at ~20k ops/sec. We scaled to a cluster with 3 shards, but the hashing strategy for session keys wasn't optimal. A deploy that redistributed hash keys caused connection pool misses during peak bidding. Fixed by pinning Redis config and testing under realistic concurrency.
Lesson: Caching architectures hide performance issues until production load. Test at 3x expected peak capacity during design validation.
IAM principle of least privilege - strict
Built IAM roles with specific action + resource scopes. ECS tasks can only PutObject to their specific S3 bucket prefix, not the entire bucket. This prevented accidental data leaks and lateral movement if a container was compromised. Dev velocity improved because developers understood exactly what permissions they had.
Cost · Risk · Tradeoffs
Estimated values based on production metrics and monitoring data
Transformative Results: Before vs After
- Workloads in default VPCSecurity gaps
- No IaC - manual setupError-prone
- Single-AZ RDSNo failover
- Overly permissive IAMSecurity risk
- Multi-AZ VPC topologyHigh availability
- 100% CDK IaC coverageReproducible
- RDS Multi-AZ + read replicas<60s failover
- Least-privilege IAM + WAFSecure by design
What was used
Need to design production-grade cloud infrastructure from scratch?
I design AWS architectures that are secure by default, fully reproducible via IaC, and operable by real teams - not just impressive on paper.