Bidding Engine Redesign
Redesign the entire RTC architecture, achieving 1M+ daily transactions.
A single-server bottleneck with no horizontal scaling path
The bidding engine was a single Node.js process handling all WebSocket connections for live vehicle auctions. Running Socket.io v2 with no Redis adapter, every server instance was completely isolated - no shared state, no cross-container event propagation. During peak auction events when hundreds of concurrent bidders connected, the single server had nowhere to go. You couldn't add a second container without splitting the connection pool, meaning bidders on one instance couldn't receive events published by the auction engine on another. The system had been in production for 8 years. Incremental patches had held it together, but the architectural ceiling was real.
- Socket.io v2 with no Redis adapter - each container instance was a completely isolated silo
- Single-server architecture with no horizontal scaling path under peak auction load
- Peak bidding events causing connection drops and latency spikes for active bidders
- Manual deployments requiring brief outages - no zero-downtime deployment strategy
- No pub-sub mechanism between the bidding platform and RTC server
| Scaling strategy | Vertical only |
| Downtime per deploy | ~5 minutes |
| Scalability ceiling | Hard limit |
A distributed, container-native RTC architecture on ECS Fargate
- Upgraded socket.io and socket.io-client from v2.x to v4.x
- Official Redis adapter now available (v2 had workarounds)
- Improved connection reliability and reconnection algorithm
- More stable WebSocket handshake under poor network conditions
- Integrated @socket.io/redis-adapter
- Every ECS container instance shares Redis pub-sub
- Event broadcast → all clients across entire fleet
- No silo, no state drift, no missed auction events
- Containerized RTC server
- Deployed to ECS with auto-scaling on ALB connection count & CPU
- Multiple containers run in parallel
- Coordinated staging validation for realistic concurrency testing
- Added structured JSON logging across all socket events
- Logged: connect, disconnect, broadcast, error, auction state
- Before: 'users can't bid' → hours of grepping
- After: root cause identification in under 10 minutes
- AWS ECR with automated builds via GitHub Actions
- ECS rolling deployments with graceful connection draining
- New tasks drain connections before replacement (zero downtime)
- Docker layer caching + parallelized tests = 50% faster pipeline
From fragile to enterprise-grade
Peak daily transaction volume handled reliably after the redesign, up from unpredictable performance under the old architecture.
The new horizontally scaled deployment achieves near-perfect uptime, meeting all architectural and SLA requirements.
System reliability and scalability improved by 40% - measured across connection stability, error rates, and deployment consistency.
CI/CD pipeline optimisation via Docker layer caching and automated test parallelisation cut deployment time in half.
Architectural choices, tradeoffs, and what nearly broke production
Socket.io v2 → v4, not Kafka
Socket.io + Redis adapter is purpose-built for real-time WebSocket state sync with subsecond latency. Kafka is designed for durable event streaming and offline replay; overkill here. Redis pub-sub + adapter keeps operational complexity low and leverages an existing cache layer.
ECS Fargate, not Kubernetes
ECS Fargate removes container node management entirely. Since Pickles Asia already uses AWS, Fargate eliminated operational overhead compared to running a K8s cluster. For this scale, Fargate's auto-scaling policies and simpler networking were the faster path to production.
What nearly failed: Connection state explosion
During initial staging load testing, Redis connection pooling was misconfigured. Under peak load, the pub-sub layer created a new connection per event, exhausting Redis connection limits within minutes.
Root cause: connection pool size mismatch across Socket.io versions.
Lesson: Test under realistic concurrency patterns before production. Connection limits are silent killers in distributed systems.
Logging was the hidden blocker
Structured JSON logging across socket lifecycle events. Without it, P1 incidents were impossible to diagnose; just 'users can't bid.' Added comprehensive logging for connect, disconnect, broadcast, and error events. This turned hour-long incident response times into 10 minutes.
Cost · Risk · Tradeoffs
Estimated values based on production metrics and monitoring data
Transformative Results: Before vs After
- Single server architectureLow scalability
- Polling-based communicationHigh latency
- Manual deploymentsSlow releases
- Limited fault toleranceFrequent downtime
- Monolithic codebaseHard to maintain
- Distributed, container-native architectureHighly scalable
- Pub-sub with RedisReal-time updates
- CI/CD with automated deploymentsFast & reliable
- Multi-AZ with auto-scalingHigh availability
- Modular microservicesEasier maintenance
What was used
Need a scalable real-time system?
I've done this on a live production platform handling 1M+ daily transactions. Let's talk about your architecture and what it needs to handle peak traffic reliably.