AWS Lambda Regional Migration
Singapore to Malaysia. Runtime upgrades. Implementing GitOps discipline.
71 functions with old runtimes, no tests, and no documentation.
Pickles Asia had built a large serverless system over 6 years. 71 Lambda functions, mix of Node.js and Python, all running in Singapore. The problem was clear when we looked closely: almost every function was on old, unsupported runtimes. Node 8.10, 10, 12, 14, 16 - all past their end of life. Even the Python functions were on 3.7 and 3.8, both deprecated.
The company's customers were in Malaysia, and AWS had just opened a region there. Moving Lambda functions there made perfect sense for latency and compliance. But it wasn't going to be a simple copy-paste operation. Most of the functions were deployed through the AWS console or Serverless Framework. No GitHub. No CI/CD pipelines. No test coverage. Some hadn't been invoked in over 450 days and nobody knew who owned them. We needed to understand each function, upgrade it safely, write tests, and document everything.
- All 71 functions on end-of-life runtimes: Node 8 through 16, Python 3.7 and 3.8, past AWS support windows
- Zero test coverage across the entire fleet - not a single unit test for any function
- Every deployment done through the AWS console by hand - no version control, no audit trail
- ~10 functions with no invocations in 450+ days and no documented owner
- No playbook for rolling back a bad deployment. Just panic and manual fixes
- No documentation on how to safely operate any of these functions
| Runtime | Count | Status |
|---|---|---|
| nodejs8.10 | 2 | EOL 2019 |
| nodejs10.x | 15 | EOL 2021 |
| nodejs12.x | 12 | EOL 2022 |
| nodejs14.x | 13 | EOL 2023 |
| nodejs16.x | 6 | EOL 2024 |
| python3.7 | 4 | EOL 2023 |
| python3.8 | 4 | EOL 2024 |
| nodejs18.x / 22.x | 13 | Supported |
| python3.12 | 2 | Supported |
Audit, plan, validate, deploy, monitor.
- Documented triggers, permissions, runtime, dependencies
- Sorted into 3 groups: 35 critical (Phase 1), 10 dormant, rest for cleanup
- Established dependency map & ownership records
- Identified all old runtime EOL risks
- Created one repo per function
- Extracted functions from console-deployed ZIPs
- Reconstructed source code & dependencies
- Ran each function in Docker with original runtime
- Captured real production events
- Replayed events to establish baseline output
- Identified edge cases & dependency requirements
- Updated all runtimes to Node.js 22 & Python 3.12
- Updated required dependencies to latest versions
- Rewrote legacy Node 8 functions completely
- Fixed security vulnerabilities across fleet
- Created tests for all production functions
- Used Jest for Node.js, Pytest for Python
- Replayed production events on new runtime
- Confirmed identical output between old & new
- Deployed to Malaysia staging environment
- Ran Malaysia & Singapore in parallel 24 hours
- Compared logs, latency, errors, & output data
- Verified behavior parity before cutover
- Moved traffic in stages: 10% → 50% → 100%
- CloudWatch alarms monitored errors & latency
- Watched for data anomalies or performance spikes
- README for each function with ownership & triggers
- Documented permissions & deployment steps
- Created playbooks for deploy & rollback
- Enabled future maintainers to operate safely
Every function had to pass 4 checks before we turned off Singapore
I ran tests locally on both the old runtime and the new one. I used AWS SAM with Docker containers so the tests ran in the exact same environment that Lambda uses. This caught any breaking changes early before anything went to the cloud.
I captured actual events from production and ran them through both the old and new versions of each function. If the output wasn't identical, we stopped and fixed it. This found subtle bugs that normal tests would have missed.
After deploying to Malaysia, both the Singapore and Malaysia versions ran at the same time for a full day. I compared logs, response times, errors, and output between them in real time. Singapore stayed live as backup the whole time. Only after 24 hours of matching results did we proceed to the next step.
Traffic moved to Malaysia in stages: 10 percent first, then 50 percent, then all of it. At each stage, alerts watched for errors, slow responses, or bad data. If anything triggered an alarm, the system automatically switched back to Singapore without waiting for anyone to manually fix it.
4 problems that testing found early
The validation layers caught real bugs that would have broken production. Each one was found, fixed, and verified before any customer traffic moved.
A Finance Lambda returned empty fields on the new runtime. The old code used AWS SDK v2 callbacks that silently swallowed errors on Node 8. Node 22 handled it differently, so errors disappeared without throwing. Fixed by rewriting to SDK v3 with proper async/await.
An SQS processor failed 6 hours into the shadow run, sending 50 messages per hour to the dead-letter queue. The code parsed JSON twice, which worked on Node 12 but broke on Node 22. Fixed the deserialization logic and verified with 24-hour clean run.
Found 3 functions with wildcard permissions on SQS and S3. Set up years ago, never reviewed. Halted their migration, locked down permissions to specific resources, got security sign-off before proceeding.
Response times jumped from 340ms to 870ms during canary at 10%. Auto-rollback caught it immediately. Larger deployment package increased cold starts. Enabled Provisioned Concurrency on the 3 busiest functions. Resumed canary and finished at 180ms (47% faster than Singapore).
3 months of careful work instead of a quick copy-paste
Speed vs safety. Every tradeoff was worth it.
A naive migration would have taken 2 weeks. This took 3 months because we audited everything, wrote tests, ran shadow deployments, and got things right. Worth every day.
No emergency pages. No 3 AM calls. No rollbacks in production. No customers noticing anything wrong. That's what the extra time bought us.
CloudWatch alarms, traffic shifting, parallel runs, monitoring. More moving parts. But this complexity kept us safe at production scale.
Next time someone needs to migrate Lambda functions, they're not starting from zero. They have playbooks, templates, and a proven framework. The investment compounds.
35 functions moved. Modern runtimes. Everything in version control. No incidents.
All 35 production functions moved from Singapore to Malaysia. All the triggers, permissions, and settings verified and working.
Every function now runs current, supported code. Old functions on Node 8 got rewritten where needed. All dependencies updated. Security vulnerabilities fixed.
Zero incidents. Zero downtime. The testing, shadow runs, and automatic rollback safeguards meant every deploy was safe and reversible.
Every function is now in GitHub with tests, a README explaining what it does and how to deploy it, and a playbook for deployment and rollback. No more mystery black boxes in the console.
Architectural choices, tradeoffs, and what nearly broke production
Why test locally first instead of just comparing both versions in the cloud?
If the old and new runtimes produce different outputs, the cloud is the wrong place to find out. Local testing lets us catch and fix functional issues before any customers are affected. The shadow run then verifies deployment and performance, not core functionality. Testing locally saves days of production incident response.
Why 24 hours for the parallel test, not just a few hours?
Lambda functions don't run on a 2-hour cycle. Some only run at 2 AM. Some only run on weekends. A short test window might miss whole categories of traffic. 24 hours covers a full day's worth of real patterns, peak times, off-peak times, and time-dependent edge cases.
Why did we catch issues in testing that our normal tests missed?
Unit tests test happy paths and specific scenarios we think of. Event replay uses real production events, real data shapes, and real concurrency patterns. When a function handles 10,000 different message types, unit tests can't cover all of them. Real events do. The SDK error swallowing and double JSON parsing never showed up in our unit tests but failed immediately with production data.
Tests mock everything. Real deployments deal with actual AWS services, actual latency, actual error behaviors.
Why automatically roll back on a latency spike instead of investigating manually?
When customers see response times jump from normal to 800 milliseconds on an API endpoint, they stop using it. Timeouts start. Real impact happens. Auto-rollback within seconds prevents customer damage. Investigation and fixes can happen post-incident when production is stable. Your customers' experience comes first.
Transformative Results: Before vs After
- Old runtimes everywhere (Node 8 through 16, Python 3.7)Security risk
- Deployments through AWS console by handMistakes happen
- No tests at allBugs go unnoticed
- Only in SingaporeSlow for Malaysia users
- No safety checks before deployingThings break in prod
- Modern Node.js 22 and Python 3.12Secure and fast
- Deployments through GitHub with automationConsistent and safe
- Full test coverage for all functionsConfident deploys
- Running in both Singapore and MalaysiaBetter for users
- 4 layers of validation before going liveZero problems in prod
What was used
Migrating Lambda functions or serverless infrastructure?
I've executed this at production scale - 35 functions, zero downtime, zero incidents. I can design the migration strategy, build the validation and testing framework, handle the engineering work, and document everything so your team owns it afterward.