Agora Live Streaming Re-implementation
Rebuilt Pickles Asia's live auction streaming stack from scratch
AWS migration broke live streaming completely. Root cause: 8-year-old hidden service nobody knew about.
Pickles Asia migrated from IPServerOne (physical server in a data center) to AWS for the Malaysia region launch. The migration was meticulously planned: database migrated, APIs tested, infrastructure validated. But during cutover, live auction streaming went dark. All auctions came online, but the streaming component failed silently. Auctioneers couldn't broadcast. Bidders can't view the streams. For a revenue-critical auction platform, this was a P1 incident.
I traced the problem through the legacy codebase and discovered the Agora integration depended on an undocumented Node.js service running on port 8443 on IPServerOne. This service existed for 8 years but nobody on the current team knew about it. It was never documented, never maintained, never migrated. When we moved to AWS, this hidden service was left behind. Every auction tried to connect to a non-existent service. The entire streaming feature broke silently.
- Live streaming broke in production after AWS migration. All auctions affected, P1 severity
- Root cause: undocumented Node.js service running on port 8443 in IPServerOne, unknown to current team, never migrated to AWS
- Service had existed for 8 years with zero documentation, zero maintenance, zero mention in any runbook or architecture diagram
- Legacy Agora integration completely dependent on this hidden service; no fallback, no error handling
- Staging environment broken for 3+ years; nobody caught the hidden service issue because staging was never tested
- Before migration could complete, legacy implementation had to be fully understood to ensure nothing was lost
| SlimJet (Windows) | ✓ Production only |
| Chrome / Firefox / Safari | ✗ Not working |
| macOS / Linux | ✗ Not working |
| Staging environment | ✗ Broken 3yr+ |
Discovered hidden service, mapped 8-year legacy, migrated to modern agora-rtc-sdk-ng with zero downtime
- Agora integration depended on undocumented Node.js service on port 8443 in IPServerOne
- Service existed for 8 years, completely unknown to current team
- Never documented, never maintained, never migrated to AWS
- Hidden service was the single point of failure for entire streaming platform
- AWS migration left the service behind, all auctions streaming went dark
- Traced how hidden service initialized Agora channels
- Mapped how web app connected to hidden service
- Documented complete event flows and dependencies
- Analyzed why staging environment had been broken for 3+ years
- Understood every integration detail before migration
- Eliminated the hidden service dependency entirely
- Works on any modern browser (Chrome, Firefox, Safari, Edge)
- Added AgoraAudioStream component with proper event handling for stream start/stop
- Configured with updated SDK initialization, channel management, and stream subscription logic
- Made the code clean, testable, and maintainable. Updated package.json to include agora-rtc-sdk-ng, updated config.js with Agora appId and channel naming for staging/production parity.
- Aligned Agora app credentials between staging and production
- Aligned channel configuration and environment variables
- Added health checks for streaming service to CI/CD pipeline
- Staging now fully functional; QA can test before production
- Tested across Chrome, Firefox, Safari, Edge on Windows/macOS/Linux
- Validated on both auctioneer computers and bidder devices (mixed platforms)
- Deployed rolling migration strategy: new service alongside old one
- Monitored health checks, confirmed all auctions streaming successfully
- Decommissioned legacy service. Total downtime: zero
P1 incident recovered. Hidden service eliminated. All auctions streaming. Any browser.
Live auction streaming broke completely during AWS migration due to the undocumented hidden service on IPServerOne. After the solution was implemented, the P1 was resolved with zero downtime and zero data loss. All auctions resumed streaming immediately after deployment.
Live auction streaming now works on any modern browser - Chrome, Firefox, Safari, Edge - on both Windows, macOS, and Linux. The SlimJet dependency has been completely eliminated. Staff can use their preferred browser without restrictions.
The staging environment for live streaming was non-functional for over 3 years. After re-implementation and alignment staging is now a fully functional testing ground. QA can validate streaming scenarios before production for the first time.
QA can now include live streaming in their full test plan and validate all streaming-related changes in staging before production. This eliminates the blind-deploy risk on a revenue-critical feature and ensures production streaming quality.
Architectural choices, tradeoffs, and what nearly broke production
How do you discover undocumented systems hidden for 8 years?
When the P1 incident hit, all live auctions streaming went dark after AWS migration.
I started with the simplest question: what changed? The database migrated fine. APIs worked. But streaming broke everywhere. I read 8 years of code history, traced every Agora function call, and realized they all flowed to a service that didn't exist anymore. Then I found the hidden service running on port 8443 on the old server.
The lesson: undocumented systems can kill your entire migration if you don't understand them first.
Why rebuild from scratch instead of patch the legacy hidden service?
You could, but that perpetuates the problem: an undocumented, unmaintained service with legacy SDK APIs. Rebuilding meant:
(1) Eliminating the hidden dependency entirely
(2) Using current Agora APIs instead of 8-year-old ones
(3) Making the code testable and maintainable
(4) enabling staging to work for the first time in 3 years.
One week rebuilding beats years of maintaining legacy infrastructure.
How do you migrate a mission-critical system with zero production downtime?
Staged migration:
(1) Deployed new Agora integration alongside old one
(2) Monitored health checks for the new service
(3) Validated streaming worked with real test auctions
(4) Switched traffic gradually: new auctions used new service, old auctions stayed on old service
(5) Once all in-flight auctions completed on old service, decommissioned it. Total downtime: zero. Total failed auctions: zero.
Why was staging broken for 3 years without anyone noticing?
Because the hidden service only existed on the production server. It was never replicated to staging. Anyone trying to test streaming in staging would hit a timeout connecting to a service that didn't exist.
After rebuilding with agora-rtc-sdk-ng, I aligned Agora credentials and configuration between staging and production. Now staging works, and staging breakage triggers loud failures before any code ships.
How do you ensure the new implementation doesn't lose functionality from the old one?
By fully understanding the legacy system first. I mapped every function, every event, every configuration parameter. Then I rebuilt each piece in the new system with explicit tests. Validated across all browsers (Chrome, Firefox, Safari, Edge) and all platforms (Windows, macOS, Linux).
No functionality was lost. Instead, gained: cross-browser support, staging testability, clean maintainable code.
Cost · Risk · Tradeoffs
Estimated values based on production metrics and monitoring data
Transformative Results: Before vs After
- SlimJet browser onlySeverely restricted
- Staging broken 3+ yearsZero testability
- Legacy entangled codeHard to debug
- Windows-only platformPlatform locked
- Any modern browserUniversal support
- Staging fully workingFull testability
- Clean reimplementationEasy to maintain
- Cross-platform supportNo restrictions
What was used
Production broke during migration and you found an undocumented system nobody knew about?
I specialise in discovering hidden infrastructure, understanding legacy systems before migrating them, and rebuilding mission-critical features with zero downtime. Let's talk.