1. Request Body Mismatch and TCP Port Exhaustion Link to heading

Problem:
When the client’s Content-Length header didn’t match the actual request body size, the server’s JSON parser failed. Worse, unclosed requests kept TCP connections open, exhausting available ports and causing service outages.

Solution:

  • Always Close Requests: Use defer to ensure http.Request.Body is closed, even during errors.
  • Set Timeouts: Configure http.Server timeouts (e.g., ReadTimeout, WriteTimeout) to prevent hung connections.
  • Adjust EC2 Limits: Increase file descriptor limits via ulimit -n and tune net.ipv4.tcp_fin_timeout to recycle ports faster.
    • net.ipv4.tcp_fin_timeout : This controls how long a connection stays in the “FIN-WAIT-2” state after it’s gracefully closed. Lowering this value (default: 60 seconds) can free up resources faster, particularly useful for servers handling lots of short connections.

Technical Insights:

  • TCP Port Exhaustion: Each TCP connection uses a unique (source IP, source port, destination IP, destination port) tuple. Ports are finite (0–65535), and connections in TIME_WAIT state (default 60s) consume ports until recycled.
  • Content-Length Mismatch: Go’s http.Server reads exactly Content-Length bytes. Extra bytes are left in the buffer, causing parsing errors. The key is to use io.ReadAll cautiously and instead we must validate early or stream with io.LimitedReader first.

Code Snippet:

func handler(w http.ResponseWriter, r *http.Request) {
    defer r.Body.Close() // Critical!
    body, err := io.ReadAll(io.LimitReader(r.Body, maxBodySize))
    // ...
}

Read more:
https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/ https://www.tecmint.com/increase-set-open-file-limits-in-linux/


2. Goroutine Overload with surge of requests Link to heading

Problem:
We know that each request spawned is a new goroutine. Now during traffic surges, excessive goroutines causes very high context switching (visible in top and number of active goroutines using go tool pprof), degrading performance.

Solution:
Connection Pool to limit concurrent requests/go routine:
https://gist.github.com/SwatiModi/2cb143f24f97aead42826ab0ca4ba299

Technical Insights:

  • Goroutine Scheduler: Go uses an M:N scheduler (M OS threads, N goroutines). High goroutine counts force frequent context switches (~1–2µs each), increasing latency.
  • Pooling vs. Worker Pools: A channel-based pool throttles concurrency but doesn’t reuse workers. For CPU-bound tasks, worker pools (fixed goroutines + task queues) reduce scheduling overhead.

Trade-off:

  • Pool size = GOMAXPROCS * 2 balances CPU utilization and memory. Monitor with runtime.NumGoroutine().

3. Custom JSON Parser for Memory Efficiency Link to heading

Problem:
Standard encoding/json used excessive memory due to reflection and allocations.

Solution:
Schema-Specific Parser:

    1. Avoid Unnecessary Allocations:
    • Reuse buffers and avoid creating intermediate strings or slices.
    • Use a single buffer for parsing and directly write results to the output map.
  1. Use String Interning:
    • Store frequently occurring strings (e.g., "true""false""null") as constants to avoid duplicate allocations.
  2. Streaming Parsing:
    • Parse JSON data in chunks instead of loading the entire JSON into memory. This is particularly useful for large JSON files.
  3. Reduce Map Overhead:
    • Use a pre-allocated map with a known capacity to avoid resizing during insertion.
  4. Avoid Unnecessary Copies:
    • Directly reference sub-slices of the input JSON data instead of copying them.

Example Optimization:

var bufferPool = sync.Pool{
    New: func() interface{} { return new(bytes.Buffer) },
}

func ParseCustomJSON(data []byte) (Event, error) {
    buf := bufferPool.Get().(*bytes.Buffer)
    defer bufferPool.Put(buf)
    buf.Reset()
    buf.Write(data)
    // Parse manually without reflection...
}

Technical Insights:

  • Reflection Overhead: encoding/json uses reflection to map fields, which is 10–100x slower than static code. Tools like easyjson generate unmarshaling code at compile time.
  • Streaming Parsers: For large payloads, json.Decoder decodes incrementally, reducing memory from O(payload size) to O(max nested object).

4. Graceful Shutdowns with In-Memory Queues Link to heading

Solution:

  • Signal Handling: Capture SIGTERM/SIGINT to start shutdown.
  • Drain In-Memory Queues: Stop accepting new requests, process remaining items, then exit.
  • Load Balancer Coordination: Use health checks (e.g., /health endpoint returning 503) to signal unavailability.

Code Snippet:

server := &http.Server{Addr: ":8080"}
go func() {
    <-shutdownSignal
    server.Shutdown(context.Background()) // Stops new connections
    drainQueue() // Process remaining items
}()

// Health check endpoint
http.HandleFunc("/health", func(w http.ResponseWriter, _ *http.Request) {
    if isShuttingDown {
        w.WriteHeader(503)
    }
})

Technical Insights:

  • Graceful Shutdown: http.Server.Shutdown() closes listeners first, then waits for active requests to finish.
  • Queue Draining: Use a sync.WaitGroup to track in-flight requests and block shutdown until done.

5. Context specific Health Checks Link to heading

We were building a system that relied on an in-memory queue. As the queue processing slowed down, the application’s memory usage began to spike rapidly. We noticed that once memory usage hit around 60%, the system was likely to soon run into an Out-of-Memory (OOM) error, risking data loss.

To mitigate this, we introduced a health check mechanism tailored to this scenario. This enabled us to proactively remove the machine from the fleet and allow it to shut down gracefully before memory usage reached critical levels, effectively avoiding OOM errors and safeguarding the data.

Implementation:

  • Monitor Memory: Use gopsutil to track UsedPercent.
  • Hysteresis: To prevent constant switching (or “flapping”) between states (e.g., healthy and unhealthy), a buffer or gap is introduced between the thresholds. In this case, the system is considered “good” when memory usage is below 60%, but it is only marked as “failing” when memory usage exceeds 80%. This gap ensures that the system doesn’t repeatedly toggle between states due to minor fluctuations around a single threshold, providing stability and reducing unnecessary actions.

Technical Insights:

  • Atomic Updates: Use atomic.Bool for thread-safe status checks.
  • Load Balancer Integration: AWS Target Groups use HTTP health checks to route traffic. A 503 triggers instance decommissioning.

6. CPU vs. Memory Trade-offs Link to heading

Optimizations:

  • GOGC=200: Reduces GC frequency (default=100), trading memory for CPU.
  • Buffer Pools: Reuse objects with sync.Pool to limit allocations.
  • Pointers for Large Structs: Avoid copying 1KB+ structs; pass pointers instead.

Example:

type LargeStruct struct { Data [1024]byte }

// Pass by pointer to avoid copy
func ProcessEvent(event *LargeStruct) {
    // ...
}

Technical Insights:

  • GC Impact: Lower GC frequency reduces CPU spikes but increases RSS.
  • Cache Locality: Pointers can cause cache misses; profile with go tool pprof -http=:8080 cpu.out.

7. Database Scaling and Connection Management Link to heading

In an application, we initially used a single database for the request path for auth, with our application code configured to allow 40 connections. However, when we scaled to over 100 instances, this caused a sudden surge in database connections, significantly increasing the load on the database. As a result, the database became a bottleneck during scaling, even though we had implemented caching at the application level.

This approach turned out to be an anti-pattern. Ideally, your application should communicate with an API, and the API should interact with the database. This design limits the number of direct connections to the database and centralizes caching at the API layer, ensuring consistency and reducing unnecessary load on the database.

Solution:

  • API Layer: Introduce a gRPC/HTTP service to pool DB connections.
  • Centralized Caching: Use Redis with read-through caching.

Technical Insights:

  • Connection Poolers: Tools like PgBouncer (for PostgreSQL) pool connections, reducing overhead.
  • Caching Strategies: Cache hot data at the API layer to offload the DB. Use TTLs and write-through policies for consistency.

8. Handling Downstream Write Failures Link to heading

Problem Statement: Downstream Write Failures

Solution:
To address downstream write failures, the following strategies can be implemented, drawing inspiration from TCP congestion control mechanisms to ensure robustness and efficiency:

  1. Temporary Write Halting with Write-Ahead Logging (WAL):

    • When downstream failures are detected, temporarily halt writes to the downstream system.
    • Buffer the incoming write operations in an on-disk Write-Ahead Log (WAL) to ensure data durability and consistency.
    • Once the downstream system is available, replay the writes from the WAL to maintain data integrity.
  2. Exponential Backoff for Retries:

    • Implement an exponential backoff strategy for retrying failed write operations.
    • Start with an initial delay (e.g., 100ms) and double the delay after each subsequent failure (e.g., 200ms, 400ms, 800ms, etc.).
    • Cap the maximum delay to a reasonable threshold (e.g., 5s) to avoid excessive latency.
    • This approach is similar to TCP’s congestion avoidance mechanism, where the sender reduces the rate of packet transmission in response to network congestion.
  3. Circuit Breaker Pattern:

    • Introduce a circuit breaker to prevent overwhelming the downstream system with repeated failed requests.
    • Monitor the failure rate of write operations. If the failure count exceeds a predefined threshold within a specific time window, trip the circuit breaker.
    • While the circuit breaker is active, reject all new write requests immediately without attempting to contact the downstream system.
    • After a cooldown period, transition the circuit breaker to a half-open state, allowing a limited number of requests to test the downstream system’s availability. If these requests succeed, close the circuit breaker and resume normal operations.
  4. Congestion Window Adaptation:

    • We implement a congestion window mechanism to dynamically adjust the rate of write operations based on downstream system responsiveness.
    • Start with a small congestion window (e.g., 1 write operation) and gradually increase it as successful writes are acknowledged, similar to TCP’s slow start and congestion avoidance algorithms.
    • Reduce the congestion window size in response to failures or timeouts to avoid overwhelming the downstream system.

By combining these techniques, the system can effectively handle downstream write failures while maintaining data integrity, minimizing retry overhead, and preventing cascading failures. This approach aligns with principles from TCP congestion control, ensuring a balance between reliability and performance.

Code:

var cb = gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name: "downstream-service",
    ReadyToTrip: func(counts gobreaker.Counts) bool {
        return counts.ConsecutiveFailures > 5
    },
})

func SendBatch(ctx context.Context, batch []Event) error {
    result, err := cb.Execute(func() (interface{}, error) {
        return client.Publish(ctx, batch)
    })
    // ...
}

Technical Insights:

  • Backpressure: Downstream saturation requires client-side throttling.
  • TCP Congestion Control: Inspired by additive-increase/multiplicative-decrease (AIMD), adjust request rates based on success/failure signals.

Key Takeaways Link to heading

  1. Close Resources Relentlessly: Sockets, files, and goroutines leak silently.
  2. Profile Before Optimizing: Use pprof to identify bottlenecks.
  3. Design for Failure: Assume downstreams will throttle; plan retries and fallbacks.
  4. Centralize State: Databases and caches should be shared, not per-instance.

By addressing these challenges with a mix of Go-specific optimizations and systems thinking, we transformed a fragile application into a scalable, resilient service. Each solution required balancing trade-offs—a reminder that scalability is as much about compromise as it is about code.