Sidecar and ambassador patterns

Chapter 14’s anti-corruption layer lives inside order-service’s own code — a deliberate choice, since translating a legacy domain model is business-relevant logic. This chapter looks at a different category of concern: infrastructure logic that has been creeping into every service’s codebase, and pulls it out into companion containers instead.

1. Problem the Pattern Solves

A security review requires every service-to-service call in Northwind’s cluster to use mutual TLS (mTLS) — both sides present and verify a certificate, not just the server. The immediate implementation path is obvious and wrong: add a TLS certificate library, a keystore, and connection-configuration code to order-service, inventory-service, payment-service, and every other service, each team implementing (and inevitably drifting from) the same logic independently — exactly the cross-cutting-duplication problem Chapter 4’s API gateway solved for client-facing traffic, but this time for traffic between services, which never passes through the gateway at all.

Separately, order-service’s LegacyCustomerBridge (Chapters 13–14) needs connection pooling, retry logic, and circuit-breaking specifically tuned for the legacy monolith’s slower, less reliable API — configuration that has nothing to do with order-service’s own business logic, but currently lives inside its codebase anyway, coupled to a Kotlin dependency (Resilience4j, covered fully next chapter) that every team calling any external system now has to configure correctly, independently.

Forces in tension:

• Consistency vs. per-service duplication. Cross-cutting infrastructure logic (TLS, retries to a specific external system, connection pooling) implemented identically inside every service’s own code inevitably drifts as teams patch their copies independently and at different times.
• Language and library independence vs. code reuse. A shared library solves duplication only if every service is on the same language and can adopt the same library version simultaneously — a real constraint in any platform with more than one language or with services on different upgrade cadences.
• Deployment complexity vs. separation of concerns. Moving infrastructure logic into a companion container removes it from the application’s code entirely, but means every pod now runs (at least) two containers, with their own lifecycle, resource allocation, and failure modes to manage.
• Debuggability. Infrastructure logic running in a separate process, communicating over localhost, is a new layer to reason about when something goes wrong — “is the bug in my code or in the sidecar” becomes a real debugging question that didn’t exist when everything ran in one process.

2. Core Idea

Sidecar: a companion container deployed alongside a service’s main container, in the same pod, sharing its network namespace (so they can talk over localhost) and lifecycle, handling a cross-cutting concern the main application doesn’t need to implement itself — TLS termination/origination, log shipping, or (as in Northwind’s case) enforcing mTLS on outbound and inbound traffic.

Ambassador: a specific kind of sidecar that acts as an outbound proxy for calls to an external system, handling connection-level concerns (retries, connection pooling, protocol translation) on behalf of the main application, which simply calls localhost and lets the ambassador manage the actual, more complex connection to the real destination.

order-service’s own code makes a plain RestClient call to localhost:15001 (the sidecar) for service-to-service traffic, or localhost:15002 (the ambassador) for the legacy monolith — it never handles a TLS certificate or a legacy-specific retry policy itself. Both concerns move to companion processes, configured and upgraded independently of order-service’s own release cycle.

Commonly confused with:

• A service mesh (a later chapter). A service mesh is, in large part, sidecars deployed systematically across every service, managed by a central control plane that configures them all consistently — this chapter’s hand-configured sidecar is the building block; the mesh chapter is what happens when you need dozens of these, consistently managed, with less per-service manual configuration. Introducing one or two sidecars by hand, as here, is a reasonable, smaller step before committing to full mesh infrastructure.
• The API gateway (Chapter 4). The gateway handles client-to-cluster traffic at the edge, for every service behind it, as one shared component; a sidecar handles concerns for one specific pod’s traffic, deployed once per pod, not shared across services. They solve traffic-shaping problems at different points in the request path and typically coexist rather than substitute for each other.
• An anti-corruption layer (Chapter 14). The ACL translates domain models — customer status codes, address formats — and lives in application code because that translation is business-relevant. An ambassador handles connection-level concerns — retries, TLS, pooling — that have nothing to do with domain meaning. Northwind’s legacy integration actually uses both together: the ambassador manages the connection to the legacy monolith; the ACL, still inside order-service’s own code, translates what comes back.

3. When to Use It

Strong indicators for a sidecar:

• A cross-cutting concern (TLS, logging, metrics collection) needs to be applied consistently across services that may be written in different languages, or on different internal library versions, where a shared code library can’t be adopted uniformly.
• A security or compliance requirement (mTLS, as here) benefits from being enforced at the infrastructure level, verifiable independently of each team’s application code.

Strong indicators for an ambassador specifically:

• Calling a particular external system (here, the legacy monolith) needs connection-level tuning (retries, pooling, possibly a protocol translation) distinct enough from the platform’s general service-to-service traffic that it deserves its own, dedicated companion process rather than generic library configuration inside the application.

Concrete use cases:

• Regulated industries (finance, healthcare): enforcing mTLS or specific compliance logging via sidecar, verifiable independently of application code, is often easier to audit than trusting every team’s in-application implementation.
• Polyglot platforms: a platform with services in Kotlin, Go, and Python benefits enormously from sidecar-based cross-cutting concerns, since a shared Kotlin library simply doesn’t help the Go or Python services at all.
• Legacy system integration, as here: an ambassador isolates the specific connection-tuning quirks of an older, less reliable system from the calling service’s own code — useful specifically during a Strangler Fig migration window (Chapter 13) or any long-term integration with an external system that has its own operational peculiarities.
• Gradual infrastructure rollout: introducing mTLS platform-wide via sidecars lets the platform team roll it out pod by pod, verifying each one, without requiring every application team to ship a coordinated code change simultaneously.

Prerequisites:

• A container orchestration platform (Kubernetes, as used throughout this series) capable of running multi-container pods with shared networking — the pattern depends on that shared localhost namespace.
• Clarity on exactly which concerns move to the sidecar/ambassador versus which stay in application code — Section 8’s most common mistake is getting this boundary wrong in either direction.

4. When Not to Use It

• A single-language platform where a shared library genuinely works. If every Northwind service were Kotlin/Spring Boot on a synchronized release cadence, a shared Gradle library providing TLS configuration might be simpler to reason about than an extra container per pod — evaluate the actual constraint (polyglot? independent release cycles? compliance auditability?) rather than reaching for a sidecar reflexively.
• Very few services, low operational maturity for multi-container pods. A small team running two or three services may find the added per-pod complexity (two containers to monitor, restart, and reason about instead of one) outweighs the consistency benefit at that scale.
• The concern isn’t actually cross-cutting. Northwind’s legacy-specific retry tuning is a good ambassador candidate because it’s peculiar to one external system; a generic retry policy that’s genuinely the same for every outbound call might be simpler as shared application-level configuration (Resilience4j, next chapter) than a dedicated per-destination ambassador.
• Overengineering signal: deploying a sidecar for a concern with a single, simple, stable configuration that rarely changes and doesn’t need independent versioning from the application — the operational overhead of an extra container isn’t justified if there’s nothing dynamic or cross-language about the concern.

5. Implementation Example

The mTLS sidecar, using Envoy as a lightweight TLS-terminating/originating proxy, deployed alongside order-service in the same pod:

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: order-service
5
spec:
6
  template:
7
    spec:
8
      containers:
9
        - name: order-service
10
          image: northwind/order-service:2.1.0
11
          ports: [{ containerPort: 8080 }]
12
          env:
13
            - name: INVENTORY_SERVICE_URL
14
              value: "http://localhost:15001"   # calls the sidecar, not inventory-service directly
15
        - name: mtls-sidecar
16
          image: envoyproxy/envoy:v1.31-latest
17
          ports: [{ containerPort: 15001 }]
18
          volumeMounts:
19
            - name: tls-certs
20
              mountPath: /etc/envoy/certs
21
              readOnly: true
22
      volumes:
23
        - name: tls-certs
24
          secret:
25
            secretName: order-service-mtls-certs

1
static_resources:
2
  listeners:
3
    - name: outbound_to_inventory
4
      address: { socket_address: { address: 0.0.0.0, port_value: 15001 } }
5
      filter_chains:
6
        - filters:
7
            - name: envoy.filters.network.http_connection_manager
8
              typed_config:
9
                route_config:
10
                  virtual_hosts:
11
                    - routes: [{ match: { prefix: "/" }, route: { cluster: inventory_service_mtls } }]
12
  clusters:
13
    - name: inventory_service_mtls
14
      load_assignment:
15
        endpoints: [{ lb_endpoints: [{ endpoint: { address: { socket_address: { address: inventory-service.default.svc.cluster.local, port_value: 443 } } } }] }]
16
      transport_socket:
17
        name: envoy.transport_sockets.tls
18
        typed_config:
19
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
20
          common_tls_context:
21
            tls_certificates:
22
              - certificate_chain: { filename: "/etc/envoy/certs/tls.crt" }
23
                private_key: { filename: "/etc/envoy/certs/tls.key" }

order-service’s own Kotlin code — its InventoryClient from Chapter 2 — needs zero changes to gain mTLS; it still makes a plain HTTP call, just to localhost:15001 instead of directly to inventory-service. The certificate handling, TLS handshake, and protocol details are entirely Envoy’s responsibility, upgradeable independently of any Kotlin release.

The ambassador for the legacy monolith, wrapping LegacyCustomerBridge’s connection with tuned retry and pooling behavior specific to that one, less-reliable destination:

1
        - name: legacy-ambassador
2
          image: northwind/legacy-ambassador:1.0.0
3
          ports: [{ containerPort: 15002 }]
4
          env:
5
            - name: LEGACY_MONOLITH_URL
6
              value: "http://legacy-php-monolith.default.svc.cluster.local"
7
            - name: RETRY_MAX_ATTEMPTS
8
              value: "5"
9
            - name: RETRY_BACKOFF_MS
10
              value: "200"
11
            - name: CONNECTION_POOL_SIZE
12
              value: "10"

1
package `in`.o612.eng.northwind.order.internal.legacy
2

3
import org.springframework.web.client.RestClient
4
import java.util.UUID
5

6
internal class LegacyCustomerBridge(
7
    // Now points at the ambassador sidecar on localhost, not the legacy
8
    // system directly — retry/pooling tuning lives entirely in the
9
    // ambassador's own configuration, not here.
10
    private val ambassadorClient: RestClient = RestClient.create("http://localhost:15002"),
11
    private val translator: LegacyCustomerTranslator,
12
) {
13
    fun getCustomer(customerId: UUID) =
14
        translator.translate(
15
            ambassadorClient.get().uri("/internal/api/customers/{id}", customerId)
16
                .retrieve().body(LegacyCustomerRecord::class.java)
17
                ?: error("Customer $customerId not found in legacy system")
18
        )
19
}

Compare this to Chapter 14’s version: the retry configuration, connection pool sizing, and backoff strategy — all previously either absent or awkwardly embedded in application code — have moved entirely to the ambassador’s own deployment configuration. LegacyCustomerBridge is now purely about making one HTTP call and handing the result to the translator; the connection’s operational tuning is someone else’s job, configured and versioned independently.

6. Step-by-Step Flow

1. Client action. order-service’s own checkout logic calls its InventoryClient and LegacyCustomerBridge exactly as before — no change to any call site.
2. API request. Both calls go to localhost, to a sidecar or ambassador respectively, rather than to a real network destination directly.
3. Service behavior. The sidecar wraps the call in mTLS before it leaves the pod; the ambassador applies retry and pooling logic specific to the legacy system before forwarding the request onward.
4. Database interaction. Unaffected — this pattern operates entirely at the connection/transport level, below any application or domain logic.
5. Inter-service communication. inventory-service’s own pod has a matching sidecar handling the inbound side of the same mTLS handshake — both ends of the connection are equally instrumented, without either service’s code being aware of it.
6. Error or failure handling. A transient legacy-system failure is retried by the ambassador according to its own configured policy, invisible to order-service’s code — which only ever sees either an eventual success or a final failure after the ambassador’s retry budget is exhausted.
7. Observability signals. The sidecar and ambassador each expose their own metrics (Envoy’s built-in stats, the ambassador’s retry/failure counters) — a genuinely separate observability surface from the application’s own metrics, both needed for a complete picture of that pod’s behavior.
8. Final response. order-service’s code receives a plain HTTP response in both cases, with all the transport-level complexity (TLS handshakes, retries, connection reuse) handled by companion processes it never had to implement or maintain.

7. Production Concerns

• Timeouts, retries, idempotency. Retry policy now lives in the ambassador, not the application — but the underlying idempotency requirement (Chapter 1’s ongoing theme) doesn’t go away; the ambassador retrying a non-idempotent call is just as dangerous as application code doing so carelessly. Configure ambassador retries only for calls the application has confirmed are safe to repeat.
• Data consistency. Unaffected directly — this pattern is transport-level, not data-level.
• API versioning and backward compatibility. The sidecar/ambassador’s configuration (Envoy config, retry policy) can be versioned and upgraded independently of the application container’s own release cycle — a genuine operational benefit, letting the platform team roll out a new mTLS cipher policy, for instance, without coordinating an application deploy.
• Authentication and service-to-service trust. This pattern is often exactly how mTLS-based service-to-service trust gets implemented in practice — the sidecar is frequently the actual enforcement point for the “every call trusted, verified, and encrypted” goal referenced but deferred in earlier chapters (Chapters 2, 6, 7).
• Logging, metrics, tracing, correlation IDs. A correlation ID set by the application must still be forwarded by the sidecar/ambassador as a header — verify this explicitly; a naively configured proxy can silently strip custom headers it doesn’t recognize, breaking tracing continuity without any obvious error.
• Kubernetes deployment, health probes, autoscaling. Multi-container pods need per-container readiness probes — a pod shouldn’t be marked Ready until both the application and its sidecar/ambassador are healthy, since a broken sidecar with a healthy application container is still a broken pod from the network’s point of view.
• Testing strategy. Test the application code against a local, plain HTTP stub (no sidecar involved) for unit and most integration tests — the sidecar/ambassador’s behavior is tested separately, at the infrastructure/configuration level, ideally with its own smoke tests verifying mTLS actually works end to end between two real pods.
• Migration strategy. Introduce sidecars incrementally, one cross-cutting concern at a time (mTLS first, as the concrete security requirement that motivated it), rather than adopting a full service mesh (a later chapter) speculatively before the specific, individual sidecar needs are well understood.

8. Common Mistakes

1. Putting business logic in a sidecar or ambassador. Adding domain-specific validation (e.g., checking a customer’s standing) inside the ambassador’s proxy logic mixes a transport-level concern with business rules that belong in the application. Fix: keep sidecars and ambassadors strictly to infrastructure concerns — connection handling, TLS, generic retries — never business decisions.
2. Retrying non-idempotent calls at the ambassador level without checking. Configuring blanket retries for every call the ambassador proxies, without verifying which underlying operations are safe to repeat, can cause duplicate side effects the application code never anticipated. Fix: configure retry policy per route/destination based on a deliberate idempotency assessment, not a blanket default.
3. Silently stripping correlation headers. A default proxy configuration that doesn’t explicitly forward custom headers breaks distributed tracing without any visible error — the trace just goes cold at the sidecar hop, and nobody notices until debugging an unrelated issue. Fix: explicitly verify and test that correlation and trace headers pass through every sidecar/ambassador unmodified.
4. No readiness coordination between application and sidecar containers. A pod marked Ready because the application container passed its own health check, while its mTLS sidecar is still starting, briefly accepts traffic it can’t actually route correctly. Fix: configure pod readiness to require all containers healthy, not just the application’s.
5. Adopting a full service mesh before understanding the individual sidecar needs it would automate. Jumping straight to Istio or Linkerd for a two-service mTLS requirement, without first understanding what a hand-configured Envoy sidecar actually does, makes debugging the mesh’s abstractions much harder when something inevitably needs troubleshooting. Fix: understand the sidecar pattern concretely first (as this chapter does); adopt a mesh once the number of sidecars and the consistency burden of managing them by hand genuinely justifies a control plane (the mesh chapter covers exactly this threshold).
6. Duplicating a concern in both the application and the sidecar. Keeping application-level retry logic and ambassador-level retry logic for the same call risks compounding retries (an application retry triggering multiple ambassador retries each) and unpredictable total latency under failure. Fix: decide explicitly which layer owns retries for each call path, and remove the logic from the other layer entirely.

9. Decision Guide

10. Hands-On Exercise

Extend it: add a logging sidecar that tails order-service’s application logs and ships them to a centralized log aggregator, without any change to order-service’s own logging configuration — demonstrating the pattern’s value for a concern entirely orthogonal to network traffic.

Simulate a failure: kill the legacy-ambassador container (not the order-service container) in a running pod, and observe what happens to LegacyCustomerBridge’s calls. Design the appropriate failure behavior — should order-service treat a dead ambassador as “legacy system unavailable,” and is that distinguishable from the legacy system itself being down?

Decision question, with justification required: Northwind now has mTLS sidecars for every service and an ambassador for the legacy integration — six services, six mTLS sidecars, one ambassador, each hand-configured. The platform team is debating whether this is the point to adopt a full service mesh instead of continuing to hand-configure Envoy sidecars per service. What specific operational signal (not just “we have several sidecars now”) would justify that move? Name the trade-offs from Section 1.

11. Key Takeaways

• A sidecar handles a cross-cutting infrastructure concern (TLS, logging, metrics) in a companion container sharing the application’s pod and network namespace, keeping that concern out of the application’s own code and independently versionable.
• An ambassador is a sidecar specifically proxying calls to one external destination, isolating that destination’s connection-level quirks (retries, pooling) from the calling application’s code.
• Keep sidecars and ambassadors strictly to transport/infrastructure concerns — business logic (like Chapter 14’s anti-corruption translation) belongs in application code, not a companion container.
• This pattern is the concrete building block a full service mesh later automates and manages centrally — understanding it by hand first makes a mesh’s abstractions far easier to reason about when that step becomes justified.
• Explicitly verify correlation and trace headers survive every sidecar/ambassador hop — silently dropped headers break tracing without any visible error.
• Decide which layer (application or sidecar/ambassador) owns retries for each call path, and avoid duplicating that logic in both, which can compound retries unpredictably.
• Introduce sidecars incrementally, for concrete, individual needs (mTLS, a specific legacy integration) — don’t adopt a full mesh speculatively before the consistency burden of hand-configuration actually justifies a control plane.