Service Discovery & Configuration

The Problem Service Discovery Solves

In a monolith, calling another module is a function call. The address is implicit — it is the same process. In a microservices architecture, services run on different machines, potentially in different data centers, and their addresses change constantly as instances scale up, scale down, restart, or get replaced.

Hardcoding service addresses does not work:

// This breaks when payment-service moves, scales, or restarts
const PAYMENT_SERVICE = 'http://10.0.1.42:3001';

Service discovery is the mechanism that allows services to find each other dynamically, without hardcoded addresses. It is foundational infrastructure for any distributed system.

Without service discovery:
┌──────────┐                        ┌──────────────┐
│ Order    │──── hardcoded IP ─────▶│ Payment Svc  │
│ Service  │     10.0.1.42:3001     │ (if it moves,│
└──────────┘                        │  orders fail)│
                                    └──────────────┘

With service discovery:
┌──────────┐    ┌──────────────┐    ┌──────────────┐
│ Order    │───▶│  Service     │───▶│ Payment Svc  │
│ Service  │    │  Registry    │    │ 10.0.1.42    │
└──────────┘    │              │    │ 10.0.1.43    │
  "Where is     │ payment-svc: │    │ 10.0.1.44    │
  payment-svc?" │ [42, 43, 44] │    └──────────────┘
                └──────────────┘

Service Registry

A service registry is a database of available service instances and their network locations. Services register themselves when they start and deregister when they stop.

┌───────────────────────────────────────────┐
│              Service Registry              │
│                                           │
│  payment-service:                         │
│    ├── 10.0.1.42:3001  (healthy, v2.1)   │
│    ├── 10.0.1.43:3001  (healthy, v2.1)   │
│    └── 10.0.1.44:3001  (draining, v2.0)  │
│                                           │
│  user-service:                            │
│    ├── 10.0.2.10:3002  (healthy, v1.5)   │
│    └── 10.0.2.11:3002  (healthy, v1.5)   │
│                                           │
│  notification-service:                    │
│    └── 10.0.3.20:3003  (healthy, v3.0)   │
│                                           │
└───────────────────────────────────────────┘

Self-Registration Pattern

Services register themselves with the registry on startup and send periodic heartbeats to confirm they are still alive.

import Consul from 'consul';

const consul = new Consul({
  host: process.env.CONSUL_HOST || 'consul',
  port: process.env.CONSUL_PORT || '8500',
});

const SERVICE_ID = `payment-service-${process.env.HOSTNAME}`;

async function registerService() {
  await consul.agent.service.register({
    id: SERVICE_ID,
    name: 'payment-service',
    address: getLocalIP(),
    port: parseInt(process.env.PORT || '3001'),
    tags: ['v2.1', 'production'],
    meta: {
      version: '2.1.0',
      region: process.env.AWS_REGION,
    },
    check: {
      http: `http://${getLocalIP()}:${process.env.PORT}/health`,
      interval: '10s',
      timeout: '5s',
      deregistercriticalserviceafter: '30s',
    },
  });

  console.log(`Registered ${SERVICE_ID} with Consul`);
}

// Deregister on shutdown
async function deregisterService() {
  await consul.agent.service.deregister(SERVICE_ID);
  console.log(`Deregistered ${SERVICE_ID} from Consul`);
}

process.on('SIGTERM', async () => {
  await deregisterService();
  process.exit(0);
});

process.on('SIGINT', async () => {
  await deregisterService();
  process.exit(0);
});

// Start service and register
const server = app.listen(PORT, async () => {
  await registerService();
});

Client-Side Discovery

The client queries the service registry directly and chooses which instance to call. The client handles load balancing.

┌──────────┐    1. Query registry     ┌──────────────┐
│ Order    │─────────────────────────▶│   Registry   │
│ Service  │                          └──────┬───────┘
│          │◀─── 2. Returns instances ────────┘
│          │     [10.0.1.42, .43, .44]
│          │
│          │    3. Client picks one
│          │─────────────────────────▶ 10.0.1.43:3001
└──────────┘

// Client-side discovery with caching
class ServiceDiscoveryClient {
  private cache: Map<string, ServiceInstance[]> = new Map();
  private consul: Consul;

  constructor() {
    this.consul = new Consul({ host: 'consul', port: '8500' });
  }

  async getInstances(serviceName: string): Promise<ServiceInstance[]> {
    // Check cache first (refresh every 30 seconds)
    const cached = this.cache.get(serviceName);
    if (cached && cached.length > 0) return cached;

    // Query Consul for healthy instances only
    const services = await this.consul.health.service({
      service: serviceName,
      passing: true,  // Only healthy instances
    });

    const instances = services.map((entry) => ({
      id: entry.Service.ID,
      address: entry.Service.Address,
      port: entry.Service.Port,
      tags: entry.Service.Tags,
      meta: entry.Service.Meta,
    }));

    this.cache.set(serviceName, instances);

    // Refresh cache periodically
    setTimeout(() => this.cache.delete(serviceName), 30000);

    return instances;
  }

  // Round-robin load balancing
  private counters: Map<string, number> = new Map();

  async getInstance(serviceName: string): Promise<ServiceInstance> {
    const instances = await this.getInstances(serviceName);
    if (instances.length === 0) {
      throw new Error(`No healthy instances found for ${serviceName}`);
    }

    const counter = (this.counters.get(serviceName) || 0) + 1;
    this.counters.set(serviceName, counter);

    return instances[counter % instances.length];
  }

  async callService(serviceName: string, path: string, options?: RequestInit) {
    const instance = await this.getInstance(serviceName);
    const url = `http://${instance.address}:${instance.port}${path}`;

    try {
      const response = await fetch(url, options);
      return response;
    } catch (error) {
      // Remove failed instance from cache to prevent routing to it
      const instances = this.cache.get(serviceName) || [];
      this.cache.set(
        serviceName,
        instances.filter(i => i.id !== instance.id)
      );
      throw error;
    }
  }
}

// Usage
const discovery = new ServiceDiscoveryClient();

async function getPaymentStatus(paymentId: string) {
  const response = await discovery.callService(
    'payment-service',
    `/api/payments/${paymentId}`
  );
  return response.json();
}

Server-Side Discovery

The client sends requests to a load balancer or router, which queries the registry and routes the request. The client does not know about the registry at all.

┌──────────┐    1. Request      ┌──────────────┐
│ Order    │───────────────────▶│ Load Balancer│
│ Service  │                    │ / API Gateway│
└──────────┘                    └──────┬───────┘
                                       │
                               2. Query registry
                                       │
                                ┌──────▼───────┐
                                │   Registry   │
                                └──────┬───────┘
                                       │
                               3. Route to instance
                                       │
                                ┌──────▼───────┐
                                │ Payment Svc  │
                                │ 10.0.1.43    │
                                └──────────────┘

This is the pattern used by Kubernetes (kube-proxy + CoreDNS), AWS ELB, and most API gateways.

Client-Side vs Server-Side Discovery

Aspect	Client-Side	Server-Side
Client complexity	Higher (needs discovery logic)	Lower (just calls an address)
Extra hop	No (direct to instance)	Yes (through load balancer)
Language support	Need library per language	Any language works
Load balancing	Client implements it	Load balancer handles it
Examples	Netflix Eureka + Ribbon, Consul	Kubernetes, AWS ELB, NGINX

Health Checks and Liveness Probes

Health checks ensure the registry only contains instances that can actually serve traffic. Without them, requests get routed to dead or broken instances.

Types of Health Checks

Liveness probe:
  "Is the process running and not deadlocked?"
  If fails: Restart the container/process
  Example: GET /healthz returns 200

Readiness probe:
  "Can this instance accept traffic right now?"
  If fails: Remove from load balancer, but do not restart
  Example: GET /ready — checks DB connection, cache, dependencies

Startup probe:
  "Has the application finished starting up?"
  If fails: Wait (do not restart yet, do not send traffic)
  Example: GET /startup — returns 200 once initialization is complete

// Health check endpoints
app.get('/healthz', (req, res) => {
  // Liveness: is the process alive?
  // Keep this simple — no dependency checks
  res.status(200).json({ status: 'alive' });
});

app.get('/ready', async (req, res) => {
  // Readiness: can we serve traffic?
  const checks: Record<string, boolean> = {};

  try {
    await db.raw('SELECT 1');
    checks.database = true;
  } catch {
    checks.database = false;
  }

  try {
    await redis.ping();
    checks.cache = true;
  } catch {
    checks.cache = false;
  }

  try {
    // Check critical downstream service
    const response = await fetch(`${PAYMENT_SERVICE_URL}/healthz`, {
      signal: AbortSignal.timeout(2000),
    });
    checks.paymentService = response.ok;
  } catch {
    checks.paymentService = false;
  }

  const ready = Object.values(checks).every(Boolean);
  res.status(ready ? 200 : 503).json({ ready, checks });
});

Kubernetes Health Check Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: order-service
          image: order-service:2.1.0
          ports:
            - containerPort: 3000

          # Liveness: restart if unresponsive
          livenessProbe:
            httpGet:
              path: /healthz
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 10
            failureThreshold: 3
            timeoutSeconds: 5

          # Readiness: remove from service if not ready
          readinessProbe:
            httpGet:
              path: /ready
              port: 3000
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 2
            timeoutSeconds: 3

          # Startup: wait for app to initialize
          startupProbe:
            httpGet:
              path: /healthz
              port: 3000
            initialDelaySeconds: 0
            periodSeconds: 2
            failureThreshold: 30  # 30 * 2s = 60s max startup time

Dynamic Service Discovery

DNS-Based Discovery

The simplest form of service discovery. Services are resolved via DNS names that return the IP addresses of healthy instances.

Query: payment-service.default.svc.cluster.local
Answer: 10.0.1.42, 10.0.1.43, 10.0.1.44

Kubernetes automatically manages DNS records
for services. When pods scale up or down,
DNS records are updated.

# Kubernetes Service — provides DNS-based discovery
apiVersion: v1
kind: Service
metadata:
  name: payment-service
spec:
  selector:
    app: payment-service
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3001

# Other services can now call:
# http://payment-service/api/payments
# Kubernetes resolves "payment-service" to a pod IP

// In Kubernetes, service discovery is just DNS
// No special library needed
const response = await fetch('http://payment-service/api/payments', {
  method: 'POST',
  body: JSON.stringify(paymentData),
});

Service Mesh (Envoy/Istio)

A service mesh adds a sidecar proxy to every service. The proxy handles discovery, load balancing, retries, circuit breaking, mTLS, and observability — all without changing application code.

Without service mesh:
┌──────────┐     ┌──────────┐
│ Order    │────▶│ Payment  │
│ Service  │     │ Service  │
└──────────┘     └──────────┘
  App handles: retries, circuit breaking, TLS, tracing

With service mesh (Istio/Envoy):
┌──────────────────┐     ┌──────────────────┐
│ ┌──────────────┐ │     │ ┌──────────────┐ │
│ │ Order Service│ │     │ │Payment Service│ │
│ └──────┬───────┘ │     │ └──────▲───────┘ │
│        │         │     │        │         │
│ ┌──────▼───────┐ │     │ ┌──────┴───────┐ │
│ │ Envoy Proxy  │─┼─────┼▶│ Envoy Proxy  │ │
│ │ (sidecar)    │ │     │ │ (sidecar)    │ │
│ └──────────────┘ │     │ └──────────────┘ │
└──────────────────┘     └──────────────────┘
  Proxy handles: discovery, load balancing,
  retries, mTLS, circuit breaking, tracing

# Istio VirtualService — traffic routing rules
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payment-service
spec:
  hosts:
    - payment-service
  http:
    - route:
        - destination:
            host: payment-service
            subset: v2
          weight: 90
        - destination:
            host: payment-service
            subset: v1
          weight: 10
      retries:
        attempts: 3
        perTryTimeout: 2s
      timeout: 10s

---
# DestinationRule — circuit breaker and connection pool
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payment-service
spec:
  host: payment-service
  trafficPolicy:
    connectionPool:
      http:
        h2UpgradePolicy: UPGRADE
        maxRequestsPerConnection: 100
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 30s
      baseEjectionTime: 60s

Configuration Management

Services need configuration: database URLs, feature flags, API keys, timeouts, rate limits. How you manage configuration determines how quickly and safely you can change system behavior.

Configuration Hierarchy

Priority (highest to lowest):
1. Environment variables    — per-deployment overrides
2. Consul KV / etcd        — dynamic, centralized config
3. Config files             — version-controlled defaults
4. Application defaults     — hardcoded fallbacks

┌─────────────────────────────────────────────────┐
│  Environment Variables (highest priority)        │
│  DB_HOST=prod-db.internal                        │
├─────────────────────────────────────────────────┤
│  Consul KV Store (dynamic config)                │
│  /config/payment-service/retry-limit = 3         │
│  /config/payment-service/timeout-ms = 5000       │
├─────────────────────────────────────────────────┤
│  Config File (version-controlled defaults)       │
│  config/default.json                             │
├─────────────────────────────────────────────────┤
│  Application Code (hardcoded fallbacks)          │
│  const DEFAULT_TIMEOUT = 10000;                  │
└─────────────────────────────────────────────────┘

Dynamic Configuration with Consul KV

import Consul from 'consul';

const consul = new Consul({ host: 'consul', port: '8500' });

class DynamicConfig {
  private config: Map<string, string> = new Map();
  private watchers: Map<string, Consul.Watch> = new Map();

  async load(keys: string[]) {
    for (const key of keys) {
      await this.loadKey(key);
      this.watchKey(key); // Watch for changes
    }
  }

  private async loadKey(key: string) {
    try {
      const result = await consul.kv.get(key);
      if (result) {
        this.config.set(key, result.Value);
      }
    } catch (error) {
      console.error(`Failed to load config key: ${key}`, error);
    }
  }

  private watchKey(key: string) {
    const watcher = consul.watch({
      method: consul.kv.get,
      options: { key },
    });

    watcher.on('change', (data) => {
      if (data) {
        const oldValue = this.config.get(key);
        const newValue = data.Value;
        this.config.set(key, newValue);

        console.log(`Config updated: ${key}`, {
          oldValue,
          newValue,
        });
      }
    });

    watcher.on('error', (err) => {
      console.error(`Config watch error for ${key}:`, err);
    });

    this.watchers.set(key, watcher);
  }

  get(key: string, defaultValue?: string): string {
    return this.config.get(key) || defaultValue || '';
  }

  getNumber(key: string, defaultValue: number): number {
    const val = this.config.get(key);
    return val ? parseInt(val, 10) : defaultValue;
  }

  getBoolean(key: string, defaultValue: boolean): boolean {
    const val = this.config.get(key);
    return val ? val === 'true' : defaultValue;
  }

  destroy() {
    for (const watcher of this.watchers.values()) {
      watcher.end();
    }
  }
}

// Usage
const config = new DynamicConfig();
await config.load([
  'config/payment-service/retry-limit',
  'config/payment-service/timeout-ms',
  'config/payment-service/feature-new-checkout',
]);

// Config updates automatically when changed in Consul
const retryLimit = config.getNumber('config/payment-service/retry-limit', 3);
const featureEnabled = config.getBoolean('config/payment-service/feature-new-checkout', false);

Feature Flags

Feature flags let you change behavior without deployments. They are a form of dynamic configuration specifically for enabling or disabling features.

class FeatureFlags {
  constructor(private config: DynamicConfig) {}

  isEnabled(flag: string, context?: { userId?: string }): boolean {
    const key = `features/${flag}`;
    const value = this.config.get(key, 'false');

    // Simple on/off
    if (value === 'true' || value === 'false') {
      return value === 'true';
    }

    // Percentage rollout
    if (value.endsWith('%') && context?.userId) {
      const percentage = parseInt(value);
      const hash = simpleHash(context.userId + flag) % 100;
      return hash < percentage;
    }

    return false;
  }
}

// Consul KV entries:
// features/new-checkout = true           (enabled for all)
// features/dark-mode = 25%              (enabled for 25% of users)
// features/beta-search = false          (disabled for all)

Secrets Management

Secrets (database passwords, API keys, encryption keys) require special handling. They should never be in source code, environment variables (visible in process listings), or config files.

HashiCorp Vault

┌──────────┐    1. Authenticate      ┌──────────┐
│ Service  │───────────────────────▶ │  Vault   │
└──────────┘                         └────┬─────┘
                                          │
     ◀──── 2. Lease secret (time-limited) ┘

Vault provides:
- Dynamic secrets (generate DB credentials on demand)
- Secret rotation (auto-rotate passwords)
- Encryption as a service (encrypt/decrypt without exposing keys)
- Audit logging (who accessed what secret, when)

import vault from 'node-vault';

const vaultClient = vault({
  apiVersion: 'v1',
  endpoint: process.env.VAULT_ADDR || 'http://vault:8200',
  token: process.env.VAULT_TOKEN,
});

// Read a static secret
async function getSecret(path: string): Promise<Record<string, string>> {
  const result = await vaultClient.read(`secret/data/${path}`);
  return result.data.data;
}

// Get database credentials (dynamic secret)
async function getDatabaseCredentials() {
  const result = await vaultClient.read('database/creds/my-role');

  return {
    username: result.data.username,
    password: result.data.password,
    leaseDuration: result.lease_duration,
    leaseId: result.lease_id,
  };
}

// Application startup
async function initializeDatabase() {
  const creds = await getDatabaseCredentials();

  const pool = new Pool({
    host: process.env.DB_HOST,
    database: 'myapp',
    user: creds.username,
    password: creds.password,
  });

  // Renew lease before it expires
  setInterval(async () => {
    await vaultClient.lease.renew(creds.leaseId);
  }, (creds.leaseDuration / 2) * 1000);

  return pool;
}

Kubernetes Secrets

# Create secret
apiVersion: v1
kind: Secret
metadata:
  name: payment-service-secrets
type: Opaque
data:
  db-password: cGFzc3dvcmQxMjM=    # base64 encoded
  api-key: c2VjcmV0LWtleS14eXo=

---
# Mount in pod
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  template:
    spec:
      containers:
        - name: payment-service
          image: payment-service:2.1.0

          # As environment variables
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: payment-service-secrets
                  key: db-password

          # Or as mounted files
          volumeMounts:
            - name: secrets
              mountPath: /etc/secrets
              readOnly: true

      volumes:
        - name: secrets
          secret:
            secretName: payment-service-secrets

Important: Kubernetes secrets are base64-encoded, not encrypted. For production, use Sealed Secrets, External Secrets Operator, or Vault integration.

Secrets Management Best Practices

DO:
  - Use a dedicated secrets manager (Vault, AWS Secrets Manager)
  - Rotate secrets regularly (automated rotation preferred)
  - Use short-lived, dynamic credentials where possible
  - Audit all secret access
  - Encrypt secrets at rest and in transit
  - Use different secrets per environment

DO NOT:
  - Commit secrets to git (even in private repos)
  - Log secrets (mask them in log output)
  - Pass secrets via command-line arguments (visible in process list)
  - Share secrets between environments (prod key in staging)
  - Store secrets in plain text config files

Key Takeaways

• Service discovery is foundational to microservices. Without it, you are either hardcoding addresses or managing configuration that changes with every deployment.
• DNS-based discovery (Kubernetes) is the simplest option. Start there unless you need advanced features like health-aware routing or traffic shaping.
• Health checks must distinguish liveness (is it alive?), readiness (can it serve traffic?), and startup (has it initialized?). Each serves a different purpose.
• Dynamic configuration lets you change behavior without deployments. Use it for feature flags, timeouts, rate limits, and circuit breaker thresholds.
• Secrets management is not optional. Use a dedicated tool (Vault, AWS Secrets Manager). Never commit secrets, never log them, and rotate them regularly.
• A service mesh (Istio/Envoy) handles discovery, load balancing, retries, mTLS, and observability at the infrastructure level. Consider it when the complexity of managing these concerns in application code becomes a burden.