March 20, 2024

Service discovery is a crucial aspect of modern distributed systems. In this article, we'll explore how to implement service discovery in Docker Compose using SRV records, which provide a standardized way to specify the location of services.

Prerequisites

  • Basic understanding of Docker and Docker Compose
  • Familiarity with DNS concepts
  • Docker installed on your system

Understanding SRV Records

SRV (Service) records are a type of DNS record that specify information about available services. They follow this format:

_service._proto.name. TTL class SRV priority weight port target

For example:

_raft._tcp.gatewayd.local. IN SRV 0 0 2223 gatewayd-1.gatewayd.local.

Key Components of SRV Records:

  • Priority: Lower values have higher priority (0 is highest)
  • Weight: Used for load balancing when priorities are equal
  • Port: The port number where the service is running
  • Target: The hostname of the machine providing the service

Step 1: Setting Up CoreDNS

Since we can't use SRV records directly in `/etc/hosts`, we'll use CoreDNS as our DNS server. First, let's create the necessary configuration files.

Note: In this example, we're using SRV records similar to Kubernetes headless services, which don't require the traditional `_service._proto` prefix. This approach simplifies service discovery while still maintaining the core functionality of SRV records.

CoreDNS Configuration

Create a file named `Corefile`:

. {
    forward . /etc/resolv.conf
    log
    errors
    cache {
        success 1024 60 60
        denial 1024 5 5
    }
}

gatewayd.local {
    file /etc/coredns/gatewayd.db
    reload 30s
    log
    errors
}

DNS Zone File

Create a file named `gatewayd.db`:

$TTL 60
@ IN SOA gatewayd.local. admin.gatewayd.local. (
    2024032001 ; serial
    3600       ; refresh
    1800       ; retry
    604800     ; expire
    60         ; minimum
)

; SRV records for Raft service
gatewayd.local. IN SRV 0 0 2223 gatewayd-1.gatewayd.local.
gatewayd.local. IN SRV 0 0 2223 gatewayd-2.gatewayd.local.
gatewayd.local. IN SRV 0 0 2223 gatewayd-3.gatewayd.local.

; SRV records for gRPC service
gatewayd.local. IN SRV 0 0 50051 gatewayd-1.gatewayd.local.
gatewayd.local. IN SRV 0 0 50051 gatewayd-2.gatewayd.local.
gatewayd.local. IN SRV 0 0 50051 gatewayd-3.gatewayd.local.

Step 2: Basic Docker Compose Setup

Let's start with a basic Docker Compose configuration that includes CoreDNS:

dns:
    image: coredns/coredns:latest
    command: ["-conf", "/etc/coredns/Corefile"]
    volumes:
      - ./coredns:/etc/coredns
    ports:
      - "5355:53/udp"

Step 3: Testing DNS Resolution

# Get the CoreDNS container IP
docker ps | grep dns
docker inspect <container_id>

# Test SRV record resolution in one of containers
nslookup -type=srv gatewayd.local <coredns_ip>
Server:         172.28.0.5
Address:        172.28.0.5#53

gatewayd.local  service = 0 0 2223 gatewayd-1.gatewayd.local.
gatewayd.local  service = 0 0 50051 gatewayd-1.gatewayd.local.
gatewayd.local  service = 0 0 2223 gatewayd-2.gatewayd.local.
gatewayd.local  service = 0 0 50051 gatewayd-2.gatewayd.local.
gatewayd.local  service = 0 0 2223 gatewayd-3.gatewayd.local.
gatewayd.local  service = 0 0 50051 gatewayd-3.gatewayd.local.

To avoid manually passing the CoreDNS IP address, we need to:

  1. Set up a custom network with a fixed subnet
  2. Assign a static IP to the CoreDNS container
  3. Configure each service to use CoreDNS as primary DNS and Docker's built-in DNS as secondary

Step 4: Complete Docker Compose Setup

Now, let's create a complete setup with multiple services and proper DNS configuration:

version: '3.8'

services:
  dns:
    image: coredns/coredns:latest
    command: ["-conf", "/etc/coredns/Corefile"]
    volumes:
      - ./coredns:/etc/coredns
    ports:
      - "5355:53/udp"
    networks:
      gatewayd-network:
        ipv4_address: 172.28.0.5

  gatewayd-1:
    dns:
      - 172.28.0.5  # CoreDNS IP
      - 127.0.0.11  # Docker's default DNS
    networks:
      - gatewayd-network
    depends_on:
      - dns

  gatewayd-2:
    dns:
      - 172.28.0.5
      - 127.0.0.11
    networks:
      - gatewayd-network
    depends_on:
      - dns

  gatewayd-3:
    dns:
      - 172.28.0.5
      - 127.0.0.11
    networks:
      - gatewayd-network
    depends_on:
      - dns

networks:
  gatewayd-network:
    driver: bridge
    ipam:
      config:
        - subnet: 172.28.0.0/24

Step 5: Using SRV Records in Your Application

Here's an example of how to use SRV records in a Go application:

package main

import (
    "fmt"
    "net"
    "strings"
)

func main() {
    domain := "gatewayd.local"
    _, srvRecords, err := net.LookupSRV("", "", domain)
    if err != nil {
        fmt.Println("DNS lookup failed:", err)
        return
    }

    fmt.Println("Service endpoints for", domain)
    for _, srv := range srvRecords {
        host := strings.TrimSuffix(srv.Target, ".")
        addr := fmt.Sprintf("%s:%d", host, srv.Port)
        fmt.Printf("Host: %s\nAddress: %s\n", host, addr)
    }
}

Best Practices

1. DNS Configuration:

  • Always include Docker's default DNS (127.0.0.11) as a fallback
  • Use static IP addresses for DNS servers in Docker networks
  • Set appropriate TTL values in your DNS records

2. Service Discovery:

  • Use meaningful service names in your SRV records
  • Consider using different SRV records for different protocols
  • Implement proper error handling in your application code

3. Docker Compose:

  • Use explicit network configurations
  • Set proper dependencies between services
  • Consider using health checks for critical services

Conclusion

Using SRV records with Docker Compose provides a robust way to implement service discovery in your containerized applications. This approach is particularly useful for distributed systems where services need to discover and communicate with each other dynamically.

By following the steps outlined in this article, you can set up a reliable service discovery mechanism that scales well with your application's needs. Remember to implement proper security measures and monitoring to ensure the reliability and security of your service discovery system.

Additional Resources