Resilient Nameserver Infrastructure – The Architecture

In a previous post, the foundation for the basic concepts of two scenarios was laid. See Resilient Nameserver Infrastructure – The Tools [Link].

In this post, the same tools will be used, along with additional concepts, to accomplish a geographically fault-tolerant and highly available service across multiple architectural designs.

For the comparison and illustration of the architectural designs, 4 layers will be assembled.

IP ROUTING

Unicast
- Pros
  - Simpler to deploy, manage, and troubleshoot.
  - Each IP address maps to a single server or location.
- Cons
  - Offers lower resiliency and scalability.
  - Higher susceptibility to latency issues, outages, and DoS attacks.
Anycast
- Pros
  - Advertises the same IP address from multiple geographically distributed locations.
  - Improves latency, availability, and DoS resistance.
- Cons
  - Increased operational complexity in routing, monitoring, and troubleshooting.
  - Requires a consistent synchronization strategy across distributed nodes.

Reflections

While not entirely predictable, it is reasonable to assume that each of the geographically distributed servers (see map above) may:
- For unicast:
  - Each server serves roughly 1/3 of the globe with the lowest latency consistently (see the circles in the map).
- For anycast:
  - Each request, regardless of origin, may be served with the lowest latency only 1/3 of the time, and the highest latency also 1/3 of the time.
The best option depends entirely on business requirements.
- Does the business serve globally? Go anycast.
- Does the business serve regionally? Go either unicast or anycast in a less spread-out fashion (see image below).

TRAFFIC HANDLING

Incoming traffic is one of the largest attack surfaces and also offers the greatest opportunities for improving scalability.

The edge is where the first layer of defense happens. Here are things to consider when designing how traffic is handled at the edge.

Use security groups to allow only ports 53 UDP/TCP at the edge router.
While firewalls (routers) often claim to provide DoS and DDoS protection, this is not recommended when combined with a DNS-aware proxy.
A DNS-aware proxy, such as PowerDNS DNSDist, is capable of:
- Filtering and dropping maliciously crafted packets.
- Throttling abusive clients and blocking enumeration attempts.
- Balancing requests across a pool of servers.
- No more than 1 proxy per IP for proper rate-limit calculation and throttling.
A caching DNS resolver, such as PowerDNS Recursor, provides the following benefits:
- Acts as a capacity multiplier for authoritative servers, reducing the number of required backend instances.
- Cache hits are computationally cheaper than full resolution and deliver extremely low-latency responses.
- Protects the backend database and significantly reduces its overhead.
- With at least one resolver per proxy, but all part of the same pool, they can serve all proxies.

Reflections

The traffic handling layer must do the heavy lifting, serving tens of thousands of requests per second, and only forwarding legitimate and unique requests to the authoritative nameserver.
Both proxies and resolvers are essentially ephemeral (stateless) and can be destroyed and redeployed one at a time with no operational impact.

HIGH AVAILABILITY

While the first layer defines how the DNS service behaves globally or regionally, and the second layer defines how robust the service is, the third layer is where the computing happens to respond to each request.

For small environments:
- Each authoritative nameserver can be self-sufficient per point of presence.
- When using Bind9 files or SQLite, AXFR/IXFR must be used to replicate (sync) changes across instances.
- Advanced features cannot be replicated if not covered by RFC-1995 and RFC-5936.
- A primary vs. secondary hierarchy must be configured at the authoritative level.
For larger environments:
- Each geographic point of presence should have at least 2 instances of the authoritative nameserver.
- Synchronization is delegated to the backend, leaving the nameservers stateless and ephemeral.

Since the nameserver is tightly coupled with its backend, more on high availability is covered in the backend layer below.

BACKEND STRATEGIES

This is where modern DNS features shine.

AXFR (Zone Transfer Protocol) and IXFR (Incremental Zone Transfer)
- While still in use, it does not cover modern features.
- It relies on DNS Notify (RFC-1996) for faster propagation of changes.
- In 2021, Zone Transfer over TLS (RFC-9103) was introduced to offer encrypted transfers.

Database-Level Replication
- Near event-driven replication is achieved with synchronous or semi-synchronous replication.
- Nameservers no longer communicate with each other; there is no concept of primary/secondary among them.
- Because state is managed at the database level, all nameservers behave as primary.
- All features of the nameserver are replicated, along with zones and records.
- Bridges and tunnels are used to provide internal communication for replication.

Modern tools, such as PowerDNS Authoritative Server, offer a variety of backend options. MySQL/MariaDB and PostgreSQL are among the most popular.

For geographically dispersed servers, data synchronization is done through some form of replication. While not atomic, changes can propagate in under a second. Externally cached responses will only reflect changes once they expire based on their defined TTL (Time-To-Live).

Note: The topic of how to cluster and replicate databases is outside the scope of this post. See MySQL Database Replication [Link].

PROPOSED DESIGNS

Simplified Classical Architecture

Unicast
- ns1.example.com 100.1.1.10
- ns2.example.com 100.1.1.20
- ns3.example.com 100.1.1.30

This simplified design has only 1 entry point (public IP) per point of presence.
- Using a virtual IP, a failover proxy can take the load if the primary fails.
- Alternatively, the single proxy can act as the fuse (sacrificial element) that protects the rest of the infrastructure from attacks.
With only 2 resolvers in the pool, a large number of requests can be served.
- This is the first scaling point when load increases.
It is recommended to have at least 2 instances of the nameserver.
The choice of backend and its level of resilience (number of instances) depends on the organization’s requirements.

Optimized Architecture

Anycast
- ns1.example.com 100.1.1.10
- ns2.example.com 100.1.1.20

This proposed design has only 2 entry points (public IPs) per point of presence.
- Optionally, each proxy can have a failover instance using a virtual IP. Recommended for large environments.
Size the number of resolvers in the pool according to the expected load.
Start with at least 3 instances of the nameserver per point of presence, then scale up and/or out as needed.
This design calls for nothing less than a PostgreSQL or MySQL/MariaDB cluster:
- The cluster size should be 3–5 nodes for an appropriate quorum.
- Consider using a DB proxy to distribute the load across nodes.
- Prefer querying all nodes for read-only operations.
- Asynchronous replication between clusters is acceptable.

BONUS

Conceptually, a “poor man’s” design could achieve remarkable performance with less complexity and lower cost, at the expense of reliability.

Ideally deployed as anycast, though unicast may be more feasible in this case.
It has a clear single point of failure at the main point of presence, but:
- This design spreads caches across different data centres in geographically distant locations.
- Resolvers will continue to respond with cached results during an eventual outage of the authoritative servers.
- The longer the TTL of cached records, the longer an outage can occur without users noticing.
No replication or synchronization of changes is needed.
Any backend strategy is suitable.