DNS zones are typically served by more than one server. One of these is called the master or primary and the ‘copies’ are called slave or secondary servers. The slave servers obtain zone data via a process called zone transfer (AXFR) or incremental zone transfer (IXFR). (And I refuse to abstain from using the terms master/slave.)

When we provision a master server with a new zone we must update all slave servers and inform them of the existence of this new zone and which addresses the master servers for the zone have. Unless you are using PowerDNS with MySQL or PostgreSQL replication (which is off topic for this discussion) this is a procedure that is normally done manually. By manually I mean there is no standard procedure for doing this.

A typical way of provisioning slave servers with the names of zones they should slave is to obtain (out-of-band somehow) a list of zones, add them to the slave’s server configuration and tell the slave to “go get ‘em”. One such proposed method is called dper: the DNS Peering Protocol which has an extra XML configuration that needs to be transported (somehow) to the slaves. If BIND is your master, you could use BIND’s statistics server and, as Tony points out in the comments below, Paul Vixie’s metazones solve the “transport” of a zone list as well. Other typical ways include writing a small program to slurp through the provisioning system, dump a list of zones, etc.

There are some obvious problems with this:

• There typically exists a time lag between creating the master zone and having it replicated to the slaves because the slaves need to know about the new zones.
• The out-of-band method of transporting a list (or a full configuration) for slave servers is non-standard. Some people query, say, the database back-end when using a database as provisioning system, others create a list and pull it from the slaves via HTTP, yet others rsync it over, etc. The possibilities are almost endless.

When operating as a master and slave combination, PowerDNS has a pretty unique feature for its slave servers which is called ‘Superslave’. Quoting myself (again) from Chapter 6 of my book:

A unique feature of PowerDNS is that you can configure it to accept notifications from specified “trustworthy” master servers for which it does not yet carry slave zones, and have it create the slave zone(s), and provision them from the master via incoming zone transfers, all automatically. In this role, the server is said to be a Superslave. (By contrast, a normal slave will accept/action NOTIFYs only if you have manually pre-configured it as a slave for the zone. You have to create a zone on the slave, giving it the name of the zone, and create an NS record in the zone pointing to this slave itself.)

This effectively allows PowerDNS to provision its slave servers automatically.

The question is: can we do that for, say, BIND and NSD as well?

This isn’t just a theoretical question. I’m currently working with a customer who has a large amount of zones which are pretty volatile (i.e. zones come and go). They have a hidden master server (PowerDNS, but it could easily have been of a different brand) and BIND slave servers (to be augmented by a couple of NSD slave servers shortly). The issues we’re seeing here (irrespective of the brand of the hidden master server) are:

1. A new zone is provisioned on the master, whereupon PowerDNS sends out a NOTIFY to its slaves.
2. The slaves don’t yet know anything about the existence of this new zone, so they ignore the NOTIFY.
3. Meanwhile, the provisioning system creates an include file for BIND’s named.conf and reloads the server, during which the server is “deaf” for queries (also for the next bunch of NOTIFY messages the master may send out due to new zones being provisioned).
4. The server is now ready, and named goes and transfers (AXFR) the new zones at which point they are available to be queried.

Even by shortening the frequency of the periodical re-provisioning of named (step

1. above), there is always some time lag (sometimes several minutes) until the new zone(s) is (are) available on the BIND slave servers.

I’ve given this some thought, particularly in view of the fact that BIND has addzone and NSD4 also has support for addzone, allowing both brands of servers to configure new zones on-the-fly. (While there is an initiative to create some form of control protocol (e.g. Requirements for the Nameserver Communication protocol) I’m not aware of something that can be used out of the box.)

The method I propose, and for which I include a prototypical proof of concept (which is working very nicely in my portable data center) can be used to provision BIND slaves, and NSD slaves from any server which is capable of sending a DNS NOTIFY to its slaves.

A small utility called metaslave runs on the slave servers alongside BIND or NSD and, on a different port number, listens for NOTIFY requests. As soon as it receives a NOTIFY, it checks a local database (simplistically implemented as a file on the file system in this PoC) to see whether it knows of the zone. If it doesn’t, metaslave launches an external command to add the zone to the particular brand of name server. (Instead of using a database, I can check existence of the zone file proper, which ought to be safe enough; this does mean ensuring all slave servers have the same “formula” for contructing paths to zone files.)

#!/usr/bin/perl

use strict;
use warnings;
use Net::DNS::Nameserver;

my $depot = '/tmp/zones';

sub notify_handler {
    my ($qname, $qclass, $qtype, $peerhost,$query,$conn) = @_;
    my ($rcode, @ans, @auth, @add, $path);

    $path = $depot."/".$qname;
    $rcode = undef;

    print "Received NOTIFY query from $peerhost for " . $qname ."\n";

    # Slight sanity check (don't accept slashes)
    # FIXME: lowercase and do more checks.

    $rcode = 'SERVFAIL' if ($qname =~ /\//);

    # Check whether the zone already exists on this slave. Implemented here
    # as a file on a filesystem. What I'd probably do is use Redis (SET)
    # or SQLite3, whereby the former could be used to 'report back' to a
    # monitoring station via PUB/SUB, etc.

    # Actually we can check existence of the zone on the file system 
    # instead of using a separate database...

    # As Marc suggests, we can also simply query the paired DNS server (i.e.
    # the server this script is catering to) using Net::DNS to see if the zone
    # has been defined.

    $rcode = 'NOERROR'  if (-f $path);

    if (defined($rcode)) {
        return ($rcode, [], [], [],
                { aa => 1, opcode => 'NS_NOTIFY_OP'} );
    }

    # NSD: addzone ${qname} groupname
    # BIND: rndc addzone ${qname} IN '......;'

    my $command = "rndc addzone ${qname} in '{type slave; file \"${qname}\"; masters { 172.16.153.102;};};'";

    if (open(DB, "> $path")) {
        print DB $command, "\n";
        close(DB);
    }

    # FIXME: too "heavy". Ensure non-blocking, maybe as a kind of queue?
    # FIXME: maybe unreliable, but attempt to obtain return code of 'addzone'
    #        and maybe SERVFAIL (not that it's of any use)

    system($command);

    $rcode = "NOERROR";
    return ($rcode, [], [], [],
            { aa => 1, opcode => 'NS_NOTIFY_OP'} );
}

sub reply_handler {
    my ($qname, $qclass, $qtype, $peerhost,$query,$conn) = @_;
    my (@ans, @auth, @add);

    return ('SERVFAIL', \@ans, \@auth, \@add);
}

my $ns = Net::DNS::Nameserver->new(
    LocalPort     => 5353,
    ReplyHandler  => \&reply_handler,
    NotifyHandler => \&notify_handler,
    Verbose       => 0,
    Debug         => 0,
) || die "couldn't create nameserver object\n";

$ns->main_loop;

My proof-of-concept metaslave uses Net::DNS which, if you recall, was augmented with a NotifyHandler for me back when I wrote the book. Note also, that this is currently very simplistic and needs quite a bit of refinement, which I’ll probably do soon-ish.

The (hidden) master servers have to be able to NOTIFY a separate server:

• PowerDNS has ALSO_NOTIFY in its domainmetadata table with support for specifying an alternative port number (see example below).
• NSD has notify (with optional TSIG keys), and a port number can be added to the address with @number.
• BIND has also-notify with an optional port setting.

As an example, consider the following SQL I inject into my PowerDNS server:

BEGIN WORK;
INSERT INTO domains (name, type) VALUES ('b.aa', 'MASTER');
INSERT INTO records (domain_id, name, ttl, type, content)
      SELECT id, 'b.aa', 60, 'SOA', 'ns.b.aa jp.b.aa 1 10800 3600 1814400 14400'
      FROM domains WHERE name = 'b.aa';
INSERT INTO records (domain_id, name, ttl, type, content)
        SELECT id, 'b.aa', 60, 'NS', 'nsd4.prox'
        FROM domains WHERE name = 'b.aa';
INSERT INTO records (domain_id, name, ttl, type, content)
        SELECT id, 'b.aa', 60, 'NS', 'bind993.prox'
        FROM domains WHERE name = 'b.aa';
INSERT INTO records (domain_id, name, ttl, type, content)
        SELECT id, 'b.aa', 60, 'A', '172.16.153.110'
        FROM domains WHERE name = 'b.aa';

INSERT INTO domainmetadata (domain_id, kind, content)
        SELECT id, 'ALSO-NOTIFY', '172.16.153.103:5353'
        FROM domains WHERE name = 'b.aa';
INSERT INTO domainmetadata (domain_id, kind, content)
        SELECT id, 'ALSO-NOTIFY', '172.16.153.101:5353'
        FROM domains WHERE name = 'b.aa';
COMMIT WORK;

The last two INSERT statements populate the domainmetadata table and set up the additional notification to the metaslave program. After a few seconds, I see the following logged by PowerDNS (I’ve numbered the lines to discuss them.)

 1. 1 domain for which we are master needs notifications
 2. Queued notification of domain 'b.aa' to 172.16.153.101
 3. Queued notification of domain 'b.aa' to 172.16.153.103
 4. Queued also-notification of domain 'b.aa' to 172.16.153.101:5353
 5. Queued also-notification of domain 'b.aa' to 172.16.153.103:5353
 6. Received unsuccessful notification report for 'b.aa' from 172.16.153.101:53, rcode: 9
 7. Removed from notification list: 'b.aa' to 172.16.153.101:53
 8. No question section in packet from 172.16.153.103, rcode=3
 9. Unable to parse SOA notification answer from 172.16.153.103
10. Removed from notification list: 'b.aa' to 172.16.153.110:5353 (was acknowledged)
11. No master domains need notifications

NOTIFYs (2. and 3.) are sent by PowerDNS to the NS RRset (the set of NS records), whereas 4. and 5. (note the :5353) are those from ALSO-NOTIFY. The former are negatively acknowledged in lines 6. through 9. because the slave server don’t know about the b.aa zone (yet). The actual zone transfers (AXFR) from both servers follow suit and complete within a second.

A great advantage in adding zones dynamically, at least to BIND, is we keep the time frame in which named is “deaf” to queries very short.

While we’re able to provision new zones on slave servers this way, and I’m confident this is a viable method, we cannot decomission removed zones: the DNS protocol doesn’t have a REMOVE-THIS-ZONE querytype; we continue to have to do that “manually”.

Or do we? Read on!

Removing zones from slave servers

Tony Finch mentioned metazones again this morning, and this got me thinking I could use them to automatically remove zones from slaves when they’re deleted on the master server.

Recall that the particular master I’m talking about is an authoritative PowerDNS master with a database back-end. (For a different brand of master, BIND, say, we’d have to modify a number of details: I would probably create a dynamically updatable metazone and script dynamic updates when adding or removing zones from the master.)

I’m creating a master zone called meta.meta which will store a metazone (of sorts). This is what it looks like in the records table

and this is what it looks like in master zone file format after it’s been transferred to BIND or NSD:

meta.meta.        86400 IN SOA meta.meta. jp.meta. (
                         1360867149 ; serial
                         10800      ; refresh (3 hours)
                         3600       ; retry (1 hour)
                         3600       ; expire (1 hour)
                         3600       ; minimum (1 hour)
                                       )
meta.meta.        86400 IN NS bind993.prox.
meta.meta.        86400 IN NS nsd4.prox.

When a zone is deleted from the domains database table, a trigger adds a record to the metazone by an INSERT into the records table.

DELIMITER $$$

DROP TRIGGER IF EXISTS domains_del;

CREATE TRIGGER `domains_del` AFTER DELETE ON `domains`
FOR EACH ROW BEGIN

    -- Add a record to the records table for the "metazone"
    -- indicating that a zone is being deleted.

    SET @metazone = 'meta.meta';
    SET @d_id = OLD.id;
    SET @d_name = OLD.name;
    SET @txt = CONCAT('"', 'd=', @d_id, ' ', UTC_TIMESTAMP(), '"');

    -- Remove all records for this zone.
    DELETE FROM records WHERE domain_id = @d_id;

    -- Add a NEW record to the metazone detailing which zone we've
    -- just deleted.

    INSERT INTO records (domain_id, name, ttl, type, content)
        SELECT id, CONCAT(@d_name, '.', @metazone), 86400, 'TXT', @txt
        FROM domains WHERE name = @metazone;

    -- Update SOA record in metazone.

    UPDATE records
        SET content = CONCAT(@metazone, ' jp.meta ', UNIX_TIMESTAMP(NOW()), ' 10800 3600 3600 3600')
        WHERE domain_id = (SELECT id FROM domains WHERE name = @metazone)
              AND type = 'SOA';
END $$$

DELIMITER ;

Suppose I delete a zone example.com, the records database table then looks like this:

mysql> SELECT id, name FROM domains WHERE name = 'example.com';
+----+-------------+
| id | name        |
+----+-------------+
| 22 | example.com |
+----+-------------+

mysql> DELETE FROM domains WHERE name = 'example.com';
Query OK, 1 row affected (0.01 sec)

The database trigger added a new TXT record with rdata containing the original PowerDNS domain_id (for debugging) and a UTC timestamp. Simultaneously, the trigger also updated the SOA serial number of the meta.meta zone, which will cause the zone’s slaves to be notified whereupon they will transfer it. So far, so good.

A program on the slave servers can then periodically (via cron for example) perform a zone transfer (from 127.0.0.1) to obtain the meta.meta zone and remove zones from the slave server configurations as well as moving the slave zone files into a backup area. Basically the program looks like this:

#!/usr/bin/perl

use strict;
use warnings;
use Net::DNS;

my $metazone = 'meta.meta';

my $res = Net::DNS::Resolver->new;

# Perform a zone transfer from the local slave server
$res->nameservers('127.0.0.1');

my @zone = $res->axfr($metazone);

foreach my $rr (@zone) {
    # Skip records we know can't be meant for us
    next unless $rr->type eq 'TXT';
    next unless $rr->name =~ /\.${metazone}$/;

    # Strip meta zone name from origin
    my $name = $rr->name;
    $name =~ s/\.${metazone}$//;

    # Perform sanity check on name
    # Skip if zone $name doesn't exist on this slave.

    # Remove zone from slave server:
    #   if BIND:
    #     issue [ rndc delzone ] 
    #   if NSD:
    #     issue [ nsd-control delzone ]
    #   if PowerDNS slave:
    #    delete from back-end database
    #   etc.
    
    # Determine if zone file on disk; if so, move it into
    #    backup area.

    print $name, "\n";
}

When I run it on one of my slaves it reports:

example.com

Which is, indeed, the zone we deleted a few moments ago on the PowerDNS master.

To cater for the possibility that a zone is removed from the master DNS server and it is added at a later time (yes, that happens), I have a second trigger which cleans out the record of a previously “deleted” domain from the metazone:

DELIMITER $$$

DROP TRIGGER IF EXISTS domains_add;

CREATE TRIGGER `domains_add` AFTER INSERT ON `domains`
FOR EACH ROW BEGIN

    -- Remove a possibly previously 'deleted' zone by deleting its
    -- record from the "metazone" zone.

    SET @metazone = 'meta.meta';
    SET @d_name = NEW.name;

    DELETE FROM records
        WHERE domain_id = (SELECT id FROM domains WHERE name = @metazone)
             AND name = CONCAT(@d_name, '.', @metazone);

    -- Update SOA record in metazone.

    UPDATE records
        SET content = CONCAT(@metazone, ' jp.meta ', UNIX_TIMESTAMP(NOW()), ' 10800 3600 3600 3600')
        WHERE domain_id = (SELECT id FROM domains WHERE name = @metazone)
              AND type = 'SOA';

END $$$
DELIMITER ;

I think the only thing missing is a periodic cleanup of the meta.meta zone on the hidden master; a trivial task.