Servers

Table of contents
  1. Standards
  2. Network Engineering
  3. Cell Site Configuration
  4. Servers

Install basic tools

~~~~~~~~~~
sudo apt-get install mtr sudo tcpdump
~~~~~~~~~~

Participation in OSPF (Option 1, using Quagga)

  1. Install Quagga (FRR is similar as it is a drived distribution)

    sudo apt-get install quagga

  2. Create configuration files

    sudo touch /etc/quagga/zebra.conf /etc/quagga/ospfd.conf /var/log/quagga/zebra.log /var/log/quagga/ospfd.log
sudo chown quagga:quagga /etc/quagga/{zebra,ospfd}.conf
sudo chown quagga:quagga /var/log/quagga/{zebra,ospfd}.log
sudo chmod o-r /etc/quagga/{zebra,ospfd}.conf
sudo chmod o-r /var/log/quagga/{zebra,ospfd}.log

  3. Edit /etc/quagga/daemons and enable zebra and ospfd, optionally enable and configure ospf6d if your network has IPv6 connectivity

  4. Populate the config files

    sudo cat > /etc/quagga/daemons <<DONE
zebra=yes
bgpd=no
ospfd=yes
ospf6d=yes
ripd=no
ripngd=no
isisd=no
babeld=no
DONE

sudo cat > /etc/quagga/zebra.conf <<DONE
password <CONNECT PASSWORD>
enable password <ENABLE PASSWORD>
log file /var/log/quagga/zebra.log
DONE

sudo cat > /etc/quagga/ospfd.conf <<DONE
password <CONNECT PASSWORD>
enable password <ENABLE PASSWORD>
log file /var/log/quagga/ospfd.conf

interface <INTERFACE>
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 <OSPF PASSWORD>
 ip ospf priority 10

router ospf
 ospf router-id <PRIMARY SERVER IP>
 redistribute connected
 distribute-list AMPR out connected
 network <LAN NETWORK ADDRESS/BITMASK> area 0.0.0.0
 network <ANYCAST NETWORK/BITMASK> area 0.0.0.0
 area 0 authentication message-digest

access-list AMPR permit 44.0.0.0/9
access-list AMPR permit 44.128.0.0/10
DONE

sudo cat > /etc/quagga/ospf6d.conf <<DONE
password <CONNECT PASSWORD>
enable password <ENABLE PASSWORD>
log file /var/log/quagga/ospf6d.log

interface eth0
 ipv6 ospf6 priority 10

interface lo

router ospf6
 router-id <PRIMARY IPv4 SERVER IP>
 redistribute connected
 interface eth0 area 0.0.0.0
 interface lo area 0.0.0.0
 area 0.0.0.0 range <IPv6 ANYCAST SUBNET 1>
 area 0.0.0.0 range <IPv6 ANYCAST SUBNET 2>
DONE

    (Note that for Ubuntu you’ll instead need to manually edit the files rather than using cat > syntax)

    (Note that in this case, the , , and are the same; they correspond with what Mikrotik calls the OSPF Interface’s Authentication Key)

    (Your anycast network’s bitmask is almost certainly 23)

  5. Start Quagga

    sudo service quagga start

  6. Verify OSPF is working

    ip r | wc -l # Should show a large number of routes, > 300 at present.

  7. Remove previous static default route

    sudo ip r | grep default # If it contains the word "zebra", do not remove it
sudo ip r del default # Be sure you have OOB control first since this can disconnect you
sudo ip r | grep default # Should now have a default route from zebra
sudo vim /etc/network/interfaces # Remove any gateway statement if this was a static config

  8. OPTIONAL: Verify everything will work from a cold boot

    ifdown eth0 ; ifup eth0 # Be sure you have OOB control first since this can disconnect you

  9. OPTIONAL: Verify you can control zebra + ospfd

    telnet localhost 2601 # For zebra
telnet localhost 2604 # For ospfd
telnet 1 2606 # For ospf6d

Participation in OSPF (Option 2, using FRR)

FRR is a fork of the previous Quagga router and is used on more modern distributions.

  1. Install FRR

    sudo apt-get install frr

  2. Create configuration files

    sudo touch /var/log/frr/frr.log
sudo chown frr:frr /var/log/frr/frr.log
sudo chmod o-r /var/log/frr/frr.log

  3. Populate the config files (ospf6 not tested). We enable ospfd and optionally ospf6d.

    sudo sed -i -e 's/ospfd=no/ospfd=yes/' /etc/frr/daemons
sudo sed -i -e 's/ospf6d=no/ospf6d=yes/' /etc/frr/daemons

sudo cat >> /etc/frr/frr.conf <<DONE
password <CONNECT PASSWORD>
enable password <ENABLE PASSWORD>
log file /var/log/frr/frr.log

interface <INTERFACE>
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 <OSPF PASSWORD>
 ip ospf priority 10
 ipv6 ospf6 priority 10

interface lo

router ospf
 ospf router-id <PRIMARY SERVER IP>
 redistribute connected
 distribute-list AMPR out connected
 network <LAN NETWORK ADDRESS/BITMASK> area 0.0.0.0
 network <ANYCAST NETWORK/BITMASK> area 0.0.0.0
 area 0 authentication message-digest

router ospf6
 router-id <PRIMARY IPv4 SERVER IP>
 redistribute connected
 interface eth0 area 0.0.0.0
 interface lo area 0.0.0.0
 area 0.0.0.0 range <IPv6 ANYCAST SUBNET 1>
 area 0.0.0.0 range <IPv6 ANYCAST SUBNET 2>

access-list AMPR permit 44.0.0.0/9
access-list AMPR permit 44.128.0.0/10
DONE

  4. Start FRR

    sudo systemctl restart frr

  5. Verify OSPF is working

    ip r | wc -l # Should show a large number of routes, > 300 at present.

  6. Remove previous static default route

    sudo ip r | grep default # Determine if its static or dynamic from zebra. If dynamic, do not remove.
sudo ip r del default # Be sure you have OOB control first since this can disconnect you
sudo ip r | grep default # Should now have a default route from zebra
sudo vim /etc/network/interfaces # Convert to static config with no gateway statement.

  7. OPTIONAL: Verify everything will work from a cold boot

    ifdown eth0 ; ifup eth0 # Be sure you have OOB control first since this can disconnect you

  8. OPTIONAL: Verify you can control zebra + ospfd

    telnet localhost 2601 # For zebra
telnet localhost 2604 # For ospfd
telnet 1 2606 # For ospf6d

Recursive DNS Anycast Service

  1. Install unbound

    sudo apt-get install unbound

  2. Stop and disable unbound

    sudo service unbound stop
sudo update-rc.d unbound disable

  3. Configure unbound

    sudo cat > /etc/unbound/unbound.conf.d/hamwan.conf <<DONE
server:
    # The following line will configure unbound to perform cryptographic
    # DNSSEC validation using the root trust anchor.
    auto-trust-anchor-file: "/var/lib/unbound/root.key"
    interface: <ANYCAST IP 1>  # e.g. 44.24.244.1
    interface: <ANYCAST IP 2>
    interface: <ANYCAST IP 3>
    interface: <ANYCAST IP 4>
    do-ip6: no
    # interface: <IPv6 ANYCAST IP 1>  # e.g. 2604:5000:20:1::1
    # interface: <IPv6 ANYCAST IP 2>
    access-control: 44.0.0.0/9 allow
    access-control: 44.128.0.0/10 allow
    # access-control: <IPv6 ALLOCATION> allow  # e.g. 2604:5000:20::/48 allow
    outgoing-interface: <PRIMARY SERVER IP>  # e.g. 44.25.16.10
    rrset-roundrobin: yes
    logfile: /var/log/unbound.log
    log-time-ascii: yes
    val-permissive-mode: yes
    local-zone: "10.in-addr.arpa." nodefault

stub-zone:
    name: "hamwan.net"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "44.10.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "240.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "241.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "242.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "243.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "244.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "245.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "246.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "247.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "248.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "249.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "250.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "251.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "252.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "253.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "254.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "255.24.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no

stub-zone:
    name: "25.44.in-addr.arpa"
    stub-addr: 44.24.244.2
    stub-addr: 44.24.245.2
    stub-addr: 44.25.0.2
    stub-addr: 44.25.1.2
    stub-prime: no
    stub-first: no
DONE

  4. Update apparmor to allow unbound to log

    cat >> /etc/apparmor.d/local/usr.sbin/unbound <<DONE
# Site-specific additions and overrides for usr.sbin.unbound.
# For more details, please see /etc/apparmor.d/local/README.
/var/log/unbound.log rw,
DONE
apparmor_parser -r /etc/apparmor.d/usr.sbin.unbound

  5. Configure recursive DNS IPv4 anycast interface

    If using Quagga on older systems:

    sudo cat >> /etc/network/interfaces <<DONE

auto any-dns-rr
iface any-dns-rr inet manual
        pre-up ip tuntap add dev any-dns-rr mode tap
        post-up ip a add <ANYCAST IP 1>/32 dev any-dns-rr
        post-up ip a add <ANYCAST IP 2>/32 dev any-dns-rr
        post-up ip l set dev any-dns-rr up
        post-up service unbound start
        post-down service unbound stop
        post-down ip tuntap del dev any-dns-rr mode tap
DONE

    If using FRR on newer systems, check first to see if there is already a loopback (lo) interface specification and if so, just add the anycast addresses (this may work for Quagga system too):

    sudo cat >> /etc/network/interfaces

auto lo
iface lo inet loopback
        post-up ip a add <ANYCAST IP 1>/32 dev lo
        post-up ip a add <ANYCAST IP 2>/32 dev lo
<CTRL-D>

  6. OPTIONALLY: Configure recursive DNS IPv6 anycast interface

    auto lo
iface lo inet loopback
        post-up ip -6 a add <IPv6 ANYCAST IP 1>/128 dev lo
        post-up ip -6 a add <IPv6 ANYCAST IP 2>/128 dev lo

  7. Start the recursive DNS resolver service

    sudo ifup lo
sudo ifup any-dns-rr  # if using quagga
systemctl enable unbound
systemctl start unbound

  8. Verify functionality

    ip a # Should see the any-dns-rr interface with the two anycast IPs as well as the IPv6 IPs on the lo interface
dig @<ANYCAST IP 1> google.com. A # Should return google's IPs
dig @<IPv6 ANYCAST IP 1> google.com. A # Should return google's IPs

  9. Verify the service is being advertised to OSPF

    ssh <NEAREST OSPF ROUTER>
/ip route check <ANYCAST IP 1> # Should display nearest server's primary IP as nexthop
/ipv6 route check <IPv6 ANYCAST IP 1> # Should display Link Local address of nearest server's ethernet interface as nexthop

Authoritative DNS Anycast Service

Authoritative DNS is used to place names for the hostnames and the necessary PTR records for reverse-dns. It is comprised of a PowerDNS install utilizing PostgreSQL. You will amostly certainly want to install the HamWAN Management Portal alongside this.

  1. Install necessary software

    sudo apt-get install postgresql postgresql-contrib postgresql-client pdns-server pdns-backend-pgsql

  2. Setup the anycast IPs

    Similar to above add anycast addresses for authoritative DNS service. For older systems:

    sudo cat >> /etc/network/interfaces <<DONE

auto any-adns
iface any-adns inet manual
        pre-up ip tuntap add dev any-adns mode tap
        pre-up ip l set dev any-adns mtu 1418
        post-up ip a add <ANYCAST IP 1>/32 dev any-adns
        post-up ip a add <ANYCAST IP 2>/32 dev any-adns
        post-up ip l set dev any-adns up
        post-up service pdns restart
        post-down ip tuntap del dev any-adns mode tap
DONE

    and for newer systems, just add the address to the loopback interface:

    iface lo inet loopback
        ...
        post-up ip a add <ANYCAST IP 1>/32 dev lo
        post-up ip a add <ANYCAST IP 2>/32 dev lo

  3. Make sure postgres can handle our connection properly

    Where VERSION is the installed postgresql version (you will need to check first)

    sudo vi /etc/postgresql/VERSION/main/pg_hba.conf

# in this file, make sure that the following line is present (most likely you'll change auth from "peer" to "md5")

local   all             all                                     md5

  4. Update the pdns config to point to the postgres db; make sure the following are set in /etc/powerdns/pdns.d/pdns.local.gpgsql.conf. Make sure there aren’t any include-dir statements. Remove bind.conf. We don’t need it.

    rm /etc/powerdns/pdns.d/bind.conf

cat > /etc/powerdns/pdns.d/pdns.local.gpgsql.conf <<DONE
launch=gpgsql
gpgsql-host=
gpgsql-port=
gpgsql-user=powerdns
gpgsql-password=<DB PW>
gpgsql-dbname=powerdns
gpgsql-dnssec=yes
local-address=<YOUR ANYCAST AUTHORITATIVE NS IPS SEPARATED BY COMMA>
DONE

cat > /etc/powerdns/pdns.d/allow-44-axfr.conf <<DONE
allow-axfr-ips=44.0.0.0/8
disable-axfr=no
DONE

  5. Setup the DB if this is going to be a master. If this is going to be a slave replica, go replica setup below.

    sudo su postgres ; change to the postgres user
psql ; enter the postgres prompt
CREATE USER powerdns WITH PASSWORD '<DB PW>';
CREATE DATABASE powerdns;
\q

  6. Create the PDNS DB Scema

    psql -d powerdns

CREATE TABLE domains (
  id                    SERIAL PRIMARY KEY,
  name                  VARCHAR(255) NOT NULL,
  master                VARCHAR(128) DEFAULT NULL,
  last_check            INT DEFAULT NULL,
  type                  VARCHAR(6) NOT NULL,
  notified_serial       INT DEFAULT NULL,
  account               VARCHAR(40) DEFAULT NULL,
  CONSTRAINT c_lowercase_name CHECK (name = LOWER(name))
);
CREATE UNIQUE INDEX name_index ON domains(name);


CREATE TABLE records (
  id                    SERIAL PRIMARY KEY,
  domain_id             INT DEFAULT NULL,
  name                  VARCHAR(255) DEFAULT NULL,
  type                  VARCHAR(10) DEFAULT NULL,
  content               VARCHAR(65535) DEFAULT NULL,
  ttl                   INT DEFAULT NULL,
  prio                  INT DEFAULT NULL,
  change_date           INT DEFAULT NULL,
  disabled              BOOL DEFAULT 'f',
  ordername             VARCHAR(255),
  auth                  BOOL DEFAULT 't',
  CONSTRAINT domain_exists
  FOREIGN KEY(domain_id) REFERENCES domains(id)
  ON DELETE CASCADE,
  CONSTRAINT c_lowercase_name CHECK (name = LOWER(name))
);

CREATE INDEX rec_name_index ON records(name);
CREATE INDEX nametype_index ON records(name,type);
CREATE INDEX domain_id ON records(domain_id);
CREATE INDEX recordorder ON records (domain_id, ordername text_pattern_ops);


CREATE TABLE supermasters (
  ip                    INET NOT NULL,
  nameserver            VARCHAR(255) NOT NULL,
  account               VARCHAR(40) DEFAULT NULL,
  PRIMARY KEY(ip, nameserver)
);


CREATE TABLE comments (
  id                    SERIAL PRIMARY KEY,
  domain_id             INT NOT NULL,
  name                  VARCHAR(255) NOT NULL,
  type                  VARCHAR(10) NOT NULL,
  modified_at           INT NOT NULL,
  account               VARCHAR(40) DEFAULT NULL,
  comment               VARCHAR(65535) NOT NULL,
  CONSTRAINT domain_exists
  FOREIGN KEY(domain_id) REFERENCES domains(id)
  ON DELETE CASCADE,
  CONSTRAINT c_lowercase_name CHECK (name = LOWER(name))
);

CREATE INDEX comments_domain_id_idx ON comments (domain_id);
CREATE INDEX comments_name_type_idx ON comments (name, type);
CREATE INDEX comments_order_idx ON comments (domain_id, modified_at);


CREATE TABLE domainmetadata (
  id                    SERIAL PRIMARY KEY,
  domain_id             INT REFERENCES domains(id) ON DELETE CASCADE,
  kind                  VARCHAR(32),
  content               TEXT
);

CREATE INDEX domainidmetaindex ON domainmetadata(domain_id);


CREATE TABLE cryptokeys (
  id                    SERIAL PRIMARY KEY,
  domain_id             INT REFERENCES domains(id) ON DELETE CASCADE,
  flags                 INT NOT NULL,
  active                BOOL,
  content               TEXT
);

CREATE INDEX domainidindex ON cryptokeys(domain_id);


CREATE TABLE tsigkeys (
  id                    SERIAL PRIMARY KEY,
  name                  VARCHAR(255),
  algorithm             VARCHAR(50),
  secret                VARCHAR(255),
  CONSTRAINT c_lowercase_name CHECK (name = LOWER(name))
);

CREATE UNIQUE INDEX namealgoindex ON tsigkeys(name, algorithm);
GRANT SELECT ON supermasters TO powerdns;
GRANT ALL ON tsigkeys TO powerdns;
GRANT ALL ON cryptokeys TO powerdns;
GRANT ALL ON domainmetadata TO powerdns;
GRANT ALL ON comments TO powerdns;
GRANT ALL ON records TO powerdns;
GRANT ALL ON domains TO powerdns;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO powerdns;
\q

  7. Restart the deamons

    exit; go back to your normal user
sudo service postgresql restart
# if using any-adns pseudo interface...
sudo ifup any-adns

  8. Set up database replication if this is a slave server, starting with the master. References: - https://www.postgresql.org/docs/15/warm-standby.html#STREAMING-REPLICATION - https://www.postgresql.fastware.com/postgresql-insider-ha

    # update pg_hba.conf to add a replication user
# configure replication parameters in postgresql.conf

  9. Configure the slave postgresql server by editing postgresql.conf References: - https://www.postgresql.org/docs/15/warm-standby.html#STREAMING-REPLICATION - https://www.postgresql.fastware.com/postgresql-insider-ha

    systemctl postgresql stop
# The standby connects to the primary that is running on host 192.168.1.50
# and port 5432 as the user "foo" whose password is "foopass".
primary_conninfo = 'host=192.168.1.50 port=5432 user=foo password=foopass'
# ensure database location is missing or empty, then fetch a current copy of the database
# run pg_basebackup ... ('-X none' because of copy from old system)
pg_basebackup -D /var/lib/postgresql/9.1/main -X none -v -h SRV1.westin.hamwan.net -U ziply_copy -W
pg_upgradecluster

  10. Test it!

    sudo service pdns stop #stop the service for testing
sudo /etc/init.d/pdns monitor # you shouldn't see any errors during the startup;

# go back to normal

sudo service pdns start

HamWAN DNS Portal

  1. Install software

    sudo apt-get update #make sure we're up to date
#install the webserver, python, and the python libs we need
sudo apt-get install nginx python3-certbot-nginx python3-pip python3-virtualenv python3-dev git libpq-dev postgresql postfix
#we use this user to run uwsgi server; use a good password!!
sudo adduser portal

# Time to download our custom stuff.  /var/www should already exist.

cd /var/www
sudo git clone https://github.com/HamWAN/dns-portal.git
sudo git clone https://github.com/HamWAN/django-ssl-client-auth.git
sudo chown portal:portal -R dns-portal django-ssl-client-auth

sudo -s -u portal

    Now, as the portal user…

    cd dns-portal
# make a symlink for the ssl auth stuff
ln -s /var/www/django-ssl-client-auth/django_ssl_auth .

# Load addtional python modules in a virtual environment
virtualenv env  #setup our virtual environment
source env/bin/activate

# We should now be in the virtual environment
# Install server and dependencies via pip instead of apt-get to get the latest and keep contained

pip3 install -r requirements.txt
pip3 install uwsgi south django-debug-toolbar # currently missing from requirements.txt
exit

  2. Done with this part! On to the database… You can’t cut/paste this block. Work in chunks. You need to be mindful of the changing input contexts.

    sudo -i -u postgres
psql
CREATE USER portal WITH PASSWORD '<your-portal-user-pw-here>';
CREATE DATABASE portal;
\q
psql -d portal
GRANT ALL PRIVILEGES ON SCHEMA public TO portal;
CREATE OR REPLACE FUNCTION array_reverse(anyarray) RETURNS anyarray AS $$
SELECT ARRAY(
    SELECT $1[i]
    FROM generate_subscripts($1,1) AS s(i)
    ORDER BY i DESC
);
$$ LANGUAGE 'sql' STRICT IMMUTABLE;
\q
exit

  3. Now we need to configure settings.py to point to our database

    sudo -s -u portal
cd dns-portal
source env/bin/activate #get back in our virtual environment

    Now we need to edit our config/settings.py file; this is a bit complicated, but use the following as a template:

    # Django settings for this portal implementation project.

# TODO(implementer): Reset to False for Production
DEBUG = True
TEMPLATE_DEBUG = DEBUG
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'

ROOT_DOMAIN = 'hamwan.net'
DEFAULT_NETWORK = '44.25.0.0/16'

AUTH_USER_MODEL = 'auth.User'

ADMINS = (
    ('Tom Hayward', 'tom@tomh.us'),
    ('Doug Kingston', 'dpk@randomnotes.org'),
)

MANAGERS = ADMINS

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': 'portal',                      # Or path to database file if using sqlite3.
        # The following settings are not used with sqlite3:
        'USER': 'portal',
        'PASSWORD': 'secure-password-here',                  # Using uid auth, you won't need a password
        'HOST': '',                      # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP.
        'PORT': '',                      # Set to empty string for default.
    },
    'pdns': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': 'powerdns',                      # Or path to database file if using sqlite3.
        'USER': 'pdns',
        'PASSWORD': 'another-secure-password',
        'HOST': '',                      # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP.
        'PORT': '',                      # Set to empty string for default.
    },
}

# Uncomment in prod where pdns is a separate database
DATABASE_ROUTERS = ['config.dbrouter.DnsRouter', ]

# Hosts/domain names that are valid for this site; required if DEBUG is False
# See https://docs.djangoproject.com/en/1.5/ref/settings/#allowed-hosts
ALLOWED_HOSTS = ['portal.hamwan.org', 'encrypted.hamwan.org',
                 'portal.hamwan.net', 'encrypted.hamwan.net',
                 'portal.ziply.hamwan.net', 'srv2.ziply.hamwan.net',]

# Local time zone for this installation. Choices can be found here:
# http://en.wikipedia.org/wiki/List_of_tz_zones_by_name
# although not all choices may be available on all operating systems.
# In a Windows environment this must be set to your system time zone.
TIME_ZONE = 'America/Los_Angeles'

# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'en-us'

# Confirm this setting at your in your Sites table. The number here
# needs to match the index of your site in the Sites table.
SITE_ID = 2

# If you set this to False, Django will make some optimizations so as not
# to load the internationalization machinery.
USE_I18N = True

# If you set this to False, Django will not format dates, numbers and
# calendars according to the current locale.
USE_L10N = True

# If you set this to False, Django will not use timezone-aware datetimes.
USE_TZ = True

# Absolute filesystem path to the directory that will hold user-uploaded files.
# Example: "/var/www/example.com/media/"
MEDIA_ROOT = ''

# URL that handles the media served from MEDIA_ROOT. Make sure to use a
# trailing slash.
# Examples: "http://example.com/media/", "http://media.example.com/"
MEDIA_URL = ''

# Absolute path to the directory static files should be collected to.
# Don't put anything in this directory yourself; store your static files
# in apps' "static/" subdirectories and in STATICFILES_DIRS.
# Example: "/var/www/example.com/static/"
STATIC_ROOT = '/var/www/hamwan-portal/static/'

# URL prefix for static files.
# Example: "http://example.com/static/", "http://static.example.com/"
STATIC_URL = '/static/'

# Additional locations of static files
STATICFILES_DIRS = (
    # Put strings here, like "/home/html/static" or "C:/www/django/static".
    # Always use forward slashes, even on Windows.
    # Don't forget to use absolute paths, not relative paths.
    "/var/www/hamwan-portal/css",
    "/var/www/hamwan-portal/config/css",
)

# List of finder classes that know how to find static files in
# various locations.
STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
#    'django.contrib.staticfiles.finders.DefaultStorageFinder',
)

# Make this unique, and don't share it with anybody.
SECRET_KEY = 'your-strong-unique-key-here'

# List of callables that know how to import templates from various sources.
TEMPLATE_LOADERS = (
    'django.template.loaders.filesystem.Loader',
    'django.template.loaders.app_directories.Loader',
#     'django.template.loaders.eggs.Loader',
)

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [
            # insert your TEMPLATE_DIRS here
            '/var/www/hamwan-portal/templates',
        ],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                # Insert your TEMPLATE_CONTEXT_PROCESSORS here or use this
                # list if you haven't customized them:
                'django.contrib.auth.context_processors.auth',
                'django.template.context_processors.debug',
                'django.template.context_processors.i18n',
                'django.template.context_processors.media',
                'django.template.context_processors.static',
                'django.template.context_processors.tz',
                'django.contrib.messages.context_processors.messages',
                # HamWAN portal additions
                'django.template.context_processors.request',
                'portal.context_processors.encrypted44',
            ],
        },
    },
]

MIDDLEWARE = (
    'django.middleware.common.CommonMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.auth.middleware.RemoteUserMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    # Uncomment the next line for simple clickjacking protection:
    # 'django.middleware.clickjacking.XFrameOptionsMiddleware',
)

ROOT_URLCONF = 'config.urls'

# Python dotted path to the WSGI application used by Django's runserver.
WSGI_APPLICATION = 'config.wsgi.application'

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django.contrib.admin',
    'django.contrib.flatpages',
    #disabled#configuration,
    #disabled#api
    'dns',
    'portal',
    # 'map',
    #disabled#rest_framework
    'utils',
    # Uncomment the next line to enable the request and query debugging tool
    # 'debug_toolbar',
    # Uncomment the next line to enable admin documentation:
    # 'django.contrib.admindocs',
)

SESSION_SERIALIZER = 'django.contrib.sessions.serializers.JSONSerializer'

# A sample logging configuration. The only tangible logging
# performed by this configuration is to send an email to
# the site admins on every HTTP 500 error when DEBUG=False.
# See http://docs.djangoproject.com/en/dev/topics/logging for
# more details on how to customize your logging configuration.
PRODUCTION_LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        }
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
    }
}

DEBUG_LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format' : "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
            'datefmt' : "%d/%b/%Y %H:%M:%S"
        },
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
    },
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': '/tmp/mysite.log',
            'formatter': 'verbose'
        },
    },
    'loggers': {
        'django': {
            'handlers':['file'],
            'propagate': True,
            'level':'DEBUG',
        },
        'MYAPP': {
            'handlers': ['file'],
            'level': 'DEBUG',
        },
    }
}

LOGGING=PRODUCTION_LOGGING

REST_FRAMEWORK = {
    # Use Django's standard `django.contrib.auth` permissions,
    # or allow read-only access for unauthenticated users.
    'DEFAULT_PERMISSION_CLASSES': [
        'rest_framework.permissions.DjangoModelPermissionsOrAnonReadOnly'
    ]
}

  4. Use the django framework to setup the database.

    # It may prompt you to make a super user, do it! Use the same email as you put in the config file for your admin user

# This creates the schemas
./manage.py migrate

# manually change portal_ipaddress.ip to type "inet"

sudo su postgres
psql -d portal
ALTER TABLE portal_ipaddress ALTER COLUMN ip TYPE inet USING(ipinet);
\q
exit

    Next you need to fix the indexing. It will look somethiing like this. You need to find the index on ‘ip varchar_patter_ops’.

    portal=# select indexname, indexdef from pg_indexes where tablename = 'portal_ipaddress';
             indexname             |                                                    indexdef
-----------------------------------+----------------------------------------------------------------------------------------------------------------
 portal_ipaddress_pkey             | CREATE UNIQUE INDEX portal_ipaddress_pkey ON public.portal_ipaddress USING btree (id)
 portal_ipaddress_ip_key           | CREATE UNIQUE INDEX portal_ipaddress_ip_key ON public.portal_ipaddress USING btree (ip)
 portal_ipaddress_ip_0d291e7e_like | CREATE INDEX portal_ipaddress_ip_0d291e7e_like ON public.portal_ipaddress USING btree (ip varchar_pattern_ops)
 portal_ipaddress_host_id_0bfe4212 | CREATE INDEX portal_ipaddress_host_id_0bfe4212 ON public.portal_ipaddress USING btree (host_id)
(4 rows)

portal=# drop index if exists portal_ipaddress_ip_0d291e7e_like;
DROP INDEX
portal=# ALTER TABLE portal_ipaddress ALTER COLUMN ip TYPE inet USING (ip::inet);
ALTER TABLE

    Test it out! Open your ip:8000 to verify that the install worked. If so, let’s move on to setting up uwsgi and nginx

    ./manage.py runserver 0.0.0.0:8000

  5. Set up an inital superuser on the portal.

    You will need this to login to an initially empty portal instance. If you are importing an existing portal database, then this step can be skipped.

    ./manage.py createsuperuser
./manage.py runserver 0.0.0.0:8000
# login and see that is functions as expected

  6. Now we setup the webserver component.

    sudo vi /etc/nginx/nginx.conf

    You’ll almost certainly need to uncomment the following line:

    server_names_hash_bucket_size 64;

    And save the file (wq! if using vi). Now to move the uwsgi/nginx configs in place. Either install either old style init script or systemd config:

    cd /var/www/dns-portal
sudo cp ./deploy/uwsgi.conf /etc/init/

    or for systems using systemd:

    cd /var/www/dns-portal
sudo cp ./deploy/emperor.uwsgi.service /etc/systemd/system/emperor.uwsgi.service

  7. Set up SSL certificates

    This can either be with LetEncrypt (certbot) or using certs signed by another certificate provider. If using LetsEncrypt’s certbot (encouraged), just follow their instructions being sure to list all the domains you are likely to be accessed as. Here are the instructions for creating a certificate signing request and adding the resulting certificate to the system. Certbot will do this automatically for nginx:

    sudo mkdir /etc/nginx/ssl
sudo openssl req -new -newkey rsa:2048 -nodes -keyout /etc/nginx/ssl/dns-portal.key -out /etc/nginx/ssl/dns-portal.csr #Generate a CSR for your https

# Fill in the details!

sudo view /etc/nginx/ssl/dns-portal.csr # Submit your CSR to get the cert signed for web ssl; startssl is free!
sudo vi /etc/nginx/ssl/dns-portal.crt #Put your signed certificate (which you probably got from startssl) in here

    Next, set up the logging and start the services. uwsgi will run as user pdns and group portal. Its createa a socket in /var/www/dns-portal, so we need to adjust the permissions according.

    # setup uwsgi
chmod 775 /var/www/dns-portal
sudo mkdir -p /var/log/uwsgi
sudo touch /var/log/uwsgi/portal.log
sudo chown portal:portal /var/log/uwsgi/portal.log
sudo systemctl enable emperor.uwsgi
sudo systemctl start emperor.uwsgi
# and nginx
sudo cp ./deploy/portal_nginx.conf /etc/nginx/sites-available/portal.conf
sudo ln -s /etc/nginx/sites-available/portal.conf /etc/nginx/sites-enabled/
sudo service nginx restart

  8. Backup

    One potential scheme is to use crontab to make a backup and manage the files. Also consider database replication which you may choose to do anyways to enable additional authoritative DNS servers.

    # m h  dom mon dow   command
# hourly pg backup
29 * * * * /usr/bin/sudo -u postgres /usr/bin/pg_dumpall | /bin/gzip > /home/tom/pgbackup/pg_dump_`/bin/date +\%Y-\%m-\%d_\%H\%M\%S`.sql.gz
# retain 1 monthly pg backup after 60 days
32 2 * * * find /home/tom/pgbackup/ -name 'pg_dump_????-??-01_00*' -prune -o -mtime +60 -exec rm {} \;
# retain 1 daily pg backup after 10 days
33 2 * * * find /home/tom/pgbackup/ -name 'pg_dump_????-??-??_00*' -prune -o -mtime +10 -exec rm {} \;

Postgresql database replication

  1. Set up a secondary server with powerdns and optionally the portal code

    Its important to not add the anycast address for powerdns until after the replication is verified working as intended. Initially, configure powerdns to server from 127.0.0.2 so you have some means of testing. You can also configure this server to run unbound if you choose.

  2. Changes to Primary to enable publishing

    # On Primary as postgres
psql <<DONE
CREATE USER srv1_westin REPLICATION LOGIN ENCRYPTED PASSWORD 'myreplicationpassword';

GRANT CONNECT ON DATABASE powerdns TO srv1_westin;
GRANT CONNECT ON DATABASE portal TO srv1_westin;
DONE

psql -d powerdns <<DONE
GRANT USAGE ON SCHEMA public TO srv1_westin;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO srv1_westin;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO srv1_westin;
CREATE PUBLICATION powerdns_slot FOR ALL TABLES;
SELECT * FROM pg_create_logical_replication_slot('powerdns_slot', 'pgoutput');
DONE

psql -d portal <<DONE
GRANT USAGE ON SCHEMA public TO srv1_westin;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO srv1_westin;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO srv1_westin;
CREATE PUBLICATION portal_slot FOR ALL TABLES;
SELECT * FROM pg_create_logical_replication_slot('portal_slot', 'pgoutput');
DONE

pg_dump -d powerdns --schema-only > powerdns.schema
pg_dump -d portal --schema-only > portal.schema

    Copy the schema files to /etc/postgresql/VERSION/main on the secondary server.

    Update pg_hba.conf to add the replication user identity and access. To add a secondary on SRV1, with address 44.25.16.10, I created the following entry at the end of /etc/postgresql/VERSION/main/pg_hba.conf:

    host    all             srv1_westin     44.25.16.10/32          md5

  3. Changes to Secondary

    Before you can start logical replication by adding subscriptions, you need to initialize the datables. This will be done by loading the schema files you dumped on the Primary.

    # On Secondary as postgres
psql <<DONE
CREATE DATABASE powerdns;
CREATE DATABASE portal;
CREATE USER pdns;
CREATE USER portal;
DONE

psql -d powerdns -f /tmp/powerdns.schema
psql -d portal -f /tmp/portal.schema

psql -d powerdns <<DONE
CREATE SUBSCRIPTION powerdns_sub CONNECTION 'host=srv2.ziply.hamwan.net port=5432 user=srv1_westin dbname=powerdns password=LXot6oyfV798IN5T'
    PUBLICATION powerdns_slot;
DONE

psql -d portal <<DONE
CREATE SUBSCRIPTION portal_sub CONNECTION 'host=srv2.ziply.hamwan.net port=5432 user=srv1_westin dbname=portal password=LXot6oyfV798IN5T'
    PUBLICATION portal_slot;
DONE

NTP Anycast Service (V1)

  1. Assumptions

    This is for an older system with quagga based OSPF.

  2. Install ntp

    sudo apt-get install ntp

  3. Configure ntp

    sudo cat >> /etc/ntp.conf
driftfile /var/lib/ntp/ntp.drift

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable

# time.k7nvh.hamwan.net
server 44.24.255.3 iburst prefer
# time02.k7nvh.hamwan.net
server 44.24.255.5 prefer
# ntp.snodem.hamwan.net
server 44.25.142.128 prefer

server 0.us.pool.ntp.org
server 1.us.pool.ntp.org
server 2.us.pool.ntp.org
server 3.us.pool.ntp.org

restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery

restrict 127.0.0.1
restrict 1
<CTRL-D>

  4. Configure NTP anycast interface

    sudo cat >> /etc/network/interfaces
auto any-ntp
iface any-ntp inet manual
        pre-up ip tuntap add dev any-ntp mode tap
        pre-up ip l set dev any-ntp mtu 1418
        post-up ip a add <ANYCAST IP 1>/32 dev any-ntp
        post-up ip a add <ANYCAST IP 2>/32 dev any-ntp
        post-up ip l set dev any-ntp up
        post-up service ntp restart
        post-down ip tuntap del dev any-ntp mode tap
<CTRL-D>

auto lo
iface lo inet loopback
    post-up ip -6 a add <IPv6 ANYCAST IP 1>/128 dev lo
    post-up ip -6 a add <IPv6 ANYCAST IP 2>/128 dev lo

  5. Start the NTP anycast service

    sudo ifup lo
sudo ifup any-ntp

  6. Verify functionality

    ip a # Should see the any-ntp interface with the two anycast IPs
ntpdate -d <ANYCAST IP 1> # Should see the current ntp date information
ntpdate -d <IPv6 ANYCAST IP 1> # Same as IPv4

  7. Verify the service is being advertised to OSPF

    ssh <NEAREST OSPF ROUTER>
/ip route check <ANYCAST IP 1> # Should display nearest server's primary IP as nexthop
/ipv6 route check <IPv6 ANYCAST IP 1> # Should display Link Local address of local server's ethernet interface

NTP Anycast Service (V2)

  1. Assumptions

    As implemented in May 2023 on a current Debian image using chrony. New install of Debian 11 (bullseye) on a small server (real or VM).

  2. Install ntp and frr routing software

    sudo apt-get install chrony frr

  3. Disable DHCP driven ntp configuration (do not request ntp-servers)

    sudo cat > /etc/dhcp/dhclient.conf <<EOF
option rfc3442-classless-static-routes code 121 = array of unsigned integer 8;

send host-name = gethostname();
request subnet-mask, broadcast-address, time-offset, routers,
    domain-name, domain-name-servers, domain-search, host-name,
    dhcp6.name-servers, dhcp6.domain-search, dhcp6.fqdn, dhcp6.sntp-servers,
    netbios-name-servers, netbios-scope, interface-mtu,
    rfc3442-classless-static-routes;
EOF

  4. Configure frr (OSPF)

    sudo rm /etc/frr/frr.conf

sed -i -e 's/ospfd=no/ospfd=yes/' -e 's/ospf6d=no/ospf6d=yes/' /etc/frr/daemons

cat > /etc/frr/ospfd.conf <<EOF
password PASSWORD
enable password PASSWORD
log file /var/log/frr/ospfd.log

interface ens18
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 ABCDEFABCD
 ip ospf priority 10

router ospf
 ospf router-id 44.25.12.75
 redistribute connected
 distribute-list AMPR out connected
 network 44.25.142.0/24 area 0.0.0.0
 network 44.24.244.0/23 area 0.0.0.0
 network 44.25.0.0/22 area 0.0.0.0
 area 0 authentication message-digest

access-list AMPR permit 44.0.0.0/9
access-list AMPR permit 44.128.0.0/10
EOF

cat > /etc/frr/ospf6d.conf <<EOF
password PASSWORD
enable password PASSWORD
log file /var/log/frr/ospf6d.log

interface ens18
 ipv6 ospf6 priority 10

interface lo

router ospf6
 router-id 44.25.12.75
 redistribute connected
 interface eth0 area 0.0.0.0
 interface lo area 0.0.0.0
 area 0.0.0.0 range 2604:5000:20:1::4/128
 area 0.0.0.0 range 2604:5000:20:2::4/128
EOF

cat > /etc/frr/zebra.conf <<EOF
password PASSWORD
enable password PASSWORD
logfile /var/log/frr/zebra.log
EOF

  5. Configure chrony

    sudo cat > /etc/chrony/chrony.conf <<EOF
# Welcome to the chrony configuration file. See chrony.conf(5) for more
# information about usable directives.

# Include configuration files found in /etc/chrony/conf.d.
confdir /etc/chrony/conf.d

# time.k7nvh.hamwan.net
server 44.24.255.3 prefer
# time02.k7nvh.hamwan.net
server 44.24.255.5 prefer
# ntp.snodem.hamwan.net
server 44.25.142.128 prefer

server 0.us.pool.ntp.org
server 1.us.pool.ntp.org
server 2.us.pool.ntp.org

# Use NTP sources found in /etc/chrony/sources.d.
sourcedir /etc/chrony/sources.d

# This directive specify the location of the file containing ID/key pairs for
# NTP authentication.
keyfile /etc/chrony/chrony.keys

# This directive specify the file into which chronyd will store the rate
# information.
driftfile /var/lib/chrony/chrony.drift

# Uncomment the following line to turn logging on.
log tracking measurements statistics

# Save NTS keys and cookies.
ntsdumpdir /var/lib/chrony

# Log files location.
logdir /var/log/chrony

# Stop bad estimates upsetting machine clock.
maxupdateskew 100.0

# This directive enables kernel synchronisation (every 11 minutes) of the
# real-time clock. Note that it can't be used along with the 'rtcfile' directive.
rtcsync

# Step the system clock instead of slewing it if the adjustment is larger than
# one second, but only in the first three clock updates.
makestep 1 3

# Get TAI-UTC offset and leap seconds from the system tz database.
# This directive must be commented out when using time sources serving
# leap-smeared time.
leapsectz right/UTC

allow 44.0.0.0/9
allow 44.128.0.0/10
EOF

  6. Configure NTP anycast interface addresses to loopback interface

    Add anycast addresses after ‘iface lo inet loopback’ (note indent):

    iface lo inet loopback
        post-up ip a add <ANYCAST IP 1>/32 dev any-ntp
        post-up ip a add <ANYCAST IP 2>/32 dev any-ntp
        post-up ip -6 a add <IPv6 ANYCAST IP 1>/128 dev lo
        post-up ip -6 a add <IPv6 ANYCAST IP 2>/128 dev lo

  7. Restart routing and start the NTP anycast service

    sudo ifdown lo
sudo ifup lo
sudo systemctl restart frr
sudo systemctl start chrony

  8. Verify functionality

    ip a # Should see the 4 anycast addresses on lo interface
sntp <ANYCAST IP 1> # Should see the current ntp date information
sntp <IPv6 ANYCAST IP 1> # Same as IPv4

  9. Verify the service is being advertised to OSPF

    ssh <NEAREST OSPF ROUTER>
/ip route check <ANYCAST IP 1> # Should display nearest server's primary IP as nexthop
/ipv6 route check <IPv6 ANYCAST IP 1> # Should display Link Local address of local server's ethernet interface