Standardised Install Notes

[Ref: FAQ 4 - Installation Guide]

Submit DMESG

When you've completed an installation, support the project by submitting a 'dmesg'

dmesg displays the contents of the system message buffer. It is most commonly used to review system startup messages. It is also very useful to the project to get copies of system messages with each new release.
(dmesg; sysctl hw.sensors) | mail -s "brand,make/model/cpu of machine" dmesg@openbsd.org

or, if the machine does not have e-mail access then save the contents to a file for sending.

Packages

[Ref: 3. Packages

Downloading packages for installation seems to be the biggest cause for time delay during a system build process. But that doesn't mean you need to mirror the full package tree. Once you've completed your trial/test installations, you have a list of packages that are installed in your configuration (@ /var/db/pkg).

Improve deployment speed (reinstallations) by having your required installation packages available locally either on the build CDs or from a networked internal resource (use Apache, it's easier.)

With packages available locally you can significantly decrease your full build/restore process to 15 minutes or less.

Tip: Caching Packages

In true Unix fashion, GUI Environments (such as Gnome and KDE) involve installing a lot of tools, toolkits. Subsequently, installation of a single binary package generally requires a lot of downloads. For those of us with slow connections to the Internet, we either pre-cache (download) all packages or cache downloads so for re-installs and speed deployment on other hosts.

Pre-cached packages are usually made available on our LAN from another machine, and we can cache packages downloaded during the pkgadd process using the environment variable PKGCACHE.

# mkdir /var/pkg/cache
# export PKG_CACHE=/var/pkg/cache
# echo "export PKG_CACHE=/var/pkg/cache" >> ~/.profile

Set the PKG_PATH environment variable pointing to the binary package mirror we will be using. In our environment, we specify:

  1. The cache, and
  2. our local LAN mirror,
  3. a Public repository.
# export PKG_PATH=${PKG_CACHE}/:http://local-repo/pub/OpenBSD/$(uname -r)/packages/$(uname -m)/:ftp://remote-repo/pub/OpenBSD/$(uname -r)/packages/$(uname -m)/
Minimal | Common Configuration

Purposing of hosts differs, so there is no consistent set of tools that are "appropriate" for each system. However, there are some tools that might be appropriate for your maintainance of any given host.

The following a tools that may or may not be useful on your hosts. The goal is minimal functionality that can be audited (i.e. we can at least keep tabs of published vulnerability reports.)

  • gnuwatch - helps with monitoring
  • nmap - helps with diagnosing basic configuration, normally removed after installation testing is completed.
  • pstree - helps visualise process states
  • rsync - required for simplifying large/repetitive file transfers
  • vim no_x11 - Text Editor of choice (choose your own poison)
  • trafshow - helps with monitoring traffic flow

As per previous section, having your tool selection available locally will significantly improve your productivity.

User Accounts

The install process asks whether a new user account should be created.

If you are learning OpenBSD, go ahead and create your user accounts during the install process, guided by the prompts.

The default settings for this account creation, doesn't fit our normal user creation process, which we further discuss in the following pages.

System Configuration

Things to maximise security and maintainability post production rollout.

newsyslog - Trim Log Files

[Ref: newsyslog(8)]

Log files can be your single largest collection of 'growing' files. Make sure you understand, know where it is stored on your system (normally /var/log) and you maintain a vigilant watch over the standard files, and application log files (e.g. Apache in base will normally retain it's logs in /var/www/logs)

Today's large capacity drives have significant storage capacity for most server requirements, but everything is dependent on your log maintenance policies, data and log growth.

Monitor this space.

The below sample works for my lazy approach to scripted scanning of log files (simple cut-off dates 'monthly'.)

Edit: /etc/newsyslog.conf

# logfile_name          owner:group     mode count size when  flags
/var/cron/log           root:wheel      600  12    *    *     Z
/var/log/aculog         uucp:dialer     660  12    *    24    Z
/var/log/authlog        root:wheel      640  12    *    $M1D0 Z
/var/log/daemon                         640  12    *    $M1D0 Z
/var/log/lpd-errs                       640  12    *    $M1D0 Z
/var/log/maillog                        600  12    *    $M1D0 Z
/var/log/messages                       644  12    *    $M1D0 Z
/var/log/secure                         600  12    *    $M1D0 Z
/var/log/wtmp                           644  12    *    $M1D0 ZB
/var/log/xferlog                        640  12    *    $M1D0 Z
/var/log/ppp.log                        640  12    *    $M1D0 Z
/var/log/pflog                          600  12    *    $M1D0 ZB "pkill -HUP -u root -U root -t - -x pflogd"

In the context on some of the servers we continue to manage, we have additional/different entries such as the below:

For OpenBSD 4.0 ~ 4.6

/var/log/pflog                          600  12    *    $M1D0 ZB /var/run/pflogd.pid

For OpenVPN Hosts

/var/log/openvpn/status _openvpn:_openvpn 644 12   *    $M1D0 Z
/var/log/openvpn/access _openvpn:_openvpn 644 12   *    $M1D0 Z

The OpenVPN configuration updates, are related to standard changes we make for log file locations in the OpenVPN Server.

Verify

[ via How to test Newsyslog]

Verify that typing errors do not cause problems with your installation by launching newsyslog in verbose mode, without performing the log trimming.

$ sudo /usr/bin/newsyslog -nvv

inetd.conf

Remove all unnecessary services listed on inetd.conf.

Take a look at the services listed in inetd.conf and if you don't understand what it is doing, look it up. If you don't need it, then disable it. If you need the service, is there another way of securing the service (such as authentication or data encryption.)

  • disable unnecessary ports | services started with inetd , see /etc/inetd.conf
  • secure running services
grep -v "^#" /etc/inetd.conf | grep -v "^;" | grep -v "^$"

On a clean OpenBSD install, the standard available services (to be cleaned) are:

ident           stream  tcp     nowait  _identd /usr/libexec/identd     identd -el
ident           stream  tcp6    nowait  _identd /usr/libexec/identd     identd -el
127.0.0.1:comsat dgram  udp     wait    root    /usr/libexec/comsat     comsat
[::1]:comsat    dgram   udp6    wait    root    /usr/libexec/comsat     comsat
daytime         stream  tcp     nowait  root    internal
daytime         stream  tcp6    nowait  root    internal
time            stream  tcp     nowait  root    internal
time            stream  tcp6    nowait  root    internal

Remove all service settings and only reconfigure services that are specifically to be secured and used. Whether the removal of these services, provides any additional security is another debate for another forum.

In general, choose services that are secureable daemons with good security practises, or set as local access only while another secure transport is used for accessing the server (such as ssh or ssl tunnels.) For example, we often use pop3 secured as below.

$ grep -v "^#" /etc/inetd.conf | grep -v "^;" | grep -v "^$"
127.0.0.1:pop3  stream  tcp     nowait  root    /usr/sbin/popa3d popa3d

In the above example, pop3 is available only from the localhost (by implication a user has to be authenticated onto the host before they can gain access to pop3, which will again authenticate them.)

Restart the inetd daemon by passing a HUP signal.

$ sudo kill -HUP `cat /var/run/inetd.pid`

Audit the above changes using the `grep' sequence of commands as well as using 'nmap' discussed further below.

Verify

[ ]

Verify that the inetd.conf file is configured correctly by manually connecting the the specified services.

For our above configuration, a validation would be:

$ telnet 127.0.0.1 pop3

Ensuring that the responses from the server is 'correct.'

pf.conf - Packet Filter Configuration

OpenBSD 4.6 and later releases enable the firewall by default.

Remember to enable IP Forwarding if you intend to release this box as a firewall.

Refer to the PF FAQ for current details: http://www.openbsd.org/faq/pf/index.html

audit - pfctl basic

Show status summary.

$ sudo /sbin/pfctl -s info | head -1
Status: Enabled for 45 days 13:33:13          Debug: Urgent

Perform a syntax check.

$ sudo /sbin/pfctl -nf /etc/pf.conf

Check any errors that may be displayed.

sshd_config - SSH Daemon

Release installations should:

  • PermitRootLogin=No
  • Use Public Keys, no interactive passwords?

File: /etc/ssh/sshd_config

$ grep -v "^#" /etc/ssh/sshd_config | grep -v "^$"
Protocol 2
PermitRootLogin no
PasswordAuthentication no
Subsystem       sftp    /usr/libexec/sftp-server
audit - sshd_config

There are a number of different things that can go wrong with your sshd configuration that can potentially keep you off your box. It is important to test remote boxes carefully before fully enabling changes to your configuration file.

Check /var/log/authlog for current access behaviour.

If you are seeing "sshd[####]: Accepted password for ..." then your sshd configuration is using PasswordAuthentication and your user-account may have a broken public-key/private-key combination.

Basic syntax configuration changes can be validated with the -t command-line option

-t      Test mode.  Only check the validity of the configuration file and
         sanity of the keys.  This is useful for updating sshd reliably as
         configuration options may change.
$ sudo /usr/sbin/sshd -t

Other changes can have more subtle circumstances. For example, do not disable password authentication until you are sure that the public key authentication is working correctly.

Start sshd with the new configuration on another port with something similar to the below

$ sudo /usr/sbin/sshd -ddd -p 9999

SSHD is now running in the foreground to a user-defined port. Track access behaviour in /var/log/authlog.

and try connecting to it from your workstation using

$ ssh -vvv -p 9999 server-name

You can track

sshd - reload

Force sshd to reload it's configuration by sending it a SIGHUP signal

$ sudo kill -HUP `cat /var/run/sshd.pid`

ssh_config - ssh client config

Determine whether you want Agent Forwarding.

$ grep -i agent /etc/ssh/ssh_config
ForwardAgent yes

sysctl.conf - System Startup Knobs

[Ref: sysctl.conf(5), sysctl(8) ]

Disable unnecessary features.

sysctl.conf(5) contains Kernel State knobs that OpenBSD reads on startup to set these kernel knobs. The kernel knobs can be manipulated at run-time using the sysctl(8)

  • IPv6 = off

If the host is intended as a gateway or firewall, make sure you enable the required features.

File: /etc/sysctl.conf

sudo grep -v "^#" /etc/sysctl.conf
net.inet.ip.forwarding=1        # 1=Permit forwarding (routing) of IPv4 packets
audit - sysctl.conf

The `only' way to validate the syctl.conf changes are functioning is to restart the machine and verify each entry update is correct. For example, if the above net.inet.ip.forwarding entry has been made, then after a restart a successful change will show the following:

$ sudo sysctl net.inet.ip.forwarding
net.inet.ip.forwarding=1

aliases - Mail Aliases

[Ref: Postfix aliases, alias_database]

Configure root mail to go to the designated monitoring team account, in our configuration we use a special user account __monitor :

  • sendmail: /etc/mail/aliases
  • postfix: /etc/postfix/aliases

For example, ensure email for the 'root' account is delivered to the __monitor account by editing the alias file as in the below.

File extract: /etc/postfix/aliases

root:           __monitor

In Postfix, ensure alias_database is configured for your preferred aliases file above:

File extract: /etc/postfix/main.cf

alias_database = hash:/etc/postfix/aliases

Rebuild the alias hash file using newaliases

  • Sendmail: /usr/bin/newaliases
  • Postfix: /usr/local/sbin/newaliases
aliases - audit

Prevalidation (without need for checking through the mail server) would include ensuring that datestamps on the aliases file are prior to the datestamps on the hash file.

$ ls -al /etc/postfix/aliases*
-rw-r--r--  1 root  wheel   8832 Oct 22 14:56 /etc/postfix/aliases
-rw-r--r--  1 root  wheel  65536 Oct 22 15:19 /etc/postfix/aliases.db
$ sudo grep -v "^#" /etc/mail/aliases | grep -v "^$" | grep -v "^_" | grep -v "/dev/null"
MAILER-DAEMON: postmaster
postmaster: root
daemon:     root
ftp-bugs:   root
operator:   root
uucp:       root
www:        root
root:       __monitor
abuse:      root
security:   root

Auditing

Patch Review

[Ref: Errata | Current Changelog

The best way of keeping tabs of security updates to the OpenBSD ecosystem is to follow the mailing lists.

The web interface for reviewing changes, are a smaller subset of reported updates normally available in the web Errata and Current Changelog.

The Errata pages list updates (and where possible, patches) for Stable releases of OpenBSD. The url/filename for errata pages are usually in the form errataXY.html where XY is the major/minor version numbers.

For example:

Changelogs show many of the machine-independent changes between the previous release and the versioned release. This lists further service/application specific updates that may be useful in your environment.

For example:

  • Current Changelog normally has the changes between the latest "stable" release and current code development.
  • plus49.html holds the changes between the OpenBSD 4.8 release and OpenBSD 4.9.
  • plus20.html holds the changes between the OpenBSD 4.8 release and OpenBSD 4.9.

Note, that not all updates are listed on these pages. For a more complete list of changes, you can follow the source code updates at source-changes, or ports-changes mailing list.

nmap scan

External scan for open ports using nmap. There should be a clear rationale to why any specific port should be kept open.

$ sudo nmap -vv -A -P0  -d 10.9.10.100
Starting Nmap 4.53 ( http://insecure.org ) at 2008-06-26 12:48 EST
.--------------- Timing report ---------------
  hostgroups: min 1, max 100000
  rtt-timeouts: init 1000, min 100, max 10000
  max-scan-delay: TCP 1000, UDP 1000
  parallelism: min 0, max 0
  max-retries: 10, host-timeout: 0
.---------------------------------------------
[ ... ]
Host 10.9.10.100 appears to be up ... good.
Interesting ports on 10.9.10.100:
Not shown: 1713 closed ports
Reason: 1713 resets
PORT   STATE SERVICE REASON  VERSION
22/tcp open  ssh     syn-ack OpenSSH 4.8 (protocol 2.0)
MAC Address: 00:1D:09:F1:50:12 (Dell)
Device type: general purpose
Running: OpenBSD 3.X|4.X
OS details: OpenBSD 3.9 - 4.2
OS Fingerprint:
[ ... ]

Uptime: 0.000 days (since Thu Jun 26 12:48:52 2008)
Network Distance: 1 hop
TCP Sequence Prediction: Difficulty=256 (Good luck!)
IP ID Sequence Generation: Randomized
Final times for host: srtt: 281 rttvar: 290  to: 100000

Package Checklist

$ pkg_info
bzip2-1.0.4         block-sorting file compressor, unencumbered
colorls-4.2         ls that can use color to display file attributes
curl-7.17.1         get files from FTP, Gopher, HTTP or HTTPS servers
gettext-0.16.1      GNU gettext
gnupg-1.4.8         GNU privacy guard - a free PGP replacement
gnuwatch-3.2.7      GNU watch command
libdnet-1.10p2      portable low-level networking library
libiconv-1.9.2p5    character set conversion library
libidn-1.1          internationalized string handling
lua-5.1.2p1         powerful, light-weight programming language
lzo-1.08p1          portable speedy lossless data compression library
multitail-5.0.5     multi-window tail(1) utility
mutt-1.5.17p0       tty-based e-mail client, development version
nmap-4.53           scan ports and fingerprint stack of network hosts
pcre-7.6            perl-compatible regular expression library
pstree-2.27         list processes as a tree
python-2.5.2        interpreted object-oriented programming language
qdbm-1.8.77         high performance embedded database library
rsync-2.6.9         mirroring/synchronization over low bandwidth links
screen-4.0.3p1-static multi-screen window manager
vim-7.1.244p0-no_x11 vi clone, many additional features
wget-1.10.2p1       retrieve files from the web via HTTP, HTTPS and FTP
zsh-4.3.5           Z shell, Bourne shell-compatible

Maintenance

The following are really of no value outside of how I'm doing things, but I'm putting it together here, because it simplifies my list of things to review (this one document)

A basic host maintenance configuration, will include archiving:

  • Configuration Files
  • Logs

We handle the above archiving process through a set of scripts that are executed either as the 'root' user, or as the special 'monitor' account that we use explicitly for maintenance, monitoring.

As Root

The scripts that we execute as the 'root' user can be automated through root's cron as in the below:

  • Generate a current Package List
  • Archive Configuration Files
  • Archive Logs
50  18  *  *  *  /bin/ls -1 /var/db/pkg/ > /etc/packagelist.txt
00  19  *  *  *  /bin/sh /usr/local/sbin/backup.configs.sh > /dev/null 2>&1
10  19  *  *  *  /bin/sh /usr/local/sbin/archlogs > /dev/null 2>&1
Generate a Package List

We maintain a text file list of installed packages in /etc, so that we can backup this information together with standard configration file settings.

This text file is used in our Configuration Management approach to ensure:

  • Local copies of packages related to this host install are maintained.
  • Building a Restoration ISO can be scripted to include packages from the Package List.
  • Restoration of this host can be scripted (this support allows for scrpted re-installation of packages.)
/bin/ls -1 /var/db/pkg/ > /etc/packagelist.txt

On a daily schedule, we generate a list of installed packages.

Archive Configuration Files

In our simplified approach to configuration management, a basic script is executed by 'root' to archive files in configuration paths, such as /etc.

Review our Configuration Archiving for futher details of this related approach.

  • /var/syncback
  • /usr/local/sbin/backup.configs.sh
  • /etc/backup.configs
  • /etc/backup.configs.exclude

Path: /var/syncback

The destination directory for the archives.

File: /etc/backup.configs

  • Is a line-separated list of files or directories to be archived.

File: /etc/backup.configs.exclude

  • Is a line-separated list of files or directories, not to be archived.

File: backup.configs.sh

#!/bin/sh
# Archive Configuration files using 'tar' or 'gtar' depending on availability
# 
# -- configuration files are specified by line separated text file (no ending `cr')
#     /etc/backup.configs
#
#  -- `gtar` supports exclude option, 
#    /etc/backup.configs.exclude

set_env() {
    PATH=/bin:/usr/bin:/sbin:/usr/sbin
    BACKUP_USER=control
    DST="/var/syncback"
    ostype="unknown"
    OPTIONS="-czpf"
    TAR=/bin/tar
    GTAR=/usr/local/bin/gtar
    HOSTNAME=`/bin/hostname -s`
    TARGETFILE="$DST/$HOSTNAME.tgz"
    CONFIG=/etc/backup.configs
    EXC_FILE=""


    if [ -s ${CONFIG} ]; then
        INC_FILE=${CONFIG}
    else
        echo "ERROR: INCLUDE file does not exist or contain valid data"
        exit 1
    fi
    
    if [ -s ${CONFIG}.exclude ]; then
        EXC_FILE="-X ${CONFIG}.exclude"
    elif [ -x ${GTAR} ]; then
        echo "ERROR: EXCLUDE file does not exist or contain valid data" >&2
    fi

    [[ -x /bin/uname ]] && ostype=`/bin/uname`
    [[ -x /usr/bin/uname ]] && ostype=`/usr/bin/uname`

    umask 027
}


set_destination() {
    if [ ! -d "$DST" ]; then
        mkdir -p "$DST"
        chown root:${BACKUP_USER} "$DST"
        chmod 770 "${DST}"
    fi
}

run_backup() {

    echo -n "Archiving config files for [$HOSTNAME]."
    case $ostype in
        "Linux")
            echo ".on Linux";
            $TAR -T ${INC_FILE} ${EXC_FILE} ${OPTIONS} $TARGETFILE 2> /dev/null
            ;;
        "OpenBSD")
            echo ".on OpenBSD";
            if [ -x $GTAR ]; then
                echo " $GTAR -T ${INC_FILE} ${EXC_FILE} ${OPTIONS} $TARGETFILE"
                $GTAR -T ${INC_FILE} ${EXC_FILE} ${OPTIONS} $TARGETFILE 2> /dev/null
            else
                echo " $TAR -I ${INC_FILE} ${OPTIONS} $TARGETFILE"
                $TAR -I ${INC_FILE} ${OPTIONS} $TARGETFILE 2> /dev/null
            fi
            ;;
        *)
            echo ".Unknown OS - backup failed";
            exit 1;
            ;;
    esac
}
main() {
    set_env
    set_destination
    run_backup
}

main $@

The sample script may be a bit too verbose, cut it down for your own needs.

Archive Log Files

Install and review the archlog script

File: archlogs

#!/bin/sh

HOSTNAME=`/bin/hostname -s`

UNAME_OPENBSD="/usr/bin/uname"
UNAME_LINUX="/bin/uname"
UNAME="uname"
TAR="/bin/tar"

ostype=`$UNAME`

case $ostype in
    "Linux")
        echo "Running on Linux";
        ;;
    "OpenBSD")
        echo "Running on OpenBSD";
        ;;
    *)
        echo "Unknown OS - backup failed";
        exit 1;
        ;;
esac

DATA="/var/log/"
DST="/var/syncback"

umask 027

$TAR -czf $DST/$HOSTNAME.logs.tgz $DATA

The sample script may be a bit too verbose, cut it down for your own needs.

monitor's cron

20  19  *  *  *  /bin/sh /usr/local/sbin/backup.peer.configs.sh > /dev/null 2>&1

Basic monitor user configuration includes maintaining current state of peer hosts not accessible from other hosts. The backup.peer.configs.sh script is responsible for hosts hidden behind this host (such as CARP peers without accessible IP addresses, or hosts behind a firewall.)

#!/bin/sh
# 
# Use for synchronising peers

set_env() {
    umask 027
    HOSTNAME=`/bin/hostname -s`
    RSYNC="/usr/local/bin/rsync"
    SYNCPATH="/var/syncback/"
    CONFIG="/etc"

}
set_peerinfo() {
    # Configuration Files containing peer IP and peer HOSTPEER

    if [ -f /etc/backup.peer.ip ]; then
        SRC_IP=`head -1 $CONFIG/backup.peer.ip`
    elif [ -f /etc/syncback.peer.ip ]; then
        SRC_IP=`head -1 $CONFIG/syncback.peer.ip`
    elif [ -f /etc/syncpeer.ip ]; then
        SRC_IP=`head -1 $CONFIG/syncpeer.ip `
    elif [ -f /etc/syncpeer ]; then
        SRC_IP=`head -1 $CONFIG/syncpeer`
    else
        echo "Can't determine PEER IP"
        exit 1
    fi
    
    if [ -f /etc/backup.peer.name ]; then
        HOSTPEER=`head -1 /etc/backup.peer.name`
    elif [ -f /etc/syncback.peer.hostname ]; then
        HOSTPEER=`head -1 /etc/syncback.peer.hostname`
    elif [ -f /etc/synchost ]; then
        HOSTPEER=`head -1 /etc/synchost`
    else
        echo "Can't determine PEER HOSTNAME"
        exit 1
    fi
}

sync_peer() {
    $RSYNC -av $SRC_IP:$SYNCPATH/$HOSTPEER.tgz $SYNCPATH
    $RSYNC -av $SRC_IP:$SYNCPATH/$HOSTPEER.logs.tgz $SYNCPATH
}

main() {
    set_env
    eval `ssh-agent 2> /dev/null` ssh-add 2>&1 > /dev/null
    set_peerinfo
    sync_peer

    kill $SSH_AGENT_PID 2>&1 > /dev/null

}

main $@
<!--(end)-->