Validating connectivity, performance, configuration

[Ref: OpenBSD 5.1, Asterisk 1.8.11 (from ports)]

Table of Content
  1. Remote Connectivity
  2. SSL/TLS Transport
  3. Behaviour

As a conglomeration of various services, there are a number of separate tests that can be carried out to confirm, validate a systems activity, performance.

General Remote/Local connections are through the standard protocols ( smtp, pop3, and imap ) whereas service behaviour have some other attributes that can also be monitored.

The following are a number of utilities, methodologies for validating your server's behaviour, performance during development and after deployment.

Remote Connectivity

SMTP - Simple Message Transfer Protocol

telnet

Telnet is a useful tool for connectivy testing with servers that respond with Text responses and support protocols that can be entered at the command-line. Services that generally support this include smtp, imap, and pop3.

$ telnet localhost smtp

Trying ::1...
Connected to localhost.
Escape character is '^]'.
220 myhost.example.org ESMTP Postfix

EHLO example.org

250-myhost.example.org 
250-PIPELINING 
250-SIZE 10240000 
250-VRFY 
250-ETRN 
250-ENHANCEDSTATUSCODES 
250-8BITMIME 
250 DSN 

MAIL FROM: <samt@example.org>

250 2.1.0 Ok 

RCPT TO: <samt@example.org>

250 2.1.5 Ok

DATA

354 Enter mail, end with '.' on a line by itself

Subject: This is my subject line

I continue writing until I'm out of interesting things to say
which is not that far away

.

250 2.0.0 Ok: queued as 699ACBA2D7

QUIT

221 2.0.0 Bye 
Connection closed by foreign host.

I've just used capital letters for the SMTP commands, but obviously they work fine with lowercase.

If your server is not yet online with a valid DNS record, then you can test using RCPT TO: samt@localhost.

Log Entries

[ tail ]

Confirm where your MTA logs it's messages, but in general mail entries are logged in the /var/log/maillog file, and we can monitor general activities through this file.

Screen Session

# tail -f /var/log/maillog

starting the Postfix mail system
daemon started -- version 2.3.2, configuration 
/etc/postfix
connect from localhost[::1]
5E4A5BA2D4: client=localhost[::1]
5E4A5BA2D4: message-id=<20061212080251.5E4A5BA2D4@hostname.example.org>
5E4A5BA2D4: from=<samt@example.org>, size=457, nrcpt=1 (queue active)
5E4A5BA2D4: to=<samt@example.org>, relay=local, delay=77, delays=77/0.05/0/0.03, dsn=2.0.0, status=sent (delivered to mailbox)
5E4A5BA2D4: removed
disconnect from localhost[::1]

From the manpage: 'tail(1)' 

    The tail utility displays the contents of file or, by default, its
    standard input, to the standard output.

-f  Do not stop when end-of-file is reached; instead, wait for
    additional data to be appended to the input.  If the file is
    replaced (i.e., the inode number changes), tail will reopen the
    file and continue.  If the file is truncated, tail will reset its
    position to the beginning.  This makes tail more useful for
    watching log files that may get rotated.  The -f option is
    ignored if the standard input is a pipe, but not if it is a FIFO.

Using the '-f' parameter tells 'tail' to continue looking at the recent additions to the file (such that updates to the file are displayed on the screen for us.) Use Ctrl+C (i.e. hold the Ctrl key while pressing C) to break out of the log review session shown above

mail

[Ref: mail(1) ]

While we're testing with real system user accounts, we can use the unix 'mail' program to check our mail message.

Screen Session

# /usr/bin/mail -u samt

Mail version 8.1.2 01/15/2001. Type ? for help.
'/var/mail/samt': 1 message 1 new
>N 1 samt@example.org Tue Dec 12 21:03 18/605 This is my subject line

& more 1

Message 1:
From samt@example.org Tue Dec 12 21:03:54 2006
X-Original-To: samt@myhost.example.org
Delivered-To: samt@myhost.example.org
Subject: This is my subject line
From: samt@example.org
To: undisclosed-recipients:;

I continue writing until I'm out of interesting things to say
which is not that far away

& q

Saved 1 message in mbox

In the above example, we enter mail for the user samt ('-u samt') and the 'mail' client shows a list of current email for user 'samt' and then gives us the '&' ampersand prompt.

We can read the email message by typing the message number, and 'mail' supports the use of a screen 'pager' such as 'more' so that we can scroll through longer messages.

From the manpage: mail(1)

Reading mail
    In normal usage, mail is given no arguments and checks your mail out of
    the post office, then prints out a one line header of each message found.
    The current message is initially set to the first message (numbered 1)
    and can be printed using the print command (which can be abbreviated p).
    Moving among the messages is much like moving between lines in ed(1); you
    may use + and - to shift forwards and backwards, or simply enter a
    message number to move directly.

Quit. We quit out of 'mail' using the 'q' command.

The above reference to the log files and mail client is to provide you with more tools for validating your installation.

Mail Receipt

[Ref: telnet, mail, /var/log/maillog ]

Verify that the mail server will accept tcp connections on smtp from the below by sending mail to a user account on the mail host:

  • localhost
  • relay network
  • external network

With the verification, ensure that email is recieved by watching the maillog to note:

  • mail is accepted (queued) by the mail server
  • mail is delivered (sent) by the mail server (in this case, delivery is to a local mail box)

An internal client (destination email) should have successfully recieved mail from the above tests.

Mail Delivery

[ telnet, mail, /var/log/maillog ]

Verify the mail server will accept tcp connections on smtp and can send mail 'to the world' by sending messages from:

  • localhost
  • LAN

Ensure that email can be sent through and to a destination by watching the maillog to note:

  • mail is accepted from the mail server host (eg. telnet from host to remote smtp server)
  • mail is queued by the mail server
  • mail is delivered by the mail server

An external client (destination email) should have successfully recieved mail from the above tests.

Summary

We now have a fully functional SMTP server that can receive email messages, and store those messages for users.

POP3 - Post Office Protocol

[Ref: The Network People, Inc. Mail Server Testing ]

If you've successfully installed dovecot with mysql above, and have gone through the Configuring a Virtual Email Service - MySQL in our postfix installation guide, (or you have installed your own MySQL virtual user accounts) then we can perform some testing, validating whether our configuration actually works.

Screen Session

$ telnet localhost pop3

Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
+OK Dovecot ready.

user charlie@alpha.example.org

+OK
pass charlie

+OK Logged in.

list

+OK 3 messages:
1 503
2 445
3 503
.
retr 3

+OK 503 octets
Return-Path: <samt@example.org>
X-Original-To: charlie@alpha.example.org
Delivered-To: charlie@alpha.example.org
Received: from example.org (unknown [IPv6:::1])
by myhost.example.org (Postfix) with ESMTP id 9A6165A950;
Fri, 9 Feb 2007 13:50:26 +1300 (TOT)
Subject: Welcome MySQL based virtual users
Message-Id: <20070209005037.9A6165A950@myhost.example.org>
Date: Fri, 9 Feb 2007 13:50:26 +1300 (TOT)
From: samt@example.org
To: undisclosed-recipients:;

Hopefully you've received this email message without fault ?


.

QUIT

+OK Logging out.
Connection closed by foreign host.

The maillog file should show success similar to the below

File Fragment: /var/log/maillog

pop3-login: Login: user=<charlie@alpha.example.org>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
POP3(charlie@alpha.example.org): Disconnected: Logged out top=0/0, retr=1/519, del=0/3, size=1451

Again, a review of the mysql transaction log can be helpful in diagnosing errors.

File Fragment: /var/mysql/myhost.log

Connect dovecot@localhost on mail
Query SELECT password FROM mailbox WHERE username = 'charlie@alpha.example.org' AND active = '1'
Query SELECT maildir, 901 AS uid, 901 AS gid FROM mailbox WHERE username = 'charlie@alpha.example.org' AND active = '1'

Simple Errors -ERR Authentication failed.

You get an Authentication failed even though you know and swear that you have entered the correct password?

  • Check the /var/mysql/myhost.log file to ensure that the correct query is sent by dovecot to the MySQL Server (i.e. SELECT password FROM mailbox WHERE username = 'VIRTUALACCOUNT@VIRTUALDOMAIN' AND active = '1')
  • Check that your dovecot configuration is using the same encryption method for creating/reading passwords, as postfixadmin. For example, in our exercise we are using CRYPT: defaultpassscheme = CRYPT.

IMAP - Internet Message Access Protocol

We use telnet on the localhost to test imap's configuration

Screen Session

$ telnet localhost imap

Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* OK Dovecot ready.

a1 login charlie@alpha.example.org charlie

a1 OK Logged in.

a2 select inbox

* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
* 3 EXISTS
* 0 RECENT
* OK [UNSEEN 1] First unseen.
* OK [UIDVALIDITY 1170991431] UIDs valid
* OK [UIDNEXT 4] Predicted next UID
a2 OK [READ-WRITE] Select completed.

a3 fetch 3 body[text]

* 3 FETCH (BODY[TEXT] {66}
Hopefully you've received this email message without fault ?


)
a3 OK Fetch completed.

a4 close

a4 OK Close completed.

a5 logout

* BYE Logging out
a5 OK Logout completed.
Connection closed by foreign host.

Note:

a1, a2, .., a5 are randomly selected unique leaders (in this case we're just making things sequential)

"a3 fetch 3 body[text]", the number '3' refers to the '3_ EXISTS_' in the list returned by 'a2 select inbox'

Your maillog file is your friend and will give you clues to where you can check for other errors.

File Fragment: /var/log/maillog

auth-worker(default): mysql: Connected to localhost 
(mail)
imap-login: Login: user=<charlie@alpha.example.org>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
IMAP(charlie@alpha.example.org): Disconnected: Logged out

Likewise the mysql transaction log should give further assistance should the installation be having problems.

File Fragment: /var/mysql/myhost.log

Connect dovecot@localhost on mail
Query SELECT password FROM mailbox WHERE username = 'charlie@alpha.example.org' AND active = '1'
Query SELECT maildir, 901 AS uid, 901 AS gid FROM mailbox WHERE username = 'charlie@alpha.example.org' AND active = '1'

SSL / TLS Transport

[Ref: openssl s_client]

The s_client command implements a generic SSL/TLS client which connects to a remote host using SSL/TLS. It is a very useful diagnostic tool for SSL servers.

For all the above protocols (smtp, pop3, imap) we can use openssl s_client to connect to the remote host, and use the standard diagnostics commands, as if it were a cleartext connection.

SMTP is provided into two flavours, a cleartext connection which can then be negotiated into an encrypted connection, and the 2nd with encrypted connection to begin with.

SMTP using STARTTLS

openssl s_client provides a command-line mechanism for initiating a connection, and starting STARTTLS to encrypt connection link.

openssl s_client -starttls smtp -crlf -connect localhost:25

A lot of certificate information is exchanged, and shown on the screen

...

250 DSN

MAIL FROM: 

250 2.1.0 Ok

RCPT TO: mylocaluser

250 2.1.5 Ok

DATA

354 End data with .

Subject: STARTTLS Test Message

Postfix will decrypt

.

250 2.0.0 Ok: queued as XXXXXXXXXXX

quit

221 2.0.0 Bye

openssl's s_client -starttls smtp tells it to make the encryption/decrypting between our console and an SMTP server. Once connected, then we can go through our normal smtp test message commands. (Shown in CAPITAL letters above.)

SMTPS

Verify the wrapmode smtps service is working correctly using the generic openssl s_client connection.

openssl s_client -connect localhost:465

CONNECTED(0000000X)
Plenty of Certificate negotiation/information

220 mx.coco.nut.to ESMTP Postfix

At this point, we have an example of connecting SSL secured to SMTPS. The same approach is made for connecting to other SSL services:

openssl s_client -connect localhost:${PORT}

Behaviour

Performance

[ ports/benchmarks/smtp-benchmark ]

smtp-benchmark is a client/server program used to push as much data through the pipe to visit the mail server and review it's capacity to handle load.

smtp-benchmark consists of two programs, smtpsend and smtpsink. smtpsend is used to send generated e-mail messages using SMTP to a mail transfer agent, smtpsink is designed to dispose of received messages as quick as possible.

  • Server
  • Client

smtp-benchmark: smtpsink

The smtpsink Server runs on the destination side of the mail server, expecting to recieve mail FROM the mail server. It basically accepts the message delivery from the mail server and dumps it to /dev/null

smtpsink -v -p 25

smtp-benchmark: smtpsend

The smtpsend Client measures the time spent sending e-mails and the number of e-mails actually sent and outputs statistics after the program run.

smtp-send can fork one or more parallel senders each using one or more sequential connections to a SMTP server to deliver one or more messages per connection.

smtp-sink comes in handy when the relaying performance of a MTA is to be measured.

smtpsend -s 1 -m 100 -b 512 -t 120 -p 25 -F samt@example.com \
   -T user@valid-domain.com -S "Test Message" -vv SMTP_SERVER

Publicly Accessible Services

[ ports/net/nmap ]

Verify with nmap that only the expected services are running on this host.

Open Relay

Since a configuration failing to allow "Open Relay" is just not excusable.

Verify with telnet that open relay is not supported, other than those specifically included in the server configuration.

An attempt to relay should be rejected by the mail server.

tcpdump -nettti interface

Our standard configurations will require mail to traverse at least two firewalls, use tcpdump at appropriate locations to verify that the smtp traffic is tranversing the route between the client and our server.

  • NIC - verify traffic reaches the NIC and has a route back to the client.
  • Firewall External NIC - verify traffic reaches the NIC and is not blocked
  • Firewall Internal NIC - verify traffic goes in/out the Internal link and indicates return route from the client.
RSS