HA Proxy

HAProxy

[Ref: OpenBSD 5.7, 5.8, HAProxy 1.5.14 Configuration Manual ]

Why another document on configuring HAProxy? Because like so many, I also scan the Internet for HOWTOs instead of reading the project documentation.

I had a particular goal in mind when I first installed HaProxy, and since I’d got most of it working using something else Enginx how difficult could it be to rehash it in HaProxy? Why do I need to read documentation, why can’t i just get a recipe and have it up and running in minutes instead of days?

Table of Contents

It really isn’t that difficult to go from

  • basic/simple HaProxy configuration to a
  • moderately difficult configuration, to
  • your desired configuration.

After many missteps I find that new deployments work better following simple steps like the above. Get the basic environment up and running to make sure all the bits and pieces are actually working as expected before debugging the more difficult ‘beautiful’ design I had in mind 8-)

The Project documentation has a nice Section 2 configuring HAProxy

2.0. Configuring HAProxy 2.1. Configuration file format 2.2. Time format 2.3. Examples

It didn’t make sense to me when I first read it, but I know you could do better (then you can come back here to be confused.)

HAProxy's configuration process involves 3 major sources of parameters :

  - the arguments from the command-line, which always take precedence
  - the "global" section, which sets process-wide parameters
  - the proxies sections which can take form of "defaults", "listen",
    "frontend" and "backend".

Of course, after having a working system the documentation makes a whole lot more sense?

If you’re trying HAProxy for the first time, I recommend you go through at least the examples in the above reference.

Packages

The basic installation for OpenBSD is through the ports/package system.

Once up and installed, and we’re ready to test our installation the process I generally use re-emphasises the first configuration process:

HAProxy's configuration process involves ... :

  - the arguments from the command-line, which always take precedence
  • When verifying a configuration change or new updates I normally run ‘haproxy’ in console debug mode with the following commands.
$ sudo haproxy -c -f /etc/haproxy/haproxy.conf
  • Command-line:
    • -c checks the config file and exits.
    • -f specify the configuration file
$ sudo haproxy -d -f /etc/haproxy/haproxy.conf
  • Command-line:
    • -d Start in foreground with debugging mode enabled.
    • -f specify the configuration file

The above spits out a lot of debug information useful for visualising what the proxy is doing.

Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use kqueue.
Using kqueue() as the polling mechanism.
....

Basic Example

A simple adaptation of the first example from the documentation highlights how simple (flexible?) HAProxy can be.

TO help you familiarise with the syntax, and using the manual we’ll look at how/what the directives in the sample configuration.

File: /etc/haproxy/haproxy.conf

    global
        daemon
        maxconn 256

    defaults
        mode http
        timeout connect 5000ms
        timeout client 50000ms
        timeout server 50000ms

    listen http-in
        bind *:80
        server websvr1 192.168.0.5:80 maxconn 32
        server websvr2 192.168.0.6:80 maxconn 32

Presuming that 192.168.0.5 and 192.168.0.6 are web servers with a simple web page “This is HTTP Server websvrX”, then you could expect to have a result such as:

$ sudo haproxy -c -f /etc/haproxy/haproxy.conf
$ sudo haproxy -f /etc/haproxy/haproxy.conf
$ while true; do curl http://localhost; sleep 1; done
This is HTTP server websvr1 (192.168.0.5).
This is HTTP server websvr2 (192.168.0.6).
This is HTTP server websvr1 (192.168.0.5).
This is HTTP server websvr2 (192.168.0.6).
...

The general settings are configured in the (line 1) “global” and (line 5) “defaults” section, and the real work is configured in the (line 11)“listen” section labelled “http-in”

The ‘global’ parameters are categorised into three separate groups.

  • Process management and security
  • Performance tuning, and
  • Debugging

Process Management and Security

For Process Management and Security we specify daemon

daemon
Makes the process fork into background. This is the recommended mode of
operation. It is equivalent to the command line "-D" argument. It can be
disabled by the command line "-db" argument.

Performance Tuning

For Performance tuning we set the maxconn rate

maxconn <number>
Sets the maximum per-process number of concurrent connections to <number>. It
is equivalent to the command-line argument "-n". Proxies will stop accepting
connections when this limit is reached. The "ulimit-n" parameter is
automatically adjusted according to this value. See also "ulimit-n". Note:
the "select" poller cannot reliably use more than 1024 file descriptors on
some platforms. If your platform only supports select and reports "select
FAILED" on startup, you need to reduce maxconn until it works (slightly
below 500 in general).

Debugging

There is no debugging specified in the configuration file, and we’ve shown above and below that we can specify this on the command-line.

(you can look those up at: debug or quiet)

Proxies

The Proxy configuration can be located in a set of sections :

  • defaults <name>
  • frontend <name>
  • backend <name>
  • listen <name>

Most examples you’ll find involve the ‘frontend’ and ‘backend’, but in our simplified example, the required services are provided in a ‘listen’ section.

Defaults

Right now, two major proxy modes are supported : "tcp", also known as layer 4,
and "http", also known as layer 7. In layer 4 mode, HAProxy simply forwards
bidirectional traffic between two sides. In layer 7 mode, HAProxy analyzes the
protocol, and can interact with it by allowing, blocking, switching, adding,
modifying, or removing arbitrary contents in requests or responses, based on
arbitrary criteria.

For this example’s defaults, we are going to be dealing with http (layer 7) traffic, with some preset timeouts related to http traffic.

Listen

A "listen" section defines a complete proxy with its frontend and backend
parts combined in one section. It is generally useful for TCP-only traffic.
bind [<address>]:<port_range> [, ...] [param*]
bind /<path> [, ...] [param*]

Define one or several listening addresses and/or ports in a frontend.

HAProxy will listen (bind) on all interfaces (".") at port 80. All traffic for that port will be load balanced to two web servers:

  • websvr1 which is at host: 192.168.0.5 port 80, and
  • websvr2 which is at host: w192.168.0.6 port 80

Below we show how we can start the HaProxy server to review the above configuration.

Command-line Interface

Two ports/packages are important for using HAProxy on OpenBSD are:

Socat isn’t really required on OpenBSD, as Netcat nc(1) is in ‘base’ and supports ‘-U’ UNIX-domain socket connections. But you will find the manual and majority of online references using socat.

Parse the Configuration File

The basic command-line option is to parse/check the configuration file.

$ sudo haproxy -c -f /etc/haproxy/haproxy.conf

Run in the foreground

Command-line options:

-d    Start in foreground with debugging mode enabled. When the proxy
      runs in this mode, it dumps every connections, disconnections,
      timestamps, and HTTP headers to stdout. ...
$ sudo haproxy -d -f /etc/haproxy/haproxy.conf
Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use kqueue.
Using kqueue() as the polling mechanism.
[WARNING] ..
[ALERT] .. Starting frontend absdefg: cannot bind socket [0.0.0.0:80]
[ALERT] .. Starting frontend abcdefghij: cannot bind socket [0.0.0.0:443]

The errors are well categorised, [ALERTS] highlight sections that will definitely cause problems with your installation (such as in the above example, it is not going to work as you expect.) [WARNING]s are likely to cause problems and you should probably investigate and make a determination whether you need to fix or it can be safely ignored.

Soft Reconfiguration

[Ref: HAProxy hot-reconfiguration, HAProxy reloading your config with minimal service impact]

HAProxy supports ‘soft-reconfiguration,’ a mechanism for pausing or disabling haproxy without effecting existing connections.

-sf 
      Send FINISH signal to the pids in pidlist after startup. The
      processes which receive the signal will wait for all sessions
      to finish before exiting. This option must be specified last,
      followed by any number of PIDs. Technically speaking, SIGTTOU
      and SIGUSR1 are sent.

-st 
      Send TERMINATE signal to pids in pidlist after startup. The
      processes which receive this signal will wait immediately
      terminate, closing all active sessions. This option must be
      specified last, followed by any number of PIDs. Technically
      speaking, SIGTTOU and SIGTERM are sent.
$ sudo haproxy -f /etc/haproxy/haproxy.conf -sf
$ sudo haproxy -f /etc/haproxy/haproxy.conf -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.conf)

Statistics on the Command-line

[Ref: Unix Socket commands, HAProxy stats from the command line]

The command-line, live query is based on HAProxy’s ‘stats’ features which are configured using something like the below (more details later)

[ Ref: stats socket]

stats socket [<address:port>|<path>] [param*]
Binds a UNIX socket to <path> or a TCPv4/v6 address to <address:port>.
Connections to this socket will return various statistics outputs and even
allow some commands to be issued to change some runtime settings. Please
consult section 9.2 "Unix Socket commands" for more details.

All parameters supported by "bind" lines are supported, for instance to
restrict access to some users or their access rights. Please consult
section 5.1 for more information.

stats timeout <timeout, in milliseconds>
The default timeout on the stats socket is set to 10 seconds. It is possible
to change this value with "stats timeout". The value must be passed in
milliseconds, or be suffixed by a time unit among { us, ms, s, m, h, d }.

To get a list of available options, send a ‘blank’ to the socket, such as:

$ echo "" | sudo nc -U /var/run/haproxy/admin.sock
Unknown command. Please enter one of the following commands only :
  clear counters : clear max statistics counters (add 'all' for all counters)
  clear table    : remove an entry from a table
  help           : this message
  prompt         : toggle interactive mode with prompt
  quit           : disconnect
  show info      : report information about the running process
  show pools     : report information about the memory pools usage
  show stat      : report counters for each proxy and server
  show errors    : report last request and response errors for each proxy
  show sess [id] : report the list of current sessions or dump this session
  show table [id]: report table usage stats or dump this table's contents
  get weight     : report a server's current weight
  set weight     : change a server's weight
  set server     : change a server's state or weight
  set table [id] : update or create a table entry's data
  set timeout    : change a timeout setting
  set maxconn    : change a maxconn setting
  set rate-limit : change a rate limiting value
  disable        : put a server or frontend in maintenance mode
  enable         : re-enable a server or frontend which is in maintenance mode
  shutdown       : kill a session or a frontend (eg:to release listening ports)
  show acl [id]  : report avalaible acls or dump an acl's contents
  get acl        : reports the patterns matching a sample for an ACL
  add acl        : add acl entry
  del acl        : delete acl entry
  clear acl  : clear the content of this acl
  show map [id]  : report avalaible maps or dump a map's contents
  get map        : reports the keys and values matching a sample for a map
  set map        : modify map entry
  add map        : add map entry
  del map        : delete map entry
  clear map  : clear the content of this map
  set ssl  : set statement for ssl
$ echo "show info" | sudo nc -U /var/run/haproxy/admin.sock
Name: HAProxy
Version: 1.5.14
Release_date: ...
...

Using ‘socat’ is similar

$ echo "show errors" | sudo socat /var/run/haproxy/admin.sock

Global

[Ref: Global parameters]

Parameters in the "global" section are process-wide and often OS-specific. They
are generally set once for all and do not need being changed once correct. Some
of them have command-line equivalents.

As mentioned earlier, HAProxy’s global section can be logically grouped into three groups:

  • Process Management and Security
  • Performance tuning
  • Debugging

We have covered some of the configuration settings earlier, and the more interesting parts from our production environment are the “log” specifications, and the “statistics” directives.

Logging

[Ref: log]

When we enable logging using log <address> <facility> [max level] we are sending startup, exit notifications.

log <address> [len <length>] <facility> [max level [min level]]
Adds a global syslog server. Up to two global servers can be defined. They
will receive logs for startups and exits, as well as all logs from proxies
configured with "log global".

By default, configuring a log address sends all events to the log host. We can separate log entries from HAProxy with a configuration such as in the above.

  • line 2 - send logs to address 127.0.0.1 syslog facility local0 and severity level info
  • line 3 - send logs to address 127.0.0.1 syslog facility local1 and severity level notice

Statistics

As shown in the earliest example, you can specify the statistical view in the Global section. There are options, one described further below.

Proxies

[Ref: Proxies]

Right now, two major proxy modes are supported : "tcp", also known as layer 4,
and "http", also known as layer 7. In layer 4 mode, HAProxy simply forwards
bidirectional traffic between two sides. In layer 7 mode, HAProxy analyzes the
protocol, and can interact with it by allowing, blocking, switching, adding,
modifying, or removing arbitrary contents in requests or responses, based on
arbitrary criteria.
All proxy names must be formed from upper and lower case letters, digits,
'-' (dash), '_' (underscore) , '.' (dot) and ':' (colon). ACL names are
case-sensitive, which means that "www" and "WWW" are two different proxies.

Defaults

[Ref: Proxies]

A "defaults" section sets default parameters for all other sections following
its declaration. Those default parameters are reset by the next "defaults"
section. See below for the list of parameters which can be set in a "defaults"
section. The name is optional but its use is encouraged for better readability.

Front End

[Ref: Proxies]

A "frontend" section describes a set of listening sockets accepting client
connections.

Back End

[Ref: Proxies]

A "backend" section describes a set of servers to which the proxy will connect
to forward incoming connections.

Listen

[Ref: Proxies]

A "listen" section defines a complete proxy with its frontend and backend
parts combined in one section. It is generally useful for TCP-only traffic.

The example ‘listen’ section we have, highlights the general feature, but is better described below.

Statistics

[Ref: Proxies]

[Ref: Statistics and monitoring]

It is possible to query HAProxy about its status. The most commonly used
mechanism is the HTTP statistics page. This page also exposes an alternative
CSV output format for monitoring tools. The same format is provided on the
Unix socket.

Logging

[Ref: Logging

Configuring Syslog

[Ref: syslog.conf(5)]

You should have something like the below in your configuration file:

Format

File extract: /etc/syslog.conf

The format is explained in the manpage, with some of the relevant extracts below:

Line 1: Blocks of rules(?) are separated with a tag line beginning with an exclamation mark.

manpage extracts: syslog.conf(5)

Each block of lines is separated from the previous block by a tag.
The tag is a line beginning with !prog and each block will be
associated with calls to syslog from that specific program. When a
message matches multiple blocks, the action of each matching block
is taken. If no tag is specified at the beginning of the file, every
line is checked for a match and acted upon (at least until a tag is
found).
  • local0 and local1 are the facilities
  • ‘*’ says to use all levels
The selectors are encoded as a facility, a period ('.'), and a
level, with no intervening whitespace. Both the facility and the
level are case insensitive.

The action field is /var/log/haproxy/info.log and /var/log/haproxy.notice.log

The action field of each line specifies the action to be taken when
the selector field selects a message. There are six forms:

• A pathname (beginning with a leading slash). Selected messages are
  appended to the file.

Note: the ‘air-gap’ between the ‘selector’ and ‘action’ are separated by ‘tabs’ not invisible blank spaces.

Tab Stops

You must use ‘tabs’ between the ‘selector’ field and the ‘action’ field. You can guarantee yourself a disfunctioning logging environmnet if you ignore this and blindly presume the ‘whitespaces’ are correct (even if it is correct for /etc/newsyslog.conf)

manpage extract: syslog.conf(5)

The selector field is separated from the action field by one or more tab characters

The simplest way to view whether you used ‘tabs’ or spaces, is to open the file in ‘vim’ and expose tabs using the command “:set list”

References