HA Proxy

HAProxy

[Ref: OpenBSD 5.7, 5.8, HAProxy 1.5.14 Configuration Manual ]

Why another document on configuring HAProxy? Because like so many, I also scan the Internet for HOWTOs instead of reading the project documentation.

I had a particular goal in mind when I first installed HaProxy, and since I’d got most of it working using something else Enginx how difficult could it be to rehash it in HaProxy? Why do I need to read documentation, why can’t i just get a recipe and have it up and running in minutes instead of days?

It really isn’t that difficult to go from

After many missteps I find that new deployments work better following simple steps like the above. Get the basic environment up and running to make sure all the bits and pieces are actually working as expected before debugging the more difficult ‘beautiful’ design I had in mind 8-)

The Project documentation has a nice Section 2 configuring HAProxy

2.0. Configuring HAProxy
2.1. Configuration file format 2.2. Time format
2.3. Examples

It didn’t make sense to me when I first read it, but I know you could do better (then you can come back here to be confused.)

HAProxy's configuration process involves 3 major sources of parameters :

  - the arguments from the command-line, which always take precedence
  - the "global" section, which sets process-wide parameters
  - the proxies sections which can take form of "defaults", "listen",
    "frontend" and "backend".

Of course, after having a working system the documentation makes a whole lot more sense?

If you’re trying HAProxy for the first time, I recommend you go through at least the examples in the above reference.

Packages

The basic installation for OpenBSD is through the ports/package system.

Once up and installed, and we’re ready to test our installation the process I generally use re-emphasises the first configuration process:

HAProxy's configuration process involves ... :

  - the arguments from the command-line, which always take precedence

$ sudo haproxy -c -f /etc/haproxy/haproxy.conf

$ sudo haproxy -d -f /etc/haproxy/haproxy.conf

The above spits out a lot of debug information useful for visualising what the proxy is doing.

Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use kqueue.
Using kqueue() as the polling mechanism.
....

Basic Example

A simple adaptation of the first example from the documentation highlights how simple (flexible?) HAProxy can be.

TO help you familiarise with the syntax, and using the manual we’ll look at how/what the directives in the sample configuration.

File: /etc/haproxy/haproxy.conf

global
    daemon
    maxconn 256

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

listen http-in
    bind *:80
    server websvr1 192.168.0.5:80 maxconn 32
    server websvr2 192.168.0.6:80 maxconn 32

Presuming that 192.168.0.5 and 192.168.0.6 are web servers with a simple web page “This is HTTP Server websvrX”, then you could expect to have a result such as:

$ sudo haproxy -c -f /etc/haproxy/haproxy.conf $ sudo haproxy -f /etc/haproxy/haproxy.conf $ while true; do curl http://localhost; sleep 1; done

This is HTTP server websvr1 (192.168.0.5).
This is HTTP server websvr2 (192.168.0.6).
This is HTTP server websvr1 (192.168.0.5).
This is HTTP server websvr2 (192.168.0.6).
…

The general settings are configured in the (line 1) “global” and (line 5) “defaults” section, and the real work is configured in the (line 11)“listen” section labelled “http-in”

The ‘global’ parameters are categorised into three separate groups.

Process Management and Security

For Process Management and Security we specify daemon

    daemon
daemon
Makes the process fork into background. This is the recommended mode of
operation. It is equivalent to the command line "-D" argument. It can be
disabled by the command line "-db" argument.

Performance Tuning

For Performance tuning we set the maxconn rate

    maxconn 256
maxconn <number>
Sets the maximum per-process number of concurrent connections to <number>. It
is equivalent to the command-line argument "-n". Proxies will stop accepting
connections when this limit is reached. The "ulimit-n" parameter is
automatically adjusted according to this value. See also "ulimit-n". Note:
the "select" poller cannot reliably use more than 1024 file descriptors on
some platforms. If your platform only supports select and reports "select
FAILED" on startup, you need to reduce maxconn until it works (slightly
below 500 in general).

Debugging

There is no debugging specified in the configuration file, and we’ve shown above and below that we can specify this on the command-line.

(you can look those up at: debug or quiet)

Proxies

The Proxy configuration
can be located in a set of sections :

Most examples you’ll find involve the ‘frontend’ and ‘backend’, but in our simplified example, the required services are provided in a ‘listen’ section.

Defaults

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
Right now, two major proxy modes are supported : "tcp", also known as layer 4,
and "http", also known as layer 7. In layer 4 mode, HAProxy simply forwards
bidirectional traffic between two sides. In layer 7 mode, HAProxy analyzes the
protocol, and can interact with it by allowing, blocking, switching, adding,
modifying, or removing arbitrary contents in requests or responses, based on
arbitrary criteria.

For this example’s defaults, we are going to be dealing with http (layer 7) traffic, with some preset timeouts related to http traffic.

Listen

A "listen" section defines a complete proxy with its frontend and backend
parts combined in one section. It is generally useful for TCP-only traffic.
listen http-in
    bind *:80
    server websvr1 192.168.0.5:80 maxconn 32
    server websvr2 192.168.0.6:80 maxconn 32
bind [<address>]:<port_range> [, ...] [param*]
bind /<path> [, ...] [param*]

Define one or several listening addresses and/or ports in a frontend.

HAProxy will listen (bind) on all interfaces (“.”) at port 80. All traffic for that port will be load balanced to two web servers:

Below we show how we can start the HaProxy server to review the above configuration.

Command-line Interface

Two ports/packages are important for using HAProxy on OpenBSD are:

Socat isn’t really required on OpenBSD, as Netcat nc(1) is in ‘base’ and supports ‘-U’ UNIX-domain socket connections. But you will find the manual and majority of online references using socat.

Parse the Configuration File

The basic command-line option is to parse/check the configuration file.

$ sudo haproxy -c -f /etc/haproxy/haproxy.conf

Run in the foreground

Command-line options:

-d    Start in foreground with debugging mode enabled. When the proxy
      runs in this mode, it dumps every connections, disconnections,
      timestamps, and HTTP headers to stdout. ...

$ sudo haproxy -d -f /etc/haproxy/haproxy.conf

Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use kqueue.
Using kqueue() as the polling mechanism.
[WARNING] ..
[ALERT] .. Starting frontend absdefg: cannot bind socket [0.0.0.0:80]
[ALERT] .. Starting frontend abcdefghij: cannot bind socket [0.0.0.0:443]

The errors are well categorised, [ALERTS] highlight sections that will definitely cause problems with your installation (such as in the above example, it is not going to work as you expect.) [WARNING]s are likely to cause problems and you should probably investigate and make a determination whether you need to fix or it can be safely ignored.

Soft Reconfiguration

[Ref: HAProxy hot-reconfiguration, HAProxy reloading your config with minimal service impact]

HAProxy supports ‘soft-reconfiguration,’ a mechanism for pausing or disabling haproxy without effecting existing connections.

-sf 
      Send FINISH signal to the pids in pidlist after startup. The   
      processes which receive the signal will wait for all sessions
      to finish before exiting. This option must be specified last,
      followed by any number of PIDs. Technically speaking, SIGTTOU
      and SIGUSR1 are sent.
      
-st 
      Send TERMINATE signal to pids in pidlist after startup. The
      processes which receive this signal will wait immediately
      terminate, closing all active sessions. This option must be
      specified last, followed by any number of PIDs. Technically
      speaking, SIGTTOU and SIGTERM are sent.

$ sudo haproxy -f /etc/haproxy/haproxy.conf -sf

$ sudo haproxy -f /etc/haproxy/haproxy.conf -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.conf)

Statistics on the Command-line

[Ref: Unix Socket commands, HAProxy stats from the command line]

The command-line, live query is based on HAProxy’s ‘stats’ features which are configured using something like the below (more details later)

global stats socket /var/run/haproxy/admin.sock mode 600 level admin stats timeout 2m

[ Ref: stats socket]

stats socket [<address:port>|<path>] [param*]
Binds a UNIX socket to <path> or a TCPv4/v6 address to <address:port>.
Connections to this socket will return various statistics outputs and even
allow some commands to be issued to change some runtime settings. Please
consult section 9.2 "Unix Socket commands" for more details.

All parameters supported by "bind" lines are supported, for instance to
restrict access to some users or their access rights. Please consult
section 5.1 for more information.

stats timeout <timeout, in milliseconds>
The default timeout on the stats socket is set to 10 seconds. It is possible
to change this value with "stats timeout". The value must be passed in
milliseconds, or be suffixed by a time unit among { us, ms, s, m, h, d }.

To get a list of available options, send a ‘blank’ to the socket, such as:

$ echo “” | sudo nc -U /var/run/haproxy/admin.sock

Unknown command. Please enter one of the following commands only :
  clear counters : clear max statistics counters (add ‘all’ for all counters)
  clear table    : remove an entry from a table
  help           : this message
  prompt         : toggle interactive mode with prompt
  quit           : disconnect
  show info      : report information about the running process
  show pools     : report information about the memory pools usage
  show stat      : report counters for each proxy and server
  show errors    : report last request and response errors for each proxy
  show sess [id] : report the list of current sessions or dump this session
  show table [id]: report table usage stats or dump this table’s contents
  get weight     : report a server’s current weight
  set weight     : change a server’s weight
  set server     : change a server’s state or weight
  set table [id] : update or create a table entry’s data
  set timeout    : change a timeout setting
  set maxconn    : change a maxconn setting
  set rate-limit : change a rate limiting value
  disable        : put a server or frontend in maintenance mode
  enable         : re-enable a server or frontend which is in maintenance mode
  shutdown       : kill a session or a frontend (eg:to release listening ports)
  show acl [id]  : report avalaible acls or dump an acl’s contents
  get acl        : reports the patterns matching a sample for an ACL
  add acl        : add acl entry
  del acl        : delete acl entry
  clear acl  : clear the content of this acl
  show map [id]  : report avalaible maps or dump a map’s contents
  get map        : reports the keys and values matching a sample for a map
  set map        : modify map entry
  add map        : add map entry
  del map        : delete map entry
  clear map  : clear the content of this map
  set ssl  : set statement for ssl

$ echo “show info” | sudo nc -U /var/run/haproxy/admin.sock

Name: HAProxy
Version: 1.5.14
Release_date: …
…

Using ‘socat’ is similar

$ echo “show errors” | sudo socat /var/run/haproxy/admin.sock

Global

[Ref: Global parameters]

Parameters in the "global" section are process-wide and often OS-specific. They
are generally set once for all and do not need being changed once correct. Some
of them have command-line equivalents.

As mentioned earlier, HAProxy’s global section can be logically grouped into three groups:

global log 127.0.0.1 local0 info log 127.0.0.1 local1 notice maxconn 50000 chroot /var/haproxy user _haproxy group _haproxy daemon pidfile /var/run/haproxy.pid stats socket /var/run/haproxy/admin.sock mode 660 level admin stats timeout 30s

We have covered some of the configuration settings earlier, and the more interesting parts from our production environment are the “log” specifications, and the “statistics” directives.

Logging

[Ref: log]

When we enable logging using log <address> <facility> [max level] we are sending startup, exit notifications.

log <address> [len <length>] <facility> [max level [min level]]
Adds a global syslog server. Up to two global servers can be defined. They
will receive logs for startups and exits, as well as all logs from proxies
configured with "log global".

By default, configuring a log address sends all events to the log host. We can separate log entries from HAProxy with a configuration such as in the above.

    log 127.0.0.1   local0 info
    log 127.0.0.1   local1 notice

Statistics

    stats socket /var/run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s

As shown in the earliest example, you can specify the statistical view in the Global section. There are options, one described further below.

Proxies

[Ref: Proxies]

Right now, two major proxy modes are supported : "tcp", also known as layer 4,
and "http", also known as layer 7. In layer 4 mode, HAProxy simply forwards
bidirectional traffic between two sides. In layer 7 mode, HAProxy analyzes the
protocol, and can interact with it by allowing, blocking, switching, adding,
modifying, or removing arbitrary contents in requests or responses, based on
arbitrary criteria.
All proxy names must be formed from upper and lower case letters, digits,
'-' (dash), '_' (underscore) , '.' (dot) and ':' (colon). ACL names are
case-sensitive, which means that "www" and "WWW" are two different proxies.

Defaults

[Ref: Proxies]

A "defaults" section sets default parameters for all other sections following
its declaration. Those default parameters are reset by the next "defaults"
section. See below for the list of parameters which can be set in a "defaults"
section. The name is optional but its use is encouraged for better readability.

defaults log global # use the log definition from the ‘global’ section mode http balance roundrobin

    timeout http-request 50s
    timeout connect 5000s
    timeout client 50000s
    timeout server 50000s

Front End

[Ref: Proxies]

A "frontend" section describes a set of listening sockets accepting client
connections.

Virtual IP use hdr_* and reqsni* to switch depending on domain name

frontend ft_vip_http80 bind *:80 mode http log global

    option httpclose    # add "Connection:close" header if it is missing
    option forwardfor   # insert x-forwarded-for header so that app servers can see both proxy and
    tcp-request inspect-delay 5s

    acl lb_nomoa_com_www_80 hdr_dom(host) -i nomoa.com
    acl lb_nomoa_com_www_80 hdr_dom(host) -i www.nomoa.com

    use_backend bk_nomoa_com_www_80  if lb_nomoa_com_www_80

frontend ft_vip_tcp443 bind *:443 mode tcp log global

    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }

    acl lb_nomoa_com_www_443 req_ssl_sni -i nomoa.com
    acl lb_nomoa_com_www_443 req_ssl_sni -i www.nomoa.com

    use_backend bk_nomoa_com_www_443  if lb_nomoa_com_www_443

Back End

[Ref: Proxies]

A "backend" section describes a set of servers to which the proxy will connect
to forward incoming connections.

#

BACK ENDS bk_

# backend bk_nomoa_com_www_80 mode http option httplog balance roundrobin

    server svr1-nomoa.com-80 192.168.0.5:80 check
    server svr2-nomoa.com-80 192.168.0.6:80 check
    server svr3-nomoa.com-80 192.168.0.7:80 check backup

backend bk_nomoa_com_www_443 mode tcp option tcplog balance roundrobin

    option httpchk HEAD / HTTP/1.1\r\nHost:www.nomoa.com
    option ssl-hello-chk

    stick-table type binary len 32 size 30k expire 30m

    acl clienthello req_ssl_hello_type 1
    acl serverhello rep_ssl_hello_type 2

    tcp-request inspect-delay 5s
    tcp-request content accept if clienthello
    tcp-response content accept if serverhello

    stick on payload_lv(43,1) if clienthello
    stick store-response payload_lv(43,1) if serverhello

    server svr1-nomoa.com-443 192.168.0.5:443 check
    server svr2-nomoa.com-443 192.168.0.6:443 check
    server svr3-nomoa.com-443 192.168.0.7:443 check backup

Listen

[Ref: Proxies]

A "listen" section defines a complete proxy with its frontend and backend
parts combined in one section. It is generally useful for TCP-only traffic.

#

STATS

#

listen HAProxy-Statistics *:1936 mode http stats enable stats refresh 20s stats auth username:mypassword

The example ‘listen’ section we have, highlights the general feature, but is better described below.

Statistics

[Ref: Proxies]

[Ref: Statistics and monitoring]

It is possible to query HAProxy about its status. The most commonly used
mechanism is the HTTP statistics page. This page also exposes an alternative
CSV output format for monitoring tools. The same format is provided on the
Unix socket.

#

STATS

#

listen HAProxy-Statistics *:1936 mode http stats enable stats refresh 20s stats auth username:mypassword

Logging

[Ref: Logging

Configuring Syslog

[Ref: syslog.conf(5)]

You should have something like the below in your configuration file:

Format

File extract: /etc/syslog.conf

!haproxy local0.* /var/log/haproxy/info.log local1.* /var/log/haproxy/notice.log

The format is explained in the manpage, with some of the relevant extracts below:

Line 1: Blocks of rules(?) are separated with a tag line beginning with an exclamation mark.

!haproxy

manpage extracts: syslog.conf(5)

Each block of lines is separated from the previous block by a tag. 
The tag is a line beginning with !prog and each block will be 
associated with calls to syslog from that specific program. When a 
message matches multiple blocks, the action of each matching block 
is taken. If no tag is specified at the beginning of the file, every 
line is checked for a match and acted upon (at least until a tag is 
found).

local0.* /var/log/haproxy/info.log local1.* /var/log/haproxy/notice.log

The selectors are encoded as a facility, a period ('.'), and a 
level, with no intervening whitespace. Both the facility and the 
level are case insensitive.

The action field is /var/log/haproxy/info.log and /var/log/haproxy.notice.log

The action field of each line specifies the action to be taken when 
the selector field selects a message. There are six forms:

• A pathname (beginning with a leading slash). Selected messages are 
  appended to the file.

Note: the ‘air-gap’ between the ‘selector’ and ‘action’ are separated by ‘tabs’ not invisible blank spaces.

Tab Stops

You must use ‘tabs’ between the ‘selector’ field and the ‘action’ field. You can guarantee yourself a disfunctioning logging environmnet if you ignore this and blindly presume the ‘whitespaces’ are correct (even if it is correct for /etc/newsyslog.conf)

manpage extract: syslog.conf(5)

The selector field is separated from the action field by one or more tab characters

The simplest way to view whether you used ‘tabs’ or spaces, is to open the file in ‘vim’ and expose tabs using the command “:set list”

References

global log 127.0.0.1 local0 info log 127.0.0.1 local1 notice maxconn 50000 chroot /var/haproxy user _haproxy group _haproxy daemon pidfile /var/run/haproxy.pid stats socket /var/run/haproxy/admin.sock mode 660 level admin stats timeout 30s

defaults log global # use the log definition from the ‘global’ section mode http balance roundrobin

    timeout http-request 50s
    timeout connect 5000s
    timeout client 50000s
    timeout server 50000s

#

FRONT ENDS: ft_

#

Virtual IP use hdr_* and reqsni* to switch depending on domain name

frontend ft_vip_http80 bind *:80 mode http log global

    option httpclose    # add "Connection:close" header if it is missing
    option forwardfor   # insert x-forwarded-for header so that app servers can see both proxy and
    tcp-request inspect-delay 5s

    acl lb_nomoa_com_www_80 hdr_dom(host) -i nomoa.com
    acl lb_nomoa_com_www_80 hdr_dom(host) -i www.nomoa.com

    use_backend bk_nomoa_com_www_80  if lb_nomoa_com_www_80

frontend ft_vip_tcp443 bind *:443 mode tcp log global

    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }

    acl lb_nomoa_com_www_443 req_ssl_sni -i nomoa.com
    acl lb_nomoa_com_www_443 req_ssl_sni -i www.nomoa.com

    use_backend bk_nomoa_com_www_443  if lb_nomoa_com_www_443

#

BACK ENDS bk_

# backend bk_nomoa_com_www_80 mode http option httplog balance roundrobin

    server svr1-nomoa.com-80 192.168.0.5:80 check
    server svr2-nomoa.com-80 192.168.0.6:80 check
    server svr3-nomoa.com-80 192.168.0.7:80 check backup

backend bk_nomoa_com_www_443 mode tcp option tcplog balance roundrobin

    option httpchk HEAD / HTTP/1.1\r\nHost:www.nomoa.com
    option ssl-hello-chk

    stick-table type binary len 32 size 30k expire 30m

    acl clienthello req_ssl_hello_type 1
    acl serverhello rep_ssl_hello_type 2

    tcp-request inspect-delay 5s
    tcp-request content accept if clienthello
    tcp-response content accept if serverhello

    stick on payload_lv(43,1) if clienthello
    stick store-response payload_lv(43,1) if serverhello

    server svr1-nomoa.com-443 192.168.0.5:443 check
    server svr2-nomoa.com-443 192.168.0.6:443 check
    server svr3-nomoa.com-443 192.168.0.7:443 check backup

#

STATS

#

listen HAProxy-Statistics *:1936 mode http stats enable stats refresh 20s stats auth username:mypassword