Getting your logging strategy right, may not be a choice
For the umpteenth time one of our squid boxes went down due to the logs filling up all available disk space. As we have 3 sites, plus a special client access network, we have FOUR squid boxes that have been problematic for a long time.
The first major problem we had (it just seemed way too slow) was identified after trawling the logs to find the problem was always there, we need to fixup the number of available file descriptors.
grep WARNING /var/squid/logs/cache.logYYYY/MM/DD HH:MM:SS| WARNING: Your cache is running out of filedescriptors
In this particular case, we had a lot of error messsages with the same text “Your cache is running out of filedescriptors.” How many file descriptors do I have for my cashing service/daemon?
Using your shell, you can determine the number of file descriptors by something like the below:
This will show you what your login class settings give to your squid daemon.
# usermod -s /bin/ksh _squid
# su _squid
$ ulimit -atime(cpu-seconds) unlimited file(blocks) unlimited coredump(blocks) unlimited data(kbytes) 2097152 stack(kbytes) 8192 lockedmem(kbytes) 296673 memory(kbytes) 887008 nofiles(descripters) 128 processes 1310
$ exit
# usermod -s /sbin/nologin _squidMore simply
$ ulimit -n128
As in the earlier example, you can also use squidclient to view what the running squid process sees as available file descriptors. This view will also let you watch over time as your file descriptors deplete etc.
# squidclient mgr:info | grep -i file
File descriptor usage for squid:
Maximum number of file descriptors:
Largest file desc currently in use:
Number of file desc currently in use:
Files queued for open:
Available number of file descriptors:
Reserved number of file descriptors:
Store Disk files open:
The solution? Increase the number of file descriptors available to your daemon and a clean way of doing this is the login class as described here
But where is the green sheep? What is there to fix our current persistent log over grown problems?
I just love that saying oft attributed to Einstein defining Insanity.
Doing the same thing over and over, expecting a different result.
It’s crazy that part of our solution was to just delete the old logs, and restart squid service.
The problem we have with our Squid Caching Proxies, which is quite different from all our other services, is that we just don’t have much disk space (strange how that should make one think more.)
We can’t just replicate our log management strategy used on other boxes. The Squid access logs really do grow quickly on these boxes. It isn’t good enough to collect a year’s worth of logs, rotating them at the end of each month.
Squid’s default log rotation process, just ‘moves’ the file, without compression. So, last month’s 8GB log file, is 8GB permanently lost on the HDD. We need to compress those old log files (do we really need them?)
Looking at our log analysis … we don’t need to keep that length of archiving on disk.
[Ref: newsyslog(8)]
There are various ways of using newsyslog(8) to administer your squid logs, one being below
File extract: /etc/newsyslog.conf
#log file name owner:group mode count size when flags options /var/squid/logs/access.log _squid:_squid 640 12 * $M1D0 ZB /var/squid/logs/squid.pid SIGUSR1 /var/squid/logs/cache.log _squid:_squid 640 12 * $M1D0 ZB /var/squid/logs/squid.pid SIGUSR1 /var/squid/logs/store.log _squid:_squid 640 12 * $M1D0 ZB /var/squid/logs/squid.pid SIGUSR1
Confirm the location of the log files and squid.pid (the above works for some of my hosts.)
The above configuration works on keeping 12 (count) previous logs of unlimited (* size) and runs the archive/compress on the 1st day of the month ($M1D0) This strategy works for our servers with more disk space, but is going to be problematic for some of us with smaller disks.
Note: squid 3.1 and later(?) changed default creation for cache_store_log OFF.
Your strategy should consider the tools you use for analysing the log files. If you can get your logs off the server on a daily basis, then you can probably go for a more frequent ‘when’ such as $D0 daily at midnight, same as @T00.
Examples from the manpage newsyslog(8) include:
PID and SIGUSR1
Newsyslog will send SIGUSR1 to squid, which in this case instructs squid to close, reopen the log file.
With the above configuration, we need to tell squid that we’re using newsyslog using the logfile_rotate value of 0
File extract: /etc/squid/squid.conf
logfile_rotate 0