Web Proxy Log Analysis Reports
[This modification has been verified to work with Smoothwall Express 2.0.
I haven't yet tested this mod with on Smoothwall GPL 1.0, but it should work fine.]
This page describes how install some web proxy anaysis reporting scripts on Smoothwall,
allowing you to analyse the data in the web proxy logs.
Overview:
The generated report includes information such as:
- summary of request method usage (ie, GET, POST, etc)
- summary of protocol usage (ie, http, ssl, etc)
- summary of client usage (ie, totals for each PC on your green network)
- summary of server usage (ie, totals for websites visited)
- summary of url types (ie, query, image, directory, html, executable, etc)
- summary of url top-level domains (ie, .com, .org, .au, etc)
- and an ascii graph showing usage over time
For each of the above (other than the graph), the report also details the
percentage of traffic that is coming out of the cache, thus providing you
with a very good idea of the re-used content your web proxy is serving.
Note that the scripts will only analyse data in the current proxy log
(ie, /var/log/squid/access.log), and Smoothwall rotates
the proxy logs at the beginning of each week.
Screen Shots:
Here's a screen shot of the summary page, showing daily graphs for the
green and red interfaces.
sections of a sample web proxy log analysis report
Requirements:
You'll need:
-
A Smoothwall Express 2.0 installation (obviously...).
-
An SCP client (such as WinSCP,
or SSH Secure File Transfer Client, part of
SSH Secure Shell)
for copying files onto your Smoothwall box.
-
A way of getting a command-line prompt on your Smoothwall box
(either by logging directly onto your Smoothwall box,
using a SSH client such as
PuTTY
or SSH Secure Shell,
or via the Smoothwall web interface).
-
sw_squidlogs.tgz
(11KB - contains report generation scripts, web page for viewing report)
Hopefully it's obvious enough that these scripts won't provide any benefit if
you're not using Smoothwall's web proxy server...
Installing the Components:
SCP the sw_squidlogs.tgz tarball into /tmp
on your Smoothwall, and untar it using
tar -zxvf /tmp/sw_squidlogs.tgz -C /
Make a copy of the page title image:
cp /home/httpd/html/ui/assets/3.6/img/pagetitles/page-proxylog.dat.png \
/home/httpd/html/ui/assets/3.6/img/pagetitles/page-squid.png
Edit /var/smoothwall/main/ui/alertboxes.en.pl
and immediately after the line containing
'proxylog.dat' => '<font class=\'pagetitle\'>Web Proxy Log Viewer</font>
<br>Check logs for the web proxy service.',
insert a line containing the following (note that it should be on a single line):
'squid.cgi' => '<font class=\'pagetitle\'>Web Proxy Log Analysis Report</font>
<br>Analyse traffic through the web proxy service.',
Edit /var/smoothwall/langs/base.pl, and after
the line containing:
'ssweb proxy' => 'web proxy',
insert a new line containing:
'ssweb proxy log' => 'web proxy analysis',
Also in /var/smoothwall/langs/base.pl, immediately
after the line containing:
'proxy log viewer' => 'Proxy log viewer',
insert a new line containing:
'proxy log analysis' => 'Web proxy log analysis',
Edit /var/smoothwall/header.pl and immediately
after the line containing:
&subsectiontab('/cgi-bin/logs.cgi/proxylog.dat', $tr{'ssweb proxy'}, 0);
insert a line containing the following:
&subsectiontab('/cgi-bin/logs.cgi/squid.cgi', $tr{'ssweb proxy log'}, 0);
To restrict viewing access to the the graphs, edit
/home/httpd/cgi-bin/logs.cgi/.htaccess and add the following to the end:
<Files squid.cgi>
require user admin
</Files>
Testing Report Generation:
Test the report generation scripts by manually running them from a command prompt:
/usr/local/squid-logs/run.pl
and you shouldn't see any output, but the scripts should have created the following files:
/usr/local/squid-logs/report.txt
/usr/local/squid-logs/summary
The report.txt is a text version of the report,
and can be viewed from a command prompt using
more /usr/local/squid-logs/report.txt
Viewing the Report:
Now point a web browser at Smoothwall's web interface, and navigate to the
"logs" menu item, and then to the "web proxy analysis" menu item to
view the report.
Automate Report Generation:
Automate the report generation process by scheduling this
script to run every day, by adding the following to the end of
/etc/crontab:
# generate web proxy analysis report
00 3 * * * root /usr/local/squid-logs/run.pl > /dev/null
Note that you can run the report more frequently, but depending on the hardware
specifications of your Smoothwall box, the size of your proxy log, etc, it
can take quite a while to execute (partiuclarly towards the end of the week),
and tends to use quite a bit of CPU time.
Note:
Note that most of the Smoothwall fixes are likely to overwrite header.pl
with a newer version, so if you install a new fix, it may overwrite your changes.
As a result, you'll probably need to reapply your changes to
header.pl after installing any fixes.
Similarly, when doing a fresh installation of Smoothwall,
only make these changes once you've fully patched the installation.
References:
Scripts written and used by NLANR
Squid Web Proxy Cache
last updated 19 May 2005
|
