

dlog v1.0.0


If you're in a hurry, read INSHORT first - but you may need to come back
again :-)


Introduction
============

This suite of logfile analyzers examine the logfiles of the following tools: 
axfrdns, dnscache, qmail, qsmtp (the tcpserver output for qmail-smtpd), 
rbldns, tinydns and publicfile if they use multilog to store their logs.

The suite is broken up into many pieces, and for fully understanding what 
you should use and when, please continue reading. 



Usage 
=====

dlog can be used in three different ways:

1)
You only use the data-collecting tools and do the reports and graphing 
with some other tools, like munin (plugins for munin are provided in 
this package).

2)
You use the data-collecting tools and use dlog to store the data in rrdtool 
databases. You will create your own graphs and webpages to make use of the 
data. This requires some scripting on your part - but you can tailor your
graphs.

3)
You use the data-collecting tools and use dlog to store the data in rrdtool 
databases. Then you configure different parts of dlog to run at given 
intervals to provide webpages and graphs.

As this README will be concentrating on method 3, you should refer to the
SPECS if you choose method 1 or 2, after finishing this document.



Requirements
============

Tools:

1)
	Perl, Lex, Yacc, make & a C compiler for compilation

2)
	Perl, Lex, Yacc, make, a C compiler for compilation and Perl & 
	RRDtool afterwards

3)
	Perl, Lex, Yacc, make, a C compiler for compilation and Perl, rrdtool, 
	sort, uniq & head afterwards (sort, uniq and head are only required 
	if you use the list-generating tools).

As long as you have a POSIX compliant make, Lex, Yacc and C compiler, dlog 
should work fine. The sort, uniq and head commands are (as far as I know) 
part of most UNIX systems. 


Configuration:

The only requirement for your environment is that your logfiles are created 
by multilog with the "t" parameter - usually the log service for a tool is 
being started like this:

exec setuidgid <username> multilog t ./main


Disk space:

You need a considerable amount of free space to keep the data that dlog
generates. It's difficult to give an estimate, as it depends on how you use
dlog, but the rrdtool databases should not take more than 400MB even if use 
every tool provided in this package. If you use the "list" feature you are 
completely on your own as it solely depends on the size of the input you
give it - but here you need approximately two times the size of the input 
available to generate the lists you want.



Compiling and Installing
========================

$ tar xvzf dlog-1.0.0.tar.gz
$ cd dlog-1.0.0
$ perl configure.pl
$ make
...
$ make install


Using dlog
==========

There are two different parts in dlog. One simply sums up the activity, 
and saves the data in rrdtool databases. The other generates lists from 
your logfiles, and tells you exactly which clients, queries or domains are 
most active. 

dodlog.pl can do all the work for you. During the installation, all 
adjustable parameters should have been set correctly, but they can be
altered afterwards.
The most important parameters are the location of the RRDTool databases 
($rrddatabasebase) and where dodlog should put stat-files and graphs 
($statfiles).
To make full use of the graphing and reports that dlog provide, you
should have the $statfiles directory reachable through a webserver.


Enough already.. Examples, please!

./dodlog.pl tinydns init
	- This will create an rrd database for holding tinydns statistics

./dodlog.pl tinydns update /service/tinydns/log/main/ 
	- This will update the rrd database with information from tinydns
	  logfiles within the last 5 minutes (you can specify timestamps
	  if you need to, but you it asssumes you want the last 5 minutes
	  if you don't specify anything).

./dodlog.pl tinydns graph
	- This will create graphs for tinydns. Graphs for the last 24 hours,
	  week, month and year are created and stored in $statfiles. 


(
Instead of tinydns, the following tools can be specified: axfrdns dnscache 
publicfile qmail qmailqueue qsmtp rbldns. But if you specify qmailqueue you
have to specify the path to qmail-qstat instead of the logfile directory, e.g:

./dodlog qmailqueue update /var/qmail/bin/qmail-qstat
)

The init command should only be issued once. If you run it again, all the
statistics in the tinydns rrd database will be erased. When the data is a 
year old, RRDTool deletes the information. So don't worry about cleaning up.

The update can be executed whenever you want - but internally, the dodlog.pl
scripts works in 5 minute intervals. (You can change this but it has great 
effect on how RRDTool displays your data). The raw tools (everything tool
that starts with dlog) works with whatever interval you give at runtime.

The graph command can be issued as often as you like; but probably not 
more often than you run the update command. I run it every time I update
the statistics.

The tool you specify for dlog reflects which logfiles you are giving, and 
the names gives themselves except for qsmtpd which is used for analyzing 
the tcpserver output for qmail-smtpd.

There is also this derivative:

./dodlog.pl tinydns raw /service/tinydns/log/mail/
	- This will give you the data from the tool directly - the only 
	  difference between running the tool (in this case dlogtiny) 
	  directly is that here you don't have to specify that you want to 
	  analyze the last 5 minutes of data. Please read SPECS to make sure 
	  you use the tool optimally - this feature is primarily for 
	  integration with other tools.

Pfew. A few more tools to go.


Another part of dodlog.pl (and dlog in general) is the tools that generates
lists of which queries, clients or domains that are most active. 

Again, examples shows it best:

./dodlog.pl axfrdnsclients list /service/axfrdns/log/main 1036688500 1037293300

This will generate a list of which clients that within the last week, had most 
frequently requested information from the axfrdns setup. In the above example 
a text file will be created in the $statfiles directory named 
axfrdnsclients.1037293300.txt.

Similar statistics can be created for other services if you use pass other 
commands to dodlog.pl:

axfrdnsclients dnscacheclients rbldnsclients tinydnsclients

Another example:

./dodlog tinydnsqueries list /service/tinydns/log/main 1037206900 1037293300

This will also generate list, but it will contain a list of which queries 
that within the given period, had been most popular. 

Similar statistics can be created for other services with the following
commands:

axfrdnsqueries dnscachequeries tinydnsqueries rbldnsqueries

The last three commands to dodlog.pl are:

qmailadd qmaildom publicdom

They (believe it or not) also create a list. qmailadd creates a list of which 
email-addresses that are most active on your system - both in (From:) and 
out (To:). qmaildom creates a list of which domains are most active, and again
that goes for both in and out (if you only host one domain, that domain
will probably be the most active domain for ingoing mail). publicdom does
the same as qmaildom, but for publicfile logfiles.


Common for the list-generating tools is that you have to clean up after you 
made a report. I don't know if you want to save the reports, or throw them 
out immediately after you have seen them - so that's up to you.



Automation
==========

After you checked that dlog seems to run fine, you should automate the
update and graphing of the statistics. The following entries in your 
crontab should work for some of you, others will have to alter it a bit:

*/5      *       *       *       *       root    /path/to/dodlog.pl tinydns update /service/tinydns/log/main/ 
*/5      *       *       *       *       root    /path/to/dodlog.pl tinydns graph

I make reports about queries, clients and domains once a week:

*/5      *       *       *       *       root    /path/to/dodlog.pl tinydnsqueries list /service/tinydns/log/main/ `perl -e 'print time-606900, " ", time;'`



That's it
=========

I hope that get's you going. If you need further info, please see SPECS.. 
and check the website:

http://dlog.gal.dk/




Links
=====

Information about axfrdns, dnscache, qmail, rbldns, tinydns and publicfile can 
be found on D. J. Bernsteins page
http://cr.yp.to.

RRDTool is the work of Tobie Oetiker
http://www.rrdtool.org

qpsmtpd can be found here
http://qpsmtpd.develooper.com

