Home  |  Map  |  Index  |  Search

Convert to GutenPalm or to PalmDoc by Logreport, Egon Willighagen About the author: Besides working for the LinuxFocus project, Egon is also working for Logreport Foundation, which develops the lire software. Content: Introduction Installation Lire as a cron job Lire as a stand-alone tool Roundup Talkback form for this article

Analyzing your internet applications' log files

Abstract:

This article is the first in a series about using lire to analyze log files of internet server applications. This is not limited to one service, e.g. Apache, but is an integrated analyzer for many different services. Included are DNS, WWW and email. This article explains how to get started with lire. It discusses installation and configuration to generate reports.

Introduction

Most internet services have the ability to log their activity. For example, the Apache web server adds for each web page request a line with information to a log file. Depending on the log format the line includes information like the page that was requested, the size of the page, which web browser was used, and much more. In case of your email server, a similar log file is made. It contains the email address that sent the email, who received it, how large it was, etc. As a matter of fact, all internet services have this capability.

These log files contain a enormous amount of information, but the format is hard to interpret by hand. You need a tool that makes summaries of the data to help you analyze the content. In case of www services this converts to TopX lists for web browsers, domains and platforms, and a hits versus time plot. Most counters from third party products show these kinds of overviews.

For most log file type tools are available to analyze the content. lire is such a tool. But lire is different from most other tools. Lire is an integrated system which is able to analyze not just one type of internet service, but many. And the reports that summarize the interesting information from the log file are plugged in. You can add custom report types yourselves.

Lire can be used in different ways. You can run it from the command line or have a crontab job installed that send you reports by email. In the former case you can choose the output format for the report. Current output formats include plain text, HTML, DocBook, PDF and LogML. In case of the crontab the only format is plain text at this moment.

Currently, the log files for these services can be analyzed:

• www
  • Apache
  • Boa
• email
  • Exim
  • Postfix
  • Sendmail
  • Qmail
• dns
  • Bind8/9

Lire is in full development, currently with three payed people working on it. Support is one of their tasks, so if you have a special request (new services for example) or general support questions please leave a message on LogReport's SourceForge site.

Installation

./configure
make
make install

The configure program has options:

If you plan to make use of output formats other than plain text, you need XML tools installed like the DocBook XML stylesheets, Jade, and XML processor like Xalan or Sablotron. With the third option for the configure program you can tell it where your SGML files are installed.

Lire as a cron job

Did you start this script as this user?

The script needs to be run by a user which is able to read the log files. The log files are often only readably to root and the users in the group adm. Check if the user is able to read the log files. An example:

~> cd /var/log
/var/log> ls -al apache/access.combined.log
-rw-r----- 1 root root 70902 Jul 27 13:23 apache/access.combined.log
/var/log> whoami
egonw
/var/log> groups
egonw adm dialout

In this example the current user egonw cannot read the Apache log file. Only root can currently read the file. The user is, however, member of the adm group so we can solve this problem by having the root change the group for the log file:

/var/log# whoami
root
/var/log# chgrp adm apache/access.combined.log
/var/log# ls -al apache/access.combined.log
-rw-r--r-- 1 root adm 70902 Jul 27 13:23 apache/access.combined.log

Temporary files are by default not kept. Do you want to keep these?

For normal use you should say no.

Are you planning to run an online responder?

The online responder is a tool that makes it possible to send your log file as email to this responder and get a report back from it. This is useful for administrators of larger networks. For normal use you should say no.

By default, temporary files are stored in ~/tmp. Do you want to change this?

By default the temporary files are kept in your home dir. You can overwrite this setting by answering "yes" and giving the directory you want them to be written. Keep in mind that this has to be a directory to which the user running lire must have write permission.

Do you want to sent a disclaimer with the generated reports?

If you send your reports to others, you might want to add a disclaimer and answer "yes" to this question. If the reports are only for internal use, you can answer "no" here.

By default, status and error messages are sent to stderr. Do you want to change this to syslog?

By answering "yes" to this question, you choose to have error and information messages sent to logger(1) instead of standard error (STDERR). You must have syslog's logger(1) installed then.

By default, informative and debug messages are suppressed. Do you want to run the system in debug mode? For normal use, you should say no.

Do you want to use some default settings for all services?

For each service you can set an email address and subject separately. For convenience you can also set a global default. Answer "yes" to set this default:

Would you like to use a default to-address?

If you answer "yes", this email address will be used to send reports to. It can be overwritten for specific services.

Would you like to use a default subject stem for the generated email?

If you answer "yes", you can give a template from which the email subjects are made. The default looks like: "[LogReport] www / apache report" for reports on Apache log files.

Which services are running?

All further questions ask you about which services you want lire to report about. The questions are all alike, and will not be discussed one by one. But for each service the questions look like:

Are you collecting ***** logfiles on this host which you'd like to get processed?

The programs lire does support are grouped in so called super services. The first question for each super service is always if you are collecting log files of some program that you want to have lire report about. If you answer "no", it will skip further questions about this super service. The super services lire currently supports are given in the Introduction of this article. There is one that was not mentioned then: apachemodgzip. This one deals with log files about Apache running in compressed mode.

The configuration program will then ask you questions about which services are run one-by-one. For email, for example, these services might be exim, postfix, qmail and sendmail. For each program lr_config asks you where lire can find the logfiles. These might be compressed, because lire will ask you if you want to specify with which program it should read the logfiles: zcat or gunzip -c can thus be used.

Are any of your ***** logfiles rotated weekly?

If you answer "yes" the crontab jobs for lire will be configured to run only once a week. The other option is daily.

If all questions are answered, it writes all setting to configuration files (*/defaults.local). These files are used by the lire cronjob. Unfortunately, this cronjob is not activated automatically. You need to do this yourself, by typing the one-liner:

~> { echo '0 10 * * * /usr/local/logreport/bin/lr_cron daily';
     echo '0 10 * * 0 /usr/local/logreport/bin/lr_cron weekly'; } | crontab -

Once activated like this, report(s) will be sent on a weekly and/or daily basis. You can fine tune your configuration, by editing the configuration files mentioned above.

Lire as a stand-alone tool

~> lr_run lr_log2report /tmp/err www apache combined <
   /var/log/apache/access.combined.log > apache.txt

The lr_log2report program takes at least three arguments. The first one is a file to which errors are redirected. In this case /tmp/err. The second argument is the super service of the log file. There are currently three such super services: www, dns and email. The third required argument is the service, that is the type of log file. In this case it is an Apache combined file, so the service is apache. There is an additional argument which is a flag that says that the apache log file uses the "combined" format. Like with most lire programs, you can get additional information with man:

~> man lr_log2report

To change the output format you can use the "-i <format>" option. Valid formats currently are txt (default), logml (LogML website), DocBook XML (DocBook Open Repository), HTML and PDF. The latter two need DocBook stylesheets and Jade installed. All formats except plain text also need a XSLT processor like Gnome's xsltproc For example, to generate PDF:

~> lr_run lr_log2report -o pdf /tmp/err www apache combined
   < /var/log/apache/access.combined.log > apache.pdf

Roundup

This article shortly introduced the lire project, and showed you how to install and configure the software. The next articles in this series will further describe the software and how to use it. For now you can find more information about it at our web site: http://www.logreport.org/.

If you want to get in touch with us, you can best meet us at IRC. The developers can often be found at the #logreport channel at the OpenProjects.org IRC network. Questions, comments, and support requests are welcomed.

Talkback form for this article

2001-08-20, generated by lfparser version 2.17