Per chi come me ha dovuto mettere in piedi il logserver per il DPS (Documento programmatico sicurezza) ho deciso di mettere in rete questa guida sommaria su come implementare attraverso software opensource il log server con Ubuntu Server 8.04 e comunque funzionante anche con Ubuntu Server 9.10 ed il software logzilla che attraverso il database mysql registra ogni collegamento come amministratore ai server Microsoft e/o Linux.
About
This page will help guide you through a standard installation process.
For this demo, I will be using version (v2.9.9o) located at The Google code site
If you’re looking for an install guide for v3.0, go to the Install Guide for LogZilla v3.0.
Name Change
You may have noticed that I’ve started using the name LogZilla.
Moving forward, the php-syslog-ng application will be changing it’s name to LogZilla in order to facilitate future plans of moving to an Ajax front-end.
Obtaining LogZilla
Sudo to root and then change directories to your web root(we’ll want to do most of this as root, so su first):
sudo su -
cd /var/www
Download: Grab the latest package from The Google code site and use wget to download it to your local system:
wget http://php-syslog-ng.googlecode.com/files/logzilla_v2.9.9n.tgz
Extract:
tar xzvf logzilla_v2.9.9n.tgz
Rename the php-syslog-ng directory to logzilla (see #Name_Change)
mv php-syslog-ng logzilla
Make sure you have your log directory created for system logs:
mkdir -p /var/log/logzilla
Requirements
Please note that you should have a decent understanding of Linux, Apache, Mysql and PHP before attempting to use this software. I’ve made every attempt to make it as easy as possible, but due to the nature of this stuff, it does require a decent skill base to implement and operate properly.
The following tools/software must be installed in order to use Php-Syslog-NG:
Syslog-ng
This logging system is based on data collected from a program called Syslog-NG which is “is an open source implementation of the Syslog protocol for UNIX and UNIX-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using TCP for transport.”
In the case of LogZilla, we’re going to use it to collect syslog messages from our network and store the information into a MySQL database for reporting capabilities.
Here’s a simple diagram of how these pieces fit together: 
Installing syslog-ng
This section covers syslog-ng using Ubuntu, if you use a different distro you’re on your own…
From a console, type:
sudo aptitude install syslog-ng
You will likely see an error in ubuntu like this:
Remove the following packages:
ubuntu-minimal
It’s a bug in Ubuntu and it’s safe to continue.
Once the syslog-ng installation completes, modify /etc/syslog-ng/syslog-ng.conf and add the following lines to the bottom:
NOTE: a copy of this config file is located under scripts/contrib/system_configs
#
# http://nms.gdd.net/index.php/LogZilla_Installation_Guide#Installing_syslog-ng
# This config works with v2.x of syslog-ng, you will need to make a few changes to make it work with v3.x
# for v3.x - change the following entries in your syslog-ng config:
# Change:
# source(s_all);
# to:
# source(s_local);
# source(s_net);
# destination(d_logzilla);
#
# http://www.syslog.org/syslog-ng/v2/
# modify /etc/syslog-ng/syslog-ng.conf and add the following lines to the bottom:
###########################################################################################
# Clay's LogZilla config below
###########################################################################################
# July 20, 2009 Added by cdukes for LogZilla
###########################################################################################
options {
long_hostnames(off);
# doesn't actually help on Solaris, log(3) truncates at 1024 chars
log_msg_size(8192);
# buffer just a little for performance
# sync(1); <- Deprecated - use flush_lines() instead
flush_lines(1);
# memory is cheap, buffer messages unable to write (like to loghost)
log_fifo_size(16384);
# Hosts we don't want syslog from
#bad_hostname("^(ctld.|cmd|tmd|last)$");
# The time to wait before a dead connection is reestablished (seconds)
time_reopen(60);
#Use DNS so that our good names are used, not hostnames
use_dns(yes);
dns_cache(yes);
#Use the whole DNS name
use_fqdn(yes);
keep_hostname(yes);
chain_hostnames(no);
#Read permission for everyone
perm(0644);
# The default action of syslog-ng 1.6.0 is to log a STATS line
# to the file every 10 minutes. That's pretty ugly after a while.
# Change it to every 12 hours so you get a nice daily update of
# # how many messages syslog-ng missed (0).
# stats(43200);
};
# Create destination to LogZilla
destination d_logzilla {
program("/var/www/logzilla/scripts/db_insert.pl"
template("$HOST\t$FACILITY\t$PRIORITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n")
);
};
# Tell syslog-ng to log to our new destination
log {
source(s_all);
destination(d_logzilla);
};
# Clay's LogZilla config above
###########################################################################################
Also, the default syslog-ng installation on Ubuntu has UDP disabled by default, so be sure to change this:
# udp();
To this:
udp();
Somewhere around line 93 of the file…
Note: Change the paths above to reflect your actual installation path!
Installing All Prerequisites
We’ll need to install:
- Apache
- PHP
- MySQL
- php-gd (for graphs)
- php-cli for (for command line scripts)
- php5-mysql
- msttcorefonts (for graph fonts)
- build-essential (you’ll need this later for building perl modules and other system stuff)
so type:
sudo aptitude install apache2 php5 php5-gd php5-cli php5-mysql mysql-server msttcorefonts build-essential
From the command line to install everything at once.
(Yes, it’s that easy – see why I like Ubuntu?)
You will also need to install the LevenshteinXS perl module in order to use this version.
To install LevenshteinXS from the CPAN archive, simply type:
sudo cpan Text::LevenshteinXS
Apache
Naturally, if we’re going to use a web interface, we’ll need a web server :-)
First, edit the /etc/apache2/apache2.conf file and add a ServerName directive
sudo vi /etc/apache2/apache2.conf
When you’re done, it should look like this (replace logzilla with your server’s name):
ServerRoot "/etc/apache2" <<- Existing line
ServerName logzilla
Replace 'logzilla' with your actual server name
Note: You can skip the following section if you are installing to the root web server.
Not all systems will need to use the following config. This is only provided as an example.
Next, create a file in /etc/apache2/sites-available called “logzilla” and add the following to it:
# LogZilla
Alias /logs "/var/www/logzilla/html/"
<Directory "/var/www/logzilla/html/">
Options Indexes MultiViews FollowSymLinks
AllowOverride All
Order allow,deny
Allow from all
</Directory>
Note: AllowOverride should be set to “All” so we can modify php variables using a .htaccess file (more on that later)
Now save this file in the appropriate location. For Ubuntu users, it would be saved as:
/etc/apache2/sites-available/logzilla
Then, for Ubuntu, you would type:
a2ensite logzilla
If you are on a distro other than Ubuntu, you’ll have to look up the documentation to see how to implement this (you might be able to simply add it to the default config file)
Now restart Apache and make sure you don’t see any errors.
sudo /etc/init.d/apache2 restart
* Restarting web server apache2 [ OK ]
Finally, go browse to http://<ipaddress>/<directory> and see if it’s working…
MySQL
LogZilla is designed to work with Mysql v5.x
It may work fine with older versions, but I’m not sure so stick with 5.x when possible.
You shouldn’t have to do anything from the command line for installing MySQL other than what we did in the prerequisites section.
All table modifications will be made for you during the LogZilla install.
PHP
This section covers PHP using Ubuntu, if you use a different distro you’re on your own…
LogZilla is designed to work with PHPv5.x
If you are using PHP v4, and plan to use the CEMDB (Cisco Error Message Database), then you will need to convert the CEMDB.class file to an older format as noted in the issues list on the code site:
http://code.google.com/p/php-syslog-ng/issues/detail?id=13&can=1&q=cemdb
If you don’t, you will get errors like this:
Parse error: syntax error, unexpected T_STRING, expecting T_OLD_FUNCTION or
T_FUNCTION or T_VAR or '}' in
/var/www/php-syslog-ng/includes/CEMDB.class.php on line 11
After you’ve installed everything as noted in the prerequisites section above, edit the /etc/php5/apache2/php.ini file and change a few settings that we’ll need during the web portion:
sudo vi /etc/php5/apache2/php.ini
Change:
memory_limit = 16M
to:
memory_limit = 128M
Change:
max_execution_time = 30
to:
max_execution_time = 300
You should also do the same for /etc/php5/cli/php.ini
Don’t forget to restart apache to start using PHP
sudo /etc/init.d/apache2 restart
* Restarting web server apache2 [ OK ]
PERL
Perl is used for some backend functions – most of these files are located in the scripts/ directory.
It’s important that you have a working copy of perl installed.
If you’re using a minimal Ubuntu system and you want to run the dbgen.pl script (to populate the database with some sample data), you might need to add Digest::SHA1 and Net::MySQL PERL modules to the default installation.
See the DBGen section for more details.
Permissions
Make sure you set the html/ directory to the apache web owner.
For example, in Ubuntu you would do:
chown -R www-data:www-data /var/www/logzilla/html
Misc Requirements
While not absolutely necessary to have a working system, the following tools/programs should be added for maintaining your operating system:
Logrotate
A sample logrotate.d script is included in the scripts/contrib/system_configs/ directory which should (depending in your distro) be placed in /etc/logrotate.d
The file contents should be similar to:
# http://nms.gdd.net/index.php/LogZilla_Installation_Guide#Logrotate
# LogZilla logrotate snippet for Ubuntu Linux
# contributed by Clayton Dukes
#
/var/log/logzilla/*.log {
missingok
compress
rotate 5
daily
postrotate
/etc/init.d/syslog-ng reload > /dev/null 2>&1 || true
endscript
}
Note, again, that you will want to be sure you’ve created the directories used to store log data:
(you can skip this step if you already did it before)
mkdir -p /var/log/logzilla
Cron
Cron is used for regular data maintenance, the following entries should be made in root’s crontab by typing
crontab -e
as the root user on your system
# http://nms.gdd.net/index.php/LogZilla_Installation_Guide#Cron
# LogZilla
@daily php /var/www/logzilla/scripts/logrotate.php >> /var/log/logzilla/logrotate.log
@daily find /var/www/logzilla/html/jpcache/ -atime 1 -exec rm -f '{}' ';'
0,5,10,15,20,25,30,35,40,45,50,55 * * * * php /var/www/logzilla/scripts/reloadcache.php >> /var/log/logzilla/reloadcache.log
# Demo LogZilla
# CHANGE TO MATCH YOUR DIRECTORY PATHS
# @hourly /var/www/logzilla/scripts/dbgen.pl >> /var/log/logzilla/dbgen.log
Advanced Features
This section covers advanced options for implementation
Authentication Methods
Some alternate authentication methods are available which include:
LDAP
To enable LDAP, simply edit the config.php file located in config/ after the install is complete and set the following variables:
define('LDAP_ENABLE', "NO");
define('LDAP_SRV', "ldap.company.com");
define('LDAP_BASE_DN', "ou=active, ou=employees, ou=people, o=company.com");
define('LDAP_CN', "uid");
MS AD
To enable MS Active Directory, simply edit the config.php file located in config/ after the install is complete and set the following variables:
define('LDAP_MSAD', "NO");
define('LDAP_DOMAIN', "mydomain.com");
Web Basic Auth (htaccess)
To enable htaccess support using mod_auth_krb5, simply edit the config.php file located in config/ after the install is complete and set the following variables:
define('WEBBASIC_ENABLE', FALSE);
Note that you will still need to add the appropriate user to the MySQL database before this will work.
For assistance with setting up apache .htaccess, please visit http://home.golden.net/htaccess.html
For more information on this feature, please visit http://code.google.com/p/php-syslog-ng/issues/detail?id=62
Squeeze
Squeeze is a de-duplication feature of LogZilla. It works great, but has some caveats that you should be aware of prior to using it.
Squeeze uses an algorithm similar to Levenshtein’s and is used to compare the number
of “edits” it would take to make each of the compared strings become the same.
The result is displayed as a “distance” (a variable you can alter in the config.php file)
This allows LogZilla to compare incoming messages to those stored in the database.
When a message arrives, SQZ will check the incoming message and compare that to messages in the database that are from the same:
- host
- facility
- priority
- level
- tag
- within the last X minutes (this window is configurable in config.php)
If rows in the database are found to match the incoming message, the Squeeze function will update the original row
with a new “First Occurrence”, “Last Occurrence” and “Count” to reflect the new information and will discard
the incoming message and any other duplicates in the database it finds (within the specified time range).
Caveats
- If your company has to comply to PCI standards, then de-duplication is not for you (you may, however, consider
splitting up messages to two separate servers, one for analyzing and reporting, and one for storing ALL messages for PCI records)
- If you are getting a lot of messages per day (millions), de-duplication may actualy be too slow for you to use since it
has to do table lookups for each incoming message.
I would encourage you to at least test it though as it does provide some major advantages.
Advantages
Enabling the Squeeze feature allows you to save important DB storage space to allow for much faster analyzation and reporting.
For example, a recent analysis of a customer of mine had the following in just 5 days worth of logs:
- There were 675 Hosts with a total of 675109 messages.
- Of the 675109 messages, 99.84% were duplicates
- Their Top 10 reporting devices had more than 25,000 single messages repeated – the top device had almost 60,000.
This means that if they were de-duplicating events before they got stored into the Syslog server’s database,
then only 2846 rows would be used in that database instead of 675109 individual rows.
The benefits here should be obvious…searching across 3,000 rows is a lot faster then searching through 700,000…
Before implementing this feature, please be aware of the potential damage you can do (i.e. deleted rows).
I encourage you to read more about it here
DBGen
The dbgen.pl file located in the scripts/contrib/dbgen/ directory is used to generate random fake events for testing.
This file can be used to test whether or not LogZilla is working without having syslog-ng installed. It’s written in PERL and you may need to install a couple of modules in order for it to work properly.
Generate a few fake events to verify that the LogZilla part works:
sudo perl /var/www/logzilla/scripts/contrib/dbgen/dbgen.pl
I got an error on my system:
Can't locate Net/MySQL.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.8 /usr/local/share/perl/5.8.8 /usr/lib
/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at /var/www/logzilla/scripts/contrib/dbgen/dbgen.pl line 22.
BEGIN failed--compilation aborted at /var/www/logzilla/scripts/contrib/dbgen/dbgen.pl line 22.
Which means I don’t have the proper perl libraries installed, so I used cpan:
cpan -i Digest::SHA1 (required to install Net::MySQL)
cpan -i Net::MySQL
NOTE: If you get a make error, you probably have a new system and haven’t installed the Ubuntu build-essentials package.
sudo apt-get install build-essential
Now try dbgen.pl again:
sudo perl /var/www/logzilla/scripts/contrib/dbgen/dbgen.pl
And viola’
Debug off, showing only inserted data...
Host: server-5
Facility: authpriv
Priority: debug
Level: alert
Tag: Tag
YMDHMS: 2008-05-10 21:42:38
Program: DBGen
Message: DBGen: %FABRIC-5-FABRIC_MODULE_ACTIVE: Module 3 is online
Affected row: 1
Now check your url to see the entries:
http://<url>
Logreplay
I’ve included a new in script in version 2.9.9 that will allow you to “replay” a log file taken from another server. The script is located in:
- /var/www/logzilla/scripts/contrib/logreplay/logreplay.pl
To use the sample logs included simply un-gzip it:
gzip -d syslog.sample.gz
And run the logreplay script:
./logreplay.pl -h
Which will give you help on the script:
This program is used to replay a standard *Cisco* syslog dumpfile into the local syslog receiver (syslog-ng)
usage: ./logreplay.pl [-hvfs]
-h : this (help) message
-v : verbose output
-f : Filename to import (required)
-s : path to the spoof program (required)
example: ./logreplay.pl -v -f ./syslog.sample -s ./spoof
So, to run it, you would do:
./logreplay.pl -v -f ./syslog.sample -s ./spoof
The “spoof” program that I’ve included will rewrite the outgoing syslog packet and insert the hostnames
from the syslog.sample file so that when syslog-ng receives the messages they appear to come from that host instead of your local machine.
I did not include the source code for “spoof” in the distribution because it could be used maliciously by bad people to insert fake events into other systems.
If you want a copy of the source, please email me and explain why you’d like to have it and I’ll be happy to send it to you.
Installation Process
This section will guide you through a step-by-step installation process, the end of the section includes screencasts so that you can get an idea of how it should work if things go wrong :-)
NOTE: Before starting this process, be sure you’ve completed the items listed in the #Requirements section.
Web-based Install
Make sure everything on the pre-installation check screen is green, if not, fix it before continuing!
Screen 1: Click Next at the top right to begin install
Screen 2: Accept the license agreement and click next
Screen 3: Enter the mysql ROOT user’s password Leave everything else as default unless you really need to change something. (You may want to uncheck the “install sample data” box) Click Next to continue:
Click ok to accept the notice and continue…
Screen 4: Enter a site name, eg: “My Syslog Server” and click Next
Screen 5: Leave the default fields as is unless necessary. You may need to change the ‘Site URL to something like /logs/ Enter email address into email field Enter a password for the local admin acount or leave the random one there (but write it down so you can get into the site later!) When you’re done, click next
Screen 6: If you opted to install the CEMDB, then you will be presented with the following screen:
click Install CEMDB to continue…
special note for Internet Explorer users: 2 people have reported that this button (Install CEMDB) does not work for them. You will need to use Firefox in order to make it work, or manually import the sql data from the command line. Sorry, maybe someday MS will decide to make a standards-based browser, or someone will fix this incompatibility :-)
If you need to, installing the CEMDB data from the command line is very easy.
Once you’ve completed the steps above, if clicking on the Install CEMDB button doesn’t work:
mysql -usyslogadmin -psyslogadmin syslog < /var/www/logzilla/html/install/sql/cemdb.sql
Be sure to replace “syslogadmin” and “syslog” with your username, password and database name.
Assuming the button works as expected, you should now see a page that looks like the following: Click the “Start Import” link to begin inserting CEMDB data into the database.
Once it completes, click on “Continue” near the bottom.
Once you click continue, you should be at the main login screen:
Main Site: Login using admin and the password you selected on screen 5
If you installed the sample data there will be a couple of entries, if not, you’ll get an error message like this:
Not to worry, that error just means that log messages have yet to arrive in the database.
This concludes the Web based portion of the install, hooray!
Console Portion (part 2)
After the web-based install is completed, you will need to modify the paths set in some of the scripts in the scripts/ directory, so run the following script:
cd /var/www/logzilla/scripts
./fixpaths.sh
This script will automatically update all paths for files located in your logzilla directory.
Screencasts
This screencast walks through a normal installation of the Web-based portion of the install:
http://www.youtube.com/watch?v=Q99ovrHWzsE
Caveats
If you get an error running logrotate.php that looks like this:
Query failed: DROP,ALTER command denied to user 'syslogadmin'@'localhost'
Then try running this in mysql:
USE mysql;
update user set Drop_priv='Y',Alter_priv='Y' where User='syslogadmin';
FLUSH PRIVILEGES;
Please note, that because there are many changes in this release, I would highly recommend that you start with a fresh install. If you need to leave your old server running in parallel, please do that rather than trying to upgrade if you can.
Upgrade Procedures
If I *had* to, here’s the method I use to upgrade my servers:
1. Rename the current directory to .old
mv logzilla logzilla.old
2. Untar the new one
tar xzvf php-syslog-ng.<ver>.tgz
3. Get a backup of your whole syslog DB in case something goes wrong:
mysqldump syslog > syslog.sql
5. Get a text listing of all your current log tables
echo "show tables" | mysql syslog | grep "^logs" > list
6. Dump a sql file for each log table in the “list” file created in step 5
for table in `cat list`; do mysqldump syslog $table > $table.sql; done
7. Do a fresh install and tell it to drop the old database
8. Verify that new install works properly
9. Import old data
for t in `ls logs*.sql`; do mysql syslog < $t ; done
Verify everything works as expected. If something blows up, you can always just import your original syslog.sql file :-)
Special note:
Some versions are incompatible with old releases so this method may not work for you.
This is the case with v2.9.9 – I’ve made several changes to the table structures in this version so following the method outlined above will not work.
I would highly recommend you just install the new version and leave the old one running until you no longer require the data in it.
Appendix/FAQ
If you ever need to reset the admin password, you can do it from a mysql shell:
update users set pwhash=md5('MYNEWPASSWORD') where username='admin';
nnadalca Documenti Tecnici, Tips Linux e Microsoft, Tutorial DPS, logzilla
![[Schema di Rete]](http://www.pluto.it/files/journal/pj0704/images/fw-pfsense-net.png)
Commenti recenti