Log Analysis

Discuss and get help to implement a CacheGuard Gateway into your networks
miguelp
Posts: 46
Joined: 17 Aug 2015 13:06

Log Analysis

Post by miguelp »

Hi,
Now I'm able to get web access log via TFTP.
Any recommendation of software to use to generate reports based on that log file ?
Thanks,
Miguel
User avatar
david
Posts: 163
Joined: 08 Aug 2015 20:38

Re: Log Analysis

Post by david »

Dear Miguel

Logs are generated using a common format which should be recognised by almost all known log file analysers. Unfortunately I can't recommend you a software in particular.

Best Regards,
David Janeway
CacheGuard Technical Team
https://www.cacheguard.com
miguelp
Posts: 46
Joined: 17 Aug 2015 13:06

Re: Log Analysis

Post by miguelp »

Dear David,

I have 2 more questions:

- What is the standard format of the log?
I've tried with:
# 1 - Apache or Lotus Notes/Domino native combined log format (NCSA combined/XLF/ELF log format)
# 2 - IIS or ISA format (IIS W3C log format). See FAQ-COM115 For ISA.
# 3 - Webstar native log format.
# 4 - Apache or Squid native common log format (NCSA common/CLF log format)
but is not any of this.

- Can you configure the log format ?, for example a field delimiter would be very useful

This is how my logs looks like, just in case.

Thanks,
Miguel


192.168.15.62 - [21/Aug/2015:13:41:58 -0400] "POST http://clients1.google.com/ocsp HTTP/1.1" 200 813 TCP_MISS HIER_DIRECT
192.168.15.62 - [21/Aug/2015:13:41:59 -0400] "POST http://clients1.google.com/ocsp HTTP/1.1" 200 813 TCP_MISS HIER_DIRECT
192.168.15.62 - [21/Aug/2015:13:42:00 -0400] "POST http://clients1.google.com/ocsp HTTP/1.1" 200 813 TCP_MISS HIER_DIRECT
192.168.15.62 - [21/Aug/2015:13:42:07 -0400] "POST http://ocsp.digicert.com/ HTTP/1.1" 200 855 TCP_MISS HIER_DIRECT
192.168.15.62 - [21/Aug/2015:13:42:11 -0400] "CONNECT www.google.com:443 HTTP/1.1" 200 0 TCP_MISS HIER_DIRECT
192.168.15.62 - [21/Aug/2015:13:42:11 -0400] "CONNECT www.google.com:443 HTTP/1.1" 200 0 TCP_MISS HIER_DIRECT
192.168.15.62 - [21/Aug/2015:13:42:11 -0400] "CONNECT www.google.com:443 HTTP/1.1" 200 0 TCP_MISS HIER_DIRECT
User avatar
david
Posts: 163
Joined: 08 Aug 2015 20:38

Re: Log Analysis

Post by david »

Hello Miguel

You can find the log format at http://www.cacheguard.net/doc/command/log.html. This format is used by Squid (an open source proxy).

Actually the delimiter is space and the request field which may contain spaces is enclosed with quotation marks.

Best Regards,
David Janeway
CacheGuard Technical Team
https://www.cacheguard.com
miguelp
Posts: 46
Joined: 17 Aug 2015 13:06

Re: Log Analysis

Post by miguelp »

Hello,
Thanks, I've tried http://www.webalizer.org that supports squid format but does not recognize the file. Do you know any software that works with the format of CG ?

Now that AD auth. is working my log file looks like:

192.168.110.4 prueba [07/Sep/2015:22:57:27 +0300] "GET http://www.ibm.com/ HTTP/1.1" 302 221 TCP_MISS HIER_DIRECT
192.168.110.4 miguelp [07/Sep/2015:22:58:10 +0300] "GET http://www.eldeber.com.bo/files/article ... 0-420.jpeg HTTP/1.1" 200 5146 TCP_MISS HIER_DIRECT

Cheers,
Miguel
User avatar
charles
Site Admin
Posts: 41
Joined: 06 Nov 2014 16:23
Location: Paris
Contact:

Re: Log Analysis

Post by charles »

Dear Miguel

Actually CG uses a custom log format which is similar to Apache log format with additional information related to the caching provided by Squid. As David mentioned the CG log format is described at http://www.cacheguard.net/doc/command/log.html.

Webalizer allows you to use a CLF (Custom Log Format). The LogType and ApacheLogFormat statements should allows you to configure the right format to use. Please test the following webalizer configuration:

LogType clf
ApacheLogFormat %h %u %t \"%r\" %>s %b - -


Refer to http://www.stonesteps.ca/projects/webalizer/README.asp for further information.

Best Regards,
Charles Tajvidi
IT Technical Architect
http://www.cacheguard.com
miguelp
Posts: 46
Joined: 17 Aug 2015 13:06

Re: Log Analysis

Post by miguelp »

Hello Charles,

This command:
ApacheLogFormat %h %u %t \"%r\" %>s %b - -

Should be set in the Apache server, is not a directive for webalizer.

(see below extract from: ftp://ftp.mrunix.net/pub/webalizer/README)

That´s why I'm asking if I can configure the format of the log of the CG.

I've tested webalizer with:
LogType clf
and
LogType squid

and it tells me:
Skipping bad record for all the records in my log file.

Any ideas ?
Thanks,
Miguel








CLF format logs by default. For Apache, in order to produce the
proper log format, add the following to the httpd.conf file:

LogFormat "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\""

This instructs the Apache web server to produce a 'combined' log
that includes the referrer and user agent information on the end of
User avatar
david
Posts: 163
Joined: 08 Aug 2015 20:38

Re: Log Analysis

Post by david »

Dear Miguel

I think that Charles is talking about "Stone Steps Webalizer" (http://www.stonesteps.ca/) which is a fork of Webalizer. Webalizer development was stopped in 2002. I suggest you to upgrade to Stone Steps Webalizer which allows you to use the ApacheLogFormat statement as described by Charles.

Best Regards,
David Janeway
CacheGuard Technical Team
https://www.cacheguard.com
miguelp
Posts: 46
Joined: 17 Aug 2015 13:06

Re: Log Analysis

Post by miguelp »

Hello David,
Thanks for the answer, but still not working.

I suppose this issue belongs to the Stone Steps Webalizer forum, I've posted there:

https://stonestepswebalizer.codeplex.co ... ons/644734

If you have any ideas, let me know.

Thanks,
Miguel
miguelp
Posts: 46
Joined: 17 Aug 2015 13:06

Re: Log Analysis

Post by miguelp »

Hello,

I will really appreciate your help, because the other forum (stone steps), is totally inactive.

Thanks,
Miguel.
Post Reply