Log centralisation / mining - swinog - vmaill01.sys

List overview All Threads
Download

Log centralisation / mining

Problems reaching large Websites

RE: [swinog] Problems reaching...

Marcel Prisi

21 Jan 2008 21 Jan '08

8:47 a.m.

Hi all, I am looking for a good log centralisation / alerting / mining solution. I know about syslog-ng / rsyslog+phpLogCon, I'd like something more complete ... Something with a bit of realtime analysis (regexp ?) and correlation ... and a nice interface where you could get some useful details fast ... What solution do swinoggers use ?? Thanks !

Reply

Show replies by date

Olivier Beytrison

21 Jan 21 Jan

8:52 a.m.

Hello, Maybe have a look at splunk. It's not free, but it seems to do what you're looking for. I'd like to ask at the same time if anyone here is using it. Because I thinking about installing it on our network. So some feedbacks would be great. www.splunk.com Regards, Olivier B. Marcel Prisi a écrit :

Hi all, I am looking for a good log centralisation / alerting / mining solution. I know about syslog-ng / rsyslog+phpLogCon, I'd like something more complete ... Something with a bit of realtime analysis (regexp ?) and correlation ... and a nice interface where you could get some useful details fast ... What solution do swinoggers use ?? Thanks ! _______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Reply

Tobias Koenig

9:03 a.m.

Hi There is actually a free version of splunk, with a few restrictions though (no access controls). But you still can handle up to 500MB Log per day and it's quite easy to install and configure. The search engine seems to be quite powerful, we run it on a debian system with 256mb ram, got approx. 7'500'000 log entries on it and a searching for a host takes just a few secs. Regards Tobias Olivier Beytrison schrieb:

Hello, Maybe have a look at splunk. It's not free, but it seems to do what you're looking for. I'd like to ask at the same time if anyone here is using it. Because I thinking about installing it on our network. So some feedbacks would be great. www.splunk.com Regards, Olivier B. Marcel Prisi a écrit :

Hi all, I am looking for a good log centralisation / alerting / mining solution. I know about syslog-ng / rsyslog+phpLogCon, I'd like something more complete ... Something with a bit of realtime analysis (regexp ?) and correlation ... and a nice interface where you could get some useful details fast ... What solution do swinoggers use ?? Thanks ! _______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

_______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Reply

Raffael Marty

9:06 a.m.

Splunk. Definitely Splunk ;) If you have any questions or you want to talk more about your use- cases, I am happy to have a chat with you. On a serious note, I think you should try it. And it is free up to 500MB/day! That's quite a bit. After that it's fairly reasonably priced! One other thing that you might want to take into consideration is that other log management solutions don't cope with configuration files or multi-line information very well, if at all. I could list you a few very interesting use-cases around that: configuration management comes to mind. Also have a look at my blog where I talk a bit about the difference between IT Search (splunk) and the log management tools: blogs.splunk.com/raffy. Seisch, wenn'd irgendwelchi Frogae hesch! Raffy -- Raffael Marty Chief Security Strategist @ Splunk> Security Visualization: http://secviz.org raffy.ch/blog On Jan 20, 2008, at 11:52 PM, Olivier Beytrison wrote:

Hello, Maybe have a look at splunk. It's not free, but it seems to do what you're looking for. I'd like to ask at the same time if anyone here is using it. Because I thinking about installing it on our network. So some feedbacks would be great. www.splunk.com Regards, Olivier B. Marcel Prisi a écrit :

Hi all, I am looking for a good log centralisation / alerting / mining solution. I know about syslog-ng / rsyslog+phpLogCon, I'd like something more complete ... Something with a bit of realtime analysis (regexp ?) and correlation ... and a nice interface where you could get some useful details fast ... What solution do swinoggers use ?? Thanks ! _______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

_______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Reply

Michele Capobianco

9:28 a.m.

New subject: AW: Log centralisation / mining

Too bad that Splunk does not run on Windows :( We are a Windows Company and if i tell them that we want to run a Linux Server, our Management would kill me ;) Is there anything out in the Net for Log management witch is Windows Based? Regards Capo -----Ursprüngliche Nachricht----- Von: swinog-bounces(a)lists.swinog.ch [mailto:swinog-bounces@lists.swinog.ch] Im Auftrag von Raffael Marty Gesendet: Montag, 21. Januar 2008 09:07 An: swinog(a)swinog.ch Betreff: Re: [swinog] Log centralisation / mining Splunk. Definitely Splunk ;) If you have any questions or you want to talk more about your use- cases, I am happy to have a chat with you. On a serious note, I think you should try it. And it is free up to 500MB/day! That's quite a bit. After that it's fairly reasonably priced! One other thing that you might want to take into consideration is that other log management solutions don't cope with configuration files or multi-line information very well, if at all. I could list you a few very interesting use-cases around that: configuration management comes to mind. Also have a look at my blog where I talk a bit about the difference between IT Search (splunk) and the log management tools: blogs.splunk.com/raffy. Seisch, wenn'd irgendwelchi Frogae hesch! Raffy -- Raffael Marty Chief Security Strategist @ Splunk> Security Visualization: http://secviz.org raffy.ch/blog On Jan 20, 2008, at 11:52 PM, Olivier Beytrison wrote:

Hello, Maybe have a look at splunk. It's not free, but it seems to do what you're looking for. I'd like to ask at the same time if anyone here is using it. Because I thinking about installing it on our network. So some feedbacks would be great. www.splunk.com Regards, Olivier B. Marcel Prisi a écrit :

Hi all, I am looking for a good log centralisation / alerting / mining solution. I know about syslog-ng / rsyslog+phpLogCon, I'd like something more complete ... Something with a bit of realtime analysis (regexp ?) and correlation ... and a nice interface where you could get some useful details fast ... What solution do swinoggers use ?? Thanks ! _______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

_______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

_______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Reply

Raffael Marty

9:38 a.m.

New subject: AW: Log centralisation / mining

Too bad that Splunk does not run on Windows :(

Not yet! There is a preview version out that runs on Windows, but it's still a bit unstable. By the end of the month, we should have something that is releasable! Hang tight or try the preview! Cheers -raffy

We are a Windows Company and if i tell them that we want to run a Linux Server, our Management would kill me ;) Is there anything out in the Net for Log management witch is Windows Based? Regards Capo -----Ursprüngliche Nachricht----- Von: swinog-bounces(a)lists.swinog.ch [mailto:swinog-bounces@lists.swinog.ch ] Im Auftrag von Raffael Marty Gesendet: Montag, 21. Januar 2008 09:07 An: swinog(a)swinog.ch Betreff: Re: [swinog] Log centralisation / mining Splunk. Definitely Splunk ;) If you have any questions or you want to talk more about your use- cases, I am happy to have a chat with you. On a serious note, I think you should try it. And it is free up to 500MB/day! That's quite a bit. After that it's fairly reasonably priced! One other thing that you might want to take into consideration is that other log management solutions don't cope with configuration files or multi-line information very well, if at all. I could list you a few very interesting use-cases around that: configuration management comes to mind. Also have a look at my blog where I talk a bit about the difference between IT Search (splunk) and the log management tools: blogs.splunk.com/raffy. Seisch, wenn'd irgendwelchi Frogae hesch! Raffy -- Raffael Marty Chief Security Strategist @ Splunk> Security Visualization: http://secviz.org raffy.ch/blog On Jan 20, 2008, at 11:52 PM, Olivier Beytrison wrote:

Hello, Maybe have a look at splunk. It's not free, but it seems to do what you're looking for. I'd like to ask at the same time if anyone here is using it. Because I thinking about installing it on our network. So some feedbacks would be great. www.splunk.com Regards, Olivier B. Marcel Prisi a écrit :

Hi all, I am looking for a good log centralisation / alerting / mining solution. I know about syslog-ng / rsyslog+phpLogCon, I'd like something more complete ... Something with a bit of realtime analysis (regexp ?) and correlation ... and a nice interface where you could get some useful details fast ... What solution do swinoggers use ?? Thanks ! _______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

_______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

_______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog _______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Reply

Rainer Duffner

10 a.m.

New subject: AW: Log centralisation / mining

Michele Capobianco schrieb:

Too bad that Splunk does not run on Windows :( We are a Windows Company and if i tell them that we want to run a Linux Server, our Management would kill me ;)

Then, don't expect a free (OSS) solution ;-) I'd look into some of the UTM (Unified Threat Management) or (specialized) IDS solutions. I haven't tried it, but if I'd have a budget, I'd take a look at Tenable's log-correlation products: http://www.tenablesecurity.com/ They actually don't run on Windows, either, but they can analyze Windows-logs. See these links: http://www.networkintrusion.co.uk/consoles.htm BTW: I'd be interested to hear from people running one of those.

Is there anything out in the Net for Log management witch is Windows Based?

I guess there is a system-management solution from MSFT, too. Call your MSFT-sales rep ;-) cheers, Rainer

Reply

Roman Hochuli

12:45 p.m.

Hello Raffy

Splunk. Definitely Splunk ;) -- Raffael Marty Chief Security Strategist @ Splunk> Security Visualization: http://secviz.org raffy.ch/blog

I see. A totally unbiased position. ;) -- Best regards, Roman Hochuli Operations Manager nexellent ag Saegereistrasse 29 CH-8152 Glattbrugg Phone: +41 44 562 30 40 Fax: +41 44 562 30 41 URL: www.nexellent.ch X-NCC-RegID: ch.nexellent Imagination is the one weapon in the war against reality. -- Jules de Gaultier

Reply

Reza Kordi

8:56 a.m.

Hi! Give us more details... What is your log volume? How many systems? Are you looking for a opensource solution or a commercial one? -----Original Message----- From: swinog-bounces(a)lists.swinog.ch [mailto:swinog-bounces@lists.swinog.ch] On Behalf Of Marcel Prisi Sent: Montag, 21. Januar 2008 08:48 To: swinog(a)swinog.ch Subject: [swinog] Log centralisation / mining Hi all, I am looking for a good log centralisation / alerting / mining solution. I know about syslog-ng / rsyslog+phpLogCon, I'd like something more complete ... Something with a bit of realtime analysis (regexp ?) and correlation ... and a nice interface where you could get some useful details fast ... What solution do swinoggers use ?? Thanks ! _______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Reply

Marcel Prisi

9:05 a.m.

Le lundi 21 janvier 2008 à 08:56 +0100, Reza Kordi a écrit :

Hi! Give us more details... What is your log volume? How many systems?

For now, 20-30 systems (growing), and we also use syslog from some of our applications.

Are you looking for a opensource solution or a commercial one?

I would of course better like an opensource one, but I will evaluate every interesting solution. Thanks.

Reply

Peter Baumann

9:55 a.m.

On Mon, 2008-01-21 at 08:47 +0100, Marcel Prisi wrote:

Hi all, I am looking for a good log centralisation / alerting / mining solution. I know about syslog-ng / rsyslog+phpLogCon, I'd like something more complete ...

Hi, If you need a commercial solution and need to be compliant (SOX, Basel II etc.) I was working the last week with RSA enVision. It supports all kind of log-interfaces and got very good reporting/alarming/reporting functions. I also like the opensource solutions but I couldn't find any solution yet for a good reporting and alarming. If you need to know more about enVision you can contact me directly. Peter

Reply

Reza Kordi

5:26 p.m.

The most professional solution on market is surely EMC/RSA envision, if you see it you won't want to bother with anything else. If you wanna a demo let me know of list. Best Regards Mit freundlichen Grüssen Reza Kordi Managing Director Clue AG Blegistrasse 9 CH - 6340 Baar/Zug tel. +41 41 240'49'49 fax. +41 41 240'49'59 mob. +41 78 870'02'30 www.clue.ch - On with Virtualization -----Original Message----- From: swinog-bounces(a)lists.swinog.ch [mailto:swinog-bounces@lists.swinog.ch] On Behalf Of Marcel Prisi Sent: Montag, 21. Januar 2008 08:48 To: swinog(a)swinog.ch Subject: [swinog] Log centralisation / mining Hi all, I am looking for a good log centralisation / alerting / mining solution. I know about syslog-ng / rsyslog+phpLogCon, I'd like something more complete ... Something with a bit of realtime analysis (regexp ?) and correlation ... and a nice interface where you could get some useful details fast ... What solution do swinoggers use ?? Thanks ! _______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Reply

Raffael Marty

6:16 p.m.

On Jan 21, 2008, at 3:45 AM, Roman Hochuli wrote:

Hello Raffy

Splunk. Definitely Splunk ;) -- Raffael Marty Chief Security Strategist @ Splunk> Security Visualization: http://secviz.org raffy.ch/blog

I see. A totally unbiased position. ;)

I assumed that was obvious... That's why I also said:

On a serious note, I ...

Reza wrote:

The most professional solution on market is surely EMC/RSA envision, if you see it you won't want to bother with anything else.

I would totally disagree. If you really want to go down that route, ArcSight is the one you want to go for. But again, be clear on what you are trying to do. All of these solutions are slightly different and should match your use. -raffy

Reply

Reza Kordi

28 Jan 28 Jan

9:19 a.m.

Raffy, What do you like about ArcSight? The policy engine? Compliance? Which version of Arcsight did you look at? Cheers, Reza -----Original Message----- From: swinog-bounces(a)lists.swinog.ch [mailto:swinog-bounces@lists.swinog.ch] On Behalf Of Raffael Marty Sent: Montag, 21. Januar 2008 18:17 To: swinog(a)swinog.ch Subject: Re: [swinog] Log centralisation / mining On Jan 21, 2008, at 3:45 AM, Roman Hochuli wrote:

Hello Raffy

Splunk. Definitely Splunk ;) -- Raffael Marty Chief Security Strategist @ Splunk> Security Visualization: http://secviz.org raffy.ch/blog

I see. A totally unbiased position. ;)

I assumed that was obvious... That's why I also said:

On a serious note, I ...

Reza wrote:

The most professional solution on market is surely EMC/RSA envision, if you see it you won't want to bother with anything else.

I would totally disagree. If you really want to go down that route, ArcSight is the one you want to go for. But again, be clear on what you are trying to do. All of these solutions are slightly different and should match your use. -raffy _______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Reply

Raffael Marty

6:45 p.m.

Well... I used to lead the solutions team at ArcSight. I was responsible for building all the compliance and "other" solutions. I have worked with up to ESM version 4.0. There is no policy engine. It's a rules-engine. And yes, it's one of the best out there, if you want to do real time correlation. In terms of compliance, it's what they sell as solutions. I am not going to comment on them here. It's what I built in my prior life. They are obviously stellar *grins*. Sorry for being sparse on details, but I think you understand based on my bias. -raffy -- Raffael Marty Chief Security Strategist @ Splunk> Security Visualization: http://secviz.org raffy.ch/blog On Jan 28, 2008, at 12:19 AM, Reza Kordi wrote:

Raffy, What do you like about ArcSight? The policy engine? Compliance? Which version of Arcsight did you look at? Cheers, Reza -----Original Message----- From: swinog-bounces(a)lists.swinog.ch [mailto:swinog-bounces@lists.swinog.ch ] On Behalf Of Raffael Marty Sent: Montag, 21. Januar 2008 18:17 To: swinog(a)swinog.ch Subject: Re: [swinog] Log centralisation / mining On Jan 21, 2008, at 3:45 AM, Roman Hochuli wrote:

Hello Raffy

Splunk. Definitely Splunk ;) -- Raffael Marty Chief Security Strategist @ Splunk> Security Visualization: http://secviz.org raffy.ch/blog

I see. A totally unbiased position. ;)

I assumed that was obvious... That's why I also said:

On a serious note, I ...

Reza wrote:

The most professional solution on market is surely EMC/RSA envision, if you see it you won't want to bother with anything else.

I would totally disagree. If you really want to go down that route, ArcSight is the one you want to go for. But again, be clear on what you are trying to do. All of these solutions are slightly different and should match your use. -raffy _______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog _______________________________________________ swinog mailing list swinog(a)lists.swinog.ch http://lists.swinog.ch/cgi-bin/mailman/listinfo/swinog

Reply

5608

days inactive

5615

days old

swinog@lists.swinog.ch

Manage subscription

14 comments

9 participants

tags (0)

participants (9)

Marcel Prisi
Michele Capobianco
Olivier Beytrison
Peter Baumann
Raffael Marty
Rainer Duffner
Reza Kordi
Roman Hochuli
Tobias Koenig