[PLUG] Solaris syslog behavior when overloaded
Aaron Burt
aaron at bavariati.org
Tue Jul 13 18:48:56 UTC 2010
On Tue, Jul 13, 2010 at 08:13:19AM -0700, Michael wrote:
>
> We're looking into a problem with VOIP phone freezing and have several
> hundred phones doing debug level logging to a Solaris box. Analysis
> shows between 1000 and 1500 messages a second being logged.
Cool problem! I used to have fun working on VoIP equipment.
> However after a few hours we see a backwards drift in timestamps - as if
> the syslog server is falling behind. I also see that syslog is
> resolving IP addresses and writing FQDN to the log.
So timestamps in logged messages are falling behind current-time?
Do you suppose the UDP receive queue is filling up with messages before
they hit syslogd, or syslogd is backing up behind the disk I/O?
At your message rate, you're receiving a lot of bits.
> The man pages only reference to buffering is that if syslog is sent a HUP
> it will attempt to flush all pending messages. There is no reference
> about controlling the DNS lookup.
Ah, the joy of Solaris manpages. You might find other settings in
/etc/default/syslogd or /etc/netconfig or /etc/net/transport/ but I think
Solaris syslogd behaviour is pretty much set in stone.
Have you considered using syslog-ng and/or a more powerful (Linux- or
OpenSolaris-based) log-server?
I <3 CSW,
Aaron
More information about the PLUG
mailing list