[PLUG] Ports and Processes

[Shannon C. Dealy]

On Fri, 26 Jul 2002, Alot of different people wrote :-)

With heavy snippage & rearrangement

Since there was a great deal of overlap, I decided to respond to all at
once.

> If you're quick, you might be able to get an lsof listing to show you the
> offending program.  Since your program completes so quickly, this will
> ====
>  Why not run "netstat --tcp --udp -lp >> logfile" in a loop?
> ====
> watch -n 0 netstat --tcp --udp -lp >> logfile
> watch -n 0 netstat --tcp --udp -lp | tee logfile.
> ====
> Or for even less overhead and disk usage, create a FIFO, and read from  that
> continuously.  You could then include/exclude  whatever you want, grep  for
> the PID of the program, look up the owner of the process, have it return that
> and the terminal they logged in on.

This approach would have worked for the latest irritation that I was
trying to track down, because the problem was occuring multiple times per
minute, and while it had occurred to me to try this, it most likely will
not work for another problem I have been trying to track down because it
only once every few hours for a day or so, then goes away (sometimes for
weeks), then shows up again.  Basically this loop approach is a
statistical method, eventually things will time out so that you capture
one, but, I'm trying to find a general solution that will work for the
rare packets as well.  I say this would have worked, because after
several days of irritation, the problem went away (more on that below)
before I could capture it.

> If you can afford to disrupt the traffic, an invalid static route for the
> destination of those mystery packets sometimes helps hold the connection
> open long enough for lsof to collect the information.  (I.e. the packet
> gets dropped and the TCP connection needs to time out, keeping it open
> longer than if it completed successfully.)

This is one that hadn't occurred to me, I'll have to give it a try.

> The other option (which is *way* more hackerly) would be to modify the
> tcp_output.c portion of the kernel to log all connect() calls and build a
> new tcp-logging kernel.  This is probably best done from the console.
> ;-)  The hard part would be figuring out how to get access to the calling
> program's process table structures from within the TCP kernel code.

Given what I do for a living, I'm more than up to the "way more hackerly"
aspect, but I hate reinventing the wheel (aside from being both busy and
lazy :-) so I was hoping someone else had already done something along
these lines, or some similar alternative approch.

> ===
> Could you describe the process of creating and using the FIFO in more
> detail.  I would like to know how to do that, but have never seen it
> before.

To create a FIFO (also called a named pipe), use the mkfifo command,
something like:

   mkfifo <some_arbitrary_file_name>

This creates on your file system a pipe, similar to what you get when you
pipe the output of one command into another such as "ls -l | less"
the difference is that the pipe has a name and is attached to your file
system.  To make use of the pipe, you can simply redirect the ouput of a
command to it, then have another command read from it like this:

  mkfifo /var/tmp/shannons_pipe
  cat /etc/passwd >/var/tmp/shannons_pipe &
  cat </var/tmp/shannons_pipe

this is obviously a stupid and trivial example, but fifos are often used
for things like a simple way for a daemon to provide status information,
or to pass data between independent processes (that may be running as
different users).  One thing to be aware of, once the pipe is full the
process which is filling it may block (if that is how it's I/O handling is
written), until some process is attached to the other end and starts
reading data from the pipe.

> =====
> I've seen all the other replies so I'll suggest ipchains/iptables.  I
> can't offer an example but I swear there is an option to log activity.
> In fact I just checked and see a --log option.  I've never tried it but it
> has been offered as a solution to me.  (Turned out my problem was PEBKAC
> so I never had to use it. :-)

While this will give me access to the packets, so far as I'm aware, it
will not give me information on the process which generated the packets,
and I don't think it will give me anything the packet sniffers

> =====
> If you can define the traffic enough to identify it, I assume you can
> filter for it.  It is simple enough to use tcpdump writing raw traffic
> to a file (just make sure to increase snaplen enough to get the
> *whole* packet w/ payload, not just the header).
> If nothing else you can do a raw of all traffic on that interface.
>
> Even assuming it is a raw TCP session examining the payload should
> give you a good idea of the application.  Reconstructing the full
> bi-dir stream even more.

Actually, this is what I have been doing, the problem is that what I am
trying to track down is sending very generic packets and doesn't give any
real clues.  The packets I was dealing with today were DNS lookups and
ICMP errors (relating to the DNS lookups), but they weren't coming from
the DNS server (I shut it down just to be sure), and I can't figure out
who was making the request, email was my first guess, but I scanned all
email messages and log files and found no references to the IP addresses
or domains that were involved in this.  Unfortunately (or fortunately
depending on your perspective), when I restarted BIND, the problem went
away.

The intermittent packets that I mentioned previously are unfortunately
even more generic, they are TCP echo requests (not ICMP)

> As mentioned in the thread, the BSD accounting option in the kernel
> would be useful.  grab the tools, and you get a nice timestamped log
> of every proc including name, uid, walltime, cpu timer, etc.  That
> compared with timestamped tcpdump logs...
---
> You might also see if you could make some combination of BSD
> process accounting and ip accounting work for your purposes.

With regard to the BSD accounting, I'm afraid I don't know much about it's
capabilities, so I'll have to look into this, but if it can just give a
simple time stamped record of processes and their ports, that might be
sufficient when coupled with a filtering packet sniffer.

Thanks to everyone for their suggestions, by using some of these, it looks
like I just might be able to remain lazy and avoid writing code! :-)

Shannon C. Dealy      |               DeaTech Research Inc.
dealy at deatech.com     |          - Custom Software Development -
                      |    Embedded Systems, Real-time, Device Drivers
Phone: (800) 467-5820 | Networking, Scientific & Engineering Applications
   or: (541) 451-5177 |                  www.deatech.com