[PLUG] Capture of CSV data

Ronald Bynoe ronald at bynoe.us
Mon Jun 23 21:51:04 UTC 2014


Netcat (nc) should be able to store the stream of data, we use it for snmp
and netconsole streams. Otherwise curl is a good basic solution as well.

Pleasantly,
Ronald Bynoe
On Jun 23, 2014 2:49 PM, "Chuck Hast" <wchast at gmail.com> wrote:

> To be a bit more clear, I am not sure how to get the data off of the
> communications medium and into a db, that is where I am at loss.
> I assume that I have to create a table in the db that is laid out like
> each row, and then have a tool that can pipe the data from the
> source into the db.
>
> The format of the connection is
> http://URL/ipaddy:PortNum
> PortNum may be 9010, 9030 or 9050, the last one spits out the data
> in binary format, I will deal with that later, right now I just want to get
> my data in the CSV format stuffed into a db.
>
>
> On Mon, Jun 23, 2014 at 2:44 PM, Chuck Hast <wchast at gmail.com> wrote:
>
> > Rich,
> > Yes, I want to store each row in a db or a flat file, but I fear the
> > FF may get messy, then once I have it somewhere for posterity,
> > I will start to process it. Generally we look at it in the following
> > form,
> > Rejects by mould number
> > Reject type by mould number
> >
> > Rejects last hour
> > Rejects last X hours
> > Rejects last shift
> > Rejects last 24 hours
> > Other time periods
> > Also we look at a breakdown of reject types on those time
> > periods.
> >
> > At this point I am just trying to get the data stream into a db or a
> > file, once there I can start playing with how to graph it, I used to
> > do RF coverage studies and we graphed all the different parameters
> > on a RF system, signal level, BER, BERT, S/N, etc both historically
> > and real time. It has been a while, but I have a lot of machines here
> > that are trying to supply all manner of status info, and that format is
> > more or less standardized among them all.
> >
> > At this point I am not worried about the graphing, I am trying to figure
> > how to get live data into a file/db so I can graph it from there.
> >
> >
> >
> > On Mon, Jun 23, 2014 at 2:30 PM, Rich Shepard <rshepard at appl-ecosys.com>
> > wrote:
> >
> >> On Mon, 23 Jun 2014, Chuck Hast wrote:
> >>
> >> > Hmmm I guess I did not get the data sample in it.
> >>
> >> Chuck,
> >>
> >>    You did include the data sample, but not what questions you had about
> >> it.
> >>
> >> > I know how to get csv files into spead sheets and i have sort of a
> idea
> >> of
> >> > getting them into a db, but how do I handle a stream of data like that
> >> > below. All of them are CSV formated, so once I can get one into
> >> something
> >> > I can graph and twiddle, the rest should be easy.
> >>
> >>    So, you want to store each row in a database table. Then you want to
> >> produce some sort of plot (histogram? scatter plot? box-and-whisker
> plot?)
> >> from selected rows or continuously? What sort of twiddling do you
> >> envision?
> >>
> >>    I would approach a solution by using a Python script (the psycopg2
> >> module
> >> is appropriate for the interface between python and postgres, and
> wxPython
> >> is the GUI I use) to store the rows in a postgres database table as each
> >> row
> >> was sent from a machine, then use pandas to analyze those data and
> produce
> >> plots using matplotlib. There are probably a gazillion alternatives,
> >> including using R for the analyses, but I suspect that would be overkill
> >> for
> >> your needs.
> >>
> >> Rich
> >>
> >> _______________________________________________
> >> PLUG mailing list
> >> PLUG at lists.pdxlinux.org
> >> http://lists.pdxlinux.org/mailman/listinfo/plug
> >>
> >
> >
> >
> > --
> >
> > Chuck Hast  -- KP4DJT --
> > Glass, five thousand years of history and getting better.
> > The only container material that the USDA gives blanket approval on.
> >
> >
> >
>
>
> --
>
> Chuck Hast  -- KP4DJT --
> Glass, five thousand years of history and getting better.
> The only container material that the USDA gives blanket approval on.
> _______________________________________________
> PLUG mailing list
> PLUG at lists.pdxlinux.org
> http://lists.pdxlinux.org/mailman/listinfo/plug
>



More information about the PLUG mailing list