[PLUG] Capture of CSV data

Benjamin Foote plug at bnf.net
Tue Jun 24 02:34:28 UTC 2014


Chuck, All,

As you approach analysis of that data I highly suggesting sending it to an
ELK stack (ElasticSearch, Logstash, Kibana).

http://logstash.net
http://logstash.net/docs/1.4.1/filters/csv
http://www.elasticsearch.org/overview/kibana/

More than just excellent graphing and log analysis, the interface is simple
yet powerful.  The ability to make custom dashboards that I can hand to a
developer and offer them "insight" into what's going on has been
instrumental in "effecting change" :)

ben


Benjamin Foote
Linux System Administration and Development
503-313-5379
ben at bnf.net
http://bnf.net
@bnf


On Mon, Jun 23, 2014 at 3:34 PM, Rich Shepard <rshepard at appl-ecosys.com>
wrote:

> On Mon, 23 Jun 2014, Chuck Hast wrote:
>
> > To be a bit more clear, I am not sure how to get the data off of the
> > communications medium and into a db, that is where I am at loss. I assume
> > that I have to create a table in the db that is laid out like each row,
> > and then have a tool that can pipe the data from the source into the db.
>
> Chuck
>
>    Ah! You're not familiar with databases. Each table has a series of
> columns
> that store the attributes; e.g., mould number, through-put rate, number of
> rejects, etc. Each row in the table is a unique set of those attributes
> identified by (perhaps) mould number and time.
>
> > The format of the connection is
> > http://URL/ipaddy:PortNum
> > PortNum may be 9010, 9030 or 9050, the last one spits out the data
> > in binary format, I will deal with that later, right now I just want to
> get
> > my data in the CSV format stuffed into a db.
>
>    Off-hand I cannot give you an answer because I've not done this before.
> However, I know it can be done quickly and easily with python and psycopg2.
>
>    You might consider contracting with a python/postgres coder to quickly
> write the application for you. There are a bunch available locally.
>
> Rich
> _______________________________________________
> PLUG mailing list
> PLUG at lists.pdxlinux.org
> http://lists.pdxlinux.org/mailman/listinfo/plug
>



More information about the PLUG mailing list