[PLUG] Capture of CSV data

Chuck Hast wchast at gmail.com
Tue Jun 24 03:49:38 UTC 2014


Well folks, this is what is neat about Linux Open Source and all of the
people
associated with it. I have a bundle of info here and plenty of entry points
at
all levels to start poking at it.

I figured that a flat file was a no-no, but wanted to make sure. The row
rate is based on the number of bottle inspections the machine is doing,
today we are running heavy bottles so the machines are only doing about
180 bpm (bottles per minute) this is more or less a weaponized wine
bottle, if you are in a bar fight you WANT one of these in your hand.

When we run lighter ware we may run as high as 280bpm, this is just
one of three outputs, the one that is available on port 9010, is almost 3
times as long and has more info in it, and there is a binary output on
port 9050, but for right now the data on port 9030 is enough to get started.

Tomorrow at work I will start poking at the data capture end and see what
I can do there, then I will start looking at setting up a db (I have muddled
through some work on them, I run ZoneMinder and every once in a wile
i have to go in and clean the MySQL db on some of the ZM machines
though of late I have not had to do it, so they must be doing some work
there.

I figure that once I can get those rows of data into a place where they can
be used, then I can move on to other machines, the format is more or less
the same for all of them, so once I can work with that one I can move to
the hot end (the moulding part of the plant) and capture that data for
processing, capture the data off of the cold end, and then they can get
even more precise data on which moulds are causing issues etc.

Again, I am all ears and will give it all  a try.



On Mon, Jun 23, 2014 at 7:34 PM, Benjamin Foote <plug at bnf.net> wrote:

> Chuck, All,
>
> As you approach analysis of that data I highly suggesting sending it to an
> ELK stack (ElasticSearch, Logstash, Kibana).
>
> http://logstash.net
> http://logstash.net/docs/1.4.1/filters/csv
> http://www.elasticsearch.org/overview/kibana/
>
> More than just excellent graphing and log analysis, the interface is simple
> yet powerful.  The ability to make custom dashboards that I can hand to a
> developer and offer them "insight" into what's going on has been
> instrumental in "effecting change" :)
>
> ben
>
>
> Benjamin Foote
> Linux System Administration and Development
> 503-313-5379
> ben at bnf.net
> http://bnf.net
> @bnf
>
>
> On Mon, Jun 23, 2014 at 3:34 PM, Rich Shepard <rshepard at appl-ecosys.com>
> wrote:
>
> > On Mon, 23 Jun 2014, Chuck Hast wrote:
> >
> > > To be a bit more clear, I am not sure how to get the data off of the
> > > communications medium and into a db, that is where I am at loss. I
> assume
> > > that I have to create a table in the db that is laid out like each row,
> > > and then have a tool that can pipe the data from the source into the
> db.
> >
> > Chuck
> >
> >    Ah! You're not familiar with databases. Each table has a series of
> > columns
> > that store the attributes; e.g., mould number, through-put rate, number
> of
> > rejects, etc. Each row in the table is a unique set of those attributes
> > identified by (perhaps) mould number and time.
> >
> > > The format of the connection is
> > > http://URL/ipaddy:PortNum
> > > PortNum may be 9010, 9030 or 9050, the last one spits out the data
> > > in binary format, I will deal with that later, right now I just want to
> > get
> > > my data in the CSV format stuffed into a db.
> >
> >    Off-hand I cannot give you an answer because I've not done this
> before.
> > However, I know it can be done quickly and easily with python and
> psycopg2.
> >
> >    You might consider contracting with a python/postgres coder to quickly
> > write the application for you. There are a bunch available locally.
> >
> > Rich
> > _______________________________________________
> > PLUG mailing list
> > PLUG at lists.pdxlinux.org
> > http://lists.pdxlinux.org/mailman/listinfo/plug
> >
> _______________________________________________
> PLUG mailing list
> PLUG at lists.pdxlinux.org
> http://lists.pdxlinux.org/mailman/listinfo/plug
>



-- 

Chuck Hast  -- KP4DJT --
Glass, five thousand years of history and getting better.
The only container material that the USDA gives blanket approval on.



More information about the PLUG mailing list