[PLUG] Capture of CSV data

Chuck Hast wchast at gmail.com
Mon Jun 23 21:49:35 UTC 2014


To be a bit more clear, I am not sure how to get the data off of the
communications medium and into a db, that is where I am at loss.
I assume that I have to create a table in the db that is laid out like
each row, and then have a tool that can pipe the data from the
source into the db.

The format of the connection is
http://URL/ipaddy:PortNum
PortNum may be 9010, 9030 or 9050, the last one spits out the data
in binary format, I will deal with that later, right now I just want to get
my data in the CSV format stuffed into a db.


On Mon, Jun 23, 2014 at 2:44 PM, Chuck Hast <wchast at gmail.com> wrote:

> Rich,
> Yes, I want to store each row in a db or a flat file, but I fear the
> FF may get messy, then once I have it somewhere for posterity,
> I will start to process it. Generally we look at it in the following
> form,
> Rejects by mould number
> Reject type by mould number
>
> Rejects last hour
> Rejects last X hours
> Rejects last shift
> Rejects last 24 hours
> Other time periods
> Also we look at a breakdown of reject types on those time
> periods.
>
> At this point I am just trying to get the data stream into a db or a
> file, once there I can start playing with how to graph it, I used to
> do RF coverage studies and we graphed all the different parameters
> on a RF system, signal level, BER, BERT, S/N, etc both historically
> and real time. It has been a while, but I have a lot of machines here
> that are trying to supply all manner of status info, and that format is
> more or less standardized among them all.
>
> At this point I am not worried about the graphing, I am trying to figure
> how to get live data into a file/db so I can graph it from there.
>
>
>
> On Mon, Jun 23, 2014 at 2:30 PM, Rich Shepard <rshepard at appl-ecosys.com>
> wrote:
>
>> On Mon, 23 Jun 2014, Chuck Hast wrote:
>>
>> > Hmmm I guess I did not get the data sample in it.
>>
>> Chuck,
>>
>>    You did include the data sample, but not what questions you had about
>> it.
>>
>> > I know how to get csv files into spead sheets and i have sort of a idea
>> of
>> > getting them into a db, but how do I handle a stream of data like that
>> > below. All of them are CSV formated, so once I can get one into
>> something
>> > I can graph and twiddle, the rest should be easy.
>>
>>    So, you want to store each row in a database table. Then you want to
>> produce some sort of plot (histogram? scatter plot? box-and-whisker plot?)
>> from selected rows or continuously? What sort of twiddling do you
>> envision?
>>
>>    I would approach a solution by using a Python script (the psycopg2
>> module
>> is appropriate for the interface between python and postgres, and wxPython
>> is the GUI I use) to store the rows in a postgres database table as each
>> row
>> was sent from a machine, then use pandas to analyze those data and produce
>> plots using matplotlib. There are probably a gazillion alternatives,
>> including using R for the analyses, but I suspect that would be overkill
>> for
>> your needs.
>>
>> Rich
>>
>> _______________________________________________
>> PLUG mailing list
>> PLUG at lists.pdxlinux.org
>> http://lists.pdxlinux.org/mailman/listinfo/plug
>>
>
>
>
> --
>
> Chuck Hast  -- KP4DJT --
> Glass, five thousand years of history and getting better.
> The only container material that the USDA gives blanket approval on.
>
>
>


-- 

Chuck Hast  -- KP4DJT --
Glass, five thousand years of history and getting better.
The only container material that the USDA gives blanket approval on.



More information about the PLUG mailing list