[PLUG] Translating ^M to \n

[Rich Shepard]

On Mon, 12 Aug 2019, Rodney W. Grimes wrote:

> My guess.. this was a browser downloaded .csv file? So line endings have
> been mangled by the web.

Rodney,

Yes, these files were downloaded from the agency's web site using the
'Download' button.

Last week I downloaded the database as a single Access .mdb file, and the
metadata PDF which has the table names and type (in Access types, not
standard SQL), but no entries in the Description column. I converted 66
tables to postgres SQL using mdbtools; all tables starging with 'A' through
'R'. There are no tables starting with 'S' through 'W' in the file although
the names are in the metadata PDF. Access is a flat file database so there
are neither primary nor foreign keys. It would take a while to reverse
engineer it building an E-R diagram and making a functional relational
database; worth while if all tables are available. (It's a very poorly
designed database, too.)

I sent them an e-mail message asking about this and no one's respnded so I
tried downloadint the data from their web site.

> You might fix it by simply deleting the first byte... then doing the cr ->
> lf remap
>
> dd if=oldfile bs=1 iseek=1 | tr '\r' '\n' >newfile
>
> Note this is gona be really slow due to single byte I/O used by the DD
> command to take the first byte off.

Those files are on my replacement desktop server/workstation (which still
needs a few more things working before it's put into service). That host has
a Ryzen 7 2700 8-core CPU and 32G RAM so byte-by-byte crunching should go
comparatively quickly.

Will report results later this morning.

Thanks,

Rich