[PLUG] Using wget/curl to download all files in directory

Rich Shepard rshepard at appl-ecosys.com
Sat Mar 8 16:50:14 UTC 2014


On Sun, 5 Feb 2012, Paul Mullen wrote:

> wget can do that for you.  My preferred wget command line:
>  wget -nc -k -r -np YOUR_URL_HERE
> With that command line, wget will download the document pointed to in
> the specified URL, plus (potentially) any documents linked from it.

Paul et al.:

   That's close to what I need, and today I developed the needed changes
(starting from scratch, candidly). For others who might need this
capability, and just in case I lose the text file with the command, I'll
record it for posterity here.

   The context: when upgrading Slackware64 distributions the multilib
libraries that permit running (and building) 32-bit applications on a 64-bit
system also need to be upgrade. AlienBOB makes these available on a couple
of Web sites; I use http://alien.slackbook.org/slackware/multilib/14.1/ (and
will change the version number as appropriate.) That directory contains many
system files and a subdirectory, slackware64_compat32 with others. I want
them stored in ~/slackware-multilib/.

   To get the latest builds, I cd to ~/slackware-multilib/ and run this
command:

wget -r -c -nH --cut-dirs=3 http://alien.slackbook.org/slackware/multilib/14.1/

   As above, -r to recurse directories, -c to continue from where the process
stopped (as it does all too frequently here), -nH for 'no host directories'
since I do not want to download the entire site, only the terminal directory
in the URL and any subdirectories it has, --cut-dirs=3 to store the desired
directories and their contents in the pwd where the command was invoked,
rather than re-creating the source Web site's directory structure locally.

   Curiously, file download speeds are 50-70% slower than from within firefox
or opera. But, eventually everything gets here.

Rich




More information about the PLUG mailing list