Download Queue (DQ) v0.2

September 12, 2007

A fairly big update to the script, I've added several new features:

• Notifications via libnotify


• Download History


• Reprioritization via move, up, and down commands


• Added the ability to add multiple urls at once


• Added the ability to skip a download via CTRL+C (moves it to end of queue)


• -f option for custom queue.txt at command line


• -c option for custom .dqcfg at command line

Get the updated script here: Download Queue v0.2

If some of the new code seems a little hack-ish just remember, I'm still learning.

Next on the agenda:

• Figure out how to account for wget getting 404, 503, etc, while still providing real-time output to the command line.


• Download grouping and/or contexts (i.e. if a download falls within a certain context, check against a certain set of filters)


• Modified "start" command (download a certain number of urls, a certain range, or those matching a certain regex)

Download Queue (DQ) v0.1b

September 2, 2007

[update]

Download Queue v0.2 Available!

See this blog post for details: Download Queue (DQ) v0.2
[/update]

I've been spending more and more time at the Linux command line lately and have been looking at ways of improving the experience, customizing and personalizing it. At the same time, I've been using it as an opportunity to learn new languages, one of those being bash.

I've used bash in a very simplistic way ever since I first got Linux, simple one-off commands here and there, aliases, .bashrc, etc., but I only recently decided to take a serious, in-depth look at it. I've been reading tutorials here and there, studied up a bit on common useful commands, but in order to really get to know a language you have to use it, so that's what I'm doing.

Download Queue (DQ) v0.1b

As is often the case, the idea for this program came from a need. I needed a command line download manager. I wanted it to be lightweight, I wanted it to be quick and easy to use, and above all I wanted it to work with plain text (a notion inspired by the wonderfully useful todo.txt, by Gina Trapani and contributors).

todo.txt's principles have influenced this project significantly, and as DQ grows I hope it becomes as useful to me as todo.txt has.

As with todo.txt, this script will focus primarily on manipulating and reading a single, human readable text file named "queue.txt". Each line in queue.txt is a single url to be downloaded. The queue is prioritized from top to bottom, meaning urls that appear first in the list are the first to be downloaded.

The script uses wget to download each url, and has the -c flag set by default. -c is short for continue, meaning even if you interrupt the download it will pick up where it left off, so you won't have to start over. Those familiar with wget might think that the script simply passes the queue.txt file on to wget, and lets it handle the rest, but that would make it more of a download enabler than a download manager. Instead, it grabs the first line of the file and passes that to wget, removing that line once the download completes.

The advantage to this is that you're still able to manipulate the file while the downloads are going. It also means you're able to analyze each url and make decisions accordingly, this means you're able to create filters.

The filters are implemented simply. You provide a regex, a directory, and flags that will be passed to wget. Both the directory and the flags are optional (though leaving both out would be a bit pointless). If you leave the directory out then the default directory, which is set in the included config file, will be used.

An example:
addfilter ".*\.mp3" "$HOME/Downloads/music/" ""
addfilter ".*\.(jpe?g|gif|png|bmp)" "$HOME/Downloads/images/" ""
addfilter ".*\.(com|org|net|html?|asp|php)/?" "$HOME/Mirrors/" "-E -H -k -K -p"

As you can see, thanks to the first filter all files ending in .mp3 will be saved to the $HOME/Downloads/music/ directory. At the same time, all files that end in .jpeg, .jpg, .gif, .png, or .bmp will be sent to an images folder in my Downloads directory. The trailing "" on the first two lines is where flags would go. It is not necessary to include an empty "", I do it here for demonstration purposes.

The third line implements a (very crude and untested) regex for detecting the download of a web page. If it matches it passes flags to wget that localizes appropriate links and grabs page requisites as well (css files, images, etc).

Once configured, the script can be used fairly simply. To add a url simply use
dq.sh add

To remove a url, use
dq.sh rm

And to begin the download process, use
dq.sh start

This is only the beginning of what DQ is and will be capable of. I'm working on a list of features (included in the downloaded .rar file as TODO) that includes download skipping, reprioritization, and the ability to specify custom queue.txt and .dqcfg files when calling the script. Any suggestions for features are welcome, and may be left as a comment here or sent to me via email at thezikes@gmail.com.

Tags, Search, and Archives

April 10, 2007

Lots of new features added today:

Search

I have a basic search system set up, which recognizes phrases in quotes and space separated key words.

Tags

Each blog can have tags associated with it. Users can click on a tag to see all blogs that use that tag.

Archives

A basic archive setup that lists blogs by month and year.

Credit goes to poeticcode for the formula I used for my tag cloud sizing.