Newsbeuter - Now With Podcatching Powers

There are plenty of RSS aggregators available, as well as plenty of podcatchers available but like anything, often the simpler the better. With that in mind, I present you with a wonderful little CLI app called Newsbeuter. It is an RSS aggregator with podcatching abilities.

The thing that held me back from discovering the full potential of this little gem is that the podcatcher abilities are either not well explained, or my brain could not interpret what the instructions were telling me to do. Like any small project their IRC channel was very quiet and took a while to get a response. In fairness they do mention this on the site. I did however mange to figure it out, and have decided to write this blog and record an HPR episode to explain. You can download my HPR episode on Newsbeuter here.

It always amazes me how many people don't know what RSS feeds are, or how they can use them and benefit from them. This is not new PC users but experienced users who have been using their PC for years but just never had their PC lives intersect with the wonders of an RSS feed. So I'll do a primer to start with.

RSS Feeds

Many sites update their stories or posts irregularly. They are not TV schedules with a new post going live at exactly 8pm or Wednesday night. Without RSS feeds you'd have to bookmark the site in question and keep remembering to manually click on it in the hope that a new post had been made. As soon as a new blog post or news story appears on a site, the RSS feed will be updated accordingly. RSS feeds allow you to subscribe to that site, and when you refresh your RSS reader it will tell you when there's a new post, and allow you to read it.

Some sites give you a teaser content and want you to click through to the site to read the whole thing, while others put the full post in the feed. The content will be displayed very like an email client, with the title of the post, and the body text. You can look at the title or the teaser of the post and decide you're not interested in that post, so just mark it as "read".

The difference between RSS and an email news update is that with an RSS feed it's as simple as adding the feed URL to your reader, or deleting it when you get bored with the content the site puts out. The site only ever gets the stats on which reader you use as part of a larger count. With email lists you give the site your email address, which it may or may not respect.

RSS feeds are often shown in the right hand side of the address bar of the web browser though not always. They may show it in multiple places on the site for easy access. Most sites have some variation on the three orange quarter circle theme for their feed icon. A podcast RSS feed is just a normal RSS feed that's been formatted differently to carry a link to the audio or video file in a way that podcatcher applications can read and download.

Newsbeuter

The numbers of different ways RSS feed readers (also known as aggregaters) have been designed is great. It means there's one that works the way you like. You can have them as standalone applications, or built into other applications. In the case of Newsbeuter it's a CLI (Command Line Interface) application. It runs in the terminal. It has the keys listed along the bottom in case you forget, but in reality you only need to remember a few. These commands are all case sensitive so d is different from D.

You'll notice that my list of feeds are a mix of podcast and regular blog and news feeds. Newsbeuter can be told to run with a different profile so you can split those functions up, I've decided it's not worth doing that, so I just have my podcasts after the blogs on the URL list. Everything from the 5 Live Football onwards is a podcast.

Feeds that have new posts have the letter N next to them. The post count next to the feed shows how many new posts are on that feed. In the time I took this screenshot, Slashdot had 9 new posts. You'll notice at the top it states there are 6 unread, but only 5 feeds with new posts are showing. I have a total of 34 feeds, only the first 21 are showing. The page up and page down buttons let you move up and down the list in large jumps, or the up arrow and down arrow keys do the same for the highlighted blue line.

To refresh all your feeds it's R, or alternatively you can set newsbeuter to refresh all your feeds as soon as it starts with newsbeuter -r instead of just newsbeuter which starts newsbeuter without refreshing anything. Now that you have some feeds with new posts how do you move about and read them? The n key moves from one unread post to the next, it will jump to the next post in the next feed for you. When you read a post and hit n, it will mark the last one as read and move on. You can of course move about manually if you like, with q and enter acting as back and forward through the feeds and posts. When it runs out of unread posts it will return to the main feed page. If you skip forward a bit and start in the middle, it will cycle back to the start when it reaches the end so you won't miss anything.

In the posts themselves, you'll notice that some have images etc that need to be seen in something other than a text format. Clicking o will open that post in the web browser you have chosen in the config file. I'll get to the config file later. When you close the browser the post reappears in the terminal as before.

Podcast feeds have an additional piece of information in them ie the audio or video file location. In the example you'll see the Link option pointing to a .ogg file. This is the audio file for this audiocast episode. To download it, you simply press e to enqueue it. At the bottom of the screenshot you'll see the results of that command, telling you it's been added to the queue. It's important to note that it's not actually downloading anything yet. That's where Podbeuter comes in.

I have an ISP deal where my bandwidth is counted during the day but unlimited at night, so I usually queue up the days new podcast episodes and get them all in one swoop overnight. I also refresh my RSS feeds several times during the day. Using Newsbeuter and Podbeuter I can read new podcast posts any time and simply enqueue them, therefore marking them as read, and run podbeuter at night to do the actual downloading. It's great.

Podbeuter is a function of Newsbeuter, it's not a separate application. To run Podbeuter it's simply podbeuter in the terminal. This will show your queue of waiting episodes. To download a single episode you simply highlight the line it's on and press d. The easier way is to tell it to download the whole queue in one swoop, which is done by simply pressing a.

You'll notice at the top there's additional information about totals and the number of parallel downloads. The total will keep count of everything it has not purged from the list, whether they are downloaded or not. To clear this list I usually P to purge the list clean after each group of new episodes have been finished. This will drop the total count back to 0. The list itself will clear when podbeuter is next started. Note that the purge command does not delete the file itself, only the queue. You can remove a file from the queue by D.

So how do you get your feeds into Newsbeuter and set up download locations etc? This is where it got confusing for me, as there are not one but two config files for Newsbeuter.

gedit .newsbeuter/urls

This is a hidden file in your home folder hence the name starting with a dot. This is a simple plain text file. Copy your RSS feed urls in there, one feed per line and save the file. When you enqueue any audiocasts it will create and start using a file called queue which is also in that folder. This is how it remembers your enqueued files to download when you close and reopen Podbeuter.

If you find your feeds get a bit messed up, you can delete the cache.db database file. It'll create a fresh one when it starts again and give you a fresh start. You won't have to re-enter your feed information as it reads from the urls file. What you will notice is that it marks everything as unread, as that information was in he old cache.db file. You can import lists in too rather than adding them by hand. They will show up in the order you list them.

gksu gedit /etc/newsbeuter/config

This is where you set the number of parallel downloads, the default download location and web browser to open when you hit the o command on a post. I just copied mine from the newsbeuter.org documentation and modified it a little. In my case I have Firefox 3.5, as my browser, Audacious as my player, a different download location and 4 parallel downloads. This file needs root privileges to edit, hence the gksu part.

The documentation for Newsbeuter in general is excellent, as is the application itself, I just had major issues trying to figure how to get it working with podcasts and downloading the enclosed media files. The documentation seemed to be lacking in that, or I just didn't understand it.

I heard klaatu was looking for a good CLI RSS reader and recommended it to him. I noticed he has trouble with the pronunciation. I suspect he his not alone. I think of beut as in "beautiful" or "you beauty". Or for the Aussies out there, think of it as "newz bewt ah" No matter how you pronounce it, it's an incredible little application, which I now use for both my regular feeds and podcasts.