Downloading Podcasts

2-minute read
rust

Today I decided to write a little program to download podcasts for me. Turns out Rust has an excellent rss crate which does most of the heavy lifting. The rest is done by reqwest.

I had something rudimentary working in short order

async fn get_feed(url: &str) -> Result<Channel, Box<dyn std::error::Error>> {
    let content = reqwest::get(url).await?.bytes().await?;
    let channel = Channel::read_from(&content[..])?;
    Ok(channel)
}

and I started happily adding the podcasts I wanted to it. But when I got to My Favorite Theorem, it stopped working. The only feedback from rss was Error: InvalidStartTag. Hrm. I looked at the RSS file, but it seemed okay. What's up?

I had a look at the git repo and there was only one place where that error appears I cloned the repo and added a debug statement there.

    dbg!(element.name());

It produced "104, 101, 97, 100" which spells "head." What? There isn't a <head tag in that file anywhere. Then I realized I was trying to parse the bytes of a response failure as an RSS feed. This example assumes success! So much for testing a bugfix in rss.

So why was reqwest giving me a failure for just that one podcast? What was different about it? I found the answer in the closed issues: there is no user-agent header. None of the other podcast feeds cared, but My Favorite Theorem (really Squarespace, I guess) did. Thanks to Bauke for asking my question last year and thanks to seanmonstar for answering it!

There are a number of ways to add a header to a reqwest. Here's one

async fn get_feed(url: &str) -> Result<Channel, Box<dyn std::error::Error>> {
    let client = reqwest::Client::new();

    let content = client
        .get(url)
        .header(reqwest::header::USER_AGENT, "feedread 0.1")
        .send()
        .await?
        .bytes()
        .await?;

    let channel = Channel::read_from(&content[..])?;

    Ok(channel)
}

But it turns out the ClientBuilder struct has a user_agent method with an example that does exactly what I want: name the user agent after the app.

// Name your user agent after your app?
const APP_USER_AGENT: &str = concat!(
    env!("CARGO_PKG_NAME"),
    "/",
    env!("CARGO_PKG_VERSION"),
);

Then, in main

let client = reqwest::Client::builder()
    .user_agent(APP_USER_AGENT)
    .build()?;

and pass that client in to the get_feed routine

async fn get_feed(
    client: &reqwest::Client,
    url: &str,
) -> Result<Channel, Box<dyn std::error::Error>> {
    let res = client.get(url).send().await?;
    if res.status() != 200 {
        return Err(format!("Response status {}", res.status()).into());
    }
    let content = res.bytes().await?;
    let channel = Channel::read_from(&content[..])?;
    Ok(channel)
}

I'm guessing I will want that almost every time I use reqwest. And using ClientBuilder seems better than using Client, as I had been doing.

And now it looks like the snow has stopped, so it's time to go shovel!