Web Scraping with Rust: Step-by-Step Guide with Working Code

Yes, Rust is a good fit for scrapers that run for a long time or fetch many pages. It uses little memory, handles many concurrent requests efficiently with tokio, and compiles to a single binary that's easy to deploy. Python is quicker for one-off scripts and has more mature browser tooling, so the right choice depends on the job.

Choose Rust when:

• The scraper runs continuously, on a schedule, or across thousands of pages.
• You want the compiler to catch type errors and missing fields before production does.
• You deploy to small servers or containers and want one self-contained binary.

Choose Python when:

• You're prototyping, exploring a site, or writing a one-off script. Our guide to web scraping with Python covers that path.
• You need heavy browser automation or a specialized library that has no Rust equivalent yet.

Many teams do both: they explore a site in Python, then move the long-running crawl to Rust once the selectors are stable.

You need an HTTP client, an HTML parser, and an async runtime. reqwest, scraper, and tokio cover those three jobs for most projects, and a few helper crates handle URLs, output, and concurrency.

Skip a few crates in new code. kuchiki 0.8.1 is unmaintained (kuchikiki is the maintained fork), and soup 0.5.1 and voyager 0.2.1 haven't been updated in years.

Setting up takes four steps: install the toolchain, check it, create a project, and add the crates.

1. Install Rust with rustup. On macOS and Linux, the installer is one shell command. On Windows, download and run rustup-init.exe.
2. Check that the compiler and Cargo are on your path:

rustc --version
cargo --version

3. Create a new binary project:

cargo new rust-scraper
cd rust-scraper

4. Add the crates with the features this guide uses:

cargo add reqwest --features query
cargo add tokio --features full
cargo add serde --features derive
cargo add scraper serde_json url csv futures anyhow

You only need the query feature for the Scraper API section, but turning it on now saves you a confusing compile error later. Your Cargo.toml should end up with a dependencies block like this:

[dependencies]
anyhow = "1.0.104"
csv = "1.4.0"
futures = "0.3.34"
reqwest = { version = "0.13.5", features = ["query"] }
scraper = "0.29.0"
serde = { version = "1.0.229", features = ["derive"] }
serde_json = "1.0.154"
tokio = { version = "1.54.1", features = ["full"] }
url = "2.5.8"

reqwest 0.13 changed several defaults and feature names, so code written for 0.11 or 0.12 often stops compiling after the upgrade. Here are the changes you're most likely to hit, and the error each one produces:

• rustls is now the default TLS backend. You no longer need a feature flag to get rustls. If you specifically need the platform's native TLS, enable native-tls.
• The rustls-tls feature was renamed to rustls. Cargo stops with "package depends on reqwest with feature rustls-tls but reqwest does not have that feature." Remove the feature, since rustls is already the default.
• .query() and .form() are now opt-in features. Without them, the compiler reports error E0599, "no method named query found for struct RequestBuilder." Enable query or form in Cargo.toml.
• reqwest::blocking still needs the blocking feature. Without it, you get "could not find blocking in reqwest."

The full list is in the reqwest changelog.

Build one reqwest::Client, give it a User-Agent and a timeout, and reuse it for every request. The client holds a connection pool, so creating a new one per request throws that pool away. Replace the contents of src/main.rs with this:

use anyhow::Result;
use reqwest::Client;
use std::time::Duration;

fn build_client() -> Result<Client> {
    let client = Client::builder()
        .user_agent("rust-scraper-tutorial/0.1 (+https://example.com/contact)")
        .timeout(Duration::from_secs(30))
        .build()?;
    Ok(client)
}

async fn fetch(client: &Client, url: &str) -> Result<String> {
    let body = client
        .get(url)
        .send()
        .await?
        .error_for_status()?
        .text()
        .await?;
    Ok(body)
}

#[tokio::main]
async fn main() -> Result<()> {
    let client = build_client()?;
    let html = fetch(&client, "https://books.toscrape.com/").await?;
    println!("downloaded {} bytes", html.len());
    Ok(())
}

Run it with cargo run. A few details matter:

• #[tokio::main] starts the tokio runtime so main can be async.
• error_for_status() turns 4xx and 5xx responses into errors. Without it, reqwest happily returns the body of a 404 page, and your parser silently finds nothing.
• The User-Agent names your scraper and gives a contact URL. That's polite on a practice site. On real sites, you'll often need to set realistic request headers as well.
• anyhow::Result lets you use ? on errors from reqwest, url, csv, and serde without writing your own error type.

The reqwest documentation covers headers, cookies, proxies, and the rest of the ClientBuilder options.

The scraper crate parses an HTML string into a document you can query with CSS selectors. You parse once with Html::parse_document, compile each selector with Selector::parse, and then call .select() to iterate over matching elements. Add this function below fetch:

use scraper::{Html, Selector};

fn first_look(html: &str) {
    let document = Html::parse_document(html);
    let card = Selector::parse("article.product_pod").expect("valid selector");
    let title_link = Selector::parse("h3 > a").expect("valid selector");
    let price = Selector::parse("p.price_color").expect("valid selector");

    println!("books on this page: {}", document.select(&card).count());

    if let Some(first) = document.select(&card).next() {
        let link = first.select(&title_link).next().expect("title link");
        let title = link.value().attr("title").unwrap_or_default();
        let price_text: String = first.select(&price).next().expect("price").text().collect();
        println!("first book: {title} ({})", price_text.trim());
    }
}

Then call first_look(&html); in main after the fetch. The output shows the 20 product cards on the first page:

books on this page: 20
first book: A Light in the Attic (£51.77)

Three methods do most of the work:

• .select(&selector) returns an iterator of ElementRef values. You can call .select() again on each one to search inside it, which is how you scope a query to a single card.
• .value().attr("title") reads an attribute. It returns Option<&str>, because the attribute may not exist.
• .text() iterates over the text nodes inside an element, and .collect::<String>() joins them.

Book titles on this site are truncated in the link text, so we read the full title from the title attribute instead. If you need a refresher on CSS selector syntax, our CSS selectors cheat sheet covers the patterns scrapers use most. The scraper crate docs list the full API.

Use Html::parse_document for full pages you download. It builds the complete tree, including <html>, <head>, and <body>. Html::parse_fragment is meant for HTML snippets, such as a chunk you already cut out of a page or the markup inside a JSON field. Passing a whole page to parse_fragment can give you a tree that doesn't match what the browser shows, and selectors that depend on the document structure stop matching.

Define a struct for the record you want, fill it from each card, and derive Serialize so serde can write it out later. For each book, keep the title, the price as a number, the star rating as 1 to 5, the stock status, and an absolute URL.

use scraper::{Html, Selector};
use serde::Serialize;
use std::sync::LazyLock;
use url::Url;

#[derive(Debug, Serialize)]
struct Book {
    title: String,
    price: f64,
    rating: u8,
    in_stock: bool,
    url: String,
}

struct BookSelectors {
    card: Selector,
    link: Selector,
    price: Selector,
    rating: Selector,
    stock: Selector,
    next: Selector,
}

static SEL: LazyLock<BookSelectors> = LazyLock::new(|| {
    let s = |css: &str| Selector::parse(css).expect("valid selector");
    BookSelectors {
        card: s("article.product_pod"),
        link: s("h3 > a"),
        price: s("p.price_color"),
        rating: s("p.star-rating"),
        stock: s("p.instock.availability"),
        next: s("li.next > a"),
    }
});

fn rating_from_class(class: &str) -> u8 {
    match class.split_whitespace().find(|c| *c != "star-rating") {
        Some("One") => 1,
        Some("Two") => 2,
        Some("Three") => 3,
        Some("Four") => 4,
        Some("Five") => 5,
        _ => 0,
    }
}

/// Parses one listing page. Kept sync on purpose: scraper::Html is not Send.
fn parse_books(html: &str, page_url: &Url) -> (Vec<Book>, Option<Url>) {
    let doc = Html::parse_document(html);

    let books = doc
        .select(&SEL.card)
        .filter_map(|card| {
            let a = card.select(&SEL.link).next()?;
            let title = a.value().attr("title")?.trim().to_string();
            let href = a.value().attr("href")?;
            let price_text: String = card.select(&SEL.price).next()?.text().collect();
            let price = price_text.trim().trim_start_matches('£').parse::<f64>().ok()?;
            let rating = card
                .select(&SEL.rating)
                .next()
                .and_then(|p| p.value().attr("class"))
                .map(rating_from_class)
                .unwrap_or(0);
            let in_stock = card
                .select(&SEL.stock)
                .next()
                .map(|p| p.text().collect::<String>().contains("In stock"))
                .unwrap_or(false);
            let url = page_url.join(href).ok()?.to_string();
            Some(Book { title, price, rating, in_stock, url })
        })
        .collect();

    let next_url = doc
        .select(&SEL.next)
        .next()
        .and_then(|a| a.value().attr("href"))
        .and_then(|href| page_url.join(href).ok());

    (books, next_url)
}

This replaces first_look from the previous step, so delete that function and its separate scraper import. What the code does:

• Selectors are built once. LazyLock compiles every selector the first time SEL is used and reuses them on every page after that. Calling Selector::parse inside the card loop would recompile the same selector 20 times per page.
• The rating comes from a class name. The site marks ratings as class="star-rating Three", so rating_from_class skips star-rating and maps the word to a number.
• The price is parsed into an f64. Whitespace and the £ sign come off first. Numbers stored as numbers are easier to sort and filter later.
• Relative links become absolute with Url::join. The href values on the page are relative, such as catalogue/a-light-in-the-attic_1000/index.html. Joining them to the page URL resolves them the way a browser would, including ../ segments on deeper pages.
• filter_map skips broken cards. If a card is missing a title or price, the ? operators return None and that card is dropped instead of crashing the run.
• The function also returns the next page URL, which the pagination step needs.

To print the first two books, update main:

#[tokio::main]
async fn main() -> Result<()> {
    let client = build_client()?;
    let url = Url::parse("https://books.toscrape.com/")?;
    let html = fetch(&client, url.as_str()).await?;
    let (books, next) = parse_books(&html, &url);
    println!("{}", serde_json::to_string_pretty(&books[..2])?);
    println!("next page: {:?}", next.map(|u| u.to_string()));
    Ok(())
}

The output is clean, typed JSON with absolute URLs:

[
  {
    "title": "A Light in the Attic",
    "price": 51.77,
    "rating": 3,
    "in_stock": true,
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
  },
  {
    "title": "Tipping the Velvet",
    "price": 53.74,
    "rating": 1,
    "in_stock": true,
    "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
  }
]
next page: Some("https://books.toscrape.com/catalogue/page-2.html")

To scrape every page, follow the "next" link until it disappears. Each loop fetches a page, parses its books, and reads the next URL that parse_books already resolved with Url::join. The loop stops when there's no next link, or when it hits a page cap you set as a safety net.

const MAX_PAGES: usize = 60;

async fn scrape_all(client: &Client, start: &str) -> Result<Vec<Book>> {
    let mut next = Some(Url::parse(start)?);
    let mut all = Vec::new();
    let mut pages = 0;

    while let Some(url) = next.take() {
        if pages >= MAX_PAGES {
            eprintln!("stopping at the {MAX_PAGES}-page cap");
            break;
        }
        let html = fetch(client, url.as_str()).await?;
        let (books, next_url) = parse_books(&html, &url);
        all.extend(books);
        pages += 1;
        next = next_url;
        tokio::time::sleep(Duration::from_millis(500)).await;
    }

    println!("pages: {pages}, books: {}", all.len());
    Ok(all)
}

Call it with scrape_all(&client, "https://books.toscrape.com/catalogue/page-1.html").await? and it walks all 50 pages of the catalog, collecting 1,000 books:

pages: 50, books: 1000

while let Some(url) = next.take() is a compact way to say "keep going while there's a next page." The half-second sleep keeps the loop from hitting the server as fast as your connection allows, and the page cap protects you from a site whose "next" link points back to itself.

This is still scraping, not crawling: you follow one known chain of pages rather than discovering every link on a site. Not sure which you need? See our explainer on scraping vs crawling. When the job grows into collecting many related pages on a schedule, a managed, bounded crawl can take over the URL discovery and limits.

When you already know the page URLs, you can fetch several at once instead of one by one. Use futures::stream with buffer_unordered(n), which keeps at most n requests in flight. The catalog's page URLs follow a pattern, so we can build all 50 up front:

use futures::stream::{self, StreamExt};

async fn scrape_concurrently(client: &Client) -> Vec<Book> {
    let urls: Vec<Url> = (1..=50)
        .map(|n| {
            Url::parse(&format!("https://books.toscrape.com/catalogue/page-{n}.html"))
                .expect("valid URL")
        })
        .collect();

    let results: Vec<(Url, Result<Vec<Book>>)> = stream::iter(urls)
        .map(|url| {
            let client = client.clone();
            async move {
                let result = async {
                    let html = fetch(&client, url.as_str()).await?;
                    Ok(parse_books(&html, &url).0)
                }
                .await;
                (url, result)
            }
        })
        .buffer_unordered(4)
        .collect()
        .await;

    let mut books = Vec::new();
    for (url, result) in results {
        match result {
            Ok(page_books) => books.extend(page_books),
            Err(e) => eprintln!("failed {url}: {e:#}"),
        }
    }
    books
}

Why it's built this way:

• client.clone() is cheap. A reqwest Client is a handle to a shared connection pool, so every task reuses the same connections.
• buffer_unordered(4) caps concurrency at four requests. A higher number fetches faster, but it also loads the server harder. Start low on any site you don't own.
• Each page keeps its own Result. One failed page is logged with its URL instead of aborting the other 49. You can collect the failed URLs and retry them later.
• Results arrive out of order. If order matters, sort the books afterward or use buffered instead of buffer_unordered.

If async Rust is new to you, the tokio tutorial explains tasks and runtimes, and the docs for buffer_unordered cover the other stream combinators.

scraper::Html isn't Send, so a task started with tokio::spawn can't keep one alive across an .await. tokio may move a spawned task to another thread at any .await, and the compiler refuses anything that isn't safe to move. This version fails:

// Does not compile: `doc` is still alive at the second .await
let handle = tokio::spawn(async move {
    let html = fetch(&client, "https://books.toscrape.com/").await?;
    let doc = Html::parse_document(&html);
    let next = fetch(&client, "https://books.toscrape.com/catalogue/page-2.html").await?;
    Ok::<_, anyhow::Error>(doc.root_element().text().count() + next.len())
});

The compiler reports:

error: future cannot be sent between threads safely
note: future is not `Send` as this value is used across an await
    let doc = Html::parse_document(&html);
        --- has type `Html` which is not `Send`

The fix is to parse inside a regular sync function, or a block that ends before the next .await, and return owned data. That's why parse_books is a plain fn that returns Vec<Book>:

// Compiles: parsing happens in a sync function that returns owned data
let handle = tokio::spawn(async move {
    let url = Url::parse("https://books.toscrape.com/")?;
    let html = fetch(&client, url.as_str()).await?;
    let (books, _) = parse_books(&html, &url);
    let next = fetch(&client, "https://books.toscrape.com/catalogue/page-2.html").await?;
    Ok::<_, anyhow::Error>(books.len() + next.len())
});

The Html value is created and dropped inside parse_books, so the task only holds String and Book values across the .await.

Retry only the errors that might succeed a second time: 429 Too Many Requests and 5xx server errors. Wait longer after each failure, and if the server sends a Retry-After header, wait that long instead. Don't retry 404 or other 4xx responses, because asking again won't change the answer.

use reqwest::StatusCode;

async fn get_with_retry(client: &Client, url: &str, max_tries: u32) -> Result<String> {
    let mut delay = Duration::from_millis(500);

    for attempt in 1..=max_tries {
        match client.get(url).send().await {
            Ok(res) if res.status().is_success() => return Ok(res.text().await?),
            Ok(res)
                if res.status() == StatusCode::TOO_MANY_REQUESTS
                    || res.status().is_server_error() =>
            {
                let wait = res
                    .headers()
                    .get(reqwest::header::RETRY_AFTER)
                    .and_then(|v| v.to_str().ok())
                    .and_then(|s| s.parse::<u64>().ok())
                    .map(Duration::from_secs)
                    .unwrap_or(delay);
                eprintln!("attempt {attempt}: {} for {url}, waiting {wait:?}", res.status());
                tokio::time::sleep(wait).await;
            }
            Ok(res) => anyhow::bail!("{} for {url}", res.status()),
            Err(e) if attempt < max_tries => {
                eprintln!("attempt {attempt}: {e}");
                tokio::time::sleep(delay).await;
            }
            Err(e) => return Err(e.into()),
        }
        delay *= 2;
    }

    anyhow::bail!("gave up on {url} after {max_tries} tries")
}

The delay doubles after each failed attempt: half a second, then one second, then two. Network errors such as timeouts are retried too, until the last attempt. A missing page fails fast:

404 Not Found for https://books.toscrape.com/does-not-exist.html

To use it everywhere, swap fetch for get_with_retry(&client, url.as_str(), 3). This helper reads only the numeric form of Retry-After. Some servers send an HTTP date instead, and in that case the helper falls back to its own delay. If you want a steady request rate across many tasks instead of reacting to errors, add the governor crate and wait on its rate limiter before each request.

Because Book derives Serialize, the csv and serde_json crates can write it without any mapping code. The csv crate uses the struct's field names as the header row.

fn save_csv(books: &[Book], path: &str) -> Result<()> {
    let mut writer = csv::Writer::from_path(path)?;
    for book in books {
        writer.serialize(book)?;
    }
    writer.flush()?;
    Ok(())
}

fn save_json(books: &[Book], path: &str) -> Result<()> {
    std::fs::write(path, serde_json::to_string_pretty(books)?)?;
    Ok(())
}

Wire everything together in main:

#[tokio::main]
async fn main() -> Result<()> {
    let client = build_client()?;
    let books = scrape_all(&client, "https://books.toscrape.com/catalogue/page-1.html").await?;
    save_csv(&books, "books.csv")?;
    save_json(&books, "books.json")?;
    Ok(())
}

The first lines of books.csv look like this:

title,price,rating,in_stock,url
A Light in the Attic,51.77,3,true,https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
Tipping the Velvet,53.74,1,true,https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html

The csv crate quotes fields that contain commas or quotes, so titles with punctuation stay in one column. Call writer.flush() before the writer goes out of scope, or a write error at the end gets lost. For very large runs, write each page's books as they arrive instead of holding everything in memory.

reqwest only sees the HTML the server sends. If a page builds its content with JavaScript after it loads, that content isn't in the response, and your selectors return nothing. Confirm that's what is happening before you change tools, then pick the lightest fix that works: embedded data first, a headless browser second, and a managed API when rendering is only part of the problem.

Fetch the page with reqwest and count the elements you expect. quotes.toscrape.com/js/ shows 10 quotes in a browser. For this part, keep build_client and fetch in src/main.rs and replace the rest with the code below. Here's what reqwest sees:

use scraper::{Html, Selector};
use serde::Deserialize;

fn count_quotes(html: &str) -> usize {
    let doc = Html::parse_document(html);
    let quote = Selector::parse("div.quote").expect("valid selector");
    doc.select(&quote).count()
}

div.quote in raw HTML: 0

Zero matches in the raw HTML and 10 on screen means JavaScript builds the quotes. You can confirm the same thing by hand: View Source shows what the server sent, while the DevTools Elements panel shows the page after scripts ran. If your target text appears only in DevTools, the page needs JavaScript or a different data source.

Many JavaScript pages ship their data inside the HTML, in an inline <script> that the page's code reads. quotes.toscrape.com/js/ does this with var data = [...]. You can slice that array out of the raw HTML and deserialize it with serde, with no browser involved. Add these types and the function to the same file:

#[derive(Debug, Deserialize)]
struct Author {
    name: String,
}

#[derive(Debug, Deserialize)]
struct Quote {
    text: String,
    author: Author,
    tags: Vec<String>,
}

fn quotes_from_inline_script(html: &str) -> Result<Vec<Quote>> {
    let marker = "var data =";
    let start = html
        .find(marker)
        .ok_or_else(|| anyhow::anyhow!("no `var data` in the page"))?
        + marker.len();
    let rest = &html[start..];
    let end = rest
        .find("];")
        .ok_or_else(|| anyhow::anyhow!("end of the data array not found"))?
        + 1;
    Ok(serde_json::from_str(rest[..end].trim())?)
}

Call both functions on the HTML from fetch:

#[tokio::main]
async fn main() -> Result<()> {
    let client = build_client()?;
    let html = fetch(&client, "https://quotes.toscrape.com/js/").await?;
    println!("div.quote in raw HTML: {}", count_quotes(&html));
    let quotes = quotes_from_inline_script(&html)?;
    println!("quotes from the inline script: {}", quotes.len());
    let first = &quotes[0];
    println!("{} ({}) {:?}", first.text, first.author.name, first.tags);
    Ok(())
}

div.quote in raw HTML: 0
quotes from the inline script: 10
“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.” (Albert Einstein) ["change", "deep-thoughts", "thinking", "world"]

That's all 10 quotes, typed and structured, from a single HTTP request. serde ignores JSON fields your struct doesn't declare, so you only model what you need. String slicing like this is brittle if the script changes, so keep the marker specific and fail loudly when it's missing.

The same idea works for data loaded after the page opens. Open the DevTools Network tab, filter by Fetch/XHR, and reload. If the page calls a JSON endpoint, you can often request that endpoint directly with reqwest and deserialize the response, which is faster and more stable than parsing HTML.

If the data only exists after rendering, drive a real browser. The headless_chrome crate controls Chrome or Chromium through the DevTools Protocol with a synchronous API, so it doesn't need tokio. Add it with cargo add headless_chrome. It needs Chrome or Chromium installed on the machine that runs the scraper. Save this as src/bin/headless.rs and run it with cargo run --bin headless.

use headless_chrome::{Browser, LaunchOptions};
use scraper::{Html, Selector};

fn main() -> anyhow::Result<()> {
    let browser = Browser::new(LaunchOptions::default_builder().headless(true).build()?)?;
    let tab = browser.new_tab()?;
    tab.navigate_to("https://quotes.toscrape.com/js/")?;
    tab.wait_for_element("div.quote")?;
    let html = tab.get_content()?;

    let doc = Html::parse_document(&html);
    let text = Selector::parse("div.quote span.text").expect("valid selector");
    let quotes: Vec<String> = doc.select(&text).map(|e| e.text().collect()).collect();
    println!("rendered quotes: {}", quotes.len());
    Ok(())
}

rendered quotes: 10

wait_for_element blocks until the selector matches, so you don't read the page before the scripts finish. get_content returns the rendered HTML, which you parse with the same scraper code as before. Rendering costs far more CPU and memory than a plain request, so use it only for the pages that need it. The headless_chrome docs cover clicks, typing, screenshots, and request interception.

chromiumoxide also speaks the DevTools Protocol, but with an async API that fits into a tokio scraper. It returns a browser and a handler, and you must spawn a task that keeps polling the handler, or every browser call hangs. fantoccini and thirtyfour take the WebDriver route instead. They drive Chrome or Firefox through chromedriver or geckodriver, which you have to install and run alongside your scraper. All of these mean running and patching browsers yourself. If you need rendering, waits, and screenshots, you can also render pages without running browsers by sending the work to a hosted browser runtime.

The scraper you just built works because books.toscrape.com and quotes.toscrape.com are practice sites that welcome automated requests. Many production sites don't. The same code that returns 1,000 clean books can start failing with 403 responses, CAPTCHA pages, per-IP rate limits, browser fingerprint checks, or content that changes by country. Rust's speed doesn't help here, because the limits are about who's asking, not how fast.

Each DIY fix adds infrastructure to run: proxy pools, browser fleets, retry queues, and monitoring. Our guide on how to avoid getting blocked goes deeper on headers, pacing, and proxies. At some point, it makes more sense to hand the fetching to a service and keep your Rust code for parsing and storage.

Scraper API fetches the page for you, handles access and retries, and can render JavaScript. Your Rust code keeps the same reqwest client and the same scraper parsing. You send one GET request with your API key and a URL-encoded target URL, then add options such as render_js or country only when the page needs them.

This example renders the JavaScript quotes page through the API, from a US location, and parses the result. It reads the key from an environment variable, so it never ends up in your source code:

use anyhow::{Context, Result};
use scraper::{Html, Selector};
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<()> {
    let api_key = std::env::var("WSA_API_KEY").context("set WSA_API_KEY first")?;
    let client = reqwest::Client::builder()
        .timeout(Duration::from_secs(120))
        .build()?;

    let res = client
        .get("https://api.webscrapingapi.com/v2")
        .query(&[
            ("api_key", api_key.as_str()),
            ("url", "https://quotes.toscrape.com/js/"),
            ("render_js", "1"),
            ("country", "us"),
        ])
        .send()
        .await?;

    let status = res.status();
    let body = res.text().await?;
    if !status.is_success() {
        anyhow::bail!("API returned {status}: {}", &body[..body.len().min(300)]);
    }

    let doc = Html::parse_document(&body);
    let quote = Selector::parse("div.quote").expect("valid selector");
    println!("quotes via Scraper API: {}", doc.select(&quote).count());
    Ok(())
}

Save it as src/bin/scraper_api.rs, set the key in your shell, and run it:

export WSA_API_KEY="your_api_key"
cargo run --bin scraper_api

What to notice:

• .query() encodes the target URL for you. That's the reqwest method that needs the query feature you enabled during setup.
• The timeout is longer than before. Rendering a page takes longer than a plain fetch, so 120 seconds leaves room for the API to load and render it.
• The status check prints the error body. An invalid key returns 401 Unauthorized with {"error":"Invalid API key"}, which is far easier to debug than a parser that finds zero elements.
• The parsing code doesn't change. The response is HTML, so the selectors and structs from earlier sections work as they are.

Keep the API key on the server side. Don't ship it in client apps, commit it to a repository, or print it in logs.

Most Rust scraping errors come from crate feature changes, async rules, or pages that don't contain what you expect. Here's each symptom with its fix.

When an error isn't in the table, print the status code and the first few hundred bytes of the response body. Most scraping bugs get obvious once you see what the server actually sent.

Scrape in a way the site owner can live with. Read the site's robots.txt, which follows the Robots Exclusion Protocol (RFC 9309), and skip the paths it disallows for your user agent. Keep concurrency low, add delays, and honor Retry-After when the server asks you to slow down. Identify your client with a User-Agent that includes a contact URL. Follow the site's terms of service, and avoid collecting personal data unless you have a clear legal basis for it. For more on how we approach this, see our responsible web data practices.

Yes, especially for scrapers that run for a long time or fetch many pages. Rust uses little memory, handles many concurrent requests efficiently with tokio, and ships as a single binary. Python is quicker for one-off scripts and has more browser tooling, so many teams prototype in Python and move the hot loop to Rust.

Most projects use reqwest to send requests and scraper to parse HTML with CSS selectors, running on tokio. Add serde for structured output, csv for spreadsheets, and futures for bounded concurrency. For JavaScript-rendered pages, add headless_chrome or chromiumoxide. Use spider when you need a full crawler framework instead of a custom loop.

First confirm the data is missing from the raw HTML. Then look for it in an inline script or a JSON request the page makes, and parse that with serde. If the data only exists after rendering, drive Chrome with headless_chrome or chromiumoxide, or call a scraping API that renders JavaScript for you.

reqwest 0.13 changed its features. The .query() and .form() methods now need the query and form features, the rustls-tls feature was renamed to rustls, and rustls is the default TLS backend. Blocking requests still need the blocking feature. Update the features in Cargo.toml and most errors disappear.

The scraper::Html type isn't Send, so a tokio task can't keep it alive across an .await. Parse the page in a regular function or a block that ends before the next .await, and return owned values such as String or your own struct. The task then only holds data that is safe to move.

Send a clear User-Agent, reuse one client, limit concurrency, add delays, and retry 429 responses with backoff that honors Retry-After. Respect robots.txt and the site's terms. Sites with CAPTCHAs, fingerprinting, or per-IP limits may still block you, which is where rotating IPs or a managed scraping API helps.

Yes. Enable reqwest's blocking feature and use reqwest::blocking::Client, or use the smaller ureq crate, which is synchronous by design. Parsing with scraper works the same either way. Blocking code is simpler for small scripts, but async with tokio scales better when you fetch many pages at once.

You now have a Rust scraper that fetches, parses, paginates, runs concurrently, retries politely, and exports clean data. A few next steps make it production-ready:

• Add steady rate limiting with the governor crate, so every task shares one request budget instead of relying on sleeps.
• Render only where you must. Keep reqwest for static pages and embedded JSON, and send only the pages that truly need JavaScript to headless Chrome.
• Hand off fetching when blocks start. Once you're spending more time on proxies and CAPTCHAs than on parsing, move the fetch step to a managed API and keep the rest of your Rust code.

Point the same reqwest client at Scraper API and get the page back rendered, from the country you choose, without running browsers or proxies yourself. Start with 5,000 free requests. No extra credits for JavaScript rendering or geotargeting, and failed requests aren't billed.

See Scraper API pricing