Web Scraping with Ruby: Step-by-Step Guide with Working Code

Yes. Ruby has mature gems for every step of a scraper: Faraday to fetch pages, Nokogiri to parse them, Ferrum to drive headless Chrome, and Mechanize for forms and sessions. The code stays short and readable, and it fits naturally when your app already runs on Ruby or Rails. Python has a larger scraping ecosystem and more browser tooling.

Choose Ruby when:

• Your app, team, or job queue already runs on Ruby or Rails, and you want the scraper in the same codebase.
• You want a readable scraper with a handful of well-known gems rather than a large framework.

Choose Python when:

• You need a library or browser tool that has no Ruby equivalent. Our guide to web scraping with Python covers that path.
• Your team works mostly in data notebooks and wants the scraper next to the analysis.

You need an HTTP client and an HTML parser. Faraday and Nokogiri cover both jobs, and a few small gems add retries, CSV output, and a browser when a page needs one.

Watir's latest release dates from August 2023, so prefer Ferrum or Selenium in new code. To compare Ruby HTML parsers beyond Nokogiri, see our guide to Ruby libraries for parsing HTML and XML.

A Ruby scraping project needs Ruby 3.4 or newer, Bundler, and a Gemfile that pins the gems above. Setup takes a few commands.

First, install Ruby if you don't have a current version, then check it:

ruby -v

Ruby 4.0 is the current stable series, and 3.4 works too. Ruby 3.3 receives security fixes only, and Ruby 3.2 reached end of life in April 2026.

Create the project and a Gemfile:

mkdir book_scraper && cd book_scraper
bundle init

Replace the generated Gemfile with this one:

# frozen_string_literal: true

source "https://rubygems.org"

ruby ">= 3.4"

gem "faraday", "~> 2.15"
gem "faraday-retry", "~> 2.4"
gem "nokogiri", "~> 1.19"
gem "csv", "~> 3.3"

group :browser do
  gem "ferrum", "~> 0.20"
end

group :test do
  gem "minitest", "~> 6.0"
  gem "webmock", "~> 3.34"
end

The ~> pins accept minor releases but not the next major version. Ferrum sits in a :browser group because only the JavaScript section needs it. Install everything:

bundle install

The scraper lives in lib/book_scraper.rb, and small scripts in scripts/ call it. Run each one with bundle exec ruby scripts/<name>.rb.

Since Ruby 3.4, csv is a bundled gem instead of a default gem, so under Bundler it loads only if your Gemfile lists it. Without it, bundle exec prints this warning:

warning: csv used to be loaded from the standard library, but is not part of the default gems since Ruby 3.4.0.
You can add csv to your Gemfile or gemspec to fix this error.

Then it stops with cannot load such file -- csv (LoadError). The gem "csv", "~> 3.3" line in the Gemfile above fixes it. Plain ruby script.rb outside Bundler still finds csv, which is why the error often appears only once a project gains a Gemfile.

Faraday sends the request and returns a response with a status, headers, and body. Start with one line in irb to confirm the site responds:

require "faraday"
Faraday.get("https://books.toscrape.com/").status # => 200

A real scraper needs one reusable connection that identifies your client, stops waiting on slow servers, raises errors for failed responses, and retries when the server asks you to slow down. Learn how to set realistic request headers before you scrape sites that check them.

Create lib/book_scraper.rb and start the module:

# frozen_string_literal: true

require "faraday"
require "faraday/retry"
require "nokogiri"
require "uri"
require "csv"
require "json"

module BookScraper
  START_URL = "https://books.toscrape.com/catalogue/page-1.html"
  USER_AGENT = "BookScraper/1.0 (+mailto:you@example.com)"
  RATINGS = { "One" => 1, "Two" => 2, "Three" => 3, "Four" => 4, "Five" => 5 }.freeze

  module_function

  # One reusable Faraday connection: timeouts, retries, and errors for non-2xx.
  def client
    Faraday.new(headers: { "User-Agent" => USER_AGENT }) do |f|
      f.options.open_timeout = 5
      f.options.timeout = 15
      f.response :raise_error # outer: raises once retries are used up
      f.request :retry,
                max: 3,
                interval: 0.5,
                backoff_factor: 2,
                interval_randomness: 0.5,
                retry_statuses: [429, 500, 502, 503, 504],
                max_interval: 30 # wait no longer than this per retry
    end
  end

  def fetch(conn, url)
    response = conn.get(url)
    # No charset in Content-Type, so Faraday returns binary; tag it UTF-8 for string methods
    response.body.force_encoding(Encoding::UTF_8)
  end

Here's what each part does:

• The User-Agent names your scraper and gives a contact address, so site owners can reach you. Replace you@example.com with a real address.
• open_timeout and timeout cap the wait to connect (5 seconds) and to read (15 seconds), so one slow page can't hang the run.
• raise_error turns 4xx and 5xx responses into exceptions such as Faraday::ResourceNotFound, so a missing page fails loudly.
• retry retries rate-limited and server-error responses with growing waits. The retry section explains each option and why the order of these two lines matters.

The fetch method returns the body tagged as UTF-8. When the Content-Type header carries no charset, as on this site, Faraday hands back a binary (ASCII-8BIT) string. Nokogiri still reads the page's meta charset, but your own string checks don't. Calling include?("£") on the binary body raises Encoding::CompatibilityError: incompatible character encodings: BINARY (ASCII-8BIT) and UTF-8. Tagging the body with force_encoding fixes that. The Faraday documentation covers adapters, middleware, and connection options in more depth.

Nokogiri turns an HTML string into a document you can search with CSS selectors or XPath. css returns every match, at_css returns the first one, [] reads an attribute, and text reads the content.

On books.toscrape.com, each book sits in an article.product_pod card. Test the selector in irb before you build anything on it:

doc = Nokogiri::HTML(Faraday.get("https://books.toscrape.com/catalogue/page-1.html").body)
doc.css("article.product_pod").size # => 20

Twenty cards match, one per book. The same query in XPath is doc.xpath("//article[@class=\"product_pod\"]"). CSS is shorter for class-based markup, while XPath can match on text and walk up the tree. If you're choosing between CSS selectors or XPath, our comparison covers the trade-offs, and Nokogiri's searching tutorial covers mixed queries. Testing in irb first means a broken selector costs seconds, not a 50-page run that writes an empty file.

Turn each card into a hash with typed values: a String title, a Float price, an Integer rating, a boolean stock flag, and an absolute URL. Typed values sort, filter, and export cleanly. Add parse_books to the module:

  def parse_books(html, page_url)
    doc = Nokogiri::HTML(html)
    doc.css("article.product_pod").map do |card|
      link = card.at_css("h3 a")
      rating_class = card.at_css("p.star-rating")["class"].split.last
      {
        title: link["title"],
        price: card.at_css("p.price_color").text.delete("^0-9.").to_f,
        rating: RATINGS.fetch(rating_class, nil),
        in_stock: card.at_css("p.availability").text.include?("In stock"),
        url: URI.join(page_url, link["href"]).to_s
      }
    end
  end

Why it's built this way:

• The title comes from the link's title attribute, because the visible text is cut short for long titles.
• delete("^0-9.") keeps only digits and the decimal point, so £51.77 becomes 51.77 before to_f.
• The rating lives in a class name such as star-rating Three, which the RATINGS hash maps to an Integer.
• URI.join resolves relative links against the page URL, so every row carries a full URL.

Try it on the first page with scripts/first_page.rb:

require_relative "../lib/book_scraper"

conn = BookScraper.client
html = BookScraper.fetch(conn, BookScraper::START_URL)
books = BookScraper.parse_books(html, BookScraper::START_URL)

puts "books on page 1: #{books.size}"
puts JSON.pretty_generate(books.first(2))
puts "next: #{BookScraper.next_page_url(html, BookScraper::START_URL)}"

The last line uses next_page_url, which you'll add in the next section. Once it's in place, the script prints:

books on page 1: 20
[
  {
    "title": "A Light in the Attic",
    "price": 51.77,
    "rating": 3,
    "in_stock": true,
    "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
  },
  {
    "title": "Tipping the Velvet",
    "price": 53.74,
    "rating": 1,
    "in_stock": true,
    "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
  }
]
next: https://books.toscrape.com/catalogue/page-2.html

To paginate, read the "next" link on each page, resolve it to a full URL, and keep going until the link disappears. Add a delay between pages and a page cap, so a looping site can't keep your scraper running forever.

The catalog marks the next page with li.next a. Add these two methods:

  def next_page_url(html, page_url)
    href = Nokogiri::HTML(html).at_css("li.next a")&.[]("href")
    href && URI.join(page_url, href).to_s
  end

  # Follow "next" links until there are none, with a delay and a page cap.
  def crawl(conn, start_url = START_URL, max_pages: 100, delay: 0.5)
    url = start_url
    books = []
    pages = 0
    while url && pages < max_pages
      html = fetch(conn, url)
      books.concat(parse_books(html, url))
      pages += 1
      url = next_page_url(html, url)
      sleep(delay) if url
    end
    [books, pages]
  end

On the last page there's no link, so the safe navigation operator (&.) returns nil and the loop ends. The max_pages: 100 cap protects you from loops, and the half-second sleep protects the site from bursts. On this catalog, crawl visits 50 pages and returns 1,000 books, which the export section saves to files.

Following links from page to page moves you from scraping toward crawling. Our guide on scraping vs crawling explains where the line sits.

When you already know the page URLs, a small pool of threads fetches several pages at once. Ruby threads help here because the interpreter releases its global lock while a thread waits on the network. A fixed pool size keeps the load on the site predictable.

The catalog's page URLs follow a pattern, so build the list up front and add crawl_concurrently to the module:

  # Fetch a known list of pages with a fixed number of worker threads.
  def crawl_concurrently(urls, workers: 4)
    queue = Queue.new
    urls.each { |u| queue << u }
    queue.close
    results = Queue.new
    errors = Queue.new

    threads = Array.new(workers) do
      Thread.new do
        conn = client # one connection per thread
        while (url = queue.pop)
          begin
            results << parse_books(fetch(conn, url), url)
          rescue Faraday::Error => e
            errors << [url, e.class.name]
          end
        end
      end
    end
    threads.each(&:join)

    books = []
    books.concat(results.pop) until results.empty?
    failed = []
    failed << errors.pop until errors.empty?
    [books, failed]
  end

How the pieces fit together:

• A closed Queue ends the workers cleanly. After close, pop returns the remaining URLs and then nil, which ends each loop, with no sentinel values needed. See the Thread::Queue docs.
• Each thread builds its own connection, so no connection state is shared between threads.
• Errors are caught per page. The rescue records the failed URL and moves on, so one bad page doesn't stop the batch.
• Queue is thread-safe, so workers push results without a mutex.

Run it on all 50 pages plus one URL that doesn't exist, using scripts/scrape_concurrently.rb:

require_relative "../lib/book_scraper"

urls = (1..50).map { |n| "https://books.toscrape.com/catalogue/page-#{n}.html" }
urls << "https://books.toscrape.com/catalogue/page-999.html" # this page doesn't exist

books, failed = BookScraper.crawl_concurrently(urls, workers: 4)
puts "books: #{books.size}, failed: #{failed.inspect}"

The output shows every book and the one page that failed:

books: 1000, failed: [["https://books.toscrape.com/catalogue/page-999.html", "Faraday::ResourceNotFound"]]

Results arrive in whatever order the threads finish, so sort them if order matters. Keep workers small on sites you don't own: the pool exists to put a ceiling on load.

faraday-retry's defaults retry only timeouts and connection failures, with no wait between attempts and no HTTP status codes. A 429 Too Many Requests or a 503 Service Unavailable fails on the first try unless you configure the middleware yourself.

These options in the client method make retries work:

• retry_statuses: [429, 500, 502, 503, 504] tells the middleware which responses to retry. Without it, none are.
• max: 3 allows three retries after the first attempt.
• interval: 0.5, backoff_factor: 2, and interval_randomness: 0.5 start with a half-second wait, double it each time, and add jitter.
• max_interval: 30 caps any single wait at 30 seconds.

When a retried response carries a Retry-After header, the middleware waits as long as the server asks. If that wait is longer than max_interval, it gives up at once: a server that answers Retry-After: 120 gets an immediate Faraday::TooManyRequestsError from this client. Raise max_interval if you'd rather wait. Only idempotent methods such as GET are retried, and a 404 never is.

The order of the two middleware lines matters too. Register f.response :raise_error before f.request :retry. Faraday runs the first-registered middleware on the outside, so retry sees the raw 429 or 503 response, retries it, and raise_error only fires after the retries run out. Swap the two lines and raise_error turns the 429 into a Faraday::TooManyRequestsError before retry ever sees a status code, so nothing gets retried, even with retry_statuses set.

With this setup, a server that returns 429 twice and then 200 gives your code a normal 200. A server that keeps returning 503 gets four requests, then your code receives Faraday::ServerError. The faraday-retry README lists every option, including retry_block for logging attempts.

Write CSV with an explicit header list and JSON with JSON.pretty_generate. Both keep the typed values from the parsing step.

Add the export methods:

  HEADERS = %w[title price rating in_stock url].freeze

  def write_csv(books, path)
    CSV.open(path, "w", write_headers: true, headers: HEADERS) do |csv|
      books.each { |b| csv << b.values_at(*HEADERS.map(&:to_sym)) }
    end
  end

  def write_json(books, path)
    File.write(path, JSON.pretty_generate(books))
  end

values_at pulls the fields in HEADERS order, so rows line up with the header even if you add keys later. Tie the crawl and exports together in scripts/scrape_books.rb:

require_relative "../lib/book_scraper"

books, pages = BookScraper.crawl(BookScraper.client)
BookScraper.write_csv(books, "output/books.csv")
BookScraper.write_json(books, "output/books.json")
puts "pages: #{pages}, books: #{books.size}"

Create the output folder and run it:

mkdir -p output
bundle exec ruby scripts/scrape_books.rb

The script prints pages: 50, books: 1000. output/books.csv has a header row plus 1,000 data rows, and starts like this:

title,price,rating,in_stock,url
A Light in the Attic,51.77,3,true,https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html

output/books.json holds the same 1,000 records as an array of objects. The Ruby 3.4 release notes list csv among the gems that became bundled gems, so this is the step where a missing Gemfile line shows up. Close the module with a final end after the last method.

Save one real page as a fixture, stub the request with WebMock, and assert on the parsed output. These tests run offline and catch selector drift before a scheduled job ships an empty CSV.

Save the first catalog page:

mkdir -p test/fixtures
curl -o test/fixtures/page-1.html https://books.toscrape.com/catalogue/page-1.html

Then write test/book_scraper_test.rb:

require "minitest/autorun"
require "webmock/minitest"
require_relative "../lib/book_scraper"

class BookScraperTest < Minitest::Test
  PAGE = "https://books.toscrape.com/catalogue/page-1.html"

  def setup
    stub_request(:get, PAGE).to_return(status: 200, body: File.read(File.join(__dir__, "fixtures/page-1.html")))
  end

  def test_parses_twenty_books_with_typed_fields
    books = BookScraper.parse_books(BookScraper.fetch(BookScraper.client, PAGE), PAGE)
    assert_equal 20, books.size
    assert_equal "A Light in the Attic", books.first[:title]
    assert_in_delta 51.77, books.first[:price]
    assert_equal 3, books.first[:rating]
  end

  def test_finds_next_page
    html = BookScraper.fetch(BookScraper.client, PAGE)
    assert_equal "https://books.toscrape.com/catalogue/page-2.html", BookScraper.next_page_url(html, PAGE)
  end

  def test_retries_503_then_succeeds
    stub_request(:get, "https://example.test/flaky").to_return({ status: 503 }, { status: 200, body: "<p>ok</p>" })
    assert_equal 200, BookScraper.client.get("https://example.test/flaky").status
  end
end

Run the tests:

bundle exec ruby test/book_scraper_test.rb

The summary line reads 3 runs, 6 assertions, 0 failures, 0 errors, 0 skips. The first two tests pin down the parser and pagination. The third stubs a 503 followed by a 200 and checks that the client returns the 200, so it fails if someone swaps the middleware order or drops retry_statuses. WebMock also blocks real network calls, so a missing stub shows up as a clear error.

Faraday and Nokogiri see only the HTML the server sends. If the page builds its content with JavaScript in the browser, that content is missing from the response. Check first, then pick the lightest fix: embedded data, then a headless browser, then a managed API.

Fetch the page and count the elements you expect. quotes.toscrape.com/js/ shows 10 quotes in a browser. Count them in the raw HTML:

require "faraday"
require "nokogiri"
require "json"
url = "https://quotes.toscrape.com/js/"
html = Faraday.get(url, nil, "User-Agent" => "BookScraper/1.0").body.force_encoding("UTF-8")
puts "div.quote in raw HTML: #{Nokogiri::HTML(html).css('div.quote').size}"
# The page ships its data in an inline script: var data = [ ... ];
json = html[/var data = (\[.*?\]);/m, 1]
quotes = JSON.parse(json).map do |q|
  { text: q["text"], author: q.dig("author", "name"), tags: q["tags"] }
end
puts "quotes from embedded JSON: #{quotes.size}"
puts JSON.pretty_generate(quotes.first)

The first line of output is div.quote in raw HTML: 0: the quotes aren't in the server's HTML. You can see the same thing in the browser. View Source shows the server's HTML, while the DevTools Elements panel shows the page after JavaScript ran.

Many JavaScript pages ship their data in an inline script and render it in the browser. Reading that data directly is lighter and sturdier than running a browser. The second half of the script above does that: the regex captures the array assigned to var data, and JSON.parse turns it into Ruby hashes.

The script then prints:

quotes from embedded JSON: 10
{
  "text": "“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”",
  "author": "Albert Einstein",
  "tags": [
    "change",
    "deep-thoughts",
    "thinking",
    "world"
  ]
}

That's all 10 quotes, with tags, and no browser. The same idea applies when a page loads its data from an API: find the JSON response in the DevTools Network tab and request that URL with Faraday.

When the data exists only after JavaScript runs, render the page in headless Chrome. Ferrum drives Chrome directly over the DevTools protocol, so there's no driver binary to install or keep in sync. It needs Chrome or Chromium on your PATH, or a path in the BROWSER_PATH environment variable.

require "ferrum"
require "nokogiri"
browser = Ferrum::Browser.new(headless: true, timeout: 30)
begin
  page = browser.create_page
  page.go_to("https://quotes.toscrape.com/js/")
  page.wait_for_selector(css: "div.quote") # Ferrum 0.19+
  doc = Nokogiri::HTML(page.body)
  quotes = doc.css("div.quote").map do |q|
    { text: q.at_css("span.text").text, author: q.at_css("small.author").text }
  end
  puts "rendered div.quote: #{quotes.size}"
  puts quotes.first
ensure
  browser.quit
end

Save it as scripts/ferrum_quotes.rb and run it with bundle exec ruby scripts/ferrum_quotes.rb. It prints rendered div.quote: 10 followed by the first quote. wait_for_selector blocks until the quotes exist, which beats a fixed sleep. page.body returns the rendered HTML, so the Nokogiri code is the same as for a static page, and ensure closes Chrome even when something fails.

Each browser instance uses real memory and CPU, so keep browser concurrency low and render only the pages that need it. The Ferrum project page documents clicks, typing, and screenshots.

selenium-webdriver works through WebDriver, so it's the choice when you need Firefox or cross-browser code, and Capybara adds a friendlier DSL for clicking and filling fields on top of it. Cuprite is a Capybara driver built on Ferrum, useful if you already write Capybara code. Kimurai wraps browsers and Nokogiri into a full framework, covered below. If you need rendering, waits, and screenshots and would rather render pages without running browsers on your own servers, a hosted browser takes that work off your hands.

Mechanize acts like a simple browser without JavaScript. It stores cookies, follows redirects, and fills and submits forms, which suits logins and multi-step flows on server-rendered sites.

The practice site has a login form at quotes.toscrape.com/login that accepts any username and password. Add gem "mechanize", "~> 2.14" to your Gemfile, run bundle install, and log in:

require "mechanize"

agent = Mechanize.new
agent.user_agent = "BookScraper/1.0 (+mailto:you@example.com)"

page = agent.get("https://quotes.toscrape.com/login")
form = page.form_with(action: "/login")
form.username = "demo"
form.password = "demo"
home = agent.submit(form)

puts "logged in: #{!home.link_with(text: 'Logout').nil?}"
puts "quotes on page: #{home.search('div.quote').size}"

The script prints logged in: true and quotes on page: 10. form_with finds the form by its action, hidden fields such as the CSRF token are submitted with it, and the session cookie stays in the agent's cookie jar for later requests. search runs Nokogiri, so the same selectors work. For logins that depend on JavaScript, use Ferrum instead. Only automate logins on sites where you have an account and permission.

Most jobs don't need one. Faraday, Nokogiri, and a thread pool handle a few sites with custom logic, and you keep full control. A framework pays off when you run many spiders and want scheduling, browser engines, and pipelines in one place.

Kimurai is the main Ruby option. Version 2.x is maintained again and requires Ruby 3.2 or newer. Before adopting it, weigh two things:

• It brings a large dependency tree, including Capybara, Selenium, and Active Support.
• Many tutorials still target Kimurai 1.x, so check that examples match the 2.x API.

Both clients work. HTTParty is the quickest way to make a one-off request and parses JSON responses into Ruby hashes for you:

require "httparty"
require "nokogiri"

response = HTTParty.get(
  "https://books.toscrape.com/",
  headers: { "User-Agent" => "BookScraper/1.0 (+mailto:you@example.com)" },
  timeout: 15
)
doc = Nokogiri::HTML(response.body)
puts "status #{response.code}, books #{doc.css('article.product_pod').size}"

That prints status 200, books 20. Faraday takes more setup, but its middleware keeps retries, timeouts, and error handling in one reusable connection. Use HTTParty for quick scripts and Faraday for scrapers that run on a schedule.

The scraper in this guide works because books.toscrape.com and quotes.toscrape.com are practice sites built for automated requests. Many production sites aren't. The same code can start failing with 403 responses, CAPTCHA pages, per-IP rate limits, browser fingerprint checks, or content that changes by country. More threads don't help, because these limits are about who's asking, not how fast.

Each DIY fix adds infrastructure to run and monitor: proxy pools, browser fleets, retry queues. Our guide on how to avoid getting blocked covers headers, pacing, and proxies. At some point it makes more sense to hand off fetching and keep your Ruby code for parsing.

Scraper API fetches the page for you, handles proxies and retries, and can render JavaScript, while your Ruby code keeps the same Faraday and Nokogiri steps. You send one GET request with your API key and the target URL, and add options such as render_js or country only when a page needs them.

Add this method to the BookScraper module, before the final end:

  # Scraper API: same Faraday, the target goes in params and gets URL-encoded for you.
  def scraper_api(url, render_js: false, country: nil)
    api = Faraday.new(url: "https://api.webscrapingapi.com") do |f|
      f.options.timeout = 120
    end
    params = { api_key: ENV.fetch("WSA_API_KEY"), url: url }
    params[:render_js] = 1 if render_js
    params[:country] = country if country
    api.get("/v2", params)
  end

Then call it from scripts/scraper_api.rb and check the status before you parse:

require_relative "../lib/book_scraper"

response = BookScraper.scraper_api("https://quotes.toscrape.com/js/", render_js: true, country: "us")

if response.status == 200
  doc = Nokogiri::HTML(response.body)
  puts "quotes via Scraper API: #{doc.css('div.quote').size}"
else
  puts "Scraper API returned #{response.status}: #{response.body[0, 200]}"
end

Set your key in the shell and run it:

export WSA_API_KEY="your_api_key"
bundle exec ruby scripts/scraper_api.rb

Why the code looks like this:

• The params hash encodes the target URL for you, so its :, /, ?, and & characters can't break the request. Pasting the URL into the query string by hand is a common bug.
• ENV.fetch fails fast. If the variable isn't set, Ruby raises KeyError: key not found: "WSA_API_KEY" before any request goes out.
• The timeout is longer, because rendering takes longer than a plain fetch.
• This connection has no retry middleware. The API already retries on its side, so client retries would only repeat the job.
• The error branch prints the body. With an invalid key, the script prints Scraper API returned 401: {"error":"Invalid API key"}.

With HTTParty, the same call passes the options as query:, which encodes them the same way. Keep the API key on the server side: don't commit it, ship it in client code, or print it in logs.

Most errors in web scraping with Ruby come from a missing gem, a retry setup that doesn't retry, an encoding mismatch, or content that isn't in the HTML. Here's each symptom and its fix.

When an error isn't in the table, print the status code and the first few hundred bytes of the response body. Most scraping bugs become obvious once you see what the server sent.

Read the site's robots.txt, which follows the Robots Exclusion Protocol (RFC 9309), and skip the paths it disallows for your user agent. Keep the worker pool small, add delays, and honor Retry-After when the server asks you to slow down. Identify your client with a User-Agent that includes a contact address. Follow the site's terms of service, and avoid collecting personal data unless you have a clear legal basis for it. For how we approach this, see our responsible web data practices.

Yes. Ruby has mature gems for every step: Faraday or HTTParty to fetch pages, Nokogiri to parse HTML with CSS selectors or XPath, Ferrum to drive headless Chrome, and Mechanize for forms. Python has a larger scraping ecosystem and more browser tooling, but Ruby is a natural choice when your app or team already runs on Ruby or Rails.

Most Ruby scrapers pair Faraday for HTTP requests with Nokogiri for parsing. Add faraday-retry for retries, the csv gem for spreadsheets, and the built-in JSON library for JSON files. Use Ferrum when a page needs a real browser, Mechanize for forms and logins, and Kimurai if you want a full crawling framework.

Both work. HTTParty is the quickest way to make a one-off request and parses JSON responses for you. Faraday takes a little more setup but builds a reusable connection with middleware, so retries, timeouts, and error handling live in one place. For a scraper that runs on a schedule, Faraday's middleware usually pays off.

No. Nokogiri parses the HTML the server sends and never runs JavaScript, so content built in the browser is missing from the response. Check the raw HTML first, then look for the data in an inline script or a JSON request the page makes. If it only exists after rendering, use Ferrum or Selenium to load the page.

Yes. Put the scraper in a plain Ruby class or service object, then run it from a background job with Active Job so web requests never wait on a crawl. Store the results with Active Record, schedule recurring runs with your job backend, and keep the HTTP and parsing code testable outside Rails with saved HTML fixtures.

Kimurai 2.x is maintained again and requires Ruby 3.2 or newer, so it is an option when you want spiders, browser engines, and pipelines in one framework. It pulls in a large dependency tree, including Capybara and Selenium. For smaller jobs, Faraday, Nokogiri, and a thread pool give you the same results with fewer moving parts.

Identify your client with a clear User-Agent, keep concurrency low, add delays between pages, and retry 429 and 503 responses while honoring Retry-After. Respect robots.txt and the site's terms. Sites with CAPTCHAs, fingerprinting, or per-IP rate limits can still block a well-behaved scraper, which is where rotating IPs or a managed scraping API helps.

You now have a tested Ruby scraper that fetches, parses, paginates, runs pages concurrently, retries politely, and exports clean data. A few next steps make it production-ready:

• Run it from a background job. In Rails, wrap the crawl in an Active Job and schedule it, so no web request waits on a scrape.
• Add a per-host rate limit, so every worker shares one request budget instead of relying on sleeps and pool size alone.
• Render only where you must. Keep Faraday for static pages and embedded JSON, and send only the pages that need JavaScript to Ferrum.
• Hand off fetching when blocks start. Once you spend more time on proxies and CAPTCHAs than on parsing, move the fetch step to a managed API and keep the rest of your Ruby code.

Point the same Faraday client at Scraper API and get the page back rendered, from the country you choose, without running browsers or proxies yourself. Start with 5,000 free requests. No extra credits for JavaScript rendering or geotargeting, and failed requests aren't billed.

See Scraper API pricing