Web Scraping with PHP: Libraries, Code & Best Practices (2026)

PHP might not be the first language that comes to mind when people think about scraping, but it has several practical advantages. First, if your existing stack already runs on PHP, adding a scraper means zero new runtime dependencies. Your team can maintain the code, your deployment pipeline stays the same, and you avoid the cognitive overhead of context-switching to another language.

Second, PHP's built-in extensions are surprisingly well suited to this task. The curl extension handles HTTP requests, dom and libxml give you a standards-compliant HTML/XML parser, and mbstring takes care of character-encoding headaches. You do not need to install anything extra for a basic scrape.

Third, the Composer ecosystem fills every remaining gap. Guzzle provides a modern HTTP client with middleware support. Symfony DomCrawler adds CSS selector queries on top of DOMDocument. Symfony Panther drives a real Chrome or Firefox instance for JavaScript-heavy pages. The tooling is mature and actively maintained.

What about PHP vs Python for scraping? Python has a larger scraping-specific community and libraries like Beautiful Soup and Scrapy, but that does not make PHP a poor choice. If PHP is your strongest language, you will write a working scraper faster than you would in a language you are still learning. The best scraping language is the one you can debug at 2 AM.

Before writing code, it helps to know which tools exist and when to reach for each one. The table below compares the major PHP scraping libraries across the criteria that matter most: what they do, whether they handle JavaScript, and how much effort they take to learn.

A few things to note. Goutte was the go-to "all-in-one" scraping library for years, but it has been deprecated. At the time of writing, the recommended migration path is Symfony HttpBrowser, which provides an almost identical API backed by Symfony's BrowserKit and HttpClient components. If you are starting a new project, skip Goutte entirely and go straight to HttpBrowser.

For most static-page scraping tasks, Guzzle (for fetching) plus Symfony DomCrawler (for parsing) is a solid, lightweight combination. Reserve Symfony Panther for pages that genuinely require JavaScript execution, because spinning up a headless browser is significantly slower and more resource-intensive.

Let's get the prerequisites out of the way. You need PHP 8.1 or newer (for enum and fiber support in modern libraries), Composer, and a handful of extensions.

Check your PHP version and loaded extensions:

php -v
php -m | grep -E 'curl|dom|mbstring|json'

If any of those four extensions are missing, enable them in your php.ini or install them via your system package manager (for example, sudo apt install php-curl php-xml php-mbstring on Debian/Ubuntu).

Next, initialize a project directory and pull in the libraries you will use throughout this tutorial:

mkdir php-scraper && cd php-scraper
composer init --no-interaction
composer require guzzlehttp/guzzle symfony/dom-crawler symfony/css-selector symfony/browser-kit symfony/http-client

That single composer require line gives you Guzzle for HTTP, DomCrawler for parsing, and Symfony HttpBrowser for the combined crawling workflow. We will add Symfony Panther later when we need headless browser support.

Create a scrape.php file and add the Composer autoloader at the top:

<?php
require __DIR__ . '/vendor/autoload.php';

You are ready to fetch your first page.

PHP's cURL extension is the lowest-level HTTP tool in your toolbox. It is verbose, but it gives you full control over every request detail, which is useful when you need to mimic a specific browser fingerprint or debug connection issues.

Here is a basic GET request that fetches the front page of a public book catalog (we will use http://books.toscrape.com as our demo target throughout):

$ch = curl_init();
curl_setopt_array($ch, [
    CURLOPT_URL            => 'http://books.toscrape.com/',
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_FOLLOWLOCATION => true,
    CURLOPT_HTTPHEADER     => [
        'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
        'Accept-Language: en-US,en;q=0.9',
    ],
    CURLOPT_TIMEOUT        => 30,
    CURLOPT_COOKIEJAR      => '/tmp/cookies.txt',
    CURLOPT_COOKIEFILE     => '/tmp/cookies.txt',
]);

$html = curl_exec($ch);

if (curl_errno($ch)) {
    echo 'cURL error: ' . curl_error($ch);
}

curl_close($ch);

A few things worth noting. CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE enable cookie persistence across requests, which is essential for multi-step scraping flows where the server tracks session state. Setting a realistic User-Agent header makes your request look like ordinary browser traffic rather than a bare PHP script. And CURLOPT_FOLLOWLOCATION handles 301/302 redirects automatically so you don't have to chase them manually.

For a POST request (for example, submitting a search form), swap in CURLOPT_POST => true and add CURLOPT_POSTFIELDS with your form data. The rest of the boilerplate stays the same.

cURL works, but it is low-level enough that you'll end up writing wrappers for headers, retries, and error handling. That is where Guzzle comes in.

Guzzle wraps PHP's cURL (or stream) layer in a clean, object-oriented API. Install it via Composer if you haven't already, then fetch the same page:

use GuzzleHttp\Client;

$client = new Client([
    'timeout' => 30,
    'headers' => [
        'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)',
        'Accept-Language' => 'en-US,en;q=0.9',
    ],
]);

$response = $client->get('http://books.toscrape.com/');
$html = (string) $response->getBody();

That is noticeably less boilerplate. Guzzle also gives you middleware hooks for logging, retry logic, and header injection, which means you can centralize cross-cutting concerns instead of scattering curl_setopt calls everywhere.

When you need to scrape multiple pages, firing requests one at a time is painfully slow. Guzzle supports promise-based concurrency through its Pool class, which lets you send multiple requests in parallel while controlling the level of concurrency.

use GuzzleHttp\Client;
use GuzzleHttp\Pool;
use GuzzleHttp\Psr7\Request;

$client = new Client(['timeout' => 30]);

$urls = [
    'http://books.toscrape.com/catalogue/page-1.html',
    'http://books.toscrape.com/catalogue/page-2.html',
    'http://books.toscrape.com/catalogue/page-3.html',
];

$requests = function () use ($urls) {
    foreach ($urls as $url) {
        yield new Request('GET', $url);
    }
};

$pool = new Pool($client, $requests(), [
    'concurrency' => 5,
    'fulfilled'   => function ($response, $index) {
        echo "Page $index fetched: " . $response->getStatusCode() . "\n";
    },
    'rejected'    => function ($reason, $index) {
        echo "Page $index failed: " . $reason->getMessage() . "\n";
    },
]);

$pool->promise()->wait();

With a concurrency level of 5, Guzzle fires up to five requests simultaneously instead of waiting for each one to complete. On a 50-page scrape, this can reduce total runtime from minutes to seconds. According to the Guzzle documentation on concurrent requests, the Pool API uses cURL's multi-handle under the hood, so the performance gain is real, not just syntactic sugar.

Once you have raw HTML in a string, you need to pull structured data out of it. PHP's built-in DOMDocument class loads HTML into a tree structure, and DOMXPath lets you query that tree with XPath expressions.

libxml_use_internal_errors(true); // suppress malformed-HTML warnings

$doc = new DOMDocument();
$doc->loadHTML($html);

$xpath = new DOMXPath($doc);

// Select every book title on the page
$titles = $xpath->query('//article[@class="product_pod"]//h3/a/@title');

foreach ($titles as $node) {
    echo $node->nodeValue . "\n";
}

The libxml_use_internal_errors(true) call is important. Real-world HTML is almost never valid XML, and without that flag, PHP will throw warnings for every unclosed tag or mismatched attribute. Suppressing them lets you parse messy pages without flooding your logs.

XPath is powerful for complex queries. Want to grab every book priced under £20? You can combine axes and predicates:

$products = $xpath->query('//article[@class="product_pod"]');

foreach ($products as $product) {
    $title = $xpath->query('.//h3/a/@title', $product)->item(0)->nodeValue;
    $price = $xpath->query('.//p[@class="price_color"]', $product)->item(0)->textContent;

    $numericPrice = (float) str_replace('£', '', $price);
    if ($numericPrice < 20.00) {
        echo "$title: $price\n";
    }
}

DOMDocument plus XPath gives you full control and zero external dependencies. The tradeoff is verbosity: even a simple query requires several lines of setup. That is where Symfony DomCrawler earns its keep.

Symfony DomCrawler sits on top of DOMDocument but exposes a much friendlier API. Instead of writing XPath by hand, you can use CSS selectors (which most web developers already know) and chain methods in a jQuery-like style.

use Symfony\Component\DomCrawler\Crawler;

$crawler = new Crawler($html);

$crawler->filter('article.product_pod')->each(function (Crawler $node) {
    $title = $node->filter('h3 a')->attr('title');
    $price = $node->filter('.price_color')->text();
    echo "$title: $price\n";
});

Compare that to the DOMXPath version above. The intent is identical, but the DomCrawler code is half as long and easier to read. The filter() method accepts any valid CSS selector, text() returns the text content, and attr() pulls an attribute value.

When should you use CSS selectors vs XPath for scraping? CSS selectors cover 90% of practical cases and are more intuitive for anyone who writes front-end code. XPath wins when you need to traverse upward (select a parent based on a child's text), perform string functions inside the query, or navigate sibling axes. A good rule of thumb: start with CSS selectors and drop down to XPath only when CSS can't express what you need.

It is tempting to reach for preg_match() when you just need one value from a page. Resist the urge. HTML is not a regular language, and regex-based extraction breaks the moment the markup changes in trivial ways: a new attribute, a switched quote style, or extra whitespace.

// Fragile — breaks if class order changes or attributes are added
preg_match('/<h3 class="title">(.+?)<\/h3>/', $html, $match);

A DOM parser handles all of those variations gracefully. Save regex for genuinely flat text (log files, CSV rows) and use DOMDocument or DomCrawler for anything that came out of an HTML document.

Goutte was the library that made PHP web scraping feel approachable. It combined Guzzle's HTTP client with Symfony's DomCrawler into a single class, letting you fetch and parse in one call. However, Goutte has been officially deprecated. Its maintainers recommend migrating to Symfony HttpBrowser, which ships as part of the Symfony BrowserKit component and offers an almost identical API.

Here is a complete scraper built with Symfony HttpBrowser that fetches book listings across multiple pages:

use Symfony\Component\HttpClient\HttpClient;
use Symfony\Component\BrowserKit\HttpBrowser;

$browser = new HttpBrowser(HttpClient::create([
    'headers' => [
        'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)',
    ],
]));

$books = [];
$url = 'http://books.toscrape.com/catalogue/page-1.html';

while ($url) {
    $crawler = $browser->request('GET', $url);

    $crawler->filter('article.product_pod')->each(function ($node) use (&$books) {
        $books[] = [
            'title' => $node->filter('h3 a')->attr('title'),
            'price' => $node->filter('.price_color')->text(),
            'stock' => trim($node->filter('.availability')->text()),
        ];
    });

    // Follow the "next" pagination link, or stop
    $nextLink = $crawler->filter('li.next a');
    $url = $nextLink->count() > 0
        ? 'http://books.toscrape.com/catalogue/' . $nextLink->attr('href')
        : null;
}

echo count($books) . " books collected.\n";

Notice how the pagination logic works. After parsing each page, the scraper checks whether a "next" link exists. If it does, the scraper follows it and repeats the process. If not, $url is set to null and the loop terminates. This pattern is reusable for any paginated listing.

The migration from Goutte is minimal. If your existing code uses $goutte = new \Goutte\Client(), replace it with $browser = new HttpBrowser(HttpClient::create()). The request(), filter(), and selectLink() methods remain the same. The underlying HTTP layer switches from Guzzle to Symfony HttpClient, which gives you native async support and better integration with the rest of the Symfony ecosystem.

One more advantage of HttpBrowser: it automatically tracks cookies and sessions across requests. When you call $browser->request() multiple times, the client behaves like a real browser session, carrying cookies forward without extra configuration.

Static-page scrapers break down when the content you need is injected by JavaScript after the initial page load. Single-page applications, infinite-scroll feeds, and lazy-loaded product grids all require a real browser engine to render. Symfony Panther fills that gap by driving Chrome or Firefox via the WebDriver protocol.

Install Panther and a ChromeDriver binary:

composer require symfony/panther
# Panther can auto-detect a locally installed ChromeDriver,
# or you can install one explicitly:
composer require dbrekelmans/bdi
vendor/bin/bdi detect drivers

Now scrape a page that relies on dynamic content rendering with PHP:

use Symfony\Component\Panther\Client as PantherClient;

$panther = PantherClient::createChromeClient();
$crawler = $panther->request('GET', 'https://example.com/dynamic-page');

// Wait until the data container is visible in the DOM
$panther->waitFor('.results-container', 10);

$crawler->filter('.results-container .item')->each(function ($node) {
    echo $node->filter('.item-title')->text() . "\n";
});

$panther->quit();

The waitFor() method pauses execution until the specified CSS selector appears in the rendered DOM, with a timeout (10 seconds here) to prevent infinite hangs. This is essential for dynamic content scraping with PHP because the HTML you need may not exist in the initial response at all.

Panther is powerful but expensive. Each request launches a real browser process, consuming memory and CPU. Use it only when JavaScript rendering is genuinely required. For pages that load data via a simple XHR/API call, it is often faster to find that API endpoint in your browser's Network tab and hit it directly with Guzzle.

At some point, the engineering cost of maintaining your own scraper (proxy rotation, CAPTCHA solving, browser fingerprinting, retry logic) exceeds the cost of outsourcing that infrastructure to a dedicated service. That is the sweet spot for a scraping API.

The integration pattern is simple. You send a URL to the API endpoint, and it returns the page's HTML (or structured JSON) with all the anti-bot handling done server-side:

$client = new \GuzzleHttp\Client();

$response = $client->get('https://api.webscrapingapi.com/v1', [
    'query' => [
        'api_key' => 'YOUR_API_KEY',
        'url'     => 'http://books.toscrape.com/',
    ],
]);

$html = (string) $response->getBody();
// Parse $html with DomCrawler as usual

When does a scraping API make sense over a DIY approach? Consider it when you are scraping at scale (thousands of pages per day), targeting sites with aggressive anti-bot defenses, or when your team does not have time to maintain proxy pools and browser infrastructure. The tradeoff is cost per request versus engineering hours.

A managed service also shines in maintenance burden. When a target site changes its anti-bot stack, a scraping API provider updates their infrastructure. Your code stays the same. If you are evaluating options, look for a provider that charges only for successful responses so you are not paying for failed requests.

Collecting data is only half the job. You need to persist it in a format that downstream processes (analytics, ML pipelines, dashboards) can consume.

CSV is the simplest option and works well for flat, tabular data:

$fp = fopen('books.csv', 'w');
fputcsv($fp, ['Title', 'Price', 'Stock']); // header row

foreach ($books as $book) {
    fputcsv($fp, [$book['title'], $book['price'], $book['stock']]);
}

fclose($fp);

JSON preserves nested structures and is easier to import into APIs and NoSQL stores:

file_put_contents(
    'books.json',
    json_encode($books, JSON_PRETTY_PRINT | JSON_UNESCAPED_UNICODE)
);

MySQL via PDO is the right choice when you need queryable, relational storage:

$pdo = new PDO('mysql:host=127.0.0.1;dbname=scraper', 'user', 'pass', [
    PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
]);

$stmt = $pdo->prepare(
    'INSERT INTO books (title, price, stock) VALUES (:title, :price, :stock)'
);

foreach ($books as $book) {
    $stmt->execute([
        ':title' => $book['title'],
        ':price' => $book['price'],
        ':stock' => $book['stock'],
    ]);
}

Using prepared statements with PDO is not optional. It protects you from SQL injection, which is a real risk when inserting user-generated or externally scraped text into a database.

For document-oriented data or schemas that change frequently, MongoDB is another viable option. The mongodb/mongodb Composer package provides a straightforward insertMany() method that accepts arrays of associative arrays directly. The choice between relational and document storage depends on how structured your scraped data is and what will consume it.

A scraper that works on your laptop is not the same as a scraper that runs reliably in production. Network timeouts, 5xx responses, connection resets, and rate-limit errors are inevitable when you make thousands of HTTP requests. Building resilience into your scraper from the start saves you from silent data loss.

Wrap every HTTP call in a try-catch with exponential back-off:

function fetchWithRetry(\GuzzleHttp\Client $client, string $url, int $maxRetries = 3): string
{
    for ($attempt = 1; $attempt <= $maxRetries; $attempt++) {
        try {
            $response = $client->get($url);
            return (string) $response->getBody();
        } catch (\GuzzleHttp\Exception\GuzzleException $e) {
            if ($attempt === $maxRetries) {
                throw $e;
            }
            $wait = (int) pow(2, $attempt); // 2s, 4s, 8s
            sleep($wait);
        }
    }
}

For structured logging, Monolog is the de facto standard in the PHP ecosystem. Adding a rotating file handler takes two lines:

use Monolog\Logger;
use Monolog\Handler\RotatingFileHandler;

$log = new Logger('scraper');
$log->pushHandler(new RotatingFileHandler('logs/scraper.log', 7, Logger::INFO));

$log->info('Fetching page', ['url' => $url]);
$log->error('Request failed', ['url' => $url, 'error' => $e->getMessage()]);

Log every request URL, status code, and any exceptions. When a scrape job fails at page 847 out of 1,000, logs are the only thing that will tell you what went wrong. This kind of production-readiness focus is what separates a prototype from a reliable pipeline.

Websites do not appreciate bots hammering their servers. If your scraper sends hundreds of identical requests per minute from a single IP, expect to get blocked. Polite scraping is both an ethical obligation and a practical necessity for long-running projects.

Rotate User-Agent strings so each request does not fingerprint as the same client:

$userAgents = [
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 13_5) AppleWebKit/605.1.15',
    'Mozilla/5.0 (X11; Linux x86_64; rv:115.0) Gecko/20100101 Firefox/115.0',
];

$headers = ['User-Agent' => $userAgents[array_rand($userAgents)]];

Add random delays between requests to avoid predictable timing patterns:

function politeDelay(int $minMs = 1000, int $maxMs = 3000): void
{
    usleep(random_int($minMs, $maxMs) * 1000);
}

Respect robots.txt programmatically. Before scraping a domain, fetch its robots.txt and check whether your target path is disallowed. You can parse this manually or use a library like spatie/robots-txt:

// Pseudocode — check before scraping
$robots = file_get_contents('http://example.com/robots.txt');
if (str_contains($robots, 'Disallow: /private/')) {
    echo "Skipping disallowed path.\n";
}

Proxy rotation is the most effective defense against IP-based blocking. If you are scraping at any meaningful volume, routing requests through a pool of residential proxies makes your traffic virtually indistinguishable from organic users. You can configure Guzzle to use a proxy with a single option:

$client = new \GuzzleHttp\Client([
    'proxy' => 'http://user:pass@proxy-host:port',
]);

Combining all of these techniques (varied headers, polite delays, robots.txt respect, and proxy rotation) gives you the best chance of scraping reliably without getting flagged.

Web scraping occupies a legal gray area that varies by jurisdiction. A few principles apply broadly.

Robots.txt is a voluntary standard, not a legal contract, but ignoring it weakens any good-faith argument you might make if challenged. Treat it as a baseline you always respect.

Terms of Service on the target site may explicitly prohibit automated access. Violating ToS can expose you to breach-of-contract claims, particularly in the United States after cases like hiQ Labs v. LinkedIn, which clarified that scraping publicly accessible data is not necessarily a violation of the Computer Fraud and Abuse Act, but did not address ToS enforcement.

GDPR is relevant if you scrape personal data belonging to EU residents (names, email addresses, profile details). Under GDPR, web scraping can constitute data processing, which means you need a lawful basis (typically legitimate interest) and must handle that data according to GDPR requirements: purpose limitation, storage minimization, and honoring data-subject access requests. When in doubt, consult a legal professional, especially if your scraping targets user-generated content.

The ethical floor is straightforward: do not scrape at a rate that degrades the target site's performance, do not collect data you have no legitimate use for, and be transparent about your intentions when possible.

• Pick the right tool for the page type. Use Guzzle plus DomCrawler for static HTML, Symfony Panther for JavaScript-rendered content, and a scraping API when anti-bot infrastructure outpaces your DIY setup.
• Goutte is deprecated. Start new projects with Symfony HttpBrowser, which provides the same crawling workflow backed by actively maintained Symfony components.
• Build resilience from day one. Exponential-backoff retries, structured logging, and input validation are not optional in production scrapers.
• Store data in the format your downstream consumers need. CSV for quick analysis, JSON for APIs and document stores, MySQL/PDO for relational queries.
• Scrape politely and legally. Rotate headers and proxies, respect robots.txt, add delays between requests, and understand the GDPR implications of collecting personal data.

Neither is objectively superior. Python has a larger scraping ecosystem (Beautiful Soup, Scrapy, Selenium bindings), which means more tutorials and community answers. PHP has strong HTTP and DOM extensions built in, and Composer libraries like Guzzle and DomCrawler are production-grade. Choose the language your team knows best. A well-written PHP scraper will outperform a poorly maintained Python one every time.

Yes, but you need a headless browser. Symfony Panther controls Chrome or Firefox via the WebDriver protocol and can render fully dynamic pages. For simpler cases where the page fetches data from an XHR endpoint, you can skip the browser entirely and call that API endpoint directly with an HTTP client, which is faster and uses fewer resources.

Legality depends on jurisdiction, the target site's terms of service, and the type of data collected. Scraping publicly accessible, non-personal data is generally permissible in many jurisdictions. GDPR applies when you process personal data of EU residents, requiring a lawful basis such as legitimate interest. Always review the target site's ToS and consult legal counsel before scraping personal data at scale.

Combine several techniques: rotate User-Agent strings, add random delays between requests (1 to 3 seconds is a reasonable range), respect robots.txt directives, and route traffic through a pool of rotating proxies. Avoid sending bursts of requests from a single IP. If you are scraping at high volume, a managed proxy or scraping API service handles rotation and anti-detection for you.

Submit credentials via a POST request (or through a form submission with Symfony HttpBrowser) and maintain the resulting session cookie across subsequent requests. With HttpBrowser, session cookies persist automatically. With raw cURL, set CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE to the same path. Always check that your login did not trigger a CAPTCHA or two-factor challenge, and be aware that scraping behind a login may have stricter legal implications under the site's terms of service.

Web scraping with PHP is a practical, well-supported workflow once you know which libraries to reach for. Start with cURL or Guzzle for fetching, layer on DomCrawler or DOMXPath for parsing, and escalate to Symfony Panther only when JavaScript rendering is unavoidable. Persist your data in the format your consumers expect, wrap everything in retry logic and logging, and always scrape politely.

The examples in this tutorial covered the full lifecycle: from a raw HTTP request through pagination handling, concurrent fetching, data storage, and anti-block strategies. Every technique maps to a real production concern, not just a toy demo.

If you find yourself spending more time battling anti-bot defenses than writing parsing logic, it may be worth offloading the request infrastructure to a service like WebScrapingAPI's Scraper API, which handles proxy rotation, CAPTCHAs, and retries so you can focus on the data extraction code that actually matters.