WebCrawler
in package
Tools for crawling/scraping web pages
This is really just to get you started.
To work with the crawler, you can reference the following tools:
- https://symfony.com/doc/current/components/dom_crawler.html
- https://symfony.com/doc/current/components/browser_kit.html
- https://symfony.com/doc/current/components/css_selector.html
- https://symfony.com/doc/current/http_client.html
Extracting data from specific elements:
$crawler->filter('ul > li')->each(function ($node) { print $node->text()."\n"; });
Click a link
$link = $crawler->selectLink('Make a booking')->link(); $crawler = $client->click($link);
Submit a form:
$crawler = WebCrawler::crawlUrl('https://github.com/'); $crawler = $client->click($crawler->selectLink('Sign in')->link()); $form = $crawler->selectButton('Sign in')->form(); $crawler = $client->submit($form, ['email' => 'test@karmabunny.com.au', 'password' => 'xxxxxx']); $crawler->filter('.flash-error')->each(function ($node) { print $node->text()."\n"; });
Table of Contents
Methods
- crawlUrl() : Crawler
- Crawl a URL, return the crawler object. Use $crawler->html() to get the raw HTML.
- stripTags() : void
- Remove any instances of specified tags from the crawler result.
Methods
crawlUrl()
Crawl a URL, return the crawler object. Use $crawler->html() to get the raw HTML.
public
static crawlUrl(string $url[, string $method = 'GET' ][, HttpClientInterface|null $client = null ]) : Crawler
Parameters
- $url : string
-
The URL to crawl
- $method : string = 'GET'
-
The HTTP method to use (GET, POST, etc)
- $client : HttpClientInterface|null = null
-
An existing client to use, or null to create a new one
Return values
CrawlerstripTags()
Remove any instances of specified tags from the crawler result.
public
static stripTags(Crawler $crawler, array<string|int, mixed> $tags) : void
Parameters
- $crawler : Crawler
-
The crawler object
- $tags : array<string|int, mixed>
-
An array of tags to remove (e.g. head, script, etc)
Return values
void —as Crawler is passed by reference