A highly flexible and efficient Laravel 5.x scraper package. Easily scrape data from anywhere on the web and build content.
Scavenger provides the following features and more out of the box.
- Ease of use
- Scavenger is super-easy to configure. Simple publish the config file and set your targets.
- Scrape data from multiple sources at once.
- Convert scraped data into useable Laravel model objects.
- eg. You may scrape an article and have it converted into an object of your choice and saved in your database. Immediately available to your viewers.
- You can easily perform one or more operations to each property of any scraped entity.
- eg. You may call a paraphrase service from a model or package of your choice on data attributes before saving them to your database.
- Data integrity constraints
- Scavanger uses a hashing algorithm of your choice to maintain data integrity. This hash is used to ensure that one scrap (source article) is not converted to multiple output objects (model duplicates).
- Console Command
- Once scavenger is configured, a simple artisan command launches the seeker. Since this is a console command it is more efficient and timeouts are less likely to occur.
- Artisan command:
php artisan scavenger:seek
- Schedule ready
- Scavenger can easily be set to scrape on a schedule. Hence, creating a someone autonomous website is super easy!
- Scavenger can be used to flexibly scrape Search Engine Result Pages.
Full documentation can be found at scavenger.reliqarts.com.
v2.2.3 (Feb. 28, 2019) [delayed] - Raw HTML is now scraped instead of text. This allows more flexibility and usability. - Stability and performance updates
v2.1.0 (Jan. 14, 2019) - Improved scraping algorithm - Stability and performance updates
v2.0.0 (Feb. 21, 2018) - Introduces SERP support.
v1.0.6 (Jan. 13, 2018) - Stability update.
v1.0.5 (Sept. 2, 2017) - Doc update.
v1.0.4 (Aug. 23, 2017) - Stability update.
v1.0.3 (Aug. 22, 2017) - Initial release.