icrawler - 内置多个图像下载引擎的多线程爬虫框架

icrawler - 内置多个图像-引擎的多线程爬虫框架

icrawler

Introduction

Documentation: http://icrawler.readthedocs.io/

Try it with pip install icrawler or conda install -c hellock icrawler.

This package is a mini framework of web crawlers. With modularization design, it is easy to use and extend. It supports media data like images and videos very well, and can also be applied to texts and other type of files. Scrapy is heavy and powerful, while icrawler is tiny and flexible.

With this package, you can write a multiple thread crawler easily by focusing on the contents you want to crawl, keeping away from troublesome problems like exception handling, thread scheduling and communication.

It also provides built-in crawlers for popular image sites like Flickr and search engines such as Google, Bing and Baidu. (Thank all the contributors and pull requests are always welcome!)

Requirements

Python 2.7+ or 3.5+ (recommended).

Examples

Using built-in crawlers is very simple. A minimal example is shown as follows.

from icrawler.builtin import GoogleImageCrawlergoogle_crawler = GoogleImageCrawler(storage={'root_dir': 'your_image_dir'})google_crawler.crawl(keyword='cat', max_num=100)

You can also configurate number of threads and apply advanced search options. (Note: compatible with 0.6.0 and later versions)

from icrawler.builtin import GoogleImageCrawlergoogle_crawler = GoogleImageCrawler( feeder_threads=1, parser_threads=2, downloader_threads=4, storage={'root_dir': 'your_image_dir'})filters = dict( size='large', color='orange', license='commercial,modify', date=((2017, 1, 1), (2017, 11, 30)))google_crawler.crawl(keyword='cat', filters=filters, max_num=1000, file_idx_offset=0)

For more advanced usage about built-in crawlers, please refer to the documentation.

Writing your own crawlers with this framework is also convenient, see the tutorials.

Architecture

A crawler consists of 3 main components (Feeder, Parser and Downloader), they are connected with each other with FIFO queues. The workflow is shown in the following figure.

url_queue stores the url of pages which may contain imagestask_queue stores the image url as well as any meta data you like, each element in the queue is a dictionary and must contain the field img_urlFeeder puts page urls to url_queueParser requests and parses the page, then extracts the image urls and puts them into task_queueDownloader gets tasks from task_queue and requests the images, then saves them in the given path.

Feeder, parser and downloader are all thread pools, so you can specify the number of threads they use.

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。