Webster是一个用Node.js编写的可靠的网页抓取爬虫框架

Webster是一个用Node.js编写的可靠的网页抓取爬虫框架

Webster

Overview

Webster is a reliable web crawling and scraping framework written with Node.js, used to crawl websites and extract structured data from their pages. Which is different from other crawling framework is that Webster can scrape the content which rendered by browser client side javascript and ajax request.

Docker quick start

pull the example docker image:

docker pull zhuyingda/webster-demodocker run -it zhuyingda/webster-demo

here is a simple demo for crawler about Baidu search result web page:

node demo_producer.js && node demo_consumer.js

Requirements

Node.js 8.x+, redisWorks on Linux, Mac OSX

Or you can deploy on Docker.

Install

npm install webster

Architecture overview

Documentation

You can see more details from here.

Contributors

Code Contributors

Financial Contributors

Become a financial contributor and help us sustain our community. [Contribute]

Individuals

Organizations

Support this project with your organization. Your logo will show up here with a link to your website. [Contribute]

License

GPL-V3

Copyright (c) 2017-present, Yingda (Sugar) Zhu

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。