Scrapyd部署scrapy爬虫, Scrapyd部署, Scrapyd经验。

1、sudo pip install scrapyd

2、sudo pip install scrapyd-client

3、切换目录到项目所在路径：

修改scrapy.cfg设置

[deploy:xxxx]
url = http://192.168.2.239:6800
project = xxxspider

发布工程到scrapyd

scrapyd-deploy xxxx -p xxxspider --version 54(version为版本号,可以方便控制爬虫版本)

Packing version 1470208884
Deploying to project "xxxspider" in http://192.168.2.239:6800/addversion.json
Server response (200):
{"status": "ok", "project": "xxxspider", "version": "1470208884", "spiders": 18, "node_name": "ubuntu-16"}

4、验证是否发布成功

scrapyd-deploy -L xxxx

会输出类型“xxxspider”的回显

scrapyd-deploy -l

会输出类似“target http://192.168.2.239:6800”的回显

5、启动爬虫

curl http://192.168.2.239:6800/schedule.json -d project=Crawler -d spider=CommonSpider

4、实时监控爬虫
通过页面实时监控：http://192.168.2.239:6800

6、终止爬虫

curl http://192.168.2.239:6800/cancel.json -d project=Crawler -d job= 8270364f9d9811e5adbf000c29a5d5be

参考：Scrapyd文档 C/C++程序员之家，Scrapy