怎么部署

• scrapyd + supervisord + crontab + redis

可以用的一些lib

• weibo的登录可以学下
• 这个爬虫的架构可以学下，基于队列的master和jobber模式来实现分布式
• 怒赞的一个爬虫框架，基于tornado，妈的，这个得好好学习
• 去重的逻辑用下这个库来搞
• 部署、控制相关的用这个

分布式

• 基于redis的方案

参考的blog

• 这个家伙专门搞爬虫的样子
• scrapy的优劣

入门

• 入门文章

结合

• 与机器学习结合，值得参考

行业资料

• xitu

行业需求

• [lagou]

其他

• 比如如何防止被ban掉

  1 2 3 4 5 6 7

  Here are some tips to keep in mind when dealing with these kinds of sites: - rotate your user agent from a pool of well-known ones from browsers (google around to get a list of them) - disable cookies (see COOKIES_ENABLED) as some sites may use cookies to spot bot behaviour - use download delays (2 or higher). See DOWNLOAD_DELAY setting. - if possible, use Google cache to fetch pages, instead of hitting the sites directly - use a pool of rotating IPs. For example, the free Tor project or paid services like ProxyMesh - use a highly distributed downloader that circumvents bans internally, so you can just focus on parsing clean pages. One example of such downloaders is Crawlera