基于Python的crawler考察垂直爬虫的几个原则：scrapy的架构图：scrapy 包含的component如下：Frequently Asked QuestionsReferences基于Python的crawler考虑到垂直爬虫及站内搜索的重要性，重新思考一下项目爬虫的技术架构及实现方案考察垂直爬虫的几个原则：性能较高：较好支持多线程并发处理；支持异步、非阻塞socket；支持分布式爬取；爬取调度算法性能较高；内存使用效率较高，不要老是出现out of memory问题；架构优美：组件式设计式架构，扩展方便；架构设计精巧。至少值得花时间去学习架构设计思想。扩展方便：能够与现有框架较好集成；由于是垂直爬虫，需要针对不同的网页定制爬取规则集逻辑，需要能够方便测试，不要老是重新编译，因此最好支持python等脚本语言功能全面：内置支持ajax/javascript爬取、登录认证、深度爬取设置、类似heritrix的爬取过滤器（filter）、页面压缩处理等管理功能：提供爬虫管理接口，能够实时监控和管理爬取厌烦了基于java的爬虫方案，尤其是考虑到python在网络编程上的易用性，因此打算考察基于python做新版本爬虫的可行性，刚好把久不使用的python捡起来。整理了一下目前基于python的crawler，大致有如下一些现成的项目方案可供参考：Mechanize：http ...继续阅读 (45)

作者：范军 (Frank Fan）新浪微博：@frankfan7 Twitter:@franfan7看到这个标题，你一定会觉得纳闷。弱点是必须要弥补的，怎么利用呢？人无完人，我们每个人都有弱点。有的不是特别明显，有的看似致命伤。我也一直为我的众多弱点困扰和挣扎着。我们从小的教育，都是和榜样看齐，以人之长来衬托自己之短。所谓的木桶短板理论，不知误导了多少人！一直木桶的盛水量是有最短的那条木板决定的，这虽然没错。可有的情况下，与其费尽全力补那只短板，未必能达到平均水平，出类拔萃就难上加难。不如打碎木桶，抽出那条最长的木板，或为栋梁，独当一面。或成船桨，遨游四海，不亦乐乎？！你说我这是喝醉了说疯话了吧，说的容易，做起来可太难了！别急，咱慢慢聊。TED上一段著名的演讲Power of Vulnerability，我开始看了两遍都不知所然，除了一些小幽默外，没觉得怎么特别。后来一哥们跟我猛推荐，又把台词读了一遍。才觉得意味深远。The people who have a strong sense of love and belongingbelieve they're worthy of love and belonging.Brene博士在十年间采访了无数的人，那些有强烈幸福感和归属感的人们有一个共同点，那就是他们都认为自己是值得被人爱，有权得到一个美好的生活。They believed ...继续阅读 (24)

Here is the list of 43 short tips you can use for writing an optimized and more efficient PHP code:If a method can be static, declare it static. Speed improvement is by a factor of 4.echois faster thanprint.Use echo’s multiple parameters instead of string concatenation.Set the maxvalue for your for-loops before and not in the loop.Unset your variables to free memory, especially large arrays.Avoid magic like __get, __set, __autoloadrequire_once()is expensiveUse full paths in includes and requires, less time spent on resolving theOSpaths.If you need to find out the time when the script sta ...继续阅读 (21)

Install and Configure GNS3First, You should install gns3:sudo emerge --autounmask-write gns3 sudo dispatch-conf sudo emerge gns3Then you need to prepare for ios images, You can download them from:ftp://ftp.unikon-ua.net/pub/Cisco/IOS/ http://115.com/file/clsclmjbThen you should configure them:Edit>IOS Images and HypervisorsAnd calculate ‘Idle PC’。You may want to get rid of warning ofbaseconfig.txt, refer to:http://forum.gns3.net/post12717.htmlIf you are using KDE, you may use konsole’s preconfigured terminal commands.Edit>Preferences>General>Terminal SettingsYou need to install and configure v ...继续阅读 (17)

As a biologist, suppose you would like to construct a phylogeny tree on several species, what kind of "raw data" you may have? Usually, you might have DNA sequences or morphology data. As a computer scientist, these are the input to our phylogeny reconstruction algorithms, which can be represented by a matrix. For DNA sequences each row represents a species and each column stands for a particular position on the sequence, and each entry is a nucleotide. For morphology data, rows have the same meaning, columns stands for different characters and each entry stands for the state of the character ...继续阅读 (30)

In this post I am goIing to summarize some of the things that I learned at Strata Santa Clara 2013. For now, I will only discuss the conference sessions as I have a much longer post about the tutorial sessions that I am still working on and will post at a later date.I will add to this post as the conference winds down.The slides for most talks will be availableherebut not all speakers will share their slides.Thisis/was my first trip to Strata so I was eagerly awaiting participating as an attendant. In the past, I had been put off by the cost and was also concerned that the conference would be ...继续阅读 (82)

Much has been made of the lack of diversity in the technology field. While there are many causes for it and not all of the factors represent intentional or even systemic discrimination, there are from time to time disturbing incidents that make me despair. One such has just been revealed to me, and I am proud to be the one with the scoop.You read here aboutHilbert’s Grand JavaScript School. After a stunningly successful first week offering its revolutionaryLearn javaScript in Five Dayscourse, authorities in Thailand have closed it for failing to offer seating on a fair and equitable basis.Oper ...继续阅读 (12)

Much has been made of the lack of diversity in the technology field. While there are many causes for it and not all of the factors represent intentional or even systemic discrimination, there are from time to time disturbing incidents that make me despair. One such has just been revealed to me, and I am proud to be the one with the scoop.You read here aboutHilbert’s Grand JavaScript School. After a stunningly successful first week offering its revolutionaryLearn javaScript in Five Dayscourse, authorities in Thailand have closed it for failing to offer seating on a fair and equitable basis.Oper ...继续阅读 (19)

去年 4 月份的时候我在Simplenote里记下过一段话：试想一下你到了一个陌生的地方，或者一个旅游景区，或者展览。你只需要将手持移动设备（手机）在入口的地方或者一个类似电话亭的地方，近距离打开无线数据传输，停留几秒，之后所有这个区域你所需要了解的信息都会直接在你的手持设备上展示。例如，当地的地图，景点介绍，实时的展览信息等。其实这些都已经不在陌生，你到任何一个地方，通过智能手机联网，Google Maps 可以给你带来地理信息、线路，Foursquare 可以看到当地景点和生活场所的推荐。前几天看了几段 2012 Google I/O 的视频，然后对 Android 系统 Jelly Beans 感觉很棒。顺着就在 Youtube 上找测评和体验视频看。很多报道关于 Google 的 Project Glass，两个月前我也有过一篇专门介绍这个「玻璃计划」的文章。事实上看了 I/O 后我倒对那个 Glass 没增加多少兴趣，倒是特别关注这次 I/O 大会上发布的另一款伴随新 Jelly Beans 系统的产品：「Google Now」。如标题所说，我感觉这两款产品分别就是代表「现实」和「未来」。Google Now（现实）据 Google Now 官方介绍，当你在 Jelly Beans 下第一次使用 Google 搜索时，系统会弹出一个介绍有关 Google Now 的屏幕。 ...继续阅读 (41)

IntroductionThis is a short introduction to Microsoft's Surface technology.What is Surface?The best way to quickly get an impression of the actual thing would be to watchthis short video. I guess, formally, Surface is a combination of hardware that has some new input qualities and software APIs that come along with it to support them.The parody(also a recommended watch) describes it as 'a computer that is a big ass table’. For those of you who haven’t watched the movie, Surface is a table that has a computer screen on its surface. You can interact with it by touching the screen, wh ...继续阅读 (34)

旧的一年结束的时候，我们往往回收过去一年。这篇文章就是回首2011年的各种纹理设计，向大家展示全年内最为新鲜最为有用的纹理设计！希望大家稀饭！ 纹理包 18 Hand Stained and Colored Paper Textures Get them here! Crumpled Paper Textures Get them here! Bamboo Texture Pack Get them here! 15 Natural Stone Textures Get them here! 10 Exclusive High Resolution Wood Textures by TextureQualityPro Get them here! 5 Subtle Blue Grunge Textures Get them here! Autumn Leaves Textures Get them here! 8 Free Original Hi-Res Old Brown Paper Textures Get them here! 27 Free High Resolution Watercolor Textures Get them无觅猜您也喜欢：wordpress 2010 全年免费主题精选介绍几款创意免费的手机壁纸2011年40个新鲜的字体免费下载~牛嚓嚓的免费PS笔刷大放 ...继续阅读 (35)

I have been waiting for the KDD conference to come to California, and I was ecstatic to see it held in San Diego this year.AdMelddid an awesome job displaying KDD ads on the sites that I visit, sometimes multiple times per page. That’s good targeting!Mining and Learning on Graphs Workshop 2011I had originally planned to attend the 2-day workshopMining and Learning with Graphs (MLG2011)but I forgot that it started on Saturday and I arrived on Sunday. I attended part of MLG2011 but it was difficult to pay attention considering it was my first time waking up at 7am in a long time. The first ...继续阅读 (69)

I woke up early and cheery Wednesday morning to attend the2011 Hadoop Summitin Santa Clara, after a long drive from Los Angeles and theBig Data Campthat lasted until 10pm the night before. Having been toHadoop Summit 2010, I was interested to see how much of the content in the conference had changed.This year, there were approximately 1,600 participants and the summit was moved a few feet away to the Convention Center rather than the Hyatt. Still, space and seating was pretty cramped. That just goes to show how much the Hadoop field has grown in just one year.KeynotesWe first heard a series of ...继续阅读 (83)

Some time over the past 6 weeks I randomly saw a tweet announcing the “Data Scientist Summit” and shortly below it I saw that it would be held in Las Vegas at theVenetian. Being a Data Scientist myself is reason enough to not pass up this opportunity, but Vegas definitely sweetens the deal! On Wednesday I woke up at 6am to partake on the5.5 hour voyage to Las Vegas.The Pre-PartyThe Venetian and all close hotels were booked, so I ended up at theAria; a new experience. The hotel is beautiful and very ritzy. I had heard that the rooms were very technologically advanced but I wasn ...继续阅读 (189)

下文是肖庆的原创翻译，转载请注明出处，英文截取自谷歌分析的一个PDF文档《Improving Online Conversions For Dummies》，这是谷歌分析的一个入门教程，共68页，其中有对谷歌分析的一个简要概述，功能描述，基本操作，等等。Be aware of the most common website problems and ensure your site doesn’t suffer from them:注意以下最常见的网站问题并确保你的网站免受其害：✓ Not having a clear call to action. Website visitors need to be told what they need to do next. If you want visitors to buy something or sign up you need to explicitly tell them to do so.没有明确的行为号召。网站访问者需要被告知他们接下来要做什么。如果你想访问者购买或者注册，你需要明确地告诉他们这样做。✓ Too many choices. People can only process so many options at a time. Conversion rate suffers when you offer to ...继续阅读 (15)

牛年已经在眼前，祝各位朋友牛年大吉，子去丑来腾锦銹，金牛奋地涌春潮。PS:前几天给国外的同事发了祝福，收到了一个美好的回复祝福，同时也加了一句，what is ox?wiki了一下，发了给他，也留下来自己记录。what is ox year?来源：http://en.wikipedia.org/wiki/Ox_(zodiac)TheOx(牛) is one of the 12-year cycle of animals which appear in theChinese zodiacrelated to theChinese calendar. TheYear of the Oxis denoted by theearthly branchcharacter丑.The Ox is the sign of prosperity through fortitude and hard work. This powerful sign is a born leader, being quite dependable and possessing an innate ability to achieve great things. As one might guess, such people are dependable, calm, and modest. Like their ani ...继续阅读 (16)

A poem some times will be falshing,which we used in a drama called "Go left, leave right". its name is love at first sight. I really love it.Love at First Sight------Wislawa SzymborskaBoth are convincedthat a sudden surge of emotion bound them together.Beautiful is such a certainty,but uncertainty is more beautiful.Because they didn't know each other earlier, they suppose thatnothing was happening between them.What of the streets, stairways and corridorswhere they could have passed each other long ago?I'd like to ask themwhether they remember-- perhaps in a revolving doorever being face to fac ...继续阅读 (24)

I am writing to you as a fellow programmer and software developer. I write in friendship and brotherhood. My heart is heavy, and the news I impart is not good: We have lost control of the apparatus.I know, the IT department soldiers on grimly. They lost a great battle to the PCs more than twenty years ago, and it took years of struggle with NT Domain Controllers and web proxies before control could be wrestled back from the users. In some places I hear battles are still being waged over USB ports and Bluetooth. IT was lucky to find a new ally against the users in MSFT just when their ancient s ...继续阅读

真假难辨的机器小强by chinarobot法国、比利时和瑞士的科学家们已经成功研制成功了一种机器蟑螂，可以让真的蟑螂不能辨别真假，把它当同类来对待。这个机器人叫做InsBot。科学家们相信，在不久的将来，为类可以使用这种机器人渗透进蟑螂的群体中去，警告小强们注意自己的行为。最初，图卢兹[法国南部城市]的一个动物认知研究中心着手研究蟑螂的行为。一个学生花了三年的时间观察了蟑螂行动，并将其拍摄下来，并用电脑建模，重现蟑螂的运动。研究发现，蟑螂同蚂蚁一样，是一种彼此间平等的昆虫，他们的群体没有一个领导。他们因为一种群体的智慧而聚集在一起。"蟑螂们希望与同类接触，当它们碰面时，它们会很安静，并且非常乐意交很多的交小强朋友。朋友越多，它们待的时间也会越长。"该计划的Jean-Louis Deneubourg.说。然后，一个叫做Leurre的计划开始实施，这个计划要花费大约３３０万美元。Leurre的目的是制造一个机器人，要求它能够检测到蟑螂，并能像蟑螂一样运动。他们成功制造出来了，这个机器人就是InsBot。这是一个绿色的，和火柴盒般大小，装备有位于洛桑的瑞士联邦工艺学校研制的激光传感器。当InsBot碰到一只蟑螂时，它能和真的蟑螂一般运动。它会安静下来，蟑螂越多，它了就能安静的待的时间就越长。最后，由布列塔尼(半岛)[法国西北部一地区]的一个科学研究中心研究出一种类似蟑螂体味的物质，并将 ...继续阅读 (11)

Them, robotsby Prakash ChandraIt's official now: this planet is about to be invaded by robots. The UN's annual World Robotics Survey says the use of robots to do domestic work will surge seven-fold by 2007. That means over four million robots doing chores that have been the exclusive preserve of human beings: from mowing lawns and cleaning floors to babysitting and pulling guard duty. They may only be automated vacuum-cleaners that drive this domestic helper boom, but they'll boost orders for industrial robots and ‘leisure' robots like Sony's all-dancing, voice-recognising, dog-like Aibo. High ...继续阅读 (51)

What will your content look like? See the styles below!HeadingsStart a section with one of these beauties. Use##to quickly create a second-level heading (h2).h1is reserved for the title.Heading 2Heading 3Heading 4Heading 5Heading 6TextHere's a sample paragraph fromThe War of the Worldsby H. G. Wells.Read the book. Smart uses Work Sans for body text and Archivo for headings, all loaded locally.No one would have believed in the last years of the nineteenth century that this world was being watched keenly and closely by intelligences greater than man’s and yet as mortal as his own; that as men bu ...继续阅读 (34)