Thinking about all of the things in the statistical world that we can estimate, the one that has always perplexed me is estimating the size of an unknown population \(N\). Usually when we compute estimates based on samples, we involve the size of the sample \(n\) somewhere, thus we take “size” for granted — the size of a sample is known. We also make inferences based on sample statistics using theory such as the Central Limit Theorem, but seem to never care about the population size \(N\), we either know it, or assume it is infinite. But, in fields lik ...继续阅读 (23)

The first few hundred registrations received a mug.As a machine learning practitioner in the Los Angeles area, I was ecstatic to learn thatNIPS 2017would be inLong Beachthis year. The conference sold out in a day or two. The conference was held at theLong Beach Convention Center (and Performing Arts Center), very close to theAquarium of the Pacificand about a mile from theQueen Mary. The venue itself was beautiful, and probably the nicest place I’ve ever attended a conference. It’s also the most expensive place I’ve ever had a conference. $5 for a bottle of Coke? $11 for two ...继续阅读 (21)

NOTE 1:This is part 1 in a series that will probably contain 3 or 4 parts. Then I will return to the usual data science etc. posts.NOTE 2:This post was intentionally delayed until I received final approval on the submission of my final dissertation.On March 14, I passed my final oral defense for my Ph.D. in Statistics.It was the moment I had been waiting for. The moment I dreaded for so many years and the moment that I thought would never come. I had very few days and times to choose from. It came down to the 13th (bad luck?) and the 14th (Pi Day). My defense began at 10am and was supposed to ...继续阅读 (30)

I’ve never been very big on New Year’s resolutions. I’ve tried them in the past, and while they are nice to think about, they are always overly vague, difficult to accomplish in a year, trite, or just don’t get done (or attempted). This year I decided to try something different instead of justnotmaking resolutions at all. I set out some professional goals for myself as a Data Scientist. So without further ado…1.Don’t Complain about It, Fix It:Contribute to Open Source Software (More)Open source software is only as good as its community and/or developer(s). D ...继续阅读 (36)

This past three years has really flown. It’s time for me to finally get back to my roots and also start blogging more, like I did previously.My last post was about Strata 2013. During this time period, I was taking a break from working full-time to finish a Ph.D. dissertation that I had neglected during my previous two positions. I learned my lesson the hard way,never work externally if you want a Ph.D. in a reasonable amount of time!I quickly got my dissertation from an intro to the first 65 pages or so during this gap. I then received an offer from Facebook. I was ready to move to Sili ...继续阅读 (40)

In this post I am goIing to summarize some of the things that I learned at Strata Santa Clara 2013. For now, I will only discuss the conference sessions as I have a much longer post about the tutorial sessions that I am still working on and will post at a later date.I will add to this post as the conference winds down.The slides for most talks will be availableherebut not all speakers will share their slides.Thisis/was my first trip to Strata so I was eagerly awaiting participating as an attendant. In the past, I had been put off by the cost and was also concerned that the conference would be ...继续阅读 (80)

Wishing you all a very Merry Christmas, Happy Holidays and Happy New Year!An update on me. In October, I began working atRiot Games, the developers ofLeague of Legends. It has been an amazing experience and has occupied the majority of my free time as has my dissertation. My New Year’s resolution this year is to dust the cobwebs off this blog!Have a safe holiday season!Here in California, I will be havingChristmas in the Sand ...继续阅读 (50)

Last week I received twoRaspberry Pis in the mail fromAdaFruitand just now have some time to play with them. The Raspberry Pi is a minimal computer system that is about the size of a credit card. In the embedded systems community, the excitement is for obvious reasons, but I strongly believe that such a device can help collect and use data to help us make better decisions because not only is it a computer, but it is small and portable.For development, Raspberry Pi can connect to a television (or other display) via HDMI or composite video (the “yellow” plug for those still stuck in ...继续阅读 (38)

During the past few decades that I have been in graduate school (no, not literally) I have boycotted JSM on the notion that “I am not a statistician.” Ok, I am a renegade statistician, a statistician by training.JSM 2012was held in San Diego, CA, one of the best places to spend a week during the summer. This time, I had no excuse not to go, and I figured that in order to get my Ph.D. in Statistics, I have to have been toat least oneJSM. The conference itself was 5 days, but I did not think I could hang with statisticians for five days, and more importantly, taking three days off th ...继续阅读 (42)

OpenPaths is a service that allows users with mobile phones to transmit and store their location. It is an initiative by the New York Times that allows users to use their own data, or tocontribute their location data for research projectsand perhaps startups that wish to get into the geospatial space. OpenPaths brands itself as “a secure data locker for personal location information.” There is one aspect where OpenPaths is very different from other services like Google Latitude:Only the user has access to his/her own data and it is never shared with anybody else unless the user cho ...继续阅读 (45)

Note: This would have been up a lot sooner but I have been dealing with a bug on and off for pretty much the past month!From April 26-28 I had the pleasure to attend theSIAM Data Mining conference in Anaheim on the Disneyland Resortgrounds. Aside fromKDD2011, most of my recent conferences had been more “big data” and “data science” oriented, and I wanted to step away from the hype and just listen to talks that had more substance.Attending a conference on Disneyland property was quite a bizarre experience. I wanted to get everything I could out of the conference, but the ...继续阅读 (35)

Recently, I participated in an email interview about what being a Statistics major entailed, how I got interested in the field and the future of Statistics. I figured this might be of interest to those that are contemplating majoring in Statistics, or considering a career in Data Science.Q1:Why did you decide to pursue a major in statistics in college?A:“When I was a kid, I really enjoyed looking at graphs, plots and maps. My parents and I could not make of what was behind the interest. At the same time, I was also heavily interested in education. My mother was a teacher and the first se ...继续阅读 (35)

Whenever I tell people in my family that I study Statistics, one of the first questions I get from laypeople is “do you count cards?” A blank look comes over their face when I say “no.”Look, if I am at a casino, I am well aware that the odds are against me, so why even try to think that I can use statistics to make money in this way? Although I love numbers and math, the stuff flows through my brain all day long (and night long), every day. If the goal is to enjoy and have fun, I do not want to sit there crunching probability formulas in my head (yes that’s fun, b ...继续阅读 (31)

To all of my readers and followers, I wish you a very Merry Christmas and a very joyous and safe Happy New Year! This year, I am thankful for the community that has sprung up around Data Science and open-source data collection and processing. This blog is almost two years old, and like with Twitter, I have been able to communicate with many data scientists, enthusiasts and some of the most prolific contributors to the data science software community. I am thankful for all of the wonderful people I have met and have yet to meet, and for your comments and reading. ...继续阅读 (60)

Lately I have doing a lot of work with theWikipedia XML dumpas a corpus. Wikipedia provides a wealth information to researchers in easy to access formats including XML, SQL and HTML dumps for all language properties. Some of the data freely available from the Wikimedia Foundation includearticle content and template pagesarticle content with revision history (huge files)article content including user pages and talk pagesredirect graphpage-to-page link lists: redirects, categories, image links, page links, interwiki etc.image metadatasite statisticsThe above resources are available not only for ...继续阅读 (42)

A month ago, I wrote aboutalternatives to the Hadoop MapReduce platformand HPCC was included in that article. For more information,see here.LexisNexishas open-sourced its alternative toHadoop, calledHigh Performance Computing Cluster. The code is available onGitHub. For years the code was restricted to LexisNexis Risk Solutions. The system contains two major components:Thor(Thor Data Refinery Cluster) is the data processing framework. It “crunches, analyzes and indexes huge amounts of data a la Hadoop.”Roxie(Roxy Radid Data Delivery Cluster) is more like a data warehouse and is des ...继续阅读 (50)

<< My review of Day 1.I am summarizing all of the days together since each talk was short, and I was too exhausted to write a post after each day. Due to the broken-up schedule of the KDD sessions, I group everything together instead of switching back and forth among a dozen different topics. By far the most enjoyable and interesting aspects of the conference were the breakout sessions.KeynotesKDD 2011 featured several keynote speeches that were spread out among three days and throughout the day. This year’s conference had a few big names.Steven Boyd, Convex Optimization: From Embedded R ...继续阅读 (78)

I have been waiting for the KDD conference to come to California, and I was ecstatic to see it held in San Diego this year.AdMelddid an awesome job displaying KDD ads on the sites that I visit, sometimes multiple times per page. That’s good targeting!Mining and Learning on Graphs Workshop 2011I had originally planned to attend the 2-day workshopMining and Learning with Graphs (MLG2011)but I forgot that it started on Saturday and I arrived on Sunday. I attended part of MLG2011 but it was difficult to pay attention considering it was my first time waking up at 7am in a long time. The first ...继续阅读 (68)

It’s been a while since I have posted… in the midst of trying to plow through this dissertation while working on papers for submission to some conferences.Hadoop has become the de facto standard in the research and industry uses of small and large-scale MapReduce. Since its inception, an entire ecosystem has been built around it including conferences (Hadoop World,Hadoop Summit), books, training, and commercial distributions (Cloudera,Hortonworks,MapR) with support. Several projects that integrate with Hadoop have been released from theApache incubatorand are designed for certain u ...继续阅读 (46)

I woke up early and cheery Wednesday morning to attend the2011 Hadoop Summitin Santa Clara, after a long drive from Los Angeles and theBig Data Campthat lasted until 10pm the night before. Having been toHadoop Summit 2010, I was interested to see how much of the content in the conference had changed.This year, there were approximately 1,600 participants and the summit was moved a few feet away to the Convention Center rather than the Hyatt. Still, space and seating was pretty cramped. That just goes to show how much the Hadoop field has grown in just one year.KeynotesWe first heard a series of ...继续阅读 (82)

It has been a while since I have been to Silicon Valley, butHadoop Summitgave me the opportunity to go. To make the most of the long trip, I also decided to check outBigDataCampheld the night before from 5:30 to 10pm. Although the weather was as predicted, I was not prepared for the deluge of pouring rain in the end of June. The weather is one of the things that is preventing me from moving up to Silicon Valley.The food/drinks/networking event must have been amazing because it was very difficult to get everyone to come to the main room to start the event! We started with a series of lightning ...继续阅读 (49)

Recently, I have been thinking about alternate ways of specifying search queries other than with text. A couple of weeks ago I came across a piece of music that I could not identify. I thought it would be a huge win for a search engine to allow me to upload this piece, and it would present me with matches, or near matches to other pieces that sound similar, or have similar characteristics. Some services already exist.Shazamallows a user to place a microphone near playing music and it will identify the artist and song. Some uses of search-by-sound:Music identification (“solved” R ...继续阅读 (47)

I am usually pretty reserved with cash, but after working full-time for six months, I finally decided to spend some of my money on building a new research development server. This process was long overdue and the reason it took me so long to commit to this project was all of the new technology developed since building my last server. This “new technology” can be pretty confusing unless one specializes in computer architecture. I want to share what I have learned throughout this process, while giving some background. These are only my opinions, and I may be wrong on some things as I ...继续阅读 (37)

Some time over the past 6 weeks I randomly saw a tweet announcing the “Data Scientist Summit” and shortly below it I saw that it would be held in Las Vegas at theVenetian. Being a Data Scientist myself is reason enough to not pass up this opportunity, but Vegas definitely sweetens the deal! On Wednesday I woke up at 6am to partake on the5.5 hour voyage to Las Vegas.The Pre-PartyThe Venetian and all close hotels were booked, so I ended up at theAria; a new experience. The hotel is beautiful and very ritzy. I had heard that the rooms were very technologically advanced but I wasn ...继续阅读 (188)

Elastic Compute Cloud (EC2)is a service provided aAmazon Web Servicesthat allows users to leverage computing power without the need to build and maintain servers, or spend money on special hardware. The idea is simple, the user “boots” up one or more machines and then accesses those machines as if they were logged into any other machine remotely. I used EC2 andElastic MapReduceextensively for my M.S. thesis last spring, but mainly used its large memory capabilities rather than its potential for explicit parallelism.Recently, I ran a crawling job on EC2 using a parellel crawler I wr ...继续阅读 (31)

This weekit was revealed that the iPhone stores users’ locations, and this immediately caused a huge firestorm of commentary by tech geeks, panic among privacy advocates, and delight to data geeks like myself. Even better/worse, it seems that the iPhone caches location traces long-term, possibly back to the date the phone was activated.I ditched my iPhone this past December (good riddance) in favor of the Droid X (Android). I figured, on such an open source OS, Google must be doing the same thing. After surfing through Hacker News, it turns out I was right.Compared to the iPhone though, ...继续阅读 (31)

Numpy and SciPy are packages for numerical computation and scientific computing, for Python.One wrinkle with NumPy/SciPy that needs to be ironed out is the difficulty of installation on certain OSes, and particularly, architectures.TheSciPy SuperPackhas done a good job of taking care of this issue, but it has not yet been updated for 2.7.1 and manually hacking away at its script has not worked for me.I cannot take credit for the instructions in this article. A brave warrior, Jeremy Conlin, somehow managed to figure out how to install 64-bit NumPy and SciPy, with 64-bit Python 2.7.1 on Snow Leo ...继续阅读 (36)

As most readers are probably aware, the free IDE forR, calledRStudio, was recently released for general use and it immediately made huge waves within the R community. IDE stands forIntegrated Development Environment. IDEs typically provides a rich set tools developing in some target language. For standard programming languages like C++ (VisualStudio) and Java (Eclipse or NetBeans), IDEs contain:an editor tailored to the target language. The editor typically has tab/auto-complete for variable names, functions and class methods and properties and also featuressyntax highlighting.amultiple docume ...继续阅读 (38)

Programming defensively requires knowing the input that your code should be able to handle. Typically, the programmer may be intimately familiar with the type of data that his/her code will encounter and can perform checks and catch exceptions with respect to the format of the data.Web mining requires a lot more sophistication. The programmer in many cases does not know the full formatting of the data published on a web site. Additionally, this format may change over time. There are certain standards that do apply to certain types of data on the web, but one cannot rely on web developers to fo ...继续阅读 (33)

I am happy to report that ByteMining is listed on “40 Fascinating Blogs for the Ultimate Statistics Geek“!Some of the ones that I frequently read, or are written by Twitter friends/followers (in no particular order):R-bloggers, an aggregate site containing blog posts tagged as posts about R. High quality content.Statistical modeling, causal inference and social science. This one is a no brainer, as it is the blog forAndrew Gelman‘s group.FlowingDataby Nathan Yau (@flowingdata), fellow Statistics Ph.D. student at UCLA. Focuses on the data and information visualization side of ...继续阅读 (36)