Saturday, June 1, 2013

Being economical with the data - Tim Harford

Economics will have to change what it recognises as a question, and what it recognises as an answer

Not everyone means the same thing when they talk about “big data”, but here are a few common threads. First, the dataset is far too big for a human to comprehend without a lot of help from some sort of visualisation software. The time-honoured trick of plotting a scatter graph to see what patterns or anomalies it suggests is no use here. Second, the data is often available at short notice, at least to some people. Your mobile phone company knows where your phone is right now. Third, the data may be heavily interconnected – in principle Google could have your email, your Android phone location, knowledge of who is your friend on the Google Plus social network, and your online search history. Fourth, the data is messy: videos that you store on your phone are “big data” but a far cry from neat database categories – date of birth, employment status, gender, income.