Institute for Ethics and Emerging Technologies

The IEET is a 501(c)3 non-profit, tax-exempt organization registered in the State of Connecticut in the United States. Please give as you are able, and help support our work for a brighter future.

Search the IEET
Subscribe and Contribute to:

Technoprogressive? BioConservative? Huh?
Quick overview of biopolitical points of view

whats new at ieet

Douglas Rushkoff joins IFTF as Research Fellow

This virtual lab will revolutionize science class

William James: Once Born and Twice Born People

Will World War 3 Be Prevented Because of Global Interdependence?

The Injustice of Sexism

NASA Can Get Humans to Mars by 2033 (Without a Budget Increase!)

ieet books

Philosophical Ethics: Theory and Practice
John G Messerly


almostvoid on 'This virtual lab will revolutionize science class' (May 28, 2016)

instamatic on 'NASA Can Get Humans to Mars by 2033 (Without a Budget Increase!)' (May 26, 2016)

almostvoid on 'Where does intelligence come from?' (May 26, 2016)

almostvoid on 'The Future of PR in Emotionally Intelligent Technology' (May 25, 2016)

almostvoid on 'Rituals Improve Life According to Ancient Chinese Philosophers' (May 25, 2016)

almostvoid on 'Optimize Brain Health by Balancing Social Life with Downtime' (May 23, 2016)

instamatic on 'Faithfulness--The Key to Living in the Zone' (May 22, 2016)

Subscribe to IEET News Lists

Daily News Feed

Longevity Dividend List

Catastrophic Risks List

Biopolitics of Popular Culture List

Technoprogressive List

Trans-Spirit List


Enframing the Flesh: Heidegger, Transhumanism, and the Body as “Standing Reserve”

Moral Enhancement and Political Realism

Intelligent Technologies and Lost Life

Hottest Articles of the Last Month

Ethicists Generally Agree: The Pro-Life Arguments Are Worthless
May 17, 2016
(4324) Hits
(10) Comments

Artificial Intelligence in the UK: Risks and Rewards
May 12, 2016
(3351) Hits
(0) Comments

Nicotine Gum for Depression and Anxiety
May 5, 2016
(3046) Hits
(0) Comments

3D Virtual Reality Is the Best Storytelling Technology We’ve Ever Had
May 5, 2016
(2859) Hits
(1) Comments

IEET > Vision > Futurism > Contributors > Patrick Tucker

Print Email permalink (0) Comments (11422) Hits •  subscribe Share on facebook Stumble This submit to reddit submit to digg

The Three Things You Need to Know About Big Data, Right Now

Patrick Tucker
By Patrick Tucker
World Future Society

Posted: Mar 11, 2012

Okay. You got me. I can’t really tell you everything you need to know about big data. The one thing I discovered last week – as I joined more than 2,500 data junkies from around the world for the O’Reilly Strata conference in rainy Santa Clara California—is that nobody can, not Google, not Intel, not even IBM. All I can guarantee you is that you’ll be hearing a lot more about it.

What is big data? Roughly defined, it refers to massive data sets that can be used to predict or model future events. That can include everything from the online purchase history of millions of Americans (to predict what they’re about to buy) to where people in San Francisco are most likely to jog (according to GPS) to Facebook posts and Twitter trends and 100 year storm records.

With that in mind, here’s the three most important things you need to know about big data right now:

1. The data experts are organizing and they want a revolution!

Data mining, (the primordial ancestor to what is today called predictive analytics) used to be considered a company or organization-specific problem. The data and the people who worked on it were “siloed” in effect. What would a statistics expert in the military and a number cruncher working in retail marketing have to talk about?

These days, it turns out, there’s a lot to discuss. First, new open source data crunching tools like Hadoop (a distributed operating system that lets you gang together thousands of computers to solve problems) are helping organizations big and small develop their own data departments at a small fraction of how much specialized software used to cost a few years ago. That means that the skills that miners are acquiring in one industry like retail are increasingly applicable across sectors, like in government. Second, combining data sets yields new insights, and the number of available sets (in some easily crunchable form like XML or just Excel) is growing.

“Over the last couple of years we’ve seen the horizontalization of data scientist,” says Alistair Croll of Bitcurrent, one of the organizers of the conference.

There was a considerable (but not surprising) consensus among attendees that data and analytics should drive a lot more decision-making within organizations, even if that better-informed strategizing comes at the expense of traditional managers, who will argue that their hard-won expertise is much more valuable than any model based on statistics. More and more often, they’ll be shown to be wrong.

There’s plenty of debate over whether everyone who works with large data sets in a technical way should get to call themselves a “data scientist.” It may be a matter of the uniqueness of the research, or just a price point.

According to JC Hertz, “if you’ve got someone in your organization that can do analytics, don’t call them a data scientist. They’ll ask you for a $20,000 raise and then get a job down the road.”

What that means: according to Hertz: “Data driven decisions have consequences. There can be political and cultural fallout. This is a gating condition that you need in the beginning. You have to say, this might [anger] x, y, z, and know that in the beginning. Not just outside the organization, but within. You need to know the political consequences of any given data-driven decision and who that decision will tick off.”

2. You’re going to be asked to opt-in to sharing your data a lot more.

A major topic for discussion this week was the Target Snafu. As originally reported in the New York Times (reg req.), Target raised a lot of eyebrows when the company used customer data and predictive analytics to figure out that one of their customers was pregnant, and, more remarkably, what trimester she was in. They emailed her some promotional material and the girl’s father discovered his daughter was pregnant based on the coupons she started receiving from a big box retailer, which gave rise to an awkward conversation, no doubt.

Most of the people I spoke with here agreed that Target made a mistake in that case, but they believed the error wasn’t in collecting the data and then using it for marketing so much as doing so without permission.

Big organizations are just beginning to realize the huge upside potential of using massive amounts of data to predict everything from what their customers are going to start buying to which of their employees will complete a certain project on time. More importantly, that data is getting increasingly easy and cheap to collect, and there’s already an enormous storehouse of it to aid in pattern extrapolation.

At the same time, customers are becoming more and more aware and alarmed by how their data is being used. This conference took place on the eve of two extremely important developments on this front, the Obama Administration putting forward a privacy bill of rights and Google HTML Tutorial changing its privacy policy to allow the company to use your browsing and search history in an assortment of weird ways, well beyond ad targeting.

So where is the middle ground? According to many of the folks here, it’s the point where people knowingly agree to contribute data. As one programmer put it, “Spying is the act of collecting data secret. Transparent data collection with defined boundries is NOT spying.”

What that means: more companies will look to make the case that allowing them to track your behavior will benefit you. If enough people buy the pitch, societal attitudes about data tracking will change. There are a lot of things organizations can do to make the offer a good one for consumers, but they haven’t yet.

As Alistair Croll of Bitcurrent put it, “Imagine if that [New York Times] article had said, Target figured out that 1% of its customer base had cancer and it told them. I would sign up for a program that tracked my purchases to let me know if there was a correlation between what I bought and what people that got colon cancer bought.”

3. The stuff you can predict is amazing, the stuff you can’t is frustrating.

This conference was full of amazing case examples of people using big data to predict things. According to Google’s Hal Varian, unemployment query volume on “Sign up for unemployment” can predict future unemployment claims with a high degree of accuracy one week before official numbers are released from the U.S. government. Coupon and rebate search queries are an excellent predictor of weak economic times ahead.

Having said that, the hype on big data is likely to grow faster than the actual capabilities, as are incidents of “data washing” or making some especially considering how early we are on the hype cycle.(see graph above)

“The most prevalent model in the industry to address this problem is MCU, make crap up,” according to marketing guru Avinash Kaushik.

What that means: Too many organizations are too focused on collecting data without a clear sense of what to do with it. The order should be reversed, according to several presenters. If you want to get started with data-driven decision making first set goals and then start amazing and crunching data sets around those goals.

Most importantly, many agreed that having great data collection and analysis capability is useless if an organization doesn’t have internal processes in place to allow people to use the new info, and not just at the top of the corporate pyramid.

“You’ve got to empower every person to make decisions with data” according to Kaushik. “Say, ‘You, Janitor! You will be in charge of using Data to make your job better!”

Bottom line: Big data is going to change the way organizations and individuals deal with information and plan ahead. Many of those transitions will be difficult; but, ten years from now, we’ll wonder how we got along without it. Even after the hype cycle on big data goes from peak to valley, there’s still a lot to look forward to.

Patrick Tucker is a senior editor and writer for THE FUTURIST magazine, an international magazine about technological, environmental, and societal trends.
Print Email permalink (0) Comments (11423) Hits •  subscribe Share on facebook Stumble This submit to reddit submit to digg


YOUR COMMENT (IEET's comment policy)

Login or Register to post a comment.

Next entry: Exciting Progress in Commercialization of Stem Cell Therapies

Previous entry: ‪Project KARA (tech demo from Quantic Dream)


RSSIEET Blog | email list | newsletter |
The IEET is a 501(c)3 non-profit, tax-exempt organization registered in the State of Connecticut in the United States.

East Coast Contact: Executive Director, Dr. James J. Hughes,
56 Daleville School Rd., Willington CT 06279 USA 
Email: director @     phone: 860-428-1837

West Coast Contact: Managing Director, Hank Pellissier
425 Moraga Avenue, Piedmont, CA 94611
Email: hank @