LAK11 Week 2: The Rise of Big Data

This was great stuff, from the concrete (10 Ways Data is Changing How we Live,) to the extremely abstract (Computing a Theory of Everything)  and everything in between.  In fact, there’s something for the paranoid, the optimist,  the experimenter, the database/computing guru, the physicist or chemist, and more!

Given my propensity for fascination (i.e. distraction) , many other links were followed.  Now I’m faced with the dilemma of how to synthesize all of that into something I can post that has meaning, yet represents the pool of resources (data) traversed. Should I summarize, find patterns, correlate, or simply comment on the outliers that I either especially enjoyed, or deplored? Ah…I get it now – a microcosm of what I’m studying – but an up close experience-within-an experience of how a human handles “big data” which, in my case, is MANY orders of magnitude LESS than what a computer is capable of handling. Since I’m doing the processing, I’ll forgo the exhaustive approach for the efficient. :-) How about a list of appetizing sample quotes from the resources?  [Kind of like Costco on Saturdays .  You can go find (click-on) the whole package if you like the sample.]  Here goes, and don’t forget to properly dispose of your toothpicks on the way out.

"Number-crunchers have even uncovered match-fixing in Japanese sumo wrestling."

"Celent, an insurance consulting arm of Marsh & McLennan Cos., recently published a study suggesting insurers could use social-networking data to help price policies and aid in fraud detection....' Whether people actually realize it or not, they are significantly increasing their personal transparency,' (Mike Fitzgerald a Celent senior analyst) says. "It's all public, and it's electronically mineable."

"Where traditional businesses generally collect information about customers from their purchases or from surveys, internet companies have the luxury of being able to gather data from everything that happens on their sites. The biggest websites have long recognised that information itself is their biggest treasure. And it can immediately be put to use in a way that traditional firms cannot match. "     (editorial comment: The same should be true (but is not-yet) of educational web-sites )

"In this case we can see how the whole becomes not only more than but very different from the sum of its parts…in general, the relationship between the system and its parts is intellectually a one-way street.  Synthesis is expected to be all but impossible; analysis on the other hand, may be not only possible but fruitful in all kinds of ways"

"The planet has grown a central nervous system"

“This is a world where massive amounts of data and applied mathematics replace every other tool that might be brought to bear. Out with every theory of human behavior, from linguistics to sociology. Forget taxonomy, ontology, and psychology. Who knows why people do what they do? The point is they do it, and we can track and measure it with unprecedented fidelity. With enough data, the numbers speak for themselves.”

' “Revolutions in science have often been preceded by revolutions in measurement,” says Sinan Aral, a business professor at New York University. Just as the microscope transformed biology by exposing germs, and the electron microscope changed physics, all these data are turning the social sciences upside down, he explains. Researchers are now able to understand human behaviour at the population level rather than the individual level.' (editorial  comment:  note that the examples given here were more than measurements -- they were VISUALIZATIONS of measurements!)

"Displaying information can make a difference by enabling people to understand complex matters and find creative solutions. Valdis Krebs, a specialist in mapping social interactions, recalls being called in to help with a corporate project that was vastly over budget and behind schedule. He drew up an intricate network map of e-mail traffic that showed distinct clusters, revealing that the teams involved were not talking directly to each other but passing messages via managers. So the company changed its office layout and its work processes—and the project quickly got back on track. "  


And that last sample quote will make the perfect seqway into my post about SNAPP.