• Home
• Media
• News Article

Predicting black swans

Could the Arab Spring have been predicted? More significant, could we have predicted that Saudi Arabia will remain stable throughout the turmoil? Or even where Bin Laden was hiding? New data mining techniques suggest we can.
Twitter, Facebook and other social media websites are a data miner's heaven. Gauging public moods and opinions, new techniques are combining social media with tones and chatter from mainstream media to create a more accurate trajectory of world events.
"News is increasingly being produced and consumed online, supplanting print and broadcast to represent nearly half of the news monitored across the world today by Western intelligence agencies," writes Kalev Leetaru, who conducted the study. His full title: Assistant Director for Text and Digital Media Analytics at the Institute for Computing in the Humanities, Arts, and Social Science at the University of Illinois and Center Affiliate of the National Center for Supercomputing Applications.
"Recent literature has suggested that computational analysis of large text archives can yield novel insights to the functioning of society, including predicting future economic events."
Applying tone and geographic analysis to a 30-year worldwide news archive, global news tone is found to have forecasted the revolutions in Tunisia, Egypt, and Libya, including the removal of Egyptian President Mubarak, predicted the stability of Saudi Arabia (at least through May 2011) and estimated Osama Bin Laden's likely hiding place as a 200-kilometer radius in Northern Pakistan that includes Abbotabad.
The study makes use of a 30-year translated archive of news reports from nearly every country of the world, "applying a range of computational content analysis approaches including tone mining, geocoding, and network analysis, to present 'Culturomics 2.0,'" notes the author.
If you think this is a gold mine for intelligence agencies, you are right - and they are already way ahead of us, for data mining by intelligence agencies is not a new concept.
In the lead up to the Second World War, British and American intelligence agencies set up the Foreign Broadcast Information Service (FBIS -- now the Open Source Center) and Summary of World Broadcast (SWB). And one of the first analysis by the agencies was a day before the Japanese struck Pearl Harbour:

"Japanese radio intensifies still further its defiant hostile tone; in contrast to its behavior during earlier periods of Pacific tension, Radio Tokyo makes no peace appeals. Comment on the United States is bitter and increased."
The analysis may have come too late to avoid the catastrophe, but it does point to the huge potential data mining offers.
Predicting Egypt
SWB and FBIS continue to live and aid western intelligence agencies and are now being combined with twitter and other social media feeds to predict new developments.

Leetaru claims that 'Culturomics 2.0' had correctly predicted Egypt's political upheavals:
"On 25 January 2011, popular dissent with the Egyptian state culminated in mass protests that continued through President Mubarak's resignation on 11 February. Figure below shows the average tone by month from January 1979 to March 2011 of all 52,438 articles captured by SWB mentioning an Egyptian city anywhere in the article....
"To normalize the data, the Y axis reports the number of standard deviations from the mean, with higher numbers indicating greater positivity and lower numbers indicating greater negativity. January 2011 reports only the tone for 1 January through 24 January, capturing the period immediately preceding the protests.
"Only twice in the last 30 years has the global tone about Egypt dropped more than three standard deviations below average: January 1991 (the U.S. aerial bombardment of Iraqi troops in Kuwait) and 1-24 January 2011, ahead of the mass uprising. The only other period of sharp negative moment was March 2003, the launch of the U.S. invasion of neighboring Iraq.

Tunisia
Similarly, the two-week period prior to Tunisian President Ben Ali's resignation was the sixth-most negative period in the last 30 years, coming after a decade-long plunge towards increasing negativity.

Saudia Arabia
Unlike Egypt and Tunisia, articles about Saudi Arabia hovered around negative levels, but did not plunge like Egypt and Tunisian data shows.

"But this level had been reached repeatedly over the previous ten years, whereas tone for Egypt, Libya, Tunisia, and other countries undergoing revolution reached sharply lower levels than previously seen. Thus, substantial tonal movement towards negativity is an indicator of possible unrest, while absence of such movement indicates greater likelihood of stability," notes Leetaru.

Locating Bin Laden
News stories on Osama Bin Laden, who was later killed by U.S. commandos in the Pakistan city of Abbotabad, also offered clues to his whereabouts, the study claims.
While Abbotabad was mentioned only once, 49% of the news on Bin Laden associated him to a city in Pakistan, and both Islamabad and Peshawar rank in the top five non-Western cities associated with him.
This is a tenuous example, and the author admits it: "While far from a definitive lock on Bin Laden's location, global news content would have suggested Northern Pakistan in a 200 km. radius around Islamabad and Peshawar as his most likely location, and that he was nearly twice as likely to be making his residence in Pakistan as Afghanistan."
Future Shock
The animated chart below tracks the tone of global news from 1979 to the present, as mapped by the study's author. Green represents positive news and red negative - the collection of news stories suggest more red than green in our future
(click to see animated graphic)

© alifarabia.com 2011

---

© Copyright Zawya. All Rights Reserved.

---