The Social Media Macroscrope: a simpler way to conduct social media data research and analysis

Illinois team develops a comprehensive, user-friendly tool for improving social media research and invites the wider research community with open access.

social media macroscope home page

Written by: Mariana Seda, Technology Services


In the past decade, the explosion of data capture from social media has catapulted data science to the forefront of businesses and research institutions alike. This data offers a wealth of insight about consumer sentiment, brand awareness and loyalty, and perceptions of social movements. Traditionally, analyzing that data requires expertise in the field of computational science, leaving some social scientist with limited resources to conduct their research. Fortunately, Illinois data scientists have developed a tool that bridges the gap between social researchers and data analytics, and are now poised to share this resource with the broader academic community.

The Social Media Macroscope (SMM) is a joint project of the University of Illinois Technology Services and the National Center for Supercomputing Applications (NCSA) with the goal of making social media data, analytics, and visualization tools accessible to researchers and students of all levels of expertise.

It was developed by a team of data scientists and social media analysts who looked at the current available model for social media data research, and realized that it was not helping the people who need it the most.

“Right now, companies who want to know more about consumers buy the data from social networks and build tools to analyze that data,” said Joe Yun, Leader of Social Research and Technology Innovation at Technology Services, and the mastermind behind the Social Media Macroscope. “How they do this, and their formulas are proprietary. These companies can’t give up their tools or they will lose their business.” he said. Since these algorithms are kept secret, they are not always verifiable.

“But data science isn’t magic; it is very mathematical,” said Yun. “So, if something in your formula is wrong, then the output, the results, will be wrong too.”

The existing model for analyzing data can often be problematic for social data researchers. Many do not necessarily have a computer science background and cannot build their own algorithms to conduct their research, limiting researchers’ abilities to get correct and verifiable information. There are many computational experts writing formulas capable of analyzing vast amounts of data in various ways, but matching the right researcher with the best analytic tool for their problem is seemingly impossible when the levels of expertise are disparate.

The Social Media Macroscope removes that limitation by acting as the “middle person connecting all these realms,” explained Yun. It is a gateway that provides a single point of access to a collection of open-source tools and resources for performing social media data collection, analysis, and visualization. Within it social scientists can process and store large datasets and collaborate with other researchers by sharing ideas, data, and methods.

The first tool developed for the SMM is the Social Media Intelligence and Learning Environment, or SMILE. SMILE helps researchers to create Twitter and Reddit data sets and analyze the data for natural language processing, sentiment analysis, and network analysis. There are plans to integrate more social media data sources in the future.

What’s next for the SMM?

“The Social Media Macroscope will hopefully become the world’s most powerful research environment for social media,” said Yun. The SMM is currently in a proof-of-concept stage. But it already has more than 100 contributing researchers doing real-time data analysis. And that number is expected to grow.

The Social Media Analytics team is scheduled to present at the 2018 Gateway Computing Environments Conference, at the University of Texas at Austin this year. They hope to spread the word about the Macroscope and connect more users with tools that enable better data research. The Social Media Macroscope is now available as an open source software.

“We are releasing the proof-of-concept to the wider academic community and we want to start partnering closer with researchers to incorporate their algorithms into the environment. We are also working on the next tool within the Macroscope called the brand analytics environment.”

Additionally, the team is launching a laboratory that will focus on advertising research and testing new technologies for campus. For more information on the Social Research and Technology Innovation Lab visit their website.