Social Harvest® was originally closed source software as a service (SaaS). It was only available privately for a few clients and assisted a few social media marketing campaigns. It was also the source of a few research projects, such as the analysis of the 2012 US Presidential Elections across Twitter as well as some other infographics, studies, college lectures, and visualizations.
Social Harvest was created by, me, Tom Maiaroto (@Shift8Creative) and was the basis for my talk at 10gen's MongoSV 2011 on machine learning algorithms (sentiment analysis) within MonogDB since the Social Harvest software was designed to analyze sentiment. The software also prompted the research and discovery of the Virality Score algorithm which remains proprietary and protected. So I'm no stranger to online marketing and social media analytics. Nor am I a stranger to open-source. Now I'd like to focus my efforts on helping an industry grow from a broader perspective.
So Why Free?
During the several years of research, construction, and feedback it was evident that the landscape had moved in an unfortunate direction. Before long, the number of social media analytics companies had more than doubled. Each of these services varied greatly in terms of accuracy, features, and cost. In other words, we have a very muddy and very expensive landscape. This led to doubt in existing tools and a high cost to play. Only the larger brands and marketing agencies can afford the thousands of dollars per month to access this, otherwise free, data. Or companies would have to hire developers to build something custom that ultimately was too expensive to maintain or didn't quite work as intended.
There is no secret sauce in free data alone.
So why and how are people reselling all this free data? Shouldn't we be paying Facebook and Twitter instead? Well, the answer to that is convenience, user experience, interpretation, and visualization. In other words, they made all the API calls, stored some data, and graphed it for you. Some social media networks, such as Facebook, Twitter, Pinterest, and Google already do provide some of this (Facebook with its Insights and now Twitter's new analytics that launched with its ads), but it's more focused around advertisements on their platforms.
Generally speaking, I believe our options today are far too expensive. The services would need to do more, somehow add that missing secret sauce to their solutions (and sentiment analysis alone is not secret sauce either) and improve accuracy. The existing solutions could be improved but, with so many high paying customers, there simply is little motivation to improve.
I don't consider businesses "overcharging" to be the real problem. Afterall, they are only charging what the market will allow. The real underlying problem is a lack of customer education and intense anxiety. Big data analytics for online marketing is new to be frank. Even newer when it relates to social media. So people are still figuring it out and how it fits into their business. This takes time. Unfortunately, time waits for no man (or woman) and companies feel the pressure to connect with their audience through social media now more than ever. They have an intense desire to understand and analyze all this data despite it being new, untested, and complex. So they use an analytics service without fully understanding the value of what they are getting or if it is even what they need. They also have very few employees who understand how it works (if any). This leaves the customer unable to validate or verify the results from the service they are paying for. They simply must take it at face value and make business decisions with it. Or worse yet, instruct their clients based on the information.
We can't keep everyone in the dark.
We need an open-source community and open-source tooling around this problem in order to fully expose it. Only then can we solve it. We've missed a step somewhere along the way in our education and it's time to address that. The best way, in my opinion, is through open-source where software and process evolve and get collaboratively improved. Let's lay all our cards out on the table. It allows us to ask why without compromising anyone's integrity. We need a safe, nuturing, environment.
Social Harvest now aims to change the landscape of big data analytics. By making an "enterprise grade" (I dislike this ambiguous term) open-source tool and community, not only will accuracy and quality increase but, the barrier to enter big data analytics for social media will be lowered. I firmly believe that there is a need for SaaS and businesses to interpret and analyze big data for others, but I also see a completely underserved set of users. Social Harvest will act as a counter weight to this unbalanced industry and take away some of the smoke and mirrors.
Open data needs open tools.
Those who want to do it themselves will have an easier time and those who want to pay for a service won't have to break the bank to do so. It will also set a standard so customers paying for services will have the means to better understand and justify their decisions. More people will be able to effectively use social media because the accessibility to tools and knowledge will be greater. It's a win win for everyone. Even the existing analytics companies, because open-source starts conversations and focuses interest. Think about WordPress for example. Now think about how many more bloggers exist thanks to open-source tools like it. When barriers are lower, more people get in the game. However, it will force existing services to respond and this will also be part of the change.