The Web Data Services Blog

While web data, especially social media data, grows exponentially, the vast array of opportunities for using real-time web data to improve analysis and decision making is limited only by your imagination.  These days, companies must incorporate Web data into their intelligence and analysis tools in order to compete. In some industries it’s a matter of survival.

Real-time data is where the answers are.  It’s where market and customer trends are immediately identifiable.  It’s where deals will be won and where winners will claim their trophies.

Reality Buzz

We recently built something 30M Americans can relate to – a way to predict American Idol and other reality show results based on data harvested from popular social media sites.

Every week on American Idol, contestants perform, their fans dial-in their support for their favorites, and the next day contestants are voted off the show.  During the performances, and for several hours after, fans tweet about and discuss their favorites online, showing support for the ones they want to see voted through to the next show.

We scrape thousands of pieces of web data from twitter, Facebook, forums and discussion sites around the web, apply sentiment analysis, analyze the data, and make predictions about the person(s) to be eliminated from the show, all in the span of a few hours.

We built the robots (automated web data collection processes) in a matter of hours.  Now they are automated to collect the data, transform unstructured data into structured data, and load it into a MySQL database.

In the last two weeks starting with the top 12 contestants, we’ve successfully predicted the American Idol contestant to be eliminated hours before the elimination show aired.  For more information on our latest predictions and to learn more, please visit Reality Buzz on Facebook.

Imagine what you could do for your business with Kapow’s Web Data Server and a few hours creating Kapow robots.  Real-time web data can fuel predictive analytics capabilities to give your company an unfair advantage.

Over 400 Kapow customers are jumping in with both feet.  What’s stopping you?  Learn more on our Kapowtech.com website, or contact us for a Free Trial of the Kapow Web Data Server.

By:  Rick Kawamura

There has been a lot of buzz around the increasing need for APIs, especially around “free” and “open” APIs.

As Dion Hinchcliffe writes in his latest blog on ebizQ, (Open APIs Mature Into a Next-Generation Business Model and Is the Future of SOA Open Source?), APIs (or the lack thereof) are the biggest obstacle for developing next-generation business applications and SOA interoperability.

Why?  First, the APIs have to be available, but they also need to be simpler (for example using REST) and easier to consume by BI tools, agile application environments and mashup builders.

Dion describes how more and more companies are providing open or free APIs to their data as an important part of their business model. These APIs are supplemented by a new line of companies, like StrikeIron and Xignite, who provide APIs to other’s data through an easy data-as-a-service (DaaS) model (check out this article in WSJ, The New Information Goldmine).

This is well aligned with the Open Government Initiative and the new US government data site data.gov where more and more government data will become available. You can read more about this initiative in this article about Digital Democracy.

But can we realistically wait for all relevant data to become “API enabled”?

With more than 5 billion websites today, there is a vast amount of growing, relevant data that is not going to have an API any time soon – if ever. Add to this data locked internally in legacy applications and at your business partners and you can see how unrealistic it is to have APIs for all this data.

This is where Web Data Services and a product like the Kapow Web Data Server become critical.

Web Data Services allow business analysts and agile application developers to instantly create APIs where none exist. The only prerequisite is that you can navigate to and see the data in a normal web browser like FireFox, IE or Safari. This even includes data behind secure, password protected sites, and data on very complex websites powered by AJAX and Flash.

With the powerful combination of open APIs, free government data and the ability to rapidly turn any web application into an API on-demand, we finally have access to any data we need.

This lays the foundation to a new way of working, where business analysts and other decision makers can spend their time building better algorithms, better data visualization, and better analysis because the most critical ingredient behind any business decision today, the data, has become so easily accessible.

By:  Stefan Andreasen