The Web Data Services Blog

More and more organizations are benefiting from the treasure-trove of data available on the Web to feed their text analytics applications and power information solutions for voice of the customer, buzz metrics, brand protection, anti-piracy and open source intelligence (OSINT).

Tim Berners-Lee’s vision for the internet as a place to find and share data has – for the most part – come true. Unfortunately most of the data on the Web is locked into HTML and AJAX code with no easy way for a text analytics application to access it, at least not without getting a lot of noise that will spoil the results.

Traditionally companies use inefficient, expensive, old-fashion methods to collect Web data.

Unfortunately, as you can see in the figure, these methods typically cannot deliver valuable data a business analyst discovered when browsing the web for new data.

Today more and more companies are opening their eyes to a brand new and much more efficient way to extract precise data from thousands of websites in near real-time.

Companies are using Web Data Services to “free” the gold-mine of Web data.

Web Data Services accesses the data exactly the same way as a web browser, and can thus deliver 100% of the data an analyst wants. Now your company can leverage the gold-mine of web data to better understand your customers, competitors and market trends.

So my advice is: don’t waste your money on buying data from data providers. Get the data yourself!

To learn more about how Web data is a powerful tool for gathering real-time intelligence, I recommend you read my article, The Power of Social Media Data in Business, that I wrote for the Semantic Analytics 2010 conference in San Francisco.

Powerful, right!

By:  Stefan Andreasen

Today we announced a partnership with Composite Software to simplify and accelerate the integration of critical web data in large-scale data virtualization environments.

This is exiting!

In today’s world where data is rapidly spreading into data silos inside and outside company boundaries, the days of traditional ETL data consolidation to a central database or data warehouse are starting to fail with more frequency.

This is true for many reasons.  For example, data synchronization and data timeliness issues, and all the expensive database and data warehouse licenses and storage hardware needed for the consolidation.

Today it’s critical to be able to access any data on the fly as efficiently as if the data was consolidated, but access at its source where it lives best and is accurate and fresh.

This is exactly the problem that Composite Software solves. Their smart virtualization and data access algorithms mean that you can access data as fast (or sometimes even faster) than if you consolidate the data, and at the same time get fresh, real-time data directly from the source.

But without Kapow, Composite can only access the data if you have documented SQL or API access, and as I wrote in my previous blog Why Distribution of Data is exploding and what we can do about it, more and more relevant data does not have documented programmatic access.

Now, combining the power of Kapow Web Data Server with Composite Software, you get a complete one-stop solution to access your data at its source – a solution that covers 100% of all relevant data sources for your business applications or analytics/BI solutions.

All this without expensive recoding of existing applications, expensive data warehouse solutions, nor needing documented API or feed access.

This is a valuable new partnership for Kapow and Composite, which our customers, and the market at large, will greatly benefit from!

By:  Stefan Andreasen

Last week I was in Washington DC with John Yapaola, our CEO, to receive our ComputerWorld Honors Laureate Medal based on our nomination by Morgan Stanley.

Morgan Stanley has been a long time customer of Kapow Technologies and in 2008 they even became one of our investors.

Morgan Stanley’s nomination highlights in detail how the Kapow Web Data Server is critical to the research team’s ability to “advance their stock recommendations [and] manage data access and extraction for its complex research projects in orders of magnitude more efficiently compared with previous methods”.  Access to real-time web data simplifies the complexities of analyzing business opportunities and provides a significant advantage for Morgan Stanley.

Morgan Stanley continues to evolve and redefine the potential and impact of Web Data Services, and we are proud to have them as a customer and investor.

By:  Stefan Andreasen

We all know the amount as well as the distribution of data and applications is exploding around us. This explosion occurs not only within our company, but also globally on the internet, with government agencies and with business partners.

This causes huge problems for CIOs and IT departments all over the world. They are increasingly challenged to deliver projects that drive increased IT project backlog and budget overrun:

• Automated B2B integrations
• Relevant and timely data for analytics
• Innovative new business applications
• Mobilization of existing applications

What is causing the Distribution of Data to explode?

To understand this we need to look back to the late nineties where two extremely important and closely tied technology innovations occurred:

• The Web Browser: A new one-fits-all interface to access any application, independent of underlying architecture and platform.
• The Internet: An extremely easy way to reach data and applications anywhere at any time.

Today, employees in companies all over the world perform their daily work through a web browser. Jobs in Business Development, Marketing, Sales, Business Intelligence, Market analysis, Financial management, Procurement, Research, HR or just about any other job function can be performed accessing applications and tools through a web browser.

The ease of reach to applications creates an explosion in demand for new data and new ways to work with data and applications, which in turn creates a huge pain for CIO’s and IT departments all over the world.

How to address Distribution of Data complexities?

Looking at this figure we see the explosion in distribution of data (the red area).  Up until the late 1990s, application and data integration required standard interfaces (SQL, ODBC, JDBC, APIs, etc.).  But with the explosion of content and data on the internet and the emergence of SaaS and Cloud computing, IT is faced with an exponential growth of web applications they can’t control and that don’t have standard API interfaces.

The problem is that there is no easy access to documented APIs.

Companies are spending an increasing amount of money to make those applications compatible with the known data integration world by creating APIs or data feeds.

I suggest we look at this challenge from a new perspective. If we could programmatically access the data and applications the same way a web browser does, we’d have access to everything instantly.

Today more than 400 companies globally are doing exactly this using Web Data Services from Kapow Technologies.  It’s a one-stop solution to easily access data and applications with incredible success.

Not only can they now get to all the data their LOB is demanding, but also achieve huge cost savings around:

• Decreased spending on information providers – they simply grab the data they need themselves.
• Increased automation of manual processes such as data-entry and B2B interactions.
• Elimination of the timely and often impossible negotiation and implementation of documented B2B integration protocols with suppliers and retailers.
• Lowered expenses on consultants for legacy re-write projects – instead they simply leverage the existing web interface and meet the demand for new business and mobile apps.

Try it for yourselves, it really works.

By:  Stefan Andreasen

Social Media and BI are the sweet and sour, yin and yang, oil and vinegar topics of interest in BI these days.  Can the real-time, user-generated, free flowing tweets and online conversations of social media benefit traditional enterprise BI?

In the past week, Information Management published the following, Social Media Will Play a Big Part in BI’s Future.

No doubt the volume of social media is growing exponentially.  And surely, this data contains valuable information on competitive intelligence, product feedback, customer service, and even market trends.

But there’s a gap in social media data access.  Traditional BI tools can’t access all this unstructured data and present it in a usable format, let alone filter out all the noise.

What’s needed is an automated, flexible way to access hundreds or even thousands of sites in real-time, extract only the relevant content, add structure to the data, and load it easily into a database.  What’s needed is Web Data Services, and it exists today.

Social Media Data Access:

With hundreds of sites to monitor (most having no API access) and an already overburdened IT department, accessing social media data becomes the foremost hurdle to overcome.  With Web Data Services, all of this can be achieved with no coding.  Kapow robots (automated data collection processes) are easily created with visual point-and-click technology eliminating the need for complex, time-consuming coding and scripting.  If you can see the data in a web browser, Web Data Services can extract it.

Enriching Unstructured Data:

The trick is taking disparate text based tweets, comments, blog posts, online conversations, etc. and structuring them in a way that lets your analyst understand when it occurred, who said it, and how it applies to your keywords or hypothesis.  But getting there is harder than you might think.  Web Data Services surgically transforms unstructured social media web data to provide superior data quality without the noise.  Included, but not often talked about, is the ability to perform regular expressions (through a graphical interface), encoding and decoding, date formatting, string calculations, conditional expressions, numeric calculations, and multiple language support.

Making the data readily available:

Web Data Services makes it easy to output the structured social media data into multiple formats, such as a SQL database, vendor hosted database, Java or C# data structure, SOAP or REST Web service, RSS, CSV, or XML.

Social media is BI 2.0. It opens the doors to listen in on what people are saying about your brand, products and services, and also taps into untapped market opportunities and customer pain points.  So rather than reacting, you are out in front predicting future events and gaining first mover advantage.

By:  Rick Kawamura

I just read an interesting blog post by Richard MacManus, “10 Ideas For Web of Data Apps” that explains the fundamentals of how people generate ideas for new applications.

My guess is that all examples described in the blog came from people combining what they saw on multiple websites into an idea of how to leverage the data in a new, valuable application.

However there is a problem here!

It’s not a given that all that data is available as Linked or Open data.  In other words, not all data necessarily has a documented method of programmatic access as, for example, an XML feed, RSS feed, or a REST or SOAP service. Without this programmatic access, no existing application or Mashup builder can get to the data which prevents these great ideas from ever materializing. WHAT A BUMMER!

More often than not the data you need to combine into your great new application idea is only available in a web browser. This means you have to either drop the idea or settle for a subset of the data available with documented programmatic access.

Wouldn’t it be cool if all the data you see in a web browser were always available?

Well that is what Web Data Services is all about. Check more of this blog to learn more.

As always, please send me your comments, my email is sa at kapowtech.com

By:  Stefan Andreasen

Kate Gosselin on Dancing with the Stars Photo Credit: ABC

Can Social Media be used to predict the outcome of Reality TV shows such as American Idol and Dancing with the Stars?  We created Reality Buzz based on our real-time automated web data collection platform to find out.

Jennifer Zaino over at Semantic Web wrote a nice article that captures the essence of Reality Buzz and our process of using real-time social media web data to build intelligence in to predictive analytics:  Taking Sentiment Analysis to Dancing with the Stars and American Idol

Check it out.  And if you have the need to automate the access, collection, harvesting, scrubbing, grabbing or scraping or real-time web data to improve market or competitive analysis to improve your strategic decision making, we’re here to help.

By:  Rick Kawamura

Can oddball metrics from Craigslist apartment listings, Subway ridership tallies, Broadway ticket sales, city parking garage count of empty stalls, cardboard box production, or diesel fuel consumption be better economic predictors than traditional, months old government reports from the labor department?  Absolutely!

The Wall Street Journal just published “New Ways to Read Economy – Experts Scour Oddball Data to Help See Trends Before Official Information Is Available” where author Cari Tuna provides numerous examples of Economists around the country using non-traditional methods to better predict economic trends and direction.

The reason for this trend is that traditional reports and data are out of date and often not very accurate.  Who has six months to wait for a government report to make a decision?

Enter Web Data Services

What would make these oddball metrics more valuable and accurate?  Automating the collection process over multiple sources of data and loading it in to the database or BI tool of your choice.

Imagine you had the ability to automate the monitoring of hundreds of sources of data in real time and could react to changes overnight?  What data would you monitor?

Interest rates?  Gold Prices?  Credit Score reports?  Salesforce data?  Apartment listings?  Competitor’s pricing?  Product Buzz?  Customer complaints?  Financial transactions?  Bank balances?  Twitter?  Facebook?  Google Trends?  Linkedin profiles?  Partner inventory?  Shipment dates?

If you can see it in a web browser, whether on the public web, behind a login screen, or behind your firewall, that data can be accessed with Web Data Services to provide you with improved predictive analytics and strategic decision making.

As a fun example, Reality Buzz uses Kapow’s Web Data Server to monitor popular social media sites to evaluate America’s sentiment towards contestants on American Idol and Dancing with the Stars.  Overnight, data is collected and evaluated, and predictions are made about the fate of the contestants before the elimination show the following night.

Hundreds of businesses incorporate Kapow’s Web Data Server solutions to improve competitiveness, product offerings, and strategic decision making.  You can too.  What are you waiting for?

By:  Rick Kawamura

Together with my product management team, I am currently out speaking with our customers to gather input for future product roadmap priorities.

It’s always great to meet up with our customers, but in particular, what’s been so rewarding this time is hearing one amazing story after another from our customers on the huge time and cost savings they’ve achieved with the Kapow Web Data Server.

Let me share two of their stories.

First, I talked to a large telecommunications company, a long time customer of ours, who has been using Kapow on 80 data automation projects. One project, with expected revenue of $4.9 million over 3 years, included the assembly of 129,000 listings into a directory, all to be finished within 6-8 weeks.

Performed manually, one listing took 15 minutes to complete, so one person was expected to complete 30 listings per day, or 1200 listings in 8 weeks. Basically 100 employees were required to finish the project on time using traditional manual order-entry methodologies.

Using Kapow they successfully built data automation “robots” that took 44 data fields from the source application, transformed it into 69 target fields, and then automatically loaded them into the target system. The Kapow robots completely automated the interaction with the source and target application Web front-ends, as well as automatically performed all the advanced transformations to convert the source data format to the target data format. With robots they were able to complete 5000 automatic listings per day with a 95% success rate and finish the project in only 4 weeks.

This resulted in an astonishing total savings of $1.3 million and at the same time eliminated the risk of the project not getting done in time.

The second company, a business information provider, was extracting energy related data from a dozen government web sites using manual cut-and-paste. With Kapow they built automated data acquisition robots that saved 2868 hours per year of manual labor for each data source. For 12 data sources this resulted in an astonishing savings of 34,416 hours per year.

I love to hear these stories and it’s always very satisfying to hear a customer tell you directly how much they like and benefit from your solutions.

By:  Stefan Andreasen

While web data, especially social media data, grows exponentially, the vast array of opportunities for using real-time web data to improve analysis and decision making is limited only by your imagination.  These days, companies must incorporate Web data into their intelligence and analysis tools in order to compete. In some industries it’s a matter of survival.

Real-time data is where the answers are.  It’s where market and customer trends are immediately identifiable.  It’s where deals will be won and where winners will claim their trophies.

Reality Buzz

We recently built something 30M Americans can relate to – a way to predict American Idol and other reality show results based on data harvested from popular social media sites.

Every week on American Idol, contestants perform, their fans dial-in their support for their favorites, and the next day contestants are voted off the show.  During the performances, and for several hours after, fans tweet about and discuss their favorites online, showing support for the ones they want to see voted through to the next show.

We scrape thousands of pieces of web data from twitter, Facebook, forums and discussion sites around the web, apply sentiment analysis, analyze the data, and make predictions about the person(s) to be eliminated from the show, all in the span of a few hours.

We built the robots (automated web data collection processes) in a matter of hours.  Now they are automated to collect the data, transform unstructured data into structured data, and load it into a MySQL database.

In the last two weeks starting with the top 12 contestants, we’ve successfully predicted the American Idol contestant to be eliminated hours before the elimination show aired.  For more information on our latest predictions and to learn more, please visit Reality Buzz on Facebook.

Imagine what you could do for your business with Kapow’s Web Data Server and a few hours creating Kapow robots.  Real-time web data can fuel predictive analytics capabilities to give your company an unfair advantage.

Over 400 Kapow customers are jumping in with both feet.  What’s stopping you?  Learn more on our Kapowtech.com website, or contact us for a Free Trial of the Kapow Web Data Server.

By:  Rick Kawamura