The Web Data Services Blog

Social Media and BI are the sweet and sour, yin and yang, oil and vinegar topics of interest in BI these days.  Can the real-time, user-generated, free flowing tweets and online conversations of social media benefit traditional enterprise BI?

In the past week, Information Management published the following, Social Media Will Play a Big Part in BI’s Future.

No doubt the volume of social media is growing exponentially.  And surely, this data contains valuable information on competitive intelligence, product feedback, customer service, and even market trends.

But there’s a gap in social media data access.  Traditional BI tools can’t access all this unstructured data and present it in a usable format, let alone filter out all the noise.

What’s needed is an automated, flexible way to access hundreds or even thousands of sites in real-time, extract only the relevant content, add structure to the data, and load it easily into a database.  What’s needed is Web Data Services, and it exists today.

Social Media Data Access:

With hundreds of sites to monitor (most having no API access) and an already overburdened IT department, accessing social media data becomes the foremost hurdle to overcome.  With Web Data Services, all of this can be achieved with no coding.  Kapow robots (automated data collection processes) are easily created with visual point-and-click technology eliminating the need for complex, time-consuming coding and scripting.  If you can see the data in a web browser, Web Data Services can extract it.

Enriching Unstructured Data:

The trick is taking disparate text based tweets, comments, blog posts, online conversations, etc. and structuring them in a way that lets your analyst understand when it occurred, who said it, and how it applies to your keywords or hypothesis.  But getting there is harder than you might think.  Web Data Services surgically transforms unstructured social media web data to provide superior data quality without the noise.  Included, but not often talked about, is the ability to perform regular expressions (through a graphical interface), encoding and decoding, date formatting, string calculations, conditional expressions, numeric calculations, and multiple language support.

Making the data readily available:

Web Data Services makes it easy to output the structured social media data into multiple formats, such as a SQL database, vendor hosted database, Java or C# data structure, SOAP or REST Web service, RSS, CSV, or XML.

Social media is BI 2.0. It opens the doors to listen in on what people are saying about your brand, products and services, and also taps into untapped market opportunities and customer pain points.  So rather than reacting, you are out in front predicting future events and gaining first mover advantage.

By:  Rick Kawamura

I just read an interesting blog post by Richard MacManus, “10 Ideas For Web of Data Apps” that explains the fundamentals of how people generate ideas for new applications.

My guess is that all examples described in the blog came from people combining what they saw on multiple websites into an idea of how to leverage the data in a new, valuable application.

However there is a problem here!

It’s not a given that all that data is available as Linked or Open data.  In other words, not all data necessarily has a documented method of programmatic access as, for example, an XML feed, RSS feed, or a REST or SOAP service. Without this programmatic access, no existing application or Mashup builder can get to the data which prevents these great ideas from ever materializing. WHAT A BUMMER!

More often than not the data you need to combine into your great new application idea is only available in a web browser. This means you have to either drop the idea or settle for a subset of the data available with documented programmatic access.

Wouldn’t it be cool if all the data you see in a web browser were always available?

Well that is what Web Data Services is all about. Check more of this blog to learn more.

As always, please send me your comments, my email is sa at kapowtech.com

By:  Stefan Andreasen

Kate Gosselin on Dancing with the Stars Photo Credit: ABC

Can Social Media be used to predict the outcome of Reality TV shows such as American Idol and Dancing with the Stars?  We created Reality Buzz based on our real-time automated web data collection platform to find out.

Jennifer Zaino over at Semantic Web wrote a nice article that captures the essence of Reality Buzz and our process of using real-time social media web data to build intelligence in to predictive analytics:  Taking Sentiment Analysis to Dancing with the Stars and American Idol

Check it out.  And if you have the need to automate the access, collection, harvesting, scrubbing, grabbing or scraping or real-time web data to improve market or competitive analysis to improve your strategic decision making, we’re here to help.

By:  Rick Kawamura

Can oddball metrics from Craigslist apartment listings, Subway ridership tallies, Broadway ticket sales, city parking garage count of empty stalls, cardboard box production, or diesel fuel consumption be better economic predictors than traditional, months old government reports from the labor department?  Absolutely!

The Wall Street Journal just published “New Ways to Read Economy – Experts Scour Oddball Data to Help See Trends Before Official Information Is Available” where author Cari Tuna provides numerous examples of Economists around the country using non-traditional methods to better predict economic trends and direction.

The reason for this trend is that traditional reports and data are out of date and often not very accurate.  Who has six months to wait for a government report to make a decision?

Enter Web Data Services

What would make these oddball metrics more valuable and accurate?  Automating the collection process over multiple sources of data and loading it in to the database or BI tool of your choice.

Imagine you had the ability to automate the monitoring of hundreds of sources of data in real time and could react to changes overnight?  What data would you monitor?

Interest rates?  Gold Prices?  Credit Score reports?  Salesforce data?  Apartment listings?  Competitor’s pricing?  Product Buzz?  Customer complaints?  Financial transactions?  Bank balances?  Twitter?  Facebook?  Google Trends?  Linkedin profiles?  Partner inventory?  Shipment dates?

If you can see it in a web browser, whether on the public web, behind a login screen, or behind your firewall, that data can be accessed with Web Data Services to provide you with improved predictive analytics and strategic decision making.

As a fun example, Reality Buzz uses Kapow’s Web Data Server to monitor popular social media sites to evaluate America’s sentiment towards contestants on American Idol and Dancing with the Stars.  Overnight, data is collected and evaluated, and predictions are made about the fate of the contestants before the elimination show the following night.

Hundreds of businesses incorporate Kapow’s Web Data Server solutions to improve competitiveness, product offerings, and strategic decision making.  You can too.  What are you waiting for?

By:  Rick Kawamura

While web data, especially social media data, grows exponentially, the vast array of opportunities for using real-time web data to improve analysis and decision making is limited only by your imagination.  These days, companies must incorporate Web data into their intelligence and analysis tools in order to compete. In some industries it’s a matter of survival.

Real-time data is where the answers are.  It’s where market and customer trends are immediately identifiable.  It’s where deals will be won and where winners will claim their trophies.

Reality Buzz

We recently built something 30M Americans can relate to – a way to predict American Idol and other reality show results based on data harvested from popular social media sites.

Every week on American Idol, contestants perform, their fans dial-in their support for their favorites, and the next day contestants are voted off the show.  During the performances, and for several hours after, fans tweet about and discuss their favorites online, showing support for the ones they want to see voted through to the next show.

We scrape thousands of pieces of web data from twitter, Facebook, forums and discussion sites around the web, apply sentiment analysis, analyze the data, and make predictions about the person(s) to be eliminated from the show, all in the span of a few hours.

We built the robots (automated web data collection processes) in a matter of hours.  Now they are automated to collect the data, transform unstructured data into structured data, and load it into a MySQL database.

In the last two weeks starting with the top 12 contestants, we’ve successfully predicted the American Idol contestant to be eliminated hours before the elimination show aired.  For more information on our latest predictions and to learn more, please visit Reality Buzz on Facebook.

Imagine what you could do for your business with Kapow’s Web Data Server and a few hours creating Kapow robots.  Real-time web data can fuel predictive analytics capabilities to give your company an unfair advantage.

Over 400 Kapow customers are jumping in with both feet.  What’s stopping you?  Learn more on our Kapowtech.com website, or contact us for a Free Trial of the Kapow Web Data Server.

By:  Rick Kawamura

There has been a lot of buzz around the increasing need for APIs, especially around “free” and “open” APIs.

As Dion Hinchcliffe writes in his latest blog on ebizQ, (Open APIs Mature Into a Next-Generation Business Model and Is the Future of SOA Open Source?), APIs (or the lack thereof) are the biggest obstacle for developing next-generation business applications and SOA interoperability.

Why?  First, the APIs have to be available, but they also need to be simpler (for example using REST) and easier to consume by BI tools, agile application environments and mashup builders.

Dion describes how more and more companies are providing open or free APIs to their data as an important part of their business model. These APIs are supplemented by a new line of companies, like StrikeIron and Xignite, who provide APIs to other’s data through an easy data-as-a-service (DaaS) model (check out this article in WSJ, The New Information Goldmine).

This is well aligned with the Open Government Initiative and the new US government data site data.gov where more and more government data will become available. You can read more about this initiative in this article about Digital Democracy.

But can we realistically wait for all relevant data to become “API enabled”?

With more than 5 billion websites today, there is a vast amount of growing, relevant data that is not going to have an API any time soon – if ever. Add to this data locked internally in legacy applications and at your business partners and you can see how unrealistic it is to have APIs for all this data.

This is where Web Data Services and a product like the Kapow Web Data Server become critical.

Web Data Services allow business analysts and agile application developers to instantly create APIs where none exist. The only prerequisite is that you can navigate to and see the data in a normal web browser like FireFox, IE or Safari. This even includes data behind secure, password protected sites, and data on very complex websites powered by AJAX and Flash.

With the powerful combination of open APIs, free government data and the ability to rapidly turn any web application into an API on-demand, we finally have access to any data we need.

This lays the foundation to a new way of working, where business analysts and other decision makers can spend their time building better algorithms, better data visualization, and better analysis because the most critical ingredient behind any business decision today, the data, has become so easily accessible.

By:  Stefan Andreasen

Boris Evelson wrote and interesting  blog, “How Do We Define a BI Vendor”, on the Information Management site. It’s a great summary of the features that characterize a BI vendor (or product), but IMHO it’s missing the most vital characteristic of the all, data access to the right data.

BI analytics is useless without the right data.  No advanced BI feature can compensate for not having access to the most relevant and timely data.

Unfortunately for business analysts, a growing volume of relevant BI data is locked in Web sites beyond the reach of any standard data access method available in BI products on the market today.

Fortunately there is a solution; it’s called Web Data Services, a convenient, economical way to access data through an existing Web presentation layer (the same interface used by normal Web browsers like Internet Explorer, Firefox, Chrome and Safari). Web Data Services can both consolidate Web data into a standard database or wrap it into standard Web services, both of which can then be easily consumed by any BI product on the market.

Suddenly business analysts can access all the data they know from their application interfaces directly in the BI tools.

Jim Ericson recently wrote a nice article ( Net Expectations -What a Web data service economy implies for business ) about the value of Web Data Services in which he digs in to all the aspects of using the Web as a new way to get data that’s faster and more cost-effective.

The article includes this quote by longtime BI analyst Howard Dresner:

“The nice thing about Web data services… is that it’s easy, it’s relatively inexpensive to create or consume and it’s immediate. Business doesn’t want to wait until next quarter, and IT is gravitating this way too because they have only so much budget and so many people.”

What a powerful statement. With Web Data Services, it suddenly becomes easy, inexpensive and immediate to obtain data access, something I am sure many IT departments will praise while they struggle to deliver the necessary data in-time and within-budget to their increasingly data-hungry Lines of Business.

What is unique about the Kapow Web Data Server is that it can get you the data even when no APIs exist by leveraging the Web presentation layer interface that always exists on the Web.

Even if there is an API, the no coding robotic approach by Kapow Technologies is typically a lot faster and  an easier way to access the data.

Think about it. If I am a business analyst and need Salesforce.com data in my BI dashboard, I really don’t want to learn about Salesforce APIs and program lines of code. I’d much rather just point at the data with my mouse in the Kapow RoboMaker Visual IDE, and get data access directly the way I am used to. Here at Kapow Technologies we have dozens of these “robots” integrating SalesForce with Marketo (our marketing automation tool), our Emails in Microsoft Exchange, Jigsaw, our customer bug tracking system, our ERP systems, and so on… and we would never dream about using the APIs.

To make my final point, go back to Jim Ericson’s article to learn more about how Fiserv uses the Kapow Web Data Server to integrate with 300 banks in 10 countries, all with no coding. It’s an impressive real-life story about the value of Web Data Services.

As always, please send me your comments, my email is SA at kapowtech dot com

By:  Stefan Andreasen

In any kind of data access scenario, transforming and applying business logic to your extracted data is critical to structure the data to the exact format you need.

In ETL terminology this is the “T” in ETL.

When accessing data from Web based applications, this capability is critical since the data is typically unstructured and the more structure you can add, the higher the value.

Most Web Scraping and Screen Scraping tools on the market today typically lack adequate transformation capabilities.

Here are some examples as to how the Kapow Web  Data Server delivers full transformation:

Below is a small extract of a blog list from PW Forum:

On the first line in blue you see the timestamp set to “Today 15:15:13”. This timestamp denotes the time and date when this blog entry was posted, but it would need to be transformed into a fixed timestamp like “2010-02-14 3:15:13 pm Pacific time” to be useful in comparing it to other blog entries over the internet.

Here’s another example from Ebay. When you bid on an item on Ebay, the price of the item is red or green depending on whether you are the high bidder or not. This is important information “hidden” in the color that you would like to capture along with the price.  Once transformed, you’ll know not only the price, but whether it is the “highest bid” or not.  It’s a simple step to define the business logic as “if price is ‘green’ then set status to ‘high bidder’ otherwise set status to ‘not high bidder’.”

The Kapow Web Data server and its powerful visual programming IDE allows you to  apply any business logic and data transformation you can think of giving you the most powerful ETL for the Web product on the market today. And it’s all done visually with no need for any coding.

Try it out next time you need Web data for your BI or analytics tools.

By:  Stefan Andreasen

“Information, thanks to the internet, is growing at breakneck, exponential speeds. This has proven to be both an opportunity and a curse for businesses. With all this data abundance, organizations of all sizes struggle to access and act upon it in a timely and cost effective way. Just think about the impact to your business if you could automatically add high-value Web data to your market intelligence, pricing intelligence, financial intelligence or any other business intelligence application. Until recently, this seemed like an impossible feat, or at least cost prohibitive based on the man hours involved.”

The above is an outtake of an article I recently wrote for ebizQ, the full article is available here and provides more insight into the emerging web data services market, common web data access challenges and real world use cases and benefits of web data services. Feel free to check it out and let me know what you think.

Web Data – Why Top Business Leaders Depend On It And You Should To

By:  Ron Yu

Many of you have asked for a deeper dive on the meaning of “Web Data Services”, so let me answer it here.

First, it’s important to understand the terms “Data Integration” and “Application Integration”.

Data Integration (DI) and (Enterprise) Application Integration (EAI) are not the same, though many vendors often confuse the two. Application Integration focuses on managing transactions or messages between applications while Data Integration focuses on managing the flow of data and providing standardized API’s to access the information. For more details, refer to Mark Madsen’s blog, Key Differences Between Data Integration and App Integration.

There are essentially three different types of Data Integration:

• Consolidation means moving all the data from the original data sources to a new repository, much like an ETL tool.

• Propagation means moving only the necessary data to a local storage for each application consuming the data.

• Federation means leaving the data at the original source and accessing it as needed in real-time.

Web Data Services is in reality all forms of Data Integration as well as Application Integration, with two distinct differences.  With Web Data Services:

• You primarily access data and business logic residing on the web (any application or data source you can access from a Web browser like Internet Explorer, Firefox or Safari). This includes applications inside your own organization and even at your business partners.

• You do not need to recode or have programmatic access to any of the data sources. As long as you have access from a Web Browser, you can access the data with no coding and be up and running in a matter of hours rather than weeks or months.

Web Data Services is the new highly productive way to access almost any of the data you need for Business Intelligence (BI), Data Validation and Acquisition, Enterprise Mashups, Partner Integration, or basically any solution that needs agile access to data or business logic. Web Data Services gives unheard of business agility and competitive advantage compared to traditional Data Integration or Enterprise Application Integration methods.

Try it free with our Kapow Web Data Server Trial Offer

Also check the Wikipedia entry on Web Data Services to read the definition from leading industry analysts.

By:  Stefan Andreasen