The Web Data Services Blog

Today we announced a partnership with Composite Software to simplify and accelerate the integration of critical web data in large-scale data virtualization environments.

This is exiting!

In today’s world where data is rapidly spreading into data silos inside and outside company boundaries, the days of traditional ETL data consolidation to a central database or data warehouse are starting to fail with more frequency.

This is true for many reasons.  For example, data synchronization and data timeliness issues, and all the expensive database and data warehouse licenses and storage hardware needed for the consolidation.

Today it’s critical to be able to access any data on the fly as efficiently as if the data was consolidated, but access at its source where it lives best and is accurate and fresh.

This is exactly the problem that Composite Software solves. Their smart virtualization and data access algorithms mean that you can access data as fast (or sometimes even faster) than if you consolidate the data, and at the same time get fresh, real-time data directly from the source.

But without Kapow, Composite can only access the data if you have documented SQL or API access, and as I wrote in my previous blog Why Distribution of Data is exploding and what we can do about it, more and more relevant data does not have documented programmatic access.

Now, combining the power of Kapow Web Data Server with Composite Software, you get a complete one-stop solution to access your data at its source – a solution that covers 100% of all relevant data sources for your business applications or analytics/BI solutions.

All this without expensive recoding of existing applications, expensive data warehouse solutions, nor needing documented API or feed access.

This is a valuable new partnership for Kapow and Composite, which our customers, and the market at large, will greatly benefit from!

By:  Stefan Andreasen

We all know the amount as well as the distribution of data and applications is exploding around us. This explosion occurs not only within our company, but also globally on the internet, with government agencies and with business partners.

This causes huge problems for CIOs and IT departments all over the world. They are increasingly challenged to deliver projects that drive increased IT project backlog and budget overrun:

• Automated B2B integrations
• Relevant and timely data for analytics
• Innovative new business applications
• Mobilization of existing applications

What is causing the Distribution of Data to explode?

To understand this we need to look back to the late nineties where two extremely important and closely tied technology innovations occurred:

• The Web Browser: A new one-fits-all interface to access any application, independent of underlying architecture and platform.
• The Internet: An extremely easy way to reach data and applications anywhere at any time.

Today, employees in companies all over the world perform their daily work through a web browser. Jobs in Business Development, Marketing, Sales, Business Intelligence, Market analysis, Financial management, Procurement, Research, HR or just about any other job function can be performed accessing applications and tools through a web browser.

The ease of reach to applications creates an explosion in demand for new data and new ways to work with data and applications, which in turn creates a huge pain for CIO’s and IT departments all over the world.

How to address Distribution of Data complexities?

Looking at this figure we see the explosion in distribution of data (the red area).  Up until the late 1990s, application and data integration required standard interfaces (SQL, ODBC, JDBC, APIs, etc.).  But with the explosion of content and data on the internet and the emergence of SaaS and Cloud computing, IT is faced with an exponential growth of web applications they can’t control and that don’t have standard API interfaces.

The problem is that there is no easy access to documented APIs.

Companies are spending an increasing amount of money to make those applications compatible with the known data integration world by creating APIs or data feeds.

I suggest we look at this challenge from a new perspective. If we could programmatically access the data and applications the same way a web browser does, we’d have access to everything instantly.

Today more than 400 companies globally are doing exactly this using Web Data Services from Kapow Technologies.  It’s a one-stop solution to easily access data and applications with incredible success.

Not only can they now get to all the data their LOB is demanding, but also achieve huge cost savings around:

• Decreased spending on information providers – they simply grab the data they need themselves.
• Increased automation of manual processes such as data-entry and B2B interactions.
• Elimination of the timely and often impossible negotiation and implementation of documented B2B integration protocols with suppliers and retailers.
• Lowered expenses on consultants for legacy re-write projects – instead they simply leverage the existing web interface and meet the demand for new business and mobile apps.

Try it for yourselves, it really works.

By:  Stefan Andreasen

Tune in and join Stefan as a guest on DM Radio’s broadcast, “A Tactical Approach to Enterprise Data Integration“, this Thursday, January 28, at 3PM EST.  Learn more about how Web Data Services delivers timely, trusted, web data for enterprise data integration.

On the front lines of enterprise data integration, practitioners employ a whole host of tactics in order to deliver timely, trusted, valuable data. These days, a wide range of options are available for gathering, cleansing and delivering enterprise data. From Web scraping to federated data management, ETL to ELT, open-source to the traditional heavy lifting done by major vendors – the data integration practice is more mature than ever. Tune into this episode of DM Radio to learn how the modern enterprise can mix and match tactics to suit practically any information need. We’ll talk to industry analyst Philip Russom of TDWI, plus Ilan Sehayek of Jitterbit, Peter Tran of Composite Software, and Stefan Andreasen of Kapow Technologies.

Click over to DM Radio on the Information Management site for more details.

Many of you have asked for a deeper dive on the meaning of “Web Data Services”, so let me answer it here.

First, it’s important to understand the terms “Data Integration” and “Application Integration”.

Data Integration (DI) and (Enterprise) Application Integration (EAI) are not the same, though many vendors often confuse the two. Application Integration focuses on managing transactions or messages between applications while Data Integration focuses on managing the flow of data and providing standardized API’s to access the information. For more details, refer to Mark Madsen’s blog, Key Differences Between Data Integration and App Integration.

There are essentially three different types of Data Integration:

• Consolidation means moving all the data from the original data sources to a new repository, much like an ETL tool.

• Propagation means moving only the necessary data to a local storage for each application consuming the data.

• Federation means leaving the data at the original source and accessing it as needed in real-time.

Web Data Services is in reality all forms of Data Integration as well as Application Integration, with two distinct differences.  With Web Data Services:

• You primarily access data and business logic residing on the web (any application or data source you can access from a Web browser like Internet Explorer, Firefox or Safari). This includes applications inside your own organization and even at your business partners.

• You do not need to recode or have programmatic access to any of the data sources. As long as you have access from a Web Browser, you can access the data with no coding and be up and running in a matter of hours rather than weeks or months.

Web Data Services is the new highly productive way to access almost any of the data you need for Business Intelligence (BI), Data Validation and Acquisition, Enterprise Mashups, Partner Integration, or basically any solution that needs agile access to data or business logic. Web Data Services gives unheard of business agility and competitive advantage compared to traditional Data Integration or Enterprise Application Integration methods.

Try it free with our Kapow Web Data Server Trial Offer

Also check the Wikipedia entry on Web Data Services to read the definition from leading industry analysts.

By:  Stefan Andreasen

Is Data Integration really “un-sexy”?

Peter Giblett recently wrote a great post, 7 Key Challenges for Data Integration, in his CIO Community blog that started off saying that Data Integration is often seen as the “un-sexy” part of IT.  But in many respects, I think that view is changing.

The reason IT thinks Data Integration is un-sexy is because it’s de-motivating and usually a waste of time to try solve a problem that is of little importance or is not yet really a problem.

Solving data integration for data sources without knowing when and if they are needed, or even if it will ever be critical to the company, is definitely not very appealing.

The future of data integration has to be much more agile, allowing IT to quickly deliver the right data at the right time to make timely critical business decisions.

This process of creating urgency and relevance for data access and delivery transforms data integration from being un-sexy to, for a lack of a better word, sexy.

It’s a lot more fun to solve data integration problems when the immediate benefits are known than to try to guess the needs of the future.

The definition of “sexy” is a relative term.  What is sexy to one person may not be the same for another.  With Web Data Services, coders and developers will be able to satisfy the time-critical data needs of their business users, and thus have not only a sexy job, but also make Data Integration a sexy part of IT.

So what tools can help IT think Data Integration is sexy?  What tools can deliver agile data access of timely, relevant data that helps the business immediately make better business decisions?

I definitely think Web Data Services is one of those tools.

What do you think?

By:  Stefan Andreasen

I really enjoyed being a guest on DM Radio’s broadcast last week where we discussed “Innovation in Information Management”.

We live in a time of rapid change through an accelerated flow of information.  The time span between market fluctuations and subsequently the time to make appropriate decisions in direction or strategy is shortening each day.  More than ever it’s become necessary to look into the “crystal ball” to know what’s ahead so we can react quicker (and correctly).

The “old school” solution is to pre-build all the data access you could ever think of so it’s ready when you need it. This is the normal approach of SOA, Data Warehousing and Data Integration products.  It’s very resource intensive with a “something’s bound to stick” approach to providing the right data at the right time.

The “new approach” is to provide IT with the tools so they can deliver relevant, timely data from anywhere in an on-demand fashion when it is needed.

When I was asked, “What is the biggest innovation you have seen in data governance?” I responded that I’ve seen a common thread where most of the innovation is powered by enabling technologies with Web Data.  And it’s not simply a “better, cheaper, faster” approach, but quite honestly a paradigm shift in taking advantage of the accelerated world to turbo-charge business intelligence tools with web data services.

That’s real innovation in an accelerated world.

By:  Stefan Andreasen

A few weeks ago, I had a great chat with Jamie Thomson from EMC about Web Data Services.  I noticed Jamie recently wrote an interesting blog post titled, “ETL for HTML”.  ETL is a well known term for anyone working with Data Integration or Data Warehousing. It stands for Extract, Transform and Load, and describes a one-way process of extracting data from a source, transforming the data into a new format and then loading the data into a destination. Traditional ETL vendors like Informatica are most effective for extracting and loading data from sources which can be accessed in traditional ways through SQL, XML or program APIs. This is where Web Data Services products like Kapow Web Data Server come in as a next-generation ETL tool. The Kapow Web Data Server allows users to Extract and Load data to and from all the data sources, including those that cannot be accessed in traditional ways, with the only prerequisite being that users are able to access and see the data in a normal Web Browser.

We live in a browser-centric world today where “ETL for HTML” encompasses the 2 extremes:  Web2.0 (e.g. web scraping, mashups, etc.) and Enterprise Data Management (e.g . data extraction, data collection, data mining, data conversion, data integration, etc.).  “ETL for HTML” is the perfect universal term that best describes working with all the data we work with and see in our Web browsers. This gives us fast and automated access to any data in applications like SalesForce or NetSuite or any of the millions of other web-based applications that exist inside our firewall, at our business partners, with the government, or just out on the public web.

Jamie is spot-on with the term “ETL for HTML” as a way to describe how most of us will access web data.  Although ETL traditionally describes a one-way process of moving data from point A to point B, Web Data Services provides two-way access to data. This means we can leave the data where it resides best (like in your HR or ERP applications) and get full programmatic access by using a product like the Kapow Web Data Server to “wrap” the applications into standard service APIs like REST, SOAP or .NET.

Why is this so important? Well for two reasons.  First, with the data explosion around us it becomes impractical to move and synchronize data into one common data repository.  Second, the data we need to perform our analysis and drive business decisions will change more and more rapidly. We will need new data sources daily, or at least weekly, to react to the ever changing business needs of the future.

So what is a good replacement for the term “ETL for HTML”? I suggest something like “Access, Enrich and Serve Web data”. This is a superset of ETL that also covers the way we want to access data in the future.

What term do you think we should use?

By:  Stefan Andreasen