The Web Data Services Blog

Tune in and join Stefan as a guest on DM Radio’s broadcast, “A Tactical Approach to Enterprise Data Integration“, this Thursday, January 28, at 3PM EST.  Learn more about how Web Data Services delivers timely, trusted, web data for enterprise data integration.

On the front lines of enterprise data integration, practitioners employ a whole host of tactics in order to deliver timely, trusted, valuable data. These days, a wide range of options are available for gathering, cleansing and delivering enterprise data. From Web scraping to federated data management, ETL to ELT, open-source to the traditional heavy lifting done by major vendors – the data integration practice is more mature than ever. Tune into this episode of DM Radio to learn how the modern enterprise can mix and match tactics to suit practically any information need. We’ll talk to industry analyst Philip Russom of TDWI, plus Ilan Sehayek of Jitterbit, Peter Tran of Composite Software, and Stefan Andreasen of Kapow Technologies.

Click over to DM Radio on the Information Management site for more details.

Many of you have asked for a deeper dive on the meaning of “Web Data Services”, so let me answer it here.

First, it’s important to understand the terms “Data Integration” and “Application Integration”.

Data Integration (DI) and (Enterprise) Application Integration (EAI) are not the same, though many vendors often confuse the two. Application Integration focuses on managing transactions or messages between applications while Data Integration focuses on managing the flow of data and providing standardized API’s to access the information. For more details, refer to Mark Madsen’s blog, Key Differences Between Data Integration and App Integration.

There are essentially three different types of Data Integration:

• Consolidation means moving all the data from the original data sources to a new repository, much like an ETL tool.

• Propagation means moving only the necessary data to a local storage for each application consuming the data.

• Federation means leaving the data at the original source and accessing it as needed in real-time.

Web Data Services is in reality all forms of Data Integration as well as Application Integration, with two distinct differences.  With Web Data Services:

• You primarily access data and business logic residing on the web (any application or data source you can access from a Web browser like Internet Explorer, Firefox or Safari). This includes applications inside your own organization and even at your business partners.

• You do not need to recode or have programmatic access to any of the data sources. As long as you have access from a Web Browser, you can access the data with no coding and be up and running in a matter of hours rather than weeks or months.

Web Data Services is the new highly productive way to access almost any of the data you need for Business Intelligence (BI), Data Validation and Acquisition, Enterprise Mashups, Partner Integration, or basically any solution that needs agile access to data or business logic. Web Data Services gives unheard of business agility and competitive advantage compared to traditional Data Integration or Enterprise Application Integration methods.

Try it free with our Kapow Web Data Server Trial Offer

Also check the Wikipedia entry on Web Data Services to read the definition from leading industry analysts.

By:  Stefan Andreasen

Is Data Integration really “un-sexy”?

Peter Giblett recently wrote a great post, 7 Key Challenges for Data Integration, in his CIO Community blog that started off saying that Data Integration is often seen as the “un-sexy” part of IT.  But in many respects, I think that view is changing.

The reason IT thinks Data Integration is un-sexy is because it’s de-motivating and usually a waste of time to try solve a problem that is of little importance or is not yet really a problem.

Solving data integration for data sources without knowing when and if they are needed, or even if it will ever be critical to the company, is definitely not very appealing.

The future of data integration has to be much more agile, allowing IT to quickly deliver the right data at the right time to make timely critical business decisions.

This process of creating urgency and relevance for data access and delivery transforms data integration from being un-sexy to, for a lack of a better word, sexy.

It’s a lot more fun to solve data integration problems when the immediate benefits are known than to try to guess the needs of the future.

The definition of “sexy” is a relative term.  What is sexy to one person may not be the same for another.  With Web Data Services, coders and developers will be able to satisfy the time-critical data needs of their business users, and thus have not only a sexy job, but also make Data Integration a sexy part of IT.

So what tools can help IT think Data Integration is sexy?  What tools can deliver agile data access of timely, relevant data that helps the business immediately make better business decisions?

I definitely think Web Data Services is one of those tools.

What do you think?

By:  Stefan Andreasen

I really enjoyed being a guest on DM Radio’s broadcast last week where we discussed “Innovation in Information Management”.

We live in a time of rapid change through an accelerated flow of information.  The time span between market fluctuations and subsequently the time to make appropriate decisions in direction or strategy is shortening each day.  More than ever it’s become necessary to look into the “crystal ball” to know what’s ahead so we can react quicker (and correctly).

The “old school” solution is to pre-build all the data access you could ever think of so it’s ready when you need it. This is the normal approach of SOA, Data Warehousing and Data Integration products.  It’s very resource intensive with a “something’s bound to stick” approach to providing the right data at the right time.

The “new approach” is to provide IT with the tools so they can deliver relevant, timely data from anywhere in an on-demand fashion when it is needed.

When I was asked, “What is the biggest innovation you have seen in data governance?” I responded that I’ve seen a common thread where most of the innovation is powered by enabling technologies with Web Data.  And it’s not simply a “better, cheaper, faster” approach, but quite honestly a paradigm shift in taking advantage of the accelerated world to turbo-charge business intelligence tools with web data services.

That’s real innovation in an accelerated world.

By:  Stefan Andreasen

A few weeks ago, I had a great chat with Jamie Thomson from EMC about Web Data Services.  I noticed Jamie recently wrote an interesting blog post titled, “ETL for HTML”.  ETL is a well known term for anyone working with Data Integration or Data Warehousing. It stands for Extract, Transform and Load, and describes a one-way process of extracting data from a source, transforming the data into a new format and then loading the data into a destination. Traditional ETL vendors like Informatica are most effective for extracting and loading data from sources which can be accessed in traditional ways through SQL, XML or program APIs. This is where Web Data Services products like Kapow Web Data Server come in as a next-generation ETL tool. The Kapow Web Data Server allows users to Extract and Load data to and from all the data sources, including those that cannot be accessed in traditional ways, with the only prerequisite being that users are able to access and see the data in a normal Web Browser.

We live in a browser-centric world today where “ETL for HTML” encompasses the 2 extremes:  Web2.0 (e.g. web scraping, mashups, etc.) and Enterprise Data Management (e.g . data extraction, data collection, data mining, data conversion, data integration, etc.).  “ETL for HTML” is the perfect universal term that best describes working with all the data we work with and see in our Web browsers. This gives us fast and automated access to any data in applications like SalesForce or NetSuite or any of the millions of other web-based applications that exist inside our firewall, at our business partners, with the government, or just out on the public web.

Jamie is spot-on with the term “ETL for HTML” as a way to describe how most of us will access web data.  Although ETL traditionally describes a one-way process of moving data from point A to point B, Web Data Services provides two-way access to data. This means we can leave the data where it resides best (like in your HR or ERP applications) and get full programmatic access by using a product like the Kapow Web Data Server to “wrap” the applications into standard service APIs like REST, SOAP or .NET.

Why is this so important? Well for two reasons.  First, with the data explosion around us it becomes impractical to move and synchronize data into one common data repository.  Second, the data we need to perform our analysis and drive business decisions will change more and more rapidly. We will need new data sources daily, or at least weekly, to react to the ever changing business needs of the future.

So what is a good replacement for the term “ETL for HTML”? I suggest something like “Access, Enrich and Serve Web data”. This is a superset of ETL that also covers the way we want to access data in the future.

What term do you think we should use?

By:  Stefan Andreasen