The Digital Integration Hub Turbocharges Your API Strategy

On June 26th, 2018 my esteemed colleague Eric Thoo and I published an Innovation Insight research about the digital integration hub (DIH). Innovation Insight is a type of Gartner research that we use to provide our clients with an initial assessment and analysis of technologies, approaches or services that are just emerging in the market. That is, innovations still in the early stages of the Hype Cycle.

The DIH is, in fact, an emerging architectural paradigm that we have observed being adopted by a growing number of organizations to support customer experience and digital transformation initiatives.

These organizations have typically implemented a DIH to enable high-scale API-based access to system of record application and data, while minimizing workload on these system and delivering additional value via analytics.

Gartner defines the DIH as an advanced application architecture that aggregates multiple back-end system of record data sources into a low-latency and scale-out, high-performance data store. A DIH typically supports access to data via an API services layer. The high-performance data store is synchronized with the back-end sources via some combination of event-based, request-based, and batch integration patterns (see figure below).

The goal of a DIH is to enable the implementation of large-scale, high-throughput and low-latency front-end API services by:

• Preventing the back-end systems from being overwhelmed with excessive API services-generated workloads.
• Avoiding intractably complex integrations to support the API services implementation.
• Decoupling the back-end data sources from the front-end API services.

The DIH can be seen as an advanced form of push-caching between the front-end API services layer and the system of record layer. In contrast to classic caching, however, the DIH stores "all" of the relevant data (for example, information about customers, products and employees) and not just "the most frequently accessed." Moreover, additional business value can be obtained by extending the architecture with capabilities that support additional use cases, such as analytics.

What's Going On?

• Implementing APIs that provide fast and high-scale access to back-end system of record applications and data sources may imply:

1) Exposing these systems to a potentially massive, often low-value, workloads.

2) Implementing complex integration between the API layer and the systems of record.

• Supporting these scenarios may be quite costly (for example, in terms of hardware upgrades of the back-end systems) and difficult to implement (for example, if the data needed by the front-end API services are dispersed across a large number of systems).
• To address these challenges, application leaders have at times implemented various approaches that replicate back-end data into caching layers, but they have done this in a tactical fashion that impedes a broader utilization of this valuable business data.

What Should You Do?

• Reduce the cost and complexity of the API service layers, decouple them from the system of record, and enable 24/7 operations by consolidating back-end data sources into a scale-out, high-performance digital integration hub (DIH).
• Enable synchronization between the DIH data store and the back ends by fully supporting event-driven, request-driven and batch interactions in their hybrid integration platform.
• Empower repurposing of DIH data for other usages by tracking and sharing metadata and lineage of all data flow supporting the DIH implementation.

The research features a case study about the DIH implementation at Macquarie Bank. Many thanks to Rajay Ray, Head of Digital Engineering & Applied Innovation at Macquarie Group for his help on putting together the case study!

The full report is available here (Warning: Paywall - you must be a Gartner client to access the report).