4 Approaches To Systems Integration

Introduction

As businesses evolve and new technologies come to market, inefficient software platforms can begin impacting performance and efficiency across departments and functions. When looking to upgrade or integrate such systems, there are vast array of options to consider which will vary depending upon the type of product (i.e. off-the-shelf, bespoke, cloud hosted, etc.) and the level of integration services provided. This paper will look at some of these options and has been written based on a combination of research undertaken and direct experience in various software projects, across multiple industrial sectors, with budgets ranging in size from £50,000 to £1,000,000.

Understanding the Risks

Businesses who engage in IT software projects do so in the face of some pretty terrifying statistics relating to project failures, budget overruns and software misbehaving after release. Research undertaken by the McKinsey and the BT Centre for Major Programme Management at the University of Oxford [1] of large scale projects, being projects with an initial budget exceeding £10 million, discovered that 17 percent of projects failed so badly that they “threatened the very existence of the company”. Such projects are labelled within the industry as “Black Swan” projects, being defined as projects with budget overruns of at least 200 percent, based on a term originally coined by Nassim Nicholas Taleb [2] to define low probability events, with a large impact, which, with hindsight, are accepted as being rational and foreseeable.

Different Approaches

#1 Product Extensions

Scenario

“Our sales teams are exporting pricing data from our mainframe systems and generating their own bespoke proposals. Senior management has no visibility of the value or structure of pending opportunities, and operations are complaining that proposals are deviating from the products and services we offer.”

Description

Extending a product can introduce automated workflows over existing systems that can ensure consistency across users with the ability to capture information and provide detailed reporting. There are multiple ways of extending both off-the-shelf and bespoke products. Product extensions may be bespoke tools, depending upon the integration options available (i.e. programming interfaces, data import/export,etc.), or developed using inbuilt programming languages, where the legacy system is a product offering such support, such as SAP or Salesforce, which offer ABAP and Apex respectively.

#2 Cross-Platform Interfaces

Scenario

“At the moment customers email us spreadsheets of orders, if the customer is new we then create them in our CRM package to get a customer reference, then add the order to our production system, then add the order to our finance system. I have a separate user account for each system.”

Description

Wrapper interfaces can be developed to make internal operations more efficient but also to provide user interfaces for customers and supplier to increase productivity (by automating processes) and reducing risk (by auditing workflow activity). It may also be possible to support real-time cross-platform transactions across the underlying existing systems (where suitable programming interfaces exist). From a security perspective, relationships between user roles across systems will need to be considered. Also, where internal systems are being opened up to customers or suppliers planning around hosting and security will be required.

#3 Data Warehousing & Business Intelligence

Scenario

“Our engineers use a stand-alone application on their laptops, our finance system is hosted on an internal server and our HR system is in the cloud.”

Description

Aggregating data across multiple systems can provide businesses with the ability to make informed decisions based on consolidated, real-time information captured from multiple disparate systems. Data can also be stored and modelled over time to measure impact of decisions.

#4 Modular Refactor

Scenario

"We need to upgrade our current VB6/Access system. It doesn’t meet the current business needs and performance is terrible. However, the system has 10 years of undocumented tweaks and changes made by multiple developers that may or may not still be relevant to the business.”

Description

With some systems the only option is to refactor or rebuild the existing application; reducing the system into modular components helps reduce the complexity of the task. Systems with complex business logic will require a significant amount of input from the business to remodel, refine and validate the processes and workflows required. A major benefit of a modular approach is to ensure that dependencies on the business can be allocated per module. Finally, remember that a bad VB6 system upgraded directly to VB.NET without reviewing the functionality will still be a bad system.

Common Attributes of Successful Projects

Below is a summary of features and attributes we have found to be common within successful integration projects, based on our experience, which we hope will help readers enjoy success and real commercial returns, whilst avoiding those Black Swans:

#1 Avoid database level integration

Integrate using services and programming interfaces where possible, integrating at a database level potentially bypasses a significant amount of business logic and data validation. Updating databases directly can cause major impact to applications running on this database. There can also be issues when reading the data, where the data may be transformed within an application, for example, where an ID for a department is still being stored as an enumeration in the application itself; this mapping information must then also be maintained by the integration layer. Finally, a service interface is a contract, when releasing a new version of a system all endeavours will be made to support existing interfaces, this is not the case with the database structure; which may result in failure, or more concerning, the undetected incorrect performance of the integration software.

#2 Research off-the-shelf product APIs

Where there are one or more off-the-shelf products available research, and where possible prototype against, integration interfaces. There is often a considerable difference between pre-sales integration options and post-sales integration options, and poor interfaces will quickly turn an efficiently integrated platform into an isolated suite of stand-alone products. Understand the API restrictions: for example; is there a limit on the number of calls per day, and also any licencing implications of accessing the software via an integration layer?

#3 Understand integration

Analysis is required to decide whether integrating with one or more systems will involve the replication of data or the implementation of cross-system business processes. Real-time distributed transactional systems with message queues are considerably more powerful than replicating data between systems, however, they are also more complicated and require a much greater analysis and understanding of the process flows within and between each of the integrating systems.

#4 Minimise data duplication

Avoid having data residing in multiple systems, and where the data does need to be shared across systems, where possible only enable updates to be committed in one system with those changes being published out. Having multiple systems responsible for maintaining the same entities will lead to a number of race conditions and difficulties from an auditing perspective.

#5 Modular releases

Where possible, release early and often, and only move back release dates as a last resort. Research has shown large projects, projects with a budget of £650 thousand or more, have a failure rate 50 percent higher than small projects, projects with a budget of £230 thousand or less [5]. Although, as it is not always possible to deliver projects in stages, consider implementing proof of concepts, prototypes, minimum viable products and beta releases to introduce functionality to end users as soon as possible.

References

• McKinsey – Delivering large-scales IT projects on time, on budget, and on value
• Nassim Nicholas Taleb – The Black Swan: The Impact of the Highly Improbable
• Harvard Business Review - Why Your IT Project May Be Riskier Than You Think
• Linklaters – BSkyB vs EDS: Time to Reasses the Risks?
• Gartner – Why Projects Fail