Stuart Steel, Founding Director of Insimbi Consulting Ltd, explains how harnessing modernised ETL processes and DPA as part of your digital transformation strategy can unleash the full potential of your data, drive significant cost and time savings and elevate your business above the competition
“Data is the new oil. It’s valuable, but if unrefined it cannot really be used.” – Clive Humby OBE, 2006
As the public and private sectors continue to adopt digital transformation strategies to enhance business efficiency and improve the customer experience, it is increasingly important to be able to extract maximum understanding, usability – and value – from your company’s data. This is where developing ETL pipelines and combining them with DPA can be crucial.
What is ETL?
ETL (Extract, Transform, Load) is an end-to-end process by which raw data from one or more disparate sources is extracted (for example queried from a database or read from a file), transformed (including filtering, validation, cleansing, enriching and homogenisation) and then loaded into an output container such as a data warehouse, or application for further use, analysis, and distribution.
Essentially, ETL processes enable consolidation of data from multiple sources into a business-useful, single source of truth, increasing visibility and simplifying access whilst helping to guarantee data integrity, validity, and currency.
Traditional versus modernised ETL
Traditionally, in-house IT staff would be tasked with developing bespoke batch scripts to perform specific on-premise ETL processes, which often proved time consuming, cost ineffective and with notable limitations:
- Single purpose and/or run once.
- Resource hungry with poor performance and scalability.
- Unable to accommodate changes in data sources or technologies without significant modification or rewriting.
- Unsuitable for real-time or big data.
- Inappropriate for an Agile environment.
In contrast, modernised ETL processes can be quickly created in off-the-shelf, GUI-based software with little or no IT staff involvement and without many of the limitations of traditional ETL scripts:
- Modular and reusable.
- Cloud based, fast, and scalable.
- Can accommodate changes in data sources and technology.
- Able to handle real-time and big data.
- Suitable for Agile environments.
What is DPA?
DPA (Digital Process Automation), an evolution of Business Process Management (BPM), seeks to reduce overheads, increase employee productivity, and improve overall operational efficiency through digital automation of manual, repeatable tasks, and facilitation of greater inter-departmental connectivity.
Utilising DPA to streamline business processes enables
- Improved collaboration throughout your organisation.
- Employee freedom from repetitive, mundane tasks.
- Operational stability, reducing user input (and therefore, human error) and ensuring processes are followed correctly.
- Greater insight into business performance and output quality.
- Market adaptability and improved customer experience.
One solution to rule them all
Well-designed DPA and ETL processes can underpin a vast range of digital transformation and business-critical activities, however introducing new technologies into an established ecosystem can be challenging, particularly in large organisations with multiple software stacks and siloed, legacy systems. But what if these challenges can be mitigated by introducing just one new software platform to implement all of your DPA and ETL process changes, plus much more, across the board?
Safe Software’s FME® is an enterprise integration solution that allows you to achieve your digital transformation goals with minimal IT staff involvement and without writing a single line of code. With support for over 450 different systems and formats it is virtually guaranteed to seamlessly integrate with your company’s data infrastructure, regardless of how, when and where your information is stored.
This is where we come in
The team at Insimbi Consulting has well over a decade of experience utilising FME® to assist organisations with their ETL tasks, information management, process automation, business intelligence, self-service applications, cloud migration and more, with a proven track record of enhancing our clients’ performance and reputation whilst saving them time and money.
Insimbi’s Certified FME® Professionals have designed, built, and supported numerous ETL solutions that have been used for delivery of contracts of all sizes, including on Europe’s largest infrastructure project.
For instance, the Enabling Works Contractor for that particular high speed rail project, Fusion JV (a joint venture between Morgan Sindall, BAM Nuttall, and Ferrovial Construction), was able to employ ETL to migrate away from a spreadsheet-based, legacy desktop information management system to a modern, cloud-based CDE (Common Data Environment).
With consultation from Insimbi, this CDE was able to comprise data seamlessly extracted from multiple internal and external sources, including Fusion’s own Document Management Systems and Asset Management System, their enterprise GIS (Geo- graphic Information System) and their client’s Electronic Documents & Records Management System, greatly simplifying stakeholder access to – and use of – this critical information.
Using FME® to harness the power of DPA, Insimbi were then able to help automate multiple business processes, providing Fusion with performance dashboards to monitor progress of deliverables and contract performance, real-time monitoring of field surveys, delivery notifications and automatic validation, consumption and ingestion of client and sub-contractor datasets, and numerous self-service tools from data validation and quality control through to report generation.
“Insimbi’s knowledge across multiple disciplines facilitated greatly in the streamlining and updating of business processes to provide a data rich vision for delivery teams, SLT, and the client. Savings were exponential in terms of resource time, a single automated workflow developed over 40 hours saved close to 4000 hours” – Graham Starling, Senior Information Manager, Ferrovial Construction UK & Ire.
Another of our key clients, FarrPoint, a UK connectivity and smart technology consultancy, enlisted Insimbi Consulting to aid delivery on one of their major connectivity projects.
We designed and built an FME®- based ETL solution that could ingest telecoms infrastructure provider datasets, validate them against template schemas, detect changes from previous versions and automatically produce summary assessment reports detailing the results.
Insimbi then further enhanced the value and utility of this data by using DPA to feed the validation results into Microsoft’s Power BI, enabling enhanced analysis and more easily digestible data and trend visualisations.
“Insimbi’s solution of automating the checks and transformations has resulted in significant time savings, especially for national scale datasets, and ensured accuracy and repeatability on a project with demanding timescales.” – Chris Bond, Principal Consultant, FarrPoint.
To find out how Insimbi Consulting Ltd can save you time and money on your company’s digital transformation journey, visit our website or contact us.
This work is licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International.
