Mike Ellis, Managing Director of Finworks, explains what data fabric is and what are its benefits to public organisations and data professionals
Data can be said to be an organisation’s most valuable asset. Organisations are finding more interesting and innovative things to do with data. A data fabric brings all this data together, catalogues it, subjects it to rules and policies and makes sure that data is accessible where and when it’s needed. Public sector organisations are planning or have implemented a data fabric architecture to power strategic, data-driven services critical to their organisation’s digital transformation mission. Success is often due to recognising the human side of creating a data fabric as it teaches people new ways to think about, analyse, and use data.
Getting trusted data
Data Discovery functionality allows the ingestion and organisation of structured and unstructured data including complex and qualitative information that is often difficult to reduce into database records. When data gets onboarded and incorporated into a data fabric, it gains value by consolidating all that data under a single data catalogue. This value is a critical reason to implement a data fabric. The idea is to enrich, activate, and govern data to drive and improve business outcomes. Data discovery accelerates the delivery of new data for analytics efforts, improves data quality, and increases trust in data.
A data fabric handles the work of assembling and assuring data, so when applications consume it, it’s been subjected to a standard set of checks and tools for data quality. Data validation is a standard set of checks and tools for data quality. Enrichment is primarily about analysing data to add or update meaning to the content in a data fabric. As data is ingested it is cleaned, validated and enriched and ready for consumption by services and applications but most importantly by data professionals to subject the data to ad hoc queries, or use it in big data models and analytics.
Data compliance and governance
Effective use of data fabric automation makes data more accessible and available to data users. Some of the most important metadata created during enrichment deals with compliance and governance. Metadata means “data about data” and addresses the relationships between and among data items. The overarching nature of a data fabric makes it ideal for incorporating, accommodating, and enforcing policy and governance on the organisation’s data. This means that data professionals can incorporate compliance and governance requirements into data fabric policy.
Data governance is an important part of getting the right data to the right people. Data should be available to those who need it and are authorised to see it. Governance allows the right access to the data fabric catalogue, analytics tools, and data holdings, so self-service users can find data, create queries and run their analytics.
Self-service an emerging trend
Self-service is an emerging trend in organisations where a data fabric is available. Self-service users understand the data they’re using, and a data fabric allows them to find new and valuable ways to put data to work by themselves. The data fabric solution allows the democratisation of data and self-service users find an astonishing number of ways to use data. Instead of waiting for IT, data analysts and professionals in other departments turn to the data fabric’s catalogue to explore data items of interest. Using efficient metadata- based search capabilities, they can quickly and easily assemble a collection of data items they wish to explore, investigate, and analyse.
The benefits of data transformation
Data transformation is the process of consolidating and aggregating existing fragmented and siloed approaches to managing, storing, and situating data, to be governed and consumed as a single data fabric. Transformation is made less nebulous and more real by addressing the question “what’s in it for me?” Focusing on the benefits of increased business volume, improved customer satisfaction and relation- ships, and new business opportunities and value-adds allow the buy-in from stakeholders, users and developers for a data transformation project. Applications come with stakeholders users, and developers, all of whom must understand and buy into that fabric. They must understand how to move data, where to store it, how to protect it, and how to make it accessible to authorised users.
Data fabric benefits
Data fabric brings worthwhile change and numerous benefits to those who buy into the vision including:
- Using a data fabric, organisations can find, access, and combine data from all available sources and all data types
- The data fabric can provide the agility, speed, scale, and reliability needed for enterprise-grade data systems
- A data fabric can process and provision data at all velocities from streaming data in real-time to scheduled batch jobs
- The data fabric provides a consistent, coherent view of, metadata for, and controls over data to meet organisational security, privacy, and compliance needs
The move to using a data fabric also allows change and transformation to organisational culture, processes, and operations metrics. The data and database professionals want to make sure data is properly tagged and labelled, accessible to the right users, and in line with compliance and governance policies and requirements. Stakeholders want to get the best and most value from their data, as well as the applications and services that consume them. Through collaboration, all groups help each other meet goals and objectives.
Data fabric use case
Finworks currently model a universe of 650 million financial instruments for a major public European institution. The solution includes daily processes to update 2 million dynamic data sets and perform associated complex calculations. The system detects and reports on anomalies across 12 months of data history, comprising 780 million entries.
The implementation achieved a substantial improvement in both volume and quality of outputs that are produced close to real-time. The solution eliminated duplication of work across affiliates and departments. Reference instruments data is disseminated to all national banks, Market Operations, other applications and self-service users.
Conclusion
A modern data fabric offers a single, coherent view of data repositories and resources wherever they reside. By eliminating duplication, offering clean, enriched data sources, and optimising access, organisations benefit from more and better insights and require less human resources to achieve it. For human data consumers, a data fabric provides a consistent, coherent and governed view of the data needed thus ensuring efficiency and maintaining stakeholder trust and confidence.
Please note: This is a commercial profile
© 2019. This work is licensed under CC-BY-NC-ND.
