Data is scattered across data silos

Technically, all data you need is stored within your organisation. But, as long as your data remains scattered in silos across multiple departments and isn’t analysed, it will be useless. Unifying data, will deliver value to your organisation. Unified data supports upstream data consumers like data scientists and analysts to run queries, to get all of the data they need. Unifying data from across complex systems, is however one of the hardest hurdles to take. Many enterprises have over hundreds if not thousands of systems and using ETL is no go.

Unifying your data fully automated with the Smart

Data Fabric

The Smart Data Fabric solves the most difficult challenge in data management: “How to unify data from across complex systems and data sources in an automated way?” The first step is to collect (extract) data. This is the easy part. But just collecting data, isn’t enough. To make data unified, your data need to be connected. Optimally, the result should be “golden records”: trusted data that is accurate and correct. Data that you can rely on. To achieve this, the Smart Data Fabric creates connected data and improves the data quality by cleansing the data, de-duplicating and normalising the data and completing empty records in an automated manner. With this unique and automated way of data integration, it doesn’t matter if just a dozen data sources need to be integrated or several thousand! Only the ingestion time will increase.

Unify your data with the Smart Data

Fabric

Fully automated (!), the Smart Data Fabric collects all of your data and put it into one central place, cleans it, deduplicates it, keeps your data updated and relevant constantly and makes it available to upstream consumers.

The engines that drive this process

By using ‘ eventual connectivity’ , all data (structured, unstructured and image only files that will be OCR-ed) from internal and external sources will be collected by crawlers and blended automatically via an ingenious process by means of i.a. our merging engine. We build connected data without a need for having to know schemas upfront. The data blending will be done on the fly (during the data integration process itself) and relations between data be put in place automatically. Graph technology is at the heart of this process but search cluster, blob store, relation and distributed cache store are equally important for speed and overall functionality. Our inference engine helps you to infer connections out of even the dirtiest of data. To infer connections will take some time, but provides better quality results. Our weighted decision engine makes decisions only when it is statistically confident that a decision is correct. If the confidence level is too low, we wait until more data is ingested and revisit this decision again. We can show you why decisions are taken which also allows our engine to learn from the decisions taken. This engine contributes to constantly re- evaluating, updating and enriching your data. In fact, the more data that will be ingested, the higher the quality will become. Our cleansing engine cleanses and normalises data. It will correct spelling mistakes, and will correct incorrect identifiers such as emails phone numbers and addresses. For this the Smart Data Fabric uses i.a.: fuzzy merging of i.a names, companies and locations named entity extraction for determining the statistical likelihood of matches parse trees for understanding the context behind text external lookups for validating input. The cleansing and formatting process is done automatically. With this step, your data is prepared optimally for further data processing the Smart Data Fabric does. Our de-duplication engine provides you a generic way of de- duplicating absolutely anything, from documents to tasks. The Smart Data Fabric consolidates the duplicates and simply let you know about the different locations of the same documents. Our reinforcement learning allows you with human interaction and input, to further improve the quality of your valuable data. Once your data flows through the Smart Data Fabric, we stream some questions that need to be answered so the Data Fabric can learn e.g. regarding your specific product names. With this, it helps to make (future) decisions on your data. Our processing engine (pipeline) is a large combination of processing steps to make sense of any type of data and to cleanse and enrich it. Processes are supported by dashboards and intuitive interfaces. Among other tools, our 18 data quality metrics allow you to see the quality of your data per metric. By adjusting the levels, automated tasks can be approved groupwise by your data engineers and your data stewards can support assigned tasks (yes/no-questions) as part of the reinforcement learning process.

Making unified data available: data streaming

In the Smart Data Fabric, all unified data is available to you as a data stream. The Smart Data Fabric uses graph-based modelling and supports all use cases! As mentioned above, the Smart Data Fabric utilises five different types of databases allowing you to model and process the data you need. You “subscribe” to a certain subset of data, and that data will be delivered to the application or platform you use. New processed data in your enterprise, matching this subset will be delivered near real-time. Every application will benefit from receiving “live” data and data that is increased in value. Similar functionality is supported by “keep me in the loop”, which allows you to receive information near real-time in e.g. your mailbox, allowing you to act on this new and relevant information. The Smart Data Fabric unifies data in an automated manner and creates with this a solid data foundation from which all data is queryable! It can stream high-quality data for further processing (analysis, data science, BI, AI, innovation etc.). You have full control over how you want to use your data. The Smart Data Fabric simply “returns” your data cleaner and enriched in a flexible way. With this, efficiency will be improved and time freed up to be spend on business use cases and better decisions can be taken!