Knowledge High quality dimensions
Taking a client viewpoint of information high quality is undoubtedly a invaluable preliminary step. However it may not cowl the completeness of the take a look at scope. In depth literature evaluations have addressed this challenge for us, providing a range of data quality dimensions which can be related to most use circumstances. It’s advisable to assessment the listing with information customers and collectively decide which dimensions are relevant and create checks accordingly.
| Accuracy | Format | Comparability |
| Reliability | Interpretability | Conciseness |
| Timeliness | Content material | Freedom from bias |
| Relevance | Effectivity | Informativeness |
| Completeness | Significance | Stage of element |
| Foreign money | Sufficiency | Quantitativeness |
| Consistency | Usableness | Scope |
| Flexibility | Usefulness | Understandability |
| Precision | Readability | |
You would possibly discover this listing too lengthy and marvel begin with it. Knowledge merchandise or any info system could be noticed or analyzed from two views: exterior view and inside view.
Exterior view
The exterior view is about the usage of the information and its relation with the group. It’s typically thought-about a “black field” with performance to signify the real-world system. The size that fall into the exterior view are extremely business-driven. Typically, the analysis of these dimensions could be subjective, so it’s not all the time straightforward to create automated checks for them. However let’s try just a few well-known dimensions:
- Relevancy: The extent to which information are relevant and useful for the evaluation. Contemplating a market marketing campaign geared toward selling a brand new product. All information attributes ought to straight contribute to the success of the marketing campaign akin to buyer demographic information and buy information. Knowledge like metropolis climate or inventory market costs are irrelevant information on this case. One other instance is the extent of element (granularity). If the enterprise desires the market information to be on the day degree, however it’s delivered on the weekly degree, then it’s not related and helpful.
- Illustration: The extent to which information is interpretable for information customers and the information format is constant and descriptive. The significance of the illustration layer is commonly ignored when accessing information high quality. It consists of the format of the information — being constant and user-friendly, and the that means of the information — being comprehensible. As an example, take into account a state of affairs the place information is anticipated to be accessible in a CSV file with descriptive column descriptions, and the values are anticipated to be in EUR foreign money fairly than in cents.
- Timeliness: The extent to which information is recent for information customers. For instance, the enterprise wants the gross sales transaction information with a most delay of 1 hour from the purpose of sale. It signifies that the information pipeline ought to be refreshed continuously.
- Accuracy: The extent to which information is compliant with enterprise guidelines. Knowledge metrics are sometimes related to difficult enterprise guidelines akin to information mapping, rounding modes, and many others. Automated checks on information logic are extremely advisable and the extra, the higher.
Out of the 4 dimensions, relating to creating information checks, timeliness and accuracy are extra simple. Timeliness is achieved by evaluating the timestamp column with the present timestamp. Accuracy checks are possible by means of buyer queries.
Inside view
In distinction, the interior view is anxious with the operation that is still impartial of particular necessities. They’re important whatever the use circumstances at hand. Dimensions within the inside view are extra technical-driven versus business-driven dimensions within the exterior view. It additionally implies that information checks are much less depending on customers and could be automated more often than not. Listed below are just a few key views:
- High quality of information supply: The standard of the information supply considerably impacts the general high quality of the ultimate information. The information contract is a superb initiative to make sure supply information high quality. As information customers of the supply, we are able to make use of the same method to watch the supply information as information stakeholders do when evaluating the information merchandise.
- Completeness: The extent to which info is retained in its entirety. Because the complexity of the information pipeline will increase, there’s a greater probability of knowledge loss occurring inside the intermediate phases. Let’s take into account a monetary system that shops buyer transaction information. The completeness take a look at ensures that every one transactions efficiently traverse your entire lifecycle with out being omitted or not noted. For instance, the ultimate account steadiness ought to precisely mirror the real-world scenario, capturing each transaction with none omissions.
- Uniqueness: This dimension goes hand-in-hand with the completeness take a look at. Whereas completeness ensures that nothing is misplaced, uniqueness ensures that no duplication happens inside the information.
- Consistency: The extent to which information is constant throughout inside techniques each day. The discrepancy is a standard information challenge that always stems from information silos or inconsistent metric calculation strategies. One other facet of the consistency challenge happens between days when information is anticipated to have a gentle development sample. Any deviation ought to increase a flag for additional investigation.
It’s value noting that every dimension could be related to a number of information checks. What’s essential is knowing the suitable software of dimensions to particular tables or metrics. Solely then, the extra checks employed, the higher.
So far, we’ve mentioned the scale of exterior views and inside views. In future information take a look at designs, it’s necessary to contemplate each the exterior and inside views. By asking the fitting inquiries to the fitting individuals, we are able to improve effectivity and scale back miscommunication.
