Spring time update, we’re running forward here at Black Oak Analytics. As regulation increases and interoperability in health care continues to be a growing concern, we’re taking a moment to look back at last year.
Testing in Hadoop MapReduce
Black Oak Analytics co-founder Dr. John R. Talburt, lead data scientist Dr. Daniel Pullen, testing framework designer Dr. Pei Wang, and technical writer Joshua Johnson all teamed up a couple of years ago to write a description of the testing process used to validate HiPER during the extensive design and development of the product which can link data across disparate data silos.
The process can be found in chapter 17 “An Automated Regression Testing Framework for a Hadoop-Based Entity Resolution System” of Analytic Methods in Systems and Software Testing.
The challenges of testing software in both Big Data and while performing entity resolution have some unique problems that must be addressed in order to validate the data. The chapter describes both the challenges and the strategies our team used to overcome them.
Specifically, the authors describe the process of how we handled the shortcomings of unit testing when taking into consideration a software designed for Hadoop MapReduce. This forced our design team to consider limitations of a distributed environment, data volume, then the additional complexity of how to measure the accuracy of entity resolution results.
At Black Oak Analytics, we have the HiPER system which we use to provide our clients with the highest possible match quality, for prices under traditional matching systems while guaranteeing a faster implementation than internal systems with results backed by experts in matching and entity resolution. HiPER is Big Data enabled and able to conquer complex match problems across multiple attributes including structured, semi-structured, and unstructured data sets.
To find out more about your company’s data and to have a discussion about our HiPER software and what it can do for you, contact us at Black Oak Analytics (info@blackoakanalytics.com) today.