We are working on data preparation activity for a Datawarehousing project to build an enterprise level data warehouse. As an intermediate layer, we are bringing all the source database information into Hadoop datalake as hive tables.
In our use case, we have around 3000+ source tables. We would like to consolidate the source data into a new schema in bigdata data lake. As part of this, we need to do the data profiling / data preparation for all the source database tables by analyzing each and every columns to map/group into the respective meaningful destination table.
Requesting suggesting from the experts on any tools (open source or licensed) to easy the process.
Thanks, Sakthivel
