As data can be extracted from different data sources, so integrate different data entities into a single data entity is very crucial and complex step for us.

This will include following steps

• Data Criteria: What data needs to be integrated
• Data Mapping: Define mapping, what data values of a data entity will be mapped with other data entity.
• Data Refining: Define data refining rules after integration, what data will not be required as there are chance of duplicated data entities, incomplete data entities or some corrupted data entities.

Data can be unstructured or in raw format so we have to define data verification steps also which will determine the accuracy of data.

This step will include following steps

• Format Verification: Verify format of values like if data has Price column that must be decimal values.
  Another example is validation of Phone Numbers.
• Unique Verification: Verify uniqueness of data like if data is related to Products then SKU code must be unique for each product.
• Data Cleaning: Data cleaning is the most important part of this step. Which includes smaller steps to exclude fake data, remove unwanted data and duplicated data.

Data cleaning is the most important part of this step. Which includes smaller steps to exclude fake data, remove unwanted data and duplicated data.
In this step, basically we will have data values mapping Meta data where we identify invalid values or short values and replace them with corresponding values.

This step will include following steps

• Invalid Values: Remove invalid values or replace invalid values to most similar values.
• Short Values: Format values into a standard value like convert “F” value of Gender column to “Female” value.
• Missing Values: Define default values for missing values and fill the gap.

Data generating is required in some cases like we have date field and wants to summarize data according to Month so we have to generate other data values (day, month & year) based on this data value so we can summarize data easily.

Basically this kind of split can be following;

• Different date fields can be added based on actual Date field.
• Start and end range can be added based on Numeric field and some other constants.
• Percentage field can be added based on Numeric field and some other constants.
• Indicator field can be added based on multiple fields like if customer is from UK and sold products are high means “IsPremium” can be added to identify those customers.

Basically we can divide joins into three steps in a repeated mode

• Select: We can select some data entities from one dataset and load into memory.
• Append: We can find related entities from other dataset, append entities with related entities.
• Write: Write combined entities into third dataset which will be called Joined Dataset.
• The main focus of this platform is Data organization and collaboration.

Summarizing data and generating new data will cover in this steps. We can give new identities for summarized entities also.