Analytics with Machine learning

How can an organization get into the data resale business game?

In order to have the capability to perform analytics using automated processes, you will be required to have the ability to perform: 

1. Descriptive analytics

2. Diagnostic analytics

3. Predictive analytics

4. Prescriptive analytics

using qualitative and quantitative methods.

Descriptive analytics - Begins with determining key performance measures of your business. Then you can employ BI tools and data-mining capabilities on these performance measures to determine past and current events. 

Diagnostic analytics - This is also called forensics analytics where you determine why good or bad things happened. This can be achieved through pattern extraction from the events identified in descriptive analytics phase.

Predictive analytics -  Here, business models are created to simulate pattern of events discovered in the diagnostics phase. These models are used to extrapolate future events. 

Prescriptive analytics - This phase optimizes the performance key measures by putting into consideration the prediction models and business goals; for example minimizing response latency or maximized public reach.

Data is not always precisely organized especially when dealing with data collected from many sources. Therefore, before beginning data transformation you have to categorize your raw data as complete data; incomplete data; or fragmented data.

• Complete data - can be personal identifiable and contacts information.

• Incomplete data - can be purchase history such as consumer data.

• Fragmented data - this can include crawled data and social data.

You can then transform the data by adding value to raw data by filling in data gaps and highlighting embedded information. This is accomplished by processing fragmented and incomplete data into a common data repository. The final product serves as foundation data for your clients to gain knowledge and make better decisions.

For the clients, the cost of acquiring a complete data set is minimized because you will have absolved the overall infrastructural cost i.e. the cost of collecting; processing; and storing data. 

Automated Data Processing.

The idea is to process data from different sources from input to storage automatically. Data is fed into the system in the different forms the source allows: i.e. bulk, streams, messaging etc.

Given a source of raw data an analyst can recommend collection of specific data elements which can be stored as a configuration. Multiple configurations are required so as to correspond with respective data sources and policies. 

Standardized data is then contextualized by matching against indexes or 3rd party services. For example a tweethandle-to-name can be matched against consumer data or an email feed from a third party.

Adding data to the repository requires verification of duplicate data and some business use cases to decide on whether to insert or update data. 

Machine learning and data processing

Where does machine learning fall into all this?

Data source description and corresponding data reformulation will require keeping memory of past events, thus the learning process. Learning can be manual, semiautomated, or fully automated. An example of manual learning is an analyst doing a study and updating the formulation/configuration. A semi-autonomous learning process can involve supervised learning of source classifications. Fully automated learning can be an evolution of the supervised learning or complete arbitrary classification of data sources through online learning. 

Data matching can also benefit from machine learning. To eliminate the network cost between matching process and indexing engines, a smart matcher can be implemented. This matcher can be created from a learning process to classify some data elements and can be loaded in memory during execution time.