# Login - Innhold

## Comments

## Transcription

Login - Innhold

Faculty of Science and Technology MASTER’S THESIS Study Program / Specialization Spring Semester, 2014 Computer Science Open / Restricted Access Writer: Piyush Duggal ………………………………………… (Writer’s signature) Faculty supervisor: Chunming Rong, (UiS) External supervisor : Yann Chagourin, (Accenture) Thesis title: SAP PA: What is the Inside Chemistry? Predicting future of Predictive Analysis Credits (ECTS): 30 Key words: Pages: Enclosure : CD Stavanger 15/06/2014 SAP, Predictive analysis, Association, Apriori, Regression, PAL, Cluster Analysis, K-Means, SOM, ABC, Scaling Front page for master thesis Faculty of Science and Technology Decision made by the Dean October 30th 2014 1 How Worth is Predictive Analysis? Predicting the Future & Exploring Inside Chemistry of SAP PA Uncovering value in the data Piyush Duggal Department of Electrical and Computer Engineering University of Stavanger E-mail: [email protected] Thesis submitted in partial fulﬁllment of the Requirements for MASTER DEGREE In Computer Science June 14, 2014 2 Acronyms BI Business Intelligence BIT Business Intelligence Tools DB Database DM Data Mining DS Data Sources DSMS Data Stream Management System DW Data Warehouse EAI Enterprise Application Integration ETL Extract Transform and Load HANA High Performance Analytic Appliance KPI Key Performance Indicator OLAP Online Analytical Processing PA Predictive Analysis PAL Predictive Analysis Library PMML Predictive Modelling Markup Language RDBMS Relational Database Management System RT Real Time SOM Self Organizing Maps SVID SAP Visual Intelligence Document SPAR SAP Predictive Analysis Archive file 3 Abstract With enormous growth in analytical data and insight about advantage of managing future brings Predictive Analysis in picture. It really has potential to be called one of efficient and competitive technologies that give an edge to business operations. The possibility to predict future market conditions and to know customers’ needs and behavior in advance is the area of interest of every organization. Other areas of interest may be maintenance prediction where we tend to predict when and where any equipment or component will break or fraud detection for insurance and banking sector companies. SAP predictive analysis tool is sum of all efforts and investments SAP has made through support of open source statistics language R, many inbuilt predictive algorithms. This tool thus supports definition, visualization, processing and deployments of predictive analysis processes in way it was never done or imagined so effectively before. There are other tools available, like SAS, infinite insight in market for quite some time now but SAP has now strategically came up with an impressive investment; a concept with Hana (in-memory database) combined with PA gives them the edge to competitors as they have a powerful selling case allowing business users to do predictive analysis on huge amounts of data, with a user-friendly tool that still, via R support, offers the possibility for expert users to develop their own algorithms Decision support systems based on predictive models are increasing in popularity as organizations collect more data than decision makers can handle manually. These predictive models can be applied to find potentially valuable patterns in the data, or to predict the outcome of some event. This report talks about PA as concept, understanding it’s necessity, the value it adds to business and how can analytical users trying to predict future of their business operations, algorithms involved, hot topics and trends, challenges & criteria of success and more. Predictive analytics thus enables having an insight of future outcomes and trends based on extracted information with a probability of outcome from existing data sets with help of models, pattern recognition, statistical algorithms. Software for PA process can be deployed on-premises for enterprise users or can be accessed via cloud and there are various solutions both proprietary & ones based on open source technologies available in market for instance Angoss, IBM Predictive Analytics, KXEN, Oracle Data Mining, SAP PA, Statistica etc. 4 Acknowledgements I would like to thank Prof. Chunming Rong & Yann Chagourin, my supervisors for their valuable advises, support and contributions in every phase of thesis work. My deepest gratitude goes to Chunming for his relentless positive suggestions and insightful comments throughout. He was always available whenever, I needed help. I would like to extend my sincere thanks to Yann Chagourin, Analytics manager with Accenture having vast experience with data mining and analytics for his support. The thesis would have never been possible without his help. I would like to thank my friends Bikash Agarwal, Tormod Lea, Raji Khushi and Marina Samohvalova Wifi for their inspirations and help. I would also like to thanks all my wonderful colleagues in Accenture for their encouraging words. Last but not the least, I would like to thank all my family and friends in Norway and India for making my thesis completing successfully. I also greatly thank God for giving me the courage and energy to finish my Master program despite my other responsibilities. Without his blessings, I will never be able to reach the achievements I have until now. Piyush Duggal University of Stavanger 5 Preface This thesis i submitted in partial fulfilment of the requirements to complete the Master of Science (M.Sc.) degree at the Department of Electrical & Computer Engineering at the University of Stavanger (UiS), Stavanger Norway. The work can be seen as a case study trying to understand and explore the inner mechanism of new tool from SAP called Predictive Analysis. Thanks to my supervisors and the books I referred to achieve the goals set at time of starting the thesis. The report contains work done from February 2014 to June 2014. This might be helpful for those thinking of switching their IT skills towards data mining and analytics. It might have a solid ground for the students and statisticians who are curious to dive in this wonderful contribution from SAP to computer world. Data visualization, prediction, exploration and analysis techniques are covered with focus on Apriori and K-means algorithms implemented in PA. Self-growing Maps can also be area of interest for those who are new to analytics. Piyush Duggal University of Stavanger 6 Table of Contents Chapter 1: Overview of Predictive Analysis ........................................... 11 1.1 Definition .......................................................................................................... 11 1.2 Potential and what value PA can bring ............................................................ 12 1.3 Five Kinds of Analysis for 5 questions .............................................................. 14 1.3.1 Time Series Analysis .................................................................................. 15 1.3.2 Classification Analysis ............................................................................... 16 1.3.3 Cluster Analysis ......................................................................................... 17 1.3.4 Association Analysis .................................................................................. 18 1.3.5 Outlier Analysis ......................................................................................... 19 1.4 Predictive Analysis as a process ....................................................................... 20 1.5 User’s Classification .......................................................................................... 22 1.6 Challenges & Criteria for Success ..................................................................... 22 Chapter 2: PA as a product from SAP .................................................... 26 2.1 Intro to SAP HANA (Based on 3rd Semester Project work)............................... 26 2.2 SAP HANA Predictive Analysis Library ............................................................... 28 2.3 R Integration ....................................................................................................... 29 2.4 Interface walkthrough of SAP Predictive Analysis as a tool ............................. 30 2.4.1 Step 1: Accessing and viewing the Data Source .......................................... 32 2.4.2 Preparing Data for Analysis ........................................................................ 34 2.4.3 Step 3: Applying Algorithms for data analysis ............................................ 36 2.4.4 Step 4: Running the model and viewing the Results................................. 40 2.4.5 Step 5: Deploying Model in Business Application ....................................... 43 Chapter 3: Predictive Analysis Applied .................................................. 46 3.1 Initial Data Exploration ....................................................................................... 46 3.1.1 Sampling .................................................................................................... 48 3.1.2 Scaling.......................................................................................................... 50 3.1.3 Binning.......................................................................................................... 52 7 3.1.4 Outliers ........................................................................................................ 55 3.2 Which Algorithm When ....................................................................................... 56 3.3. Challenges & Resolutions ................................................................................... 61 Chapter 4: Cluster & Association Analysis Explored............................... 65 4.1 Association Analysis ............................................................................................ 65 4.1.1 Applications of Association Analysis ........................................................... 66 4.1.2 Apriori Association Analysis ........................................................................ 67 4.1.3 Apriori Association Analysis in PAL ............................................................. 68 4.1.4 Strength & Weakness with Apriori Lite ....................................................... 69 4.2 Cluster Analysis ................................................................................................... 70 4.2.1 Introduction & Applications of Cluster Analysis .......................................... 70 4.2.2 ABC Analysis in PAL ...................................................................................... 71 4.2.3 K-Means Cluster Analysis in PAL .................................................................. 75 4.2.4 Silhouette ..................................................................................................... 78 4.2.5 Self-Organizing Maps ................................................................................... 80 Chapter 5: Conclusion ........................................................................... 88 5.1 Problem Set: Burn that Churn ............................................................................ 88 5.2 Results & Analysis ............................................................................................... 89 5.2.1 Clustering ......................................................................................................... 92 5.2.2 Decision Tree .................................................................................................... 94 5.2.3 Apriori............................................................................................................... 95 5.2.4 Neural Network ................................................................................................ 96 5.3 Discussion & Issues ........................................................................................... 100 5.3.1 SAP PA compared to Hadoop....................................................................... 100 5.3.2 Sharing your own R component................................................................. 101 5.3.3 Configuring HANA PAL to use with SAP PA ............................................... 101 5.4 Future Work ...................................................................................................... 102 5.5 Conclusion ......................................................................................................... 102 References .......................................................................................... 104 8 Table of Figures Figure 1: PA utilizing approaches from many disciplines ............................................................. 12 Figure 2: Competitive Advantage goes well with Analysis ........................................................... 13 Figure 3: Five main questions of PA .............................................................................................. 14 Figure 4: Historic points as base for future points plot ................................................................ 15 Figure 5: Classification Analysis .................................................................................................... 16 Figure 6: Cluster Analysis .............................................................................................................. 18 Figure 7: Association Analysis ....................................................................................................... 19 Figure 8: Two Dimensional showing Outlier……………………………………………………………………………..20 Figure 9: Steps of PA Process…………………………………………………………………………………………………….21 Figure 10: SAP HANA Internal Architecture……………………………………………………………………………….27 Figure 11: R Integration of PA…………………………………………………………………………………………………..30 Figure 12: Welcome Screen for PA…………………………………………………………………………………………….31 Figure 13: Select Input Source for PA………………………………………………………………………………………..32 Figure 14: Window to search for database………………………………………………………………………………..34 Figure 15: Merge option in Step 1……………………………………………………………………………………………..34 Figure 17: Preparing Data for Analysis ………………………………………………………………………………………35 Figure 18: Possibility to apply and configure algorithms…………………………………………………………..36 Figure 19: Configuring the attributes for algorithms………………………………………………………………….38 Figure 20: An Advanced Analysis in PA……………………………………………………………………………………..40 Figure 21: Dialogue to create a new R Component for PA…………………………………………………………40 9 Figure 22: Predict Results Grid View………………………………………………………………………………………….41 Figure 23: Cluster Parallel Coordinate Chart……………………………………………………………………………..42 Figure 24: Scoring the saved model in PA………………………………………………………………………………….44 Figure 35: Share View in PA for outputs……………………………………………………………………………………45 Figure 36: Table versus Charts…………….……………………………………………………………………………………47 Figure 37: Input & Output Systematic Sampling………………………………………………………………..………50 Figure 38: The Sample Component in PA…………………………………………………….……………………………50 Figure 39: Scaling types and their results compared…………………………………………………………………51 Figure 40: Normalization Component in PA………………………………………………………………………………53 Figure 41: Input Output tables for Binning table in PAL.……………………………………………………………54 Figure 42: Algorithm Categories with tasks and examples…..……………………………………………………57 Figure 43: Four Data Sets in Anscombe’s Quartet...................................................................…...62 Figure 44: Process of Overfitting the models……………………………………………….……………………………63 Figure 45: Examples of Multicollinearity………………..…………………………………………………………………64 Figure 46: Apriori Principle.………………………………………………………………………………………………………66 Figure 47: Parameter Table Definition for Apriori………..……………………………………………………………67 Figure 48: An Example of ABC Analysis………………………………..……………………………………………………68 Figure 49: ABC Analysis Input & Output tables…...........................................................................73 Figure 50: Parameter Table Definition for K-Means….………………………………….……………………………76 Figure 51: Decision Tree Analysis of Clusters …..…….……………….……………………………………………….80 Figure 52: Data Set Records to the Map...…………………………………………………………………………………83 Figure 53: Four Clusters in the 4 * 4 Map……………………..……………………………………………………………86 10 Chapter 1: Overview of Predictive Analysis 1.1 Definition SAP defines its predictive analysis tool as ‘SAP Predictive Analysis is a statistical analysis and data mining solution that enables you to build predictive models to discover hidden insights and relationships in your data, from which you can make predictions about future events by allowing you to perform various analyses on the data, including time series forecasting, outlier detection, trend analysis, classification analysis, segmentation analysis, and affinity analysis’. In most simple words it is quantitative analysis supporting predictions and steps involved. It is a trending term in computer science terminology but not a new topic as we in past few decades can find many prediction attempts like product sales, costs, headcount, customer churn, advertising campaign response, possible fraud etc. One can argue that it involves data mining in contrary to involving knowledge discovery, whatever be end of debate, and it can prove to be business changing methodology if skilled to best of its potential. It is essentially a process of finding meaningful correlations, patterns and trends by interpreting and analyzing over through large amounts of data stored in data repositories deploying statistical/mathematical techniques or pattern recognition concepts. Inferential statistics and statistical sampling not only enforces requirement of very large data sets for prediction but also provides possibility to analyze smaller data sets for efficient sampling of correlations among datasets. Wikipedia defines it to be an area of statistical analysis in which you extract information from data to predict patterns and trends. This can then be used to predict an unknown, be it past, present or future; for example identifying fraud that has been committed or as it is actually occurring, through to forecasting future sales. The heart of predictive analytics is finding the relationship between known variables and a predicted variable, using past occurrences. This relationship is then used to predict an unknown outcome. Naturally, in such an analysis the quality of the data analysis and the assumptions made, will greatly affect the accuracy and usability of the predictions. Predictive analysis is a blend of multi quantitative analytics disciplines and Venn diagram below may describe the contribution of these disciplines to PA. Predictive analytics enriches decision makers and analysts with the potential to make accurate predictions about future events based on complex statistical algorithms applied to data under investigation. In other words, PA is synergy of interdisciplinary methodologies and prospective and combination of useful approaches to problem solving from diff professions. Statisticians find analysis methods like inferential statistics, regression and other multivariate methods as key concepts while operational researchers prefer simulation & optimization methods contrary to 11 Chapter 1: Overview of Predictive Analysis Artificial intelligence and information extraction approach followed by data miners. No matter what approach one goes for, this will always be an analytics process which initializes with data selection, acquisition and explorations using visualizations or sampling, finding validity of results, possibly reiterate whole result set and then dissemination in end to implement improved business processes. Predictive Analytics thus now can be seen as a broader term describing a variety of statistical and analytical algorithms/techniques used in order to develop models that can predict future behaviors or events. Figure 1.1: PA utilizing approaches from many disciplines 1.2 Potential and what value PA can bring White paper titled ‘The Business Value of Predictive Analytics’ by IDC research reported an asset management firm increased its marketing offer acceptance rate by300%; an insurance company identified fraudulent claims 30 days faster than before; a bank was able to identify 50% of fraud cases within the first hour and a communications company increased customer satisfaction by 53%. During the 2009 pandemic of H1N1 influenza virus or swine flu, Google was able to leverage search term activities to predict the spread of the H1N1 disease two weeks ahead of the government’s reports. This knowledge enabled state and local healthcare to ensure the availability of medicine resources and treatment for patients. What can describe better the advantage of being powered with information of what may happen in future depending on model efficiency? Management becomes when you have an insight of future provided the 12 Chapter 1: Overview of Predictive Analysis predictions are accurate. Better and accurate analysis of future happenings better is the control over it. Figure below describes the competitive advantage we get progressing from simply reporting the past to predicting the future and clearly advantage increases considerably. SAP Predictive Analysis was launched in late 2012 as a supplement to SAP Lumira (formerly Visual Intelligence), a tool to allow users to run R, HANA PAL, and HANA-R algorithms through a userfriendly interface. It will be quiet interesting to note here the results of a survey by Ventana research titled ‘Predictive Analytics: Improving performance by making future more visible’ and the results were as below 55% use predictive analytics to create new revenue opportunities. 68% who use predictive analytics claimed a competitive edge. 86% asserted that predictive analytics will have a major positive impact. Measurement of benefit from PA is not easy to calculate as theoretically it is difference between what happened from using PA to what would have been happened if PA was not used and we don’t have value known for later but the fact that market for predictive analysis software is estimated at over 2 billion dollar can give an idea about its potential, worth and relevance to business today. Figure 1.2: Competitive Advantage goes well with Analysis Users for PA can be data scientists, data analysts or business users. Data scientists are less than 1% of an organization’s head count and generally create complex predictive models, validate 13 Chapter 1: Overview of Predictive Analysis predictive requirements and publish results to management. Data analysts contribute to around 3% of head count and assists data scientists in transforming & enriching data sources, creating simple models and visualize results to publish to BI tools. Rest 97% are direct or indirect consumers of this analysis information and collaborate with each other for further business actions. Data scientists generally encompasses traditional terms like data miner, statisticians or data researchers and have deep knowledge & expertise to build predictive models for analysis, data collection, validation, exploration, selection and finally prediction. Business users don’t have technical knowledge and just need output of analysis. 1.3 Five Kinds of Analysis for 5 questions Whatever be the business or reason to deploy Predictive Analysis in that business, technically PA tries to find answers to following 5 questions as shown in figure below. Figure 1.3: Five main questions of PA Finding trends in historical data can be used to project future data by applying times series analysis, by utilizing historical data points to see how they might continue. This can be applied to predict demands or sales forecasting. Keeping track of key influencers of an event or an outcome can prove to be worthy for churn analysis as we can try to follow purchasing trend of customers. There can also exist significant segments or groups in data which are of more interest to us, finding them can be a key for further analysis. Are there any clear groupings of data or some main influencers? PA also tries to find associations or links between products by analyzing market baskets to trigger recommendation engines and lastly what and why some anomalies exist in 14 Chapter 1: Overview of Predictive Analysis data, are they errors or actual variations to be further analyzed. Off course PA is being deployed for large set of applications in cross industry but the key questions to be investigated remain same are thoroughly used. They are actually as basic to PA as a methodology that we can even group classes of applications. Each of 5 question we discussed above correspond to one of 5 classes of Predictive Analysis helping to describe structure of data for analysis. We classify predictive analysis applications to one of following 5 classes 1.3.1 Time Series Analysis Time series analysis accounts for the fact that data points taken over time may have an internal structure such as autocorrelation, trend or seasonal variation that should be accounted for where Time Series is an ordered sequence of values of a variable at equally spaced time intervals. It helps to obtain an understanding of the underlying forces and structure that produced the observed data and it helps to predict a model and proceed to forecasting, monitoring or even feedback and feed forward control. The intent is to discern whether there is some pattern in the values collected to date, with the intention of short term forecasting. Past data points are used as basis for predicting future ones. Time Series is an ordered sequence of values of a variable at equally spaced time intervals which give an understanding of the underlying forces and structure for data under observation. This is also a major weakness because it relies on the assumption that past behavior will be repeated which may not be true always and thus should be used with caution. There is actually no real argument to say that decision trees are a better algorithm than neural networks to classify data but still very common. It will depend on the fit the data set at hand, and also on the demands of the client as decision trees are easy to understand, and can help with understanding data patterns, whereas neural networks are black boxes. Figure 1.4: Historic points as base for future points plot 15 Chapter 1: Overview of Predictive Analysis 1.3.2 Classification Analysis This is largest group of applications and tend to predict a variable using data of other variables that is believed to affect the value of variable we want to predict. Prediction variable is also called as output variable or target variable as it depends on few independent variables or input variables. Studying churn analysis or target marketing is most used result. It is one of most common data mining techniques for finding hidden patterns in data along with clustering analysis. Classification is different to clustering as it also segments customer records into distinct segments called classes but unlike to cluster approach classification analysis requires that the end-user/analyst know ahead of time how classes are defined. A common approach to classify is to use decision trees for segmenting & partitioning records when better records are obtained by traversing the tree from the root via branches and nodes, to the leaf as it is a class instance. The path takes through a decision tree is a rule, as in "Income<$30,000 and age<25, and debt=High. Due to the sequential nature of the way a decision tree splits the records, it can result in a decision tree being overly sensitive to initial splits. It is thus advisable to find error rate of each leaf node.it is easy to express as paths can be shown as rules making it possible to use measures for evaluating the usefulness of rules such as Support, Confidence and Lift to also evaluate the usefulness of the tree. We don't use these values practically to measure the quality of a decision tree model, they go more well with Apriori algorithms. On decision tree models you can just check the accuracy of the model on known past data. Figure 1.5: Classification Analysis 16 Chapter 1: Overview of Predictive Analysis 1.3.3 Cluster Analysis Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups. Cluster analysis itself is not one specific algorithm, but the general task to be solved. The greater the similarity (or homogeneity) within a group, and the greater the difference between groups, the “better” or more distinct the clustering. Cluster analysis is a classification of objects from the data, where by classification we mean a labeling of objects with class (group) labels. Cluster analysis is distinct from pattern recognition or the areas of statistics know as discriminant analysis and decision analysis, which seek to find rules for classifying objects given a set of pre-classified objects and hence can be considered an alternative to Factor Analysis. As the groups are not know in advance, it can be difficult as results don’t make sense in the context of the research being conducted. Hierarchical Clustering is efficient and groups data over a variety of scales by creating a cluster tree which not a single set of clusters, but rather a multilevel hierarchy, where clusters at one level are joined as clusters at the next level. K-Means Clustering is a partitioning method and partitions data into k mutually exclusive clusters, and returns the index of the cluster to which it has assigned each observation. Unlike hierarchical clustering, k-means clustering operates on actual observations rather than the larger set of dissimilarity measures by creates a single level of clusters. Gaussian Mixture Models form clusters by representing the probability density function of observed variables as a mixture of multivariate normal densities. Mixture models of the Gaussian mixture distribution class use an expectation maximization (EM) algorithm to fit data, which assigns posterior probabilities to each component density with respect to each observation. Clusters are assigned by selecting the component that maximizes the posterior probability and often considered as soft clustering method. It helps to understand the attributes of smaller subsets more effectively. Patterns in data or any further relationships are easy to find when we focus on these clusters and it is also possible to cluster data in a way that allows us to focus on a specific group within dataset. Cluster Analysis is actually pattern recognition without a priori knowledge of the data set. When we have groups of similar customers, based on some attributes, it can be utilized to improve the business processes. For example, if the algorithms find a cluster of high value customers, there might be idea to target those with specific campaigns. Contrary to the classification analysis where all observations are known to be a part one of a number of groups and the objective is to predict the group to which a new observation belongs, cluster analysis tries to find the number and composition of the groups. 17 Chapter 1: Overview of Predictive Analysis Figure 1.2: Cluster Analysis 1.3.4 Association Analysis Given a set of transactions, we try to find rules that will predict the occurrence of an item based on the occurrences of an item based on the occurrences of other items in the transaction. The purpose of association analysis is to find patterns in particular in business processes and to formulate suitable rules, of the sort "If a customer buys product A, that customer also buys products B and C". Thus association is a data mining function that discovers the probability of the co-occurrence of items in a collection. The relationships between co-occurring items are expressed as association rules. In transactional data, a collection of items is associated with each case. The collection could theoretically include all possible members of the collection. For example, all products could theoretically be purchased in a single market-basket transaction. However, in actuality, only a tiny subset of all possible items are present in a given transaction; the items in the market-basket represent only a small fraction of the items available for sale in the store. In transactional data, a collection of items is associated with each case. The collection could theoretically include all possible members of the collection. For example, all products could theoretically be purchased in a single market-basket transaction. However, in actuality, only a tiny subset of all possible items are present in a given transaction; the items in the market-basket represent only a small fraction of the items available for sale in the store. The associations necessarily don’t need to be products in shopping baskets, it can even be people in social network or telephone calling patterns etc. 18 Chapter 1: Overview of Predictive Analysis Figure 1.3: Association Analysis 1.3.5 Outlier Analysis An outlier is a data point which is signiﬁcantly diﬀerent from the remaining data i.e. is an observation which deviates so much from the other observations as to arouse suspicions that it was generated by a different mechanism and can also be referred to as abnormalities, deviants or anomalies. An outlier often contains useful information about abnormal characteristics of the systems and entities, which impact the data generation process. Most outlier detection algorithm output a score about the level of “outliereness” of a data point. This can be used in order to determine a ranking of the data points in terms of their outlier tendency. This is a very general form of output, which retains all the information provided by a particular algorithm, but does not provide a concise summary of the small number of data points which should be considered outliers. A second kind of output is a binary label indicating whether a data point is an outlier or not. While some algorithms may directly return binary labels, the outlier scores can also be converted into binary labels. This is typically done by imposing thresholds on outlier scores, based on their statistical distribution. A binary labeling contains less information than a scoring mechanism, but it is the ﬁnal result which is often needed for decision making in practical applications. Predictive model should be capable of differentiating between outlier caused due to errors in data or genuine variations of data. It is mostly applied in fraud detection, clinical trials, voting irregularity etc. 19 Chapter 1: Overview of Predictive Analysis Data sets with multiple outliers or clusters of outliers are subject to masking and swamping effects. Masking occurs when a cluster of outlying observations skews the mean and the covariance estimates toward it, and the resulting distance of the outlying point from the mean is small. Swamping occurs when a group of outlying instances skews the mean and the covariance estimates toward it and away from other non-outlying instances, and the resulting distance from these instances to the mean is large, making them look like outliers. This is the second main usage for outlier analysis, i.e. "improving" the quality of a data set before running other algorithms on it as algorithms that would "suffer" from the presence of outliers, like regression algorithms should not be brought in. Two Dimensional with an outlier point 1.4 Predictive Analysis as a process Like every other process, Predictive analysis also is series of defined logical steps and can be defined well with following steps. 1. Requirement Analysis – What is reason behind your prediction and what is motivation behind this attempt to predict? What outcomes are expected and who are prospective participants, timelines and resources. It is important step and requires on average around 20% of total time to come up with good analysis of requirement. A flaw here will always affect the last prediction output. 2. Data Identification - What are data requirements and what sort of data will support our prediction model the best? What are various sources available and which one can be most reliable? Validation of data after acquiring from various sources is also a good practice. Initial 20 Chapter 1: Overview of Predictive Analysis data exploration is conducted and some data transformations like sampling, binning or rescaling the data may be performed in preparation for model building. It will be interesting to note here that data selection, acquisition and preparation is most time consuming step in whole process. A lot of questions like what is needed, in what terms it should be measured, where it can be obtained from, how good resultant data set is etc. determine data to be captured and transformed. On an average this step accounts for 36% of total time spent on a PA process. 3. Model Building – Here algorithms come in picture and they are applied to identify data along with chosen parameters to find the best analysis. Training the model or testing the model on one set of data and then reapplying on another or unseen part of data is generally done to evaluate algorithm fit (how good is the algorithm at solving the problem at hand) for particular case. This also gives an idea about model performance in terms of robustness, usability and goodness. 20% time is average consumption in this step. 4. Deployment – Here comes the opportunity to apply selected models in various business applications and is sometimes also referred to as model scoring. Business rules may sometimes need to be integrated with business rules & fundamentals to get a better prospective of business context. Here we also monitor model performance over time. 5. Reiterate – Predictive analysis requires iteration to any stage of process back thus making it wrong to define as single pass through sequential well defined steps. It is because data is needed further to transform data so as to alternate the analysis for better results and provides an option to see a set of data from multi prospective scale. Figure 1.9: Steps of PA Process 21 Chapter 1: Overview of Predictive Analysis 1.5 User’s Classification We can have PA users classified in diverse categories with different skill sets and domain experience ranging from data scientists to consumer of business applications. Data Scientists form the minority group as they come under 0.01% of total users and are responsible to create predictive models, validate predictive business requirements and publish results to management. Models built by data scientists are used by data analysts through an interface like some wizard to explore and analyze data related to a particular application say marketing campaign. They generally have functional and business domain knowledge but generally don't want to get engaged with technical process of applying algorithms and creating models. They generally need guidance for understanding these models and basically are interested in output of these models and most of times are market researchers, market campaign managers or analysts. People who just want the benefits of predictive analysis simply embedded in their business processes are classified as business users. They are only interested to analyses the output of algorithms for decisions. With Service Pack 6 for HANA, SAP has now introduced the Application Function Modeler tool, that is a graphical interface for running advanced algorithms & accessing results from the Application Function Libraries and thus helping these business users be more effective. 1.6 Challenges & Criteria for Success Before we look Predictive Analysis as a product from SAP and its potential, let’s talk about some challenges of PA and myths involved. It has to be understood that PA don’t guarantee the successful & consistent prediction of future. In classical business organizations, enormous amount of data is collected without knowing where & when it will be used with an approach in mind to save everything because you never know when you need something. But for analyzing data, quality of data is more important than quantity of data. It may be interesting and efficient if we store data with a metadata defining the purpose and possible decision making it may support. Identifying the variable correctly that has biggest impact on prediction or output is most of times very difficult. Objectives of analysis should be very clear in starting if we want a successful predictive analysis project implemented. There are many myths about PA in market and are propagated by innocent or biased parties. Below fiver are most common misconceptions held about PA as per SAP published book on Predictive Analysis. 1. 2. 3. 4. 5. PA is all about algorithms. PA is all about accuracy. PA requires a data warehouse. PA is all about vast quantities of data. PA is done by predictive experts. 22 Chapter 1: Overview of Predictive Analysis First myth that PA is all about algorithms is not totally wrong as algorithms form heart of this process. Good and efficient algorithms are a part of story only. In previous section, we saw that only 20% of this PA process is devoted to generating models. Other important things that make core of PA other than algorithms are defining project goals, acquiring understanding and manipulating data, analyzing, evaluating, modeling and presenting the results. Believing this myth is something like driving a car that only has engine but no steering wheel, fuel or brakes. Everyone wants the best model and thus there are various measures to model quality but that doesn’t mean PA is always accurate. Spending continuous energy and time to refine a model in order to get very last model of precision always creeps in extra cost. It is business decision that how worth that extra efforts are to get more and more accurate analysis. A part of predictive accuracy may discover an interesting pattern in data which will prove to be very crucial in business decision, but this usefulness doesn’t depend on accuracy of model. Usefulness of PA algorithms depend on understandability & deplorability of it in business model and not fully on its accuracy. Third myth is based on assertion that we need to have a fully functional data warehouse to start with predictive analysis process. It is not true but for sure PA process will be more efficient and easy to implement if organizational data is relatively clean & easy to access. While planning to start a data warehouse requisite data for any analysis should be considered. Management reporting is basic purpose of data warehouse and not data analysis. That’s why it is a considered a myth and having no data warehouse can’t stop anyone anyhow to start predictive analysis. Its era of Big Data and thus computers memories, database sizes, performance factors etc. All are many times bigger these days; but it is definitely a myth that we cannot predict or analyze without having a vast dataset. We can't "predict" if there are no patterns in the data, whatever the amount of data. The idea with "big data" is that the more data you have, the more likely you are to see the patterns, but that pre-supposes the existence of patterns. PA is equally relevant & successful for very small volume of data. As far as statistical inference is concerned, data volumes can be small but yet of very high importance for analysis. In most of cases like churn analysis, credit risks test, loan defaulting etc., even though we have thousands of records in dataset, PA still depends on just very few key variables. It is not wrong to say data analysis of bigger dataset is increasing in popularity but for sure analysis of small data volumes may be equally beneficial and popular to next business decision. Last myth is true bit not only case and hence PA can be done by Predictive experts as well as newbies in this area. With tools like PA, the idea is that you do not need to be an expert in R, or in the inner workings of the provided algorithms, in order to build predictive models. You still need some knowledge in order to know how to use the algorithms, their strengths and weaknesses, but yes actually the most important part might be business knowledge to be able to interpret the results. It actually depends on what has to be done and complexity of this task. When prospective borrower is browsing a bank website to analyze the bank rules for passing a 23 Chapter 1: Overview of Predictive Analysis loan, he is doing his predictive analysis but definitely need not be an expert in this case. PA is best performed by someone who has relevant business domain knowledge. Soon we will be exploring how to use predictive analysis so it becomes wise to go through common pitfalls that every user should avoid to help saving him from any unapparent source of trouble. Firstly predictive analysis will make no sense by simply throwing in data without any thoughts. Rational thinking treats it illogical which makes no sense to just dump all accessible data but some data scientists prefer doing it and want to reply on some intelligent and reliable algorithm that can work out to find important variables and ignore all irrelevant ones. It is somehow related to myth 4 mentioned above that PA is all about vast quantities of data. It can be a good practice to keep dumping all data if you believe on an algorithm that can sort out noise from signal and has potential to reject some variables if algorithm thinks it to be irrelevant to analysis. However experienced business users find this approach dangerous and counterintuitive. Predictive analysis will off course be of no use if user don’t have basic to intermediate knowledge of related business domain. Without business knowledge of application area it is ideally impossible to guide predictive analysis process towards useful results and make a decision based on that results once we have them. Thus it involves teams working together with diverse skillsets from business knowledge to analysis knowledge working together otherwise neither results nor variables and dependencies will sound understandable to someone with no business knowledge when making prediction. Lack of data knowledge is another common pitfall to be avoided and thus approach should be to have detailed answers about data, data types, authenticity, source, provider, measurements, and interpretation in terms of business rules etc. need to be found out at first place. Has data come from sample or survey or was it unbiased, such questions have deep significance. Irrelevant data or lack of data knowledge can be as worse as having no data. Without data knowledge we can be misled even and tend to make erroneous invalid assumptions too. Some assumptions need to be verified twice for an instance if a customer can hold multiple accounts or if a class attendance is mandatory. In case of legacy and outsourced data, it is difficult task as even data experts need to be sure about these assumptions. In Short, sources should always be questioned and drilled before we finalize any assumption while data verification. After these pitfalls and myths it would be easy to summarize and understand this section as key factors for success for any predictive analysis process. Expectations should not be kept very high and data mining does not guarantee finding gold. It depends on your expectations to see if finding 10 when you promised 12 is a success or failure. It is always advisable to steer any predictive analysis project after agreeing the first steps of setting objectives, business case and desired outcomes and not to go other way to start process and wait for something to be found if we are lucky. Working in team is also crucial with business domain knowledge experts available at every step while data analysts supporting them and understanding their requirements. 24 Chapter 1: Overview of Predictive Analysis Sensitivity analysis is to question the impact of assumptions made and to verify result of these assumptions on analysis output. It is actually a very influencer of success. Solution can be considered unstable or model is considered unhealthy if small changes in assumptions bring large alterations in results. 25 Chapter 2: PA as a product from SAP Intro to SAP HANA (Based on 3rd Semester Project work) 2.1 Before we talk about SAP HANA & Predictive Analysis library, here is a quick introduction to SAP HANA and in-memory computing basics from report of 3rd semester work. An in-memory database system also known as main memory database system contrast traditional systems which rely on disk storage mechanism and are claimed to be faster relying on faster internal optimization algorithms which eliminates seek time. SAP HANA is a powerful platform providing libraries for predictive, planning, text processing, spatial and business analytics combining data processing, application platform and database capabilities in memory. SAP HANA is a powerful platform providing libraries for predictive, planning, text processing, spatial and business analytics combining data processing, application platform and database capabilities in memory. SAP HANA is an innovative in-memory data platform that is deployed on-premise as an appliance, in the cloud or as hybrid of two. The key lies in its unique ability to converge database and application logic within in-memory engine to perform advanced, real-time analytics. HANA stores a table in column store as sequence of columns in consecutive memory locations maximizing spatial locality of table columns. CPU execution speeds are high without need of internal waits for memory address operations. Data is compressed in two-fold making it a less costly database allowing speedy searches and calculations. Hana Database also called SAP in-memory database follows hybrid approach and consists of two relational database engines. Column bases store arranges data in columns and is optimized to hold huge amount of data, which can be aggregated in real time. Row based storage is more optimized for insert and updates and stores data in rows. To achieve the desired performance, in-memory computing follows these basic concepts: Keep data in main memory to speed up data access. Minimize data movement by leveraging the columnar storage concept, compression, and performing calculations at the database level. Divide and conquer leverage the multi-core architecture of modern processors and multiprocessor servers, or even scale out into a distributed landscape, to be able to grow beyond what can be supplied by a single server. All standard features expected from any relational database like views, triggers, indexes etc. are supported by HANA database engines. At time of table creation, administrator can select either of two options. It is always possible later to convert tables from one form to another. Both engines share common persistency layer which is responsible for page management and logging. 26 Chapter 2: PA as a product from SAP Logger saves every transaction committed on HANA database in a log entry written on persistent storage. Log volumes use low latency flash technology for storage. Modeling capabilities to define in memory transformation of analytical views from relational tables are also provided. Analytical views always provide real time results as views are never materialized. In-memory computing allows the processing of massive quantities of real time data in main memory to provide immediate results from analysis and transaction. In order to support developers in creating applications and services directly within this new SAP HANA Extended Application Services, SAP has enhanced the SAP HANA Studio to include all the necessary tools. SAP HANA Studio was already based upon Eclipse; therefore we were able to extend the Studio via an Eclipse Team Provider plug-in which sees the SAP HANA Repository as a remote source code repository similar to Git or Perforce. This way all the development resources (everything from HANA Views, SQLScript Procedures, Roles, Server Side Logic, HTML and JavaScript content, etc.) can have their entire lifecycle managed with the SAP HANA Database. These lifecycle management capabilities include versioning, language translation export/import, and software delivery/transport. SAP HANA Internal Architecture 27 Chapter 2: PA as a product from SAP 2.2 SAP HANA Predictive Analysis Library More & more are getting aware of SAP huge efforts and contributions in the area of predictive analysis ranging from SAP HANA (in-memory computing database) to modern user interface for visualizing, defining and executing the whole process efficiently. SAP is also recognized as a leader in big data predictive analysis by Forrester in their report ‘The Forrester Wave: Big Data, Predictive Analysis Solutions’ just because of innovative solution and research contributions providing business users with powerful predictive assets as data preparation, data predictive algorithms, developer tools and a workbench to execute, visualize and share analysis accelerating the business applications. SAP allows its predictive tool to support many data sources like HANA or data from SAP BO along with non-SAP solutions like Hadoop (via SAP data services), CSV or even normal excel files. The fundamentals behind these PA assets will always be the powerful predictive analysis algorithms and Predictive Analysis Library (PAL) in HANA which is C++ built in library to perform in-database data mining and in-database statistical calculations. An enterprise class solution is delivered by SAP Data Services for data integration, quality management, text analytics, data profiling and metadata management. Unstructured data sources are also supported through combination of data services. PAL contain a lot of defined predictive analysis algorithms that execute in-database to process large datasets. Point here being data is not extracted out of SAP HANA to another analysis placed somewhere and thus reducing data movement time allowing performing calculations with in HANA server and database. These algorithms are called from within HANA SQLScript procedures and are generally grouped together using following classes of applications. Listed below are all algorithms provided by PA. a. Association Analysis - Apriori - Apriori Lite b. Cluster Analysis - ABC Classification - DBSCAN - K-Means - Kohonen Self Organized Maps c. Classification Analysis - C4.5 Decision Tree Analysis - CHAID Decision Tree Analysis - K Nearest Neighbor/(KNN) - Multiple Linear Regression - Polynomial Regression - Exponential Regression - Bi-Variate Geometric Regression 28 Chapter 2: PA as a product from SAP d. e. f. g. - Bi-Variate Logarithmic Regression - Logistic Regression - Naïve Bayes Time Series Analysis - Single Exponential Smoothing - Double Exponential Smoothing - Triple Exponential Smoothing Outlier Detection - Inter-Quartile Range Test (Tukey’s Test) - Variance Test - Anomaly Detection Link Prediction - Common Neighbors - Jaccard’s Coefficient - Adamic/Adar - Katz_ Data Preparation - Sampling - Binning - Scaling - Convert Categorical to Binary Link prediction is emerging set of group of algorithms to analyze social networks finding links between entities on social networks. It will not be wrong to consider PAL to be table based because each algorithm PA supports, three tables are maintained for each algorithm, an input table which contains data for analysis, a parameter or control table containing various parameter combinations for particular algorithm and an output table for the output of the analysis. The SQLScript which calls PAL contains code which first generates specific procedure followed by definitions of table for input data, parameter settings & results and finally calls the procedure. All these procedures are defined in AFL schema which stands for Application Function Library Schema. 2.3 R Integration R, open source statistics language with around 3500 plus packages/algorithms, is one of most used predictive analysis tool approximately used by 60% of data miners. Allowing use of R from with HANA offer breadth of algorithms available for business calculations in addition to specific algorithms defined in PAL. A high level architecture for SAP predictive assets and their association with R can be shown as figure below. SAP HANA platform being core along with PAL provides flexibility to involve R. HANA studio provides development environment while the client tool SAP Predictive Analysis is used by business analysts and data scientists. R and SAP HANA 29 Chapter 2: PA as a product from SAP resides on separate servers side by side and R servers takes in data from data stored in HANA tables which is transformed by R into R vectors or R frames which is default data format used by R. SQLScript embeds within R script code which is passed over to R for R processing on R server and the results are transferred back. These results given by R server are again in data vectors format and are thus needed to be converted back to HANA table. All these transformations and transfer are performed by HANA Platform. SQLScript containing R script first calls code to initiate specific procedure and calls parameters, input & output tables before calling the procedure. No doubt, predictive analysis gets huge flexibility and comprehensiveness due to this R support & Integration with HANA. If you want to use PAL algorithms you should know SQLScript similarly like you need knowledge of R to use open source R algorithms and packages. SAP PA is a simple tool with nice user interface allowing business users to get best benefit of predictive analysis without knowledge of R or SQLScript. SAP PA capability increases to a great extent with this feature to add R algorithms. With use of R in SAP PA, data mining capabilities can be extended with many new algorithms. It also enhances further charts/visualization capabilities. Prerequisite is off course R software to be installed on host machine with necessary libraries and R algorithms. R Integration for PA 2.4 Interface walkthrough of SAP Predictive Analysis as a tool SAP Predictive Analysis (PA) in most simple terms is a tool or solution from SAP that serves like user interface which defines and executes all predictive analysis processes. These processes can be on in-database PAL in HANA or on predictive algorithms in R or even traditional data sources such as SAP BO, XLS or CSV. PA has another advantage of being fully integrated with Lumira to enhance and ease of sharing the results after data acquisition, visualization and manipulation is done. PA fully supports all analysis processes for prediction mentioned earlier in this report. This section below gives a detailed description of SAP PA as a product with 30 Chapter 2: PA as a product from SAP screenshots and functions possible explaining data preparation, applying algorithms and deployment of models. All stages starting from accessing/viewing input data to performing required data preparation and then to finally applying algorithms to analyze this data is covered in following section of report. The first screen you get when you initiate or invoke PA is called Welcome Screen. Immediately, a simple five-step getting Started guide comes up and displays possible five steps. It is really simple by design and interface too as it seems to be in text. Welcome screen also contains collection of SAMPLES to help new users learn and understand the product and get used to the tool. Connecting to a data source. Select a data source. Prepare your data. Explore the input data. Analyze your data. Click on the Predict View. Visualize the analysis results. Click on Results. Save the analysis: an optional step. WELCOME SCREEN FOR PA 31 Chapter 2: PA as a product from SAP 2.4.1 Step 1: Accessing and viewing the Data Source When we select NEW DOCUMENT button from PA welcome Screen, a new dialogue box opens up giving us possibility to SELECT A SOURCE. PA supports seven different kind of data sources and this list is populated under the NEW DATA SOURCE COLUMN while on right RECENT DATA SOURCES column list recently acquired data sources for convenience and fast access. Unique feature in this list of ‘New Data Source’ is SAP HANA ONLINE which acts as a data source helping to acquire data from SAP HANA tables, views and analysis views to perform in-database predictive analysis functions using PAL algorithms and R integration of SAP HANA. Rest all data sources in list are off course non in-database and exclude PAL algorithms & R integrating for HANA as analysis part runs outside HANA. SAP HANA Source Input Following seven different data sources appearing under New Data Source are 1. CSV file: This option gives the possibility to acquire data from comma-separated value data file and perform in-process analysis using native PA algorithms and R integration for PA. 2. Free hand SQL: This option helps to create user’s own data provider allowing manual entry of SQL values to a target data source to perform in-process analysis with help of native PA algorithms and R integration for PA. 32 Chapter 2: PA as a product from SAP 3. SAP HANA Offline: This option lets you to acquire data from SAP HANA tables, views and analysis views and allows performing in-process analysis of data using native PA algorithms and with help of integration R to PA. It is only option that allows the predictive models to be run on the HANA database, all the other options have model run locally. 4. SAP HANA Online: This option can acquire data from HANA tables, views and analysis views to perform in-database analysis using SAP HANA PAL algorithms and R Integration for SAP HANA. 5. MS Excel: Microsoft excel spreadsheet can be used as a data source and after acquiring data from an excel spreadsheet, we can perform in-process analysis using native PA algorithms with integrating R to PA. 6. Universe 3.x: This option allows you to acquire data from SAP BusinessObjects Universe which are available on X1 3.x platform and perform in-process analysis using native PA algorithms and by integrating R to PA. 7. Universe 4.x: This option allows you to acquire data from SAP BusinessObjects Universe which are available on BI 4.x platform and perform in-process analysis using native PA algorithms and by integrating R to PA. Selecting SAP HANA online populates a dialogue box asking for SAP HANA connection information and selecting desired table to fetch data as shown in figure above. Once user select HANA table and gets a successful connection to HANA, PA directs to PREPARE view where you can opt to PREDICT view by using all in-database PAL algorithms listed as shown in figure below, with other data source components in the analysis editor. Data writer and preparation components are always run in-database in HANA when you opt for SAP HANA online as your data source and it is important to note that various PAL algorithms like Apriori, K-Means, CNR Tree or Multiple linear regression are supported for R Integration in such cases. For other 6 data sources other than HANA online, for an instance CSV file, prompt for data source selection looks like as figure below where you can browse for data file located in hard disk of your machine. Once data is acquired for analysis, similar to HANA Online case PA takes you to PREPARE view from where you can opt PREDICT view and can utilize all the native PA algorithms and PA supported R algorithms from algorithms section. All supported algorithms are listed under algorithms tab and can be selected to use based on your requirement. SAP PA also allows you to combine data from two different data sets from within PREPARE view. You have two options for combining data sets, either you can merge or union data sets. MERGE functionality creates a combined table to match a key column from two given data sets while UNION appends to target dataset the selected columns of source data set based on an identifier provided matching columns in two data sets have same data type. Figure below gives you an idea of functionality to MERGE two data sets in the Prepare View of PA. Once you have acquired input data from any of 7 possible data sources, you are ready to analyze and perform initial data exploration along with preparation before applying any algorithm. 33 Chapter 2: PA as a product from SAP Window to search for input file database Merge Data in Step 1 2.4.2 Preparing Data for Analysis Once data input is finished, PA moves to prepare view where we have possibility to review data in grid format or apply to columns rich features like sorting, filtering, renaming, merging, 34 Chapter 2: PA as a product from SAP creating as geographical hierarchy, creating a time hierarchy, reformatting or converting to a different data type. Data in prepare stage can be viewed in both grid or facets display option as shown in figures below. In facets view, the data is shown by distinct value equivalent to horizontal bar chart but by value. It is useful in case we have few distinct values. Data manipulators available are similar in both views. Preparing Data for Analysis A unique and useful functionality available in Prepare View is the Visualize view with help of the available extensive chart library. After accessing the data and then exploring it using many 35 Chapter 2: PA as a product from SAP visualizations and choosing one or more of extensive chart options, we get better control of data and thus can perform further data preparation to apply PA algorithms in an effective way. 2.4.3 Step 3: Applying Algorithms for data analysis After data preparation is over, all components that can be added to create an analysis are grouped under tabs into Algorithms, Data Preparation and Data Writers are seen under the Predict view of PA. Components available to use vary depending on whether you are building an in-database analysis using HANA algorithms or alternatively adding an in-process analysis. Actual construction of analyses is similar in case of both methods even when components available vary. Building an analysis is quiet straight forward, simply select a component and then drag selected component to analysis editor workspace and you will see it getting automatically connected to component in focus. Second way is to double click desired next component instead of dragging which also make it connect to component in focus automatically. Input and output anchors also called as connection points are contained by each component and are useful to get connected to other components. Data source output always have only single output connection point. Connected data components always work in a fashion that data transmits from predecessor component to their successor component or we can say output of predecessor component in a connection acts as source of input to successor component in that connection. Structure of component is shown in figure below and has options to rename, run, delete or configure its properties. Figure below also shows different states a component can be in. ‘Not Configured’ refers to scenario when user drag a component on analysis editor workspace and it needs to be configured before analysis can be run. ‘Configured’ refers to case when all mandatory properties of components are configured and analysis can be run. ‘Success’ is displayed after successful execution of analysis and ‘Failure’ refers to case when component causes execution of analysis towards failure state. Possibility to apply and configure Algorithms 36 Chapter 2: PA as a product from SAP Now we will discuss both in-data base analysis with data sourced from HANA and algorithms based on PAL along with case of in-process analysis with data sourced from a CSV file and algorithms based on integrating R to PA using approach to building an analysis and with help of screenshots. Case 1: In-Database analysis using HANA tables and PAL We start with selecting the data in SAP HANA and then choose predict view in PA to run this analysis. Key difference and point to mention here is that in this case when data source is SAP HANA online, the data does not leave SAP HANA i.e. whole analysis is run in-database. We take an example to run analysis aiming to segment or cluster the retails stores data into similar groups based on sales turn over, profit margins, staff numbers and store size and we choose to implement HANA K-means algorithm for this analysis. Choosing component as explained in previous step is simple and it requires just to drag and connect the component in analysis editor workspace as shown in figure below. PAL on HANA Data source Next we proceed by configuring the properties of HANA K-means component as shown in figure below and we are given options to change all primary properties. We can choose what variables we want to use in the analysis e.g. Turnover, Size, margin etc. and the number of clusters that is the value of K, which in this example is taken to be 5. Clicking the Advanced properties tab in dialogue box will display rest of control parameters for this algorithm component as shown in figure below which are clearly defined for business analysts contrary to writing SQLScript. Fields marked with an asterisk are mandatory inputs. Generally default values for advanced properties 37 Chapter 2: PA as a product from SAP are displayed and can be changed, for instance, the maximum no of literation’s should be changed to some lower value than default 100 if data volumes are too large and processing time is crucial factor. After primary and advanced properties are set, analysis is ready to run. Configuring the attributes for algorithms It starts with selecting the data in SAP HANA and then choose predict view in PA to run this analysis. Key difference and point to mention here is that in this case when data source is SAP HANA online, the data does not leave SAP HANA i.e. whole analysis is run in-database. We take an example to run analysis aiming to segment or cluster the retails stores data into similar groups based on sales turn over, profit margins, staff numbers and store size and we choose to implement HANA K-means algorithm for this analysis. Choosing component as explained in previous step is simple and it requires just to drag and connect the component in analysis editor workspace as shown in figure below. Case 2: In-Process analysis using CSV file and R Integration 38 Chapter 2: PA as a product from SAP We start with selecting the data from a CSV file for this case of in-process analysis and then go to predict view. To explain this case we refer to same example of aiming to cluster or group the retail stores into similar groups based on sales, turnover, store-size etc. We similar to previous case again proceed to select the R K-Means algorithm and drag to connect the component in analysis editor workspace. Next we have options available to configure the properties of the R K-means components which are very similar to editable properties in previous case. We can select the variables for the analysis and value of K which represents the number of clusters to be created during the analysis. Going to Advanced properties in dialogue box displays rest of control parameters for R K-Means algorithm which are defined for business analysts as opposed to writing R language script. Values displayed under advanced properties are default values and can be left untouched by user. Analysis is now ready to be run as we have configured right data source with required algorithms and given desired parameters. Before we move on to next step when we describe how to run the analyses, let’s have a look to a more advanced and more realistic analysis as compared to analyses we have built in upper cases just by dragging in two components. In figure below which represents a realistic analysis scenario, Stores.csv is data source which then runs the inter-quartile range test on each variable to filter for outliers before running cluster analysis on data. Cluster analysis is then run which results writing to a database table the source data with assigned cluster numbers, while specific results for cluster one are under execution. Data as a result of cluster analysis is then analyzed with help of a decision tree when target or dependent variable is from a set of previously derived cluster number and independent variables still represent store turnover, margin, staff and size giving us an insight to find specific rules and patterns explaining why such cluster sexist. Finally the results are exported to a filtered subset. It is quiet beneficial as we have an advantage of saving our decision tree models to reapply to a new data to predict a new stores cluster assignment. Saving models offer an option to export it to another application using Predictive Modeling Markup Language (PMML) standard which is explained more lately in this report. We start with selecting the data from a CSV file for this case of in-process analysis and then go to predict view. To explain this case we refer to same example of aiming to cluster or group the retail stores into similar groups based on sales, turnover, store-size etc. We similar to previous case again proceed to select the R K-Means algorithm and drag to connect the component in analysis editor workspace. New feature added to PA is the ability to define and run R algorithm from the PA analysis editor. GUI to add R script as new component in A is provided by tool based on either R integration for PA or R integration for SAP HANA to run such scripts. Even capability to add your own custom algorithm written in C++ or JAVA is available. Figure below displays a part of wizard to write custom R script components which can later be included in any analysis. This integration features SAP PA to thousands of algorithms from R libraries. An expert R user can write new components or algorithms which another business user can then easily embed into his analysis. 39 Chapter 2: PA as a product from SAP An Advanced Analysis in PA Dialogue to create a new R Component for PA 2.4.4 Step 4: Running the model and viewing the Results 40 Chapter 2: PA as a product from SAP Running in analysis whether in in-database or in-process method is exactly same. Let’s take an example when we run in-process analysis we developed in last step using CSV file to explain the PA functionality using screenshots. We can run analysis after we generate it in two ways. Either we can run using the ‘Run till here’ option or R K-Means component or we can also invoke it from the RUN ANALYSIS icon on the analysis editor toolbar. Once analysis run is completed successfully, we can switch to predict results view for tabular or grid output along with specific charts of used algorithms and default ad hoc chart viewer for user defined visualizations. Figure below shows a new column added corresponding to each record, its assigned cluster number in the results view for first few records of our cluster analysis with input data listed to left on new column. Predict Results Grid View If you click CHARTS option, just on right of Grid button, then for used K-Means algorithm you can see cluster chart which provides four different visualizations of results of this cluster analysis for further exploration of data by user and to get a better view of analysis. Vertical bar chart compares each cluster by showing size of each cluster which can also be changed to a horizontal bar chart or pie chart. There is also generated a cluster density chart with distance chart where color code scale of dark to light for dense to sparse clusters is shown following the fact that thicker line means closer the clusters. Small-sized clusters that seem to be close can be entertained for combining while other small-sized clusters distant from other clusters may be considered as outliers. Two chars generated at bottom can be used by user to compare clusters based on any specific variable to differentiate properties of each cluster. For every algorithm available, an algorithm summary is generated when it is used in an analysis, which in this example can be shown as figure below representing output from R for this R K-means algorithm applied. This summary includes the cluster center coordinates, the within cluster sum of squares which is 41 Chapter 2: PA as a product from SAP like the squared sum of the distance between individual records in the cluster and the cluster center for all records in the cluster and finally the size of each cluster. Cluster Chart in PA Under CHARTS option, a third chart is available with name CLUSTER PARALLEL COORIDINATES CHART where each record is plotted as horizontal line connected by its value on vertical axis of each displayed variable and then colored coded based on its cluster number. As with many of visualizations in PA, we can drill down into specific data to examine it in more details. Each algorithm has a default visualization which is discussed under algorithms sections of this report. We can also use the visualize option for ad hoc user defined chart creation. In figure below, a trellis chart is created showing each variable by cluster group of comparison. Going further we have possibility to use the model to predict new data either with in PA or in an external application. Cluster Parallel Coordinate Chart 42 Chapter 2: PA as a product from SAP 2.4.5 Step 5: Deploying Model in Business Application We have several options available in SAP PA to deploy models: Scoring models in PA and exporting the results. Exporting the model as PMML. Sharing the analysis in the Share View in PA. Exporting and importing analyses between PA users. Exporting an SAP HANA PAL model from PA as a stored procedure. Most commonly used option of these possibilities is to use PA to predict new data or score the model. PA provides this functionality to save the built model after you have built this model and then when you want to make predictions using same model but with new set of data. Saved model can be run again with new data from which the target or dependent variable can be predicted. It got its name scoring models based on phrase of scoring a customer’s credit worthiness or their probability to churn. You can extend the analysis by adding the decision tree algorithms R-CNR tree to derive rules that describe why records have been assigned to specific clusters. Independent or input variables are retail store turnover, margin, staff number and shop size and target or dependent variable is cluster number. Figure below demonstrates this, possibility of an option to save model with a model name and corresponding description. Saved models are shown in a new tab, along with the Algorithms, Data Preparation and Data Writers tab once the model is saved. It makes easy to build new analysis using saved models to predict new data set. In prepare view, directly add saved model to predict along with new data set. Alternative approach can be to export saved model from current analysis and after this to create a new analysis with new data for scoring and importing saved model. Below two screenshots show the saved model tab with name and description in PA tool and also how the scoring the saved model can be done. Above section explained the procedure to export the predictions of a model for reuse and so in following section we will talk about exporting the model as PMML. PA gives you the opportunity to export a saved model as the Predictive Modell ling Markup Language (PMML) which is now seen as an industry standard for model sharing between various applications. The option to export model as PMML appears when we right click on saved model component in PA analysis editor where we can specify the output file to store the XML. Figure below is a screenshot from such an XML of PMML generated for a saved model. PMML can then easily be read by another application which describes the created model and then can exploit it to bring up predictions. This was detailed view of exporting model as PMML and now in next section we will see how to share the analysis with other users. Off course sharing the analysis is as important step as developing analysis. 43 Chapter 2: PA as a product from SAP Scoring the saved model in PA There is a share view option given in PA tool which gives us the following functionalities to use on data, chart of both of them together. We can share our charts, export our generated data set to a file, publish generated data set for an analysis view to SAP HANA, publish generated data set and charts to Stream work, and publish generated data set to SAP Lumira cloud or publishing our data set to Business Objects information space to be accessed in SAP BO Explorer. Diagram below gives an idea of how we can share the Customers.xlsx data along with associated visualizations. Following section will cover exporting and importing analyses between PA users. PMML Output for decision tree 44 Chapter 2: PA as a product from SAP To Export a model for analysis by another PA user, .spar file can be generated in a SVID document which can later be used in another SVID document once imported is successful. It can be done with a series of simple steps. From PA tool in the predict view, choose ‘Export Model’ and provide name of .spar file when prompted and save it. This saved model in .spar file can be reused by another SVID document by importing it from saved .spar file. To Import model is also as straightforward as export model. Choose ‘Import Model’ under PA tool bar in Predict view of PA tool and when asked choose the path and file of desired .spar file and click Open. This will display the model in saved model tab after importing completes. SVID stands for SAP Visual Intelligence Document where Visual Intelligence is earlier name of Lumira and was a product to store data set along with visualizations generated by users. Saving model in PA automatically makes a copy and stores it in SVID also. The purpose is to make it available to share with other users or to share on Lumira cloud from where they can be accessed directly in PA or Lumira. SPAR is an acronym for SAP Predictive Analysis Archive file and forms the proprietary format to export models created in PA. Current scenario sees it mainly for transporting model purpose only but eventually plans are to cover analysis and custom created components also in this format. It is very helpful and core of business operations when one user creates the model and other uses it or share it or modify it to his own needs. To use a saved PA model in SAP HANA, it is possible to export saved model as SAP HANA PAL model too. We can export and save model using wizard as shown in figure below after creating an in-database SAP HANA model and saving it in correct format. Exported procedure along with associated objects of tables, types, procedures appear under the selected schema in SAP HANA. Share View in PA for output 45 Chapter 3: Predictive Analysis Applied It all starts from initial data exploration i.e. historic and current data sets form basis of all predictions. Following section talks about importance and methods of initial data exploration along with data preparation for predictive analysis. Quality of any analysis in general is directly dependent on the quality of input data sets agreeing with age old concept of garbage n garbage out. It don’t seem true that pushing algorithms on any collected data will give useful predictions thus making data exploration very difficult and crucial step. Prediction depends on identifying first what data might be useful, then finding out where it might be available, analyzing it, reviewing it to understand and validate, to propose key element that effects the outcome etc. 3.1 Initial Data Exploration There are two types of data: qualitative and quantitative. Qualitative data also called as categorical data in statistics is mostly expressed by means of natural languages and not in terms of numbers. Examples include text color is red, tallest in class is Anna, male elephant etc. These categories generally are associated with some structure. Nominal categories are which has no natural ordering like race, gender, religion while ordinal variables are ones which can have their categories ordered in some way like small, medium and large. Numerical measurements expressed in terms of numbers instead of natural languages are categorized as quantitative data. Case can be some numbers which can’t be continuous or measurable like post codes and tax codes. We can differentiate on fact that such numerical data can’t be added or subtracted. Quantitative variables either are discrete such as number of students in class or continuous such as weight, height salary etc. they are more like integers. There can be further categorization like binary variables like 0 or 1, on or off etc. or date formats which look like numerical digits. In a sense, categorical, string or text variables are qualitative while numeric which can go under algebraic functions are quantitative variables. Data type is crucial and important for any analysis as non-numeric values cannot be predicted by linear regression while some decision tress need input data in a standard format. Data types should be chosen before test as most statistical test and analysis are sensitive to data types. Database types are generally reflected by data types in PAL. In PA there are string or varchar data types for qualitative variables and only integer and double for quantitative variables. ‘Date’ data type is also supported by PA giving facility to convert data types, format them to appropriate formats. There are functions available in PAL which converts data types, for an 46 Chapter 3: Predictive Analysis Applied instance, CONV2BINARYVECTOR converts categorical input data to numeric data for use of algorithms which accept only numeric data like K-Means. Second way to convert categorical variables to numeric ones is to use Formula component and clauses such as If (‘Name’ == ‘ANNA’) Then (1). Under data types, we may sometimes need to construct new variables from existing variables. A case can be finding ratio of bank transactions made on weekend compared to transactions made on weekdays may be more helpful than using those two values independently. Missing value also forms a very considerable data type when we talk about predictive analysis. There can be lot of reasons behind getting missing values in a data set like mistakes, reluctance to provide confidential information or simply unavailable values. We can approach these missing values with different ways, either ignoring them or substituting them with some values based on similar records or can be interpolated from within set of known data values. Data type is crucial and important for any analysis as non-numeric values cannot be predicted by linear regression while some decision tress need input data in a standard format. Everyone will believe when you say that data is understood better when represented in a visual form rather than through lists or tables of data. Following figure proves the power of visualizing data makes understanding easy over tables or lists. Most frequently call pairing and group of callers can clearly be analyzed with figure on right compared to same data represented in table for call traffic. Credit to invent idea of line charts, bar charts, pie charts etc. goes to William Playfair who published first version of these charts in 1786. Once data is explored and visualized, we come to next stage of data preparation for predictive analysis: sampling, scaling and binning in following section. Table versus Chart 47 Chapter 3: Predictive Analysis Applied 3.1.1 Sampling Sampling refers to process of creating subsets of all of the date in order to produce inferences about al of given data. All data is referred to as the population by statisticians and hence we also refer this process as sampling from the population and is pushed where efforts to access and interpret all data is too high. It also helps when some of data goes missing for any reason. Generally speaking sampling is done mainly to explore data initially with different analysis points before choosing one and focusing in detail, as it gives us an idea about which data is actually unnecessary for the specific analysis. ‘Sample is representative of all data.’ Sampling when used in predictive analysis technique is called as cross-validation and we can also sample because the sheer amount of data makes running/training models on the complete data set too time consuming. Simply it involves creating of subset of data to build a model and then use the left over data to test this model; that implies a condition that test data has to be removed from initial creation of model so that we can compare later to see how good our analysis and prediction is. Most of times, model works poorly on new data as contrary to given data thus we need model over fitting which means to adjust an excellent model using trading data and poor model for the test data. Overfitting a model is the issue one could have by getting a predictive model that is too good on the data set it was trained upon. That could render it useless on other samples of the data. For example you have a 100 000 rows data set, take a sample of 10000 to train a model, and get a predicted accuracy of 97%. You think it's pretty good, but when you run the model on another (control) sample, you get an accuracy of below 50% => your model was over fit. Sampling is easy to use with simple train before test splitting of data, called as holdout, through to k-fold cross-validation where data set is composed of k subsets and hold method is called k times in iteration. All k-1 subsets are amalgamated to form training set every time one of k subsets is used as test set which helps to compute average across all k trials. A different approach to this sampling can be to randomly distribute data into attest and training each set k times. In PAL, there are eight sampling methods, invoked from function to sample data. It is thus easy but you still have several available methods with the underlying issue when you're not looking at the whole data set. What you gain in processing times, you lose in data coverage. 1. First N 2. Middle N 3. Last N 4. Every Nth 5. Simple random with replacement 6. Simple random without replacement 7. Systematic sampling 8. Stratified sampling 48 Chapter 3: Predictive Analysis Applied Simple random with replacement method initiates random sampling of N or N% of all records with replacement i.e. for potential further section selected record is allowed to come back and join all left data. Simple random without replacement also works same as smile random with replacement but a select record here cannot be returned back to all data for further selection. In systematic sampling, little structure is required for random sampling. It is also called as interval sampling sometimes. We first determine sampling interval (k) and then divide number of units in the population by this k to create samples. For an instance, selecting from a population of 600 a sample of 50, we would require a sampling interval of 600 ÷ 50 = 12. Therefore, k will be set to 12, which allows you to choose one record from every 12 records to total as 100 records in the sample. Given a random number between one and k, which would be the first number included in your sample is referred to as random start. The 25 records that comprise population or data which we will use in this example to select samples are shown below in figure. Code below represents main elements of PAL Script for systematic sampling while full code is available in the file SAP_HANA_PAL_SAMPLING_SYSTEMATIC_Example_SQLScript on the SAP PRESS website. This sampling is based on method 6 of eight methods mentioned above. When we select a sampling size i.e. k as 5 output is sown in figure below with 5 records showing an increment of 5 as sampling interval when starting value was chosen randomly as 2. // The procedure generator Call SYSTEM.afl_wrapper_generator ('SAMPLING_TEST','AFLPAL','SAMPLING',PDATA); // The Control Table parameters INSERT INTO #CONTROL_TAB VALUES ('SAMPLING_METHOD',6,null,null); INSERT INTO #CONTROL_TAB VALUES ('SAMPLING_SIZE',5,null,null); //Assume the data as in figure below and calling the procedure CALL SAMPLING_TEST(DATA_TAB, "#CONTROL_TAB", RESULT_TAB) WITH OVERVIEW; SELECT * FROM RESULT_TBL; There is one more sampling method available in PAL called Stratified sampling which targets to find out those attributes that may divide up a data set or population in subpopulations or strata. This should be in a way that selected sample from population can still be considered as representative of population. Stratified sampling takes samples from each stratum of population. Requirement while sampling population in such case is that the proportion of each stratum in sample be same as in population. These takes are applied more when population is heterogeneous and dissimilar but still gives a possibility to find homogeneous subpopulations. On other hand, when data is homogeneous, simple random sampling methods are more appropriate. 49 Chapter 3: Predictive Analysis Applied Input & Output Systematic Sampling PA allows us to sample data both for in-database in SAP HANA or on non-SAP HANA data sources, with sample component option available in Data Preparation tab of Predict view. The Sample Component in PA 3.1.2 Scaling Scaling of data is done before we run any of predictive algorithms with intention to be sure that every variable in the model gets equal weight and emphasis as an input to the model which needs 50 Chapter 3: Predictive Analysis Applied to define a common data scale for all these variables. For instance, we can scale all the data used as input to be classified within a selected range, such as, -2.5 to 2.5, or 0.4 to 2.9. Another scaling approach close to normalization or standardization the data use a z-score and a variable is rescaled to have a mean of zero and a standard deviation of one. Calculation is thus in case taken as the average of the variable subtracted from the value for each record, giving in result the mean of the standardized variable of zero, divided by the standard deviation, which results in a standard deviation of one. In simple words a value of 2 indicates that the value for that record is two standard deviation above the mean, while a value of -3 indicates that a record has a value three standard deviations less than the mean. Scaling or normalization plays a vital role to classify algorithms which involve neural networks, or distance measurements such as nearest neighbor classification and clustering, where independently scaled data, for an instance, few entities in millions and some in tens or hundreds, can influence the analysis to a considerable value and hence a need for a common scale among the numeric variables arises. The PAL supports three methods, and these function can be called using SCALINGRANGE: EE Min-max normalization. EE Z-Score normalization EE Normalization by decimal scaling Let’s take an example of scaling assuming we have data table DATA_TAB in HANA as shown in figure below. To scale this data in range A to B, formula we use is (B – A) * (Xi – Min Xi) / (Max Xi – Min Xi) + A which when considering A as 0 and B as 1 simplifies to (Xi – Min Xi) / (Max Xi – Min Xi). Procedure to scale can be called as with the following PAL SQLScript for scaling. // Calling the procedure CALL SCALINGRANGE_TEST (DATA_TAB,"#CONTROL_TAB", RESULT_TAB) with overview; SELECT * FROM RESULT_TAB; Scaling types and their results compared 51 Chapter 3: Predictive Analysis Applied Figure above shows us the result when we scale that data from 0 to 1 by setting the parameter NEW_MAX to 1 and NEW_MIN to 0 i.e. maximum value 1 and minimum value 0. Other method to scale in PAL is z-score normalization and can be called by setting scaling method control parameter to value 1, and then choosing one of 3 possibilities which are ‘Mean and standard deviation’, ‘Mean and mean absolute deviation’, ‘Median and median absolute deviation’. Consider Z_Score_Method as Zero with same input table as in previous example results are shown in figure above where scaling of data is on mean zero with standard deviation of 1. We can scale with Normalization component in PA also with an interface available in Data Preparation tab under Predict view. Scaling can be done on both HANA and non-SAP HANA data sources. Below figure gives an overview of Normalization component is PA using inbuilt function for normalization. Normalization Component in PA 3.1.3 Binning Binning of data is used to summarize or group for better visualizations when data volume to be analyzed is large. To construct a histogram for an instance is not easy without data binning. To visualize huge amount of data points need data binning first which can further ask for subsequent interactive drill. Generally data binning is done prior to run any predictive algorithm as an 52 Chapter 3: Predictive Analysis Applied attempt to reduce complexity of model. Complex models are no one’s cup tea as they are difficult to understand and thus aim is to achieve concept of parsimony i.e. simplest model with very few variants. Binning of numeric data is also called discretization of continuous data and it is important to have it done effectively otherwise it will lead to complex models. You can imagine a situation trying to construct decision trees based on variables with huge set of numbers where each branch of tree is considering every number thus making a complex decision tree difficult to implement. Before discussing binning functions available in PAL, it is good to know that PAL also allows three methods of smoothing. Noise is a random error or variance in a measured variable. Given a numerical attribute such as, say, age, how can we “smooth” out the data to remove the noise? Data Smoothing uses an algorithm to remove noise from a data set thus facilitating important patterns in dataset to stand out. Random, random walk, moving average, simple exponential, linear exponential and seasonal exponential smoothing are some common ways in data mining for smoothing Firstly smoothing by bin where mean value of bin replaces all other values in bin, secondly smoothing by bin medians where bin median replaces all other values in bin and lastly smoothing by bin boundaries where closest boundary value replace each bin value and represents minimum or maximum values in given bin. It is also trying to find threshold on continuous variables. For example a person's income might have an effect on where the person is going on vacation and maybe the threshold is on 500K a year. Those with less stay in Norway, those with more go abroad etc and if you use a continuous variable in a decision tree the threshold will be difficult to spot, but if you bin that data in "less than 500k" and "more than 500k" then it will be easier. The problem is that often we would not be knowing beforehand what binning strategy should be applied i.e. how many groups, and with what rules? PAL supports following four methods to achieve binning. Equal widths based on the number of bins Equal widths based on the bin width Equal number of records per bin Mean/Standard Deviation bin boundaries Let’s try to implement an example of this binning process on table DATA_TAB as shown in figure below. Main elements of the SQLScript are shown in code below while full code is available in file SAP_HANA_PAL_BINNING_Example_SQLScript on the SAP PRESS web site. 53 Chapter 3: Predictive Analysis Applied Input Output tables for Binning table in PAL // The procedure generator Call SYSTEM.afl_wrapper_generator('BINNING_TEST','AFLPAL','BINNING',PD ATA); // The Control Table parameters INSERT INTO #CONTROL_TAB VALUES ('BINNING_METHOD',0,null,null); INSERT INTO #CONTROL_TAB VALUES ('SMOOTH_METHOD',0,null,null); INSERT INTO #CONTROL_TAB VALUES ('BIN_NUMBER',4,null,null); //Assume the data as shown in table DATA_TAB from input table in figure below //Calling the procedure CALL BINNING_TEST(DATA_TAB, "#CONTROL_TAB", RESULT_TAB) with overview; SELECT * FROM RESULT_TAB; In this example of binning method, numbering starts from zero of equal widths depending on number of bins here set to 4 and smoothing is done using smoothing by bin mean. The bin widths calculated as (max – min) / k, which in this case becomes (38 – 6) / 4 equals 8, so the bin ranges are >=6 to <14; >=14 to <22; >=22 to <30; >=30 to <=38 which makes first bin getting the values 6, 12, 13 and 10, which has a mean of 10.25. Same way we can have second bin containing the value of 15 only while the third bin possesses the values 23, 24 and 25, with a mean value of 24.The last bin fourth one, contains the values 30, 32 and 38and thus man value of 33.33. Results are shown in figure up. We can choose binning method based on which one of us appeals most in that particular scenario, most of times found by hit & trials. Best approach can be to try all binning and smoothing methods to see individual impact of each on model. Binning method is not important when model gets robust to changes with binning method. It is always practical to analyze the reason behind variations by looking data in detail if model solution does vary significantly. 54 Chapter 3: Predictive Analysis Applied 3.1.4 Outliers Outliers in data can always influence any algorithm’s performance, model’s parameters and confidence in predictions to a significant level thus a crucial practice as a part of initial data exploration is always to check existence of outliers or unusual values in given data set to understand the cause and decide next action items concerning them. Some algorithms are more sensible to outliers than others. An outlier is an observation which deviates so much from the other observations as to arouse suspicions that it was generated by a diﬀerent mechanism. Outliers can easily be visualized in scatter plots, although difficult to scale for large data volumes. Box plot as show in in figure is most popular used visualization approach for outlier detections. Box plots are thus an excellent tool for conveying location and variation information in data sets, particularly for detecting and illustrating location and variation changes between different groups of data. Every value is specified in the box with upper and lower quartiles on y axis scale, induced white line in the box representing median value. Fences on top and bottom of box signifies a factor time’s interquartile range. A single box plot can be drawn for one batch of data with no distinct groups. Alternatively, multiple box plots can be drawn together to compare multiple data sets or to compare groups in a single data set. For a single box plot, the width of the box is arbitrary. For multiple box plots, the width of the box plot can be set proportional to the number of points in the given group or sample (some software implementations of the box plot simply set all the boxes to the same width). All the dots plotted outside these fences represent the outliers. Data volumes and dimensions affect the outlier detection and visualization. PAL offers specific algorithms for outlier detection namely variance Test, the InterQuartile Range Test, the K Nearest Neighbor Outlier Test, and Anomaly Detection using Cluster Analysis. The Inter-Quartile Range test is a simple and popular test for outlier detection and is the basis of the very useful box plot. It is also a robust test in that the outliers do not themselves affect the statistics of the test, as opposed to the Variance test, where outliers clearly affect the limits given that they are measured in terms of standard deviations. That is the weakness of the Variance test, but, again, its simplicity makes it popular. K Nearest Neighbors looks for local outliers, as opposed to global outliers, which is very useful as these are often harder to find because they are not so obvious. The weakness of the test is that the value of K may affect the solution, but this can be minimized by exploring the solution using several values of K. The other weakness is that by specifying the number of outliers, you ensure that you get that number, and some may not really be outliers. 55 Chapter 3: Predictive Analysis Applied 3.2 Which Algorithm When In Chapter 2 of this report, we mentioned vast number of algorithms provided by PAL but an interesting question is out of all many algorithms available, which one should be used when? It seems to be a whole big task for new users to decide which algorithm will should be used by them to get the result & analysis they want. The task become worse if we consider 3500 plus packages or algorithms contained in R. Below section discusses the criteria and main factors to consider before we select the right algorithm. Also we will talk here about accuracy factor of an algorithm and trying to summarize with general set of rules to select efficient and right algorithm. Basic questions that drive the decision of selecting an algorithm are What is purpose of algorithm and what you want to see as an analysis result? For example: group the data, look for associations in the data, or predict a series of data values. What data do you have and what are the attributes of that data? For example: numeric, categorical, Boolean, etc. The answers to these questions help you finding the best algorithm to apply as you have a better idea which algorithm fits best in your purpose. Below table lists some common tasks with corresponding algorithm category and example algorithms as mentioned in SAP Predictive analysis book which makes it to pick algorithm i.e. if we want to look for unusual values outliers we can use variance test and inter-quartile range test. To build a predictive model on a variable using data of second variable for model building we can use decision trees or neural networks and regression models. Thinking about second main factor about kind of data and its attributes is also crucial as some algorithms work only on numeric data while others on categorical data while others can be modified to support both of data types. Off course the table above can help but still we will try to classify algorithms based on five main classes of application in PA. Algorithms in PA can broadly be classified in following 5 groups and this classification gives a better understanding of purpose and use of algorithms. 1. Association analysis trying to find for associations or affinities in the data. 2. Segmentation or cluster analysis, trying to segment or group the data into similar clusters. 3. Classification analysis trying to classify or predict new data based on a model built by an algorithm. It is the largest group of algorithms in PA to predict a variable using the data of other variables that are believed to affect the values of the variable that we are trying to predict 4. Time-series analysis trying to use data with an inherent periodicity to predict values for future time periods. 5. Outlier analysis trying to find unusual values in the data. 56 Chapter 3: Predictive Analysis Applied Task Example Algorithms Algorithm Category Example Algorithms Summary statistics Descriptive statistics Mean, median, variance… Outlier detection Statistical tests Variance test, IQR test, anomaly detection… Preparation of the data for analysis Data preparation Sampling, scaling, binning… Statistical inference Sampling theory T tests, F tests, ANOVA… Relationships, cause and effect Correlation and regression Multiple linear regression, non-linear regression… Clustering or grouping data Cluster analysis ABC Analysis, K-Means, Kohonen SOMs… Time series forecasting Time series analysis Exponential smoothing, regression… Association or affinity analysis Association analysis Apriori Prediction, model building Classification analysis Decision trees, neural networks, regression… Social network analysis Network analysis Jaccard’s coefficient, common neighbors… Optimization Optimization Linear and non-linear programming Risk analysis, modelling Simulation Monte Carlo analysis Algorithm Categories with tasks and examples In Association analysis, most common and powerful algorithm is Apriori which is discussed in detail later in this chapter. In second group of segmentation most popular one is KMeans algorithm and it is known for simplicity and positive correlation. Classification analysis has largest group of algorithms indicating its importance in PA and are further sub classified in 3 groups: regression algorithms, decision trees algorithms and neural network algorithms. Regression algorithms is essentially fitting of a model either linear or non-linear, of the form Y is a function of X1, X2…XN, where Y is the dependent variable and Xi are the independent variables, which minimizes the difference between the fitted data and the actual data. Bivariate linear and non-linear, multiple linear, polynomial & logistic regression are main regression algorithms. Decision trees recursively part the data, initializing with the most divisive split of the input variable values compared to the target variable, and keep doing same till any of many stopping criteria is met. Result then defines the relationships between input & target variables. 57 Chapter 3: Predictive Analysis Applied Class of Problem and Algorithm Group Association Input or Independent Variables Categorical Output or Target or Dependent Variable Algorithms Categorical : Association rules with support, confidence and lift Apriori, Apriori Lite Cluster Numeric NA : Cluster groupings, cluster quality K-Means, Analysis, SOMs Classify - Regression Numeric Numeric : Best fit regression equation Multiple Linear & Non-Linear Regression Classify- Regression Numeric/Categoric al Numeric/Categorical : Best fit logistic curve, probabilities of outcomes Logistic Regression Classify -Decision Trees Numeric/Categoric al Numeric/Categorical : Decision tree and rules with confidence level C4.5, CHAID Classify Networks Numeric/Categoric al Numeric/Categorical : Black box model for prediction Neural Network Numeric Numeric/Categorical : Classification of new data K Nearest Neighbor Numeric : Best fit and projected values Exponential Smoothing, Regression NA : Detected outliers IQR, Variance Test, Anomaly Detection -Neural Classify -Other Time Series Analysis Outlier Detection Numeric Numeric ABC Kohonen Neural network algorithms are closer to the way human brain processes information. Two neural network algorithms, sourced from R namely Monmlp package and Nnet package are supported by PAL. Functionally they work by simulating huge number of interconnected simple processing units which are arranged in layers; input, hidden and output layer attached with varying connections strengths or weights. The network adapts by analyzing individual records, to give prediction for each record, and adjusting the value of weights whenever it sees an incorrect prediction. K-nearest neighbor is final sub category under classification algorithms which predicts or classifies objects based on their similarity or closeness to other objects with prediction calculated as average classification. Time series algorithms are significant as business applications need advantage of time series forecasting. Data is generally constant, trending or seasonal and thus smoothing goes hand in hand here. Outlier analysis algorithms come under last group and seek unusual values. Best known algorithm here is Inter-Quartile range test which is also ground of Box Plot. Variance test is also commonly used following the simple concept that unusual data 58 Chapter 3: Predictive Analysis Applied is distant from average of data. With this much knowledge, we can start applying algorithms on hit & trial basis to find out best suitable for our purpose but still it is advised to simply try all algorithms under same group to see which provides best fit for your analysis. Table below summarize what we just discussed. To check and analyze which algorithm is working best with our problem set, easy and logical approach is to apply and run all algorithms on input data and choose the best one but what factors will decide what is best and how to measure it? Answer to this question will be different for each group as we can’t compare two algorithms in different groups. For association analysis, the choice of algorithms is between Apriori and Apriori Lite. Apriori lite being a subset of the Apriori, is restricted to find single pre and post rules. The choice thus depends on rule requirements and performance, as Apriori Lite will be faster than the generic Apriori but is restricted in terms of the rules extracted from the data. For cluster analysis, finding better algorithm is difficult as for example in ABC Analysis, different values of A, B or C can’t be judged good or worse. User can only find the best value and thus no model is best. The K-Means algorithm may be poorer to analyze cluster better than Kohonen Self Organizing Maps (Kohonen SOM) but is easier to understand & flexible while Kohonen SOMs lack functionality to determine the number of clusters in advance. It is thus logical to try both K-Means and Kohonen SOMs, with varying cluster numbers, to explore the solutions in order to decide which is the most appropriate for the application. For time series analysis, numeric predictions as in classification analysis assume to have same measure of model quality, except that the analysis is done based on time periods. The Variance test and Inter-Quartile Range (IQR) test help to find overall outliers in the data set for the outlier tests. The Variance test is trivial and well-known but the outliers themselves influence the analysis. So an algorithm using the median and quartiles is more popular IQR test, measuring in a way that an identified outlier is not affected by the actual outliers anyhow. Local outliers in the data set can easily be found by the Anomaly Detection algorithm. The Variance test and Inter-Quartile Range (IQR) test help to find overall outliers in the data set for the outlier tests. The Variance test is trivial and well-known but the outliers themselves influence the analysis. So an algorithm using the median and quartiles is more popular IQR test, measuring in a way that an identified outlier is not affected by the actual outliers anyhow. Local outliers in the data set can easily be found by the Anomaly Detection algorithm. For classification models, it is more logical to compare algorithms based on their input being numerical or categorical. For numeric predictions, the residual error which is the sum of square of actual minus fitted for each data point is the most common measure along with another common way of presenting as MSE that is Mean Square Root. MSE scales back to original data RMSE on associations. Statistical measure of goodness of fit like R Squared, analysis of variance 59 Chapter 3: Predictive Analysis Applied or F value are used to produce numeric predictions for regression analysis. Classifier confusion matrices which are how often each category correctly predicts and how often incorrectly, generally evaluate categorical prediction. Model quality measure can then be made based on these matrices and measures of quality like Sensitivity or true Positive rate and Specify or True Negative rate. Gain and lift charts are plotted to compare model and algorithm performance in case of binary classification models. Based on this discussion in section above, we can have following rules as a base to start o compare algorithms their performance and applications based on user requirement as follows. If objective is to find associations in the data, Use Apriori. If all multiple item associations are required. Use Apriori Lite if only single pre and post item rules are required Use Apriori Lite sampling if performance with Apriori is too slow If objective is to find clusters or segments in the data, Use ABC Analysis if the cluster sizes are user defined. Use K-Means if the desired number of clusters is known. Use Kohonen SOMs if the number of clusters is unknown. If objective is to find outliers or unusual values, Use the Variance and IQR tests if you are looking for global outliers. Use IQR test if there are significant outliers. Use Anomaly Detection If you are looking for local outliers. If objective is to classify data when the target variable is numeric and only one independent numeric variable exists, Use Bivariate Linear Regression if a linear relationship is considered. Use Bivariate Exponential or Geometric or Natural Logarithmic Regression if a non-linear relationship is considered. Use Multiple Linear and Non-Linear Regression for linear and non-linear models respectively if more than one independent numeric variable is considered. If objective is to classify data when the variables are categorical or a mixture of categorical and numeric, Use either C4.5 or CHAID or CNR and choose the best fit if Output of decision tree rules is desired. Use Logistic Regression when preference is to find output of the probability. 60 Chapter 3: Predictive Analysis Applied Use Neural Networks and choose the best fit if model quality is of primary concern. If objective is to predict time series data, Use Single Exponential Smoothing when the data is Constant or stationary. Use Double Exponential Smoothing when the data is trending. Use Triple Exponential Smoothing when the data is seasonal. Use K Nearest Neighbor when objective is to classify data with simple easy to understand approach and the target variable is numeric or categorical in case of numeric input variables. 3.3. Challenges & Resolutions This section will discuss and elaborate following four common difficulties faced in any predictive analysis process. 1. 2. 3. 4. Cause & effect Lies, dammed lies & statistics Model Over fitting Correlation between independent variables We cannot always conclude a relation to be a cause & effect relationship if we find out a good mathematical relationship between variables as it is not always easy to interpret every mathematical relationship as cause & effect relationships. If we plot a graph between numbers of jobs in market to number of cars newly bought in city, we can see a mathematical relationship but we cannot summarize it that with every car sold in market there is a new job created. We can understand second challenge of lies & statistics by an example which covers dangers of looking only at statistical measures and is known as Anscombe’s quartet. He used 4 datasets having simple similar statistical properties to prove the importance of plotting data before analyzing it and impact of outliers as they all appear to be very different from one another when plotted. First dataset appears as a well-behaved dataset having clean and well-fitting linear model and can be plotted using y = 3 + 0.5x having mean of X as 9 and mean of Y as 10. Second dataset does not have a linear correlation strangely has same equation y = 3 + 0.5x but with R squared value of 0.67. Third dataset does have linear relation but the linear regression is thrown off by an outlier which means if the outliers were spotted and removed before plotting it would have been easy to fit a correct linear model. Last dataset does not fit any kind of linear model but the single outlier makes keeps the alarm from going off. This implies that it is wise to understand data before applying any algorithm. Graphs and 4 data sets used are shown in figure below. 61 Chapter 3: Predictive Analysis Applied I x II y x III y x IV y x y 10.0 8.04 10.0 9.14 10.0 7.46 8.0 6.58 8.0 6.95 8.0 8.14 8.0 6.77 8.0 5.76 13.0 7.58 13.0 8.74 13.0 12.74 8.0 7.71 9.0 8.81 9.0 8.77 9.0 7.11 8.0 8.84 11.0 8.33 11.0 9.26 11.0 7.81 8.0 8.47 14.0 9.96 14.0 8.10 14.0 8.84 8.0 7.04 6.0 7.24 6.0 6.13 6.0 6.08 8.0 5.25 4.0 4.26 4.0 3.10 4.0 5.39 19.0 12.50 Four Data Sets in Anscombe’s Quartet 12.0 10.84 12.0 9.13 12.0 8.15 8.0 5.56 7.0 4.82 7.0 7.26 7.0 6.42 8.0 7.91 5.0 5.68 5.0 4.74 5.0 5.73 8.0 6.89 Overfitting signifies a condition when data under analysis fits a model “too good” that it can be thought to describing your sample nearly perfectly and is too rigid to fit any other sample. This condition thus makes it loose enough to serve our predictive needs by fitting badly on new data. Over fit specifically needs to be watched when you’ve got small sample sizes or your data is too small & limited in some way and defining as phenomenon where the predictive model may well describe the relationship between predictors only but may fail to provide valid predictions in new data. It is generally due to high expectations and need for accuracy requiring an extra good job to fit the sample data by introducing too many input variables. Most of times it is case when a model has to many data points compared to number of data points. Including test data and analyzing it from every angle set is crucial when building a predictive model to have it more accurate and stable over time. Figure below explains the context with help of two figures representing two graphs based on same data points. Left graph off course is doing a decent job 62 Chapter 3: Predictive Analysis Applied as it captures general nature & characteristic of the relationship between the X and Y variables. While right hand side graphs is clearly attempting too hard to capture every subtle change in the relationship between the two variables; It makes model on left outperforming model on right when new data points are fed into the model as the right hand side model will not be able to generalize well the data it has not seen before. To avoid Overfitting, words of advice is to use a proportion and balance of the available data to train the model and the rest of the data which is unseen or hold out data to test the model. This is a key methodology in PA and definitely an important one in classification analysis and time series analysis. Process of Overfitting the models ‘Multicollinearity’ is problem & comes in picture when you’re trying to fit a regression model or other linear model. It indicates a case of predictors correlated with other predictors in the model. Unfortunately, the effects of Multicollinearity can feel unsure and intangible, which makes it unclear about how to fix if you are able to decide that it should be fixed. Statisticians define multicollinearity as a strong correlation between two or more independent variables. It is quite difficult to remove effects on dependent variables because of linear relation making model easily assuming the existence of multicollinearity in dataset. Estimates made on parameter may alter significantly in response to small changes in the model or the data which means Multicollinearity effects the calculations regarding individual predictors without minimizing the predictive power or reliability of the model as a whole specially at least within the sample data itself indicating that a multiple regression model with correlated predictors can definitely show you the degree of relation between bundle of predictors predicts the outcome variable, but will not produce always a valid results about any individual predictor and about extent of redundancy of predictors with regards to each other. Multicollinearity to an extent is normal but if it has higher value it becomes a problem because i the variance of the coefficient estimates increase which make the estimates very sensitive to minor changes in the model. Following can be seen as main sources of multicollinearity; method used for data collection, constraints pushed in the population, Model specification or an over fitted over defined model. Removing multicollinearity 63 Chapter 3: Predictive Analysis Applied fully is not possible but can be reduced by several remedial measures such as collecting the additional data or new data, re-specification of the model, ridge regression or by using data reduction technique like principal component analysis. Examples of Multicollinearity Figure above shows two graphs X1 & X2 that are highly positively correlated and value of correlation coefficient between them is 0. 9771 as computed by data on left. Trying to find a model now that describe the relationship between Y and independent variables X1 & X2 is difficult because we can merely differentiate because of them being so close and hence higher value of multicollinearity becomes a problem because i the variance of the coefficient estimates increase which make the estimates very sensitive to minor changes in the model. Mitigation, adding more data sampling gives an advantage can’t solve it completely. Omitting one of correlated variables can be another interesting approach if you can decide which variable to ignore risking the danger of ignoring real casual variable. 64 Chapter 4: Cluster & Association Analysis Explored Although all groups of algorithms supported and analysis techniques used by PA are crucial and have their own importance. Keeping time limit in mind, I thought to go through in detail of only two classes of Analysis techniques instead of covering all briefly. It was quiet interesting to go through code behind and implementation of these algorithms so neatly on SAP PA interface. 4.1 Association Analysis As name suggests, Association Analysis looks for associations between objects and also known as affinity analysis. Output of this analysis is generally in form of rules like ‘if item A is purchased by customer A, he has a very high probability to purchase item B and item C’, ‘75% of those who buy comics on-line also buy music on-line’, ‘60% of those who have high blood pressure and are overweight have high levels of cholesterol’ etc. Following the original definition by Agrawal the problem of association rule mining is defined as: Let I = {i1, i2, ..., in} be a set of n binary attributes called items. Let D = {t1, t2... tn} be a set of transactions called the database. Each transaction in D has a unique transaction ID and contains a subset of the items in I. A rule is defined as an implication of the form X→Y where X, Y ⊆ I and ∩ = ∅. The sets of items (for short item sets) X and Y are called antecedent (left-handside or LHS) and consequent (right-hand-side or RHS) of the rule respectively. Quality of these rules can be calculated by finding the number of cases when this rule was proved to be true divided by total number of sales from that store and is referred as rule support. Support of an item set is defined as the percentage of the data set which contains that particular item set. Rule confidence is related important statistical measure and calculates the efficiency of rule to calculate prediction of right hand side of rule, here item B in our example when left hand side of rule, item A in this example is triggered thus giving number of baskets in which A & B both exist divided by number of baskets with only expressed in percentage. Confidence of combination of items divided by support of result is called as lift and gives a ratio of how often B is bought along with A to how often B is bought independent. This is nice calculation as it gives better picture to association than rule support as we can see clearly that B is more often bought with A when lift value is more than one otherwise not. We will elaborate more about these statistical terms and their calculations in detail with help of examples later in this report. As calculations are simple, the challenge comes with performance as generally data under analysis is huge. Even to interpret results is difficult without deep business domain knowledge as you can lead in wrong impression of rules to be either trivial associations or apparent nonsensical 65 Chapter 4: Cluster & Association Analysis Explored associations. It is most often called as market bucket analysis based on its most common application of finding out rules of products getting sold together in a supermarket. Using the data gathered from baskets, list of products sold together, we can have an analysis of patterns or strong relations between products to recommend product placement in store, suggest additional product purchases to buyers or identify unusual combinations of fraud management. Different objective measures deﬁne different association patterns with different properties and applications. For instance, the purchase of an electronic device that does not include batteries often implies the purchase of batteries or charger. Apriori principle: If an itemset is frequent, then all of its subsets are frequent. 4.1.1 Applications of Association Analysis Netflix based on previous rating of movies compared to other users watching patterns predicts for you movies of interest to you. Associations generally depend on finding patterns that can be evaluated through subjective arguments. It is considered uninteresting for data analysis if it don't reveal unexpected information about the data or give some new unknown information that can lead to proﬁtable actions. To include subjective knowledge into pattern evaluation needs lot of efforts and knowledge from domain experts and an extensive amount of prior information from historic data. Pattern evaluation gets more challenging when partial associations among items within the pattern are also present. For an instance, few associations & relationships keep appearing and disappearing when conditioned with the value of certain items. Support(X) = no. of transactions which contain the itemset X / total no. of transactions Confidence (X>>Y) = Support(X U Y) / Support (X) Lift (X>>Y) = Support(X U Y) / Support (Y) * Support (X) 66 Chapter 4: Cluster & Association Analysis Explored 4.1.2 Apriori Association Analysis It is an influential algorithm to find associations in market basket data or sales transaction data giving some Boolean association rules as an output based on is calculations of three statistical values, support, confidence and lift. It continues to identify the frequent individual items and extend them to larger data sets till these item sets appear sufficiently often in the analysis. Apriori is designed to handle databases that hold transactional data like list of items bought by customers or details of a website frequentation. Apriori algorithm works on following general process by splitting association rule generation into two separate steps: 1. Minimum support is applied to find all frequent itemsets in a database. 2. These frequent itemsets combined with minimum confidence constraint are considered to output rules. 67 Chapter 4: Cluster & Association Analysis Explored Let’s take the example dataset as shown above to illustrate these three terms and algorithms. Support is calculated as ratio of total number of baskets that support rule i.e. a combination exists to total number of baskets expressed in percentage. Note that support is bidirectional that is 'if 10 then 20' will be similar to 'if 20 then 10' as both will have same rule percentage. Confidence is defined as ratio of number of baskets in which both items 1 & 2 exist divided by the number of baskets with only item 1 in them expressed as percentage. Confidence is not bidirectional as support. Both support & confidence give an idea about rule’s validity but cases exist when value of both of them is high and concluded rule is of no use. This shortcoming of these 2 measures bring into picture one more measure to find accuracy association called a lift or improvement and is defined by the ratio of how often when item 1 is bought item 2 is also bought divided by how often item 2 is bought independent. Value less than one indicates item 2 is more often bought independently on own while value greater than one tells item 2 is often bought with item one. Calculations are as shown in figure and it is not challenging to conclude that Support can be used to see most popular rules, Confidence gives you most useful rules while overall most useful & popular rules are given by Lift. Now as we understand terms to measure and compare associations in a dataset, we will now talk about how to implement Apriori association analysis. 4.1.3 Apriori Association Analysis in PAL In PA library for SAP, the function name to perform Apriori association analysis is APRIORIRULE and the algorithm name is Apriori. The input is always of two variables, first being the transaction ID representing Basket ID and second being Item ID signifying the product name. Data types for both these variables can be Integer, varchar or char. The output is comprised of two tables where first one contains the association rules with leading items or pre-rule or lefthand side in the first column and dependent items also called as post rules in the second column. Some applications allow and demand combining the pre-rule & post-rule item columns in order to construct the total rule i.e. table will now display support, confidence and lift values. Second table shows PMML definition of Apriori model which are forced to be the calculated rules and their measures. Following figure shows the definition of Parameter table for Apriori algorithm. 68 Chapter 4: Cluster & Association Analysis Explored Parameter Table Definition for Apriori The following text displays main components of the SQLScript for Apriori. The full code can be accessed from the file SAP_HANA_PAL_Apriori_Example_SQLScript on the SAP PRESS website. // The procedure generator CALL "SYSTEM".afl_wrapper_generator('PAL_APRIORI_RULE', 'AFLPAL','APRIORIRULE', PDATA); // The Control Table parameters INSERT INTO PAL_CONTROL_TAB VALUES ('MIN_SUPPORT',null,0.01,null); INSERT INTO PAL_CONTROL_TAB VALUES ('MIN_CONFIDENCE',null,0.01,null); INSERT INTO PAL_CONTROL_TAB VALUES ('PMML_EXPORT',2,null,null); INSERT INTO PAL_CONTROL_TAB VALUES ('THREAD_NUMBER',2,null,null); // Assume the data has been stored in the table PAL_TRANS_TAB // Calling the procedure CALL PAL_APRIORI_RULE(PAL_TRANS_TAB, PAL_CONTROL_TAB, PAL_RESULT_TAB,PAL_PMMLMODEL_TAB) with overview; SELECT * FROM PAL_RESULT_TAB; SELECT * FROM PAL_PMMLMODEL_TAB; // Merging Prerule & Postrule DROP VIEW TMP_RESULT_V; CREATE VIEW TMP_RESULT_V AS SELECT CONCAT(PRERULE, ' => ') AS PRERULE, POSTRULE, SUPPORT, CONFIDENCE, LIFT FROM PAL_RESULT_TAB; DROP VIEW RESULT_V; CREATE VIEW RESULT_V AS SELECT CONCAT(PRERULE, POSTRULE) AS RULES , SUPPORT, CONFIDENCE, LIFT FROM TMP_RESULT_V; SELECT * FROM RESULT_V; As discussed, we get two output tables one containing the measures and one showing associations in PMML file. This PMML output can be used to transfer model rules to a business application such as recommendation engine. 4.1.4 Strength & Weakness with Apriori Lite Apriori Lite can be considered as an alternate to Apriori but actually is a specific instance of Apriori algorithm in a way that it looks pre-rule and post-rule rules only for a single item. This makes it more efficient and faster but can be applied only when we try to seek one-to-one rules instead of getting all associations. Another plus point of this algorithm is the possibility to sample data. LITEAPRIORIRULE is the function name to call it in SAP Predictive analysis library and the input to this algorithm is exactly same as Apriori full. In parameter table of Apriori Lite, 69 Chapter 4: Cluster & Association Analysis Explored MAXITEMLENGTH does not exist but include two extra parameters namely OPTIMIZATION_TYPE and IS_RECALCULATE. Reason behind association analysis being so popular is the ability to produce clear results. Calculations are most of times so straightforward that anyone in management position without detailed technical knowhow can understand it easily and thus speeding up faster and better decision making. One of biggest drawback with Apriori is its heaviness and it requires more and more computations exponentially with increase in data. Apriori lite thus is an alternate when one-to-one rules have to be found. Sometimes results are of no value and misleading. No matter how many weakness we can count for this association analysis and related algorithms, we can never utilize PA to fullest without them in most of cases. There may be cases and problem sets that can be best solved with regression analysis & never use Apriori. 4.2 Cluster Analysis This chapter involves the concepts around cluster analysis and how it is being implemented in PAL, R and SAP PA. Cluster analysis is also referred to as segmentation analysis and is a very popular application of SAP PA. We start this section explaining simplest of all algorithms i.e. ABC Classification which groups records in data set based on specific parameter into top X % then top Y % and all remaining in Z % to total up to 100%. We will also talk about popular and efficient statistical analysis algorithm for comparing, machine-learning clustering, self-organizing maps etc. and is called K-Means cluster analysis. Task of grouping and classification started since days of early man who needed to distinguish between edible & poisonous food, pet & wild animals and we still do in daily life by grouping students at university by country of origin, previous studies aggregates, age, gender etc. giving us a reason to do it. It is clear that we have a better understanding of data even when data size is large if we break it down in groups. Attribute values which describe the objects are used for assessing the dissimilarities among clusters. 4.2.1 Introduction & Applications of Cluster Analysis Cluster Analysis intends to organize and segment data into groups with close characteristics and features in a way that data within a group is closely matching to other data in same group and is different from data in other groups in some respect. In other words, objects within one cluster are shortly distant and compact within cluster while distance between intracluster objects & clusters is more as they look disparate. Cluster analysis or clustering thus can be defined as the job of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters). We may need to do this clustering for many reasons, for an example, identifying 70 Chapter 4: Cluster & Association Analysis Explored people with similar shopping pattern to find better marketing strategies, grouping movie shows into similar categories based on viewer ratings, making task of cluster analysis considered as intellectually satisfying, profitable, and sometimes both. Cluster analysis is a concept, a method which do not represent or identify a particular statistical method or model, as factor analysis, and regression. There can be many ways to cluster data into groups and the choice depends on various factors and requirement. Cluster analysis encompasses a variety of algorithms and methods to classify objects of similar kind into given categories. Cluster analysis as on output finds for you structures in data without knowledge of why they exist. Cluster analysis can be seen as most widely used class of predictive analysis methods with diverse applications including criminal pattern analysis, medical research, social services, psychiatry, education, archaeology, astronomy, and taxonomy making it indeed ubiquitous and significant for data analysis. Market segmentation is one of most talked about application and helps to make better decisions by making different business plans for different group of buyers with different promotional offers. A nice example to highlight the importance of clustering is that there is an exponential decrease in number of different sizes of clothes available in stores because we after analyzing many measurements of body size and came up with a generalized system of body measurements whereby individuals are allocated to specific sizes/clusters. 4.2.2 ABC Analysis in PAL Cluster Analysis intends to organize and segment data into groups with close characteristics and features in a way that data within a group is closely matching to other data in same group and is different from data in other groups in some respect. In other words, objects within one cluster are shortly distant and compact within cluster while distance between intracluster objects & clusters is more as they look disparate. Cluster analysis or clustering thus can be defined as the job of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters). We may need to do this clustering for many reasons, for an example, identifying people with similar shopping pattern to find better marketing strategies, grouping movie shows into similar categories based on viewer ratings, making task of cluster analysis considered as intellectually satisfying, profitable, and sometimes both. Cluster analysis is a concept, a method which do not represent or identify a particular statistical method or model, as factor analysis, and regression. There can be many ways to cluster data into groups and the choice depends on various factors and requirement. Cluster analysis encompasses a variety of algorithms and methods to classify objects of similar kind into given categories. Cluster analysis as on output finds for you structures in data without knowledge of why they exist. 71 Chapter 4: Cluster & Association Analysis Explored ABC Analysis, K-means and self-organizing maps are three cluster analysis algorithms supported by SAP Predictive analysis library. Three user defined clusters are given by ABC Analysis while K-means creates K clusters based on data memberships. Self-Organizing maps use a map, usually an M*N matrix, to map the given data item to some coordinates on the map which later takes form of clusters when multiple records get mapped to specific coordinates. ABC which stands for Artificial Bee Colony Analysis clusters data depending on what that particular data item contributes to total to find top X % of items based on characteristic A or top Y % of items based on characteristic B etc. was first proposed by Karaboga. ABC classification hence gives functionality to an organization to segregate units into three groups: A, the most important; B, important; and C, the least important. The intention behind such classification of items into groups is to have a better control and understanding over each item based on their group. The ease of use and simplicity to understand make it more popular. Data generally is initially sorted in descending numeric order and then grouped into first A %, the second B % and finally remaining C % to total up as complete hundred. It can be considered as one weakness of this algorithm as it don’t support more than 3 groups. The Artificial Bee Colony (ABC) algorithm treats the search space representing data set as it were a foraging environment, and each point in this search space relates to a food source which we take as solution and has to exploit by the artificial bees. This can be useful but basically the algorithm just sorts the data based on a continuous variable i.e. is my customer in the 20%, 50%, 80% most spending customers for an example? The fitness of the solution is represented as the nectar amount of a food source. According to this algorithm, there are 3 kinds of bees employed bees, onlooker bees, and scout bees. Specific food sources are first exploited by Employed bees before and then forwarded for the quality information of the food sources to the onlooker bees. Information about the food sources is received by onlooker bees who then will exploit a particular food source based on the information of nectar quality will be chosen by them. The more nectar the food source contains, the larger probability the onlooker bees will choose it [23] [24]. “Limit” is the quality parameter controlling the employed bees whose food should be abandoned. Food sources is responsibility of scout bees by searching & analyzing whole environment. ABC algorithm can be defined as below steps: 1. Initialization phase when each food source X i,j available in environment is initialized by scout bees after setting up the control parameters. The number of food sources equals to half of the colony size. D the dimension represents the number of parameters to be optimized. 2. Employed bees phase when Employed bees start searching for more food sources having more nectar i.e. increased fitness value in the neighborhood of the food sources in their memory. Once a neighbor food source is encountered, these employed bees calculate its 72 Chapter 4: Cluster & Association Analysis Explored fitness. Greedy algorithm is applied on the new food source to the original food source and the best will be placed in memory. If the food source is improved, the trials counter of this food will be reset to zero else incremented by one. 3. Onlooker bees phase when Onlooker bees resting till now in their hive are given all this food source information from employed bees which based on their probabilistically calculations on fitness values on given information choose their food sources. An onlooker bee chooses a food source depending on its probability value which may allow multiple onlooker bees choosing a same food source if that food source has a higher fitness. Once food sources have been selected by onlooker bees, each of onlooker bee will now find a new food source in the neighborhood and will compute Fitness values of these new food sources same way as the employed bees did in their phase. That means more onlooker bees will be used to find richer food sources. 4. Scout bees phase is last phase when the value of trials counter of each food source is used to decide. If the value is more than the limit parameter, the food source will be abandoned and the bee there will become a scout bee going back to initialization phase and a new food source will be produced randomly in the search space for these new scout bees and the trials counter will be reset to zero for them. The first three phases will be repeated until some end criterion is met and the best food source showing the best optimal value will be considered as final solution. An example is shown in figure below where values of A, B and C are 20%, 30% and 50% respectively which help us to visualize that A segment being 20% of total is accounted for 5 items out of 70 or 7.1% of all items. Segment B being 30 % of total is accounted by 9 items and this being 12.9% of all items whereas last segment of 50% is accounted by 56 items and thus relates to 80% of item population. An Example of ABC Analysis 73 Chapter 4: Cluster & Association Analysis Explored Algorithm name to call it for grouping in PAL is ABC ANALYSIS and corresponding function name is ABC. There are two columns in input able when item or record names are contained in first column while numeric values to be used for analysis are stored in second column. Item name has data type as char or varchar while corresponding value is inputted as Double always. The parameter table contains four parameters namely ’PERCENT_A’ having data type as Double representing Interval for A class, ’PERCENT_B’ having data type as Double representing Interval for B class, ’PERCENT_C’ having data type as Double representing Interval for C class and ’THREAD_NUMBER’ having data type as an integer value representing the total number of threads. Values of ABC always should add up to 100 and is a check done by PAL before calling the algorithm. Output table has 2 columns again holding assigned values of A, B or C to items and other one holding item name in output table as shown in figures below. ABC Analysis Input & Output tables The main elements of the SQLScript are as follows, with the control parameters set as A=35%, B=20% and C=45%. The full code is available in the file SAP_HANA_PAL_ABC_Example_SQLScript on the SAP PRESS website. // the procedure generator Call SYSTEM.afl_wrapper_generator ('PAL_ABC','AFLPAL','ABC', PDATA); // The Control Table parameters INSERT INTO #CONTROL_TBL VALUES ('PERCENT_A', null, 0.35, null); INSERT INTO #CONTROL_TBL VALUES ('PERCENT_B', null, 0.20, null); INSERT INTO #CONTROL_TBL VALUES ('PERCENT_C', null, 0.45, null); INSERT INTO #CONTROL_TBL VALUES ('THREAD_NUMBER', 1, null, null); //Assume the data has been stored in table TESTABCTAB 74 Chapter 4: Cluster & Association Analysis Explored //Calling the procedure CALL PAL_ABC (TESTABCTAB, "#CONTROL_TBL", RESULT_TBL) with overview; SELECT * FROM RESULT_TBL; 4.2.3 K-Means Cluster Analysis in PAL The K-Means algorithm is one of the best known predictive analysis algorithms and very famous for cluster analysis as it efficiently clusters the records or observations into K clusters such that each record belongs to the cluster with the nearest mean. This algorithm works over continuous data and can be applied in different kinds of domains. As k-means needs initial partitions to initialize the task, best results can be expected only when the initial partitions keep getting closer to the final solution. This algorithm tries to identify relatively homogeneous groups of values in given dataset based on chosen parameters and the specified number of clusters. This algorithm processes a set of data to cluster them into a predefined number of clusters represented by K. k-means initialize itself with random cluster centroids as a starting point and as it progresses it keeps replacing the data objects in the dataset to cluster centroids depending on closeness between the cluster centroids and the data objects. This reassignment procedure and algorithm terminates As soon as the any of finishing convergence criterion like the number of iterations, or the cluster results being unchanged even after a certain number of loops) is encountered. Performance is dependent significantly on random selection of centroids to initialize the process. The k-means clustering process can be seen as by the four following steps: 1. Randomly picking K centroids to give an initial dataset partition, which most of times is struggle to find, how to choose value of K, and calculate centers with distances? 2. Placing each value from the dataset under analysis to the closest cluster centroid. The measure nearest can be looked differently as there can be several inter-object distance ways to it which can affect the assignment. 3. Recalculating the centroid of each of K clusters to get new mean. 4. Repeating above two steps until exit criteria is met. In Predictive Analysis Library, associated function name is KMEANS and algorithm name is KMEANS. Input table is not fixed in structure but there is always a column for initial ID and then subsequent columns to contain variables for analysis which are always numeric in nature. This is because clusters are calculated based on inter-object distance measures which don’t make sense for non-numeric data type. The table represents the definition of the parameter table for K-means algorithm in PAL. Value of K is simply an integer signifying the quantity of clusters or segments we wish to derive. Manhattan distance which is also called city block distance calculates distance between two points horizontally or vertically on a grid. Euclidean distance is unique shortest path and is used 75 Chapter 4: Cluster & Association Analysis Explored as most common approach to calculate distance between any two points for clustering. Murkowski generalize both of above mentioned Euclidean and Manhattan distances. Maximum number of iterations can be used to define an exit criteria thus giving some control over processing time and complexity of algorithm and saving from process going relatively very long. For initialization or seeding, first K records method takes first K records as initial cluster centers and makes illogical processing if data is sorted. Random with replacement method selects randomly K records from data set to be used as initial centers. SAP patent method to find K random clusters is based on max-min approach where initial center is chosen very close to minimum point and then subsequent centers chosen. Threshold value indicates when iterative process should end and default value is set as 0.00001. Name Data Type Description GROUP_NUMBER Integer The value of K, the number of clusters. DISTANCE_LEVEL Integer Computes the distance between the item and cluster center, can be Manhattan distance, Euclidean distance or Murkowski distance MAX_ITERATION Integer The maximum number of iterations. Integer Center initialization method: 4 options First K, Random with replacement, Random without replacement or one which is SAP’s patent for selecting the initial centers NORMALIZATION Integer Normalization method with three options No, Yes for each point or Yes for each column EXIT_THRESHOLD Double The threshold (actual value) for exiting the iterations. THREAD_NUMBER Integer The number of threads. INIT_TYPE Parameter Table Definition for K-Means To save our analysis from a situation where very large numbers dominating very small numbers, it’s a good practice to normalize or standardize data which ensures all variables having equal weight in subsequent calculations. The output for K-Means is in form of two tables. The first table holds results of the analysis, which maps each record or item in the data set to an assigned cluster 76 Chapter 4: Cluster & Association Analysis Explored number along with the distance from that item to the cluster center this data item is part of. The co-ordinates of each cluster center are listed in second table as the Center Points row. This distances between item & cluster centers for all data items gives a way to measure the compactness of the clusters and to identify unusual values or outliers. The main elements of SQLScript to call PAL K-Means algorithm are as follows with defined parameters settings while the full code is available in the file SAP_HANA_PAL_KMEANS_Example_SQLScript on the SAP PRESS website. // the procedure generator Call SYSTEM.afl_wrapper_generator ('PAL_KMEANS','AFLPAL', 'KMEANS', PDATA); //The Control Table parameters INSERT INTO PAL_CONTROL_TAB VALUES ('GROUP_NUMBER', 4, null, null); INSERT INTO PAL_CONTROL_TAB VALUES ('INIT_TYPE', 4, null, null); INSERT INTO PAL_CONTROL_TAB VALUES ('DISTANCE_LEVEL', 2, null, null); INSERT INTO PAL_CONTROL_TAB VALUES ('MAX_ITERATION', 100, null, null); INSERT INTO PAL_CONTROL_TAB VALUES ('EXIT_THRESHOLD', null, 0.000001, null); INSERT INTO PAL_CONTROL_TAB VALUES ('NORMALIZATION', 0, null, null); INSERT INTO PAL_CONTROL_TAB VALUES ('THREAD_NUMBER', 2, null, null); //Assume the data has been stored in table PAL_KMEANS_DATA_TAB //Calling the procedure CALL _SYS_AFL.PAL_KMEANS (PAL_KMEANS_DATA_TAB, PAL_CONTROL_TAB, PAL_KMEANS_RESASSIGN_TAB, PAL_KMEANS_CENTERS_TAB) with overview; SELECT * FROM PAL_KMEANS_CENTERS_TAB; SELECT * FROM PAL_KMEANS_RESASSIGN_TAB; 77 Chapter 4: Cluster & Association Analysis Explored The results of running the SQLScript are shown above with input data, which show the assignment of each record to a specific cluster and the data points for each cluster center. 4.2.4 Silhouette The cluster viewer window in SAP PA displays four charts. A horizontal bar chart showing the size of each cluster and A cluster density and distance chart with a color coded scale of dark to light, for dense to sparse clusters. Thicker the line, the closer are the clusters. Other two charts at the bottom allow the user to compare clusters by user chosen variable for better customized understanding and analysis. How to choose value of K is a key question as it impacts performance and output of analysis. In some business cases we can think of a value based on requirement and application like t shirt size groups for sale in stores, still for many cases the value of K is difficult to pre-set before analysis. One of commonly used approaches is to use the square root of N halved where N is total number of records in dataset. This becomes challenging when data set is too big as an instance we would need 700+ clusters to cluster a million records based on this approach and this big number of clusters can’t be considered easy to manage. In that case, data visualization is a big aid and stored can be seen and plotted in a bubble plot. Cluster Analysis comes under undirected data mining as here is no target or dependent variable to be predicted, so another quantitative approach can be to determine value of K which measures cluster quality is silhouette. Silhouette is thus a practical way to measure quality of cluster analysis without which it is not possible as we can never compare or find the output of analysis. Good clusters can be thought of groups where cluster members are close to each other as well as far from members of other clusters. The average for all the records in the dataset of (b−a) / max (a, b), is calculate here by silhouette method where a is the average distance of the record to all other records within the same cluster i.e. cohesion while b is the average distance of the record to all the other records in the nearest cluster center that it does not belong to i.e. separation. This calculated value indicates all records to be located directly on cluster centers when value is 1 while indicating all records to be located on cluster centers of some other cluster when value is -1. When records will be equidistant from their own cluster center as well as cluster center of nearest other cluster, it will have silhouette coefficient set to 0. These values off course are ideal values but still can help to summarize a general guide as per which value less than 0.2 means very poor clustering while value of 0.5 or above is considered good and meaningful. In PAL, it can be called with function name VALIDATEKMEANS and has two input tables to run. First table is required for cluster analysis while second one is used to assign cluster numbers to each record. For table 1, the first column which is for Record ID can either hold Integer or 78 Chapter 4: Cluster & Association Analysis Explored String as data type while all other columns in table one which represent attribute data store values either in integer or double value. For table 2, the first column which is again used to store Record ID here hold only Integer values while second columns in table two which represent assigned cluster number store values integer form only. Parameter table and output table for Validate K-means when called in PAL are defined below. The silhouette measured value will increase and, finally, when K = N it will become equal to 1, as the value of K approaches the number of records (N). Name Data Type Description VARIABLE_NUM Integer The number of variables THREAD_NUMBER Integer The number of threads Name Data Type Description Result 1 Varchar or char Name Result 2 Double The Silhouette value Parameter Table Definition for Validate K-Means The SQLScript to call Validate K-Means in the PAL is as follows while the full code available in the file SAP_HANA_PAL_VALIDATEKMEANS_Example_SQLScript on the SAP PRESS website. // the procedure generator Call SYSTEM.afl_wrapper_generator ('palValidateKMeans','AFLPAL','VALIDATEKMEANS',PDATA); // The Control Table parameters INSERT INTO #CONTROL_TAB VALUES ('VARIABLE_NUM', 2, null, null); INSERT INTO #CONTROL_TAB VALUES ('THREAD_NUMBER', 1, null, null); // Calling the procedure CALL palValidateKMeans(PAL_KMEANS_DATA_TAB, V_KMEANS_TYPE_ASSIGN,"#CONTROL_TAB", KMEANS_SVALUE_TAB) with overview; SELECT * FROM KMEANS_SVALUE_TAB; To choose the initial cluster center for K-means algorithm is second key question and is answered by seeding strategy applied. Generally many different seeding strategies are applied and solutions are compared for robustness to see if the solution change or remains constant with change in values. Clusters need to be understood and analyzed again to check reason of differences if solution is changing frequently. This was a discussion when data set to be clustered 79 Chapter 4: Cluster & Association Analysis Explored was all numeric but imagine a scenario when we have to cluster categorical data. Does PA allow us to handle it? For sure, categorical data cannot be clustered based on inter-object distances as we cannot say the difference between sun and moon. An approach can be to convert each category into a new variable in binary format. Once we have them we can rescale them by multiplying by SQRT (0.5) to reduce influence when we have binary variable mapped with categorical data items. It is also a good practice to merge categories when there are many categorical variables and within each variable more sub categories exist or alternatively we can consider other non-distance based clustering algorithms. Decision tress to associate analysis close to clustering is a win situation. Decision Tree Analysis of Clusters 4.2.5 Self-Organizing Maps Self-Organizing Maps or SOMs also called as Kohonen SOMs after their inventor professor Teuvo Kohonen, of the Academy of Finland, are a type of neural network that can be used to cluster a dataset into distinct groups. One or two dimensions in a vector or matrix, known as a map are generally used to represent multi-dimensional data in much lower dimensional space. Once network gets trained records in data set that are different will appear far apart while records which are similar will appear close together on the output map. More populated units are shown by the number of records or observations captured by each cell or unit in the map indicating groupings of the records or segments initializing the existence of a sense of the appropriate number of clusters in the dataset. The value of ‘K’ is not predetermined as in K-Means cluster analysis. They are based on unsupervised learning, which means that no human intervention is needed during the learning and that little needs to be known about the characteristics of the input data. 80 Chapter 4: Cluster & Association Analysis Explored A network created from a 2D lattice of “nodes”–the map is shown in figure below which is fully connected to the input layer. A small SOM network of 3 * 3 nodes connected to the input layer of a two dimensional vector i.e. a two-variable dataset is also shown below. Generally specific topological position is assigned to each node, an x, y coordinate in the lattice or map, which also contains a vector of weights of the same dimension as the dimensions of input vectors. In the input vector/dataset there are 2 dimensions/variables in our example which means each node will have a corresponding weight vector W, of 2 dimensions: W1, W2 there to represent adjacency we have lines connecting the nodes but they do not signify a connection. 3*3 SOM connected to a 2-variable Dataset Following steps occurring over many iterations represent training of the SOM: 1. Each node in the map gets its weights initialized with random values between -0.05 and 0.05 set by PAL. 2. From inside the set of training data, a vector is chosen generally starting with the first one and is presented on the map. 3. The ”winning” node or the Best Matching Unit (BMU) is calculated such that nodes’ s weights are most similar or closest to the input vector using any distance measure such as the Euclidean distance for each and every node in set. 4. The radius of the neighborhood of the BMU is then calculated which represents a value that starts large, typically chosen to the radius of the lattice but decreases with each iteration and all nodes found within this radius are deemed to be inside the BMU’s neighborhood. 81 Chapter 4: Cluster & Association Analysis Explored 5. Weights of each neighboring node’s found in step 4 are adjusted to make them more similar to the input vector. Weights get altered more when node is closer to the BMU. 6. Steps 2 to 5 are repeated for the next vector in the data set and then for N iterations or until the weights stop changing. Figure below highlights with an example the size of a typical neighborhood when its time near to the commencement of training. The area of the neighborhood shrinks with time, which is accomplished by making the radius of the neighborhood shrink too with help of decay function, a unique characteristic of the Kohonen SOM. Decreasing Neighborhood Size during SOM Iterations Eventually with time, the neighborhood shrinks to the size of just one node which we call as the BMU. The goal is to discover some underlying structure of the data. Thus we can say that SOMs have two phases: Learning phase when map is built and network organizes using a competitive process with help of training set and secondly Prediction phase when new vectors are quickly given a location on the converged map, easily classifying or categorizing the new data. The weight vector of node is adjusted as follows when a node is found to be within the neighborhood, else it is left alone. W (t+1) = W (t) + λ (t) * (V (t) – W (t)) Where T represents the iteration and λ, which is a small variable and called the learning rate, reduces with each iteration that means the new adjusted weight for the node gets equal to the old weight (W) when added to a fraction of the difference (λ) between the old weight and the input vector (V). Λ (t) = λ0 exp (–t / λ) λ0, denotes the width of the lattice at iteration t = 0 and the Greek letter lambda, λ, denotes a constant. λ0 is set to 0.5 by default in PAL. 82 Chapter 4: Cluster & Association Analysis Explored The effect of learning should also be proportional to the distance of a node from the BMU with every decay in leering rate over time. The learning process should barely have any effect at all at the edges of the BMUs neighborhood. The quantity of learning generally fades with distance same way as it do in the Gaussian decay which is shown below: λ(t) = exp (–dist2 / 2λ2(t)) With each iteration, records from the training data set are allocated to cells on the map, and closely related records grouped together, as shown in figure below. Self-organizing maps are different from other artificial neural networks in the sense that they use a neighborhood function to preserve the topological properties of the input space. Assignment of the Data Set Records to the Map, Showing the Clusters We can call this method by associated function name ‘SELFORGMAP’ In the Predictive Analysis Library where it comes under set of algorithms and the algorithm name is Self-Organizing Maps. The Input table consists of an initial ID column, and all subsequent columns contain the variables to be used for the cluster analysis. These variables must be numeric to be used with the cluster analysis. It’s because SOMs cluster data objects using the inter object distance which cannot be computed if the input variables are not-numeric. The Parameter Table Definition for Self-Organizing Maps is shown in table below. There can be 3 methods to normalize the data as discussed below: 1. The data stays as default setting and the parameter is set to 0. 83 Chapter 4: Cluster & Association Analysis Explored 2. For each variable X (x1,x2,...,xn), the minimum and maximum value of X is found and then X[i] = (X[i]-min)/(max-min) is calculated to rescale the data between 0 (min value) and 1 (max value) and and the parameter is set to 1. 3. For each variable X (x1,x2,...,xn), the normalized values depend on the mean and standard deviation of X. x1, is normalized to X’ by computing X’ = (xi – Mean(X)) / S.D.(X) for each X when the parameter is set to 2. Name Data Type Description SIZE OF MAP Integer The self-organizing map is made up of n × n unit cells. This parameter defines the value n. MAX_ITERATION Integer The maximum number of iterations. NORMALIZATION Integer Normalization method with 3 methods : No or Transform to new range (0.0, 1.0) or Z-score normalization THREAD_NUMBER Integer Number of threads. The Parameter Table Definition for Self-Organizing Maps Self-Organizing Maps outputs two tables. The first table is called SOM Map & holds the final weights corresponding to each of the map cell IDs, along with the number of records or tuples assigned to each map cell ID i.e. cluster size. It is stored in last column of table and is always integer type. The weight vectors to stimulate original tuples are always outputted as double in middle column. The second output table is called SOM Assign row and maps each Cell ID assigned to each record displaying the membership of the clusters. ID for tuples can be integer or string. Name Data Type Description 1st column Integer Unit cell ID. Other columns except the last one Double The weight vectors used to simulate the original tuples. Last column Integer The number of original tuples that every unit cell contains. Output Tables Defined for Self-Organizing Maps See below an example of self-organizing maps in the PAL implemented on same data set as shown in figure above on which we implemented K-Means and thus an interesting attempt to 84 Chapter 4: Cluster & Association Analysis Explored comparison of the two cluster analysis algorithms also. The PAL SQLScript to call for Self Organizing Maps is as follows while full code available in the file SAP_HANA_PAL_SELFORGMAP_Example_SQLScript on the SAP PRESS website. // PAL set-up Call SYSTEM.afl_wrapper_generator('PAL_SELF_ORG_MAP', 'AFLPAL', 'SELFORGMAP', PDATA); // Preparing application data for calling procedure INSERT INTO PAL_CONTROL_TAB VALUES ('MAX_ITERATION', 200, null, null); INSERT INTO PAL_CONTROL_TAB VALUES ('SIZE_OF_MAP', 4, null, null); INSERT INTO PAL_CONTROL_TAB VALUES ('NORMALIZATION', 0, null, null); INSERT INTO PAL_CONTROL_TAB VALUES ('THREAD_NUMBER', 2, null, null); //Assume the data has been stored in table PAL_SOM_DATA_TAB // Calling the procedure CALL PAL_SELF_ORG_MAP(PAL_SOM_DATA_TAB, PAL_CONTROL_TAB, PAL_SOM_MAP_TAB, PAL_SOM_RESASSIGN_TAB) with overview; Select * from PAL_SOM_MAP_TAB; Select * from PAL_SOM_RESASSIGN_TAB; This script outputs two tables as shown below. The final weights assigned to each of the map cell IDs along with the number of records or tuples corresponding to specific map cell ID (represents the cluster sizes) is shown in first output table. Cell ID assigned to each record which is the membership of the clusters is output shown in second table. E.g., Cell ID 0 has 2 records with trans_id 11, 12 and Cell ID 13 has 2 records again with trans_id 18, 19 and Cell ID 15 has 5 records (Trans ID 0,1,2,3,4)etc. 85 Chapter 4: Cluster & Association Analysis Explored Generally after this the results are presented in a visualization to understand and analyze the clustering of the data, for instance, cell 7 has five records, cell 8 has two records, and so on. We can run this SOM in N*N grid. Smaller the number of N, more close the cells and hence more closed grouping can be seen. Comparing the outputs of SOM with K-Means give most of times same number of clusters but in K-Means algorithm user predetermines the value of K prior to running the algorithm but this is not case with Kohonen SOMs as we allow the data suggest the value of K. SOM if have less variables than number of cells, it will result a sparse output with lot of empty regions in middle of classified objects while in alternate case, cells will be forced to share with variables and thus giving groups or clusters. Following figure show the visualization of results when we run SOM on input data in figure XX and plot the table data for 4 clusters in 4*4 map. The Four Clusters in the 4 * 4 Map Cluster analysis definitely is capable to be called one of most popular methods of predictive analysis but is more involved for business operations because of its uniqueness in splitting down huge amounts of data into smaller manageable clusters giving better understanding compared to analyzing data for predictions. Most commonly used application is market segmentation based on fact that focused marketing is more effective than a generic approach. Biggest strength of cluster analysis in general is it's easiness to understand as intention in this step is not to make predictions but to cluster existing data. Each cluster analysis algorithm discussed above has individual pros 6 cons. ABC Classification is very simple, very practical and therefore very popular but limited to 3 grouping seven when a user wants more groups which maybe not necessarily add up to 100% like finding the top 15%, second 10% and third 40%. 86 Chapter 4: Cluster & Association Analysis Explored K-Means is also easy to understand and to apply if we choose value of K in advance and detection to be undirected and it being data driven. K-Means is clearly driven by the choice of K which depends on user experience and skills but if chosen wrong will make whole clustering a bad experience. In K-Means the results generally vary dependent on the choice of distance is sensitive to the initial choice of cluster centers and measure. We can never try to optimize value of K as it don’t involve maximizing or minimizing some function. Input data is assumed to be numeric from idle run but Non-numeric data can be aliased to numeric using some approaches to initiate cluster analysis such as using decision trees and association analysis to find groups in the data. Self-organizing maps find its application to create an order visualization of multidimensional data which simplifies complexity and reveals meaningful relationships and thus can be compared to a non-parametric regression technique which converts multi-dimensional data spaces into lower dimensional abstractions. They are popular and beneficial as they provide quick and concise model creation even for voluminous data sets and have outstanding prediction accuracy due to patented procedure for the extraction of non-linear relations. They can work well only if clusters exist, but they can also not work i.e. it can’t find clusters for user if there are no clusters. 87 Chapter 5: Conclusion Chapter 5: Conclusion 5.1 Problem Set: Burn that Churn To implement and have a hand on experience for SAP PA, I decided to have data predictive analysis on a telecom data for last few years and then compare the efficiency and trueness of predictions by comparing the model correctness applying same model to different data sample subsets. Huge volumes of customer data is analyzed in order to identify business factors like new revenue opportunities, reduce churn which can help make informed decisions. Customers become “churners” when they discontinue their subscription and move their business to a competitor. Credit card issuers, insurance companies and telecommunication companies always are keen to predict churning i.e. process of customer turnover. If they predict the laving customer at right time, they can always try to hold him with better offers as it is always cheaper to retain a current customer than to gain a new one. In this work, we have only considered customerinitiated churn and ignored operator-initiated churn because mostly latter is because of payments problems from customers and are of no interest. All telecom service providers must be able to respond and react in timely manner to customer’s preferences for their survival in this competitive world. They must be able to predict and prevent subscriber churn by understanding reasons of already churned customers as well as expectations of current customers. All telecom operators store enormous size of data about subscribers and their behaviors. In this thesis part of work, this data has been attempted to analyze and find some valuable insight for business decision makers. For this objective there are many approaches to begin like customers segmentation based on location, demographics, purchase history etc., understanding which market campaigns and new products were able to add in most customers and more. As it is not practical to understand personal preferences for each and every subscriber, but it is possible to make visualizations and understand patterns based on historic data in repositories to predict probability of churn in each segment. Following variables were taken in main consideration as they appear to be having highest effect on churn. Customer demographics, i.e., age, gender, marital status, location, etc. Call statistics: length of calls at different times of the day, no of long distance and local calls. Billing information for each customer – what the customer is paying for local and long distance. Extra service information, that is, what extra plan the customer is registered on, e.g. special long distances rates. 88 Chapter 5: Conclusion Complaint information: how many customer service calls are made for disputed billing, dropped calls, slow service provisioning, non-working special services, and so on. Credit history. 5.2 Results & Analysis I got the data set for this work from TERADATA CENTER FOR CUSTOMER RELATIONSHIP MANAGEMENT AT DUKE UNIVERSITY which they used for NCR Teradata 2003 Tournament. The data represents a major wireless telecommunication service. Data holds records for more than 100,000 customers who have minimum of 6 months of service history with them. I found data quiet good for churn modeling and predictions to get results and analysis for this thesis report. Dataset was named tourn_1_calibration.csv’ and is copied to attached CD. Using normal Excel program, firstly I removed noise by deleting all rows with missing or null or strange appearing values for variables that were most appealing to me to influence prediction and modelling. These variables from data source are: 1. 2. 3. 4. 5. 6. 7. 8. Age of Handset Calls to customer care in last 3 months % change in monthly outgoing calls on previous 3 months average Age of Customer Current Handset Price Income of customer Total number of months as customer Credit History New dataset after removing noise is named as tourn_1_calibration _thesis.csv and is also copied to CD. Binning is implemented too on this data set to improve the analysis results and to have data read in group’s clusters instead of integer values. New columns I introduced to the dataset under binning process are: 1. range_handprice which has 3 possible values ‘low end’, ’medium’ & ‘high end’ representing the cost group of headsets. Mobile phones costing below 79 are classified as low end, costing more than 170 as high end while remaining come under medium group. 2. range_income which again has 3 possible values ‘low’, ’medium’ & ‘high’ representing the income bracket of users. Customers earning less than or equal to 3 on index are classified as low income group, earning more than or equal to 7 as high income group while remaining come under medium group. 3. range_agemob which has 3 possible values ‘less than 1 year’, ‘around 2 years’ and ‘very old mobile’ depending on duration of handset used. As name signifies, group called less than 1 year holds all entries where handset is used less than 12 months. 13 89 Chapter 5: Conclusion to 30 months of handset use comes under around 2 years group and all others more than 30 months old are classified under very old mobile group. 4. A new column is also inserted in dataset with column name ‘range_churn’ simply representing value of churn flag in a readable text format for quick analysis. Churn value 0 which represents customer still active has been shown in this new column as ‘Remained’ while value 1 is shown as ’churned’ stating that customer has already churned. 5. range_custcare is a new column grouping the number of customer care calls a user has made in last 3 months and has 3 possible values ‘less than 20’,’20 to 60 calls’ & ’more than 60’. Values under these 3 groups are easy to guess based on the group names. Same way all crucial columns that I thought to have a major influence on churn have been held in new columns and replaced by same bracket value with help of excel program. All these new columns are named as range_xxx to quickly select them from list of variables in PA. The process starts with importing this modified data set with added columns and noise removed to PA. In PA interface you can acquire the data source file from File menu. Select New, search csv file option and navigate to dataset in question. Once the dataset got imported, I run sample function to create 4 sample datasets of 7500 records each to make modelling faster. These 4 diff samples also helped to verify correctness of any model by applying models repeatedly and comparing results for these four samples individually. I named them as ‘churnsample1.csv’, ‘churnsample2.csv’, and ‘churnsample3.csv’ and ‘churnsample4.csv’ .All 4 sampled datasets were taken by random selection in PA and are copied in attached CD. 90 Chapter 5: Conclusion Next step is to start applying algorithms on each dataset and compare the calculated predicted value of churn flag with the actual value of churn flag. Following subsections hold the results for 4 algorithms categories and comments on how we can get an idea of future churning behavior based on the results of these algorithms on datasets. All the models created to get the below results are saved in .pmml format and are copied to attached CD for reference. Under Predict tab, PA Interface allows you to drag and drop the required algorithm in front of data set and before running you can configure the settings of each algorithm by left clicking on settings icon on algorithm itself, as shown in figure below. Subsections 5.2.1 to 5.2.4 are results for analysis and calculated values by PA on 4 different datasets for four different classes of algorithms. Some algorithms allow you to predict value for a particular variable say churn in our example by letting us dependent variable under configuration settings while others don’t. In that case we cannot have direct analysis for model efficiency by comparing the correctly predicted values to total number of values, but we still can have some understanding of patterns and behavior based on visualizations and comparisons. 91 Chapter 5: Conclusion 5.2.1 Clustering Algorithm Used: R-K-Means (MacQueen) Number of clusters: 5 Dependent variables: change_mou, months, no_of_cars, income, eqpdays and custcare_mean No of Iterations: 100 As this algorithm don’t allow us to calculate a value to specific variable based on other dependent variables, we can still find clusters and common patterns based on some independent variables. As clear from result sets (visualizations) below that cluster 5 is most dense while cluster 1 is smallest. From cluster density & distance chart, we can see strong connection between cluster 2 and cluster 5. Cluster center representation diagram reveals that cluster 1, 3 and 4 are more inclined on handset price variable while biggest one cluster 5 is based on 2 values equipment age and income bracket. Parallel coordinate chart displays calculated_churns for all 5 clusters while scatter matrix chart plots relations between different chosen independent variables against each other. The summary is as change_mou months hnd_price eqpdays income custcare_mean 1 2391.470 19.55148 7998999023 431.2059 5.913963 1.7418900 2 3313.584 24.77475 3026621144 639.8905 5.955474 0.8334206 3 2213.295 18.43822 5997563536 465.2371 5.928161 1.1652299 4 1932.428 19.43715 9995865138 308.0222 5.407119 1.9365962 5 1896.514 16.94159 1556729838 304.7009 5.701856 1.7578339 92 Chapter 5: Conclusion Cluster Analysis for first dataset 93 Chapter 5: Conclusion 5.2.2 Decision Tree This algorithm allows us to make calculations for a chosen variable or target variable and thus we can easily find the efficiency and correctness of model after running the model. Algorithm Used: R-CNR Tree (Regression) Output Mode: Trend Dependent variables: range_income, range_custcare, range_months, range_handprice, range_agemo Target variable: churn New output column: calculated_churn The output chart shows the probability of churn based on above mentioned dependent variables. By analyzing in detail, we can predict customers with high churning possibility based on rules. Example, customers with more than 20 customer calls and more than 23% decrease in overall revenue over past 3 months have 69% probability to churn or customers with expensive handsets and more use of mobile data compared to voice calls have 78% probability to churn, may be they use handsets for internet gaming or browsing more and thus latest handsets are only interest to them. This analysis also under result grid view give you a calculated predicted value of each customer to churn as shown in figure below. Decision Tress showing probability and classification of dependent variables. 94 Chapter 5: Conclusion Calculated churn value for each customer based on dependent variables 5.2.3 Apriori Here also we can’t calculate value for a particular variable based on other independent variables in dataset, so we rely on visualizations analysis and rules summary. Although it is not a favorable algorithm to apply in this problem set but business decisions can be made better just by some small piece of new information. Sort Type: Ascending Transaction Size Output Mode: Rules Dependent variables: range_income, range_custcare, range_months, range_handprice, range_agemo Support: 0.1 Confidence: 0.8 Below output figure is a set of rules generated by Apriori and can be very useful for decision making if dived in. This analysis based on 5 dependent variables gave us a set of 37 rules. Interestingly, changing the values of support and confidence brings change in number of generated rules as well as value of lift for each rule comparatively. 95 Chapter 5: Conclusion Apriori rules output for first dataset 5.2.4 Neural Network This algorithm according to my experience fits best in this problem set and allows us to prepare a model to be reused with new data to predict values for target variable. Using this approach, we run a model based on training data and keep fitting it unless the drop in correctness when applied to another dataset is not huge. When we run this model, it adds a new column in the table which according to model should be the value for that target variable (churn) based on previous patterns and behavior. This allows us to compare the value of target data in real to value of target data predicted by PA. Algorithm: R-MONMLP Target Variable: Churn Output Mode: Trend Dependent variables: income, months, custcare_mean, handprice, Hidden Layer 1 Neurons: 5 Predicted column name: predicted_value The following screenshot shows the insertion of new column predicted_value when we run this algorithm with values 0 and 1 which represents the customer churning or not churning according to this model and configuration settings for chosen dependent variables. 96 Chapter 5: Conclusion New column ‘Predicted value’ in result set We can easily find how many of the values are predicted right by PA by comparing the output values to real churn values in dataset. Total number of right findings divided by total number of rows will give the percentage correctness of the model. In first attempt, I got the following figures for the predicted values compared to churn values in dataset. Confusion Matrix for predicted churn variable That means out of 3780 correct one values, our model predicted ones correctly 2082 times i.e. approx. 55% correctness. I then tried to put new values by hit & trial method, combinations of variables to configure algorithm until I got the results and a model which is relatively better. In PA any model with 80% or above correctness is considered to be good. No model can be 100% accurate and applicable in all problem sets. It needs business processes understanding and domain knowledge to plan and implement predictive analysis solution for a problem set. 97 Chapter 5: Conclusion Output model with 65% correctness Output model with 96% correctness This improvement in model can also be an outcome of model Overfitting, so to confirm the possibility of model Overfitting, I applied the same model run on other 3 datasets. The correctness percentage of model din fluctuate to large values, so I considered this to be a good model. To save a model, using interface click on save as model under predict tab. Once saved models appear under component pane as shown in diagram below. This model is one which gave us 96% correct values for churnsample1. Saving a model makes it easy to reapply it to other datasets as we don’t need to worry about configuring algorithm settings again. 98 Chapter 5: Conclusion Saving a model with high correctness Following steps show the results and figures for same model ‘neuralnetwork’ when applied to other 3 datasets. Confusion matrix for data set 2, correctness 99% Confusion matrix for data set 3, correctness 98% 99 Chapter 5: Conclusion Confusion matrix for data set 4, correctness 79% As we don’t see a big fall or deviation in correctness when this model was applied to other 3 sample sets, we can conclude it to be a good configured model and can use it to predict churn variable for customers who are still active. That’s the potential of modelling and PA. 5.3 Discussion & Issues 5.3.1 SAP PA compared to Hadoop SAP Predictive Analysis tool as per date is designed to predict and analyze structured data only, that means user should have structured data either in xls, csv, txt or HANA database tables. If data is not in structured format then user cannot proceed further for analysis and prediction using PA while Hadoop is leveraged to analyze the unstructured and semi structured data as well. Hadoop is often used to store and process big data volumes of semi structured or binary data. But it looks feasible to pass results from Hadoop to PA with 2 steps. Firstly Hadoop doing first selection and aggregation to output reduced dataset. Take a case of text documents stored in Hadoop and it returning only list of words with frequency to PA and after that SAP Predictive Analysis when inputted this information from Hadoop can then apply data mining algorithms on the Hadoop result set. Comparing them makes no sense as they drive different purposes. Hadoop is a file system which stores variety of data like Big Data i.e. Structured, Unstructured and semi-structured data; while PA being an advanced analytical tool reads the historical data from above data sources and projects the predicted results for a business query where data can be picked from any data sources like HANA, HADOOP, BW, BO. It can be a next big approach from SAP to provide possibilities of SAP & Hadoop integration to ease the customer, either by merging PA with InfiniteInsight to PA which allows this Hadoop integration or some other innovative module 100 Chapter 5: Conclusion 5.3.2 Sharing your own R component It is possible to share with colleagues or customers your own created R components and just involves few simple steps. Depending on whether you need a new library or not with your component, library has to be added first under C:\users\Public\R-3.0.1\library and then can be shared. Remember to close PA program at time of sharing components. In the folder C:\Users\piyush\SAP Predictive Components\RScript you can see all your R components created by you and this folder is to be used to paste component shared to you by someone else. Rename the components to more meaningful names from default names given by PA. After renaming, it is crucial to modify the name of folder in the ‘component.xml’ file. With help of simple text editor replace ‘automatically created’ name by your new folder name. That’s all required to make your R created components being reused by your colleagues by simply sharing the folder. Logic is encapsulated in so called “Custom R Components”, allowing users even without R skills to use them. Connecting R and PA 5.3.3 Configuring HANA PAL to use with SAP PA SAP PA while connected to HANA in an online mode; facilitates users to leverage the HANA Predictive Analysis Library with help of a user friendly interface that push all the processing operations to the HANA server. The PAL is not installed on HANA by default. To connect them, Install AFL (Application Function Library) at first place on HANA server with help of below commands. Login to Root of HANA server. Extract the files using SAPCAR - SAPCAR -xvf IMDB_AFL100_60_1-10012328.SAR and then navigate into the SAP_HANA_AFL directory which was created in step 2 and execute 'hdbinst' as ~/tmp/SAP_HANA_AFL #./hdbinst It can be downloaded from http://service.sap.com/swdc and once installed you can verify the AFL installation success with commands below: 101 Chapter 5: Conclusion SELECT * FROM "SYS"."AFL_AREAS" WHERE SCHEMA_NAME = '_SYS_AFL' AND AREA_NAME = 'AFLPAL'; SELECT * FROM "SYS"."AFL_PACKAGES" WHERE SCHEMA_NAME = '_SYS_AFL' ANDAREA_NAME = 'AFLPAL'; SELECT * FROM "SYS"."AFL_FUNCTIONS" WHERE SCHEMA_NAME = '_SYS_AFL' AND AREA_NAME = 'AFLPAL'; Add the afl_wrapper_generator and afl_wrapper_eraser procedures if they don't exist. On the HANA server, navigate to the /hanamnt//<SID>/HDB <instance_number>/exe/plugins/afl/ directory and execute the afl_wrapper_generator.sql and afl_wrapper_eraser.sql scripts as HANA user SYSTEM. (An easy way to do this is to open the files in a text editor on the Linux server and copy the code back to HANA studio for execution as the SYSTEM user in a SQL console). You now have two procedures - AFL_WRAPPER_GENERATOR and AFL_WRAPPER_ERASER which are owned by SYSTEM. Grant the EXECUTE privilege on system.afl_wrapper_generator and system.afl_wrapper_eraser to your predictive analysts. For example, if the user name is MyHANAUser, run the commands: GRANT EXECUTE ON system.afl_wrapper_generator to MyHANAUser; GRANT EXECUTE ON system.afl_wrapper_eraser to MyHANAUser. 5.4 Future Work SAP traditional Predictive Analysis tool allows to run data analysis activities but they are manual and thus repetitive and prone to human errors. SAP InfiniteInsight (formerly KXEN) introduce automation to PA activities and allow users to concentrate more on business decisions. It would be interesting to dive in and explore extra potential added to traditional PA. I wanted to implement HANA server connectivity and running PA predictions on both HANA server and local machine to show the difference in performance in figures, but couldn’t because of expensive licenses fees for HANA server. Developing mobile applications to integrate PA results and charts to other applications for better and faster business making. A predictive model which is as an equation, algorithm, or set of rules needed to predict an outcome depending on the input dataset; can simply be a set of business rules based on past observations, and can be developed more accurately using statistically rigorous predictions and statistical algorithms. Future work can also consider making a standard for these models to be used by business communities again just by importing. 5.5 Conclusion In normal daily life even, everyone takes advantage of predictive analytics, in the form of anything from weather forecasts to insurance premiums. Predictive analytics will be used more & more as businesses understand and appreciate the business benefits that this prediction tools 102 Chapter 5: Conclusion bring. Predictive analytics is a subset of data mining but indicates a focus on making predictions. SAP in 2012 announced the launch of SAP Predictive Analysis 1.0, their new solution in the predictive analytics portfolio as a replacement to the classical offering of SAP BO Predictive Workbench. With PA, HANA and HANA native predictive library (PAL) enables the execution of predictive algorithms in-database that is making the procedures running in the DB layer and then exporting just the result set, instead of exporting the whole dataset for the algorithms to run in the application layer. This gave SAP a leading edge and made SAP a big contender in the Big Data Predictive Analytics space. But in actual terms, SAP's actual portfolio was very small. HANA no doubt brought a lot of modern and ground breaking technologies to the game that weren't available before, but in terms of actual functionalities related to analytical models, it was still behind its main competitors SAS, IBM, and Tibco etc. Only a couple of dozen algorithms in PAL were available and definitely were not enough to compete. But the game changed since day SAP announced the R integration which means more than 3500 algorithms are part of library set. This HANA R integration, although powerful, still had disadvantage as it required a very specific set of development skills in order to deliver actual analytical models to the business users. To support these users with very little or almost none technical knowledge, SAP PA 1.0 latest version 11 allows implementing custom R functionalities (i.e. algorithms that weren't built in standard PA) without having to resort to developing HANA SQLScript/R procedures. Users can share their existing R scripts (just adapting it to a function model and can now visually create their analytical models with the most complex algorithms. PA interface and possibility to run analysis on in-memory databases allows analysis of very large amount of data with better performance. The basic working character for predictive analysis is the predicator, that variable whose measured value for an individual or entity gives idea about future behavior. For example, an insurance company could consider age, income, credit history, insurance claims history and other demographics as predictors when issuing an insurance policy to determine an applicant’s risk factor. Predictive model is combination of such multiple predicators which may be helping factor to forecast future possibilities or behaviors with an acceptable value of reliability when they are subjected to an analysis. Model must be kept re-validated and revised as additional data become available for further predictions based on collected data, formulations of statistical models and previous properties of models. PA always go hand in hand with business knowledge and statistical techniques for its full exploitation and prediction objective/insight must be clear before starting the process. It's a good practice to keep ready multiple related predictive models available to be run and applied to dataset for better strategic company decisions. 103 References [1] J. MacGregor, Predictive Analysis with SAP®, SAP Press, 2013. [2] C. Mankala og G. M. V, SAP HANA Cookbook, Packt Publishing, Packt Publishing. [3] I. Gordon, Managing the New Customer Relationship: Strategies to Engage the Social Customer and Build Lasting Value, 2013: Wiley. [4] E. Siegel, Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die, Wiley, 2013. [5] A. Bari, Predictive Analytics For Dummies, For Dummies, 2014. [6] T. W. Miller, Modeling Techniques in Predictive Analytics: Business Problems and Solutions with R, Pearson FT Press, 2013. [7] B. Ratner, Statistical and Machine-Learning Data Mining: Techniques for Better Predictive Modeling and Analysis of Big Data, Second Edition, CRC Press, 2011. [8] C. Carlberg, Decision Analytics: Microsoft Excel, Conrad Carlberg , 2013. [9] M. Kuhn, Applied Predictive Modeling, Springer, 2013. [10 M. Zaki, R. U. N. U. Dept. of Comput. Sci., S. Parthasarathy, W. Li og M. Ogihara, «Evaluation ] of sampling for data mining of association rules,» Research Issues in Data Engineering, 1997. Proceedings. Seventh International Workshop, 1997. [11 G. P.-S. P. S. Usama Fayyad, «From Data Mining to Knowledge Discovery in Databases,» ] Association for the Advancement of Artificial Intelligence (www.aaai.org), 2014. [12 J. S. Park, M.-S. Chen og P. S. Yu, «Efficient parallel data mining for association rules,» CIKM ] '95 Proceedings of the fourth international conference on Information and knowledge management. [13 C. Rygielski, J.-C. Wang og D. C. Yen, «http://www.sciencedirect.com/,» Elsevier Science Ltd, ] [Internett]. Available: http://www.sciencedirect.com/science/article/pii/S0160791X02000386. [Funnet 2014]. [14 V. K. J. R. Q. J. G. Q. Y. H. M. G. J. M. A. N. B. L. P. S. Y. Z.-H. Z. M. S. D. J. H. D. S. Xindong Wu, ] «Top 10 algorithms in data mining,» Springer-Verlag, 2006. 104 References [15 P. J. R. Leonard Kaufman, «Finding Groups in Data,» i An Introduction to Cluster Analysis, ] John Wiley & Sons, 2009, pp. 47-55. [16 P. J. Rousseeuw, «Silhouettes: A graphical aid to the interpretation and validation of cluster ] analysis,» http://dx.doi.org/10.1016/0377-0427(87)90125-7, p. 467. [17 A. Sturn, J. Quackenbush og Z. Trajanoski, «Genesis: cluster analysis of microarray data,» i ] Oxford University Press 2002. [18 C. Fraley og A. E. Raftery, «How Many Clusters? Which Clustering Method? Answers Via ] Model-Based Cluster Analysis,» The Computer Journal (1998), nr. 41 (8): 578-588. [19 A. J. S. a. M. Knott, «A Cluster Analysis Method for Grouping Means in the Analysis of ] Variance,» April 2008. [Internett]. Available: http://www.ime.usp.br/~abe/lista/pdfXz71qDkDx1.pdf. [Funnet 2014]. [20 T. W. H. M. Akihiro Inokuchi, «An Apriori-Based Algorithm for Mining Frequent ] Substructures from Graph Data,» Springer Link, nr. Department of Computer and Information Science, Norwegian University of Science and Technology, pp. 13-23. [21 C. B. ([email protected]), «Efﬁcient Implementations of Apriori,» ] [Internett]. Available: http://www.intsci.ac.cn/shizz/fimi.pdf. [22 Y. Ye, A. Corp. og C.-C. Chiang, «A Parallel Apriori Algorithm for Frequent Itemsets Mining ] 10.1109/SERA.2006.6».Software Engineering Research, Management and Applications, 2006. Fourth International Conference. [23 F. E. H. J. G. J. B. M. E. Y. V. J. F. H. Ewout W Steyerbergemail address, «Internal validation ] of predictive models,» Journal of Clinical Epidemiology, nr. Volume 54, Issue 8, 2000. [24 H. Bliss, «SAP BI Blog, All things Business Intelligence,» [Internett]. Available: ] http://sapbiblog.com/category/predictive-analytics/. [25 D. W. T. H. a. R. T. Gareth James, «Linear Regression,» i An Introduction to Statistical ] Learning with applications in R, Springer. [26 D. Alahakoon, S. Halgamuge og B. Srinivasan, «Dynamic self-organizing maps with ] controlled growth for knowledge discovery,» IEEE, 2002. [27 A. RAUBER, «Self-organizing maps,» [Internett]. ] http://www.ifs.tuwien.ac.at/ifs/research/pub_html/mer_dexa98/node3.html. 105 Available: References [28 F. Y og J. B. Partovi, «Emerald Insight :Using the Analytic Hierarchy Process for ABC ] Analysis,» [Internett]. Available: http://www.emeraldinsight.com/journals.htm?articleid=848735&show=abstract. [29 A. Tanwari, A. Lakhiar og A. Ghulam, «ABC Aanlysis as a Inventory Control Technique,» ] http://www.goiit.com/upload/2009/2/22/e4fac76b66664f7f346c3aaed9feb829_1302799. pdf. [30 I. Ben-Gal, «OUTLIER DETECTION,» http://www.eng.tau.ac.il/~bengal/outlier.pdf. ] [31 «Outlier detection,» [Internett]. ] http://www.molmine.com/magma/global_analysis/outlier_detection.html. [32 C. C. AGGARWAL, OUTLIER ANALYSIS, ] http://www.charuaggarwal.net/outlierbook.pdf. New Available: York USA: [33 M. N. M. S. M. O. Mansur, «Outlier Detection Technique in Data Mining: A Research ] Perspective,» http://eprints.utm.my/3336/1/Mohd_Noor__Outlier_Detection_Technique_in_Data_Mining_A_Research_Perspective.pdf?origin=publication_detail. [34 http://scn.sap.com/docs/DOC-32651, «Official Product Tutorials – SAP Predictive Analysis,» ] SAP. [35 S. D. Zengyou He, «fats Greedy Algorith for ] http://arxiv.org/ftp/cs/papers/0507/0507065.pdf, China University. Outlier Mining,» [36 C. C. A.-N. 4. I. License, «Linear Least Squares Regression,» [Internett]. Available: ] http://www.cyclismo.org/tutorial/R/linearLeastSquares.html. [37 O. Torres-Reyna, «Getting Started in Linear Regression using R,» [Internett]. Available: ] http://www.princeton.edu/~otorres/Regression101R.pdf. [38 H. C. H. J. A. W. A. K. S. a. M. J. v. d. W. Robert A van den Berg*, «Centering, scaling, and ] transformations: improving the biological information content of metabolomics data,» BMC Genomics, [Internett]. Available: http://www.biomedcentral.com/1471-2164/7/142. [39 «Data Collection,» i Centering, scaling, and transformations: improving the biological ] information content of metabolomics data, http://highered.mcgrawhill.com/sites/dl/free/0073373656/639839/doa73656_ch02.pdf. 106 References [40 D. K. a. B. Basturk, «Artiﬁcial bee colony (ABC) optimization algorithm for solving ] constrained optimization problems,» Lecture Notes in Computer Science, pp. 789-821, 2007. [41 B. A. a. C. O. D. Karaboga, «Artiﬁcial bee colony (ABC) optimization algorithm for training ] feed-forward neural networks,» Lecture Notes in Computer Science, pp. 318-329, 2007. [42 D. V. H. D. Morris, «T h e B u s i n e s s V a l u e o f P r e d i c t i v e A n a l y t i c s by IBM,» ] [Internett]. Available: http://www.spss.com.ar/MKT/Promos/2012/0612_PA/0612_businessvalue_PA.pdf. [43 R. D. Kugel, «Business Planning and Predictive Analytics,» [Internett]. Available: ] http://www.ventanaresearch.com/research/article.aspx?id=355. [44 L. M. 8, «Mining Frequent Itemsets – Apriori Algorithm,» [Internett]. Available: ] http://software.ucv.ro/~cmihaescu/ro/teaching/AIR/docs/Lab8-Apriori.pdf. [45 I. Tudor, «Association Rule Mining as a Data ] http://bmif.unde.ro/docs/20081/7%20ITudor.pdf, Romania. 107 Mining Technique,»