Detailed comparison of ModelRisk, @RISK and Crystal Ball

Detailed comparison of
ModelRisk, @RISK and Crystal Ball
Crystal Ball is a product of Oracle Corporation. @RISK is a product of Palisade Corporation.
Disclaimer: All reasonable attempts have been made to ensure that the comparison between ModelRisk and
competitive products described in this document are factually based, fair and accurate. All software was used
according to vendor specifications and versions tested were those available at the time of publication (July 2014).
If you are aware of any inaccuracies or omissions in this text that should be corrected in order to maintain a fair
and accurate comparison, please contact [email protected].
© Vose Software BVBA
www.vosesoftware.com
1|Page
Introduction
This technical review compares the three most common risk analysis Excel add-in software products:
Product
Crystal Ball:
ModelRisk:
@RISK:
Link to official intro video
Here
Here
Here
Company web site
www.oracle.com
www.vosesoftware.com
www.palisade.com
Summary of conclusions
The best product choice depends on the factors you consider most important:










Lowest cost for similar level of features – ModelRisk
Largest choice of distributions – ModelRisk
Offers a choice of correlation tools – ModelRisk
Most closely follows Excel protocol – ModelRisk
Easiest to share results – ModelRisk
Widest range of technical tools – ModelRisk
Includes insurance and finance modeling – ModelRisk
Widest range of results plots – ModelRisk, @RISK
Linking to Microsoft Project in Excel - @RISK
Language other than English - @RISK or Crystal Ball
Approach
The primary data source for this review is this1 independent and up-to-date overview of Excel risk
analysis add-ins on Wikipedia. This review only includes the Wikipedia information where there is a
difference between the software products. Aside from which technical capabilities each product has,
user-experience is also extremely important and a strong measure of the quality of the software as a
whole so, where possible, links have been provided to official videos produced by each developer so you
can compare the approaches. Please do take a look at them. This review is split into the following
sections:








1
Functionality
Technical specifications
Simulation controls
Reporting results
Help file, technical support and training
Implementation within Excel
Precision of the algorithms used in the software
Pricing
Last viewed 13 July, 2014
© Vose Software BVBA
www.vosesoftware.com
2|Page
Functionality
The following chart is plotted from the Wikipedia article which gave a numerical summary of the
number of features each product offered2
The three types of random variables that you tend to see in risk analysis modeling are:



Distributions
Correlation structures
Time series
Distributions - there are something like 150 distribution types that have a practical use and are not
simply a rehashed or renamed version of another one. You could say that 90% of users only need to use
10% of the types of distribution that are known, but which 10% varies enormously between the types of
problems that the users focus on. You can view videos of how the different products fit distributions to
data here: Crystal Ball3, ModelRisk, @RISK.
Correlation structures (copulas) are a very important and much overlooked component of risk modeling.
Failure to incorporate appropriate correlation will usually result in an underestimation of the risk. All of
the products reviewed effectively use a Gaussian copula which allows simulating a correlation structure
between any number of variables4. ModelRisk also provides the Clayton, Gumbel, Frank and Student
2
The table was removed from general view because of Wikipedia’s policy of not including original research (you can track the discussion that
resulted in its removal on this page).
3
Official video not found
4
In fact, @RISK and Crystal Ball uses rank order correlation, a simulation technique from 1982 that closely resembles a Gaussian copula. See
Iman, R. L. and Conover, W. J., (1982). 'A Distribution-Free Approach to Inducing Rank Order Correlation Among Input Variables', Commun
Statist-Simula Computa 11(3) 311-334.
© Vose Software BVBA
www.vosesoftware.com
3|Page
copulas. Bivariate versions can be rotated and all can be statistically fit to data. ModelRisk also offers an
empirical copula that will statistically reproduce unusual data patterns across any number of variables.
Correlating two Uniform distributions using the Gaussian copula with, for example, a 90% correlation
would produce a scatter pattern that looks like this:
But the different copulas will produce markedly different patterns for the same 90% correlation, for
example (reading from the top left: Clayton, inverted Clayton, Frank, Gumbel, inverted Gumbel,
Student):
For example, the Clayton copula (top left plot) is often used by banks for loan defaults because these are
strongly correlated in times of recession, but not in better times. The selection of correlation shape can
have a very significant effect on the tails of results when the correlation level is moderate (e.g. 10% to
50%). You can view the way ModelRisk fits copulas to data here, and for unusual patterns here.
Time series models are used to forecast variables across several periods, like exchange rates, inflation,
sales volumes. They incorporate at a minimum the trend and the level of randomness around that trend.
More sophisticated models incorporate characteristics like sudden jumps, cycles of high and low periods
of volatility, etc. The more sophisticated models are usually fit to data using software. You can view
videos of how the different products fit time series to data here: Crystal Ball Suite, ModelRisk, @RISK.
© Vose Software BVBA
www.vosesoftware.com
4|Page
Technical specifications
This section compares the key differences in the modeling tools that each product offers.
Technical specification
Distribution fitting
Correlation fitting
Time series fitting
Optimizer included
Database connectivity
VBA calls to functions
C++ calls to functions
Six sigma supported
UDF error analysis
Extreme value modeling
Expert elicitation tools
Data previsualizer
ODE and numerical integration
Assumption library
Converters for
@RISK
@RISK
@RISK
Standard Professional Industrial
No
No
No
No
No
No
No
Yes
No
No
Yes
No
No
No
CB
Yes
No
No
No
No
Yes
Yes
Yes
No
No
Yes
No
No
Yes
CB
Yes
No
Yes
Yes
No
Yes
Yes
Yes
No
No
Yes
No
No
Yes
CB
Crystal
Ball
Crystal Ball ModelRisk ModelRisk ModelRisk
Suite
Standard Professional Industrial
Yes
No
Yes
No
No
Yes
No
Yes
No
No
No
No
No
No
No
Yes
No
Yes
Yes
No
Yes
No
Yes
No
No
No
No
No
No
No
No
No
No
No
No
Yes
Yes
No
Yes
No
No
No
No
No
CB, @RISK
Yes
Yes
Yes
Yes
No
Yes
Yes
No
Yes
Yes
Yes
Yes
No
No
CB, @RISK
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
CB, @RISK
Source: Wikipedia
Notes














Distribution fitting – a fundamental requirement unless your risk analysis model is only based on subjective estimates of uncertainty
Correlation fitting – another fundamental requirement unless your risk analysis model is only based on subjective estimates of
uncertainty, or all your variables are unconnected (independent)
Time series fitting – a fundamental requirement if you want to make forecasts based on historic data
Optimizer – allows you to determine the optimal values of variables you control based on some target(s) like maximizing the mean
profit. All three products use the industry leader OptQuest optimizer, and @RISK includes other optimizers
Database connectivity – useful if you have databases with large datasets that you want to fit distributions, correlation structures or
time series and that will update as new data are added to the database
VBA calls – useful if you want to automatically run simulations, or functions that will internally perform simulations or probability
calculations, etc.
C++ calls – for very advanced modelers who want to build their own specialist, high performance simulation programs
Six Sigma functionality – a moderately useful feature for Six Sigma specialists that analyzes simulation results using Six Sigma
metrics
Probability calculations – a very useful capability for advanced probability modelers because, amongst other reasons, one can
combine calculation and simulation together to create more precise and faster models.
User Defined Function error analysis – an error message appears in the spreadsheet cell. For example:
Extreme vale modeling – very useful in certain circumstances - many risk analysis issues are driven by extremes, like predicting the
size of the largest wave or wind gust, the lowest temperature, the largest insurance claim, the longest blackout, etc.
Expert elicitation – tools to help the modeler get the most precise uncertain estimates from subject matter experts. If this is useful
to you, you should investigate each tool’s capabilities because there is a great deal of variation in what they offer
Data previsualizer – allows you to review and explore the data before you start fitting distributions, time series, or correlations.
Again, a large variation in what is offered, so worth investigating the differences if this is important
Differentiation and integration – a very advanced capability for either (i) building a system of ordinary differential equations with
uncertain inputs; or (ii) performing one-dimensional numerical integrations. Useful for scientists, engineers and finance/insurance
people mostly
© Vose Software BVBA
www.vosesoftware.com
5|Page

Assumption library – allows the user to store key assumptions that are used in various models. Again, a very wide range of
capabilities offered, so worth investigating the differences if this is important
Converter – used if one changes to another software product, and wishes to keep using models developed in the old product (CB =
Crystal Ball)

Simulation controls
This section compares the controls that are available to run simulations.
@RISK
@RISK
@RISK Crystal ModelRisk ModelRisk ModelRisk
Standard Professional Industrial Ball
Standard Professional Industrial
Simulation controls
Lock / unlock random variables
Apply specific sample in model
Random number generator
Spreadsheet interpreter
Sampling method
Yes
Yes
Several
No
MC, LHS
Yes
Yes
Several
No
MC, LHS
Yes
No
Yes
No
Several
MCG
No
Yes
MC, LHS MC, LHS
Yes
Yes
Twister
No
MC
Yes
Yes
Twister
No
MC
Yes
Yes
Twister
No
MC
Source: Wikipedia
Notes:





Lock/unlock sampling – switches off the Monte Carlo simulation, replacing random values with a predictable value like the median.
Useful in model auditing as one can compare values between different versions of a model
Apply sample to model – allows the user to review the results data and force Excel to show a particular selected sample. This lets
you check, for example, why a particularly high or peculiar value was produced by the model. It also demonstrates to an auditor that
specific scenarios appear within the results, which is especially useful for banks and insurance companies
Random number generator – advances in random number generators have made this essentially irrelevant. Nearly every product
uses the Mersenne Twister by default.
Spreadsheet interpreter – this creates a compiled version of the Excel model and then runs this instead of within Excel. The result is
a much faster simulation time. Mostly useful if one has a large model that needs optimizing. The drawback is that not all Excel
models can be compiled.
Sampling method – MCS = Monte Carlo simulation, LHS = Latin Hypercube sampling. We describe in a LinkedIn blog why LHS is no
longer important here.
Reporting results
Risk analysis is a decision-making tool. That means that it is vital to be able to share the results of a risk
analysis effectively. The simulation results from a risk analysis model can be complex for a decisionmaker to understand, so a wide variety of customizable graphical reports is essential.
Reporting results
@RISK
@RISK
@RISK Crystal ModelRisk ModelRisk ModelRisk
Standard Professional Industrial Ball Standard Professional Industrial
Free report viewer
Spider
Box
No
Yes
Yes
No
Yes
Yes
No
Yes
Yes
No
No
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
The greatest difference between the products is how they organize the storage and presentation of
results, and particularly how these results can be shared with people who are not licensed users of their
products, which is explained below.
© Vose Software BVBA
www.vosesoftware.com
6|Page
Crystal Ball
Crystal Ball opens a new window for each graph after clicking a ribbon button. Examples:
Image source: Oracle Axoft
Results files can only be opened by another Crystal Ball user, though the charts can be copied as bitmaps
into Word, PowerPoint, etc.
ModelRisk
Results are shown in a separate application called the Results Viewer which opens at the end of the
simulation. The user can create and save an electronic report and that file can then be distributed to
others, without the need to share the model itself. The file can be opened by anyone who has installed
the ModelRisk Results Viewer, which is available free of charge from Vose Software’s web site here.
Image source: Vose Software
Aside from saving on licenses, the main advantages of this approach is that the report is electronic and
completely customizable – labels and color schemes can be edited by the reviewer, new graphs can be
created, plots can be interrogated for – say – the 95th percentile instead of the 90th, and the whole
report can be resaved if desired. Plots can also be copied into Word, PowerPoint and other applications.
© Vose Software BVBA
www.vosesoftware.com
7|Page
@RISK
@RISK opens a new window for each graph after clicking a ribbon button. Examples:
Image source: Palisade
Results files can only be opened by another @RISK user, though the charts can be copied as bitmaps into
Word, PowerPoint, etc.
Help file, technical support and training
Technical support is a vital part of software delivery. Installation of software, particularly ad-ins to other
products, is not always simple because of access restrictions and security policies within a company and
sometimes requires liaising with the IT department. Moving software to a new computer can also be
very frustrating if one does not have ready access to technical support.
For the majority of risk analysis problems, the modeling techniques are not that complex, but some
basic training is really essential to avoid making the most common mistakes. Online videos that teach
the basics of driving the software are a good start, and a comprehensive help file saves a lot of time, but
if you are making multi-million dollar decisions based on a risk analysis model then we recommend
investing in 1-2 days of face-to-face, or at least online, training. We strongly recommend that the
training you purchase is not just about how to use the software. Instead, the course should focus on the
types of problems that you face, and the content tailored to show you the features of the software, the
particular distributions, etc. that you will need. The examples really need to be tailored to your industry
and problems to be effective, since people find it difficult to extrapolate from an example in another
field.
Help file, support
and training
UDFs linked to help file
Online videos
Online training
Onsite training
Language versions
@RISK
Standard
@RISK
Professional
@RISK
Industrial
Crystal
Ball
Yes
Yes
Yes
No
Yes
Yes
Yes
No
Yes
Yes
Yes
No
Yes
Yes
Yes
No
ZH, EN, FR, ZH, EN, FR, DE, ZH, EN, FR,
EN, JA, ES
DE, JA, PT, ES
JA, PT, ES
DE, JA, PT, ES
© Vose Software BVBA
www.vosesoftware.com
ModelRisk ModelRisk ModelRisk
Standard Professional Industrial
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
EN
EN
EN
8|Page
Notes

UDFs linked to help file – this means that when you open a dialog for a function, there is a link that will take you directly to the
relevant help file topic. Excel makes this quite easy for developers to do, and automatically adds a hyperlink to the dialog box, so it is
hard to see why this feature should not be implemented
Implementation within Excel
Risk analysis add-ins for Excel are popular because Excel is essentially ubiquitous across businesses and
government agencies, and the modeling environment is intuitive and easy to use. At the same time,
there are many features like database connectivity, the VBA macro language, pivot tables, etc. that are
used by a small minority of power users but make it possible to develop very sophisticated models if
needed. Microsoft also encourages developers to produce add-ins and have a programming structure to
ensure that the add-ins will perform well with Excel. Excel add-ins for risk analysis therefore make a lot
of sense. Monte Carlo simulation add-ins for Excel reviewed here take two approaches to adding
uncertainty to the model.
Approach 1: Covering a cell with a distribution
The first approach, used in Crystal Ball, is to add an invisible layer to the spreadsheet. One selects a
spreadsheet cell that contains a value (not a formula), selects a distribution from the menu – perhaps
with links to other cells, and clicks OK to enter the distribution. The cell background turns green, but the
original number that was in that cell remains unchanged. Some people like this because one can share
the model with non-Crystal Ball users and it still works as the original ‘unrisked’ spreadsheet model,
although the simulation component is lost unless the other user has Crystal Ball too. It has its dangers,
however. You can enter a value of 100 in the cell, for example, and ‘cover’ it with a uniform distribution
between 0 and 1. The incompatibility is not detected by the software and the distribution is only visible
if you click on a Crystal Ball button. The lack of visibility also makes it more difficult to audit and check a
model. Links within Crystal Ball are invisible to Excel, and Excel’s auditing tools do not work.
Approach 2: Inserting a random sampling function
The second approach is to provide user-defined functions, or UDFs. This is the standard method of
creating extra Excel functionality envisioned by Microsoft and employed by ModelRisk and @RISK. For
example, the following UDFs will take a random sample from a Triangular distribution with minimum,
most likely and maximum of 0, 40 and 100 respectively:
ModelRisk:
@RISK:
=VoseTriangle(0,40,100)
=RiskTriang(0,40,100)
The use of UDFs makes it possible to trace through the logic of the model using Excel’s Formula Auditing
tools as illustrated below:
© Vose Software BVBA
www.vosesoftware.com
9|Page
A fundamental Excel rule is that there is a hierarchy of calculation in any formula. For example, in this
formula:
=SUM(SQRT(A1), A2)
Excel will evaluate A1 and A2, then evaluate SQRT(A1), and finally evaluate the SUM.
These rules are critical to having a predictable calculation outcome. However, this presents problems for
developers in creating functions that perform some operation on a distribution. For example, consider
the following @RISK function:
=RiskCompound(5, RiskLognorm(10,1))
The function sums 5 independent samples from a Lognormal(10,1) distribution. However, if we follow
Excel’s rules, the RiskLognorm function – which is a random sampling function - must be evaluated
before the RiskCompound function. So, we’d get something like this as an intermediary calculation:
=RiskCompound(5, 1.032)
in which case the RiskCompound would just be adding five 1.032 values together. To avoid this, @RISK
suppresses the evaluation of the RiskLognorm function. This causes problems of predicting how the
function behaves. For example, it is not immediately evident how the RiskCompound function should
behave in these circumstances:
=RiskCompound(5, RiskUniform(RiskUniform(2,3),4))
=RiskCompound(5, A2), where A2: =RiskNormal(A1,2)
ModelRisk’s approach is to use different functions when defining a distribution, called ‘Objects’, not the
same function used for sampling. Their equivalent function works as follows:
= VoseAggregateMC(5, VoseLognormalObject(10,1))
The LognormalObject function is used to define the distribution we are intending to use. Object
functions allow ModelRisk to comply with Excel’s evaluation rules. They are used in a variety of
ModelRisk’s tools.
© Vose Software BVBA
www.vosesoftware.com
10 | P a g e
Pricing
All software vendors offer different editions of their products. In the following table, the products are
sorted according to price in US$ for a single perpetual (i.e. not time limited) license including the first
year’s maintenance and support. These prices appear on the following web sites, accessed on 10 July
2014: Crystal Ball5, ModelRisk6, @RISK7.
Price comparison
Perpetual license + 1st year support
Cost of additional year's support
# CPUs supported
Single click function view
Distribution fitting
Links to Microsoft Project
Aggregate modeling
Markov Chain simulation
Multivariate time series
Dynamic sorting
Optimizer
Stochastic data sharing
Specialist financial tools
Specialist insurance tools
Bayesian model averaging
ModelRisk
Standard
@RISK
Standard
Crystal Ball
@RISK
ModelRisk
Professional Professional
$895
$161
$1,195
?
$1,213
$218
$1,595
?
All
Yes
No
No
No
No
No
No
No
No
No
No
No
2
No
No
No
No
No
No
No
No
No
No
No
No
?
No
Yes
No
No
No
No
No
No
No
No
No
No
2
No
Yes
Yes
Yes
No
No
No
No
No
No
No
No
@RISK
Industrial
ModelRisk
Industrial
Crystal Ball
Suite
$1,645
$296
$2,195
?
$2,295
$413
$2,433
$438
All
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
No
No
No
No
All
No
Yes
Yes
Yes
No
No
No
Yes
No
No
No
No
All
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
?
No
Yes
No
No
No
No
No
Yes
No
No
No
No
Notes

Some additional factors have been included that are not already described in previous sections. The choice is somewhat subjective
but focused on features useful to a modeler.
5
https://shop.oracle.com/pls/ostore/product?p1=oraclecrystalball&p2=&p3=&p4=&sc=ocom_crystalball
www.vosesoftware.com/purchasing.php
7
www.palisade.com
6
© Vose Software BVBA
www.vosesoftware.com
11 | P a g e