MongoDB Management Suite Manual

Transcription

MongoDB Management Suite Manual
Release 1.3
MongoDB, Inc.
June 23, 2015
Contents
1
2
3
Install On-Prem MMS
1.1 Install the On-Prem Monitoring Server . . . . . . . . . . . . . . . . . . . . . . . .
Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Installation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Configure On-Prem MMS Server . . . . . . . . . . . . . . . . . . . . . . . . . . .
Start and Stop the On-Prem MMS Server . . . . . . . . . . . . . . . . . . . . . .
Advanced Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Configure On-Prem MongoDB Management Service Jetty Instances to use HTTPS
Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
3
3
4
7
8
9
12
12
12
13
On-Prem MongoDB Management Service
2.1 User and Environment Management . . . . . . . . . . . . . . . . . . . . . .
User Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Working with Multiple Environments . . . . . . . . . . . . . . . . . . . . .
2.2 Authentication Requirements . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 On-Prem MongoDB Management Service Administration Interface . . . . .
Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Sections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4 Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
My Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Group Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5 Frequently Asked Questions About On-Prem MongoDB Management Service
User and Group Management . . . . . . . . . . . . . . . . . . . . . . . . . .
Alerts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
About On-Prem MongoDB Management Service . . . . . . . . . . . . . . .
2.6 On-Prem MongoDB Management Service Server Changelog . . . . . . . . .
On-Prem MongoDB Management Service Server 1.3.1 . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
15
15
15
15
16
17
17
17
19
19
20
21
21
22
22
23
23
23
23
On Prem MMS Monitoring
3.1 Getting Started with On Prem MMS Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Install Monitoring Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Monitor Hosts with On Prem MMS Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23
24
24
34
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3.2
3.3
3.4
3.5
3.6
3.7
3.8
Connect to Hosts with Kerberos Authentication . . . . . . . .
Update On-Prem MongoDB Management Service . . . . . . .
Configure On Prem MMS Monitoring . . . . . . . . . . . . .
Using the MMS Console . . . . . . . . . . . . . . . . . . . .
Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Alerts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Dashboards . . . . . . . . . . . . . . . . . . . . . . . . . . .
Host Statistics . . . . . . . . . . . . . . . . . . . . . . . . . .
Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Diagnostic and Troubleshooting Guide . . . . . . . . . . . . .
Getting Started Checklist . . . . . . . . . . . . . . . . . . . .
Installation . . . . . . . . . . . . . . . . . . . . . . . . . . .
Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Monitoring Server . . . . . . . . . . . . . . . . . . . . . . . .
Munin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Frequently Asked Questions About On Prem MMS Monitoring
Host Configuration . . . . . . . . . . . . . . . . . . . . . . .
On Prem MMS Monitoring Agent . . . . . . . . . . . . . . .
Data Presentation . . . . . . . . . . . . . . . . . . . . . . . .
Alerts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
On Prem MMS Monitoring Reference . . . . . . . . . . . . .
Host Types . . . . . . . . . . . . . . . . . . . . . . . . . . .
Host Process Types . . . . . . . . . . . . . . . . . . . . . . .
Event Types . . . . . . . . . . . . . . . . . . . . . . . . . . .
Alert Types . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chart Colors . . . . . . . . . . . . . . . . . . . . . . . . . . .
Database Commands Used by the Monitoring Agent . . . . .
PyMongo Error Codes . . . . . . . . . . . . . . . . . . . . .
Browsers Supported by the MMS Console . . . . . . . . . . .
On Prem MMS Monitoring Agent Changelog . . . . . . . . .
Monitoring Agent 1.6.3 . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
35
36
38
40
41
43
43
47
47
50
50
50
50
51
52
52
53
53
54
54
55
55
56
59
59
59
60
60
60
60
62
62
62
62
62
63
63
63
63
63
63
63
On-Prem MongoDB Management Service is a package for managing MongoDB deployments. On-Prem MongoDB
Management Service provides MMS Monitoring and MMS Backup, which helps users optimize clusters and mitigate
operational risk.
On-Prem MongoDB Management Service High level concepts that apply to all On-Prem MongoDB Management
Service components.
Authentication Requirements Details the permissions required to use MMS agents with MongoDB 2.4 instances with access control.
2
User and Environment Management Details how to manage users and groups in MMS.
Settings Lists and explains the MMS Settings page.
On Prem MMS Monitoring High level overview of issues related to On Prem MMS Monitoring.
Getting Started with On Prem MMS Monitoring Installation and configuration tutorials for On Prem MMS
Monitoring.
Monitor Hosts with On Prem MMS Monitoring Describes how On Prem MMS Monitoring monitorings hosts
and discusses monitoring architecture options.
Update On-Prem MongoDB Management Service Manually update the Monitoring agent on UNIX/Linux and
Windows.
Configure On Prem MMS Monitoring Discusses configurable options with On Prem MMS Monitoring: hardware monitoring with Munin-Node and using On Prem MMS Monitoring with MongoDB instances running with SSL.
Using the MMS Console Discusses the pages of the MMS Console and their functionality.
Diagnostic and Troubleshooting Guide Troubleshooting advice for common issues encountered with On Prem
MMS Monitoring.
Frequently Asked Questions About On Prem MMS Monitoring On Prem MMS Monitoring frequently asked
questions.
On Prem MMS Monitoring Agent Changelog Change log detailing new features and updates for each Monitoring Agent release.
On Prem MMS Monitoring Reference Reference for types of hosts, databases, and other states that may occur
in On-Prem MongoDB Management Service.
1 Install On-Prem MMS
Install the On-Prem Monitoring Server Install the monitoring component of MMS.
Configure On-Prem MongoDB Management Service Jetty Instances to use HTTPS Configure the Jetty server that
runs the core MMS application to use HTTPS.
1.1 Install the On-Prem Monitoring Server
Requirements
Hardware
To run the On-Prem Monitoring server, you must use a 64-bit server, with requirements according to the following
table:
Number of Monitored Hosts
Up to 400 monitored hosts
Up to 2000 monitored hosts
More than 2000 hosts
CPU Cores
4+
8+
Contact MMS
RAM
15 GB
15 GB
Storage Capacity
200 GB
500 GB
Storage IOPS/s
500
10000+ (SSD)
For reference: an AWS EC2 Standard Extra Large (i.e. m1.xlarge) with a provisioned 500 IOP/s EBS volume supported the 400-host configuration above. An AWS EC2 High I/O Quadruple Extra Large (hi1.4xlarge) supported the
2000 host configuration above.
For the best results, On-Prem MMS instances require SSD-backed storage.
3
Software
On Prem MMS Monitoring has the following required dependencies:
• 64-bit Linux. On-Prem MMS servers must run any of the following distributions:
– CentOS 5 or later,
– Red Hat Enterprise Linux 5, or later, or
– SUSE 11 or Later,
– Amazon Linux AMI (latest version only,)
– Ubuntu 12.04 or later.
• MongoDB 2.2.0 or later.
• SMTP server or other email integration option.
While many Linux server-oriented distributions include a local SMTP server by default (e.g. Postfix, Exim, Sendmail,)
you may also configure On Prem MMS Monitoring to send mail via 3rd party providers including Gmail and Sendgrid.
On Prem MMS Monitoring requires email for fundamental server functionality such as password reset and alerts.
On Prem MMS Monitoring has the following optional dependencies.
• A Twilio API account for SMS alerting integration.
• A Graphite hostname / port for charting the MMS server’s internal health.
• An SNMP trap receiver for periodic heartbeat traps about MMS server’s internal health.
Browsers
On Prem MMS Monitoring supports the following browsers:
• recent versions of Firefox, Chrome, and Safari
• Internet Explorer, Versions 9 and later.
The On Prem MMS Monitoring application will display a warning on non-supported browsers.
Installation Process
Overview
At a high level, a basic installation will look like the following. The estimated setup time is less than an hour.
1. Install a standalone local MongoDB server backed by a fast, large storage volume.
2. Install an SMTP email server as appropriate for your environment.
3. Install the MMS server RPM package.
4. Configure the MMS server’s URL and email addresses.
5. Start up MMS server.
4
Prepare Server
1. For AWS users, prepare MongoDB Storage:
If you are not using AWS, skip this step and continue to the next step.
If using an AWS EBS volume for MongoDB storage, create and attach the volume to your EC2 instance. Once
the volume is successfully attached, issue the following command to determine the name of the new EBS
volume:
sudo fdisk -l
Create a filesystem on this volume using the name you found in the previous, command, using the following
form:
sudo mkfs -t ext4 /dev/xvd<letter>
Replace <letter> with the identifier for the volume, as in the following example:
sudo mkfs -t ext4 /dev/xvdf
You only need to create a filesystem the first time you initiate the drive.
Create a directory to use as the mount point:
sudo mkdir /mnt/ebs-mount-dir
Mount the volume with a command that resembles the following:
sudo mount /dev/xvd<letter> /mnt/ebs-mount-dir
Replace <letter> with the identifier for the volume, as in the following example:
sudo mount /dev/xvdf /mnt/ebs-mount-dir
2. Set Linux Kernel parameters. All users must complete this step to ensure optimal performance. Begin by using
the following commands to change the parameters of running instance:
sudo /sbin/sysctl -w net.core.netdev_max_backlog=30000
sudo /sbin/sysctl -w net.core.wmem_max=16777216
sudo /sbin/sysctl -w net.core.rmem_max=16777216
Edit the /etc/sysctl.conf file and append the lines below to ensure that these parameters are always
applied following a system reboot:
net.core.netdev_max_backlog = 30000
net.core.wmem_max = 16777216
net.core.rmem_max = 16777216
Install and Start MongoDB
This section assumes you’re installing MongoDB on an instance running Red Hat, CentOS, Fedora, or Amazon Linux:
Use the Install Mongodb on Red Hat, CentOS, or Fedora Linux tutorial for more information.
1. Add MongoDB repositories to the system’s package management tool.
Create the /etc/yum.repos.d/mongodb.repo file and add the following information about the repository:
5
[mongodb]
name=MongoDB Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64
gpgcheck=0
enabled=1
2. Install the MongoDB packages using the following operations:
sudo yum install mongo-10gen mongo-10gen-server
3. Configure data and logging directories for MongoDB.
Create directories for MongoDB’s log and data. This example assumes that the path for all MongoDB data
is beneath http://docs.opsmanager.mongodb.com//mnt/ebs-mnt-dir/mongo/. Use the following commands:
sudo mkdir -p /mnt/ebs-mount-dir/mongo/data
sudo mkdir -p /mnt/ebs-mount-dir/mongo/logs
Issue the following command to ensure that the mongod process owns all paths below the
http://docs.opsmanager.mongodb.com//mnt/ebs-mnt-dir/mongo/ path:
sudo chown -R mongod:mongod /mnt/ebs-mount-dir/mongo
Edit the /etc/mongod.conf to include the following settings:
logpath=/mnt/ebs-mount-dir/mongo/logs/mongod.log
dbpath=/mnt/ebs-mount-dir/mongo/data
This will configure the paths for the log and data directories. Adjust the paths as needed.
4. Start MongoDB.
Issue the following command:
sudo /etc/init.d/mongod start
Note: If using EBS, starting MongoDB may take several minutes to pre-allocate the journal files. This is
normal behavior.
Obtain and Install On-Prem MMS Server
Note: Contact a MongoDB representative to obtain the current stable On Prem MMS Monitoring release. MMS is
available in DEB packages for Debian, Ubuntu, and related systems; RPM for Red Hat Enterprise Linux, CentOS,
Fedora, and related systems; as well as tar.gz and zip packages.
You can also download the latest On-Prem Monitoring releases from <http://www.mongodb.com/commercialsupport/downloads>
as an RPM package.
RPM Install
Install the RPM by issuing command in the following form:
sudo rpm -ivh mongodb-mms-<version>.x86_64.rpm
6
Replace <version> with the version of the .rpm you obtained. When installed, the base directory for the MMS
software is /opt/mongodb/mms/. The RPM will also create a new system user mongodb-mms under which the
server will run.
DEB Install
Install the DEB by issuing command in the following form:
sudo dpkg --install mongodb-mms-<version>.x86_64.deb
Replace <version> with the version of the .deb you obtained. When installed, the base directory for the MMS
software is /opt/mongodb/mms/. The DEB package will also create a new system user mongodb-mms under
which the server will run.
tar.gz and zip Install
You can install On Prem MMS Monitoring from the provided tar.gz or zip archive without making any changes
to the underlying system (i.e. without creating users.) To install, extract the package, as in the following command:
tar -zxf mongodb-mms-<version>.x86_64.tar.gz
Optionally create a symlink in /etc/init.d to the included control script for convenience, as in the following:
sudo ln -s <install_dir>/bin/mongodb-mms /etc/init.d/
Note, when the app is first started, it will create and store an encryption key in $HOME/.mongodb-mms for the app
user.
Configure On-Prem MMS Server
This section describes the required configuration for the On-Prem MMS Server. See Advanced Configuration for more
configuration information.
Required Configuration
Note: By default, On Prem MMS Monitoring will use a local SMTP server listening on port 25.
Configure MMS properties, by editing the <install_dir>/conf/conf-mms.properties file. Edit the following properties according to the needs of your deployment, as in the following example:
mms.centralUrl=http://mms.example.com:8080
mms.fromEmailAddr=MMS Alerts <[email protected]>
[email protected]
mms.adminFromEmailAddr=MMS Admin <[email protected]>
[email protected]
[email protected]
These properties are blank initially, and you must define them before the On Prem MMS Monitoring instance will
start.
In addition, whether or not you enable backup, define these properties if you use authentication:
7
mongo.mongoUri=<SetToValidUri>
mongo.replicaSet=<ValidRSIfUsed>
Otherwise, MMS will fail while trying to connect to the default 127.0.0.1:27017 URL.
If you use the MMS <install_dir>/bin/credentialstool to encrypt the password used in
the mongo.mongoUri value,
also add the mongo.encryptedCredentials key to the
<install_dir>/conf/conf-mms.properties file and set the value for this property to true:
mongo.encryptedCredentials=true
For more details, see Authentication Configuration below.
Configure Email Authentication
Please refer to your SMTP provider’s documentation for the appropriate settings or for how to configure a local SMTP
server as a relay. You may configure authentication if you want to send mail using existing email infrastructure (i.e.
SMTP,) or a service such as Gmail or Sendgrid .
Set the following value in the <install_dir>/conf/conf-mms.properties file:
mms.emailDaoClass=com.xgen.svc.core.dao.email.JavaEmailDao
Then, to the same file, set the following values as defined by your provider. Defaults specified inline:
mms.mail.transport=<smtp/smtps> # (defaults to smtp)
mms.mail.hostname=<mail.example.com> # (defaults to localhost)
mms.mail.port=<number> # (defaults to 25)
mms.mail.tls=<true/false> # (defaults to false)
The following two values are optional, and unless set default to disabled authentication:
mms.mail.username=
mms.mail.password=
Optional: AWS Simple Email Service Configuration
Set the following value in <install_dir>/conf/conf-mms.properties to configure integration with
AWS’s Simple Email Service (SES:)
mms.emailDaoClass=com.xgen.svc.core.dao.email.AwsEmailDao
To configure this integration you must also provide your AWS account credentials in the following two properties:
aws.accesskey=
aws.secretkey=
Start and Stop the On-Prem MMS Server
After configuring your On Prem MMS Monitoring deployment, you can start the MMS server with the following
command: 1
1 If you installed from a tar.gz or zip archive, you must create a symlink located at the path /etc/init.d/mongodb-mms that points
to the <install_dir>/bin/mongodb-mms.
8
sudo /etc/init.d/mongodb-mms start
The On-Prem MMS server logs its output to a logs directory inside the installation directory. You can view this log
information with the following command:
sudo less <install_dir>/logs/mms0.log
If the server starts successfully, you will see content in this file that resembles the following:
[main]
[main]
[null]
[main]
[main]
[main]
[main]
[main]
[main]
INFO ServerMain:202 - Starting mms...
WARN AbstractConnector:294 - Acceptors should be <=2*availableProcessors: SelectChannelConnec
LoginService=HashLoginService identityService=org.eclipse.jetty.security.DefaultIdentityServic
INFO AppConfig:46 - Starting app for env: hosted
INFO MmsAppConfig:67 - Not loading backup components
INFO GraphiteSvcImpl:67 - Graphite service not configured, events will be ignored.
INFO TwilioSvcImpl:48 - Twilio service not configured, SMS events will be ignored.
INFO OpenDMKSnmpTrapAgentSvcImpl:91 - SNMP heartbeats hosts not configured, no heartbeat trap
INFO ServerMain:266 - Started mms in: 24979 (ms)
You can now use the On Prem MMS Monitoring instance by visiting the URL specified in the mms.centralUrl
parameter (e.g. http://mms.example.com:8080) to continue configuration:
Unlike the SaaS version of MMS, On Prem MMS Monitoring stores user accounts in the local MongoDB instance.
When you sign into the On Prem MMS Monitoring instance for the first time, the system will prompt you to register
and create a new “group” for your deployment.
After completing the registration process, you will arrive at the “MMS Hosts,” page.
Because there are no Monitoring agents attached to your account, the first page you see in On Prem MMS Monitoring
will provide instructions for downloading the Monitoring agent. Click the “download agent” link to download a preconfigured agent for your account. Continue reading this document for installation and configuration instructions for
the MMS agent.
Stopping the On-Prem MMS server is as follows:
sudo /etc/init.d/mongodb-mms stop
Advanced Configuration
Change Port Number
1. Edit <install_dir>/conf/conf-mms.properties:
mms.centralUrl=http://mms.acmewidgets.com:<newport>
2. Edit <install_dir>/conf/mms.conf
BASE_PORT=<newport>
3. Restart MMS server:
sudo <install_dir>/bin/mongodb-mms restart
Run as Different User
1. Edit <install_dir>/conf/mms.conf:
9
MMS_USER=foo_user
2. Change Ownership of <install_dir> for new user:
sudo chown -R foo_user:foo_group <install_dir>
3. Restart MMS server:
.. code-block:: sh
sudo <install_dir>/bin/mongodb-mms restart
Replication Configuration
The
backing
MongoDB
store
uses
a
connection
<install_dir>/conf/conf-mms.properties directory.
string
URI
defined
in
the
Edit conf-mms.properties to define the replication hosts. For example:
mongo.mongoUri=mongodb://host1:40000,host2:40000,host3:40000/?maxPoolSize=100
mongo.replicaSet=mmsreplset
See Connection String URI Format for more information.
Authentication Configuration
For standalone MongoDB nodes running with user authentication, simply add the username and password credentials
to the mongoUri, and specify the database as admin. For example:
mongo.mongoUri=mongodb://mongouser:[email protected]:40000/admin?maxPoolSize=25
mongo.replicaSet=mmsreplset
Important: You must modify every mongoURI connection string in the conf-mms.properties file.
This does require that you store credentials in plain text; however, following standard practice you may reduce the
permissions of the configuration file:
sudo chmod 600 <install_dir>/conf/conf-mms.properties
If you do not want to store credentials in plain text, On Prem MMS Monitoring provides a tool to encrypt the MongoDB
credentials. To encrypt authentication credentials:
1. Navigate to the On Prem MMS Monitoring server installation directory.
2. Issue the following command to create an encrypted credential pair, replacing <username> with your username:
bin/credentialstool --username <username> --password
This will prompt you to enter the password and will output the encrypted credential pair.
3. Copy the encrypted credential pair into the MongoURI connection strings of the
conf/conf-mms.properties file where needed, and add the encryptedCredentials = true
configuration option to indicate to MMS that the credentials are set as encrypted tokens.
The added line(s) should resemble the following:
10
Important: You must modify every mongoURI connection string in the conf-mms.properties file.
Optional: Configure a Required reCaptcha for user Registration
To enable reCaptcha anti-spam test on new user registration, sign up for a reCaptcha account and provide the API
credentials in the following two properties:
reCaptcha.public.key=
reCaptcha.private.key=
Optional: Configure Twilio SMS Alert Support
To receive alert notifications via SMS, signup for a Twilio account at http://www.twilio.com/docs/quickstart and enter
your account ID, API token, and Twilio phone number into the following properties:
twilio.account.sid=
twilio.auth.token=
twilio.from.num=
Optional: Configure SNMP Heartbeat Support
New in version 1.3.0.
The MIB file is available for download at: http://downloads.mongodb.com/on-prem-monitoring/MMS-MONGODBMIB.txt
You can configure the On-Prem MMS Server to send a periodic heartbeat trap notification (v2c) that contain an internal
health assessment of the MMS Server.
The MMS Server can send traps to one or more endpoints on the standard SNMP UDP port 162.
There are three configuration options that affect the heartbeat behavior:
# Listening UDP port for SNMP. (Note: Setting to less than 1024 will require running MMS server with
snmp.listen.port=11611 #default
# Period in seconds between heartbeat notifications
snmp.default.heartbeat.interval=300 #default
# Optional comma-separated list of hosts where 'heartbeat' traps will be sent on standard UDP port 16
# Leaving blank (the default) disables the SNMP heartbeat functionality
snmp.default.hosts=
Optional: Configure Advanced MMS User Password Management
You can configure the password policy for MMS user accounts with the following properties:
11
# The number of previous passwords to remember. A remembered password can not be reused.
mms.password.minChangesBeforeReuse=
# The number of failed login attempts before an account becomes locked. An account can only be
# unlocked by an MMS Administrator.
mms.password.maxFailedAttemptsBeforeAccountLock=
# The number of days before an account that has not visisted the MMS website should be locked.
mms.password.maxDaysInactiveBeforeAccountLock=
# The number of days a password can be in use before it must be changed.
mms.password.maxDaysBeforeChangeRequired=
Optional: Configure Advanced MMS User Session Management
You can configure the number of hours before a session on the MMS website expires with the following property:
mms.session.maxHours=
Optional: Configure Jetty to use HTTPS Connections
You can configure the Jetty services that run the On-Prem MongoDB Management Service application to use HTTPS.
See Configure On-Prem MongoDB Management Service Jetty Instances to use HTTPS for more information.
1.2 Configure On-Prem MongoDB Management Service Jetty Instances to use
HTTPS
Overview
You can optionally configure the Jetty instances that serve the On-Prem MongoDB Management Service application
to use HTTPS to encrypt connections between the MMS application and the MMS agent as well as the web interface.
Alternately, you can provide access to the MMS application using a load balancer that provides HTTPS access.
Prerequisites
Before configuring MMS and Jetty, you must create and prepare a valid SSL certificate. You must also have access to
the openssl library on your system:
Generate Certificate
Existing Certificate Signed by trusted Certificate Authority If you have an existing certificate, you do not need
to generate a new certificate. Proceed to the Prepare the Certificate section to complete the prerequisites.
Create New Certificate and Signing Request for a 3rd Party Certificate Authority
1. Issue the following command at the system prompty to create a new certificate and certificate signing request
(CSR):
12
openssl req -new -out mms-ssl.csr -newkey rsa:2048 -keyout mms-ssl.key
openssl will prompt you to answer questions for the certificate’s meta data. Complete all prompts. The
Common Name must have the same hostname value as the mms.centralUrl configuration.
Refer to the instructions provided by the certificate authority to ensure that they do not have any more requirements for the certificate signing authority or the certificate meta data.
2. Submit your new CSR to the 3rd-party certificate authority. The certifiacte authority will return a signed certificate.
Each certificate authority may have a different certificate signing procedure.
When you have a signed certificate, proceed to the Prepare the Certificate section to complete the prerequisites.
Self-Signed Certificate To generate a self-signed certificate, issue the following command at the system prompt.
openssl req -x509 -days 3650 -newkey rsa:2048 -keyout mms-ssl.key -out mms-ssl.crt
openssl will prompt for a private key passphrase, and for the answers to questions for the certificate’s meta data.
Complete all prompts. The Common Name must have the same hostname value as the mms.centralUrl configuration.
Prepare the Certificate
Create PEM Certificate If the signed certifcate is PEM format, convert the certificate to PEM format with the
following command:
openssl x509 -in mms-ssl.cer -inform DER -outform PEM -out mms-ssl.crt
If the certificate authority uses a certificate chain, concatenate the certificates together to create a unified certificate,
with a command that resembles the following:
cat mms-ssl.crt <intermediate-certificate> <root-certificate> > mms-ssl-unified.crt
Replace <intermediate-certificate> with the intermediate certificate chain and <root-certificate>
with the certificate authority’s root certificate.
Create PKCS12 Certificate Combine the private key and signed certificate, or certificate chain, into a PKCS12formatted keystore with the following command:
openssl pkcs12 -inkey mms-ssl.key -in mms-ssl-unified.crt -export -out mms-ssl.pkcs12
openssl will prompt for the private key passphrase as well as a new passphrase for the PKCS12 keystore.
Procedure
Create Java Keystore
Generate Keystore Convert the PKCS12 keystore into a Java Keystore, so that the MMS server can access the
required SSL infrastructure:
/opt/mongodb/mms/jdk/bin/keytool -importkeystore \
-srckeystore mms-ssl.pkcs12 \
-srcstoretype PKCS12 \
-destkeystore mms-keystore.jks
13
Replace /opt/mongodb/mms with the path of the installation directory for the MMS server.
/opt/mongodb/mms is the default installation directory.
keytool will prompt for the PKCS12 keystore passphrase, as well as for a passphrase for the new Java keystore. You can use the same passphrase for Java as for PKCS12 because you can delete the PKCS12 file (i.e.
mms-ssl.pkcs12) after generating the Java Key store.
Set Keystore Location and Permission Issue the following sequence of commands to move the Java keystore file
to the /etc/mongodb-mms directory and set the appropriate permissions:
sudo cp -a mms-keystore.jks /etc/mongodb-mms/ sudo chown mongodb-mms:root /etc/mongodbmms/mms-keystore.jks sudo chmod 600 /etc/mongodb-mms/mms-keystore.jks
If the MMS application server runs as a different user, change mongodb-mms in the chown command as needed.
Generate Credentials
Issue the following command to generate a credential pair for the MMS application to use to access the Java Keystore:
/opt/mongodb/mms/bin/credentialstool --username keystore --password
Replace /opt/mongodb/mms with the path of the installation directory for the MMS server.
credentialstool will return output that resembles the following:
Your encrypted credentials pair:
Username: abcdef1234567890-76d41ae0a98c
Password: abcdef1234567890-2cc28e525d1f543464
Configure MMS Application to use SSL
Edit the mms.conf (e.g. /opt/mongodb/mms/conf/mms.conf) file and add the following options:
JAVA_MMS_SSL_OPTS="${JAVA_MMS_SSL_OPTS} -Dxgen.webServerSslEnabled=true"
JAVA_MMS_SSL_OPTS="${JAVA_MMS_SSL_OPTS} -Dxgen.webServerSslKeyStorePath=/etc/mongodb-mms/mms-keystore
JAVA_MMS_SSL_OPTS="${JAVA_MMS_SSL_OPTS} -Dxgen.webServerSslKeyStoreEncryptedPassword=abcdef1234567890
Modify the values as required by your configuration. Edit the conf-mms.properties file to change the
mms.centralUrl value to reflect the new HTTPS information. For example:
mms.centralUrl=https://mms.example.net:8443
Restart MMS Application Server
Before you can access MMS using an HTTPS connection you must restart the MMS application server:
You can now connect to MMS by accessing the following URL in a web browser:
https://mms.example.net:8443
14
2 On-Prem MongoDB Management Service
The On-Prem Monitoring agent is at the core of MMS. You must install the Agent before you can begin monitoring
your MongoDB deployments. If you have not yet installed the On Prem MMS Monitoring Agent, see Getting Started
with On Prem MMS Monitoring.
The following pages discuss management tools and authentication requirements for On-Prem MongoDB Management
Service.
Authentication Requirements Details the permissions required to use MMS agents with MongoDB 2.4 instances with
access control.
On-Prem MongoDB Management Service Administration Interface Lists and explains the MMS Administration
page.
Frequently Asked Questions About On-Prem MongoDB Management Service On-Prem MongoDB Management
Service frequently asked questions.
On-Prem MongoDB Management Service Server Changelog A list of changes for the On-Prem MongoDB Management Service server.
2.1 User and Environment Management
This page helps you manage users that have access to your MMS groups, and also enables you to create and manage
groups.
User Management
You can grant users access to your group in the “Users” section of the MMS console. Click on the + ADD USER icon
next to your account name and enter their email in the dialogue. If the console finds a connected account, MMS will
automatically add the user to the MMS console.
Note: With MongoDB Management Service On-Prem, user accounts and groups are independent from JIRA. This is
in contrast to the MongoDB Management Service, which shares account and group information with the MongoDB
JIRA instance.
Users can create accounts at any time using the account registration page of your MMS installation.
Working with Multiple Environments
If you have multiple MongoDB systems in distinct environments and cannot monitor all systems with a single agent,
you will need to add a new group. Having a second group makes it possible to run two agents, because the API and
secret keys are unique to the group.
You may also use a second group and agent to monitor a different set of MongoDB instances in the same environment
if you want to segregate the hosts within the MMS console. A user can only view data from the hosts monitored in a
single group at once.
After adding a second group, the MMS interface will have a drop down list that will allow you to change groups.
Selecting a new group will refresh the current page with the data available from the servers in this group.
15
Create Group
To add a new group click on the “Add New Group” link on the upper left hand side of the “Users” page. Then, specify
the new group name and select the Add New Group button to create the new group. Group names must be unique
and you cannot reuse a group name. After creating a group, only the current user account is a member that group. To
add more people, click on the + Add/Invite User button and complete the form.
Rename Group
If you want to change the name of a group, click on the “Edit Group Name” button (with the pencil icon) near your
group name at the top of the “Users” page of the MMS console. Group names must be unique and you cannot reuse a
group name.
Remove Group
You can remove users from a group at any time from the Users tab, by clicking on the “garbage can” icon.
Although it’s not possible to delete a group, you can remove a group by removing all users from the group, and then
remove your own account from the group. The group will then be orphaned. It will remain for auditing purposes, but
be otherwise inaccessible.
2.2 Authentication Requirements
On-Prem MongoDB Management Service requires elevated privileges to collect complete data from MongoDB 2.4
instances with authentication enabled.
The following table lists the commands needed to grant the required permissions to the agent user. For an explanation
of what each privilege role means, see: User Privilege Roles in MongoDB.
Important: To properly configure the agent user on a sharded cluster, you must create the user on the mongos
and on each shard’s replica set primary.
This creates a local admin database with local credentials for each shard. See Sharded Cluster Security for a detailed
discussion of access control privileges in sharded clusters.
16
Single Server: in a mongo shell
connected to the server,
MMS Monitoring
MMS Monitoring with Database Profiling
MMS Monitoring without dbSats
MongoDB Deployment:
in a
mongo shell connected to either the replica set primary or a
mongos.
use admin
use admin
db.addUser( { user: "agent",
pwd: "Moon1234",
pwd: "Moon1234",
roles: [ "clusterAdmin",
roles: [ "clusterAdmin
"readAnyDatabase"
"readAnyDatab
]
]
} )
} )
use admin
use admin
pwd: "Moon1234",
pwd: "Moon1234",
"readAnyDatabase",
"readAnyDatab
"dbAdminAnyDatabase"
"dbAdminAnyDa
]
]
} )
} )
use admin
use admin
pwd: "Moon1234",
pwd: "Moon1234",
]
]
} )
} )
2.3 On-Prem MongoDB Management Service Administration Interface
The MMS Administration section of the On-Prem MongoDB Management Service application provides access to user
management, system status, and system-wide messaging.
Background
Access the MMS Administration section by clicking the Admin in the upper right hand corner of the MMS interface.
Only users with the MMS Administrator role have access to the MMS Administration section of the On-Prem MongoDB Management Service application. The first user that registers for the On-Prem MongoDB Management Service
application has the MMS Administrator role. Users MMS Administrator also have Group Admin level access for all
groups.
Sections
System Overview
The Overview page of the MMS Administration section reports system use and activity. First, a table reports totals for
numbers of:
• groups
17
• active groups
• active hosts
• users
• users active
• ping spillover queue
• increment spillover queue
Additionally two charts report:
• total page views
• total chart requests
These charts have the same zoom features as other MMS charts.
UI Messages
The Messages tab holds the UI Messages page. Use this interface to create bulletins inside of the MMS application to
announce impending maintenance windows or other calls to action for users.
Messages may be active, or visible, on all pages or a subset of pages.
Create Message To create a new message click on the Add Message button next to the page title. This will raise the
Add Message interface where you can create a message and specify:
• The text of a message.
• An optional page name (URL) or page prefix.
When you create a Message, you can specify an optional prefix of pages on which to include the message.
This prefix allows you to specify a path, beginning with a http://docs.opsmanager.mongodb.com//
character, of a single page or the URL prefix of a group of pages.
• A toggle to enable (i.e. Active) or leave the page disable.
After creating an active message, the message may take up to 60 seconds to propagate to users.
Message Table The UI Messages page holds a table of all available messages. You can use the search interface on
the upper right corner of the table to narrow the list of messages.
For each message, the message table reports:
• which page or page prefix the message will appear on.
• the text of the message.
• whether the message is active or inactive.
Active messages are also highlighted in orange on the UI Messages page.
• the creation date and time for the message.
• the date and time of the last modification for the message.
Manage Active Messages On the right side of the message table, for each message there are two buttons: a toggle
to enable or disable a UI message, as well as a button to delete a message.
After enabling a message, the message may take up to 60 seconds to propagate to users.
18
Users
The Users tab holds a user management interface. The user management table holds a list of
Use the search interface on the upper right corner of the table to narrow the list of messages.
Users Information For each user, the users table reports:
• the username
• the available administrative roles, if any
• the date and time of the last login.
• the date and time of the last access event.
• the total number of login.
• the user’s configured timezone.
• the creation date and time.
Edit Users On the right side of the users table, for each user there is an edit button that allows administrators to
modify the user account. Use this interface to:
• add or change the email address associated with the account.
• change the role, for each group. Available roles are: “group admin” or “read only.”
• toggle the “account locked” flag which prevents users from logging in.
• toggle the “MMS Administrator” flag which adds the MMS Administrator role to the user.
2.4 Settings
The “Settings” section of the On-Prem MongoDB Management Service (MMS) console enables users to personalize
their console and activate or activate a variety of features. The following sections correspond to a tab on the Settings
page.
My Settings
These settings are specific to the logged in user, and will only affect their MMS experience.
Profile
The Profile page allows users to update their personal information.
Note: The username, email address, and password are also used for jira.mongodb.org. Changing your email address
or password in MMS will also change the email address and password you use to log into Jira.
• User Name: displays the user’s name. You cannot change your username.
• Email Address: displays the email address MMS associates with your account. You can change your email
address by clicking on the “pencil” icon.
19
• Mobile Phone Number: displays the mobile phone number MMS uses for Two-Factor Authentication. You
can change the number by clicking on the “pencil” icon.
If you are outside of the United States or Canada, you will need to include ‘011’ and your country code. For
instance, for New Zealand (country code 64), you would need to enter ‘01164’, followed by your phone number.
Alternately, you can sign up for a Google Voice number, and use that number for your authentication.
Note: Two-Factor Authentication via SMS text message does not currently support Indian mobile phone
numbers (country code 91).
• Password: allows you to change your MMS password. Passwords must fulfill MMS’s password requirements.
Personalization
The Personalization page allows users to configure the console to suit their needs and preferences.
• Chart Refresh: sets the rate at which the data displayed in the charts on your dashboards or on the “Hosts
Statistics” page refreshes.
• Default Dashboard: sets the default dashboard on the “Dashboard” page. You can select from a list of all of
your dashboards.
• Homepage: sets which page of the MMS console you will see when you log into MMS. You can choose from
the “Hosts” page, or the “Dashboard” page.
• Page Shown When Switching Groups sets which page of the MMS console you will see when you select a
different group. You can select the “Hosts” page or the “Dashboard” page. Alternatively, select “Current” and
MMS will not change pages when you select a different group.
• Separate Opcounter Charts: allows you to control the presentation of Opcounter Charts. If enabled, MMS
charts each opcounter type separately. Otherwise, each opcounter type is overlaid together in a single chart.
• Display Chart Annotations: toggles the presence of chart annotations. Chart annotations overlay information
about significant system events on the charts. For example, with chart annotations MMS will draw a red vertical
line over the charts.
• User Time Zone: sets your local time zone.
• User Language: allows you to select the language the MMS Console will display in.
• User Date Format: allows you to select your preferred date format.
• Email Notifications: allows you to opt-in to, or opt-out of receiving e-mail newsletters about MMS.
Group Settings
These settings are general settings that apply to all users in the current group.
General Settings
• Log Data Collection For All Hosts: activates or deactivates the collection of log data for all hosts. This
overwrites the statuses set on the individual hosts. On Prem MMS Monitoring displays log data in the “Logs”
tab of the “Host Statistics” page.
• Group Time Zone: sets your group’s time zone.
20
• Exposed DB Host Check: when active, On Prem MMS Monitoring will periodically try to connect to your
MongoDB instances. If your configuration is secure, this should fail. If the attempt is successful, On Prem
MMS Monitoring sends you an alert.
• DB Stats: allows you to enable or disable the collection of database statistics. For more information, see “How
does MMS gather database statistics?”.
Preferred Hostnames
Preferred Hostnames allows you to specify the hostname to use for servers with multiple aliases. This prevents servers
from appearing multiple times under different names.
Duplicate Hostnames
Duplicate Hostnames allows you to reset and remove all detected duplicate hosts. This is useful if your server environment has drastically changed and you believe a host is incorrectly marked as a duplicate.
API Key
API Key displays the On Prem MMS Monitoring API Key for your MMS group. Keep this key private. Use the API
key to support automated agent installation.
Secret Key
Secret Key displays the On Prem MMS Monitoring Secret Key for your group. Keep this key private. Use the Secret
key to support automated agent installation.
Monitoring Agent
Monitoring Agent provides links for downloading the pre-configured Monitoring agent in both .zip and .tar.gz
formats. It also provides links to download a generic agent for automated agent installation.
2.5 Frequently Asked Questions About On-Prem MongoDB Management Service
See also:
Frequently Asked Questions About On Prem MMS Monitoring.
User and Group Management
How do I reset my password?
You can reset your password using the password reset form.
How do I change my password?
You can change your password by resetting your password.
21
What are the password requirements?
Passwords must be at least 8 characters long and contain at least one letter, one digit, and one special character.
Passwords for the MongoDB Jira instance and MMS are the same, although the length and character requirements are
different for Jira and MMS.
How do I add a user to my company/group?
If the user already has a MongoDB Jira or MMS account, you can add their username to your group on the admin
page.
If the user does not have a Jira account then they can create a new account. After they have created an account, you
can add their username to the company/group on the admin page.
How do I remove my company/group?
Although it’s not possible to delete a group, you can remove a group by removing all users from the group, and then
remove your own account from the group. The group will then be orphaned. It will remain for auditing purposes, but
be otherwise inaccessible.
Alerts
What is “Exposed DB Host Check,” and why did MMS send me an “MMS Found Exposed Databases!!!” Email?
If enabled on the settings page, the On Prem MMS Monitoring application servers will attempt to connect to your
MongoDB instances to help ensure that your firewall is properly configured. This is a weak security validation that
should not replace other auditing or intrusion detection system procedures.
This test runs once per day and attempts to connect using both the IP address and the non-canonical host identifier that
you see in the Name field of the Hosts page.
How do I modify my alert settings?
You can enable, disable, or modify alerts on the settings tab of Alerts page.
About On-Prem MongoDB Management Service
What open source projects does MMS use?
• Database: MongoDB
• App framework: Google Guice
• Http server: Jetty
• Web framework: Jersey
• Misc server libs: Apache Commons
• UI lib: jQuery , Bootstrap
• Charts: dygraphs
22
• Graphics: Font-Awesome
2.6 On-Prem MongoDB Management Service Server Changelog
On-Prem MongoDB Management Service Server 1.3.1
Released 2014-02-17
• Enhanced handling of custom MongoDB build versions
• Enhanced handling of situation in which monitored MongoDB has bindIP of localhost or 127.0.0.1
Released 2013-12-01
• Packaging/support for Debian and SUSE Linux
• Kerberos authentication support between MMS server and backing MongoDBs, as well as between Monitoring
Agent and the MongoDBs it monitors
• OnPrem users can be overall site administrators (MMS Admins)
• New admin section where MMS Admins can manage user roles and message banners
• Tunable advanced password and session management configurations.
• Encryption key rotation, more specific CORS policy, auth tokens removed from chart URLs, and other security
enhancements.
Released 2013-07-24
• Redesigned user interface and enhanced algorithm to auto-discover hosts and derive host topology
• SNMP monitoring
• Ability to export charts
• Option to store encrypted authentication credentials in the mmsDb property in the configuration file.
• Ability to classify users within an MMS Group as group administrators or read-only users
3 On Prem MMS Monitoring
On Prem MMS Monitoring is a service for monitoring MongoDB deployments, and an integral part of of the OnPrem MongoDB Management Service. On Prem MMS Monitoring collects statistics on all key server and hardware
indicators and presents this data through an intuitive web interface.
This manual describes the installation of the Monitoring agent and operation of the On Prem MMS Monitoring web
console. You can find answers to common questions in the FAQs, but for all other inquiries please feel free to open a
JIRA ticket.
As
an
alternative,
you
may
consider
the
full
installation
http://docs.opsmanager.mongodb.com//monitoring/tutorial.
instructions
in
the
23
Getting Started with On Prem MMS Monitoring Installation and configuration tutorials for On Prem MMS Monitoring.
Install Monitoring Agent Tutorials for installing On Prem MMS Monitoring and deploying it in production.
Monitor Hosts with On Prem MMS Monitoring Describes how On Prem MMS Monitoring monitorings hosts
and discusses monitoring architecture options.
Update On-Prem MongoDB Management Service Manually update the Monitoring agent on UNIX/Linux and
Windows.
Configure On Prem MMS Monitoring Discusses configurable options with On Prem MMS Monitoring: hardware monitoring with Munin-Node and using On Prem MMS Monitoring with MongoDB instances running with SSL.
Using the MMS Console Discusses the pages of the MMS Console and their functionality.
Diagnostic and Troubleshooting Guide Troubleshooting advice for common issues encountered with On Prem MMS
Monitoring.
Frequently Asked Questions About On Prem MMS Monitoring On Prem MMS Monitoring frequently asked questions.
On Prem MMS Monitoring Agent Changelog Change log detailing new features and updates for each Monitoring
Agent release.
3.1 Getting Started with On Prem MMS Monitoring
Installing and configuring On Prem MMS Monitoring quick and easy.
This manual will explain how to install and configure your On-Prem MongoDB Management Service server, and guide
you through the On Prem MMS Monitoring installation process specific to your operating system. Other pages will
help you to customize your configuration, keep the Monitoring agent updated, and provide you with troubleshooting
strategies.
As an alternative,
you may follow the install process from
start
to
finish
in
the
Install Monitoring Agent Tutorials for installing On Prem MMS Monitoring and deploying it in production.
Monitor Hosts with On Prem MMS Monitoring Describes how On Prem MMS Monitoring monitorings hosts and
discusses monitoring architecture options.
Connect to Hosts with Kerberos Authentication Outlines the procedure for configuring the Monitoring Agent to authenticate to hosts in a Kerberos environment.
Update On-Prem MongoDB Management Service Manually update the Monitoring agent on UNIX/Linux and Windows.
Configure On Prem MMS Monitoring Discusses configurable options with On Prem MMS Monitoring: hardware
monitoring with Munin-Node and using On Prem MMS Monitoring with MongoDB instances running with
SSL.
Install Monitoring Agent
The following tutorials will guide you through the steps necessary to install and deploy On Prem MMS Monitoring on
your system. You will need to install and configure the On-Prem Monitoring server itself, and then install and start the
Monitoring agent.
24
As an alternative,
you may follow the install process from
start
to
finish
in
the
Install the Monitoring Agent on CentOS, RHEL, Fedora, Amazon Linux
Overview Installing the MMS monitoring agent on CentOS, Fedora, Red Hat Enterprise Linux (RHEL), and Amazon Linux requires a number of Python packages and extensions, including C extensions.
Prerequisites Before you install the monitoring agent, these software packages must be available or installed on the
target system:
• Python 2.6+
• setuptools to install Python packages
• python-dev to install Python C extensions
• pymongo to install the Python driver used by the monitoring agent
• agent.py to install the MongoDB monitoring agent
Procedure
Step 1: Install Python Packages and Extensions Install python-setuptools, which you will use to install
the remaining Python dependencies.
sudo yum install python-setuptools
Step 2: Install Python C Extensions While the C extensions are not required for On Prem MMS Monitoring, they
significantly improve performance. You must have a C compiler (e.g. gcc) and Python header files installed on your
system. Type this command to install Python headers:
sudo yum install gcc python-dev
Step 3: Install and Upgrade PyMongo If you have not installed pymongo, type this command to install the latest
version:
sudo easy_install pymongo
To upgrade to the latest version of the driver, type this command:
sudo easy_install -U pymongo
For more information about PyMongo installation, see the Additional Information section below.
If PyMongo was previously installed without C extensions, install PyMongo C extensions.
If you are installing PyMongo and the Monitoring agent on systems that do not have C compilers, build PyMongo
packages with PyMongo C extensions.
25
Step 4: Install the MongoDB Monitoring Agent Download the latest MMS monitoring agent from the On-Prem
MongoDB Management Service, located on the Settings page and the Monitoring Agent tab.
With Python software requirements installed, install the MongoDB monitoring agent with these commands:
cd mms-agent
nohup python agent.py > /LOG_DIRECTORY/agent.log 2>&1 &
Replace LOG-DIRECTORY with the path to your MongoDB logs.
Next Steps Once the agent is running and configured, the next step is to deploy the monitoring agent in production.
Also, consider reading the PyMongo documentation.
Install the Monitoring Agent on Debian and Ubuntu
Overview Installing the MMS monitoring agent on Debian and Ubuntu requires a number of Python packages and
extensions, including C extensions.
Prerequisites Before you install the monitoring agent, these software packages must be available or installed on the
target system:
• Python 2.6+
• setuptools to install Python packages
• python-dev to install Python C extensions
• pip to install and uninstall PyMongo
Procedure
Step 1: Install Python Packages and Extensions Install python-setuptools, which you will use to install
the remaining Python dependencies.
sudo apt-get install python-setuptools
Step 2: Install Python C Extensions While the C extensions are not required for On Prem MMS Monitoring, they
significantly improve performance. You must have a C compiler (e.g. gcc) and Python header files installed on your
system. Type this command to install Python headers:
sudo apt-get install build-essential python-dev
Step 3: Install and Upgrade PyMongo If you have not installed pymongo, type this command to install the latest
version:
sudo easy_install pymongo
To upgrade to the latest version of the driver, type this command:
26
sudo easy_install -U pymongo
For more information about PyMongo installation, see the Additional Information section below.
Step 4: Install the MongoDB Monitoring Agent Download the latest MMS monitoring agent from the On-Prem
MongoDB Management Service, located on the Settings page and the Monitoring Agent tab.
With Python software requirements installed, install the MongoDB monitoring agent with these commands:
cd mms-agent
Install the Monitoring Agent on Windows
Overview Installing the MMS monitoring agent on Windows requires a number of Python packages and extensions
including C extensions, as well as PowerShell.
Prerequisites
Requirements Before you install the monitoring agent, these software packages must be available or installed on
the target system:
• PowerShell 2.0+
• Python 2.7
See the Python Download Page for the most up to date Python distribution downloads, or use the direct link for the
Python 2.7 64-bit Windows Installer .msi file.
If you do not have a 2.x-series Python installation on your system, install the latest version of Python 2.7.x. If your
system supports 64-bit executable, use the 64-bit version.
Ensure that you install the correct PyMongo build for your Python installation both in terms of version number (e.g
.2.6, 2.7, etc.) and system architecture (e.g. 32 or 64-bit.)
Credentials If you already have jira account you may sign in to MMS with your JIRA credentials. When you sign
into MMS for the first time, the system prompts you to create a new “group” for your deployment.
If you have not downloaded the Monitoring agent already, you can click the “Settings” link immediately after you sign
into MMS, select “Settings” and then click on “MMS Agent” to download an agent specifically configured for your
account.
27
Considerations The Monitoring agent distribution download includes a WINDOWS.txt file with instructions for
using the agent on Windows platforms. Before you can use the agent on Windows platforms, you must ensure your
system meets the system requirements.
System Resources The Monitoring agent does have some resource requirements and should run on separate systems
to avoid impacting mongod and mongos performance. To monitor five or fewer nodes, you can safely deploy on an
AWS “micro instance.” Similarly, if you are only monitoring a small number of databases, you may be able to deploy
the agent on the system running the mongos process. Running PyMongo with the native extensions, which requires
gcc, provides significant performance improvements.
Permissions Installing the monitoring agent on Windows requires the appropriate permissions to enable unsigned
script execution with PowerShell. As described below, permissions must be set at the HKEY_LOCAL_MACHINE
level or HKEY_CURRENT_USER level. In some cases, your system administrator may have used Group Policy to
block unsigned PowerShell scripts; contact your administrator if the Set-ExecutionPolicy commands below
generate errors.
Procedure
Step 1: Install PyMongo Use the Windows installer to install PyMongo from PyPi.
Step 2: Enable Script Execution in PowerShell Right click the PowerShell icon in the “Start Menu,” and run
PowerShell as administrator. Issue the following command:
Get-ExecutionPolicy
This returns the current execution policy. Save this value for future use in Step 4 below. If the policy is
“Unrestricted” you can proceed to the next step. If the policy is not “Unrestricted” issue the following
command:
Set-ExecutionPolicy -ExecutionPolicy Unrestricted
If this command returns an error denying access to a HKEY_LOCAL_MACHINE registry key, this command sets the
execution policy scope to the current local user (e.g. HKEY_CURRENT_USER), not globally for the local machine:
Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser
Confirm the policy change when prompted.
Step 3: Install the MMS Monitoring Agent Start an administrator command window. Change to the folder containing the Monitoring agent files. Run mongommsinstall.bat. If this succeeds, the Monitoring agent will be
running and will start on system boot.
Download the Python 2.7 64-bit Windows Installer from the following URL:
http://www.python.org/ftp/python/2.7.2/python-2.7.2.amd64.msi
Now you can start and stop the Windows service to control the MMS agent. Consult the WINDOWS.txt file for more
information.
28
Step 4: Disable Script Execution in PowerShell If you have changed the execution policy, after the installation has
completed successfully set it back to its original value using the following command:
Set-ExecutionPolicy -ExecutionPolicy <original value>
Replace <original value> with the value that you recorded above in Step 2.
Requirements for Python 2.4
Overview If your system is running Python 2.4, you will need to install the hmac and hashlib Python modules.
Prerequisites
• hmac to install Python package for hash-based message authentication code
• hashlib to install Python package of hash libraries
• easy_install to install Python libraries
Procedure
Step 1: Install hmac and hashlib Python modules Install these Python modules with easy_install:
sudo easy_install hmac hashlib
Do not use pip to install modules when using Python 2.4 or 2.5, as there are some compatibility issues. Some users
have reported problems installing hmac, if easy_install produces an error or you suspect a problem with these
libraries see instructions for installing hmac manually.
If easy_install produces an error when trying to install hmac, or you see the following error in your agent logs,
you will need to install the hmac package manually:
AttributeError:'builtin_function_or_method' object has no attribute 'new'
To install the hmac package, begin by downloading the latest source package. The following example uses version
20101005. Issue the following command:
curl -O http://pypi.python.org/packages/source/h/hmac/hmac-20101005.tar.gz
Always download the latest version of hmac, which you can find at the hmac page on PyPi.
Step 2: Extract hmac Files Extract the files from the archive with the following command:
sudo easy_install hmac hashlib
Step 3: Install hmac Files Extract the files from the archive with the following command:
cd hmac-20101005/
python setup.py install
You may need root (i.e. sudo) privileges to run the final install step.
29
Deploy the Monitoring Agent in Production
Overview For production deployments of On Prem MMS Monitoring, you will want to daemonize the agent process
and ensure that it restarts following a system restart. Your operating system likely has a preferred method for managing
daemon processes.
Prerequisites If your datacenter requires that you route outbound https connections via a proxy, you must use a
version of Python greater than 2.6.1.
Considerations You must configure the networking rules of your deployment so that:
• the Monitoring Agent can connect to all mongod and mongos instances that you want to monitor.
• the Monitoring agent can connect to On Prem MMS Monitoring server on port 443 (i.e. https.)
The On Prem MMS Monitoring server does not make any outbound connections to the agents or to MongoDB instances. If Exposed DB Host Check is enabled, the On Prem MMS Monitoring server will attempt to connect to your
servers occasionally as part of a vulnerability check.
Ensure all mongod and mongos instances are not accessible to hosts outside your deployment.
Procedure
Step 1: Daemonize the Process As a temporary measure, the following command will start the agent process
detached from the current terminal session:
nohup python agent.py > /[LOG-DIRECTORY]/agent.log 2>&1 &
Replace [LOG-DIRECTORY] with the path to your MongoDB logs.
This command allows the agent survive the current terminal session and writes all messages to the agent.log file.
You may include this command in your MongoDB control script or use your system’s /etc/rc.local equivalent;
however, avoid running the agent as root.
Step 2: Create Control Scripts If you need to create an initialization script to control the daemonized process,
consider the resources listed in the Additional Information section below.
You may also examine the scripts in your system’s /etc/init.d/ or /etc/rc.d/ directory. Ensure that the
agent does not run with root privileges.
Use the update-rc.d utility on Debian and Ubuntu and the chkconfig tool on Red Hat related systems to add
these scripts to the initialization process. Be sure to test the control script configuration. It is essential that you be able
to start, stop, and restart the agent following a system reboot.
Step 3: Automate Agent Deployments The agent you downloaded from the On-Prem MongoDB Management
Service site is automatically configured with the credentials for your account. You can download a configured copy of
the Monitoring agent by from the Monitoring Agent section of the Settings page by selecting Download Monitoring
Agent (zip) or Download Monitoring Agent (tar.gz).
The URL for the configured agent will resemble the following:
30
https://mms.mongodb.com/settings/mmsAgent/<hash>/mms-monitoring-agent-<group-name>.zip
Alternately, you may automate Monitoring agent deployments using an unconfigured agent and the API from the API
Settings section of the Settings page. The unconfigured agent is available at the following URLs:
https://mms.mongodb.com/settings/mms-monitoring-agent.zip
https://mms.mongodb.com/settings/mms-monitoring-agent.tar.gz
Extract this archive and edit the settings.py file, updating the @API_KEY@ with the API key for your account.
Also update the @DEFAULT_REQUIRE_VALID_SERVER_CERTIFICATES@ and @MMS_SERVER@ values.
You may embed this process in your existing deployment scripts to automatically install or redeploy new agents.
Step 4: Configure any Proxies You will need to export the “https_proxy” environment variable (or
“HTTPS_PROXY” on Windows).
Issue the following command before running the Monitoring agent to set the environment variable:
export https_proxy='http://proxyserver.example.net:port"
Replace “http://proxyserver.example.net” with the name or IP address of the proxy server and “port
with the TCP port that the proxy service runs on. You may choose to export this variable inside of your control script.
Next Steps Once the agent is running and configured to your satisfaction, see Configure On Prem MMS Monitoring.
Also see Using the MMS Console for more information about the On Prem MMS Monitoring interface.
Additional Information The following resources provide documentation for creating control scripts:
• Ubuntu Boot Up How To
• Debian Linux Control/Init Scripts
• Arch Linux rc.d Scripts
Debian and Ubuntu Systems have an example control script located at /etc/init.d/skeleton that you can use
as a template.
Red Hat Enterprise Linux and related distributions (e.g. Fedora, CentOS, etc.) provide example control scripts in the
/usr/share/doc/initscripts-*/sysvinitfiles/ directory.
Install PyMongo with C Extensions
Overview If you installed PyMongo without the C extensions (i.e. on a system without a C compiler, or not using
the Windows installer), PyMongo should be re-installed with C extensions.
While the C extensions are not required for MMS Monitoring, they significantly improve performance. You must have
a C compiler (e.g. gcc) and Python header files installed on your system.
Prerequisites Before you install the PyMongo C extensions, these software packages must be available or installed
on the target system:
• Python 2.6+
• gcc or similar to install a C compiler
31
• For Debian and Ubuntu: python-dev to install Python C extensions
• For Red Hat,
CentOS, and Fedora:
python-devel,
python-setuptools-devel to install Python C extensions
python-setuptools,
and
Procedure
Step 1: Install gcc and Python Packages For Debian and Ubuntu environments, issue this command:
For Red Hat, CentOS, and Fedora environments, issue this command:
sudo yum install gcc python-devel python-setuptools python-setuptools-devel
Step 2: Install the pip Utility
sudo easy_install pip
Step 3: Remove PyMongo
sudo pip uninstall pymongo
Step 4: Re-Install PyMongo
sudo pip install pymongo
Step 5: Restart Monitoring Agent
cd mms-agent
When you restart your agent there is a 5 minute timeout before the agent will begin sending data to MMS again.
Build PyMongo Packages with PyMongo C Extensions
Overview If you are installing PyMongo and the Monitoring agent on systems that do not have C compilers, you
can create a .egg package of PyMongo with C extensions to install on your system or systems.
While the C extensions are not required for MMS Monitoring, they significantly improve performance. You must have
a C compiler (e.g. gcc) and Python header files installed on your system.
32
Prerequisites Before you build PyMongo packages with PyMongo C extensions, these software packages must be
available or installed on the target system:
• Python 2.6+
• curl to retrieve PyMongo archived file
• gcc or similar to install a C compiler
• python-dev or python-devel to install Python C extensions
The packages distributed in PyPI for Windows include C extensions.
Considerations The .egg file created with these instructions will only work on systems that resemble the environment on which you built the package. Ensure operating systems, versions of Python, and architecture (i.e. “32” or
“64” bit) match.
Commands below use PyMongo version 2.6.3. If the current version is higher, replace 2.6.3 with the current number.
Procedure
Step 1: Install gcc and Python Packages For Debian and Ubuntu environments, issue this command:
For Red Hat, CentOS, and Fedora environments, issue this command:
sudo yum install gcc python-devel python-setuptools python-setuptools-devel
Step 2: Install PyMongo
curl http://pypi.python.org/packages/source/p/pymongo/pymongo-2.6.3.tar.gz > pymongo-2.6.3.tar.gz
Step 3: Extract the PyMongo Files
tar -zxvf pymongo-2.6.3.tar.gz
Step 4: Build the .egg File
cd pymongo-2.6.3
python setup.py bdist_egg
Once built, you can find .egg file in the dist/ sub-directory.
The file name will resemble
pymongo-2.6.3-py2.7-linux-x86_64.egg but may have a different name depending on your platform and the version of python you use to compile.
Step 5: Install the .egg File on Target System
sudo easy_install pymongo-2.6.2-py2.7-linux-x86_64.egg
33
Step 6: Install the Monitoring Agent Copy the .egg file to the target system and issue this command to install
the package:
cd mms-agent
Monitor Hosts with On Prem MMS Monitoring
The Monitoring agent automatically discovers MongoDB processes based on existing cluster configuration. You’ll
have to manually “seed” at least one of these hosts from the MMS console.
To add a host to On Prem MMS Monitoring, click the “plus” (+ ADD HOST) button at the top of the Hosts page.
This raises a query element for the hostname, port, and optionally an admin DB username and password. Provide the
necessary information and select “Add.”
Once it has a seed host, the Monitoring agent will discover any other nodes from associated clusters. These clusters,
and their respective seed hosts, include:
• Master databases, after adding slave databases.
• Shard clusters, after adding mongos instances.
• Replica sets, after adding any member of the set.
Once you add these seed nodes, the Monitoring agent will fetch this information from the MMS servers. Thus,
when configuring the monitoring environment, you may need to wait for several update cycles (e.g. 5-10 minutes) to
complete the auto-discovery process and host identification. During this period, you may see duplicate hosts in the
MMS web console. This is normal.
The Monitoring agent fetches configuration and reports to On Prem MMS Monitoring every minute, so, again, there
may be a delay of several minutes before data and host information propagate to the MMS console.
You can find immediate evidence of a working installation in the agent output or logs. For more information, check
the MMS console’s “Hosts,” section in the “Agent Log” and “Pings” tabs. Once On Prem MMS Monitoring has data,
you can view and begin using the statistics.
If the agent cannot collect information about a host for 24 hours, the On Prem MMS Monitoring system deactivates the
instance in the Monitoring agent and console. Deactivated hosts must be manually reactivated from the MMS console
if you wish to collect data from these hosts.
Monitoring Architecture
Only one Monitoring agent per group or environment will report to On Prem MMS Monitoring at a time. However,
you can run multiple instances of the agent to provide redundancy. Secondary agents act as hot standbys. If you stop
the agent process, On Prem MMS Monitoring removes the agent from the “Agents” tab within a day. If a Monitoring
agent is unavailable, the last ping times for that agent do not update. To install secondary agents, simply repeat the
installation process in the installation guide for each new agent. You can also automate agent installation.
Note: When the Monitoring agent restarts, there is a five minute delay before that agent begins collecting data sending
pings to On Prem MMS Monitoring. If you have multiple agents, this delay permits other agents in your infrastructure
to become the “primary agent” and for On Prem MMS Monitoring to determine which agent will be primary.
34
During this interval, the restarted Monitoring agent will not collect data.
Additional Considerations for MongoDB 2.4 Nodes
If you are monitoring a MongoDB 2.4 instance with authentication enabled, you must have elevated privileges to
collect complete performance data. At a minimum, the user will need to possess:
• the clusterAdmin role, and
• the readAnyDatabase role.
If MMS is running with DB Profiling enabled, the user requires additional permissions.
clusterAdmin and readAnyDatabase roles, the user must possess either:
In addition to the
• the dbAdminAnyDatabase role, or
• the dbAdmin role on the specific databases being profiled.
However, if you do not have DB Profiling enabled and you are not collecting dbStats data, the agent only needs
permissions provided by the clusterAdmin role.
See also:
User Privilege Roles in MongoDB.
Connect to Hosts with Kerberos Authentication
Kerberos is a generic authentication protocol available in MongoDB Enterprise after version 2.4. The On Prem MMS
Monitoring agent can authenticate to monitored hosts using Kerberos in addition to the default MongoDB authentication protocol.
Install the monitoring agent and all requirements before beginning to configure Kerberos.
Install Required Operating System Packages
Debian and Ubuntu Linux Install the following required packages:
sudo apt-get install krb5-user python-setuptools libkerb5-dev build-essential python-dev
Red Hat Enterprise, CentOS and Fedora Linux Install the following required packages:
sudo yum install krb5-appl-clients.x86_64 gcc python-devel krb5-devel.x86_64
Install Python Requirements
sudo apt-get uninstall python-kerberos
sudo easy_install pymongo kerberos
Configure Kerberos Environment
1. Create or configure the /etc/kerb5.conf file on the system to integrate this host into your Kerberos environment.
35
2. Ensure that the kinit binary is available at the http://docs.opsmanager.mongodb.com//user/bin/kinit
path.
Create Kerberos Principal and MongoDB User
1. Create or choose a Kerberos principal for the On Prem MMS Monitoring agent.
2. Generate a keytab for the Kerberos principal and copy it to the system where the monitoring agent runs.
Important: Ensure that the user that will run the Monitoring agent is the same user that owns the keytab file.
3. Create a MongoDB user for the new Kerberos principal. See Authentication Requirements for more information
about required authentication roles.
4. Edit the agent’s settings.py file to inform the agent about the keytab and principal identifier. Set:
• the krb5Principal to the name of the Kerberos principal.
krb5Principal = <id>
• Set the kerb5Keytab value to the complete absolute path of the keytab file:
krb5Keytab = None
Update On-Prem MongoDB Management Service
Update the Monitoring Agent
Update the Agent on UNIX/Linux To update the Monitoring agent on UNIX/Linux systems:
1. Stop the agent processes.
2. Download the latest Monitoring agent from the Settings page of the MMS console, in the Monitoring Agent
section.
3. Restart the Monitoring agent.
Update the Agent on Windows To update the agent on Windows systems:
1. Download the latest Monitoring agent from the “Settings” page of the MMS console in the Monitoring Agent
section).
2. From a command shell, issue the following:
sc stop MongoMMS
sc delete MongoMMS
This should stop and uninstall On Prem MMS Monitoring.
3. Confirm that the service no longer exists. Issue the following command:
sc query MongoMMS
This operation will return a message that resembles the following:
[SC] EnumQueryServicesStatus:OpenService FAILED 1060:
The specified service does not exist as an installed service.
36
4. When you have successfully uninstalled the Monitoring agent, you can use mongommsinstall.bat to reinstall On Prem MMS Monitoring.
5. Restart the agent.
Upgrade MMS Packages
Upgrading an RPM-Based Installation
server release.
Please contact MMS to receive the download location of the latest MMS
1. Shutdown the On-Prem MMS server and take a backup of your existing configuration:
sudo cp -a <install_dir>/conf ~/mms_conf.backup
2. Perform an RPM upgrade:
sudo rpm -Uvh 10gen-mms-<version>.x86_64.rpm
3. Reconcile any changes in configuration files.
At this point the upgrade is complete. However you may need to reconcile changes in your configuration with
new configuration options available in the latest release.
During the rpm operation, if you saw the following output, you have changes to reconcile:
warning: <install_dir>/conf/conf-mms.properties created as
<install_dir>/conf/conf-mms.proper
Compare your current configuration to the updated version, with the following sequence of operations:
diff -u <install_dir>/conf/conf-mms.properties <install_dir>/conf/conf-mms.properties.rpmnew
diff -u <install_dir>/conf/mms.conf <install_dir>/conf/mms.conf.rpmnew
Edit your configuration to resolve any conflicts between the old and new versions, being sure to take any new
changes from conf-mms.properties.rpmnew as appropriate. Changes to mms.centralUri, email
addresses, and MongoDB are the most common configuration changes.
Repeat the above reconciliation for mms.conf if the upgrade indicates a conflict.
Note: The upgrade from beta versions 1.0.1 to 1.0.2 changed several paths to make the MMS server completely
self contained. In 1.0.2 all logs, configuration, and working files are in the /opt/10gen/mms/ hierarchy. This
changes the following paths from 1.0.1:
• New logs path: <install_dir>/logs/
• New tmp path: <install_dir>/tmp/
Finally, you may also need to re-symlink your startup script:
sudo ln -s /<install_dir>/bin/mongodb-mms /etc/init.d/mongodb-mms
4. Restart the On-Prem MMS server.
Upgrading a tgz/zip Installation To upgrade a tarball installation, backup configuration and/or logs, and then
re-install the On-Prem MMS server.
Important: It is crucial that you back up the existing configuration because the upgrade process will delete existing
37
data.
In more details:
1. Shutdown the MMS server and take a backup of your existing configuration and logs.
sudo cp -a <install_dir>/conf ~/mms_conf.backup
sudo cp -a <install_dir>/logs ~/mms_logs.backup
2. Remove your existing MMS server installation entirely and extract latest release in its place:
cd <install_dir>/../
sudo rm -rf <install_dir>
sudo tar -zxf -C . /path/to/10gen-mms-<version>.x86_64.tar.gz
3. Similar to the RPM upgrade path above, compare and reconcile any changes in configuration between versions:
diff -u ~/mms_conf.backup/conf-mms.properties <install_dir>/conf/conf-mms.properties
diff -u ~/mms_conf.backup/mms.conf <install_dir>/conf/mms.conf
4. Edit your configuration to resolve any conflicts between the old and new versions, being sure to take any new
changes as appropriate.
Note: Changes to mms.centralUri, email addresses, and MongoDB are the most common configuration
changes.
5. Restart the On-Prem MMS server.
Configure On Prem MMS Monitoring
This document discusses specific configuration options for On Prem MMS Monitoring, including hardware monitoring
with Munin-Node and using On Prem MMS Monitoring with SSL.
Hardware Monitoring with Munin-Node
On Prem MMS Monitoring provides support for collecting and charting hardware statistics collected with Munin. You
must install the munin-node package on each the host system that you wish to monitor.
Note: munin-node, and hardware monitoring is only available for MongoDB instances running on Linux hosts.
On Debian and Ubuntu systems, issue the following command to install munin-node:
sudo apt-get install munin-node
To install munin-node on Red Hat, CentOS, and Fedora systems, issue the following command:
yum install munin-node
Note: For Red Hat and CentOS 6.8 systems, you will need to install the EPEL repository before installing
munin-node. To install the EPEL repository, issue the following command:
rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
38
When installation is complete, ensure that munin-node:
• is running. Use the command, “ps -ef | grep "munin"” to confirm. If the process is not running, issue
the command “/etc/init.d/munin-node start”.
• will start following the next system reboot. This is the default behavior on most Debian-based systems. Red Hat
and related distributions should use the “chkconfig” command, to configure this behavior (i.e. “chkconfig
munin-node on”)
• is accessible from the system running the agent. munin-node uses port 4949, which needs to be open on the
monitored system, so the agent can access this data source. Use the following procedure to test access:
telnet [HOSTNAME] 4949
fetch iostat
fetch iostat_ios
fetch cpu
Replace [HOSTNAME] with the hostname of the monitored system. Run these commands from the system
where the Monitoring agent is running. If these “fetch” commands return data, then munin-node is running
and accessible by the Monitoring agent.
Note: On some platforms, munin-node does not have all required plugins enabled.
For CentOS, the munin-node package does not have the iostat and iostat_ios plugins enabled. Use the
following operation to enable these plugins:
sudo ln -s /usr/share/munin/plugins/iostat /etc/munin/plugins/iostat
sudo ln -s /usr/share/munin/plugins/iostat_ios /etc/munin/plugins/iostat_ios
sudo /etc/init.d/munin-node restart
If munin-node is running but inaccessible, make sure that you have access granted for the system running the Monitoring agent and that no firewalls block the port between munin-node and the Monitoring agent. You may find the
munin-node configuration at “/etc/munin-node/munin-node.conf” or “/etc/munin-node.conf”,
depending on your distribution.
Additional considerations:
• If you have numbered disk devices (e.g.
/dev/sda1 and /dev/sda2) then you will need to
configure support for numbered disk in the munin iostat plugin.
Find the configuration file at
/etc/munin/plugin-conf.d/munin-node or a similar path, and add the following value:
[iostat]
env.SHOW_NUMBERED 1
• If you have Munin enabled and do not have iostat ios data in your Munin charts, your munin-node may
not have write access to required state files in its munin/plugin-state/ directory. See the munin-node
plugin log (i.e. /var/log/munin/munin-node.log or similar depending on your distribution) for more
information.
The full path of this state directory depends on the system,
but is typically
/var/lib/munin/plugin-state/. Run the following command sequence to correct this issue:
touch /var/lib/munin/plugin-state/iostat-ios.state
chown -R [username]:[group] /var/lib/munin/plugin-state/
chmod -R 660 /var/lib/munin/plugin-state/
Replace [username] and [group] with the username and group that the munin-node process runs with.
39
If you encounter any other problems, check the log files for munin-node to ensure that there are no errors with
Munin. munin-node writes logs files in the /var/log/ directory on the monitored system.
Using SSL with On Prem MMS Monitoring
On Prem MMS Monitoring can monitor MongoDB instances running with SSL. To use SSL with mongod and
mongos, you must enable it at compile time, or use one of the subscriber builds. MongoDB added SSL support
in version 2.0.
To monitor a host with SSL enabled, you can either:
1. Edit the settings.py file in your agent installation, so that the useSslForAllConnections value is
True, as follows:
useSslForAllConnections = True
Then restart the Monitoring agent. After restarting the agent you may observe a five minute delay before On
Prem MMS Monitoring receives data from the agent.
2. Enable support on a per-host basis in the MMS console by clicking on the edit (i.e. “Pencil”) button on the right
hand-side of the “Hosts” page. In the dialogue that pops up, click the check-box on the SSL tab.
If you enable SSL support globally you will not be able to override this setting on a per-host basis.
Next Steps with On Prem MMS Monitoring
Take this opportunity to explore the MMS interface. For a detailed explanation of the pages that form the MMS
console, continue to the usage guide. You may also want to consult the troubleshooting guide.
3.2 Using the MMS Console
The MMS web console is at the center of the On-Prem MongoDB Management Service. Through it, you can access
all of your monitored objects, set up alerts, control users, adjust your settings, and, using On Prem MMS Monitoring
dashboards, create customized collections of charts for easier data analysis.
The best way to learn about On Prem MMS Monitoring‘s capabilities is to browse the interface with live data. If you
haven’t done so already, please install the Monitoring agent before proceeding.
For an in-depth description of the top-level pages that form the MMS web console, consider the following documents:
Hosts Detailed description of the Hosts page of the MMS console, which lists all hosts that are currently being
monitored.
Host Statistics In-depth guide to the Host Statistics and the options that you can specify to customize your view.
Events An overview of the Events page of the MMS console, detailing possible event types and when they occur.
Alerts Details the alert types that the On-Prem MongoDB Management Service supports, and provides instructions
for configuring notifications.
Dashboards Instructions for modifying what dashboards are displayed on the Dashboards page of the MMS console.
40
Hosts
The “Hosts” section is the primary location for monitoring information in the MMS console. This tabbed interface
provides access to all of your monitored objects. The tabs you see depend on the types of processes in your deployment.
The tab:
• “Hosts” displays all non-arbiter mongod instances.
• “Mongos” displays all mongos instances.
• “Configs” displays all database configuration servers.
• “Arbiters” lists the mongod processes functioning as arbiters in replica sets.
• “Monitoring Agents” lists the Monitoring agents attached to this MMS account. For more information,
see “Monitoring Agents.”
The remaining “Host” section tabs contain information about the monitoring process and environment. The tab:
• “Monitoring Agent Log” displays a log of the Monitoring agent’s activity.
• “Pings” provides access to the raw JSON document of the last 20 pings sent to MMS by the agent.
• “Host Aliases” shows the mapping between system hostnames and the names provided by the monitored process
(e.g. mongod and mongos.) In most cases, hosts are automatically aliased during auto-discovery. If needed,
you may modify and configure the alias mapping in the “Settings” section of the console in the “Preferred
Hostnames” section. For more information about creating host aliases, see “Creating Host Aliases”.
Note: When you add or update a “Preferred Hostname” value, this action resets all previous hostname aliases.
• “CommandLine” shows the run time configuration options set in the configuration file and on the command
line for each host.
All of these interfaces, except for the Command Line, provide a search field in the upper right corner to filter the lists
in real time.
Note: If you see a hostname in displayed in orange on the host page, this means that:
• On Prem MMS Monitoring has detected startup warnings for this host. You can see the warning in the last ping
for the host.
• On Prem MMS Monitoring suspects that the host has a low ulimit setting that is less than 1024. On Prem
MMS Monitoring infers the host’s ulimit setting using the total number of available and current connections.
On Prem MMS Monitoring flags deactivated hosts with an amber yield sign on the host page. If you have deactivated
hosts, On Prem MMS Monitoring will add a warning icon (the amber “yield” sign) at the top of the “Hosts” page.
Important: If you have deactivated hosts, review all deactivated hosts to ensure that they are still in use, and remove
all hosts that are not active. Then click on the warning icon and select “Reactive ALL hosts”.
The Hosts Table
Host Groups From the Hosts page, you can display groups of hosts in one view. By following a link from the
“Cluster” or “Repl Set” columns, you can see a side-by-side comparison of charts from all hosts in a given cluster or
replica set.
41
Host Labels “Host Labels” provide a method to narrow the list of hosts in the MMS display. This is particularly
crucial if you monitor a large number of hosts and want to access a more limited set of hosts.
A pencil icon on the “Hosts” tab itself opens a dialogue where you may create a new host label. After adding the first
label, a drop down box appears where you can narrow the view by selecting a group.
To add hosts to a label or labels, click on the “Edit Host,” button (a pencil) in the rightmost column in the host table.
In the dialogue box that returns, the third tab allows you to select or remove labels from this host.
Hosts may belong to none, one, or multiple labels.
Create Host Aliases “Host Aliases” enables you to create host aliases for the hosts you monitor with MMS. These
aliases are arbitrary, and are useful if your machines have existing hostnames that do not sufficiently describe the
system in the context of MMS.
To set an alias for a host, click on the “Edit Host” button (a pencil) in the far right column of the hosts table. Port
numbers are not appended to host aliases.
Remove Hosts MMS cannot ignore any hosts added in the discovery process. You can always manually delete a
host from the MMS dashboard by clicking on the trash icon on the far right of the host entry in the hosts table. If
you want to add monitoring for a host that you have deleted from MMS, you must add this host manually using the
“+ ADD HOST” button at the top of the “Hosts” page. For more about monitoring hosts, see “Monitoring Hosts with
MMS”.
DB Profiling On Prem MMS Monitoring can collect data from MongoDB’s profiler to provide statistics about performance and database operations.
This data can include sensitive information, including the content of database queries. Ensure that exposing this to
On Prem MMS Monitoring is consistent with your information security practices. Additionally, be aware that the
profiler can consume resources which may adversely affect MongoDB performance. Consider the implications before
enabling profiling.
To allow On Prem MMS Monitoring to collect profile data for a specific host, click the “Enable profile data transmission,” button (the clock icon), in the far right column of the hosts table. It is the middle of the three icons. This raises a
dialogue box that describes the implications of DB profiling and allows you to enable the transmission of the profiling
data to On Prem MMS Monitoring.
Note: The Monitoring agent attempts to minimize its effect on the monitored systems. If resource intensive operations, like polling profile data, begins to impact the performance of the database, On Prem MMS Monitoring will
throttle the frequency that it collects data. See “How does MMS gather database statistics?“for more information
about the agent’s throttling process.
When enabled, On Prem MMS Monitoring samples profiling data from monitored instances: the agent only sends the
most recent 20 entries from last minute.
When you select “Enable Profile Info Transmission,” the Monitoring agent will begin sending profile data to On Prem
MMS Monitoring. All configuration changes made in the MMS console can take up to 2 minutes to propagate to the
agent and another minute before profiling data appears in the MMS interface.
However, to begin collecting profile data, you need to modify the value of setProfilingLevel on the database
itself. See the database profiler documentation for instructions for using the profiler. There is a link at the bottom of
the Host Statistics page that displays the profile levels.
If you have profiling data, and wish to delete it from the system, there is a button on the bottom of the “Profile Data”
tab, that says “Delete Profile Data.” When you click on this button, the MMS Console raises a confirmation dialogue.
When you confirm, On Prem MMS Monitoring will begin removing stored profile data from this server’s record.
42
Note: If On Prem MMS Monitoring is storing a large amount of profile data for your instance, the removal process
will not be instantaneous.
Agents
The “Agents” tab of the “Hosts” section contains information about the deployed Monitoring agents, and has the
following fields:
• Hostname: the name of the host running the agent.
• Address: the IP address of the address running the agent.
• Hosts: the number of MongoDB instances this agent monitors.
• Ping Count: the number of pings (i.e. data payloads) sent by the agent since midnight GMT. Typically agents
send pings every minute.
• Conf Count: the number of configuration requests sent by the agent since midnight GMT. Typically agents
request configuration updates every two minutes.
• Version: the version of the agent software running on this agent instance.
• Last Ping: the last time this agent sent a ping to the MMS servers.
• Last Conf: the last time the agent made a configuration request of the MMS servers.
Note: If your Monitoring agent is out of date, it will be highlighted in red on the Monitoring Agents tab of the Hosts
page.
Remember, if you have more than one Monitoring agent, only one agent actively monitors MongoDB instances at
once. See “Monitoring Architecture” for more information.
Events
The “Events” section relays information about the Monitoring agent’s operations. Possible event types are:
• “new host” occurs when the agent identifies a new MongoDB host.
• “restart” occurs when a mongod or other monitored instance restarts.
• “upgrade” occurs after upgrading mongod to a new version.
• “now secondary” occurs when a mongod instance becomes secondary in a replica set.
• “now primary” occurs when a mongod instance becomes primary in a replica set.
Alerts
The “Alerts” section provides access to On-Prem MongoDB Management Service‘s alert system. This section has
three tabs: open alerts, closed alerts, and alert settings. Following the installation of your first Monitoring agent, the
system configures an alert to send an email when the agent is down. You can delete or modify this alert as you like.
43
Types
There are four possible alert types. On Prem MMS Monitoring:
• “Host Down” activates in response to a mongod or other monitored instance restarts or fails to check in to
MMS within 90 seconds. There are no “host down” alerts if the agent is down.
• “Agent Down” activates in response to an agent that fails to report to MMS within 90 seconds.
• “Host Recovering” activates when a replica set member enters RECOVERING status.
• “Metric Min/Max Value” activates when a monitored metric passes a specified threshold.
MMS will add additional alert types in the near future.
Configure Notifications You can create notifications for any of the above alert types. Click on the + Add Alert
button next to the word “Alerts” at the top of these pages. This will raise a dialogue where you can create email, SMS,
or SNMP alerts for any of these event types.
Note: The MIB file for SNMP is available for download here.
You can configure the following options:
For all Alerts Except Metric Min/Max Value
• Address specifies an address to send alert emails. For SMS alerts, specify the telephone number that will receive
the alert. For SNMP alerts, specify the hostname that will receive the v2c trap on standard port 162.
Warning: Telephone numbers are not validated or checked, and users must ensure that they have entered the
correct number into the Alerts interface.
Note: Two-Factor Authentication via SMS text message does not currently support Indian mobile phone numbers
(country code 91).
• Type to select from text email, HTML formatted email, SMS, or SNMP alerts.
• Frequency Config (minutes) to select how often to send alerts, in minutes.
• Minimum Time Before Notification specifies how long MMS will wait before sending a notification. Raising
this value will increase the amount of time between the identification of an issue and the alert, but it may reduce
potential false positives.
You may also create multiple alerts with different minimum times before notification to implement alert escalation.
Important: The time between re-notifications increases by the frequency amount every alert cycle (e.g. 5
minutes, 10 minutes, 15 minutes, 20 minutes.)
For Metric Min/Max Value Alerts
• Host Type specifies the kind of monitored host. Options include: standalone, primary, secondary, mongos,
conf, master, slave, and individual/specific host.
44
• Replica Set specifies a monitored replica set to include in the alert.
• Field specifies the metric to use in the alert.
• Alert Value specifies the alert threshold. Use the toggles in this field to select greater than or less than and avg
per second or total per epoch.
• Contact Type specifies the method of contact. Select from text email, HTML formatted email, SMS message,
and SNMP trap.
• Contact Frequency Config (minutes) to select how often to send alerts, in minutes.
• Contact Min Time Before Notification specifies how long MMS will wait before sending a notification. Raising this value will increase the amount of time between the identification of an issue and the alert, but it may
reduce potential false positives.
Unresolved Alerts The “Open” tab displays a list of all monitored instances that have current unresolved alerts.
Check this tab to make sure that there are no outstanding alerts.
Resolved Alerts The “Closed” tab displays a record of all historical alerts. Use this tab as a record of past activity.
Alert Settings The “Settings” tab provides an interface for configuring alerts. You can delete and disable existing
alerts using the function buttons in the right hand column.
SNMP Integration The following is an example of using snmpwalk to query for SNMP health status:
$ snmpwalk -v 2c -c public mms.acmewidgets.com:11611 MMS-10GEN-MIB::mmsHeartbeatObject
MMS-10GEN-MIB::mmsHeartbeatHostnameId.0 = STRING: "[hostname]-[instance#]"
MMS-10GEN-MIB::mmsHeartbeatInterval.0 = INTEGER: 300
MMS-10GEN-MIB::mmsHeartbeatMessage.0 = STRING: "Healthy"
MMS-10GEN-MIB::mmsHeartbeatMessage.0 = No more variables left in this MIB View
(It is past the end of the MIB tree)
MIB File The MIB file is available for download at: http://downloads.mongodb.com/on-prem-monitoring/MMS10GEN-MIB.txt
Each of the main alert notification fields contain a brief description in the MIB file. an example of which follows:
Alert Trap:
mmsAlertId OBJECT-TYPE
SYNTAX DisplayString (SIZE (0.. 64))
DESCRIPTION "The alert identification"
mmsAlertGroupName OBJECT-TYPE
DESCRIPTION "The MMS group name from where the alert occurred"
mmsAlertHostId OBJECT-TYPE
45
DESCRIPTION "The MMS Host ID of the affected host"
mmsAlertHostAndPort OBJECT-TYPE
DESCRIPTION "The hostname:port of affected host"
mmsAlertStatus OBJECT-TYPE
SYNTAX INTEGER
{
new (1),
reminder (2),
clear (3)
}
DESCRIPTION "Is this alert new, a reminder of an existing alert, or clearing (closing)
an existing alert"
mmsAlertUrl OBJECT-TYPE
DESCRIPTION "The url of the corresponding MMS alerts list page"
mmsAlertMetricName OBJECT-TYPE
DESCRIPTION "The name of the triggering alert metric"
mmsAlertMetricThreshold OBJECT-TYPE
DESCRIPTION "The threshold set on the metric"
mmsAlertMetricValue OBJECT-TYPE
DESCRIPTION "The observed value of the metric which caused the alert to be triggered"
mmsAlertReplSetName OBJECT-TYPE
DESCRIPTION "The Replica Set name (if applicable) to which this affected host belongs"
mmsAlertSeverity OBJECT-TYPE
SYNTAX INTEGER
{
debug (1),
info (2),
warning (3),
error (4),
critical (5)
}
DESCRIPTION "The severity of this alert, set automatically by MMS based on alert type"
mmsAlertSummary OBJECT-TYPE
DESCRIPTION "Summary text description of the alert condition"
Heartbeat/Health Trap:
mmsHeartbeatHostnameId OBJECT-TYPE
DESCRIPTION "Hostname and instance id of MMS server."
46
mmsHeartbeatInterval OBJECT-TYPE
SYNTAX
Integer32
DESCRIPTION "Interval in seconds between successive heartbeat notifications."
mmsHeartbeatMessage OBJECT-TYPE
DESCRIPTION "Text description of current MMS server health."
Dashboards
With On Prem MMS Monitoring dashboards, you can create customized collections of charts for easier data analysis.
You can configure On Prem MMS Monitoring to automatically load a dashboard rather than the Hosts page from the
MMS settings page.
You can create multiple dashboards as your needs dictate. Use the plus icon at the top of the page to specify a name
and create a new dashboard, or select “New Dashboard...” when adding a chart to a dashboard . You can rename or
remove a dashboard from links on the bottom of a dashboard page. Dashboards have the same chart controls as the
host statistics pages.
Adding and Removing Charts from Dashboards
You can add any On Prem MMS Monitoring chart to any dashboard by selecting the plus sign (i.e. “+”) in the chart
toolbar of the desired chart on the Host Statistics page.
To remove a chart from a dashboard, navigate to the dashboard and select the “x” icon in the chart toolbar.
Advanced Dashboard Creation
When adding a new dashboard, you can select the “+ ADD CHART” button to create a dashboard that includes a
custom selection of charts, or a collection of charts from a dynamically assembled list of hosts. From this page, you
may create new dashboards or append new charts to existing dashboards. You can filter the included processes by host
type.
Specify the list of hosts to include in this dashboard by selecting a replica set or shard cluster or writing a regular
expression to match monitored processes’ hostnames. If checked, the “Host Alias In Regexp” check box allows you
to use the regular expression to select the aliased hostname you configured, rather than the actual hostname. Below
the host configuration options you may toggle an option to “group hosts in chart,” which creates a single composite
chart for all matching charts.
Below this, there are 17 chart types that you can use to select charts for this dashboard. Below the chart selection,
the final row of buttons allows you to: (optionally) test the “host regexp” to ensure that your regular expression is
sufficiently selective; preview the charts that this operation will add to the dashboard; and submit these changes to the
dashboards.
You can add and remove charts to these dashboards manually. You may also add additional charts using the “advanced
create dashboard” functionality by specifying an existing dashboard in the first field.
Host Statistics
The MMS web console provides an extensive set of features for analyzing the statistics collected by the Monitoring
agent. For a basic overview of the console, see the usage documentation. This document provides a more in-depth
guide for the “Host Statistics” page of the MMS console.
47
Accessing the Host Statistics
In the “Hosts” section, click any hostname link to view the data collected from that process. The charts on this “Host
Statistics” page are interactive, and provide tools for navigating and working with On Prem MMS Monitoring data.
Click on the “info” buttons with an i in a circle to raise informational boxes, to explore the functionality of the MMS
console and charts.
Global Page Controls
There are three to five items in the line below the MMS console’s menu bar. From left to right, these are:
• The hostname and port of the process. This is a link to the raw JSON content of the latest information gathered
from this host.
• The host type. Possible types include: “primary,” “secondary,” “master,” “slave,” “standalone,” “recovering,”
and “unknown.”
• Optionally, the name of the replica set to which this process belongs. Only replica set members have this value.
The name of the set links to a page with all of the charts from all members of a set.
• Optionally, the name of the shard cluster to which this process belongs. Only shard cluster members have this
value.
• The version of MongoDB running on this process.
The second line contains nine links that control the host statistics page. On Prem MMS Monitoring displays the current
selection in a larger font. In the second line there is a “window” selector that narrows the amount of data displayed.
These options are:
• “by minute,” which is the default setting. All charts plot one data point per minute. The “window” options are:
– “one hour,” which is the default window for this selection and charts 1 hour of data.
– “six hours,” which charts 6 hours of data.
– “twelve hours,” which charts 12 hours of data.
– “twenty-four hours,” which charts 24 hours of data.
• “by 5min,” which re-plots all charts with five-minute averages. The “window” options are:
– “six hours,” which charts 6 hours of data.
– “twelve hours,” which charts 12 hours of data.
– “twenty-four hours,” which is the default window for this selection and charts 24 hours of data.
– “forty-eight hours,” which charts 24 hours of data.
• “by hour,” which re-plots all charts with hourly averages. The “window” options are:
– “one day,” which charts 1 day of data.
– “one week,” which charts 1 week of data.
– “two weeks,” which charts 2 weeks of data.
– “one month,” which is the default window for this selection and charts 1 month of data.
– “two months,” which charts 2 months of data.
• “by day,” which re-plots the chart to display a period of time greater than 24 hours. The “window” options are:
– “one week,” which charts 1 week of data.
48
– “two weeks,” which charts 2 weeks of data.
– “one month,” which is the default window for this selection and charts 1 month of data
– “six months,” which charts 6 months of data.
– “one year,” which charts 1 year of data.
• “range,” which allows you to specify a time rage for the charts to display.
• “avg/sec,” which is the default setting. When selected, charts display the average number of events per second.
• “total,” which allows you to re-plot the charts to display the total number of events.
• “gmt,” which allows you to re-plot the charts to the GMT zone. Use this option when correlating MMS data
with server logs in GMT rather than your local timezone.
• “refresh,” which triggers a refresh of all charts.
On the next line, the “multi-chart zoom” slider allows you to change the scope of all charts at once. Move the sliders
on either end of this bar to narrow all of the charts on this page at once.
Specific Chart Controls
You may also interact with the charts individually. Using the mouse you can:
• Click-and-drag to select a portion of the chart to zoom into.
• Double-click to revert to the default zoom setting.
• Shift-click-and-drag (i.e. hold the shift key while clicking and dragging) to scroll left and right.
You can control each On Prem MMS Monitoring chart using the buttons at the top right of the chart container in the
“chart toolbar.” From left to right, these controls are:
• “Add To Dashboard,” a plus sign, takes you to a dashboard creation page where you can create a new dashboard
and add a collection of charts to the new dashboard.
• “Expand Chart,” an icon with two arrows expanding, raises a box with a larger version of the chart.
• “Chart Permalink,” a chain, links to a page that only displays this chart.
• “Email Chart,” an envelope, raises a dialogue box where you can input an email address and short message to
send the chart to an arbitrary email address.
• “View Legend,” the character i in a circle, raises a box with a key to the chart.
Chart Annotations
Annotations may appear as colored vertical lines on your charts to indicate server events. The following color/event
combinations are:
• A red bar indicates a server restart.
• A purple bar indicates the server is now a primary.
• A yellow bar indicates the server is now a secondary.
If you do not wish to see the chart annotations, you can disable them on the “Setting” page.
Clicking on any host name in the hosts table of the “Hosts” tab will give you access to detailed statistics about that
host. For detailed instructions on interacting with this data, please see the Host Statistics documentation.
49
Next Steps
The MMS web console contains a great deal of functionality beyond the scope of this overview. Continue to explore
the functionality addressed above in your own system, with your own data.
3.3 Diagnostic and Troubleshooting Guide
This document provide troubleshooting advice for common issues encountered installing the On Prem MMS Monitoring agent. Begin by working through the checklist below to ensure issues are not easily resolved. Questions and
answers also are listed below for issues not caused by easily fixed installation or connectivity problems.
For answers to other questions, see the monitoring FAQ.
Getting Started Checklist
Most problems with MMS are the result of issues with installation, connectivity, and other problems easily resolved.
To begin troubleshooting, complete these tasks:
1. Authentication Errors
2. Check Agent Output or Log
3. Confirm Only One Agent is Active and Running
4. Ensure Connectivity Between Agent and Monitored Hosts
5. Ensure Connectivity Between Agent and MMS Server
6. Allow Agent to Discover Hosts and Collect Initial Data
Installation
Authentication Errors
If your MongoDB instances run with authentication enabled, ensure MMS has these credentials. For new hosts, click
the Add Host button on the Hosts page then specify credentials for every host with authentication enabled. For hosts
already listed in MMS, click the gear icon to the right of a host name on the Host page then select Edit Host to provide
credentials.
Please consult the Authentication Requirements documentation for details about how to use authentication.
Setup Exits with command ’gcc’ failed with exit status 1 Error
This error usually indicates Python C extensions cannot be built due to missing dependencies. Type this command to
determine your system’s architecture:
uname -a
Debian and Ubuntu users should issue these commands to install any missing Python dependencies:
sudo apt-get install python-setuptools
Red Hat, CentOS, and Fedora Users should issue these commands to install any missing Python dependencies:
50
sudo yum install python-setuptools sudo yum install gcc
python-devel
If you install MMS monitoring agents on Windows, see Install the Monitoring Agent on Windows.
Agent
Check Agent Output or Log
If you continue to encounter problems, check the agent’s output or logs for errors. Here are a errors you might find
and their solutions:
AttributeError: ‘builtin_function_or_method’ object has no attribute ‘new’
This error often happens after an MMS agent software uprade. Usually the agent agent runs under Python 2.4 and the
hmac and hashlib packages are missing. To fix, either install these packages or upgrade to Python 2.5 or greater.
For more details, see Install Monitoring Agent.
TypeError: _init_() got an unexpected keyword argument ‘ssl’
This error indicates PyMongo is out of date. Upgrade to at least version 2.6.3. The agent cannot connect to hosts
without the latest version of PyMongo.
Confirm Only One Agent is Active and Running
If your monitoring agent can connect to all hosts in your deployment, a single monitoring agent is sufficient. A second
monitoring agent can act as a hot standby. Otherwise, multiple agents can cause unexpected problems.
To tell which agent is the Primary Agent, note the Last Ping value in the Monitoring Agents tab on the Hosts page. If
there is no Last Ping value for a listed agent, the agent is a standby agent.
When you upgrade a monitoring agent, do not forget to kill the old agent.
If you run a primary agent and a host standby agent, confirm both agents are the same version.
See Frequently Asked Questions About On Prem MMS Monitoring and Monitor Hosts with On Prem MMS Monitoring
for more information.
Ensure Connectivity Between Agent and Monitored Hosts
Ensure the system running the agent can resolve and connect to the MongoDB instances. To confirm, log into the
system where the agent is running and issue a command in the following form:
mongo [hostname]:[port]
Replace [hostname] with the hostname and [port] with the port that the database is listening on.
Ensure Connectivity Between Agent and MMS Server
Verify that the Monitoring agent can connect on TCP port 443 (outbound) to the MMS server (i.e.
“mms.mongodb.com”.)
51
Allow Agent to Discover Hosts and Collect Initial Data
Allow the agent to run for 5-10 minutes to allow host discovery and initial data collection.
Hosts
Hosts are not Visible
Problems with the monitoring agent detecting hosts can be caused by a few factors.
Host not added to MMS: In MMS, click the Hosts tab then click the Add Host button. In the New Host window,
specify the host type, internal hostname, and port. If appropriate, add the database username and password and whether
or not MMS should use SSL to connect with your monitoring agent. Note it is not necessary to restart your monitoring
agent when adding (or removing) a host.
Accidental duplicate mongods If you add the host after a crash and restart the monitoring agent, you might not see
the hostname in the MMS Mongos page. MMS detects the host as a duplicate and suppresses its data. To reset, select
Settings then Group Settings. Click the Reset Duplicates button.
Too many monitoring agents installed: Only one monitoring agent is needed to monitor all hosts within a single
network. You can use a single monitoring agent if your hosts exist across multiple data centers and can be discovered
by a single agent. Check you have only one monitoring agent and remove old agents after upgrading the monitoring
agent.
A second monitoring agent can be set up for redundancy. However, the MMS monitoring agent is robust. MMS
sends an Agent Down alert only when there are no available monitoring agents available. See Monitoring FAQ and
Monitoring Architecture for more information.
Cannot Delete a Host
In MMS, click the Hosts tab and click the gear icon to the right of a hostname and select Remove Host.
In rare cases, the mongod is brought down and the replica set is reconfigured. The down host cannot be deleted and
returns an error message, “This host cannot be deleted because it is enabled for backup.” Contact MMS Support for
help in deleting these hosts.
Monitoring Server
Why doesn’t the monitoring server startup and run successfully?
If you use authentication, whether or not you enable backup, confirm these properties are in the
<install_dir>/conf/conf-mms.properties file:
mongo.mongoUri=<SetToValidUri>
mongo.replicaSet=<ValidRSIfUsed>
Otherwise, MMS will fail while trying to connect to the default 127.0.0.1:27017 URL.
If you use the MMS <install_dir>/bin/credentialstool to encrypt the password used in
the mongo.mongoUri value,
also add the mongo.encryptedCredentials key to the
<install_dir>/conf/conf-mms.properties file and set the value for this property to true:
For more details, see Authentication Configuration.
52
Munin
Install and configure the munin-node daemon on the monitored MongoDB server(s) before starting MMS monitoring. The MMS agent README file provides guidelines to install munin-node. However, new versions of Linux,
specifically Red Hat Linux (RHEL) 6, can generate error messages. See Configure MMS Monitoring for details about
monitoring hardware with munin-node.
Restart munin-node after creating links for changes to take effect.
“No package munin-node is available” Error
To correct this error, install the most current version of the Linux repos. Type these commands:
sudo yum install http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
Then type this command to install munin-node and all dependencies:
sudo yum install munin-node
Non-localhost IP Addresses are Blocked
By default, munin blocks incoming connections from non-localhost IP addresses such as MMS. The /var/log/muninnode/munin-node.log file will display a “Denying connection” error for your non-localhost IP address.
To fix this error, open the munin-node.conf configuration file and comment out these two lines:
allow ^127\.0\.0\.1$
allow ^::1$
Then add this line to the munin-node.conf configuration file with a pattern that matches your subnet:
cidr_allow 0.0.0.0/0
Verifying iostat and Other Plugins/Services Returns “# Unknown service” Error
The first step is to confirm there is a problem. Open a telnet session and connect to iostat, iostat_ios, and
cpu:
telnet HOSTNAME 4949 <default/required munin port>
fetch iostat
fetch iostat_ios
fetch cpu
If any of these telnet fetch commands returns an “# Unknown Service” error, create a link to the plugin or service
in /etc/munin/plugins/ by typing these commands:
cd /etc/munin/plugins/
sudo ln -s /usr/share/munin/plugins/<service> <service>
Replace <service> with the name of the service that generates the error.
3.4 Frequently Asked Questions About On Prem MMS Monitoring
See also:
53
Frequently Asked Questions About On-Prem MongoDB Management Service
Host Configuration
How do I add a new host or server?
Click on the plus icon on the top of the hosts page.
Can I monitor Kerberos-enabled nodes?
Yes. On Prem MMS Monitoring does support monitoring for Kerberos-enabled MongoDB instances. See Connect to
Hosts with Kerberos Authentication for more information.
How does MMS gather database statistics?
In most instances, On Prem MMS Monitoring will scale its request cycle to limit more expensive statistics gathering.
The information on the DB Stats tab updates every 10 minutes, and the agent will throttle the frequency to reduce the
impact on the database. 2 Even so, the “DB stats” operation impacts the performance of your database, as is possible
when installations have a large number of databases and collections.
In these situations you must disable database stats collection. See the “DB Stats” section on the “Settings” page in the
MMS Console before starting your agent.
On Prem MMS Monitoring Agent
Should I run more than one agent?
The Monitoring agent is fairly robust, and MMS automatically sends an Agent Down alert when a Monitoring agent
becomes unavailable. For most applications, a single Monitoring agent is sufficient. However, you can choose to run
a second agent to provide redundancy. Larger numbers of agents do not provide significant benefit. See: Monitoring
Architecture for information.
Where should I run the Monitoring Agent?
The amount of resources the Monitoring Agent requires varies depending on infrastructure size, the number of nodes
and the databases it’s monitoring. Run the agent on an existing machine with additional capacity that does not run a
mongod instance or on a smaller dedicated instance.
Can I run the Monitoring agent on an AWS micro instances?
If you are monitoring five or fewer mongod instances, you can use a AWS micro instance.
2
54
The DB Stats tab will not appear until 30 minutes after you add the host to On Prem MMS Monitoring
Why can’t the Monitoring agent connect to my host?
The most common problem is that the agent is unable to resolve the hostname of the server. Check DNS and the
/etc/hosts file.
The second most common problem is that there are firewall rules in place that prohibit access to the server from the
agent.
To test the connection, login to the server running the agent and run: mongo hostname:port/test If you are
unable to connect, the agent will not be able to connect.
In addition, On Prem MMS Monitoring does currently support monitoring for Kerberos-enabled nodes: if your host is
using Kerberos for authentication, the On Prem MMS Monitoring Agent will not be able to connect to it.
How do I download the preconfigured agent?
You can download a preconfigured agent from the “Agent” section on the settings page.
How do I setup and configure the agent?
See the README file included in the agent download.
Data Presentation
What are all those vertical bars in my charts?
A red bar indicates a server restart.
A purple bar indicates the server is now a primary.
A yellow bar indicates the server is now a secondary.
Why is my hostname displayed in orange?
Your server either has startup warnings or a very low ulimit set. This can introduce problems. If your startup warning
is about NUMA, see the MongoDB documentation. If you have a ulimit of 1024 or less, see the instructions for
increasing your ulimit.
Why is my Monitoring agent highlighted in red on the Agents tab?
Your agent is out of date. For instructions on updating the agent, see: Update On-Prem MongoDB Management
Service.
3.5 Alerts
The “Alerts” section provides access to On-Prem MongoDB Management Service‘s alert system. This section has
three tabs: open alerts, closed alerts, and alert settings. Following the installation of your first Monitoring agent, the
system configures an alert to send an email when the agent is down. You can delete or modify this alert as you like.
55
Types
There are four possible alert types. On Prem MMS Monitoring:
• “Host Down” activates in response to a mongod or other monitored instance restarts or fails to check in to
MMS within 90 seconds. There are no “host down” alerts if the agent is down.
• “Agent Down” activates in response to an agent that fails to report to MMS within 90 seconds.
• “Host Recovering” activates when a replica set member enters RECOVERING status.
• “Metric Min/Max Value” activates when a monitored metric passes a specified threshold.
MMS will add additional alert types in the near future.
Configure Notifications
You can create notifications for any of the above alert types. Click on the + Add Alert button next to the word
“Alerts” at the top of these pages. This will raise a dialogue where you can create email, SMS, or SNMP alerts for any
of these event types.
Note: The MIB file for SNMP is available for download here.
You can configure the following options:
For all Alerts Except Metric Min/Max Value
• Address specifies an address to send alert emails. For SMS alerts, specify the telephone number that will receive
the alert. For SNMP alerts, specify the hostname that will receive the v2c trap on standard port 162.
Warning: Telephone numbers are not validated or checked, and users must ensure that they have entered the
correct number into the Alerts interface.
Note: Two-Factor Authentication via SMS text message does not currently support Indian mobile phone numbers
(country code 91).
• Type to select from text email, HTML formatted email, SMS, or SNMP alerts.
• Frequency Config (minutes) to select how often to send alerts, in minutes.
• Minimum Time Before Notification specifies how long MMS will wait before sending a notification. Raising
this value will increase the amount of time between the identification of an issue and the alert, but it may reduce
potential false positives.
56
For Metric Min/Max Value Alerts
• Host Type specifies the kind of monitored host. Options include: standalone, primary, secondary, mongos,
conf, master, slave, and individual/specific host.
• Replica Set specifies a monitored replica set to include in the alert.
• Field specifies the metric to use in the alert.
• Alert Value specifies the alert threshold. Use the toggles in this field to select greater than or less than and avg
per second or total per epoch.
• Contact Type specifies the method of contact. Select from text email, HTML formatted email, SMS message,
and SNMP trap.
• Contact Frequency Config (minutes) to select how often to send alerts, in minutes.
• Contact Min Time Before Notification specifies how long MMS will wait before sending a notification. Raising this value will increase the amount of time between the identification of an issue and the alert, but it may
reduce potential false positives.
Unresolved Alerts
The “Open” tab displays a list of all monitored instances that have current unresolved alerts. Check this tab to make
sure that there are no outstanding alerts.
Resolved Alerts
The “Closed” tab displays a record of all historical alerts. Use this tab as a record of past activity.
Alert Settings
The “Settings” tab provides an interface for configuring alerts. You can delete and disable existing alerts using the
function buttons in the right hand column.
SNMP Integration The following is an example of using snmpwalk to query for SNMP health status:
$ snmpwalk -v 2c -c public mms.acmewidgets.com:11611 MMS-10GEN-MIB::mmsHeartbeatObject
MMS-10GEN-MIB::mmsHeartbeatHostnameId.0 = STRING: "[hostname]-[instance#]"
MMS-10GEN-MIB::mmsHeartbeatInterval.0 = INTEGER: 300
MMS-10GEN-MIB::mmsHeartbeatMessage.0 = STRING: "Healthy"
MMS-10GEN-MIB::mmsHeartbeatMessage.0 = No more variables left in this MIB View
(It is past the end of the MIB tree)
MIB File The MIB file is available for download at: http://downloads.mongodb.com/on-prem-monitoring/MMS10GEN-MIB.txt
Each of the main alert notification fields contain a brief description in the MIB file. an example of which follows:
57
Alert Trap:
mmsAlertId OBJECT-TYPE
DESCRIPTION "The alert identification"
mmsAlertGroupName OBJECT-TYPE
DESCRIPTION "The MMS group name from where the alert occurred"
mmsAlertHostId OBJECT-TYPE
DESCRIPTION "The MMS Host ID of the affected host"
mmsAlertHostAndPort OBJECT-TYPE
DESCRIPTION "The hostname:port of affected host"
mmsAlertStatus OBJECT-TYPE
SYNTAX INTEGER
{
new (1),
reminder (2),
clear (3)
}
DESCRIPTION "Is this alert new, a reminder of an existing alert, or clearing (closing)
an existing alert"
mmsAlertUrl OBJECT-TYPE
DESCRIPTION "The url of the corresponding MMS alerts list page"
mmsAlertMetricName OBJECT-TYPE
DESCRIPTION "The name of the triggering alert metric"
mmsAlertMetricThreshold OBJECT-TYPE
DESCRIPTION "The threshold set on the metric"
mmsAlertMetricValue OBJECT-TYPE
DESCRIPTION "The observed value of the metric which caused the alert to be triggered"
mmsAlertReplSetName OBJECT-TYPE
DESCRIPTION "The Replica Set name (if applicable) to which this affected host belongs"
mmsAlertSeverity OBJECT-TYPE
SYNTAX INTEGER
{
debug (1),
info (2),
warning (3),
error (4),
critical (5)
}
DESCRIPTION "The severity of this alert, set automatically by MMS based on alert type"
58
mmsAlertSummary OBJECT-TYPE
DESCRIPTION "Summary text description of the alert condition"
Heartbeat/Health Trap:
mmsHeartbeatHostnameId OBJECT-TYPE
DESCRIPTION "Hostname and instance id of MMS server."
mmsHeartbeatInterval OBJECT-TYPE
SYNTAX
Integer32
DESCRIPTION "Interval in seconds between successive heartbeat notifications."
mmsHeartbeatMessage OBJECT-TYPE
DESCRIPTION "Text description of current MMS server health."
3.6 Events
The “Events” section relays information about the Monitoring agent’s operations. Possible event types are:
• “new host” occurs when the agent identifies a new MongoDB host.
• “restart” occurs when a mongod or other monitored instance restarts.
• “upgrade” occurs after upgrading mongod to a new version.
• “now secondary” occurs when a mongod instance becomes secondary in a replica set.
• “now primary” occurs when a mongod instance becomes primary in a replica set.
3.7 On Prem MMS Monitoring Reference
This document contains references of the different types of hosts, databases, and other statuses that may occur in On
Prem MMS Monitoring.
Host Types
The possible values for the “Type” column in the Hosts page are:
• primary
• secondary
• standalone
• master
• slave
• unknown
• recovering
The “Host Type” selector on the advanced dashboard creator also includes:
• conf
• mongos
59
Note: The host type column may also have the value “no data,” which means that On Prem MMS Monitoring has not
received any data from the Monitoring agent for this host. Possible causes for this state:
• If the Monitoring agent can’t connect to the server because of networking restrictions or issues (i.e. firewalls,
proxies, routing.)
• If your database is running with SSL. You must enable SSL either globally or on a per-host basis. See Using
SSL with On Prem MMS Monitoring for more information.
• If your database is running with authentication. You must supply On Prem MMS Monitoring with the authentication credentials either when you’re adding a host or by clicking on the edit (i.e. “Pencil” button) on the right
of the entry on the “Hosts” page.
Host Process Types
On Prem MMS Monitoring can monitor the process types:
• mongod database processes
• mongod arbiter processes
• mongos
• Monitoring agents
Event Types
Types of events in the Events section of the MMS console:
• new host
• restart
• upgrade
Alert Types
The available alert types are:
• Old Host Version
• Host Down
• Agent Down
• Now Secondary
• Now Primary
Chart Colors
• A red bar indicates a server restart.
• A purple bar indicates the server is now a primary.
• A yellow bar indicates the server is now a secondary.
60
Status Page
• cpu time
• db storage
• page faults
• repl lag
• replica
• network
• cursors
• queues
• connections
• background flush avg
• lock % 3
• btree
• non-mapped virtual memory
• memory
• asserts
• opcounters-repl
• opcounters
DB Stats Page
• collections
• objects
• average object size
• data size
• storage size
• num extents
• indexes
• index size
• file size
3 For versions of MongoDB after 2.1.1, this chart has a drop-down menu next to the tile that lists available databases, including “global” to
represent the global lock for this host. Select a database to see its lock utilization. See the documentation of lock reporting in serverStatus for more
information.
61
Database Commands Used by the Monitoring Agent
• serverStatus
• buildinfo
• getCmdLineOpts
• connPoolStats
• _isSelf
• getParameter
• ismaster
• getShardVersion
• netstat
• replSetGetStatus
• shards.find
• mongos.find
• config.chunks.group
• oplog.find
• collstats - oplog.rs
• sources.find (slave)
• config.settings.find
• dbstats
• db.locks
PyMongo Error Codes
10057
Authentication error.
Browsers Supported by the MMS Console
• Chrome 8 and greater.
• Firefox 4 and greater.
• IE 9 and greater.
3.8 On Prem MMS Monitoring Agent Changelog
Monitoring Agent 1.6.3
Released 2013-11-18
• Responsive agent. The agent can now send data to the MMS server when it detects changes, such as adding a
new host.
62
Released 2013-11-05
• Fixed bug that led to high CPU service with log collection and the agent’s connection to a mongod instance
failed.
Released 2013-10-21
• Add options in the agent’s ‘settings.py to suppress database specific statics at the agent level.
• Improved error messages in agent logs.
Released 2013-10-07
• For monitored instances on MongoDB 2.4.x series, this release fixes a bug that reported monitored mongod
instances as down during foreground index builds.
Released 2013-08-12
• Collects timestamps associated with hardware metrics, allowing more accurate minute-level charts to be displayed in MMS.
Released 2013-07-15
• Collects additional information on shard names
Released 2013-04-23
• Removes the writeBacksQueued queued call
Released 2013-03-20
• Does not log an error if unable to collect profiling stats.
63

MongoDB Management Suite Manual

Transcription

Similar documents

How to login into MMS Grade Book Parent Portal Parents,

Why Customer Service is Critical

Mulberry Middle School News Letter December 2015

Visualizing System Center Data with PowerBI and PowerView

IntelliChief JDA Integration SpecSheet_V1002.indd

SavinGreen Mobile Phone Post-‐Paid Auto Roaming (Optus 3G

ADVANTAGE 1.0

13th Annual MinnTS - Minnesota Microscopy Society