Admin Tech Clash: Discussing Best (and Worst) Administration

Admin Tech Clash: Discussing Best (and Worst)
Administration Practices from
hundreds of customers
Presented by: Christoph Adler
Ben Menesi
Speakers
Christoph Adler
 Senior Consultant at panagenda
 15 years of IBM (Lotus) solutions
experience
 Since 2007:
 IBM Notes Client Management
 ICS Infrastructure Analysis and
Optimization
 Good to know:
 Lives in Germany, hence the funny
accent
 Frequent traveler and speaker at
international conferences
 No energy drinks – squirrel alert!
 Beer or wine? Beer!
Ben Menesi
 Head of Product at Ytria
 IBM Notes Administrator and Developer
wearing both hats
 Working with Ytria since 2011
 C API Development
 Server side Administration
 Enterprise IT Security
 Good to know:
 Born Hungarian, raised in Hungary &
Switzerland, living in Canada for the
past 5 years
 Frequent traveler and speaker at
international conferences
 Whiskey or Vodka? Beer with
Whiskey!
Introduction
Your collaboration infrastructure contains a TON
of valuable data
In order to extract this data you need to know
WHERE to look
Visualization helps to see trends and patterns and
understand relationships
IBM can help with IDD (IBM Domino Doublecheck)
We (Ytria / panagenda) can help with our software
solutions, services and expertise BUT
Scout‘s honor: We won‘t talk about our
products in this session!
Agenda
User Activity
Server-side Logging
Client-side Logs & Performance
Usage Activity
Usage Activity
 Why should we care?
Understanding usage data allows us to find
unused / heavily used applications
In the light of this data we can calculate resource
requirements and optimize performance
We can profile our high impact users and
processes and optimize licensing
Usage Activity
 What data source should we use?
 Valuable high level
reports
 Not very
customizable
 Lack of
customization leads
to inaccuracies
 Very detailed logs
 Really crucial info
buried inside one
text field
 Unless through 3rd
party tools: NONO
Activity
Trends
Log &
Catalog
DB User
Activity
3rd Party
Solutions
 Great amount of
detail
 Only available per
database
 To be discussed later
 Amazing products
out there
 Some installed on
server side
 Some client side
 To be discussed later
Activity Trends
 How does it work??
Log.nsf
Catalog.nsf
Names.nsf
Activity Trend
Collector
Activity.nsf
Statistics
Activity Trends
 Activity data – advantages
 DB Usage (active / inactive), space growth, etc…
 User details (active / inactive), connection times, dbs and
network used etc...)
 Server tasks (HTTP, IMAP, LDAP, Replication etc...)
 Some visual reports
 Activity data – disadvantages
 What you get is what you get – no digging deeper
 Raw data isn’t always used properly
 Think about a DB marked “Active” with only servers and
agents using it
 Doesn’t properly handle space used percentages
Log & Catalog
 Activity data – advantages
 It’s only two databases to analyze NOT hundreds or thousands
 You’ll get the full picture instead of only application usage
 Think about usage of system databases or client overhead
 Activity data – disadvantages
 Only contains data of the last 7 days
 Actually: There is an undocumented parameter for the Log=
notes.ini line:
 LOG=logfilename, log_option, cant_touch_this, days, size,
[activitydays]
 Data structure (Later)
Log & Catalog
 How to get valueable data out of log.nsf
• Create an additional view
• View selection formula: SELECT FORM = "Activity“
• Add columns that are interesting in your scenario
Log & Catalog
 Details: Database usage document
• Note there is a 1400 activity entry maximum per database (FIFO)
• There is also a 64K size limit for the user activity
• More details in IBM Technote #1086245
Log & Catalog
 Database activity: catalog.nsf
• related information, but
different focus
• Full text index details
• Replication information
• ACL overview
Note:
Domino does not distinguish
between user, server or maintenance
tasks activity at this level
Log & Catalog
 Example visualizations: Domino Environment Overview
Log & Catalog
 Example visualizations: Network Bandwith Demand
Log & Catalog
 Example visualizations: User Impact
Log & Catalog
 Example visualizations: Database Demand Characteristics
Log & Catalog
 Example visualizations: Network demand
Log & Catalog
 Real customer example:
 15 European Locations
 8000 Users
 1200 Domino Apps
 Logistics
„We want to mobilize the 20 most heavily used applications that
sales users throughout Europe access in read only mode. How can
we find them?“
Log & Catalog
Domino Data
 Collect session logs from all servers  log.nsf
 Collect nsf inventory from all servers  catalog.nsf
 Collect person information from all users  names.nsf
Non Domino Data
 Merge organizational info to person info
 Analyze and Visualize
Log & Catalog
 Applications used from Sales in read only mode
Log & Catalog
 Applications used from Sales in read only mode
Log & Catalog
 Applications used from Sales in read only mode
Log & Catalog
 Applications used from Sales in read only mode
DB User Activity
 Recorded User Activity
 User Activity can be a *VERY* rich source of data (incl. HTTP!)
 Tip: No_Force_Activity_Logging=0 (Default value) allows on ALL
dbs
 Tip: Set confidential to prevent users with lower than Designer
access
DB User Activity
 Prior to ODS 48
 Entry size allocated 44 bytes, object size 61600 bytes (max 1400)
Date &
Time
Reads
Writes
User
 ODS 48+ (undocumented improvement)
 Entry size 92 bytes. Object size 128800 bytes (same max 1400)
Date &
Time
Reads
Writes
Updates
Deletes
User
 Curious: only two additional counters added, yet ODS48+ entry size
is over 200% the size of ODS48 – why?
DB User Activity
 Does Notes only log what it displays?
 Nope
Reads
Adds
Updates
Deletes
Σ
Σ
Σ
Σ
Data
Data
Data
Data
Non
Data
Non
Data
Non
Data
Non
Data
 Read more: http://techlab.ytria.com/5448/lotus-notesarticles/interpret-user-activity-traces-portrait-global-usage
DB User Activity
 Limitations
 Only available per database
 Free tools available to facilitate working with this data
across multiple databases:
http://www.agecom.com.au/useractivity
 Or implement via W32_NSFDbGetUserActivity
 Makes no difference between servers & users
 Filter user names using free or paid tools
 Only maintains 1400 entries
 Isn’t this the same for the log file activity documents?
 Wait, this doesn’t need to be an issue
DB User Activity
 Customer Example: calculating usage patterns from usage activity
Database B
Database A
#1
Entry Date
Entry Date
#1
Entry Date
#1400
5 days
120 days
#1400
Entry Date
3rd Party Solutions
 panagenda iDNA…
3rd Party Solutions
 3rd Party products for usage analysis
 IDD (IBM Domino Doublecheck)  free offer
http://goo.gl/maL0fo
 panagenda iDNA
https://www.panagenda.com/products/idna
 Ytria databaseEZ & consoleEZ
http://ytria.com/databaseez
http://ytria.com/consoleEZ
 Tableau
http://www.tableau.com
Server-side Logs
Mining logs
Domino Server
The Admin
Event lifecycle
1.
2.
3.
4
5
• Events take place
• Events are logged
• Some events captured
• Few events addressed
• Tons of events missed, angry users on the phone
Dissecting Log.nsf
 Server tasks can log events to multiple types of documents!
 But no duplicate logs: different information!
 Example: Server 1 replicator opens session to Server 2 as per
Connection Document settings
Misc.
Replication
Dissecting Log.nsf
 Miscellaneous events: what do we see?
 Some high level info (Server, Start & Finish time) and
 UI shows one text type item: Date + Time & Description
 What is an event?
Time
Status Code
Severity
Type
Addin Name
Target Server
Target DB
Target User
Dissecting Log.nsf
 Miscellaneous events – event attributes logged in multi-value Text
items
Event 1
Event 2
Event 3
EventList
Time &
Description
Time &
Description
Time &
Description
EventSeverity
Severity
Severity
Severity
EventStatus
HEX Status
HEX Status
HEX Status
EventType
Type
Type
Type
Dissecting Log.nsf
 Can we get more data?
 We can use formulas and the list of Severities, Events & Status
Codes to get a better idea of each Misc. log document
 Total # of events / document [Eventlist item]
– @Elements(Eventlist)
 Total # of unique events [EventStatus item]
– @Elements(@Unique(EventStatus))
Dissecting Log.nsf
 Highest severity / document [EventSeverity item]
– _sev:=@If(@Sort(@Unique(@Text(EventSeverity));[Ascendin
g])[1]="0";@Sort(@Unique(@Text(EventSeverity));[Ascendin
g])[2];@Sort(@Unique(@Text(EventSeverity));[Ascending])[1
]);
– Once we have the _sev variable, use icons to display (from DDM
resources)
– @If(_sev="0";"iconNormal";_sev="1";"iconFatal";_sev="2";"i
conFailure";_sev="3";"iconHigh";_sev="4";"iconLow";_sev="
5";"iconNormal";"")+".gif“
– What you get:
Dissecting Log.nsf
 A view displaying the highest severity encountered, # of events,
unique events and all levels of severities per document:
 More info here: http://www.slideshare.net/BenedekMenesi/ibm-
connected-2015
Client-side Logs & Performance
ClientClocking
NOTE:
All of this should only be done on “ONE” client – slows down client,
creates huge log files!
 Use the following to see and measure client performance





CLIENT_CLOCK=1
CONSOLE_LOG_ENABLED=1
Debug_Outfile=c:\debug.txt
Debug_Console=1
DEBUG_TCP_ALL=1 (use only if needed)
 Fileserver or local disk Performance debugging is virtually
impossible

If on >= 8.5 try adding DEBUGGINGWCTENABLED=16386 in Notes
ClientClocking
 Client_Clock=1 leads to a debug file of the following format:
ClientClocking
OpenNTF – NRPC Parser
Link:
http://www.openntf.org/internal/home.nsf/project.xsp?action=openD
ocument&name=Notes%20RPC%20Parser
Developed by:
 Jamie Magee, MartinScott Consulting LLC, US
 Andrew Magerman, Magerman Collaborative Software GmbH,
Switzerland
ClientClocking
OpenNTF – NRPC Parser
Cache.ndk
We've all been “advised to”/guilty of deleting the cache.ndk in an
effort to cure Notes performance problems/symptoms
• Some companies even delete cache.ndk on a scheduled basis
Cache.ndk
HOWEVER, let us explain the impact of that on Notes performance ...
 Increases traffic between client and server by 4,000% (forty times
more traffic, yes)
(10,000+% if the data directory is on a network drive)
 The more databases end users use, the worse it gets
 There is always a root cause – deleting cache.ndk only fixes
symptoms
 Let's look at some detailed traffic analysis for just ONE application
(the mail file) of ONE user ...
Cache.ndk
After deleting cache.ndk
Keeping / NOT deleting cache.ndk
100 users =
150 MB vs. 3,7 MB
1000 users =
1.5 GB vs. 37 MB
Excluding Fileservers!
ODS
On Disk Structure
ODS = On Disk Structure
 ODS 16 = Notes 2
 ODS 17 = Notes 3
 ODS 20 = Notes 4 (or Templates)
 ODS 41 = Notes 5
 ODS 43 = Notes 6 & 7
 ODS 48 = Notes 8
 ODS 51 = Notes 8.5 & 9
 ODS 52 = Notes 9.0.1
The difference between ODS 43 and 52 = up to 80% LESS FILE I/O
This helps with ANY client AND server hardware
On Disk Structure
CAUTION:
The ODS is not automatically updated when upgrading clients
 Except for names.nsf, bookmark.nsf, and desktop*.dsk/ndk – IF –
you upgrade properly (think ini:Setup/TemplateSetup=)
 ODS 51 requires Notes >= 8.5 and CREATE_R85_DATABASES=1 in
notes.ini for NEW db's
 ODS 52 requires Notes >= 9.0.1 and CREATE_R9_DATABASES=1 in
notes.ini for NEW db's
On Disk Structure
Good News for Notes > 8.5
 Use NSF_UpdateODS=1 with CREATE_R85/9_DATABASES=1 for a
one-time upgrade of most local databases
 NOTE: USE WITH EXTREME CARE IF YOUR DATA DIRECTORIES
ARE ON A NETWORK DRIVE!
 NOTE: End users can't access databases during compact
 For databases on server: no more downtime required, use:
 Compact [dbpath] –Replica to create replica in the
background (new in domino 9)
 More info: https://www01.ibm.com/support/docview.wss?uid=swg27039379&ai
d=1
On Disk Structure
Impact examples
 Startup time of a Notes 8.5.3 client with 3 ODS 20 apps in Notes
data = 10 seconds
 After ODS upgrade: 2 seconds
 Reduced File I/O of your disks/SAN/NAS after ODS 41 to 52 upgrade
by 60%
 Removing 70% of all old files in Data directories on SAN/NAS = 45%
less managed storage (backup)
The bad news: There is no way to tell what is “out there”
 3rd party tools or random inspections are highly recommended
 News for Notes > 8.5
On Disk Structure
IBM helps!
Encryption/Compression
Slow clients
Port settings are included in the notes.ini
 Type / Active / Encryption / Compression / Timeout
Bad idea
 Remove LAN0 / COMx / DisabledPorts
Good idea
Use TCPIP according to the client type




PORTS = TCPIP
TCPIP = TCP,0,15,0,,12320,  Desktops
TCPIP = TCP,0,15,0,,45088,  Laptops
TCPIP = TCP,0,15,0,,12288,  Citrix / VDI
Riverbed !?
Don‘t forget your Servers!
Q&A
Thank you