Kepler Reliability

Kepler Reliability Metrics
And Their Use in
Occurrence Rate Calculations
Steve Bryson, Kepler Science Office
With Thanks to: Tim Morton, Joe Catanzarite, Chris Burke, Mike Haas, Jason Rowe,
Natalie Batalha, Jon Jenkins, Fergal Mullally, Joe Twicken and the Kepler Team
KOI-988.01
≠
KOI-6253.01
The Kepler Planet Candidate
Population
•  Kepler planet candidates (PCs) are selected
from detected transits (TCEs)
•  Select periodic transit-like signals
•  Avoid identifiable false alarms
•  Remove identifiable false positives
•  If the signal is plausibly a transit, and it is not
identifiably a false alarm or false positive, it
becomes a planet candidate
•  Historically selection was via manual inspection, now
fully automated (Coughlin, poster #23)
Identifiable False Positives
•  Astrophysical false positives (AFPs) are
identified by
•  Light curve shape examination (grazing eclipsing binaries)
•  Pixel examination (background eclipsing binaries)
•  Works well at high SNR, fails at low SNR
Difference Image
Direct Image
Easy to see
good planet
transit shape;
determine
location from
clear signal in
the pixels
Difficult to see
transit shape;
location cannot
be determined
from pixels
Difference Image
Direct Image
So What If You Don’t Know the
Transit Source Location?
•  Morton & Johnson: Kepler PCs < 10% false alarm rate
•  But this assumed all background transit sources > 2” away
have been removed
•  Prior to such removal, KOIs have ~30% false positive rate
Many False Alarms near 372 Days !
•  Thermally dependent periodic systematics
•  Leads to orbit-coupled periodic spurious signals
KOI 5302.01
Period 372 days
Marked FP on archive
•  Leads to many many
detections near 372 days
The Case of KOI-6981.01
•  Shallow planet candidate at 593 days, 1.9 Re
•  Passed all tests!
•  Super-Earth in the habitable zone!
The Case of KOI-6981.01
•  Shallow planet candidate at 593 days, 1.9 Re
•  Passed all tests!
•  Super-Earth in the habitable zone!
•  But the pixels say:
Sudden Pixel Sensitivity Dropout (SPSD):
A discontinuous loss of sensitivity in a
pixel, usually due to a cosmic ray hit.
This is one of three transits; the other two
were marginal
What’s Wrong with High-Confidence
Thresholds?
•  Currently a detection is either a PC or not
•  Threshold is set to moderately high confidence
•  Result: very few detections in the low SNR Earth-analog
regime, causing large uncertainty
•  But those detections may have a higher than average FP
rate
From Burke et al. 2015
Reliability Metrics Coming Online
•  The Kepler Science Office has been developing various
reliability metrics
•  Astrophysical false positives:
•  Positional probability (probability transit is on target star)
•  False positive probability (probability transit is not planetary) (Morton
poster #61)
•  False alarm metrics:
•  Statistical bootstrap (Seader poster #83)
•  Image artifact flags (Clarke poster #22)
•  Machine-learning Classification
•  Into PC, AFP, FA (Catanzarite poster #20)
•  Model fit residuals, MCC chains (Rowe poster #79)
•  Have begun to appear on the exoplanet archive
•  http://exoplanetarchive.ipac.caltech.edu
Reliability Metrics Under Development
•  Shape analysis for false alarms
•  Measure whether a systematic or transit best fits the data
(Mullally posters #63, #64)
•  Auto-vetting results
•  (Coughlin poster #23, Mullally poster #65)
•  More to come as we explore injected transits
•  (Christiansen poster #21)
•  Transit Inversion should measure rate of many false
alarms
Inverted data
•  Help turn metrics into probabilities
•  (Hoffman poster #35?)
Normal data
Using Reliability Metrics
•  Positional and False Positive probabilities really are
independent probabilities of occurrence
•  So can be multiplied to form a prior or weight for each PC
•  Machine learning probabilities are probabilities of
classification, not occurrence
•  Very not independent of other metrics
•  The various false alarm metrics are not even
probabilities (yet)
•  We need to develop a way of giving more weight to a signal
best fit by a transit vs best fit by an artifact
•  Many of the metrics are not independent
•  Careful to not over-count!
The Stakes
•  Estimates of Eta-Earth are reliant on few low-reliability
detections
•  Naïve toy cartoon experiment: compare Burke et al
results from Q16 with Q17, which had a more (maybe
too) stringent filtering of false alarms
•  Being sure to use the correct detection efficiency parameters
• 
Thanks, Dan Foreman-Mackey for the wonderful Python notebook that made this comparison easy!
Q16, 154 objects
Q17, 125 objects
The Payoff
•  Better statistics at the low SNR regime
•  Proposal: more less-than-high-reliability PCs with
known reliability rates
•  How would these be incorporated into occurrence
rate calculations?
•  Weighting planet detections for inverse detection
efficiency?
•  Mixture modeling for Bayesian methods?
•  Detection efficiency is dependent on allowed reliability
•  Much work to be done here
•  Better estimate of Eta-Earth