Document 6540026

Recruit Sufficient ,Sample to .Aclirwe ?1us1F* Goals
Sample an re&n to the number of participants being evaluated in a sdd^% and it is a
cudc ai factor in detarmung how pcieeise a sWs fiad'mgs can be while remaining shtistit ally
siguv6eaut Iiaane%mr, sample size is also a key cost driver Utilities xinist weigh the beor.fits of
increased precision against the COSIS of MMRMMM# and i1111plelleubbOR to detamiae the
appropriate balance . Although sample size cakulabous, we mpLe to the type of statistical testing
being per£iume+d, the €alCUlalim will Uely include the Mowing
a
n
a
•
a- =apIE sia:lepsramis the marober ofpaticqwts wfian each groUp' being ftsird
A a E-mqnsent the amount of climige that most be Observed and the tolerable erW (+ivalue) seslotrMy; these values may be adjusted in accordance with the program's ynk
a aadP - represent the probability of ea+aor there is some flexibility in sssigoiq dose
values ytiklik certain generally wcqAed boundaries
a- standard devistiok in The planmiog shqes, Ibis is usually an esfimate and is not within
the eapeiimentefs control to 3naaipula6e (Olt & Laagnealer 2010)
AltLoirgh it is desirable to have a low A at E value and low a and P values, lowering
these x4boes increases n. Tbudcffe, The program team must eucfuIl^r assess the goals of The
evakwhosi as well as the feasibility and cost of increased sample to dekn3me the appropriate
balanceOver-Recruit in Axceunt for Dropouts
Owe the sample size goals are estikrshe1 it is important tint program, maWers
remember that not everyone recruited in the beginniog of the program will and up providing
useful data at The paog[am's conclusion Gbs6ouners, may enuofl but not follow through with
mstlllabam, name before the evaluation period is comcbuie& or iotso3xe other vanables that
cannot be controlled for For these reasms., it is important that programs recruit more parbckxmh
than required for the final aaalysis to ensure that safficimt data is available at The stody's
conclusion Appropriate rates of ov cr-raauitmeat vary, but faetars to consider include_
Demands an the castomner- Iiow eW is it to perticipabr^
Leigh of the evahiatiun period - How zwor}; dapsfmo^s^years of daft do are needed?
Participant chimicleristics - Who is being evaluated? Homeowners? Renters?
Commercial property oavaers? Each pogulatim will have unique cluractrristics
impec1iag the Iilwhlnod+of Mm4lroughP6a for an Assessment Peaioil Sufficient for Evaluation of Long-Term Impacts
Allowing tam to assess lomg-tra impacts is especially critical for behai,i;aor-bosed
savings when persistence cannot be assumed Savings observed within the fast ym of
parhcqwban my be inflated based an short-term engagement at they may mark the beginning
of 1omg-las6ag energy savings habats_ What Iamg-Eerrd means for any given Program will
depend an the zhlity's goals, but it is iaI t^t to remember that behavior-lased UvmP Cannot
be assumed to edend for a period lmpw than the period during which they weffe evaluated.
CW12/IGEEE Summer Shady ao EneFgy E1icierwy in BuldiMgs
228T
100
Combine Various Data Soorces to Improve Understanding of Impacts
I:nsMs conamWoon dab is, of emu, The most essential component m
enerU savings imqacts- Hoavever, addiaand dab can maRY expami the opportunities for
analyxis_
WeaAha data can be used to normalize eangy' consumption- fsrilifaf3ng caaapariam
between groups- Wenther nonuslindm however, may not be necesauy if fim is a
randomly asa*aed cautra1 groupDeft from o&w utility progiaans can be used to klenhfy energy savings not athiluemble to
the progaan's fteatmeni(s)Pa^icapant feedback can provide msighi into outside fs^o^s that may impact enein
cousunuphon, as well an a qmlit&vr assessment of the ausbomw expuience,
?lEP Texas SM.^LR.T View Pi-ujW; A Case Study
To illustrate what these best practices look like when appiied- a hqot4eM overview of
the AEP Texas SWRT View Project research methodology is provided belcrw. This research
plan illustrates the implementation of The best practices deacn-bed above as well as provikWig an
oppatuuzt^ to discuss alternative appmoarhes and oppohunties for unprovment of 4Le plant
Inaernal Device POK
In preparation for launch of the fsdl. sdrdy_ a pilot study was camdocted with Pia Team
employees to.
Identify= devices that are the best fit for the study
Assess the level of support that will be reqdmed for lauach of the project
Idmafy c+oaoID^m *xU&al issaes, so that
^
pmventative measures can be aipleummied
project sappm# slaff is prepared to Made common
s
•
Participants m the internal pilot were re=Jed through an email to local AEP Tem
officrL Volunteers received a fee device as well as instructions for pffovisioning dw device (Le,,
establishing a coaner.tian between the device and The me6er)- PMcipaat support was nianaged
through a dedicated email account, and all issues were logged m a simple backing syslrmSevrenl weeks a&r Pronsioning their &-me= pilot Porticipats. completed a short survey about
Ibdr expaieuce installing and interacting with the deviceAs a result of the pilot find- , the project vom made several key adjustments to the
research pLm, which will facilitate a saaoodta rollout to paticrpants- The changes mduded:
Device Selechm: The pilot revealed that one of the eligible devices had too mony
bamen to iastallatiom and too little visidality to occupants once msbdle& Far these
reasons, Or project vom eleded. not to include it in the fuIl shulg-
Q20i2 ACEEE Sunwner Shm* on EneFW Elicieecy in 6uldW
2312
101
Pre-delivery veiifiCation: AEP Texas idesifified two pm-ddivey verification skah*s
that significaldy reduced the pacerdpc of participants who encountered tecloncal issues
duinig the device, setup pw%L
Staggered Uelivray Schedule: Based an support needs encoambered daring the pulot, the
project teem detesommd. plissaug the delivery schedule over a period of 4-6 weeks would
keep the support needs of participants at a manageable level
Tbus internal pilot may be considered a type of soft launch. However, 8re rerroitouat
Phase also begam with a so81nmcL (see be6ow).
Samplmg PI= and Screening hftmdiodebp.
Because 1I1Iy programs require an opt-in recrudhmat melhodology, tbe pncWt team
dc6eammed dot the population for tLe¢ study would be likely particqmmb in an AEP Tens IED
program Knowing that AEP Texas caukd not rJann savings for an 1I1L1 tmiless a program
parbcpW had chosen. to receive one from the ati[ity, the hum knew that there was no value in
atteoipt,mg to apply the sesuhs mom geaecaUyPa:iheqmmb will be recanted through an -1
survey to c4zfim that they nod The
pwject's screening czibeii Because AEP does not lave emad addresses. targeted postcards with
a fink to the aunvy will be mailed to randomly selected siagk-faorily residemlial homeowners
with advanced me6em To encourage a high response nCe, survey invitations will promote
availability of a saeepshloes incentive (e.g-, respond to be entered to win one of tea $100 VISA
gift ^) Qudtfymg survey perbeipants will be grim an opportunity to indicate &w
willingness to participate in the AEP Texas SMART View Project', i3eezuae knaqvlydpe of the
suivey's purpose could influence the responses provided by respondents, 6e, oRniftaidy to p"
new 1erbnology will not be mmhamed until eligibility has been confirmed
ln addition to allowing the project to pre-screen pabcTonts, recruiting participants
through a survey facilitates the recruit-amd-dmy methodology for establishing a control group.
Because qualified participants will indicaDe fim willingness to participate rather than actually
applying to be a parbeipesd, the project beam will be able to confirm that members of the caotool
group aresimlar ID members of the treatment group in every respect other than the treaho®t.
Itw-rokment Soft Launch
Because this study relied an a recruitment mNhod onu before used by the oftlity it was
essential to confim the response rates and qualifying rates estimated in. the research PILL The
cost and reach of fie postcard recruitment effort made the soft launch especially critical; with a
plm to send over 100,000 postcai& it was very unpiaQaant that the team knew what ID expect
and make no mistakes- The postcard outreach was soft launched with 5°a of the total anticipated
mvitaaane to use= actual response rase and qualifying rate, as well as test =uIple icer^t
messagesTo ensm the highest possible response rme, the team developed two unique reannunmt
postcard templates to test dining the so8lnmch Both templates included essential dernmis such
as the survey ud the sweepstakes offer, and the emhq conducting the research. One template
was designed to use be least possilile copy wide The otler was fo®atled as a leflu and included
adddmmat elcoments often listed as best prachm for survey reawtmesit, including a privacy
4=2 ACEEE Sunrner Slody an IEmtHy Etisieacy in 8uitlinps
2-269
102
disclaimer and an explanation of the xseaTch pm.pose. Each tesoplaie included a uniqum idmti6uer
in The survey on to back resu[4sInd^mgly- tue team found that the more formal invitation was less effCtive in
ma>sting Lomeammis; the response rAe was one half of The response rate observed wfih The
more concise mvi,iaton- The learn also evaluated whether qualifying rain vaned based on the
postcard zzceived, but no sinifuea^t difference was obsaved As a resuit, the team sdopted the
higher response aate template as the basis for the final template
AMamigh the qualifying rate of survey respondents was qmtdeae to the eshusled value
of 35%., the response rule for the postcards was signu6canfly loner flan projected MAN. as
opposed to 1.5%). As a resal>', Or project team made several modifications to The posicard to,
Memsr response IRL- including
n
n
n
n
addition ofan expected time&amue, for response to mobvate action
adddwn of the uitiWs logo to mcmw perceived credibility
^ sin of the survey url to draw atlealion. and improve reodability
use of colored cudsbo& (ahiie was used in the initial soft lausch) to draw attention
The early discovery of the low response rate also allowed the tow. to prqme an
alternative research design in case the sample size required for the anginal design could not be
aclurued. To e9nme that suffnent sample could be obbuned. tlamo this zeenaluseut
nuedadolog,y, a second soft launch was conducted before the postcard recaudmeat was folly
launched (resolts are pmdiag)Treml®ent Groups
Became the project tusm was iaarirsted in detamnoing if the impect of HMs might be
augmented by The diahNbntioa of mqqdemeoW edacafiosat materials, the project uses a matrix
shucds+e to Wiese the tealaumt and control groups. FFacch fteameot is being tested in isolation as
well as in combination with other tteammLs, so 69 when the project twmeludrs, the ham wilt
be able to assess the impact af esch individually.
Ta61c 2. .AU Texas SAURT
Project Treatment
cawocatis
-- -,-I
Device Type
ltesot
Not PHes®t
^evioeli
Al
A2
Deaioe B
BI
B2
Nb ^ics
01
W
Pwbcqmb who meet the srneming criteria will be randamly assigned to one of lhe sus.
poaps. The number initially recrmted into each group will be 20'% `. higher than tire targeted
sample size to, account for participants who do not successfully set up &m 11M or most be
removed fain the assessment for odw reasons.
c Ple9e main fLti this is a poftmmy de9iBa md is MWt m^o®e Is
sdswlababq nfdq*lEpwtiripmtr..
C012 1UCEEE Sunmer SWdy an EneFW Efi¢iermy in Buildops
on the ^Ns of an YecaEftmeoa effim
2-M
103
Sample size
Like many ptopems (espeeiallg Fogrants with opt-in reenutmmO, the AEP Texas
SMART View Project had limitations that r+equimed the tum to make campaomsiaes regarding
necessary sample sim, Mi^ sample size needs fur the hypadiew test were catwlsted based
an The followint Sninuta for a two-sided lest oanwiag means of iadepmdicoR s®ptes_
ra=^a (Zg^`2 + ZP)W
A2
(Oft & Ianpecker 2010)
The a and P values were set at 0-05 and 0-1 respeetivety-The o was estimaled based on
past energy c.aosumption data flour the repon-2 The estimation of tl was based on the savinp
that would be required to Justify the cost +oAf 1he devices to the utility and the range of sarinp that
could be reasamd* amliopa6e,d based on previous resmwh- Due to flit relatively high Situ raao,
the catt11taled. A was qY&E ldWBecause the CitC'Otied niumimin sample size was low, a second calculation was used to
deinoustrahe the value of increasing the ssimple Sls^+2 beyond the inumminn required for the
hypodiesis test This calculation was based an tlle confidence minvat for comparing means of
^ samples-
E
1
R
1
-j- -
R
CUtk ^ Laogµecker 21i113}
The some a and o values were used m this calcoluhaa T.hm K was nonipubded upwards
to demonstrate how increased samPle size allows far a reduced aamficlmoe admvaL We*bW
the results of this calculation and the interest m poEC1Si0R (aSkdwAkd by a re&iced canfidencE
mkua^at^ apinst the compliaftms of recumbag participants and &e need to control costs, the
project fem dehsinined that a sanple size of 100 participants per cell (pAus 20% to account far
dnogouts) was required and that, when appropaiabe, cells would be oonikuned during analyns, to
provide increased precision for comparisons. between larger participant graPSAssessinen.t Period
Due to a hard deadline at the end of 2013. the AEP Texas SMART View Project will
indudr an assessment period of 14 to 16 monfLs. This Van of tine, will allow the project team to
assess impacts beyond tLc first year It will also indUde two suamris, which is a critical pmud
for madpnti demand m Texas- However. if savings are abmenu+ed and c4mdiuur into the
final mo>fis of the assessuient penud, The team will net have sufficient data to delmmiar the fill
peimlmae of 8tie impacts- (If this ocacmm the goject team may recommend a fatlow-up
evaluation in saes aqumt yews,)
H¢u®e tie so* win We '
zo*sis, aaas bmd on yew-1o-pw chop a dmMxeq
^mopnea, ndw d= total elecMatg moMmpom
CD'12 AGEEE Sumaner SWdy an EmffW Efieieneyr in Buik%nps
2-291
104
Past Tr eatnent Surr+eiA$rt the completion of the ssoessmeot periA the paojact teinn will c?on&x.^t a survey of
participants tnducling the comlrd group) to assess_
'
•
•
•
PWhCVWA perception of IHM' zmPact mcbmlinU
•
Hehavim cLang;es
•
Eqmzornd upgrades and borne effidenc)r miprmemenis
Participant satidwhon With the &wiec
Changes m awmenesshaidershm&W 1t-mls evaluated during initial su+e3^
Other facm that may have impacted energy asagr (i+enovatims, change m number of
occuqwmiz, etc-)
This survey data will be combined with pxracqmtLon data for aW MhRT View
participants who also participated in other AEP Teses energy efficiency pQoqpmms- Analyzing
this data in camAwnhon with the consmupbon data will allow the pQapet beant to develop a
deqw understmuling of the proW€t's impacts, incbxbw.
Clross-pwgcm► benefits - Are pea* with a display mffe likely to participate in other
uffd3`Vonsoffed Progmm`3
Quaktahve participant experience EeeAa& -17ro the participants parocive a bowWAdditiaoal factors that impact energy savings - Are certain sepments of the population
(yovWcad_ Lqghllm mcamr,highJiow canmumWhw etc_) mo®e likely to show savings
that others?
C"onch.sioa
There are always caukands that make it difficult for programs to achieve optimal
design, and each program tmm mat auei& The tndeoffs,. These best practices pum& an outline
of zoqatandt considerations for the program dmp proem_
•
•
•
•
•
•
•
•
Begin with a soft launch
Define the population and align the saoqpie to auaacinoim external validity
Include a€antr©1 emap thDvommiumrt internal validity
Design treatment groups In isolate impacts of research variables
Recruit suffideut sample to achieve anatg^sis goals
ChTaae+enut to account ifor dropouts
Plan for an amassment period sufficient for evaluation uf kmg-trsm impacts
Combine various data sauces to improve understanding of unpacis
While liaise rer^mdatims are especially important fior research and development
psojxta, they can improw the appostimities for asaessmmt of any program the will require
verification of energy savings.
jIf swdin=Wct is aLanaed, itwill de ueomuy to di9c+omode ervmpm nrodd doutsltaomtio&
@ID612 AiCEEE Summer Study on EnerW Elieiency in HulMMgs
2-292
105
References
El^uItIatatinea, K_, K A- Damfly, & B. A- Laibaer. 2010_ `=Advanced Mewing Initiatives
and ResidmW F®edbwJk Ptoga®s: AMeta-Redrvr fimr Household Elechicity-Saving
Oppatunoties.' Washinghw D.C..: American C:auxa1 for an EmeWr-Efficiadt FconaW_
EPR.i. 2009_ Readmlim[ ebcirncity ecsefoadback A research syvmdew and ecaanoxwfiamwwork.
Palo Alto Cali£_ Electrwal Pvaw ^. ewach IosWWe.
Ott, R L & M Lampeckeor_ 2010_ ,In IAbnvdWctian to Shwishcal Mddsa& and Dab Analysis,
Sixth E*fim Belnwt Calif.: BrooksXo1e.
Q2012IlCEEE Sumner SWdy on EntW Eiicieecy in 9uldiW
2-293
106