GreenButtonCMDWorkshop20140429

The Green Button CMD Workshop
April 21,22 at SCE
PGE
Jerry Yip (IT):
([email protected])
www.greenbuttondata.org
Ed Denson (IT): ([email protected])
Deepak Subramanian (IT): ([email protected])
Kruti Patel (BA) ([email protected])
Jaclyn Hood (Product Mgr) ([email protected])
SCE
Andrew Eisses/SCE/EIX, [email protected]
Vidyod Mavilavalappil/SCE/EIX, [email protected]
Natalie Martinez [email protected],
Genevieve Feng/SCE/EIX, [email protected]
Dale LeFevre, [email protected]
Gopal Bhageria, [email protected]
Zoe Hillenmeyer, [email protected]
Cannis Meng, [email protected]
SDG&E
Blockowicz, Brendan [email protected]
Logan, Tom [email protected]
Bristow, Johnie [email protected]
Rodoni, Madeleine [email protected]
Kundu, Partha [email protected]
Brown, Gregory [email protected]
Wiffen, Stuart [email protected]
NIST
Marty Burns, [email protected]
John Teeter, [email protected]
Don Coffin, [email protected]
Kin Lane, [email protected]
Agenda
• 1
[~2.5 hr] Architecture Guidelines
–  1.1 Each IOU Present Overview of their plans and
–  1.2 Key Definitions – SA, CISR, …
–  implementations
• 2
[~1.5 hr] Function Blocks
–  2.1 [.5 hr] Minimum FBs
–  2.2 [1 hr] Content of each FB
• 3
[~8 hr] Relationship Establishment and Management
–  3.1 [~2-3 hr] Scope
–  3.2 [~1 hr] Third Party Registration
–  3.3 [~3-4 hr] Authorization
• 4
[~5 hr] Data Transfer
–  4.1 [~2 hr] REST GETs
–  4.2 [~1 hr] SFTP
–  4.3 [~2 hr] Other
Definitions
•
Customer
– If residential it is individual (customer of record)
– If commercial it is an entity
•
Customer Account
– Organization of the bill
– A customer may multiple customer accounts
•
Service Account/Agreement
– Corresponds Customer to a meter
•
Customer Information Service Request (CISR)
– A legal agreement that authorizes a California utility to provide customer information to
a thirdparty – paper or online
•
Service Point/Location
– Logical location representing the meter at the premises level
•
ESPI
– RetailCustomer
•
Owner of one or more UsagePoints (has right to authorize)
– Subscription
•
Authorized access to a (sub) set of UsagePoints of a single RetailCustomer
– Bulk
•
Authorized access to a collection of UsagePoints governed by individual authorizations to a
third party
GBCMD ids and resources depends on
authorization level
subscriptionId
Account
relationships
Customer
1:n
retailCustomerId
Premises
Customer
Account
1:n
1:n
usagePointId
Service
Agreement
or Service
Account
1:1
Service
Point
1:1
bulkId – arbitrary
collection of
resources
Meter
1:n
HAN Device
Green Button Data Exchange
Data Custodian
Web
Service
Provider
Web
Portal
Third Party
Third Party Registration
Automated Transfer
One-time
Authorization
User
(Browser)
Retail Customer
Web
Service
Consumer
Web
Portal
Components of Connect My Data
Components of
GBCMD
Data Custodian
Third Party
Third Party Registration
Web Service
Provider
Automated Transfer
Web Service
Consumer
Web
Portal
One-time
Authorizatio
n
Web
Portal
User
(Browser)
ApplicationInformation
REST API, Notification
SFTP, TLS, Access-Token
Retail Customer
Authorization
Governed by
Function Blocks
Scope
Function Blocks for CMD
FunctionBlocks for Green Button
Connect My Data
[FB_3] Core Green Button Connect My Data
[FB_13] Security and Privacy classes
[FB_14] Authorization and Authentication (OAuth)
[FB_19] Partial update data
[FB_31] Core Rest Services
[FB_32] Resource Level REST
[FB_33] Management REST Interfaces
[FB_34] SFTP for Bulk
[FB_35] REST for Bulk
[FB_36] Third Party (Client) Dynamic Registration
[FB_37] Query Parameters
[FB_38] On Demand Requests
[FB_39] PUSH model
Description
Min
Core Services
HTTPS support
Oauth
IntervalBlocks without full data sets (Ups,MR, …)
Third Party Access to Subscription/Authorization
Third Party Access to UsagePoints, MeterReading,
… and collections
GET PUT POST DELETE individual resources …
SFTP delivery of Bulk for Bulk request
Support the REST request for Bulk
Use Case 1





Without Notification
Notification followed by GET

Allows manual Authorization without OAuth but
produces Authorization resource XML
[FB_40] Offline Authorization
[FB_42] Third Party Core REST Services
[FB_43] Third Party Management REST Services
[FB_44] Manage ApplicationInformation Resource Allows PUT and DEL of ApplicationInformation
[FB_41] Manage Authorization Resource
Allows PUT and DEL of Authorization
Access Tokens and Their Data
access-token
Obtained: During retail customer authorization process
resourceUri  https://services.greenbuttondata.org/DataCustodian/espi/1_1/resource/Batch/Subscription/{SubscriptionID}
authorizationUri 
https://services.greenbuttondata.org/DataCustodian/espi/1_1//resource/Authorization/{authorizationUri}
client_access_token
Obtained: After ThirdParty registration
resourceUri  https://services.greenbuttondata.org/DataCustodian/espi/1_1/resource/Bulk/{bulkId}
authorizationUri  https://services.greenbuttondata.org/DataCustodian/espi/1_1/resource/Authorization/{authorizationId}
registration_access_token
Obtained: During ThirdParty registration (either dynamic or manually) Always results in an ApplicationInformation and
Authorization data structure.
resourceUri 
https://services.greenbuttondata.org/DataCustodian/espi/1_1/resource/ApplicationInformation/{applicationInformationId}
authorizationUri  https://services.greenbuttondata.org/DataCustodian/espi/1_1//resource/Authorization/{authorizationId}
Review of ApplicationInformation XML content and schema
APPLICATIONINFORMATION
Review of Authorization XML content and schema
AUTHORIZATION
Scope
Term
Expansion
Scope
FBTerms
FBTerm
[ FBTerms ], [ ValueTerms ], [ ResourceTerms ];
“FB=“, { [FBTerm], ”_”} , FBTerm, ScopeDelimiter ;
“4” | “5” | “6” | “7” | “8” | “9” | “10” | “11” | “12” | “15” | “16” | “17” | “18” | “19”
| “27” | “28” | “29” | “31” | “32” | “33” | “34” | “35” | “36” | “37” | “38” | “39” |
“40” | “41” | “43”
{ ( "IntervalDuration=", namedOrNumber,{“_”, namedOrNumber}),
| ( "BlockDuration=", namedOrNumber,{“_”, namedOrNumber}),
| ( "HistoryLength=", nonNegativeNumber),
| ( "SubscriptionFrequency=", nonNegativeNumber | namedFrequency),
ScopeDelimiter };
{ (“AccountCollection=”, nonNegativeNumber) | “BR=”, brID), ScopeDelimiter}
ValueTerms
ResourceTerms
ScopeDelimiter
namedFrequency
namedOrNumber
brID
nonNegativeNumber
Digit
Character
“;”
“billingPeriod” | “daily” | “monthly” | “seasonal” | “weekly” |
nonNegativeNumber | namedFrequency;
Character, {Character}*;
digit, { digit };
0 | "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" ;
Digit | “-” | "A" | "B" | "C" | "D" | "E" | "F" | "G"
| "H" | "I" | "J" | "K" | "L" | "M" | "N"
| "O" | "P" | "Q" | "R" | "S" | "T" | "U"
| "V" | "W" | "X" | "Y" | "Z" | "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i" | "j" |
"k" | "l" | "m" | "n" | "o" | "p" | "q" | "r" | "s" | "t" | "u" | "v" | "w" | "x" | "y" | "z" ;
Scope Term Descriptions
Where:
ResourceTerms
If a Bulk resource is specified via the “BR” term, the value of the {bulkID}
is provided after the equals sign (“=”). There could be one or more terms
in this list that express the granularity of notifications about resource
changes. If the Subscription has more than one UsagePoint, the
AccountCollection term can indicate the number of UsagePoints included
FBTerms
The function blocks supported
ValueTerms
These are parameterized terms
IntervalDuration
This is the minimum default length of an interval in seconds (e.g. 900 for
15 minutes, 3600 for one hour, …)
BlockDuration
This is the length of a block that contains the intervals (based on
enumeration of MacroPeriodKind in ESPI above as namedFrequency)
HistoryLength
This is the length of history buffer seconds
BulkAccountCollection Used where the DC wants to provide for the reporting of multiple
UsagePoints in a single Subscription. The number of UsagePoints is
represented by the value in the assignment statement – e.g. 4
UsagePoints would be BulkAccountCollection=4.
Examples for Connect My Data
• Example Scope Strings
– SCE
• Scope = “FB=1_3_4_5_8_13_18_19_31_34_35_39;
IntervalDuration=900_3600;BlockDuration=Daily; HistoryLength=
34128000;SubscriptionFrequency=Daily;
 AccountCollection=5;BR=1;”
– PG&E
• Scope =
“FB=1_3_4_5_7_8_13_14_15_18_19_31_32_34_35_37_38_39_40;
IntervalDuration=300_900_3600;BlockDuration=Daily_BillingPeriod_Weekl
y_Monthly; HistoryLength=63072000;SubscriptionFrequency=Daily;
 AccountCollection=5;BR=1;”
– SDG&E
• Scope = “FB=1_3_4_5_8_13_14_18_19_31_34_35_39_40;
IntervalDuration=300_900_3600;BlockDuration=Daily_BillingPeriod_Weekl
y_Monthly; HistoryLength=94608000;SubscriptionFrequency=Daily;
 AccountCollection=5;BR=1;”
Scope Negotiation
Oversimplified sequence diagram of Use Case #2
showing essence of scope negotiation
TP
DC
RC
Logon
Logon
HTTP Redirect with
Scope={scope1} {scope2} …
Authorization request
Scope={scope2}
Authorization response
Scope={scope2}
access-token
resourceUri
authorizationUri
referenceId
…
PUSH Model – Notify/Get



When data is ready, data custodian will notify third party with Multiple URLS (1…Many)
Third party will make call (GET) based on the URL(s) provided
Data custodian will respond with the data as part of response to GET call
Deferred Response when DC does not
have data ready
DC
Request:
HTTP GET
https://localhost/DataCustodian/espi/1_1/Subscription/1
Content-type: application/xml
TP
HTTP returns 202
Request:
HTTP POST
https://localhost/ThirdParty/espi/1_1/Notification
Content-type: application/xml
Response:
Request:
HTTP GET
https://localhost/DataCustodian/espi/1_1/Subscription/1
Content-type: application/xml
HTTP returns 200 and Data
• ThirdParty makes asynchronous request
• DataCustodian does not have the data ready – returns HTTP 202
• Some time later (when ready no guarantees or time limit) DataCustodian sends
Notification with original URL in BatchList
• DataCustodian provides the result
SFTP Data Transfer



When data is ready, data custodian will notify third party with Multiple URLS (1)
Third party will make call (sFTP) based on the URL(s) provided
Data custodian will respond with the data as part of response to sFTP get call
Notification
DC
TP
Request:
HTTP POST
https://localhost/ThirdParty/espi/1_1/Notification
Content-type: application/xml
Response:
<?xml version="1.0" encoding="UTF-8"?>
<BatchList xmlns="http://naesb.org/espi" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://naesb.org/espi espiDerived.xsd">
<resources>https://localhost:8080/DataCustodian/espi/1_1/resource/Batch/Subscription/1
</resources>
<resources>https://localhost:8080/DataCustodian/espi/1_1/resource/Authorization/1
</resources>
<resources>https://localhost:8080/DataCustodian/espi/1_1/resource/Batch/Bulk/1
</resources>
</BatchList>
ReadServiceStatus
DC
TP
Request:
HTTP GET
https://services.greenbuttondata.org:443/DataCustodian/espi/1_1/resource/ReadServiceStatus
Content-type: application/xml
Response:
<?xml version="1.0" encoding="UTF-8"?>
<ServiceStatus xmlns="http://naesb.org/espi"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://naesb.org/espi espiDerived.xsd">
<currentStatus>1</currentStatus>
</ServiceStatus>
Retrieve UsagePoints for a Subscription
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:espi="http://naesb.org/espi" xsi:schemaLocation="http://naesb.org/espi espiDerived.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<id>urn:uuid:1DA5F5DD-312B-4896-B910-098F59C73965</id>
<title>Green Button Subscription Feed</title>
<updated>1899-12-30T00:00:00Z</updated>
<link rel="self" href=" https://localhost/DataCustodian/espi/1_1/resource/Subscription/1/UsagePoint "/>
<entry>
<id>urn:uuid:C8C34B3A-D175-447B-BD00-176F60194DE0</id>
<link rel="self" href="https://services.greenbuttondata.org/DataCustodian/espi/1_1/resource/Subscription/1/UsagePoint/1"/>
<link rel="up" href="https://services.greenbuttondata.org/DataCustodian/espi/1_1/resource/Subscription/1/UsagePoint"/>
<link rel="related" href="https://services.greenbuttondata.org/DataCustodian/espi/1_1/resource/Subscription/1/UsagePoint/1/MeterReading"/>
<link rel="related" href="https://services.greenbuttondata.org/DataCustodian/espi/1_1/resource/Subscription/1/UsagePoint/1/ElectricPowerUsageSummary"/>
<link rel="related" href="https://services.greenbuttondata.org/DataCustodian/espi/1_1/resource/LocalTimeParameters/01"/>
<title>Green Button Sample Data File</title>
<content>
<UsagePoint xmlns="http://naesb.org/espi">
<ServiceCategory>
<kind>0</kind>
</ServiceCategory>
<ServiceDeliveryPoint>
<name>sample tariff showing block and tier pricing</name>
<tariffProfile>./TariffSample.xml</tariffProfile>
</ServiceDeliveryPoint>
</UsagePoint>
</content>
<published>2013-09-19T04:00:00Z</published>
<updated>2013-09-19T04:00:00Z</updated>
</entry>
<entry>
<id>urn:uuid:C8C34B3A-D175-447B-BD00-176F60194DE0</id>
<link rel="self" href="https://services.greenbuttondata.org/DataCustodian/espi/1_1/resource/Subscription/1/UsagePoint/2"/>
<link rel="up" href="https://services.greenbuttondata.org/DataCustodian/espi/1_1/resource/Subscription/1/UsagePoint"/>
<link rel="related" href="https://services.greenbuttondata.org/DataCustodian/espi/1_1/resource/Subscription/1/UsagePoint/2/MeterReading"/>
<link rel="related" href="https://services.greenbuttondata.org/DataCustodian/espi/1_1/resource/Subscription/1/UsagePoint/2/ElectricPowerUsageSummary"/>
<link rel="related" href="https://services.greenbuttondata.org/DataCustodian/espi/1_1/resource/LocalTimeParameters/01"/>
<title>Green Button Sample Data File</title>
<content>
<UsagePoint xmlns="http://naesb.org/espi">
<ServiceCategory>
<kind>0</kind>
</ServiceCategory>
<ServiceDeliveryPoint>
<name>sample tariff showing block and tier pricing</name>
<tariffProfile>./TariffSample.xml</tariffProfile>
</ServiceDeliveryPoint>
</UsagePoint>
</content>
<published>2013-09-19T04:00:00Z</published>
<updated>2013-09-19T04:00:00Z</updated>
</entry>
</feed>
DC
TP
HTTP GET
https://localhost/DataCustodian/espi/1_1/
resource/Subscription/{subscriptionID}/UsagePoint
Content-type: application/atom+xml
SFTP for Bulk Transfer
•
Pertinent to the SFTP discussion are the concepts that each Third Party has a defined relationship with
the Data Custodian.
–
–
–
•
For the purposes of Bulk transfer, this URI will be:
–
–
•
How to initiate the SSH connection?
What is the role if any of the client_credentials authorization to control access to SFTP enabled resources?
Discussion –
–
–
•
sftp://hostname:port/DataCustodian/espi/1_1/resource/Batch/Bulk/{bulkId}
where {bulkId} is a unique identifier assigned by the Data Custodian and the balance of the URI is presented in the
ApplicationInformation resource that both parties share (contains all relevant URIs and data for interchange via OAuth
etc…). The Third Party would then retrieve the bulk data by using an SFTP client with that URI. This is a straw man
concept for discussion on the call. Its advantage is that it in harmony with overall architecture of the Green Button
Connect My Data RESTful architecture and simply adds SFTP as a means of transfer when a large data set is to be
returned.
Used to Retrieve the data using SFTP protocols
–
–
•
For automated exchange of information about his relationship there is a special Authorization obtained in Use Case #1
(see the Authorization.docx -http://osgug.ucaiug.org/sgsystems/OpenADE/Shared%20Documents/Testing%20and%20Certification/GreenButtonTe
stPlan/referenceMaterial/GreenButtonAuthorization.docx).
We anticipate that when the Data Custodian has data available, it sends an asynchronous Notification to the Third
Party.
This Notification provides URIs of note that it is assumed the Third Party will want to retrieve.
After authorization of TP, they use Pene test, so what is benefit of access-token?
sftp user:pw, user=<tpname>, password=<tp client-credentials access-token>
Summary
–
–
sftp://hostname:port/DataCustodian/espi/1_1/resource/Batch/Bulk/{bulkId}
sftp user:pw, user=<tpname>, password=<tp client-credentials access-token>
Separation of PII containing Resource
RetailCustomer from Subscription*
PII Containing
information
Anononymous
EUI
Subscription
RetailCustomer
UsagePoint
ServiceLocation
Customer
Agreement
EndDeviceAsset
Normal ESPI
Resources
ServiceSupplier
PostionPoint
Key
TariffProfile
Account Resource
Authorization
*This data structure is to be developed on an aggressive schedule
based on HelpDesk issue #83 and PAP10 NAESB Std REQ.18. No
single API request can retrieve both PII and Anonymous data
Existing Resource
ERP Resource
Chunking – When the DataCustodian
needs to provide data in “chunks”
• Have huge data set that DC wants to provide in
“chunks” over HTTPS
• Use common URI – Subscription or Bulk / id
– Chunked by index and date?
– Use ESPI query parameters (FB_37) to distinguish
chunks
<?xml version="1.0" encoding="UTF-8"?>
<BatchList xmlns="http://naesb.org/espi" xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance
xsi:schemaLocation="http://naesb.org/espi espiDerived.xsd">
<resources>../espi/1_1/resource/Batch/Bulk/1?start-index=1&max-results=1000&published-max=2012-04-02T04:00:00Z
&published-min=2012-04-02T04:00:00Z</resources>
<resources>../espi/1_1/resource/Batch/Bulk/1?start-index=1&max-results=1000&published-max=2012-04-02T04:00:00Z
 &published-min=2012-04-02T04:00:00Z</resources>
</BatchList>
Parking Lot
•  Clarify accesses
– client_access_token= Authorizations/{},
Batch/Bulk/{bulkid}, ServiceStatus
– registration_access_token= ApplicationInformation
– access_token = used for individual
authorizations/subscriptions
•  Chunking for RESTfull GET of Bulk data
•  Replacement of
RetailCustomer/{retailCustomerId} with
Subscription/{subscriptionId}
•  Deferred access due to request that cant be
satisfied now but results in notification later. 202 http
response to GET
Action Items
• AI: Mark “OAuth” parameters as from OAuth in Authorization
• AI: Add ex: to each annotation of Authorization and
ApplicationInformation in the schema
• AI: remove “The field MUST be a valid URI.”
• AI: URI caps in Authorization.docx, schema,
ApplicationInformation, Authorization
• AI: thirdPartyScopeSelectionScreenURI 
thirdPartyLandingScreenURI in schema, Authorization.docx,
OpenESPI code
• AI: Dale will check on how to use access-token with SFTP
• AI: Replace RetailCustomer/{retailCustomerId} with
Subscription/{subscriptionId} in all APIs and related
documents