Erlang-style Error Recovery for Concurrent Objects with Cooperative

Erlang-style Error Recovery for Concurrent Objects with
Cooperative Scheduling
Einar Broch Johnsen2
Georg Göri 1
Rudolf Schlatte
2
Volker Stolz
2,3
University of Technology, Graz, Austria
[email protected]
University of Oslo, Norway
{einarj,rudi,stolz}@ifi.uio.no
Bergen University College, Norway
http://www.envisage-project.eu
1 / 21
Introduction
Presented Work
Rollback mechanism for communication through futures
Erlang-style error handling primitives for ABS
Motivation
Erlang execution of ABS models
Step towards distributed execution
Errors inherent, consider modeling
2 / 21
ABS - Abstract Behavioral Specification
Developed as part of the HATS project, continued in ENVISAGE
Concurrent object-oriented modeling language
Active Objects
Asynchronous Calls with Futures
Concurrent Objects Groups (COG) / Processes
Cooperative scheduling by explicit scheduling points
Backends: Java, Maude, Erlang
Proofs on class invariants over scheduling points, histories
3 / 21
Erlang
Erlang as Backend
lightweight shared nothing process
asynchronous message passing
built-in distribution
Error Handling
used in high availability systems
processes linking / monitors
error propagation
supervision
4 / 21
Small Example
MainBlock
Output
{
Output o=new Output i();
Fut<Bool> f=o!print(”Hello World”);
await f?;
Bool r = f.get;
}
class Output i implements Output {
String last=””;
Bool print(String s){
last=s;
return True;
}
}
5 / 21
Bank account with transaction log
Account
Methods
interface Account {
Unit deposit (Int amount);
Unit withdraw (Int amount);
}
Unit deposit (Int amount) {
transactions =
Cons(amount, transactions);
balance = balance + amount;
}
class Account implements Account
// log of transactions
List<Int> transactions = Nil;
// current balance
Int balance = 0;
Unit withdraw (Int amount) {
transactions =
Cons( amount, transactions);
if (balance < amount)
abort ”Insufficient funds”;
balance = balance amount;
}
6 / 21
Compare mechanisms in ABS and Erlang
Erlang
Processes, Messages
ABS
Active Objects, COGs,
AsyncCalls, Futures
Linking
Exit Messages
Error Propagation
Error Handling
7 / 21
Why model Errors?
Distribution errors
Connection / packet loss
External world errors
Through foreign function invocation e.g. I/O errors
Resource errors
Models closer to real implementations
Guarding against invalid input or malfunctioning components
8 / 21
Error Handling
Error propagation through futures
Rollback to preserve invariant
New Syntax
abort <Error>
<Future>.safeget
die <Error>
9 / 21
Semantics
abort <Error>
AsyncCall terminates call, error stored in future, rollback
ActiveObject object becomes invalid, terminate all AsyncCalls
MainBlock runtime terminates
<Future>.get
An error e in the <Future>, will lead to an abort e
<Future>.safeget
Return value or error ! enables error handling
die <Error>
Same as abort <Error> for active object.
Object dies, further calls fail.
10 / 21
Semantics
abort <Error>
AsyncCall terminates call, error stored in future, rollback
ActiveObject object becomes invalid, terminate all AsyncCalls
MainBlock runtime terminates
<Future>.get
An error e in the <Future>, will lead to an abort e
<Future>.safeget
Return value or error ! enables error handling
die <Error>
Same as abort <Error> for active object.
Object dies, further calls fail.
10 / 21
Semantics
abort <Error>
AsyncCall terminates call, error stored in future, rollback
ActiveObject object becomes invalid, terminate all AsyncCalls
MainBlock runtime terminates
<Future>.get
An error e in the <Future>, will lead to an abort e
<Future>.safeget
Return value or error ! enables error handling
die <Error>
Same as abort <Error> for active object.
Object dies, further calls fail.
10 / 21
Semantics
abort <Error>
AsyncCall terminates call, error stored in future, rollback
ActiveObject object becomes invalid, terminate all AsyncCalls
MainBlock runtime terminates
<Future>.get
An error e in the <Future>, will lead to an abort e
<Future>.safeget
Return value or error ! enables error handling
die <Error>
Same as abort <Error> for active object.
Object dies, further calls fail.
10 / 21
Example
Scenario
RequestHandler reads/writes from KeyValueStore
KeyValueStore value either in readCache or accessed via a Database
File (opened per invocation)
Error Handling
KeyValueStore no special handling (= propagation, rollbacks)
File
die in case of read or write Error
11 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Example with rollback
12 / 21
Towards Linking
Link
Linkable
class Link(Linkable f,Linkable s){
Int done=0;
Unit setup(){
f!waiton(this,s);
s!waiton(this,f);
await done==2;
}
Unit waitOn(Link l,Linkable la){
Fut<Unit> fut=la!wait();
l!done();
await fut?;
case fut.safeget {
Error(e) => die e;
}
}
}
Unit done(){
done=done+1;
}
Unit wait(){
await false;
}
13 / 21
Link Idea
Nonterminating AsyncCall both ways
Object1
Object2
wait
wait
14 / 21
Link Idea
Nonterminating AsyncCall both ways
Object1
Object2
wait
Object1
Object2
error
wait
14 / 21
Implementing a Supervision tree
Hierarchical process structure:
parent $ child
Restart strategies:
one-for-one:
one-for-all:
(or propagate upwards)
Images: OTP Design Principles User’s Guide 6.2, Ericsson AB
15 / 21
Supervisor in ABS
Starting a child
Handle a deceased child
Unit died(SupervisibleStarter ss,
Unit start(SupervisibleStarter child){
String error){
SupervisorLink sl= new SupervisorLink(this,child);
case strategy {
Link l=new Link(sl,this);
RestartAll => this.restart();
await l!setup();
RestartOne => this.start(ss);
this.links=Cons(sl,links);
Prop => die error;
sl.start();
}
}
}
16 / 21
Compare mechanisms in ABS and Erlang
Erlang
Processes, Messages
Linking
Exit Messages
Error Propagation
Error Handling
No direct support
ABS
Active Objects, COGs,
AsyncCalls, Futures
AsyncCalls: Built-in,
Objects: Link
Error stored in Future
abort, die
Through get
Through safeget
Rollback
17 / 21
But. . . how much does it cost?
Rollbacks require to keep old object-state around
Each asynchronous call duplicates callee state on demand á la
copy-on-write
Release points (suspend/await) commit state
No transactions (across release points)
. . . but maybe easy to implement now?
Distributed error detection more complicated. . .
18 / 21
But. . . how much does it cost?
Rollbacks require to keep old object-state around
Each asynchronous call duplicates callee state on demand á la
copy-on-write
Release points (suspend/await) commit state
No transactions (across release points)
. . . but maybe easy to implement now?
Distributed error detection more complicated. . .
18 / 21
But. . . how much does it cost?
Rollbacks require to keep old object-state around
Each asynchronous call duplicates callee state on demand á la
copy-on-write
Release points (suspend/await) commit state
No transactions (across release points)
. . . but maybe easy to implement now?
Distributed error detection more complicated. . .
18 / 21
Conclusion
Contributions
Error Handling
Error propagating futures
Rollback
language extensions in ABS
Also: backend in Erlang
magnitude faster
smaller code
distribution easier to implement
Further work
Tests in larger case study
Extend model simulation in Maude
Error handling analysis
Fault injection
19 / 21
Implementation & more Details:
http://abs-models.org/
Thank you
Also enjoy the following talk by Ivan:
“Fault Model Design Space for Cooperative
Concurrency”
design space of faults
once more, with exceptions
20 / 21
Further Reading
Johnsen, E. B., Lanese, I., and Zavattaro, G.
Fault in the future.
In Coordination Models and Languages, LNCS 6721, pages 1–15.
Springer, 2011.
Lanese, I., Lienhardt, M., Bravetti, M., Johnsen, E. B., Schlatte, R.,
Stolz, V., and Zavattaro, G.
Fault Model Design Space for Cooperative Concurrency,
In Proc. ISoLA’14, LNCS 8803, pages 23–37, Springer, 2014.
Chang Din, C., Dovland, J., Johnsen, E. B., and Owe, O.
Observable behavior of distributed systems: Component reasoning for
concurrent objects.
J. of Log. and Alg. Prog., 81:227–256, 2012.
Johnsen, E. B., Hähnle, R., Schäfer, J., Schlatte, R., and Ste↵en, M.
ABS: A core language for abstract behavioral specification.
In FMCO, LNCS 6957, pages 142–164. Springer, 2012.
21 / 21