the Annual Report
Transcription
the Annual Report
ORCHID Developing the Science of Human-Agent Collectives FOURTH ANNUAL REPORT www.orchid.ac.uk Table of Contents INTRODUCTION .........................................................1 RESEARCH ACHIEVEMENTS ...................................4 Flexible Autonomy .......................................................................... 5 Agile Teaming ................................................................................ 8 Incentive Engineering ................................................................... 10 Accountable Information Architecture .......................................... 13 HAC Applications ......................................................................... 16 KNOWLEDGE TRANSFER ACHIEVEMENTS .........19 Progress against Objectives......................................................... 20 Key Aims ...................................................................................... 22 APPENDICES ...........................................................24 A – Publications from 2014 .......................................................... 24 B – HAC Applications ................................................................... 29 C – Deliverables from 2014 .......................................................... 30 D – Statements of Work for 2014 ................................................. 31 E – Measures of Success............................................................. 32 F – Project Management Data ..................................................... 40 Introduction The computer has come a long way from its initial role as a scientific tool in the research lab. We live in a world where a host of computer systems, distributed throughout our physical and information environments, are increasingly implicated in our everyday actions. Computer technologies impact all aspects of our lives and our relationship with the digital has fundamentally altered as computers have moved out of the workplace and away from the desktop. Networked computers, tablets, phones and personal devices are now commonplace, as are an increasingly diverse set of digital devices built into the world around us. Data and information is generated at unprecedented speeds and volumes from an increasingly diverse range of sources and via ever more sensor types. It is then combined in unforeseen ways, limited only by human imagination. People’s activities and collaborations are becoming ever more dependent upon, and intertwined with, this ubiquitous information substrate. As these trends continue apace, it is becoming apparent that many endeavours involve the symbiotic interleaving of humans and computers. Moreover, the emergence of these close-knit partnerships is inducing profound change. The ability of computer systems to sense and respond to our on-going activities in the real-world is transforming our daily lives and shaping the emergence of a new digital society for the 21st century. More specifically, rather than issuing instructions to passive machines that wait until they are asked before doing anything, we are working in tandem with highly inter-connected computational components that act autonomously and intelligently (aka agents). This shift is needed to cope with the volume, variety, and pace of the information and services that are available. It is simply infeasible to expect individuals to be aware of the full range of potentially relevant possibilities and be able to pull them together manually. Computers need to more proactively guide users’ interactions based on their preferences and constraints. In so doing, greater attention needs to be given to the balance of control between people and machines. In many situations, humans are in charge and agents predominantly act in a supporting role, providing advice and suggesting options. In other cases, however, agents are in control and humans play the supporting role (e.g., automatic parking systems on cars and algorithmic trading on stock markets). Moreover, these relationships may change during the course of an activity (e.g. a human may be interrupted by a more pressing request and so take a less hands-on approach to the current task or an agent may encounter an unexpected situation and have to ask for human assistance for a task it was planning to complete autonomously). In ORCHID, we term this emerging class of systems human-agent collectives (HACs) to reflect the close partnership and the flexible social interactions between the humans and the agents. As well as exhibiting increased autonomy, such systems are inherently open and social. This openness means participants need to continually and flexibly establish and manage a range of social relationships. Thus, depending on the task at hand, different constellations of people, resources, and information need to come together, operate in a coordinated fashion, and then disband. The openness and presence of many distinct stakeholders, each with their own resources and objectives, means participation is motivated by a broad range of incentives ̶ extrinsic (e.g., money or tax-benefit), social or image motivation (e.g., public accreditation or leader-board position) or intrinsic (e.g., personal interest in a social cause) ̶ rather than diktat. 1 HAC systems exhibit a number of distinctive features that make them particularly challenging to engineer, control and predict. Their open nature means control and information is widely dispersed among a large number of people and agents with different aims and objectives. The various system elements exhibit a range of availabilities, some are persistent others are transient. The independent actors need to coordinate flexibly with people and agents that are themselves adapting their behaviour and actions to the prevailing circumstances to best achieve their goals. The real-world context means uncertainty, ambiguity and bias are endemic and so the agents need to handle information of varying quality, trustworthiness and provenance. Thus, techniques are required to provide an auditable information trail from the point of capture (a sensor or a human participant), through the fusion and decision processes, to the point of action and the agents have to reason about the trust and reputation of their collaborators to take the best course of action. Finally, in many cases, it is important that the collective action of the volitionally participating actors results in acceptable social outcomes (such as fairness, efficiency or stability). These features of HACs require us to: understand how to provide flexible autonomy that allows agents to sometimes take actions in a completely autonomous way without reference to humans, while at other times being guided by much closer human involvement. discover the means by which groups of agents and humans can exhibit agile teaming and come together on an ad hoc basis to achieve joint goals and then disband once the cooperative action has been successful. elaborate the principles of incentive engineering in which rewards are designed so that participants are encouraged to take actions that generate socially desirable outcomes. design an accountable information infrastructure that allows the veracity and accuracy of seamlessly-blended human and agent decisions, sensor data and crowd generated content to be confirmed and audited. The ORCHID team all share the ethos of undertaking fundamental research in real-world contexts (“in the wild”) and are working closely with industrial collaborators to demonstrate and evaluate this fundamental work in the application domains of energy systems, citizen science and disaster response. To this end, a number of prototypical HACs (see appendix B for more details) have been built these include: AtomicORCHID which illustrates agent-based task co-ordination in disaster response situations. AgentSwitch a fully-automated energy tariff advice system using real-time monitoring and prediction algorithms. It has been deployed both in homes directly and to support the Centre for Sustainable Energy in advising thousands of householders on their energy tariffs. TariffAgent, a prototype smart grid system tested in trials with 20 households, has been developed to explore issues of how users might live with differential tariffs. A smart phone app to help locate the New Forest cicada – this has been downloaded by over 5,000 members of the public. A gamified version has been used in conjunction 2 with electronic cicadas (which emit a sound similar to the real cicada) at a number of public engagement events including the British Science Festival. The MyJoulo home heating advice system was deployed (2,245 users to date) as part of a DECC study into the requirements of smart heating controls, and won the British Gas Connected Homes start-up competition. During 2014, trials were conducted with three UK energy retailers and in 2015 the associated spin out was acquired by Quby. An on-line game, Apocalypse of MOP, developed to explore understandings of provenance. To date it has attracted over 850 users resulting in over 100,000 logged provenance activities. The FigureEnergy system for better understanding and analysis of home energy data, used by over 30 homes in a trial in collaboration with Kingston University, London. The CrowdScanner system that was developed for, and won, the US State Department’s TAG challenge for social mobilisation and rapid information gathering. This involved five individuals roaming the streets of New York, Washington DC, Bratislava, Stockholm, and London on a single day and the challenge was to locate them based only on a single photo of each individual released at 8am. The Bayesian Classifier Combination algorithm for the Shared Task Challenge, developed in collaboration with a Microsoft Research/Bing team, which won the Shared Task Challenge in the Crowdsourcing at Scale workshop. An automated fantasy football manager that has outperformed 2.5M players in the online Fantasy Premier league. This has been developed into the RateMyTeam app which currently has 1,000 views per day. The OutrunCancer platform, launched to evaluate networked incentive schemes in collaboration with Cancer Research UK. This attracted over 1,500 participants. Veri.ly a web application that tasks volunteers with verifying reports from the ground or on-line during major disasters. A recent pilot over 2 days attracted 70 participants and hundreds of visitors. ORCHID technologies have also resulted in several software releases in 2014: Latent force software to support an article in the Journal of Machine Learning (http://jmlr.csail.mit.edu/papers/v15/) InferNet module, developed in conjunction with Microsoft Research, which automatically generates large scale inference code for Bayesian graphical models (http://blogs.msdn.com/b/infernet_team_blog/archive/2014/06/25/community-based-bayesian-classifiercombination.aspx). Prov Python library, an implementation of the Provenance Data Model by the World Wide Web Consortium (https://pypi.python.org/pypi/prov). ProvStore which is a public online repository for provenance documents with over 13,000 documents (https://provenance.ecs.soton.ac.uk/store/). Non-intrusive Load Monitoring Toolkit which is an open-source toolkit for evaluating energy disaggregation research (https://github.com/nilmtk/nilmtk). Literatin which is a plug-in that allows users to explore the complexity of any text found within a webpage (https://chrome.google.com/webstore/detail/literatin/igpbgncfceidjgcjddcdamjgbp dcmbnl?hl=en-GB). 3 Research Achievements The project team involves a number of leading University groups: the University of Nottingham (Tom Rodden), the University of Oxford (Steve Roberts and Michael Osborne), and the University of Southampton (Nick Jennings, Luc Moreau, Alex Rogers and Gopal Ramchurn). To date, it has directly employed 17 Research Fellows and 24 PhD students, and a further 6 researchers have been associated with the project. It has the following organisations as industrial collaborators: BAE Systems, Secure Meters Ltd, the Australian Centre for Field Robotics (ACFR) and Rescue Global. It also employs a Knowledge Transfer Officer, David Nicholson, who is jointly funded by EPSRC and BAE Systems, and a project administrator (Angela Westley). The project receives advice from an Independent Scientific Panel whose members are Brian Collins (University College London), Yolanda Gil (University Southern California), Thore Graepel (Microsoft Research), David Parkes (Harvard University) and Alan Winfield (University of West of England) and an Industrial Advisory Group whose members are Antony Waldock, Henry Tse and Robert Johnston (BAE), Kaushik Ghosh (Secure), Salah Sukkarieh (ACFR) and David Jones (Rescue Global). The research within ORCHID is split into five work areas (WAs), each of which is described in more detail in the remainder of this section. Each section describes the progress to date in that area and then outlines the high-level plans for the coming year. More details of the work carried out are given in the Appendices: C lists all the deliverables produced and D the statements of work that were active in the reporting year. This year the project has produced 62 publications in journals and conferences/workshops (see Appendix A for the full listing). To date, the project has produced 228 publications, 57 in journals/books and 171 in conferences and workshops. Of these, 34 are between more than one of the partners involved in the project, 9 involve an industrial co-author, 93 involve an international co-author and 8 were either shortlisted for or won a best paper award. The project web-site is a key resource and has attracted some 15,000 users and 51,000 page views since 2012. In addition to those listed in this section, a number of research achievements relate to knowledge transfer activities, including the award of 6 patents and participation in Angel Thunder (one of the largest search and rescue exercises in the world). These are detailed in the section on Knowledge Transfer Achievements. 4 Flexible Autonomy (WA1) Progress against Objectives This Work Area is led by Rodden and its initial objectives, as identified in the proposal, were: We will focus on the establishment of the key interactional principles between human and software agents that are needed to allow a sense of flexible autonomy to be established, monitored and amended in a context sensitive manner. These principles will be built into key mechanisms and reflected in the ways we represent and reason about these systems. Key objectives in establishing these principles are as follows. (i) The development of new presentation and interaction techniques to allow users to understand the actions of large collections of independent systems as they reason and act on behalf of users. (ii) The development of new control models and representations that allow dynamic and fluid interactive arrangements of agents and users. For example, allowing authority relationship between users and agents to change in different contexts and agents to work under varying levels of supervision. (iii) The development of techniques to recognise human activity to allow agents to reason about human interaction so they might exploit this understanding to augment and support the actions of users. (iv) An elaboration of the effectiveness of different styles of interaction and their role in HACs. For example, when should users give direct orders that agents must obey? When might users express their broad desire and allow these to be re-interpreted by agents? Under what circumstances should agents instruct users what to do and when should users be able to question them? The principal aim was to continue with the real world deployment and understanding of HACs and to supplement these with the development of more conceptual work on theories and frameworks. In more detail: (i) Our work on presentation and interaction techniques will focus on the following. First, the development of new interaction techniques focusing on the programme’s applications. More emphasis will be placed on the situational awareness and disaster response, which provide particular issues in terms of multimodal interaction and the visceral nature of the environment. Second, we will complement this exploration by a closer exploration of physiological interaction where we may exploit the monitoring of user physiological data as part of interaction. Third, the development and elaboration of more generic frameworks of the presentation and interaction techniques that generalise across our different investigations. (ii) Work on new models of control and representation will focus on the analysis and understanding of the data collected from the various deployments of ArgusUAV and AtomicORCHID and the emerging fieldwork from our studies of disaster response training. These studies will shape the articulation of a range of models and frameworks that may be used to inform the future design of these systems. (iii) We will continue to exploit a range of activity recognition techniques to explore the extent to which these can best be applied. This will be guided by the project’s key applications with a particular focus on disaster response. This will be complemented by a consideration of activity recognition based on human physiology gained from a range of experiences and the analysis of network traffic information captured from domestic networks and how these might be exploited alongside energy and other captured domestic information. (iv) Work on the assessment and evaluation of different styles of interaction will be undertaken as novel interactive techniques emerge during the year. The aim is to 5 consolidate this work and shape a set of techniques and approaches to promote lowcost rapid prototyping of HACs to allow the assessment of techniques before they are fully developed. This will build upon our focus group work and animated sketching and explore the development of approaches based on “Wizard of Oz” tools to mimic agent response and involvement. The progress against the aims is as follows: (i) We continued our work on the development of new presentation techniques with an increasing focus on articulating the underlying conceptual interactive frameworks of HACs. This included an exploration of the role of trust within agent - user interactions and the impacts of different forms of discourse between agents and users. This included Consolidation of the AtomicORCHID into a full HAC demo presented at the All Hands Meeting and published at AAMAS 2015. Engagement and fieldwork with Rescue Global to outline appropriate means of presenting HAC information in disaster response settings. Resulting in a full paper to appear at CHI 2015. A third workshop on Human Agent Interaction Design and Models was hosted at AAMAS and a special issue of the Journal of Autonomous Agents and MultiAgent Systems will be published in 2015. (ii) We explored issues of control and representation by consolidating our work on novel methods and techniques for HACs to consider how we might envision future HACs and allow users to comment on these through a combination of in-situ prototyping and games-based investigation. Highlights have included Studies of planning and orienting in AtomicORCHID resulting in a series of publication in both the HCI and agent-based communities, e.g. COOP204 and CTS2014. Continued studies of the use of language in the giving of instruction as part of an on-going PhD studentship. Design deployment and analysis of a provenance based game, which led to a publication at CHI2015. (iii) Our work on capturing and representing human activity continued to focus on the significant ethical challenges that need to be addressed within HAC systems. In particular, this focused on a reconsideration of consent and ethics for HAC systems. This built upon our existing explorations of consent within energy systems to consider how future autonomous systems might be understood legally and ethically, as well as interactionally. This has included The design and development of a series of Ideation cards for involving legal and ethical considerations in the design of intelligent and agent based systems which was published at CHI 2015. A lab based study of multi UAV control using Functional New-Infrared Spectroscopy (fNIRS) brain study technology. 6 (iv) We elaborated the effectiveness of different styles of interaction through the development of a HAC pattern language looking at the various forms of human agent arrangement that have been used to date and the construction of a repository to capture these arrangements. This included the design and use of a patterns card design used in the design of game experiences including agents. Key Aims for 2015 Next year will emphasize wide scale public release and engagement with real world settings: (i) We will continue our work on the development of new presentation techniques focusing in particular on the development and demonstrations of the use of augmented projection of information within disaster responses settings in partnership with Rescue Global. (ii) We will extend our work on control and representation to consider how we might envision future HACs in real world settings using insitu prototyping and deployment “in the wild” focusing particularly on control room settings within disaster response. This will be complemented by an in-depth study of the use of fNIRS brain scanning technology as a means of capturing human activity in controlling UAVs. (iii) Our work on capturing and representing human activity will continue to focus on the significant ethical challenges that need to be addressed within HAC systems. This will include extending and rolling out our design ideation approaches. (iv) We consolidate our work on the development of a HAC pattern language by exploring how these may be presented to designers through a series of design prototyping exercises. 7 Agile Teaming (WA2) Progress against Objectives This Work Area is led by Rogers and its initial objectives, as identified in the proposal, were: We will focus on the development of key mechanisms and formalisms to deliver the agile teaming described above and how these are made available to users. Key objectives in establishing these mechanisms and formalisms include the following. (i) The development of new metaphors and design guidelines that enable effective teams to be formed by interleaving humans and agents such that they can work collectively toward a particular task, while fully exploiting the specific advantages and constraints of both parties. (ii) The development of mechanisms and algorithms that allow humans and agents within a group to effectively coordinate their activities and actions to collectively maximise their utility. (iii) The integration and demonstration of these approaches within exemplar applications within the domains of energy systems and disaster response. The stated aims for 2014 were: (i) In terms of the development of new metaphors and design guidelines, the insights from AtomicORCHID, the Cicada Hunt trials, and engagement with Rescue Global will be incorporated into a set of HAC design patterns. The generation of these patterns will incorporate key insights from the user trials conducted to date. (ii) In terms of the development of mechanisms and algorithms that allow humans and agents within a group to effectively coordinate their activities work, Maleki will focus on the scaling of both Shapley value calculation and coalition structure generation, exploiting both approximation schemes that provide bounds, and features of the coalition characteristic function. Beck and Venanzi will continue work to develop algorithms that combine principled probabilistic representations of future UAV task arrival with decentralised coordination algorithms to efficiently allocate tasks between UAVs, while Baker will investigate algorithms for UAV path planning. Salisbury will also be studying how such UAVs can be coordinated with human controllers through crowdsourcing interfaces such as the ArgusUAV app. Wu will focus on investigating algorithms for coordination of emergency responders in uncertain environments. (iii) In terms of the integration and demonstration of these approaches, this will focus on further development of the AtomicORCHID platform, specifically integrating IBCC algorithms to extract tasks from social media text feeds, incorporating data feeds from UAVs that are autonomously coordinating on the basis of these feeds, and providing software agents to assist the first responders in their own coordination and the additional allocation of tasks to UAVs. Moreover, we will work on extending the TariffAgent App to a multi-user setting to investigate human-agent interaction in collective energy purchasing scenarios. The progress against the aims is as follows: (i) A publication formalising and presenting HAC design patterns, based upon the real deployments performed during the course of the ORCHID project has been submitted for publication. 8 (ii) Fundamental research on agile teaming has continued with Maleki developing and presenting a rich understanding of approximation algorithms for calculating the Shapley to fairly distribute the rewards of collective behaviour and Beck presenting robust algorithms for multi-UAV coordination. (iii) The AtomicORCHID platform was developed into a compelling disaster management demonstration which was presented at the 2014 ORCHID All Hands Meeting. Key Aims for 2015 (i) The agile teaming approaches demonstrated within AtomicORCHID will be further developed in collaboration with Rescue Global as part of an ‘interactive bird-table’ system, which will form part of Rescue Global’s ongoing disaster management operations. (ii) The coalition formation and agile teaming approaches developed and demonstrated in simulation in the context of coordinating cooling loads in coalitions of buildings, will be further developed within an Innovate UK funded project with KiwiPower (the UK’s leading aggregate demand response provider). 9 Incentive Engineering (WA3) Progress against Objectives This Work Area is led by Roberts and its initial objectives, as identified in the proposal, were: We will focus on the development of mechanisms and methods of approach that provide a means of influencing the behaviour of individual actors (humans and software agents) and groups, where there is no direct means of controlling the internal workings of the participants. In particular, the key initial objectives for this work are as follows. (i) Evaluate a principled value for information, action and strategy to inform an agent’s assignment of utility. (ii) Develop methods to incentivise humans and agents to gather and utilize information and act in uncertain, dynamic environments. (iii) Develop methods by which communities and social networks and coalitions of humans and agents can be analysed, tracked and forecasted. (iv) Determine how and what (weak or indirect) control needs be exerted over collectives of humans and agents to achieve global objectives by engaging in joint actions. The stated aims for 2014 were: (i) In terms of developing a principled value for information, action and strategy we will refine, develop and understand the relationships between intelligent task assignment, reinforcement learning (RL), inverse RL and preference learning. We aim to modify such algorithms to accommodate humans within the loop more naturally and thus, mitigate the need for costly incentives and reduce disincentives. We will experiment, for example, on crowd-sourcing information with a limit on how often human participants can be queried and develop approaches to infer a human's utility function from observing preferred actions instead of constant direct questioning. We will investigate the impact of these algorithms on the HACs ability to complete tasks successfully through engagement with empirical demonstrators. (ii) We will further understand the theoretical underpinning of how humans and agents can be incentivised to gather and utilize information. We will consider incentives for humans to operate within a HAC as not only direct incentives (money, knowledge, etc.), but also engineering software agents to encourage humans to engage with them. We will work to establish incentive mechanisms to encourage the human to remain truthful and fully engaged in these situations. We will also investigate the design of budgetlimited crowdsourcing workflows in order to improve the accuracy and efficiency of crowdsourcing where humans may not be truthful or expert at classification or content generation tasks. (iii) We will explore the intimate relationships and methods that exist between dynamic social networks and coalitions of agents and humans. Investigations into the use of group-level, rather than individual-level, incentives and weak control will be investigated, along with further integration of provenance and inferred trust. (vi) We will drive forwards the application domains as foundations for testing and comparing incentive mechanisms and weak control. This will achieved by evaluating the effect of 'nudges' and (ethical) weak control on human agents and more extensive empirical testing via experiments on HACs using our network of demonstrators in Citizen-science (Zooniverse, BioSound, New Forest Cicada) and crowd-generated information (CollabMap, AtomicORCHID). 10 The progress against the aims is as follows: (i) In terms of developing a principled value for information, action and strategy we have continued the development and exploitation of dynamic methodologies for tracking user policies and value of information in large, real-world citizen science and crowd-sourced environments. In particular we have: Further developed our system for intelligent tasking, using a joint utility function of information value (system performance) based upon the selection of tasks and users so as to maximise overall utility. Extensions of Independent Bayesian Classifier Combination (IBCC) have been validated this year for fusion of text sentiment and time-series data and further showcased on financial prediction problems, where they outperform the existing state-of-the-art (winning a best paper award at ICICA 2014 (Levenberg et al.) and leading to commercial internships). Pioneered decentralised approaches for evaluating information, incentives and optimal aggregations of knowledge from mixed human-sensor systems, culminating in joint work with Microsoft Research (Simpson et al., 2015). Developed DynIBCC, which can track the effect of incentives on accuracy, and used this to determine the types of tasks that affect an individual's reliability. This is incorporated into our intelligent tasking framework, to select tasks that bring the most information by balancing current information needs with incentives. (ii) For the work on how humans and agents can be incentivised to gather and utilize information we have developed our theory and practice around three major topics: Financial incentives have been explored in mixed software and crowd-sourced systems; for example, in the TREC challenge work, individual workers were paid a fee for their assessment of documents. These assessments were then used as 'silver labels' to enable an automated system to learn. This learning was then used not only to make automated decisions about unseen documents, but also to select new worker-object pairings, to hire new workers and fire workers whose information flow to the algorithm was very low. We have continued to use beneficial information as an incentive. For example, work on thermal modelling and forecasting (as part of AgentSwitch and MyJoulo) has enabled users to see the benefits of changing thermostat settings. Our Citizen-science applications have exploited knowledge as an incentive. Large user bases, such as Zooniverse, are maintained via knowledge feedback and the reassurance that the value of effort given is used to maximal effect – a core driver of our developments in Bayesian decision combination. The New Forest Cicada work developed user engagement incentives and minimised disincentives by incorporating best-practice design into the smartphone app and coordination of events. The new BioSound project takes the same approach and adds research on user-friendly path-planning to coordinate information gathering with minimal disincentives. 11 (iii) For the relationships and methods that exist between dynamic social networks and coalitions of agents and humans we have developed and validated social network methods, based on probabilistic community discovery to aggregate citizen scientists into communities based on the similarity of decision-making policies. Originally, this was applied to a large, dynamic Zooniverse data set that enabled us to track and quantify community evolution taking place as citizen-scientists are presented with new objects to classify. The methodology has continued to be leveraged to understand the importance, and value, of provenance: In work on provenance graphs from the CollabMap application, we have shown that perceived trust in information can be inferred from the graph (Huynh et al., 2014). From the AtomicORCHID provenance networks we extract meaningful summary information of what happened to help participants as they consider accepting/rejecting the Planning Agent's suggestions. Specifically, across several provenance graphs, we retrospectively ran a bespoke algorithm that was able to generate meaningful messages on an any-time basis, i.e. from whatever provenance data had been collected up to the point of execution. (iv) To drive forward the application domains as foundations for testing and comparing incentive mechanisms and weak control, we have: Analysed new data sets obtained from the New Forest cicada experiments. Undertaken field trials of energy management systems (TariffAgent) where users work in collaboration with an agent to choose the cheapest tariff. Investigation has also been undertaken into interaction design to help users minimise their cost. Furthered the integration of ORCHID methodologies into the Zooniverse environment and produced results indicating the approaches taken are transferable with ease between data sets. Developed a budget-limited crowdsourcing algorithm (Tran-Thanh et al., 2014) for complex crowdsourcing workflows. Developed incentive-based approaches, based on saliency and information voracity, to underpin the engagement with Rescue Global. Key Aims for 2015 (i) We will continue to leverage ORCHID methodologies in applications, from citizen science, disaster response, crowd sourced sensing and home energy management. (ii) We plan to detect the effect of different incentives through extensions of the DynIBCC model. We aim for the complete model to learn an individual's task preferences, how this affects their reliability, which will therefore allow us to trade-off the most informative tasks with those that keep a worker engaged. We will also investigate transfer learning within decentralised database systems that use IBCC to refine worker confusion matrices by exploiting data from heterogeneous task domains. 12 Accountable Information Infrastructure (WA4) Progress against Objectives This Work Area is led by Moreau and its initial objectives, as identified in the proposal, were: We will focus on the establishment of the key principles underpinning an accountable information infrastructure, and on the design of its architectural foundations. Whereas provenance tends to be detailed, providing an explicit account of how information was derived, trust and reputation tend to be measures of the quality of such information. Given this, several approaches have been proposed to derive the latter from the former. However, their varied nature in terms of representation and size imply different techniques to gather, manage, reason over and query them. In particular, ORCHID aims to integrate them in a coherent manner in the accountable information infrastructure. To this end, key objectives include the following. (i) The specification of the information infrastructure underpinning HACs, including common data models, representations and APIs to share and access accountable information uniformly. (ii) The definition of models for provenance/trust/reputation of information and their source in HACs. Models and algorithms will also be designed to support crowd generated content, uncertain information, incomplete and conflicting provenance, and anonymity and pseudonymity for privacy concerns. (iii) The design of accountability services, for online and offline use, based on a decentralised, large-scale architecture for capturing and reasoning over provenance/trust/reputation. These will exploit existing cloud infrastructures. (iv) The development of techniques to manage provenance/trust/reputation over variable timescales with an emphasis on the computational and ethical issues that may emerge from long lived information. The stated aims for 2014 were: (i) In terms of the specification of the information infrastructure underpinning HACs, the focus will be on the definition of APIs allowing accountability information to be accessed, stored, processed and transformed. The challenge here is to design interfaces that combine ease of use, interoperability, and scalability, but also integrate the various relevant algorithms that are being developed, in a seamless manner. (ii) In terms of models for provenance/trust/reputation, work will focus on generalizing the machine learning methods and demonstrating that one can learn interesting characteristics of applications from the provenance they generate. Furthermore, a deployment of CollabMap with online decisions about the quality of data produced and allocation of tasks to users will seek to demonstrate the potential of this approach. (iii) As far as the design of accountability services are concerned, the focus will be on the engineering of scalable implementations for the APIs designed in (i). We will continue to integrate them in ORCHID applications and, in particular, focus on provenance patterns, which identify points of potential interest in the execution of HACs. A promising direction of research is the relation between such provenance patterns and the HAC design patterns being envisaged in WA1 and WA2. (iv) In terms of the techniques to manage provenance/trust/reputation over variable timescales, the focus will be on Cr0n and the associated data and provenance that can be logged and analyzed. The study will focus on questions such as: can multiple interpretations of provenance exist?, how much provenance should be exposed to players?, how does provenance influence trust in information?, and how does provenance logging impact on user actions? A mix of data analysis, interviews, and lab experiments will be used in the course of this study. 13 The progress against the aims is as follows: (i) The work on specification of information infrastructure continued with the design and implementation of an API for registering interest in some provenance events and for sending notifications when such events occur. This was successfully integrated in the AHM 2014 ORCHID Demonstrator, which published notifications whenever plans had been invalidated by new incoming information. To support processing of provenance documents as mathematical graph artifacts, converters to matrix representation were also exposed and deployed. We have also added a new visualisation to present provenance information in a flow diagram, showing flows of influences captured in provenance information. (ii) The provenance network analytic approach previously used for classifying the quality of crowdsourced data in CollabMap has been formalised into a generic methodology for analysing provenance information based on provenance network metrics. The method was successfully demonstrated on two further applications: (1) identifying the owners of provenance graphs deposited on ProvStore from their topological characteristics, and (2) identifying instructions from chat messages in the AtomicORCHID trials from their provenance. (iii) We devised a graph analysis technique based on aggregating provenance types that is capable of generating provenance graph summaries to facilitate the understanding of complex provenance information. The summary captures the frequency of provenance elements and relations in the original graph, the visualisation of which was shown to reveal outliers hidden in the complexity of the original graph. In terms of the accountable information infrastructure, we re-designed the data schema underpinning the storage of provenance information. The new schema allows for queries of provenance information to be efficiently executed across multiple documents (which may be deposited by different users or applications). (iv) In collaboration with Urban Angel, an artistic production company, we designed and deployed “The Apocalypse of MoP”, an alternate reality game around provenance, to explore issues regarding usability and comprehensibility of provenance information. We have finished collecting data from the study, capturing interactions with provenance of varying degrees of complexity by over 1,000 participants. A qualitative study was carried out with the participants (41 questionnaire responses and 8 onehour interviews) to probe their understanding and perceptions about provenance in general, the PROV standard and its graphical illustration, and the specific ways they interacted with provenance information. 14 Key Aims for 2015 (i) We will carry out quantitative and qualitative analyses of the data of user interactions with provenance in the Provenance game to understand common graph layout patterns and how they relate to user understanding of provenance. Relevant data sets will be packaged and published for the benefits of the research community. (ii) We will continue our work on identifying common provenance patterns, which will help spotting points of potential interests in HAC executions. Our initial investigation showed that existing techniques did not scale well with the size of provenance graphs common in ORCHID applications. The focus for this work will be on developing pattern extraction techniques tailored for provenance graphs with the emphasis on scalability. (iii) The matrix representation for provenance graphs has allowed us to apply matrix analyses over provenance information. In particular, we will investigate the application of matrix decomposition techniques over the corpus of provenance graphs from ORCHID applications. A potential result is the identification of invariant component matrixes from provenance graphs generated by HAC applications. 15 HAC Applications (WA5) Progress against Objectives This Work Area is led by Ramchurn and its initial objectives, as identified in the proposal, were: The specific objectives are as follows. (i) Define vignettes that generate use-cases for HAC systems that reflect the level of maturity of the developed HAC technologies. Through outreach exercises (workshops, symposia, and other domain-specific forums) and our various advisory groups, the vignettes will be enriched and enhanced. This may include looking outside the initial exemplar domains of energy systems and disaster response as new opportunities present themselves. (ii) Develop a methodology for the construction of HAC systems that can be used by researchers and practitioners that combines all the constituent components into a coherent overarching framework. (iii) Design applications that both collect domain data from deployments and simulate challenging domain-specific problems with high fidelity. Initially, user applications with rapid up-time will be developed that permit the collection of sizeable datasets with regards to user behaviour and system performance and, in the long term, act as vehicles for evaluating ORCHID technologies ‘in the wild’. In addition, simulations will be designed to contain an extensive set of use-cases, coupled with a benchmarking framework. (iv) As technologies mature from other work areas, they will be evaluated in both real-world settings (through user trials) and simulation platforms, feeding back new datasets and requirements. (v) Construct demonstrators to showcase HACs to domain experts and to obtain feedback from the public, academics and policy makers that will help identify new requirements and highlight the associated ethical issues. The stated aims for 2014 were to focus on those applications already developed in the project, rather than develop new ones. In particular, we aim to evolve more complete versions of HAC applications that demonstrate the full range of HAC capabilities and allow us to start establishing the foundational principles of HACs. In more detail: (i) We will explore the notion of HAC design patterns through extensions to the AtomicORCHID platform. Specifically, we will extend the scenario to involve notions of agile teaming where multiple humans and multiple agents need to coordinate. We will also introduce incentive engineering models into the platform and evaluate them in situ. The platform will be provenance enabled and provenance will be used in the decision making process involved in the game. Through such extensions, we will demonstrate the full potential of HACs and further establish the design principles and interaction mechanisms for them. We aim to involve Rescue Global in our deployments in order to obtain real-world validation of the mechanisms we develop. (ii) We will deepen our research into collective energy management problems. In particular, we aim to extend AgentSwitch with collective tariff switching, through the TariffAgent probe, and energy management features that will allow us to evaluate coalition formation principles where humans and agents are involved. We also aim to research new design principles for autonomous energy systems through further deployments of our tools and platforms with energy advisors based at the Centre for Sustainable Energy and through field trials with end users. (iii) We are discussing MyJoulo trials for spring 2014 with EDF, British Gas and Npower. Each trial will potentially involve 500-1000 users, and we intend to deploy a new analysis website and new logger hardware. These deployments will generate new data that will allow us to inform agent-based models of homes and human behaviour and algorithms to learn these behaviours. 16 (iv) We will run the New Forest Cicada hunt again in conjunction with BugLife and the New Forest National Park Authority using a newly updated app and develop new mechanisms to attract more users within the Forest. This will involve attracting users with an intrinsic interest in the cicada, but we will also look into attracting users in serendipitous ways as they visit the Forest, thus deploying notions of incentive engineering in situ. (v) Apocalypse of MOP will be used to study issues around the generation, curation and manipulation of provenance data. Moreover, in AtomicORCHID we will develop new ways to present provenance data and use it to correct for inconsistencies in HAC action plans. The progress against the aims is as follows: (i) We developed AtomicORCHID version 3.0 to include novel flexible autonomy interfaces and incentive engineering mechanisms based on initial trials of the system. AtomicORCHID was thus trialled with over 30 participants including members from Rescue Global and results written in a number of research papers (Ramchurn et al., 2015, Fischer et al., 2014, Jiang et al., 2014). Building upon this work, we implemented HAC-DR, a disaster response system based on HACs. HAC-DR implements HACs to solve challenges related to gathering situational awareness using crowds, deployments of multiple UAVs to gather first-hand information, AtomicORCHID, and a provenance tracking and management system. HAC-DR was demonstrated at the AHM 2014 and led to a paper at AAMAS 2015. (ii) We conducted a long-term field trial of TariffAgent to investigate issues around acceptance/adoption of autonomy over long periods of time. The results are being written up for a journal paper. Furthermore, to investigate challenges related to multi-user environments, in collaboration with researchers from the University of Zurich, we implemented the SmartThermo platform that incorporates an agent that can adjust thermostat settings taking into account users’ preferences for comfort against cost. This system is to be deployed in January/February 2015. Work with the Centre for Sustainable Energy in collaboration with the CharIoT project, has led to the construction of low-cost temperature/humidity/presence sensors for deployment in low-income households and trials are about to begin in 2015. (iii) The MyJoulo home heating advice system has been deployed within a number of trials. In January 2013 it was trialled in 750 homes as part of the initial prototype deployment. In September 2013 the project won the British Gas Connected Homes Start-up Competition and was subsequently trailed in January 2014 in 1500 home in collaboration with three UK energy companies. In September 2014, Joulo Ltd. was spun-out from the University of Southampton with Prof. Alex Rogers and Dr. Reuben Wilcock as founders, and was subsequently acquired by Quby (http://quby.nl), a smart thermostat manufacturer based in the Netherlands, in January 2015. (iv) A smart phone app to help locate the New Forest cicada – this has been downloaded by over 5,000 members of the public, and generated over 5000 reports in the New Forest. A gamified version has been used in conjunction with electronic cicadas (which emit a sound similar to the real cicada) at a number of public engagement events in17 cluding the British Science Festival and a seminar at the EPSRC office. The work will be presented at the Houses of Parliament in the 2015 SET for Britain competition. (v) Apocalypse of MOP was deployed and studied with 850 users and the results written up for a research paper (Bachour et al., 2015). Key Aims for 2015 A number of new research challenges have been identified in our deployments of various prototypes as well as through ethnographic studies of user environments. While a number of these challenges are being taken up within new projects (e.g., CharIoT and ASUR) others will build upon the infrastructure already created by this work area (e.g., AtomicOrchid, Apocalypse of MOP, TariffAgent, Multi-UAV coordination systems) to undertake deeper studies. Hence, this work area will aim to consolidate our portfolio of applications that will form part of the legacy of ORCHID. In more detail: (i) We will engineer AtomicORCHID to be accessible to the wider research community. This will involve opening up the code base of the smart planner agent and providing support to run initial field-trials by external parties. (ii) We will capitalise on the work done in the HACDR system to develop new tools in collaboration with Rescue Global. In particular, we will look at instantiating the functionalities of the HAC-DR system in terms of an augmented bird table that aims to provide enhanced capability to disaster responders while being robust to harsh physical environments. We aim to trial prototypes at Angel Thunder 2015 and at the Intelligent Disaster Management workshop in Mexico. (iii) The SmartThermo system will be trialled and studies on multi-user energy systems will be carried out. (iv) A report on HAC design patterns will be produced to capture the key lessons learnt in developing HAC applications. (v) We will work with the Centre for Ecology and Hydrology (CEH) to deploy the species recording and recognition technology developed in the New Forest Cicada project, within a new field guide app for all UK Orthoptera. In addition, we will work with the Royal Botanic Gardens, Kew, as part of a Google Impact Award, to investigate the application of these techniques for the acoustic detection of mosquitos. 18 Knowledge Transfer Achievements Progress against Objectives The Knowledge Transfer activities, led by Nicholson, identified the following communities as the beneficiaries for the ORCHID work: (i) academics; (ii) industry and (iii) policy makers and the general public. The stated aims for each of these categories for 2014 were: Academic knowledge transfer (i) Nationally: ORCHID will continue to engage with EPSRC’s AISP and Digital Economy programmes, at an individual researchers’ level, but also more widely through joint participation in at least one workshop, conference or other relevant event, to explore overlap and opportunities for collaboration. ORCHID will aim to connect with relevant UK Centres for Doctoral Training (CDT), in particular those on Next Generation Computational Modelling, Future Autonomous and Robotic Systems, Data Science, My Life in Data and Energy Demand. (ii) Internationally: ORCHID staff and researchers will organise the 3rd Human-Agent Interaction and Design and Models (HAIDM) workshop at AAMAS 2014. ORCHID staff and researchers will organise the first workshop on Demand Response at ACM E-Energy. ORCHID will continue to team and participate in new international challenges. Industrial knowledge transfer (i) ORCHID will host its third Industry Day. (ii) ORCHID will continue to work to organize further short secondments and visits (both in and out). The aim is to have 3-4 of these completed or in progress by the end of the year. (iii) ORCHID will run an AtomicORCHID trial ‘in the wild’ for industry, government or NGO participants and/or observers, who will be given the opportunity to help shape future demonstration scenarios, application, and the route to exploitation. (iv) For ArgusUAV, the aim is to put this in the hands of end-users as early as possible to give focus to further technology development. (vi) On the back of the successful CDE project with BAE Systems, ORCHID will partner with industry, government, or NGOs, to win further external funding for exploiting HAC technology in real-world applications that could derive benefit. (vi) ORCHID will progress its commercialization plans for MyJoulo and AgentSwitch. It will extend the MyJoulo business to reach more large companies, so it can help them and their consumers. ORCHID will finalize a business plan for AgentSwitch and aim to secure funding to begin implementing the plan. 19 Policy makers and general public knowledge transfer (i) ORCHID will formalize its collaboration with Rescue Global by working together to define a set of relevant and realistic knowledge transfer targets and their implementation plan. The aim is to achieve this by: workshops, tabletop exercises, testing ORCHID’s HAC algorithms on real datasets obtained by Rescue Global on its operations; and running trials together. (ii) ORCHID will produce a number of articles targeted toward policy makers. These will have a HAC application focus and will be authored with domain experts (e.g. Rescue Global) to provide real-world examples of where HACs could benefit end-users and the implications for future policy decisions. (iii) ORCHID will support the British Science Festival by delivering a stand with a variety of educational materials and hands-on demonstrations based around HAC science. (iv) ORCHID will design, develop, and release classroom materials that will help teachers to educate schoolchildren about HAC science in a compelling way. (v) ORCHID will engage with the general public and invite them to play Apocalypse of MOP. The progress against the aims is as follows: Academic knowledge transfer (i) ORCHID continued to engage with other relevant EPSRC programmes, in particular the Autonomous and Intelligent Systems Programme (AISP), the University Defence Research Collaboration in Signal Processing, and two Centres for Doctoral Training (Oxford’s Autonomous, Intelligent Machines and Systems CDT and Southampton’s Next Generation Computational Modelling CDT). (ii) ORCHID participated in the organization of the International Workshop on Human-Agent Interaction Design and Models at AAMAS 2014. This was the third such event and ORCHID contributed papers, presentations, and panel discussion. (iii) ORCHID published a HAC science piece in the Communications of ACM to promote key HAC research challenges to a wide academic audience and encourage multidisciplinary effort. Industrial knowledge transfer (i) ORCHID held its third Industry Day in July 2014 at the British Computing Society London office. The event was attended by 25 people outside ORCHID, representing a variety of industrial organisations (BAE Systems, Secure Meters, QinetiQ, Selex, Envitia, Blue Bear, British Gas, IBM, and Toshiba). It also attracted attendees from national and local government (Dstl, DECC, and Avon Fire & Rescue Service) and charities (Rescue Global). 20 (ii) ORCHID placed four researchers on secondment with external organisations: one to British Gas Connecting Homes, two to Microsoft Research Lab, and one to BAE Systems. (iii) ORCHID trialled its AtomicORCHID technology demonstrator with Rescue Global Pathfinders, who provided feedback based on their real-world disaster management experience and operational needs. This feedback provided valuable steer that helped to improve AtomicORCHID’s use-cases and application. (iv) ORCHID matured the test and evaluation of its ArgusUAV technology demonstrator through external funding from MOD’s Autonomous Systems Underpinning Research (ASUR) programme. This resulted in a quantified evidence of the benefit of flexibility autonomy in missions involving the assignment of multiple UAVs to multiple tasks. (v) ORCHID research was used to leverage external funding from the following sources: MOD (for the ASUR HAC project); EPSRC (for the CharIoT project, partnering the universities of Nottingham and Southampton with the Centre for Sustainable Energy); Google (for an Impact prize awarded to the University of Oxford to exploit HAC research for mosquito detection); Innovate UK (for the Adaptive Demand Response project, partnering the University of Southampton with KiwiPower). (vi) ORCHID’s spin-out company Joulo (formed in 2013) has continued to grow, resulting in Joulo monitors placed in 2500+ UK homes to provide home heating advice and ongoing interest from the UK’s big energy suppliers. In early 2015 it was acquired by Quby. (vii) ORCHID has begun to engage with Dstl on how relevant ORCHID technologies could be transferred and demonstrated within Dstl’s C4ISR Concepts and Solutions (CCS) programmes. A workshop was held with 12 Dstl participants and this has led to follow-up discussion around specific ORCHID technologies and military applications. (viii) ORCHID ran an Industry Master Class on Non-Intrusive Load Monitoring at Imperial College. The Class was attended by representatives from AlertMe, British Gas, Green Running, Greeniant, Navetas, ONZO and Wattgo, as well as academics from Imperial College London and the University of Southampton. (ix) ORCHID provided a consultancy service to Toyota Central R&D Labs to apply incentive mechanisms for electric vehicle charging to smart grid settings with photovoltaic generation (x) ORCHID developed a crowdsourcing module for Microsoft Research Labs and this is being tested on an experimental platform used by Bing researchers prior to further development and application Policy makers and general public knowledge transfer (i) ORCHID continued to generate popular press in 2015, with new articles appearing in: The Guardian, The Daily Telegraph, The Economist, WIRED, Financial Times, ITV News and Sky News. 21 (ii) ORCHID researchers attended Angel Thunder 2014 at the invitation of Rescue Global. This is a large-scale disaster management exercise conducted in the US and sponsored by various US government agencies. ORCHID carried out an ethnographic study. (iii) ORCHID researchers manned a stand at the British Science Festival 2014, engaging hundreds of schoolchildren and members of the public in a compelling game-based demonstration of ORCHID technology and its impact in the disaster response domain. (iv) ORCHID participated in the First Lego League event in Hampshire, engaging students in the 9-16 age range with programming Lego Mindstorm robots to perform tasks that replicate those arising in disaster settings. (v) ORCHID researchers have partnered with the Fantasy Football First blogging site to launch the RateMyTeam tool to football fans around the world. The tool uses HAC algorithms and concepts to help users pick their fantasy football teams. (vi) ORCHID delivered algorithms and software for Independent Bayesian Classifier Combination into Zooniverse, a citizen science platform (vii) ORCHID contributed algorithms and software to Microsoft’s Infer.NET community-based crowdsourcing application (viii) ORCHID’s Apocalypse of MoP (Provenance Game) attracted over 850 sign-ups and 300 active players from the general public, creating complex provenance graphs to test and evaluate ORCHID’s provenance research. Key Aims for 2015 Academic knowledge transfer (i) ORCHID will co-organise the Intelligent Disaster Management Workshop in Mexico in February 2015. Its goal is to serve as a springboard for long-term collaboration between Mexican and UK universities as well as companies interested in disaster response. (ii) ORCHID will co-organise a HAC-related workshop at AAMAS 2015 and contribute papers, presentations, and research leads for a panel discussion. (iii) ORCHID will host a technical conference in London (September 2014) to share its main results, demonstrations and finding with the UK and European academic community. (iv) ORCHID will pull together materials for a new book on HAC science and applications and seek an academic publisher with a view to final publication in 2016. 22 Industrial knowledge transfer (i) ORCHID will continue engagement with its primary industry partners: BAE Systems and Secure Meters. With BAE Systems, ORCHID will exploit its disaster management and citizen science research through BAE Systems lead on the ASUR programme and participation in the MOD’s CCS programme. With Secure Meters, ORCHID will seek to transfer its energy application research into Secure through provision of software and technology consultancy. (ii) ORCHID will continue engagement with Dstl with a goal to supporting the field trials and demonstration days that Dstl has scheduled for its CCS programme in Q1 and Q3 in 2015. (iii) ORCHID will continue engagement with Microsoft Research Lab and Zooniverse with a view to the maturation, test, evaluation and release of further crowdsourcing technology. (iv) ORCHID will seek follow-on funding through the MOD’s ASUR programme to exploit its research on UAV flexible autonomy in military applications. (v) ORCHID will continue to promote industrial placements and will target 3 of these in 2015. Policy makers and general public knowledge transfer (i) ORCHID will work closely with Rescue Global on a number of activities in 2015: transfer of software applications for a Digital Bird Table in support of Rescue Global operations; participation in Angel Thunder 2015; and delivery of a thought piece (publication or presentation) to inform policy makers about the impact and role of AI applications in future disaster response. (ii) ORCHID will pull together its key energy research findings into a white paper and/or presentation materials that will be used when and where appropriate to inform and potentially influence policy decisions regarding energy and the environment. (iii) ORCHID Artists in Residence, Steve Beard and Victoria Halford, will present their work at an exhibition to be hosted by The Hansard Gallery in Southampton 14 July to 5 September 2015. (iv) ORCHID will hold a showcase event at the Royal Academy of Engineering in September 2015, at which the popular press will be briefed on ORCHID’s key findings and their impact. 23 APPENDIX A — PUBLICATIONS IN 2014 The project produced the following publications this year; the full list is available on the project website. The following markings are used to denote publications that: *: are between more than one of the groups involved in the ORCHID project +: involve an international co-author #: involve an industrial co-author ++: were shortlisted or won a best paper award * Alan, A., Costanza, E., Fischer, J. E., Ramchurn, S. D., Rodden, T., & Jennings, N. R. (2014). A field study of human-agent interaction for electricity tariff switching. In Proceedings of AAMAS 14: 13th Int. Conf. on Autonomous Agents and Multi-Agent Systems, 965-972. * Bachour, K., Wetzel, R., Flintham, M., Huynh, D. T., Rodden, T., & Moreau, L. (2015). Provenance for the People: A User-Centered Look at the W3C PROV Standard through an Online Game. In Proceedings of ACM CHI Conference on Human Factors in Computing Systems 2015. In press. + Batra, N., Kelly, J., Parson, O., Dutta, H., Knottenbelt, W., Rogers, A., Singh, A. and Srivastava, M. (2014) NILMTK: An Open Source Toolkit for Non-intrusive Load Monitoring. In e-Energy '14 Proceedings of the 5th international conference on Future energy systems, 265-276. + ++Kelly, J., Batra, N., Parson, O., Dutta, H., Knottenbelt, W., Rogers, A., Singh, A. and Srivastava, M. (2014) NILMTK v0.2: a non-intrusive load monitoring toolkit for large scale data sets: demo abstract. In BuildSys '14 Proceedings of the 1st ACM Conference on Embedded Systems for Energy-Efficient Buildings, 182-183 + Biermann, F., Naroditskiy, V., Polukarov, M., Nguyen, T., Rogers, A., & Jennings, N. R. (2014). Task Assignment with Controlled and Autonomous Agents. Mathematical Social Sciences, 71, 116-121. + Bistaffa, F., Farinelli, A., & Ramchurn, S. D. (2015) Sharing rides with friends: a coalition formation algorithm for ridesharing. In Proceedings of AAAI Conference on Artificial Intelligence 2015.In Press. + Bistaffa, F., Farinelli, A., Cerquides, J., Rodriguez-Aguilar, J. A., & Ramchurn, S. D. (2014). Anytime Coalition Structure Generation on Synergy Graphs. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, 13-20 * Costanza, E., Fischer, J. E., Colley, J., Rodden, T., Ramchurn, S. D., & Jennings, N. R. (2014). Doing the Laundry with Agents: a Field Trial of a Future Smart Energy System in the Home. In Proceedings of ACM CHI Conference on Human Factors in Computing Systems 2014, 813-822. + de Clippel, G., Naroditskiy, V., Polukarov, M., Greenwald, A., & Jennings, N. R. (2014). Destroy to Save. Games and Economic Behavior, 1-13. 24 + Farinelli, A., Rogers, A. and Jennings, N.R. (2014) Agent-based decentralised coordination for sensor networks using the max-sum algorithm. Journal of Autonomous Agents and Multi-Agent Systems, 28, (3), 337-380. * Fischer, J. E., Reeves, S., Rodden, T., Reece, S., Ramchurn, S. D., & Jones, D. (2015). Building a Bird's Eye View: Collaborative Work. In Proceedings of ACM CHI Conference on Human Factors in Computing Systems 2015. In press. * Fischer, J. E., Costanza, E., Ramchurn, S. D., Colley, J., & Rodden, T. (2014). Energy Advisors at Work: Charity Work Practices to Support People in Fuel Poverty. In Proceedings of ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 14, 447-458. * Fischer, J. E., Jiang, W., Kerne, A., Greenhalgh, C., Ramchurn, S. D., & Reece, S., et al. (2014). Supporting Team Coordination on the Ground: Requirements from a Mixed Reality Game. In Proceedings of 11th Int. Conference on the Design of Cooperative Systems (COOP 14), 49-67. Han, T., Tran-Thanh, L., & Jennings, N. R. (2014). The cost of interference in evolving systems. In Proceedings of COIN 2014: The 17th International Workshop on Coordination, Organisations, Institutions and Norms. In press. Huynh, D. T., & Moreau, L. (2014). ProvStore: a public provenance repository. In Proceedings of the 5th International Provenance & Annotation Workshop-IPAW 2014. In press. * Huynh, D. T., Ebden, M., Ramchurn, S. D., Roberts, S. J., & Moreau, L. (2014). Data quality assessment from provenance graphs. In Proceedings of the 5th International Provenance & Annotation Workshop-IPAW 2014. In press. + Inoue, S., Moran, S., & Nakata, K. (2014). Supporting Tool for Cooperative Work Analysis Based on Distributed Cognition. In Proceedings of the 12th International Conference on Probabalistic Safety Assessment and Management (PSAM). In press. Jager, N., Moran, S., & Schnadelbach, H. (2014). Using Adaptive Architecture to Support Yoga Practices: Social Considerations for Design. In Proceedings of Third IEEE Workshop on the Social Implications of Pervasive Computing for Sustainable Living, 364 - 369. * Jennings, N. R., Moreau, L., Nicholson, D., Ramchurn, S. D., Roberts, S. J., Rodden, T., & Rogers, A. (2014). On human-agent collectives. Communications of the ACM, 57, (12), 33-42. * Jiang, W., Fischer, J. E., Greenhalgh, C., Ramchurn, S. D., Wu, F., Jennings, N. R., & Rodden, T. (2014). Social Implications of Agent-based Planning Support for Human Teams. In Proceedings of 2014 Int. Conference on Collaboration Technologies and Systems, 310-317. Keshavarz, A. S., Huynh, D. T., & Moreau, L. (2014). Provenance for online decision making. . In Proceedings of the 5th International Provenance & Annotation Workshop-IPAW 2014. In press. # ++ Levenberg, A., Pulman, S., Moilanen,K., Simpson,E. & Roberts, S. (2014). Predicting Economic Indicators from Web Text Using Sentiment Composition. In Proceedings of ICICA2014. In press. 25 # A Levenberg, S Pulman, K Moilanen, E Simpson, S Roberts, Predicting Economic Indicators from Web Text Using Sentiment Composition, In International Journal of Computer and Communication Engineering 3 (2), 109-115. Maior, H. A., Pike, M., Wilson, M., & Sharples, S. (2014). Continuous detection of workload overload: An fNIRS approach. In Proceedings of the international conference on Ergonomics & Human Factors 2014, 450–457 . Maior, H. A., Pike, M., Wilson, M., & Sharples, S. (2014). Evaluating Mental Workload with fNIRS. The Ergonomist. In press. Moran, S. (2014). Social Networks and Ubiquitous monitoring: An Application of the PSA-BI Model. In Proceedings of Third IEEE Workshop on the Social Implications of Pervasive Computing for Sustainable Living, 370 - 375. + Moran, S., Bachour, K., & Nishida, T. (2014). User Perceptions of Anthropomorphic Robots as Monitoring Devices. AI & Society: Knowledge, Culture and Communication, 1-21. Moran, S., Luger, E., & Rodden, T. (2014). An Emerging Tool Kit for Attaining Informed Consent in UbiComp. In Proceedings of 1st International Workshop on How do you Solve a Problem like Consent, 635-639. Moran, S., Luger, E., & Rodden, T. (2014). Exploring Patterns as a Framework for Embedding Consent Mechanisms in Human-Agent Collectives. In Proceedings of 8th International Workshop on Human Aspects of Ambient Intelligence, 475-486. Moran, S., Luger, E., & Rodden, T. (2014). Literatin: Beyond Awareness of Readability in Terms and Conditions. In 1st International Workshop on How do you Solve a Problem like Consent, 641646. * Moran, S., Pantidi, N., Rodden, T., Chamberlain, A., Griffiths, C., & Zilli, D., et al. (2014). Listening to the Forest and its Curators: Lessons Learnt from a Bioacoustic Smartphone Application Deployment. In Proceedings of SIGCHI Conference on Human Factors in Computing Systems, 2387-2396. Moreau, L., Huynh, D. T., & Michaelides, D. (2014). An Online Validator for Provenance: Algorithmic Design, Testing, and API. In Proceedings of 7th International Conference on Fundamental Approaches to Software Engineering (FASE'14), 291-305. + Naroditskiy, V., Jennings, N. R., Van Hentenryck, P. and Cebrian, M. (2015) Crowdsourcing Contest Dilemma. Journal of the Royal Society Interface, 11, (99), 20140532. Naroditskiy, V., Stein, S., Tran-Thanh, L., Vlassopoulos, M., & Jennings, N. R. (2014). Referral incentives in crowdfunding. In Proceedings of HCOMP2014: Conference on Human Computation & Crowdsourcing, 171-183 . + Obraztsova, S. , Markakis, E., Polukarov, M. , Rabinovich, Z. and Jennings, N. R. (2015) On the convergence of iterative voting: how restrictive should restricted dynamics be? In Proceedings of AAAI 2015: Twenty-Ninth AAAI Conference on Artificial Intelligence. In press. 26 Panagopoulos, A. A., Alam, M., Rogers, A., & Jennings, N. R. (2015). ADA-HEAT: A general adaptive intelligent agent for domestic heating control. In Proceedings of AAMAS-15 : 14th Int. Conf. on Autonomous Agents and Multi-Agent Systems. In press. + Panagopoulos, A., Chalkiadakis, G and Jennings, N.R. (2015) Towards optimal solar tracking: a dynamic programming approach. In Proceedings of AAAI-2015: 29th AAAI Conference on Artificial Intelligence. In press. * Pantidi, N., Moran, S., Bachour, K., Rodden, T., Zilli, D., Merrett, G. V., & Rogers, A. (2014). Field Testing a Rare Species Bioacoustic Smartphone Application: Challenges and Considerations. In Proceedings of Third IEEE Workshop on the Social Implications of Pervasive Computing for Sustainable Living, 376-381. Parson, O., Ghosh, S., Weal, M., & Rogers, A. (2014). An Unsupervised Training Method for Nonintrusive Appliance Load Monitoring. Artificial Intelligence, 217, 1-19. + Pawlowski, K., Kurach, K., Svensson, K., Ramchurn, S. D., Michalak, T., & Rahwan, T. (2014). Coalition Structure Generation with the Graphics Processing Unit. In Proceedings of AAMAS 14: 13th Int. Conf. on Autonomous Agents and Multi-Agent Systems, 293-300. Pike, M., Maior, H. A., Porcheron, M., Sharples, S., & Wilson, M. (2014). Measuring the effect of Think Aloud Protocols on Workload using fNIRS. In Proceedings of ACM SIGCHI Conference on Human Factors in Computer Systems (CHI 2014), 3807-3816. * Ramchurn, S. D., Simpson, E., Fischer, J. E., Huynh, D. T., Ikuno, Y., & Reece, S., et al. (2015). HAC-ER: A disaster response system based on human-agent collectives. In Proceedings of AAMAS-15 : 14th Int. Conf. on Autonomous Agents and Multi-Agent Systems. In press. * Ramchurn, S. D., Wu, F., Fischer, J. E., Reece, S., Jiang, W., & Roberts, S. J., et al. (2015). Human-agent collaboration for disaster response. Journal of Autonomous Agents and Multi-Agent Systems. In press. * Reece, S., Roberts, S. J., Ghosh, S., Rogers, A., & Jennings, N. R. (2014). Efficient state-space inference of periodic latent force models. Journal of Machine Learning Research, 1-66. * # Simpson, E., Venanzi, M., Reece, S., Kohli, P., Guiver, J., Roberts, S. and Jennings, N. R. (2015) Language Understanding in the Wild: Combining Crowdsourcing and Machine Learning. In Proceedings of 24th International World Wide Web Conference (WWW 2015). In Press. + Ströhle, P., Gerding, E. H., de Weerdt, M., Stein, S., & Robu, V. (2014). Online Mechanism Design for Scheduling Non-Preemptive Jobs under Uncertain Supply and Demand. In Proceedings of AAMAS 14: 13th Int. Conf. on Autonomous Agents and Multi-Agent Systems, 437-444. Teacy, W. L., Julier, S., Nardi, D. R., Rogers, A., & Jennings, N. R. (2015). Observation modelling for vision-based target search by unmanned aerial vehicles. In Proceedings of AAMAS-15 : 14th Int. Conf. on Autonomous Agents and Multi-Agent Systems. In press. 27 + Tran-Thanh, L., Huynh, T. D., Rosenfeld, A., Ramchurn, S.D. and Jennings, N. R. (2015) Crowdsourcing Complex Workflows under Budget Constraints. In Proceedings of Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI-15). In Press. + Tran-Thanh, L., Huynh, D. T., Rosenfeld, A., Ramchurn, S. D., & Jennings, N. R. (2014). BudgetFix: Budget limited crowdsourcing for interdependent task allocation with quality guarantees. In Proceedings of AAMAS 14: 13th Int. Conf. on Autonomous Agents and Multi-Agent Systems, 477-484. Tran-Thanh, L., Stein, S., Rogers, A., & Jennings, N. R. (2014). Efficient crowdsourcing of unknown experts using bounded multi-armed bandits. Artificial Intelligence, 214, 89-111. + Vallverdu, J., Casacuberta, D., Nishida, T., Ohmoto, Y., Moran, S., & Lazare, S. (2014). From Computational Emotional Models to HRI. International Journal of Robotics Applications and Technologies, 1, (2), 11-25. # ++ Venanzi, M., Guiver, J., Kazai, G., Kohli, P., & Shokouhi, M. (2014). Community-Based Bayesian Aggregation Models for Crowdsourcing. In Proceedings of 23rd International World Wide Web Conference (WWW 14), 155-164. Vinyals, M., Robu, V., Rogers, A., & Jennings, N. R. (2014). Prediction-of-use games: a cooperative game theory approach to sustainable energy tariffs. In Proceedings of AAMAS 14: 13th Int. Conf. on Autonomous Agents and Multi-Agent Systems, 829-836. Wetzel, R. (2014). Introducing Pattern Cards for Mixed Reality Game Design. In Proceedings of Third Workshop on Design Patterns in Games. Foundations in Digital Games Conference 2014. In press. Wu, F., & Jennings, N. R. (2014). Regret-based multi-agent coordination with uncertain task rewards. In Proceedings of AAAI-14: Twenty-Eighth Conference on Artificial Intelligence (AAAI), 1492-1499. Ye, C., & Wilson, M. (2014). A user defined taxonomy of factors that divide online information retrieval sessions. In Proceedings of IIiX '14 the 5th Information Interaction in Context Symposium, 48-57. Ye, C., & Wilson, M. (2014). The characteristics of casual sessions in search behaviour logs. In Proceedings of Search4Fun workshop in conference IIiX2014, 6-6. Zenonos, A., Stein, S., & Jennings, N. R. (2015). Coordinating measurements for air pollution monitoring in participatory sensing settings. In Proceedings of AAMAS-15 : 14th Int. Conf. on Autonomous Agents and Multi-Agent Systems. In press. Zilli, D., Parson, O., Merrett, G. V. and Rogers, A (2014) A Hidden Markov Model-Based Acoustic Cicada Detector for Crowdsourced Smartphone Biodiversity Monitoring. Journal of Artificial Intelligence Research, 51, 805-827 28 APPENDIX B — HAC APPLICATIONS Name Description Provenance game run by nottingham to investigate people's Apocalypse of MOP manipulation of provenance data and understandings of it. Atomic Orchid Agent B AgentSwitch ArgusUAV BioSound Cicada App CollabMap Crowdscanner Fantasy Football Figure Energy GalaxyZoo Hire and Fire MyJoulo Publication http://apocalypseofmop.com/ Contact Dong Huynh, Khaled Baschour, Richard Wetzel Mixed-reality game to investigate coordination in disaster response. Fischer, J. E., et al. (2012). Serious Mixed Reality Games. In Joel Fischer, The scenario involves having a planner agent guiding rescuers to ACM Converence on Computer-Supported Cooperative Wenchao Jiang, complete tasks in teams. Work Gopal Ramchurn A HAC application to help evaluate interfaces for demand-response in Costanza et al. (2014) 'Doing the Laundry with Agents' - CHI Enrico Costanza, the home. An agent advises users as to when is cheapest to schedule 2014 Joel Fischer their washing machine loads and notifies them if prices change. Personalized energy tariff-recommender system: Ramchurn, Sarvapali, et al. (2013) AgentSwitch: towards Gopal Ramchurn www.agentswitch.org. Incorporates a prediction algorithm for yearly smart electricity tariff selection. In, 12th International energy consumption, disaggregation of energy feeds, and collective Conference on Autonomous Agents and Multiagent energy purchasing algorithms. Systems (AAMAS 2013), Saint Paul, US. 8pp. Interfaces for UAV/UGV control by an emergency responder. http://player.vimeo.com/video/34837704 Feng Wu, Gopal Ramchurn Audio classification of birdsong implemented on mobile devices with http://www.robots.ox.ac.uk/~parg/biosound/doku.php Steve Roberts, user input. Timos Papadopoulos Android and iPhone apps to detect an insect in the New Forest. More Zilli, D. et al. (2013). A Hidden Markov Model-Based Alex Rogers, details at: http://newforestcicada.info Acoustic Cicada Detector for Crowdsourced Smartphone Davide Zilli Biodiversity IJCAI A crowdsourcing platform to help create high resolution evacuation Ramchurn, etMonitoring. al. (2013). Collabmap: crowdsourcing maps for Gopal Ramchurn, maps. Can be used to create maps using help from voluteers through emergency planning.. In ACM Web Science Dong Huynh gamified processes. See more at www.collabmap.org. App that won the TAG challenge. Using monetary incentives, the app A. Rutherford, M. Cebrian, I. Rahwan, S. Dsouza, J. Victor Naroditskiy was able to gather a large team of contributors to identify terrorists in McInerney, V. Naroditskiy, M. Venanzi, N. R. Jennings, J.R. Nick Jennings 4 cities across the world. The mechanism draws upon the DARPA red deLara, E. Wahlstedt, S. U. Miller. Targeted social balloon, networked incentives method. mobilization in a global manhunt. PLoS ONE, 2013 http://scailab.media.mit.edu/crowdscanner URL: www.squadguru.com - An algorithm for sequential decision Matthews, T., et al.(2013). Competing with humans at Gopal Ramchurn making in the FF domain. An interface for human input is under fantasy football: team formation in large partiallydevelopment. observable domains. . In Twenty-Sixth Conference of the Association for the Advancement for Artificial Intelligence. A web application that provides an interactive interface to your electricity consumption. See a demo at http://vimeo.com/42328926 Costanza, E., et al. (2012). Understanding domestic energy consumption through interactive visualisation: a field study. UBICOMP Crowdsourcing classification of astronomical objects. The IBCC is used E. Simpson, et al. (2011). here to help understand the types of users in the system and avoid Bayesian Combination of Multiple, Imperfect Classifiers. them getting bored or improve their performance by sending them Proceedings of NIPS 2011 workshop. tasks that fit the purpose. Enrico Costanza, Gopal Ramchurn IBCC applied to crowdsourcing to untrustworthy workers. The algorithm detects the bad workers and fires them. It also hires new workers and keeps them in if it needs more information. At myJoulo.com - an 'internet-of-things' application where a low-cost temperature logger can be used to collect home heating data and a web interface that is driven by machine learning in order to do estimates of usage/savings from the data. Edwin Simpson, Steve Reece, Gopal Ramchurn Alex Rogers Incentive mechanisms for engaging people in crowdsourcing activities. The project was carried out in partnership with Cancer Research UK. The website was viewed almost 9,000 times and over 1,500 users signed up over the course of three months A suite of python modules providing: an implementation of the W3C PROV Data model with JSON import/export support; a light-weight provenance tracker for recording provenance (in python code) with Provenance Service minimal effort; a provenance repository service that provides a REST API for storing/retrieving provenance and a Web interface for browsing the repository's content. working paper based on TREC challenge work (See below) Rogers, A., Ghosh, S., Wilcock, R. and Jennings, N. R. (2013) A Scalable Low-Cost Solution to Provide Personalised Home Heating Advice to Households [Best Paper Award BuildSys 2013]. In, 5th ACM Workshop On Embedded Systems For Energy-Efficient Buildings (BuildSys) 8pp. http://outruncancer.co.uk/ Outrun Cancer SharedTask BCC App TariffAgent Text Retrieval Crowdsourcing Challenge Ver.ily An algorithm developed in collaboration with MSR to classify tweets that won the SharedTask Challenge at Crowdsourcing at Scale competition at HCOMP 2013 An app to simulate real-time tariffs and to evaluate human-agent interaction for tariff switching on a daily basis in the face of uncertain prices due to changing weather conditions. A challenge as part of the Text Retrieval Conference run by NIST. The 'Crowdsourcing' challenge requires the use of crowdsourcing to help match topic-document pairs. The approach taken by the Soton-Oxford team involved the use of IBCC (Independent Bayesian Classifier Combination) and an Amazon Turk Deployment. A web application that tasks volunteers with verifying reports from the ground or on-line during major disasters. Steve Roberts, Edwin Simpson Victor Naroditskiy Seb Stein http://www.w3.org/2011/prov/wiki/Main_Page , Luc Dong Huynh, Luc Moreau, Paul T. Groth: Provenance: An Introduction to Moreau PROV. Synthesis Lectures on the Semantic Web: Theory and Technology, Morgan & Claypool Publishers 2013, ISBN 9781627052214, pp. 1-129 https://crowdflower.com/blog/2013/12/crowdsourcing-at- Matteo Venanzi scale-shared-task-challenge-winners/ Alan, A., et al. (2014). Mixed-initiative electricity tariff switching for dynamic environments. AAMAS 2014 Simpson, E., et al. (2013). Using a Bayesian Model to Combine LDA Features with Crowdsourced Responses. In , the Twenty-First Text REtrieval Conference (TREC 2012) Alper Alan, Enrico Costanza, Gopal Ramchurn Gopal Ramchurn, Edwin Simpson, Steve Reece http://veri.ly/ Enrico Costanza Further details of these applications can be found at http://www.orchid.ac.uk/technology-genres/citizenscience, http://www.orchid.ac.uk/technology-genres/disaster-response/, http://www.orchid.ac.uk/technologygenres/smart-grid/. 29 APPENDIX C - DELIVERABLES Date Deliverable Title Type 07/01/2014 D1.1 Flexible forms of control and influence Report 14/01/2014 D3.1 Learning and Inference in Decentralised Report Systems of Mixed Humans and Agents 16/01/2014 D5.1 User experience of crowd sourcing biodiReport versity data 05/02/2014 D2.1 Models of Consent in Autonomous Systems Report 18/02/2014 D3.1 Referral Incentives in Crowdfunding Report 19/02/2014 D5.1 New online mechanisms for settings with Report probablistic supply 11/03/2014 D4.3 Transformations and patterns discovery in Report provenance graphs 24/03/2014 D5.2 Supporting the work of the energy advisor Report 24/03/2014 D1.1 Computational planning support of disaster Report responders in a mixed-reality game probe 09/06/2014 D1.1/1.2 Report on applying decentralised coordina- Report tion and planning algorithms to disaster response scenarios with human-agent teams. 25/06/2014 D1.1 A new information gathering framework for Report co-ordinating UAVs in finding victims in disaster scenarios. 15/07/2014 D5.1 Applying decentralised hard/soft data fusion Report algorithms to crowd sourced data 01/09/2014 D3.1 Development of a coordination framework Report and presentation of the results obtained 24/09/2014 D5.1/5 Knowledge Transfer Status Report 4 Report 24/09/2014 D1.1 Multi-User Experience of Occupancy within Report a Human-Agent Collective 30/09/2014 D3.1 Theoretical properties and empirical eval- Report uation of referral incentives 01/10/2014 D2.1 Collaboration between heterogeneous Report agent groups in SAR 01/10/2014 D5.2 Developing appropriate preheating control Report methods 01/10/2014 D1.1 Consumer oriented residential demand re- Report sponse 22/10/2014 D5.3 Understanding the work of disaster reReport sponse: a field study 27/10/2014 D4.4 Trust-based algorithms for fusing Report crowdsourced estimates of continuous quantities 03/12/2014 D5.2 Deployment of Electric Vehicle Charging Report Simulation 03/12/2014 D1.1 Multi-UAV Control - a lab student using Report fNIRS 30/12/2014 D3.1 Report on new task allocation algorithms Report for the budget-limited task allocation problem in crowdsourcing systems 30 Author Feng Wu Steve Reece Stuart Moran Stuart Moran Victor Naroditskiy Sebastian Stein T Dong Huynh Joel Fischer Joel Fischer Feng Wu Chris Baker Steve Reece Alexandros Zenonos Dave Nicholson Stuart Moran Victor Naroditskiy Zoltán Beck Thanos Panagopoulos James Holyhead Joel Fischer Matteo Venanzi Sebastian Stein Joel Fischer Long Tran-Thanh ORCHID Fourth Annual Report 2014 APPENDIX D — STATEMENTS OF WORK All research undertaken in the project is covered by a statement of work. In this section, the statements that were active at some point in the report period are listed. Commenced Researcher/PhD student 14/01/2014 Edwin Simpson 31/01/2014 Amr Hussain 31/01/2014 Chris Baker 10/02/2014 Matteo Venanzi 10/02/2014 Elliot Salisbury 10/02/2014 11/02/2014 Title Work Area and Packages* Location WA1, WA3, WA5 Interactive Informative Feature Learning in HACs Scheduling and Coordination of Telescopes for Transient Events Follow-Ups WA1 Maximising Information and Utility Value in WA1: WP1. WA2: WP1. WA5: Heterogeneous Teams of UAVs and UGVs WP3 Bayesian Aggregation Models for Crowdsourced SpatialTemporal Data WA4:WP4, WP6 Oxford Southampton Southampton Southampton Feng Wu Crowd Sourcing in Disaster Response Scenarios Efficient Algorithms for Human-Agent Coordination in Disaster Response Scenarios WA5:WP1, WP3 WA1:WP1, WP2, WA5:WP1, WP3 Matthew Pike The role of fNIRS in HCI evaluation WA1:WP1, WA2: WP3, WA5:WP1, Nottingham WP5 11/02/2014 Alexandros Zenonos Development of coordination mechanisms and WA3:WP3, WA4:WP4, WA5: incentive strategies to tackle crowdsourcing problems WP3 18/02/2014 Thanos Panagopoulos 20/02/2014 Sasan Maleki 20/02/2014 James Holyhead 27/02/2014 Mark Ebden 01/03/2014 Victor Naroditskiy 03/03/2014 Southampton Southampton Southampton Towards Optimal House Preheating Fair Payoff Division Mechanisms for Scenarios with Large Number of Agents Consumer Oriented Residential Demand Response in the Smart Grid WA1:WP1, WA5:WP2 Southampton WA2 WA1:WP1, WP2, WA5:WP1, WP3 Southampton WA4:WP2, WP5, WA3:WP1 Oxford WA3:WP1,WP2 Southampton Tom Nickson Development of Provenance-based Incentives in HACs Referral Incentives in Citizen Science and Disaster Response Decentralised route planning for real world Bayesian Optimisation using UAVs 10/03/2014 Stuart Moran Patterns and Occupancy in Human-Agent Collectives 26/06/2014 Joel Fischer 27/06/2014 Justin Bewsher 02/07/2014 Long Tran-Thanh HACs in the Wild Prognostic changepoint and fault detection in electrical signals Budget-Limited Task Allocation in Crowdsourcing Systems 16/07/2014 Zoltan Beck 08/08/2014 Steve Reece *Work Areas Agent Collaboration in Search and Rescue Decentralised Heterogeneous Bayesian Classifier Combination Southampton WA1, WA2, WA5 Oxford WA1: WP1, WP2, WA5: WP3, WP4, WP5. Nottingham WA1 : WP1, WP2, WP3. WA5: WP2, WP3, WP5 Nottingham WA1 WA2 WA3 WA5 Oxford WA3: WP1, WA5: WP2 Southampton WA2 : WP1. WA5 : WP3 Southampton WA3: WP1. WA5: WP1, WP3, WP4 Oxford Work Packages WP1 Flexible forms of control and influence, WP2 Awareness and attention in HACs, WP3 New forms of agency WP1 Decentralised coordination under uncertainty, WP2 Human agent interaction in coordination, WA2 Agile Teaming WP3 Integration and demonstration WP1 Decentralised inference and control, WP2 Bayesian Games and weak control, WP3 Inference over WA3 Incentive Engineering dynamic collectives WP1 Accountable information infrastructure, WP2 Provenance Model and reasoning, WP3 Architectures for Provenance/Trust, WP4 Representing human actions in crowdsourcing, WP5 User WA4 Accountable Information Architecture exploitation of accountability info, WP6 Trust and information fusion models WP1 Defining scenarios, WP2 Energy systems applications, WP3 Disaster management applications, WA5 Applications WP4 Monitoring of new application areas, WP5 Study of ethics and privacy issues in HACs WA1 Flexible Autonomy If you would like copies of any (or all) statements of work please email [email protected] 31 ORCHID Fourth Annual Report 2014 APPENDIX E — KEY MEASURES OF SUCCESS Research Quality Number of publications: 226 @ 31 January 2015 Number of top-tier publications (Core A* conferences and top journals GEB, AIJ, JMLR, JAIR) 169 o 41 journal articles o 128 conference papers Number of citations (based on Google Scholar): 2168 Prizes and Awards (including best papers): 23 o ICICA 2014 Best paper – Levenberg et al o Buildsys 2014 Best Demo 2014 – Parson et al o World 3rd in International Microelectronics Competition 2014 - Beck o Winner Google Impact Challenge 2014 – Roberts et al. o WWW 2014 Best Paper Runner-up - Venanzi et al o BuildSys 2013 Best Paper - Rogers et al. o Winner HCOMP 2013 Crowdsourcing at Scale Shared Task Challenge - Venanzi et al o Winner 2013 British Gas Connecting Homes Start-up Competition – MyJoulo o EPSRC Doctoral Award 2013 – Parson o UbiComp 2013 Best Paper (Honourable Mention) – Luger o IJCAI 2013 Outstanding Student Paper - Zilli and Parson o CHI 2013 Best Paper - Rodden et al o Dean's Award for Research Excellence University of Southampton 2013 – Rahwan o AAMAS 2013 Best Student Paper Award – Alam et al o BCS Distinguished Dissertation Award (honourable mention) – Tran-Thanh o ECCAI 2012 Artificial Intelligence Dissertation Award (honourable mention) – Tran-Thanh o Runner-up TREC Challenge 2012 – Simpson et al o Winner TAG Challenge 2012 – Naroditskiy et al o AAAI 2012 Honourable Mention Award – Tran-Thanh et al o ECAI 2012 Nomination Best Student Paper Award –Tran-Thanh et al o IEEE International Conference on Data Mining 2012 Best paper award - Reece et al o NIPS 2012 Workshop on Human Computation for Science and Sustainability Best Contribution Award - Simpson et al 32 ORCHID Fourth Annual Report 2014 Research Impact Collaborations formed with academic groups outside ORCHID: 25 o o o o o o o o o o o o o o o o o o o o o o o o o o o o o Hans Fangohr, CDT in Next Generation Computational Modelling (Southampton) Niki Trigoni, CDT in Autonomous Intelligent Machines and Systems (Oxford) Sven Seuken, University of Zurich Ruth Retie, Kingston Business School Mike Hazas, Lancaster University James Scott, Microsoft Research Abigal Sellen, Microsoft Research Keith Edwards, Beki Grinter, Georgia Tech Kathy Willis, Oxford Biodiversity Institute, Kew Gardens Derek McAuley, Horizon Mario Berges, Carnegie Mellon University Mark Girorlami, UCL, Muffy Calder, Glasgow University Kepler pipeline team, NASA Prof Rob Fender, University of Oxford (LOFAR & SKA) Tao Qin, Microsoft Research Asia Peter Key, Microsoft Research Cambridge Botond Szabo, CREST Paris Jia Yuan Yu, IBM Research Dublin Suzanne Aigrain, Exoplanet research group, Oxford Aris Karastergiou, Radio Astronomy group, Oxford Marian Dawkins, Animal Behaviour Group, Oxford Ioannis Psorakis, Doyne Farmer, Institute for New Economic thinking, Oxford Simon Benjamin, Quantum Computing Group, Oxford Interaction Design, Chalmers University of Technology, Gothenburg, Sweden Systems Group University of Cambridge Warwick Management Group Edinburgh School of Design Royal College of Art Invited Keynotes: 11 o o o o o o o o o o o ICAART 2015 Big Data, Big Models, Big Deal - CSML 2014 MaxEnt 2013 ECML 2013 EPSRC student future workshop AAMAS HADM 2013 workshop AAMAS 2012 Computational Sustainability 2012 UKRC Digital Economy All Hands Meeting 2012 Royal Society “time series modelling for physical sciences” NIPS workshop on crowd-sourcing 2012 33 Nick Jennings Steve Roberts Steve Roberts Steve Roberts Tom Rodden Tom Rodden Alex Rogers Alex Rogers Alex Rogers Steve Roberts Steve Roberts ORCHID Fourth Annual Report 2014 Workshops in Major Venues: 24 o o o o o o o o o o o o o o o o o o o o o o o o o Massive MAS, 2015 HAIDM, AAMAS 2015 Ubicomp 2014 HAIDM, AAMAS 2014 NordiCHI 2014 RSS 2014 Provenance Week 2014 IPAW 2014 Provenance Analytics Workshop 2014 Sigcomm 2013 Ubicomp 2013 AMEC 2013 NIPS 2013 HAIDM, AAMAS 2013 OPTMAS 2013 ATES, AAMAS 2013 ECSCW 2013 ATES, AAMAS 2012 NIPS 2012 NCAF 2012 TADA/AMEC 2012 HAIDM, AAMAS 2012 OPTMAS 2012 OPTMAS 2011 ATES, AAMAS 2011 Long Tran-Thanh Gopal Ramchurn Tom Rodden/Ewa Luger Gopal Ramchurn Joel Fischer Steve Roberts Luc Moreau Luc Moreau Luc Moreau/Dong Huynh Tom Rodden Tom Rodden Seb Stein Mike Osborne Gopal Ramchurn Gopal Ramchurn Alex Rogers Joel Fischer Alex Rogers Mike Osborne Steve Roberts Seb Stein Gopal Ramchurn Gopal Ramchurn Gopal Ramchurn Alex Rogers Research Visits from other organisations to work with ORCHID: 23 o o o o o o o o o o o o o o o o o o o o o o o Filippo Bistaffa Sofia Ceppi Renzo de Nardi, Mathijs de Weerdt Geradline Fitzpatrick Keiichiro Hayakawa Francois Jean Marina Jirotka Marc Langienrich Fabio Maffioletti Areej Malibari Robert Niven Toni Penya-Alba Riccardo Reffato Avi Rosenfeld Simo Sarkka Mike Shann Arfon Smith Piotr Szczepanski Mike Twidale Twan van Laarhoven Ioannis Vetsikas Yair Zick University of Verona, Verona, Italy Politecnico di Milano, Italy UCL, UK TU Delft, The Netherlands University of Vienna, Austria Toyota Central R&D labs ENSTA-Bretagne, France University of Oxford, UK University of Lugano, Switzerland University of Verona, Verona, Italy King AbdulAziz University, Saudi Arabia CarbonCure IIIA – CSIC, Spain University of Verona, Verona, Italy Jerusalem College of Technology, Israel Aalto University, Finland University of Zurich, Switzerland Adler Planetarium, Chicago, US Warsaw University of Technology, Poland University of Illinois, US Radboud University, Nijmegen, The Netherlands IIT Demokritos, Greece Nanyang Technological University, Singapore 34 ORCHID Fourth Annual Report 2014 • Extended visits by ORCHID researchers to other organisations: 10 o Internship at MSR 2014 - Matteo Venanzi o Internship at MSR 2014 - Davide Zilli o Internship at MIT 2014 – Davide Zilli o The Moller-Maersk Institute in Denmark 2013 – Gopal Ramchurn o University of Southern California 2013 – Gopal Ramchurn o Internship at MSR 2013 - Matteo Venanzi o Internship at PlotWatt, 2013 – Oliver Parson o Internship at BAE Systems 2013 - Sam Miller o University of Verona 2012– Gopal Ramchurn o Internship at PlotWatt 2012 – Oliver Parson Software and standards released: 10 o Latent force software accompanying JMLR article (http://jmlr.csail.mit.edu/papers/v15/), 2014 o Smart Home Network (http://www.smarthomeframework.org/), 2014 o NILMTK: An Open Source Non-intrusive Load Monitoring Toolkit (https://github.com/nilmtk/nilmtk)., 2014 (40 starred on Github) o Literatin via chrome web store. This plug-in allows users to explore the complexity of any text found within a webpage. (https://chrome.google.com/webstore/detail/literatin/igpbgncfceidjgcjddcdamjgbpdcmbnl? hl=en-GB), 2014 (1,172 downloads) o Infer.NET Community-based Bayesian Classifier Combination (http://research.microsoft.com/en-us/um/cambridge/projects/infernet/), 2013 (550 downloads) o Underpinning machine learning software developed for ORCHID-funded mobility analysis work (https://pypi.python.org/pypi/vbihmm/), 2013 (1,351 downloads) o PROV Standard (http://www.w3.org/TR/prov-overview/), 2013 o ProvToolbox (http://lucmoreau.github.io/ProvToolbox/), 2013 o ProvStore (https://provenance.ecs.soton.ac.uk/store/): a public online repository for provenance documents, 2012 (Active users (those with data): 33, Total number of document stored: over 13K) o Prov Python library version 1.1.0 (https://pypi.python.org/pypi/prov), 2012 (46,000 downloads) 35 ORCHID Fourth Annual Report 2014 Industrial Impact Number of technologies pulled through: 8 o Cicada app and analysis expanded and applied by Centre for Ecology and Hydrology to cover all UK opthoptera. o Human mobility analysis technology applied by BAE Systems in security and counterterrorism context. o DEMS (Portsmouth BAE): implemented algorithms to monitor and control loads in micro grids. o IBCC technology applied by BAE Systems in context of Cyber and Aircraft fault classification. o PyIBCC software supplied to Zooniverse for use with citizen science projects (planet hunters, snapshot Serengeti, space warps). o Crowdsourcing Module developed with Microsoft Research being tested on an experimental platform used by Bing researchers prior to further development and application. o Joulo Ltd founded. Licensing analysis algorithms from the University of Southampton. Acquired by Quby early 2015. o AgentSwitch spin off in progress. Working with Hampshire County Council for community energy purchasing programme. Patents/Licenses given out: 6 o “Crowdsourcing System with Community Learning”, Inventors: Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, Milad Shokouhi, MS 340522.01, January 2014 o “Adaptive Task Assignment”, Inventors: Matteo Venanzi, John Guiver, Pushmeet Kohli, MS355469.01, August 2014 o “Sensor Data Processing”, Inventors: Reece, Roberts and Lloyd (Oxford) and Nicholson (BAE), Patent No: PCT/GB2012/000481, 2012 o “A method for inferring an output distribution using a Gaussian Process (on kernelised data)”, Inventors: Reece, Roberts and Lloyd (Oxford) and Nicholson (BAE) Patent No: GB1109209.5., 2011. o “A method for inferring a relevance for a data source using a regression technique (on kernelised data), Inventors: Reece, Roberts and Lloyd (Oxford) and Nicholson (BAE), Patent No: GB1109210.3 o “A method for inferring the relevance for a pre-processing technique using machine learning”, Inventors: Reece, Roberts and Lloyd (Oxford) and Nicholson (BAE) Patent No: GB1109206.1, 2011. 36 ORCHID Fourth Annual Report 2014 Number of external organisations engaged with: 42 o AHL ○ ASUR o Avon Fire & Rescue Service ○ Babcock o BAE Systems ○ BBC o Blue Bear ○ BP o British Gas ○ Cancer Research UK o Centre for Ecology and Hydrology ○ Centre for Sustainable Energy o Department of Energy and Climate Change ○ DSTL o EDF ○ Energy Savings Trust o Energy Technologies Institute ○ EPSRC o ESA ○ Envitia o Google ○ Halo Crisis Management o Home Office ○ IBM o Kiwi Power ○ Microsoft o NASA ○ Navetas o Npower ○ PassivSystems o PlotWatt ○ QinetiQ o Rescue Global ○ Schlumberger o Secure Meters Ltd. ○ Toshiba Research Lab o Toyota Central R&D Labs, Inc. ○ UKERC o Urban Angel ○ Ushahidi o YouGov 37 ORCHID Fourth Annual Report 2014 Outreach Media mentions (including BBC, New Scientist, The Guardian, WIRED and Economist ): 88 Deployments of trial systems (number of participants): o o o o o o o o o o MyJoulo 2245 users Cicada 5000 users AgentSwitch 3000 users AtomicORCHID 82 users TariffAgent 20 users Apocalypse of MoP 850 users Outrun Cancer 1577 Veri.ly 70 users Demand Response System (Agent B) 20 users Fukushima Heatmap 7,064 page views Demonstrations and presentations at public events: 16 Demo/Presentation Where Disaster response technologies New Forest Cicada British Science Festival, Birmingham Whe n 2014 The Wood Fair, New Forest 2014 ORCHID Industry Day London 2014 Autonomous Systems Showcase HACs in the Smart Grid Southampton 2014 Webinar 2013 Disaster response technologies New Forest Cicada First Lego League Event 2013 BBC Summer of Wildlife, Birmingham 2013 New Forest Cicada British Science Festival, Newcastle 2013 I’m a Scientist… Online 2013 ORCHID Industry Day Farnborough 2013 New Forest Cicada Big Bang Fair, Solent Region 2013 New Forest Cicada 2013 ORCHID Overview University of Southampton Science and Engineering Day Big Bang Science and Engineering Fair, Birmingham Farnborough Air Show AtomicORCHID University of Southampton Open Day 2012 CARGO igFest, Bristol 2012 ORCHID Overview 2012 2012 Master classes, learning and classroom materials provided: 9 o o o o o o o o o Researcher Links Workshop for British Council on Disaster Response (Mexico), 2015 Incentive mechanisms for electric vehicle charging for Toyota Central R&D Labs, 2014 Designed a learning activity and video based on Outrun Cancer for the University of Southampton MOOC on Digital Marketing, 2014 NILM 2014 @ London (European venue for discussing non-intrusive load monitoring research from industry and academia) Keynote on energy disaggregation at Human-Centered Energy Management workshop, 2014 Training course on energy disaggregation for Telekom Malaysia R&D, 2014 Training course on energy disaggregation for British Gas Connected Homes, 2014 Police Workshop on Provenance (NCA, South East Regional Cyber Crime Unit), 2014 Provenance Tutorials (Southampton and Nottingham), 2013 38 ORCHID Fourth Annual Report 2014 Research Capacity Building Number of PhDs graduated: 10 o o o o o o o o o o Jan-Peter Calliess Ewa Luger Matteo Venanzi James McInerney Sam Miller Oliver Parson Muddasser Alam Edwin Simpson Oleksandr Pryymak Francesco Delle Fave 2014 2014 2014 2014 2014 2013 2013 2013 2013 2012 Oxford Nottingham Southampton Southampton Southampton Southampton Southampton Oxford Southampton Southampton PDRAs appointed to faculty positions: 7 o Long Tran-Thanh 2015 University of Southampton o Joel Fischer 2014 University of Nottingham o Nadia Pantidi 2014 University of Cork o o o o Feng Wu Talal Rahwan Sebastian Stein Michael Osborne 2014 2014 2013 2012 University of Science and Technology of China Masdar Institute of Technology University of Southampton University of Oxford Academic promotions: 3 o o Michael Osborne (Lecturer to Associate Professor) (2014) Nick Jennings (Professor to Regius Professor) (2014) o Alex Rogers (Reader to Professor) (2012) Research interns employed on ORCHID: 12 o o o o o o o o o o o o Petre Andre Andrei Petre David Smith Debanjan Ghosh Alex Botev Po Ting Tse Quan Tran Sam Millar Jack Flann Po Ting Tse Quan Tran Sam Millar 2011 2012 2012 2012 2013 2013 2013 2013 2014 2014 2014 2014 39 ORCHID Fourth Annual Report 2014 Number of follow on grants: Follow on Grant 25 Funder When Where Value(£k) Southampton/CSE/ Open System solutions Ltd/Local authorities Southampton/ KiwiPower Southampton 494 Aperio: Low cost façade management in naturally ventilated buildings EPSRC 2014 Adaptive Demand Response TSB 2014 Researcher Links Workshop Grant for Disaster Response (Mexico) British Council 2014 Casma ESRC 2014 Nottingham 500 Biosound/Mosquito sensor project Google 2014 Oxford/ Southampton 500 APERIO: Low cost facade management in naturally ventilated buildings EPSRC 2014 Southampton 500 Intelligent SME energy management and trading with ancillary services. EPSRC / TSB 2014 Southampton 200 Thermal Modeling and Inference with HomeOS ASUR: Human-Agent Collaboration for Multi-UAV Control EPSRC / MSR 2014 Southampton 60 DSTL 2014 Southampton 50 KTS with BAE Systems EPSRC 2014 50 CharIoT: Leveraging the Internet of Things to Reduce Fuel Poverty EPSRC 2014 Southampton/ BAE Systems Southampton/ Nottingham/ CSE Human Mobility Analysis and Anomaly Detection CDE 2013 Southampton/ BAE Systems 77 Creating the energy for change EPSRC 2013 Southampton/ Nottingham 1076 ORCHID Artist in Residence Leverhulme Trust 2013 Southampton 15 Domestic energy management HEIF Fund 2013 Southampton 15 Referral incentives in crowdsourcing British Academy 2013 Southampton 8 Referral incentives in crowdsourcing FSHS Strategic Interdisciplinary Pilot study with Research Kingston Development Fund EPSRC 2013 Southampton 7 2013 50 2013 Southampton/ Kingston Business School Southampton 1316 Autonomous behaviour and learning in an uncertain world Prototyping open innovation models for ICT-enabled manufacturing in food and packaging Home Hub-of-all-Things as platform for multi-sided market powered by internet-ofthings User centred networking EPSRC 2013 Oxford 1200 EPSRC 2013 Nottingham 1827 EPSRC 2013 Nottingham 982 EU 2013 Nottingham 3947 Automated assessment of broiler chicken welfare using optical flow patterns in relation to behaviour, disease risk and production A robust toolbox for exoplanet analysis BBSRC 2013 Oxford 738 Leverhulme Trust 2013 Oxford 220 Biosound James Martin School 2013 Oxford 225 TOTAL 14634 Transforming Feedback –Interactive, Practice-level Visualisation of Electricity Consumption Multiagent collectives for sensing, 250 50 277 autonomy, intelligence and control 40 ORCHID Fourth Annual Report 2014 APPENDIX F — PROJECT MANAGEMENT DATA The programme started on the 1 January 2011 and received £5.5M in funding from EPSRC. This was supplemented by £3.67M of additional investments: 24 PhD studentships provided by the partner universities at a cost of £1.35M; a £1M commitment from BAE Systems to an aligned programme to pull through the results of ORCHID into their corporate research programme and relevant Business Units; a £400k commitment from Secure Meters to a parallel programme in which they allocate a full-time engineer to pull through the results of ORCHID into their commercial activities; a £255k contribution in kind from BAE Systems and Secure Meters for their senior managers and scientists to fulfill their various research and advisory roles associated with ORCHID; a £320k contribution from ACFR to support engagement with their field robotics capability; and a £250k contribution from Southampton’s Strategic Development Fund to fund a research lecturer (Dr S Stein) and co-fund a research fellow (Dr T Dong Huynh) and to interface with their University Strategic Research Groups. Spend Profile Actual December 2014 Directly incurred Staff Travel Equipment Other Seedcorn Sub total Exceptions Student Maintenance 100% Student Fees Sub Total Total 100% (includes directly allocated costs) Southampton Total Budget Actual to date £1,187,606.53 £1,076,828.89 £266,710.00 £283,587.74 £24,500.00 £9,469.23 £145,114.65 £96,544.11 £200,000.00 £163,701.21 £1,823,931.18 £1,630,131.18 £84,777.00 £84,777.00 £21,340.50 £21,340.50 £106,117.50 £106,117.50 £3,587,361.49 £2,657,762.86 Oxford Remaining Total Budget Actual to date £110,777.64 £548,728.02 £402,508.40 -£16,877.74 £84,168.00 £57,747.86 £15,030.77 £0.00 £0.00 £48,570.54 £28,755.68 £28,755.68 £36,298.79 £0.00 £0.00 £193,800.00 £635,149.70 £489,011.94 £0.00 £0.00 £0.00 £0.00 £159,176.25 £102,264.57 £0.00 £159,176.25 £98,664.57 £929,598.63 £1,728,062.67 £1,302,311.07 Nottingham Remaining Total Budget Actual to date £146,219.62 £523,192.70 £437,929.00 £26,420.14 £67,334.40 £82,655.57 £0.00 £0.00 £0.00 £0.00 £45,589.28 £43,565.27 £0.00 £0.00 £0.00 £172,639.76 £636,116.38 £564,149.84 £0.00 £127,165.50 £107,608.71 £56,911.68 £32,010.75 £26,319.66 £56,911.68 £159,176.25 £133,928.37 £425,751.60 £1,415,893.60 £999,038.16 Remaining £85,263.70 -£15,321.17 £0.00 £2,024.01 £0.00 £71,966.54 £19,556.79 £5,691.09 £25,247.88 £416,855.44 Southampton and Nottingham have overspent on their travel budgets, as researchers have actively collaborated on the development of the main HAC this year. The budgets will be actively managed to ensure that researchers can continue to actively work together as ORCHID draws to a close. 41 ORCHID Fourth Annual Report 2014 Staff Employed Oxford PI CI CI CI CI Steve Roberts CI Mike Osbourne CI Researcher Researcher Researcher Researcher Researcher Researcher Researcher Researcher Researcher Researcher Researcher Steve Reece Researcher Mark Ebden Researcher Edwin Simpson Researcher Researcher Researcher Researcher PHD Student PHD Student PHD Student PHD Student PHD Student PHD Student PHD Student PHD Student PHD Student PHD Student PHD Student PHD Student Jan Calliess PHD Student Justin Bewsher PHD Student Tom Gunter PHD Student Tom Nickson PHD Student Mark Mcleod PHD Student Jonathan Downing PHD Student PHD Student PHD Student PHD Student PHD Student PHD Student PHD Student Technical Staff Technical Staff Technical Staff Project Administrator Knowledge Transfer Officer Nottingham Associated Researchers* Derek McAuley Brian Logan Maria Polukarov Enrico Costanza Oliver Parson Siddhartha Ghosh (left 30/09/2013) Napta Gupta Amir Sezavar Keshavarz Obaid Malik Associated PhD Student* Southampton Nick Jennings Luc Moreau Alex Rogers Gopal Ramchurn Tom Rodden Seb Stein Talal Rahwan T Dong Huynh Victor Naroditskiy Feng Wu Greg Hines Meritxell Vinyals Long Tran Thanh Ruben Stranders Matteo Venanzi Joel Fischer Nadia Pantidi Khaled Bachour Stuart Moran Sasan Maleki Amr Hussein Chris Baker Hale Harding Zoltán Beck Alexandros Zenonos James Holyhead Elliot Salisbury Thanos Panagopoulos Radu Pruna Jhim Verame Wenchao Jiang Daniela Dybalova Robert Spencer Leigh Clark Richard Wetzel Chaoyu Ye Matthew Pike Yukki Ikuno Michele Roncalli Andrei Petre Angela Westley Dave Nicholson Start date 01/01/2011 01/01/2011 01/01/2011 01/01/2011 01/01/2011 01/10/2011 01/01/2011 01/01/2011 01/01/2011 12/05/2011 01/04/2011 14/10/2011 01/09/2011 01/06/2012 01/11/2011 01/09/2011 21/07/2014 01/01/2011 01/01/2011 01/10/2013 01/04/2011 01/08/2011 01/01/2012 01/02/2012 01/10/2011 26/09/2012 26/09/2012 26/09/2012 04/03/2013 01/10/2013 01/10/2013 01/10/2013 01/10/2013 01/10/2014 01/10/2014 01/10/2011 01/10/2012 01/10/2012 01/10/2012 01/10/2013 01/10/2014 26/09/2011 01/10/2011 26/09/2011 01/10/2012 01/12/2012 01/01/2013 01/01/2013 02/06/2014 01/10/2012 18/06/2013 07/03/2011 05/05/2011 End date 31/12/2015 31/12/2015 31/12/2015 31/12/2015 31/12/2015 31/12/2015 31/12/2015 31/12/2015 31/12/2013 30/12/2015 30/09/2014 30/06/2014 11/10/2013 31/07/2012 31/12/2015 29/02/2012 31/12/2015 31/12/2014 31/12/2014 31/12/2015 31/03/2014 31/07/2014 31/01/2015 31/10/2014 30/09/2014 31/03/2016 31/03/2016 31/12/2013 03/09/2016 30/03/2017 30/03/2017 30/03/2017 30/03/2017 30/03/2018 30/03/2018 30/09/2013 30/09/2015 30/09/2015 30/09/2015 30/09/2016 30/09/2017 25/09/2014 30/09/2014 25/09/2014 30/09/2015 30/11/2015 31/12/2015 31/12/2015 31/12/2015 24/06/2013 31/10/2013 06/03/2016 31/12/2015 *Associated researchers and students are those who have contributed to the project, but are not directly funded by it. 42 www.orchid.ac.uk [email protected] +44 (0) 23 8059 9512