Challenge
Transcription
Challenge
Beyond Jeopardy! Challenge: Cognitive Assistants Race Hamid R. Motahari Nezhad, PhD, Member of IBM Academy of Technology IBM Almaden Research Center San Jose, CA, USA © 2015 IBM Corporation IBM Global Technology Outlook: Driven by Research, Drives Strategy and Innovation 2012 2013 2014 2015 Cloud Mobile Social Data Internet of Things Volume Velocity Variety Veracity 4 Confluence of Social, Mobile, Cloud, Big Data, IoT and Analytics Systems of Insight Data Transforming Industries Data will disrupt entire science, technology and industries © 2013 IBM Corporation Data is the world’s new natural resource! (Ginni Rometti, IBM Annaul Report, 2014) 44 zettabytes From the dawn of civilization until 2003, humankind generated five exabytes of data. Now we produce five exabytes every two days…and the pace is accelerating. Eric Schmidt, Executive Chairman, Google unstructured data We are here structured data 2010 5 2015 2020 © 2013 IBM Corporation Big data creates new opportunities and challenges 90% of the world’s data was created in the last two years 3M+ Apps on leading App stores 4 80% of the world’s data today is unstructured 1 Trillion connected devices generate 2.5 quintillion bytes data / day By 2017 The collective computing and storage capacity of smartphones will surpass all worldwide servers © 2013 IBM Corporation A new computing paradigm is emerging Cognitive Systems Era Programmable Systems Era Tabulating Systems Era © 2013 IBM Corporation Cognitive Systems Era Discovery & Recommendation Probabilistic Big Data Natural Language as the Interface Intelligent Options 6 © 2013 IBM Corporation How we got here?: Grand AI Challenges in 1980s Challenge Where we are today? World Champion Chess Machine Mathematical Discovery Translating Telephone Accident Avoiding Car (Autonomous car) Self-Organizing Systems Self-Replicating Systems Turing Test for Humanoid Behavior (1950-) 7 R. Reddy, Foundations and Grand Challenges of Artificial Intelligence, AI Magazine, 1988 © 2013 IBM Corporation Where we are today?: Grand AI Challenges in 1980s 8 Challenge Where we are today? World Champion Chess Machine IBM DeepBlue, 1997 Won World Chess Champion Mathematical Discovery Open Challenge Translating Telephone Wildfire, Portico and Webely Translating Assistants (1992-1994) Accident Avoiding Car (Autonomous car) 2004 (Desert Challenge)-2008 (Commercial) Self-Organizing Systems Open Challenge Self-Replicating Systems Open Challenge Turing Test for Humanoid Behavior (1950-) Attempts has been made (Eliza 1966, Goostman 2014) © 2013 IBM Corporation Turing Test and Machine Intelligence Can a computer think? Are there imaginable digital computers which would do well in the imitation game?" The contest requires at least 30% (50%) of judges take machine as a human! Main variations: Standard Test (Loebner Prize), and Imitation Game Critics: focus on human behavior, and not intelligence AI has focused on intelligence, and on specific goals/tasks, and not on mimicking humans. - Some human behavior are not intelligent (typing mistakes, lie, …) - Some intelligent behavior are inhuman (solving difficult computational problems) 9 © 2013 IBM Corporation Human Intelligence in terms of Cognitive Abilities Ability to Achievable by machines today? draw abstractions from particulars. Partially, semantic graphs* maintain hierarchies of abstraction. Partially, semantic graphs* concatenate assertions and arrive at a new conclusion. Partially, relationships present reason outside the current context. Not proactively compare and contrast two representations for consistency/inconsistency. Limited reason analogically. Not automated, require domain adaptation learn and use external symbols to represent numerical, spatial, or conceptual information. Better than human in symbolic rep. & processing learn and use symbols whose meanings are defined in terms of other learned symbols. Uses and processes, limited learning invent and learn terms for abstractions as well as for concrete entities. No language development capability invent and learn terms for relations as well as things Not advanced, using symbols, not cognitive Gentner, D. (2003), In D. Getner & S. Goldin-Meadow (eds.), Language in Mind: Advances in the Study of Language and Thought. MIT Press. 195--235 (2003) 10 © 2013 IBM Corporation Human Intelligence vs. Machine Intelligence Analytical Skills Synthetic Skills Cognitive skills that machines excel at would take intellectual efforts from human Cognitive skills that human performs effortlessly but hard for machines with current AI – Mathematical calculations, making logical decisions in complex situations, chess Computational Intelligence – Manipulation of symbols through algorithmic information processing – The processing units (processing device) does not know or care about the “meaning” of symbol – Cognition by “information processing”, or cognition as computation – Interpretation of subtle facial expressions, engaging in creative conversations, etc. Conscious intelligence – Symbol manipulation also happens in the lowest level of hierarchical structure of brain function – The higher levels of hierarchical structure of brain function involve emergent concepts where higher level concepts/ideas combine, and form complex organisms (analogy with ‘cloud’, a whole, relation to air and water molecules, component) – It is at this level of cognition that “understanding of meaning” arise Ref.: Eric Lord, Science, Mind and Paranormal Experience, 2009 11 © 2013 IBM Corporation Cognitive Capability Levels Desired for an Intelligent Machine Higher level of cognition desired for an intelligent machine Discovery • Create new insights and new value Decision • Provide bias-free advice semiautonomously, learns, and is proactive Understanding • Build and reason about models of the world, of the user, and of the system itself Question Answering • Leverage encyclopedic domain knowledge in context, and interacts in natural language Cognitive Capability Touring Test* (Chinese Room) 12 © 2013 IBM Corporation A major challenge towards winning Touring Test: Building knowledge bases “For an artifact, a computational intelligence, to be able to behave with high levels of performance on complex intellectual tasks, perhaps surpassing human level, it must have extensive knowledge of the domain” The challenge of AI in making progress toward building human-like artifacts: – Knowledge representation, and (especially) knowledge acquisition Approaches – Build a large knowledge base by reading text – Distilling from the WWW a huge knowledge base EDWARD A. FEIGENBAUM, Some Challenges and Grand Challenges for Computational Intelligence, Journal of the ACM, Vol. 50, No. 1, January 2003, pp. 32–40 Semantic Web and Linked Data methods over the last decade extensively has explored building models, ontologies and rule-set that contributes to WWW knowledge representation – Manual, and semi-automated, focused on curated ontologies – Community participation in building ontologies have resulted in creation of large knowledge bases: DBPedia, Yago, Wikidata, Freebase, MediaWiki, etc. – Ontologies are expensive to build and scale, and are generic in nature 13 © 2013 IBM Corporation What changed the game for AI, and for the machine intelligence was …. Data + A New Computation Paradigm Advancing Question Answering Capability JEOPARDY! CHALLENGE 14 © 2013 IBM Corporation On February 14, 2011, IBM Watson made history . . . 15 © 2013 IBM Corporation Jeopardy Game: Automatic Open-Domain Question Answering Given – Rich Natural Language Questions – Over a Broad Domain of Knowledge Deliver – – – – 16 Precise Answers: Determine what is being asked & give precise response Accurate Confidences: Determine likelihood answer is correct Consumable Justifications: Explain why the answer is right Fast Response Time: Precision & Confidence in <3 seconds © 2013 IBM Corporation Analyzing Jeopardy against other Question-Answering Problems 17 Ref: David Ferrucci, et al. Towards the Open Advancement of Question Answering Systems. IBM Research Report RC24789. December 2008 © 2013 IBM Corporation Jeopardy Question Examples Broad/Open Domain $200 $400 The juice of this bog fruit is sometimes used to treat urinary tract infections Portland, Oregon is "The City of" these flowers Complex Language $2000 High Precision Accurate Confidence A little bird told us smoking or dentures can cause this most common fungal infection of the mouth Factoid Answers High Speed 18 $600 $800 In cell division, mitosis splits the nucleus & cytokinesis splits this liquid cushioning the nucleus Grace Murray Hopper is credited with applying this 3letter term to a mysterious computer problem © 2013 IBM Corporation DeepQA in Watson: Massively Parallel Probabilistic Evidence-Based Architecture DeepQA generates and scores many hypotheses using an extensible collection of Natural Language Processing, Machine Learning and Reasoning Algorithms. These gather and weigh evidence over both unstructured and structured content to determine the answer with the best confidence. Learned Models help combine and weigh the Evidence Evidence Sources Question Answer Sources Primary Search Question & Topic Analysis Candidate Answer Generation Question Decomposition Hypothesis Generation Hypothesis Generation 19 Answer Scoring Evidence Retrieval Hypothesis and Evidence Scoring Deep Evidence Scoring Synthesis Hypothesis and Evidence Scoring ... David Ferrucci, et al. , Building Watson: An Overview of the DeepQA Project, AI Magazine, 2010 Models Models Models Models Models Models Final Confidence Merging & Ranking Answer & Confidence © 2013 IBM Corporation Question interpretation, finding, scoring and ranking 20 © 2013 IBM Corporation Data as an Enabler: Where did it acquire knowledge? Three types of knowledge Domain Data (articles, books, documents) Wikipedia Time, Inc. New York Time Encarta Oxford University Internet Movie Database IBM Dictionary ... J! Archive/YAGO/dbPedia… Total Raw Content Preprocessed Content 21 Training and test question sets w/answer keys NLP Resources (vocabularies, taxonomies, ontologies) • 17 GB • 2.0 GB • 7.4 GB • 0.3 GB • 0.11 GB • 0.1 GB • 0.01 GB XXX • 70 GB • 500 GB Had access to 200 million pages of structured and unstructured content consuming four terabytes of disk storage Watson Machine consisted of: • 750 IBM Power servers, each 3.5 GHz POWER7 eight core processor, in total, the system has 2,880 POWER7 processor threads • 16 terabytes of RAM © 2013 IBM Corporation Lesson Learned from Watson (1): scalable knowledge model building method “The Watson program is already a breakthrough technology in AI. For many years it had been largely assumed that for a computer to go beyond search and really be able to perform complex human language tasks it needed to do one of two things: either it would “understand” the texts using some kind of deep “knowledge representation,” or it would have a complex statistical model based on millions of texts.” – James Hendler, Watson goes to college: How the world’s smartest PC will revolutionize AI, GigaOm, 3/2/2013 Breakthrough: – Developing a systematic approach for scalable knowledge building over large, less reliable data sources, and deploying a large array of individually imperfect algorithms to find right answers • Building and curating a robust, and comprehensive knowledge base and ruleset is laborious, time consuming and slow Source: Inquire Intelligent Book • Watson approach for building on massive, mixed curated and not-curated and less reliable information sources with uncertainty has proved effective 22 © 2013 IBM Corporation Lesson Learned from Watson (2): learning-based characterization of each algorithm Leveraging a large number of not always accurate techniques but delivering higher overall accuracy through characterizing and employing confidence levels Without confidence estimator With perfect confidence estimator Comparison of two QA systems with and without confidence estimation. Both have an accuracy of 40%. 23 © 2013 IBM Corporation Towards Mass Computing as the Shared Characteristic of Recent Computing Paradigm Shifts Building Stronger Super Computers Cloud Computing Scalable Computing over Massive Commodity Hardware Crowd Computing Big Data Advanced Analytics Watson: Mass computing applied to AI 24 Advanced individual algorithms Complex array of algorithms applied to make sense of data, and offer cognitive assistance © 2013 IBM Corporation Discovery Decision Understanding Question Answering BEYOND JEOPARDY!: COGNITIVE COMPUTING COGNITIVE ASSISTANTS, HEALTHCARE, WATSON SERVICES, AND DEEP LEARNING 25 © 2013 IBM Corporation Cognitive System A Cognitive System offers computational capabilities typically based on Natural Language Processing (NLP), Machine Learning (ML), and reasoning chains, on large amount of data, which provides cognition powers that augment and scale human expertise 2 Generates and evaluates evidence-based hypothesis 1 Understands natural language and human communication Watson 3 Adapts and learns from user selections and responses 26 Cognitive Systems do actively discover, learn and act © 2012 International Business Machines Corporation Gartner Hype Cycle of Technology 2015 Nuance’s Survey of 1000 users found… 27 © 2013 IBM Corporation Cognitive Assistant A software agent (cog) that – “augments human intelligence” (Engelbart’s definition1 in 1962) – Performs tasks and offer services (assists human in decision making and taking actions) – Complements human by offering capabilities that is beyond the ordinary power and reach of human (intelligence amplification) A more technical definition – Cognitive Assistant offers computational capabilities typically based on Natural Language Processing (NLP), Machine Learning (ML), and reasoning chains, on large amount of data, which provides cognition powers that augment and scale human intelligence Getting us closer to the vision painted for human-machine partnership in 1960: – “The hope is that, in not too many years, human brains and computing machines will be coupled together very tightly, and that the resulting partnership will think as no human brain has ever thought and process data in a way not approached by the information handling machines we know today” “Man-Computer Symbiosis , J. C. R. Licklider IRE Transactions on Human Factors in Electronics, volume HFE-1, pages 4-11, March 1960 28 1 Augmenting Human Intellect: A Conceptual Framework, by Douglas C. Engelbart, October 1962 © 2013 IBM Corporation History of Cognitive Assistants from the lens of AI Thinking machines DARPA PAL Program Expert Systems Touring Test, 1950 Logic Theorist (Newwell, Simon, 1955) 1965-1987 DENDRAL 1974-1984 MYCIN IRIS Checker Player (Samuel, 1956) NLS/Augment (Engelbart) Memex (Bush) 1987 Cognitive Tutors (Anderson) Eliza (Weizenbaum) Apple’s Knowledge Navigator System 1945 29 1955/6 1962 1966 1965-1987 Virtual Telephone Assistant CALO Portico, Wildfire, Webley; Speech Recognition Voice Controlled 1992-1998 2002-08 © 2013 IBM Corporation Modern Cognitive Assistants: State of the art (2008-present) Commercial Open Source/Research Personal Assistants and Bots OAQA – Siri, Google Now, Microsoft Cortana, Amazon Echo, FB M – Braina, Samsung's S Voice, LG's Voice Mate, SILVIA, HTC's Hidi, Nuance’ Vlingo – AIVC, Skyvi, IRIS, Everfriend, Evi (Q&A), Alme (patient assistant) – Viv (Global Brain as a Service) – x.ai, Telegram bots Cognitive and intelligent platforms – IBM Watson – Wolfram Alpha – Saffron 10 – Vicarious (Captcha) DeepDive OpenCog YodaQA OpenSherlock OpenIRIS iCub EU projects Cougaar Inquire* (intelligent textbook) * Curated knowledge base 30 © 2013 IBM Corporation Intelligent Assistants: Related App Landscape 31 © 2013 IBM Corporation A Society of Interacting Cognitive Agents (Bots) and Humans Cognitive Agent to Agent Human-Cog interaction Cognitive Agent to Human Human to Human Natural Language Planning a Vacation Trip Cog-Cog interaction Natural Language, or ACL? ACL: Agent Communication Language, KQML, etc. Cog-mediated Human Interaction Provider Cogs Travel Cog 1 Health Agent Personality Mediated and facilitated by Cogs Weather Cog Insight Cog. Considering preferences, experience, conditions, cost, Availability, etc. Natural Language-ACL-Natural Language Travel Cog 2 32 © 2013 IBM Corporation eAssistant: a cognitive assistant for the enterprise A mobile intelligent assistant for the enterprise that assist a user (worker) to be more productive by supporting following a methodology of monitor, process, recommend and do actions with the following capabilities – Understands human language – Monitors input channels including email, calendar chat and enterprise information sources – Builds a model of the user and the world, and is situational aware (context) – Offer assistance by • Pre-processing information, and presenting information in desired format • Categorizing and filtering information • Gathering and organizing information • Scheduling meetings • Identifying requests, and organizing to-dos of its human subject • Assists in performing tasks such as organizing events, travel assistant, and learns new tasks • And, suggest taking certain actions to its human subject that supports increasing productivity, and growth 33 © 2013 IBM Corporation eAssistant: Cognitive Assistant Types in Work Environment Personal (employee) eAssistant – Personal eAssistants have access to the data space (and applications) that the principal has access to with the same level of visibility – While eAssistant is proactive in making suggestions, it takes action under the control and direction of the principal Assistant’s eAssistant – An assistant to Human Assistants helping them to become more productive, and focus on work that require human judgment Expert/Process eAssistants – Assistants that are experts in a specific domain such as travel policy, human resources, etc. Individual cognitive agents Assistant’s Cognitive Agents Cognitive Assistant Platform 34 Systems of cognitive agents that collaborate effectively with one another to support human activities. Expert Cognitive Agents Interactions types need to be supported: • cog-to-cog interactions, • human-cog interactions, and • cog-backed human-to-human interactions © 2013 IBM Corporation Actionable Statement Identification Over Unstructured Conversations Email, Chat, and Calendaring apps are the most used channels for doing work in the enterprise Addressing the work organization and management for Knowledge workers: monitoring communication channels (email, chat), and: - 35 capturing, prioritizing and organizing work of a worker Identifying actionable statements (requests, commitments, questions) and track them over the course of conversations © 2013 IBM Corporation Cognitive Assistant for Task management Task, commitment and task lifecycle extraction from workers interactions over email and chat 36 Anup K. Kalia, Hamid R. Motahari Nezhad, Claudio Bartolini, Munindar P. Singh: Monitoring Commitments in People-Driven Service Engagements. IEEE SCC 2013: 160-167 © 2013 IBM Corporation Opportunity and challenge (2): cognitive methods and tools Cognitive Computing as a Service: Watson in IBM BlueMix Available today User Modeling Personality profiling to help engage users on their own terms. Message Resonance Communicate with people with a style and words that suits them Concept Expansion Maps euphemisms to more commonly understood phrases Relationship Extraction Intelligently finds relationships between sentences components Machine Translation Translate text from one language to another. Question and Answer Direct responses to users inquiries fueled by primary document sources Visualization Rendering Graphical representations of data analysis for easier understanding Language Identification Coming • • • • • • • • • • • • • • • • • • • • Concept Analytics Question Generation Speech Recognition Text to Speech Tradeoff Analytics Medical Information Extraction Semantic Expansion Policy Knowledge Ontology Creation Q&A in other languages Policy Evaluation Inference detection Social Resonance Answer Assembler Relationship identification Dialog Machine Translation (French) Smart Metadata Visual Recommendation Industry accelerators Identifies the language in which text is written 37 © 2013 IBM Corporation Deep Learning Deep learning is a branch of machine learning based on a set of algorithms that attempt to model high-level abstractions in data by using multiple processing layers with complex structures, or otherwise composed of multiple non-linear transformations (Wikipedia Definition) Most successful form of deep learning algorithms – Artificial Neural Networks (ANNs) • We have had good algorithm for training ANNs with 1-hidden layer, Deep Learning focuses on training ANNs with multiple hidden layers, at scale Deep learning methods use a cascade of many layers of nonlinear processing units for feature extraction and transformation Are hierarchical in learning the model (different levels of abstractions), favor unsupervised (and probabilistic) feature extraction and transformation Other examples: Deep Boltzmann Machine (DBM), a binary pair-wise Markov random field, Multi-layer Kernel Machines (MKM), Deep Q-Networks (Google DeepMind) Applications – Speech recognition, Image recognition, NLP Libraries – Torch, Theano, OpenNN, TensorFlow (Google), etc. Latest Development – Modeling Order in Neural Word Embeddings at Scale, 32nd International Conference on Machine Learning, 2015. – Digital Reasoning: training largest neural network with 160 Billion parameters (Google’s record was 11.2 Billion) – Achieved 85.8% accuracy, up from 76.2% (Google), with few hours/days of training time on commodity hardware 38 © 2013 IBM Corporation Todai Challenge: Tokyo University Entrance Exam 39 Japan National Institute of Informatics, http://21robot.org/ © 2013 IBM Corporation Open Challenges (1) Building the knowledge base and Training Cognitive Agents – How does User Train the Cog? – How to build a user model for the Cog? Adaptation and training of Cogs for a new domain – How to quickly train a cog for a new domain? Current approaches is laborious and tedious. Performance Dimensions, and Evaluation Framework – Metrics, testing and validating functionality of Cog – Are controlled experiments possible? – Do we need to test in Real environment with Real users User adoption/trust, and privacy – – – Can I trust that the Cog did what I told/taught/think the Cog did? Is the Cog working for me? Issues of privacy, privacy-preserving interaction of cogs. Team vs. Personal Cogs – – Training based on best practices vs. personalized instruction Imagine Teams of Cogs working with teams of Human Analysts Symbiosis Issues – 40 What is best for the human to do? What is best for the cog? © 2013 IBM Corporation Open Challenges (2) Teaching the Cog what to do – Learning from demonstration, Learning from documentation – Telling the Cog what to do using natural language – Interactive learning where the Cog may ask questions of the trainer – How does the Cog learn what to do, reliably? – Active learning where the Cog improves over time • Moving up the learning curve (how does Cog understand the goal/desired end state?) • Adapts as the environment (e.g., data sources and formats change) – On what conditions should the Cog report back to the Human? – Task composition (of subtasks) and reuse – Adaptation of past learning to new situation Proactive Action taking – Initiating actions based on learning and incoming requests • E.g., deciding what information sources to search for the request , issuing queries, evaluating responses – Deciding on next steps based on results or whether it needs further guidance from Human Personal knowledge representation and reasoning – Capturing user behavior, interaction in form of personal knowledge – Ability to build knowledge from various structured and unstructured information – AI Principle: expert knows 70,000+/- 20,000 information pieces, and human tasks involves 1010 rules (foundation of AI, 1988) © 2013 IBM Corporation Open Challenges (3) Context understanding, and context-aware interaction – Modeling the world of the person serving, including all context around the work/task, and being able to use the contextual and environmental awareness to proactively and reactively act on behalf of the user Learning to understand the task and plan to do it – Understanding the meaning of tasks, and coming up with a response (e.g.. How many people replied to an invite over email, accepting the offer, without asking the Cog to do so), or suggestions on how to achieve it (based on any new information discovered by the Cog) Cognitive Speech recognition, or other human-computer interfaces for communicating with Cogs – Improving the speech-to-text techniques, and personalized, semantic-enriched speech understanding – Non-speech based approaches for communicating with humans © 2013 IBM Corporation Summary and Conclusions Data is the world’s new natural resource (Ginni Rometti, IBM’s CEO) –New data types and feeds becoming available: personal data marketplace Data is an enabler of machine intelligence – Companies are offering free services to gather more data – Data owners will rule the future of IT! Machine Intelligence and AI is offered as a Service (like other business functionalities) – It has implications for countries and companies that do not own data/services Big data and AI advances also means the need for training skilled workers –Education programs at universities have to be created on big data, analytics and cognitive computing – In the US, there is estimate of the need for thousands of data scientists, and analytics over next 5-10 years Cognitive Computing, and Deep Learning are hot areas of research Natural Language will be the new interface, Cognitive Assistants with NL interfaces with be commonplace in next 5-10 years 43 © 2013 IBM Corporation The call for Defining a National Grand Challenge Choosing a challenge of national importance and visibility Involving big data analytics and AI challenges, that are achievable within a given timeframe Should involves all lifecycle of data acquisition, data pre-processing and transformation, knowledge base building, AI / cognitive computing, and adaptive and deep learning – Should exhibits characteristics of big data, and intelligence Partnership among universities, industry and government agencies 44 © 2013 IBM Corporation Questions? THANK YOU! 45 © 2013 IBM Corporation BACKUP 46 © 2013 IBM Corporation Where Watson Fits Efficient decision support over unstructured (and structured) content Deeper Understanding, Higher Precision and Broader, Timely Coverage at lower costs Shallow Understanding Low Precision Broad Coverage Unstructured Data Broad, rich in context Rapidly growing, current Invaluable yet under utilized 47 Deeper Understanding but Brittle High Precision at High Cost Narrow Limited Coverage Structured Data Precise, explicit Narrow, expensive © 2013 IBM Corporation