Percy Liang
Associate Professor of Computer Science, Senior Fellow at the Stanford Institute for HAI, and Associate Professor, by courtesy, of Statistics
Bio
Percy Liang is an Associate Professor of Computer Science at Stanford University (B.S. from MIT, 2004; Ph.D. from UC Berkeley, 2011) and the director of the Center for Research on Foundation Models (CRFM). He is currently focused on making foundation models (in particular, language models) more accessible through open-source and understandable through rigorous benchmarking. In the past, he has worked on many topics centered on machine learning and natural language processing, including robustness, interpretability, human interaction, learning theory, grounding, semantics, and reasoning. He is also a strong proponent of reproducibility through the creation of CodaLab Worksheets. His awards include the Presidential Early Career Award for Scientists and Engineers (2019), IJCAI Computers and Thought Award (2016), an NSF CAREER Award (2016), a Sloan Research Fellowship (2015), a Microsoft Research Faculty Fellowship (2014), and paper awards at ACL, EMNLP, ICML, COLT, ISMIR, CHI, UIST, and RSS.
Academic Appointments
-
Associate Professor, Computer Science
-
Senior Fellow, Institute for Human-Centered Artificial Intelligence (HAI)
-
Associate Professor (By courtesy), Statistics
Professional Education
-
BS, MIT (2004)
-
MEng, MIT (2005)
-
PhD, UC Berkeley (2011)
2024-25 Courses
- Language Modeling from Scratch
CS 336 (Spr) -
Independent Studies (15)
- Advanced Reading and Research
CS 499 (Aut, Win, Spr) - Advanced Reading and Research
CS 499P (Aut, Win, Spr) - Curricular Practical Training
CS 390A (Aut, Win, Spr) - Curricular Practical Training
CS 390B (Aut, Win, Spr) - Curricular Practical Training
CS 390C (Aut, Win, Spr) - Independent Project
CS 399 (Aut, Win, Spr) - Independent Project
CS 399P (Aut, Win, Spr) - Independent Study
STATS 299 (Aut, Win, Spr) - Independent Work
CS 199 (Aut, Win, Spr) - Independent Work
CS 199P (Aut, Win, Spr) - Industrial Research for Statisticians
STATS 298 (Aut, Win, Spr) - Part-time Curricular Practical Training
CS 390D (Aut, Win, Spr) - Research
STATS 399 (Aut, Win, Spr) - Senior Project
CS 191 (Aut, Win, Spr) - Writing Intensive Senior Research Project
CS 191W (Aut, Win, Spr)
- Advanced Reading and Research
-
Prior Year Courses
2023-24 Courses
- Artificial Intelligence: Principles and Techniques
CS 221 (Aut) - Language Modeling from Scratch
CS 336 (Spr)
2022-23 Courses
- Advances in Foundation Models
CS 324 (Win) - Artificial Intelligence: Principles and Techniques
CS 221 (Aut)
2021-22 Courses
- Artificial Intelligence: Principles and Techniques
CS 221 (Aut) - Understanding and Developing Large Language Models
CS 324 (Win)
- Artificial Intelligence: Principles and Techniques
Stanford Advisees
-
Doctoral Dissertation Reader (AC)
Kawin Ethayarajh, Rohan Taori, Tianyi Zhang -
Postdoctoral Faculty Sponsor
Sarah Cen, Sung Min Park -
Doctoral Dissertation Advisor (AC)
Yann Dubois -
Orals Evaluator
Kawin Ethayarajh, John Hewitt, Rohan Taori -
Master's Program Advisor
Sarah Chen, Ryan Chi, Joshua Francis, Soham Konar, Taran Kota, Rhea Mitr, Patricio Ortiz, Siddharth Sharma, Elena Sierra, Liz Song, Pratham Soni, Marc Soong, Aaron Villanueva, Hong Nhu Ngoc Vo, Karsen Wahal, Andrew Wang, Jason Wang, Jonathan Williams, Kevin Yang, Azure Zhou -
Doctoral Dissertation Co-Advisor (AC)
John Hewitt, Siddharth Karamcheti, Moo Kim, Rohith Kuditipudi, Joon Park -
Doctoral (Program)
Ahmed Ahmed, Rishi Bommasani, Steven Cao, Yann Dubois, Qian Huang, Lisa Li, Ken Liu, Nelson Liu
All Publications
-
Stronger data poisoning attacks break data sanitization defenses
MACHINE LEARNING
2021
View details for DOI 10.1007/s10994-021-06119-y
View details for Web of Science ID 000722108900003
-
WILDS: A Benchmark of in-the-Wild Distribution Shifts
JMLR-JOURNAL MACHINE LEARNING RESEARCH. 2021
View details for Web of Science ID 000683104605062
-
Beyond IID: Three Levels of Generalization for Question Answering on Knowledge Bases
ASSOC COMPUTING MACHINERY. 2021: 3477-3488
View details for DOI 10.1145/3442381.3449992
View details for Web of Science ID 000733621803045
-
Prefix-Tuning: Optimizing Continuous Prompts for Generation
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2021: 4582-4597
View details for Web of Science ID 000698679200153
-
Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
JMLR-JOURNAL MACHINE LEARNING RESEARCH. 2021
View details for Web of Science ID 000683104606087
-
Just Train Twice: Improving Group Robustness without Training Group Information
JMLR-JOURNAL MACHINE LEARNING RESEARCH. 2021
View details for Web of Science ID 000683104606074
-
Catformer: Designing Stable Transformers via Sensitivity Analysis
JMLR-JOURNAL MACHINE LEARNING RESEARCH. 2021
View details for Web of Science ID 000683104602046
-
Robust Encodings: A Framework for Combating Adversarial Typos
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2020: 2752–65
View details for Web of Science ID 000570978203005
-
Concept Bottleneck Models
JMLR-JOURNAL MACHINE LEARNING RESEARCH. 2020
View details for Web of Science ID 000683178505043
-
Understanding Self-Training for Gradual Domain Adaptation
JMLR-JOURNAL MACHINE LEARNING RESEARCH. 2020
View details for Web of Science ID 000683178505055
-
Feature Noise Induces Loss Discrepancy Across Groups
JMLR-JOURNAL MACHINE LEARNING RESEARCH. 2020
View details for Web of Science ID 000683178505031
-
A Tight Analysis of Greedy Yields Subexponential Time Approximation for Uniform Decision Tree
ASSOC COMPUTING MACHINERY. 2020: 102–21
View details for Web of Science ID 000554408100007
-
Enabling Language Models to Fill in the Blanks
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2020: 2492–2501
View details for Web of Science ID 000570978202069
-
ExpBERT: Representation Engineering with Natural Language Explanations
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2020: 2106–13
View details for Web of Science ID 000570978202034
-
Pretraining deep learning molecular representations for property prediction
AMER CHEMICAL SOC. 2019
View details for Web of Science ID 000525055503355
-
Inferring Multidimensional Rates of Aging from Cross-Sectional Data.
Proceedings of machine learning research
2019; 89: 97–107
Abstract
Modeling how individuals evolve over time is a fundamental problem in the natural and social sciences. However, existing datasets are often cross-sectional with each individual observed only once, making it impossible to apply traditional time-series methods. Motivated by the study of human aging, we present an interpretable latent-variable model that learns temporal dynamics from cross-sectional data. Our model represents each individual's features over time as a nonlinear function of a low-dimensional, linearly-evolving latent state. We prove that when this nonlinear function is constrained to be order-isomorphic, the model family is identifiable solely from cross-sectional data provided the distribution of time-independent variation is known. On the UK Biobank human health dataset, our model reconstructs the observed data while learning interpretable rates of aging associated with diseases, mortality, and aging risk factors.
View details for PubMedID 31538144
-
SPoC: Search-based Pseudocode to Code
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2019
View details for Web of Science ID 000535866903051
-
Inferring Multidimensional Rates of Aging from Cross-Sectional Data
MICROTOME PUBLISHING. 2019: 97–107
View details for Web of Science ID 000509687900011
-
Defending against Whitebox Adversarial Attacks via Randomized Discretization
MICROTOME PUBLISHING. 2019: 684–93
View details for Web of Science ID 000509687900071
-
On the Accuracy of Influence Functions for Measuring Group Effects
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2019
View details for Web of Science ID 000534424305027
-
Verified Uncertainty Calibration
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2019
View details for Web of Science ID 000534424303074
-
Unlabeled Data Improves Adversarial Robustness
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2019
View details for Web of Science ID 000535866902078
-
Training Classifiers with Natural Language Explanations.
Proceedings of the conference. Association for Computational Linguistics. Meeting
2018; 2018: 1884–95
Abstract
Training accurate classifiers requires many labels, but each label provides only limited information (one bit for binary classification). In this work, we propose BabbleLabble, a framework for training classifiers in which an annotator provides a natural language explanation for each labeling decision. A semantic parser converts these explanations into programmatic labeling functions that generate noisy labels for an arbitrary amount of unlabeled data, which is used to train a classifier. On three relation extraction tasks, we find that users are able to train classifiers with comparable F1 scores from 5-100* faster by providing explanations instead of just labels. Furthermore, given the inherent imperfection of labeling functions, we find that a simple rule-based semantic parser suffices.
View details for PubMedID 31130772
-
Active Learning of Points-To Specifications
ASSOC COMPUTING MACHINERY. 2018: 678–92
View details for DOI 10.1145/3192366.3192383
View details for Web of Science ID 000452469600046
-
A Retrieve-and-Edit Framework for Predicting Structured Outputs
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2018
View details for Web of Science ID 000461852004059
-
Generalized Binary Search For Split-Neighborly Problems
MICROTOME PUBLISHING. 2018
View details for Web of Science ID 000509385300163
-
Know What You Don't Know: Unanswerable Questions for SQuAD
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2018: 784–89
View details for Web of Science ID 000493913100124
-
Training Classifiers with Natural Language Explanations
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2018: 1884–95
View details for Web of Science ID 000493904300175
-
The price of debiasing automatic metrics in natural language evaluation
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2018: 643–53
View details for Web of Science ID 000493904300060
-
Prediction with a Short Memory
ASSOC COMPUTING MACHINERY. 2018: 1074–87
View details for DOI 10.1145/3188745.3188954
View details for Web of Science ID 000458175600092
-
Uncertainty Sampling is Preconditioned Stochastic Gradient Descent on Zero-One Loss
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2018
View details for Web of Science ID 000461852001049
-
Semidefinite relaxations for certifying robustness to adversarial examples
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2018
View details for Web of Science ID 000461852005046
-
Synthesizing Program Input Grammars
ASSOC COMPUTING MACHINERY. 2017: 95–110
View details for DOI 10.1145/3062341.3062349
View details for Web of Science ID 000414334200007
-
Unsupervised Transformation Learning via Convex Relaxations
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2017
View details for Web of Science ID 000452649406090
-
From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2017: 1051-1062
View details for DOI 10.18653/v1/P17-1097
View details for Web of Science ID 000493984800097
-
Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2017: 1766-1776
View details for DOI 10.18653/v1/P17-1162
View details for Web of Science ID 000493984800162
-
Naturalizing a Programming Language via Interactive Learning
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2017: 929–38
View details for DOI 10.18653/v1/P17-1086
View details for Web of Science ID 000493984800086
-
Certified Defenses for Data Poisoning Attacks
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2017
View details for Web of Science ID 000452649403057
-
Learning Overcomplete HMMs
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2017
View details for Web of Science ID 000452649400090
-
Learning Executable Semantic Parsers for Natural Language Understanding
COMMUNICATIONS OF THE ACM
2016; 59 (9): 68-76
View details for DOI 10.1145/2866568
View details for Web of Science ID 000382671100026
-
Learning Language Games through Interaction
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2016: 2368–78
View details for Web of Science ID 000493806800224
-
How Much is 131 Million Dollars? Putting Numbers in Perspective with Compositional Descriptions
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2016: 578-587
View details for Web of Science ID 000493806800055
-
Data Recombination for Neural Semantic Parsing
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2016: 12-22
View details for Web of Science ID 000493806800002
-
Unsupervised Risk Estimation Using Only Conditional Independence Structure
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2016
View details for Web of Science ID 000458973701058
-
Simpler Context-Dependent Logical Forms via Model Projections
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2016: 1456-1465
View details for Web of Science ID 000493806800138
-
Inferring Logical Forms From Denotations
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2016: 23-32
View details for Web of Science ID 000493806800003
-
Unanimous Prediction for 100% Precision with Application to Learning Semantic Mappings
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2016: 952-962
View details for Web of Science ID 000493806800090
-
Data Augmentation via Levy Processes
PERTURBATIONS, OPTIMIZATION, AND STATISTICS
2016: 343-373
View details for Web of Science ID 000521530900013
-
Tensor Factorization via Matrix Factorization
MICROTOME PUBLISHING. 2015: 507-516
View details for Web of Science ID 000508399700056
-
Learning Where to Sample in Structured Prediction
MICROTOME PUBLISHING. 2015: 875-884
View details for Web of Science ID 000508399700096
-
Environment-Driven Lexicon Induction for High-Level Instructions
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2015: 992-1002
View details for Web of Science ID 000493808900096
-
Building a Semantic Parser Overnight
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2015: 1332-1342
View details for Web of Science ID 000493808900129
-
Compositional Semantic Parsing on Semi-Structured Tables
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2015: 1470-1480
View details for Web of Science ID 000493808900142
-
Learning with Relaxed Supervision
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2015
View details for Web of Science ID 000450913100051
-
Calibrated Structured Prediction
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2015
View details for Web of Science ID 000450913100026
-
Estimating Mixture Models via Mixtures of Polynomials
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2015
View details for Web of Science ID 000450913100070
-
On-the-Job Learning with Bayesian Decision Theory
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2015
View details for Web of Science ID 000450913102009
-
Bringing Machine Learning and Compositional Semantics Together
ANNUAL REVIEW OF LINGUISTICS, VOL 1
2015; 1: 355-376
View details for DOI 10.1146/annurev-linguist-030514-125312
View details for Web of Science ID 000350994000018
-
Linking People in Videos with "Their" Names Using Coreference Resolution
13th European Conference on Computer Vision (ECCV)
SPRINGER INT PUBLISHING AG. 2014: 95–110
View details for Web of Science ID 000345524200007
-
Zero-shot Entity Extraction from Web Pages
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2014: 391-401
View details for Web of Science ID 000493814100037
-
Semantic Parsing via Paraphrasing
ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2014: 1415-1425
View details for DOI 10.3115/v1/p14-1133
View details for Web of Science ID 000493814100133
-
Altitude Training: Strong Bounds for Single-Layer Dropout
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2014
View details for Web of Science ID 000452647102063
-
Simple MAP Inference via Low-Rank Relaxations
NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2014
View details for Web of Science ID 000452647100040
-
Video Event Understanding using Natural Language Descriptions
IEEE International Conference on Computer Vision (ICCV)
IEEE. 2013: 905–912
View details for DOI 10.1109/ICCV.2013.117
View details for Web of Science ID 000351830500113
-
A Data Driven Approach for Algebraic Loop Invariants
22nd European Symposium on Programming (ESOP)
SPRINGER-VERLAG BERLIN. 2013: 574–592
View details for Web of Science ID 000342810200031
- A data driven approach for algebraic loop invariants. 2013
- Spectral experts for estimating mixtures of linear regressions. 2013
- Semantic parsing on Freebase from question-answer pairs. 2013
- Dropout training as adaptive regularization. 2013
- Video event understanding using natural language descriptions. 2013
- Feature noising for log-linear structured prediction. 2013
-
Genome Editing of Human Embryonic Stem Cells and Induced Pluripotent Stem Cells With Zinc Finger Nucleases for Cellular Imaging
CIRCULATION RESEARCH
2012; 111 (12): 1494-?
Abstract
Molecular imaging has proven to be a vital tool in the characterization of stem cell behavior in vivo. However, the integration of reporter genes has typically relied on random integration, a method that is associated with unwanted insertional mutagenesis and positional effects on transgene expression.To address this barrier, we used genome editing with zinc finger nuclease (ZFN) technology to integrate reporter genes into a safe harbor gene locus (PPP1R12C, also known as AAVS1) in the genome of human embryonic stem cells and human induced pluripotent stem cells for molecular imaging.We used ZFN technology to integrate a construct containing monomeric red fluorescent protein, firefly luciferase, and herpes simplex virus thymidine kinase reporter genes driven by a constitutive ubiquitin promoter into a safe harbor locus for fluorescence imaging, bioluminescence imaging, and positron emission tomography imaging, respectively. High efficiency of ZFN-mediated targeted integration was achieved in both human embryonic stem cells and induced pluripotent stem cells. ZFN-edited cells maintained both pluripotency and long-term reporter gene expression. Functionally, we successfully tracked the survival of ZFN-edited human embryonic stem cells and their differentiated cardiomyocytes and endothelial cells in murine models, demonstrating the use of ZFN-edited cells for preclinical studies in regenerative medicine.Our study demonstrates a novel application of ZFN technology to the targeted genetic engineering of human pluripotent stem cells and their progeny for molecular imaging in vitro and in vivo.
View details for DOI 10.1161/CIRCRESAHA.112.274969
View details for Web of Science ID 000311994700042
View details for PubMedID 22967807
View details for PubMedCentralID PMC3518748
-
Induced Pluripotent Stem Cells as a Disease Modeling and Drug Screening Platform
JOURNAL OF CARDIOVASCULAR PHARMACOLOGY
2012; 60 (4): 408-416
Abstract
Induced pluripotent stem cells (iPSCs) hold great hopes for therapeutic application in various diseases. Although ongoing research is dedicated to achieving clinical translation of iPSCs, further understanding of the mechanisms that underlie complex pathogenic conditions is required. Compared with other classical models for studying diseases, iPSCs provide considerable advantages. A newly emerging application of iPSCs is in vitro disease modeling, which can significantly improve the never-ending search for new pharmacological cures. Here, we will discuss current efforts to create iPSC-dependent patient-specific disease models. Furthermore, we will review the use of iPSCs for development and testing of new therapeutic agents and the implications for high-throughput drug screening.
View details for DOI 10.1097/FJC.0b013e318247f642
View details for Web of Science ID 000309977900012
View details for PubMedID 22240913
View details for PubMedCentralID PMC3343213
-
Modeling Pathogenesis in Familial Hypertrophic Cardiomyopathy Using Patient-Specific Induced Pluripotent Stem Cells
Basic Cardiovascular Sciences Scientific Session
LIPPINCOTT WILLIAMS & WILKINS. 2012
View details for Web of Science ID 000312506400056
- Identifiability and unmixing of latent parse trees. 2012
- Learning dependency-based compositional semantics. 2011
- Learning minimal abstractions. 2011
- Scaling up abstraction refinement via pruning. 2011
- A dynamic evaluation of static heap abstractions. 2010
- Learning programs: a hierarchical Bayesian approach. 2010
- A game-theoretic approach to generating spatial descriptions. 2010
- Type-based MCMC. 2010
- A simple domain-independent probabilistic approach to generation. 2010
- On the interaction between norm and dimensionality: multiple regimes in learning. 2010
- Learning from measurements in exponential families. 2009
- Learning semantic correspondences with less supervision. 2009
- Probabilistic grammars and hierarchical Dirichlet processes. The Oxford Handbook of Applied Bayesian Analysis 2009
- Online EM for unsupervised models. 2009
- Asymptotically optimal regularization in smooth parametric models. 2009
-
Optimal team size and monitoring in organizations
ACCOUNTING REVIEW
2008; 83 (3): 789-822
View details for Web of Science ID 000256277400008
- A probabilistic approach to language change. 2008
- Agreement-based learning. 2008
- Learning bilingual lexicons from monolingual corpora. 2008
- Structure compilation: trading structure for features. 2008
- Analyzing the errors of unsupervised learning. 2008
- An asymptotic analysis of generative, discriminative, and pseudolikelihood estimators. 2008
- A probabilistic approach to diachronic phonology. 2007
- A permutation-augmented sampler for Dirichlet process mixture models. 2007
- Structured Bayesian nonparametric models with variational inference (tutorial). 2007
- The infinite PCFG using hierarchical Dirichlet processes. 2007
- An end-to-end discriminative approach to machine translation. 2006
- Alignment by agreement. 2006
- Linear programming in bounded tree-width Markov networks. 2005
- A data structure for maintaining acyclicity in hypergraphs. Massachusetts Institute of Technology Technical Report 2005
- Efficient geometric algorithms for parsing in two dimensions. 2005
- Methods and experiments with bounded tree-width Markov networks. Massachusetts Institute of Technology Technical Report 2004
- How much of a hypertree can be captured by windmills? Massachusetts Institute of Technology Technical Report 2003
-
INTERFEROMETRIC STUDIES OF THE JOVIAN ATMOSPHERIC PROBE FIELD
AMER INST PHYSICS. 1980: 1093–94
View details for Web of Science ID A1980KP44100161
-
Saponins and sapogenins. III. The sapogenins obtained from chlorogalum pomeridianum
JOURNAL OF THE AMERICAN CHEMICAL SOCIETY
1935; 57 (1): 525-527
View details for Web of Science ID 000188361300171