Bio


Michael Bernstein is an Associate Professor of Computer Science at Stanford University, where he is a Bass University Fellow and Interim Director of the Symbolic Systems program. His research focuses on designing social, societal, and interactive technologies. This research has been reported in venues such as The New York Times, Wired, Science, and Nature. Michael has been recognized with an Alfred P. Sloan Fellowship, the UIST Lasting Impact Award, and the Computer History Museum's Patrick J. McGovern Tech for Humanity Prize. He holds a bachelor's degree in Symbolic Systems from Stanford University, as well as a master's degree and a Ph.D. in Computer Science from MIT.

Academic Appointments


Honors & Awards


  • Bass University Fellow, Stanford University
  • Patrick J. McGovern Tech for Humanity Prize, Computer History Museum
  • Lasting Impact Award, UIST
  • CASBS Fellow, Center for Advanced Study in the Behavioral Sciences
  • Teaching Honor Roll, Tau Beta Pi
  • STMicroelectronics Faculty Scholar, Stanford University
  • Sloan Research Fellowship, Sloan Foundation
  • NSF CAREER award, National Science Foundation
  • Best paper award, CHI 2019, CHI 2017, CSCW 2017, CHI 2016, UIST 2014, ICWSM 2011, UIST 2010, UIST 2006
  • Outstanding Academic Title, "Handbook of Collective Intelligence", American Library Association, Choice
  • Robert N. Noyce Family Faculty Scholar, Stanford University
  • George M. Sprowls Award for best doctoral thesis in Computer Science, MIT

Program Affiliations


  • Symbolic Systems Program

Professional Education


  • PhD, MIT, Computer Science (2012)
  • SM, MIT, Computer Science (2008)
  • BS, Stanford University, Symbolic Systems (2006)

2024-25 Courses


Stanford Advisees


All Publications


  • Social and moral psychology of COVID-19 across 69 countries. Scientific data Azevedo, F., Pavlović, T., Rêgo, G. G., Ay, F. C., Gjoneska, B., Etienne, T. W., Ross, R. M., Schönegger, P., Riaño-Moreno, J. C., Cichocka, A., Capraro, V., Cian, L., Longoni, C., Chan, H. F., Van Bavel, J. J., Sjåstad, H., Nezlek, J. B., Alfano, M., Gelfand, M. J., Birtel, M. D., Cislak, A., Lockwood, P. L., Abts, K., Agadullina, E., Aruta, J. J., Besharati, S. N., Bor, A., Choma, B. L., Crabtree, C. D., Cunningham, W. A., De, K., Ejaz, W., Elbaek, C. T., Findor, A., Flichtentrei, D., Franc, R., Gruber, J., Gualda, E., Horiuchi, Y., Huynh, T. L., Ibanez, A., Imran, M. A., Israelashvili, J., Jasko, K., Kantorowicz, J., Kantorowicz-Reznichenko, E., Krouwel, A., Laakasuo, M., Lamm, C., Leygue, C., Lin, M. J., Mansoor, M. S., Marie, A., Mayiwar, L., Mazepus, H., McHugh, C., Minda, J. P., Mitkidis, P., Olsson, A., Otterbring, T., Packer, D. J., Perry, A., Petersen, M. B., Puthillam, A., Rothmund, T., Santamaría-García, H., Schmid, P. C., Stoyanov, D., Tewari, S., Todosijević, B., Tsakiris, M., Tung, H. H., Umbres, R. G., Vanags, E., Vlasceanu, M., Vonasch, A., Yucel, M., Zhang, Y., Abad, M., Adler, E., Akrawi, N., Mdarhri, H. A., Amara, H., Amodio, D. M., Antazo, B. G., Apps, M., Ba, M. H., Barbosa, S., Bastian, B., Berg, A., Bernal-Zárate, M. P., Bernstein, M., Białek, M., Bilancini, E., Bogatyreva, N., Boncinelli, L., Booth, J. E., Borau, S., Buchel, O., Cameron, C. D., Carvalho, C. F., Celadin, T., Cerami, C., Chalise, H. N., Cheng, X., Cockcroft, K., Conway, J., Córdoba-Delgado, M. A., Crespi, C., Crouzevialle, M., Cutler, J., Cypryańska, M., Dabrowska, J., Daniels, M. A., Davis, V. H., Dayley, P. N., Delouvée, S., Denkovski, O., Dezecache, G., Dhaliwal, N. A., Diato, A. B., Di Paolo, R., Drosinou, M., Dulleck, U., Ekmanis, J., Ertan, A. S., Farhana, H. H., Farkhari, F., Farmer, H., Fenwick, A., Fidanovski, K., Flew, T., Fraser, S., Frempong, R. B., Fugelsang, J. A., Gale, J., Garcia-Navarro, E. B., Garladinne, P., Ghajjou, O., Gkinopoulos, T., Gray, K., Griffin, S. M., Gronfeldt, B., Gümren, M., Gurung, R. L., Halperin, E., Harris, E., Herzon, V., Hruška, M., Huang, G., Hudecek, M. F., Isler, O., Jangard, S., Jorgensen, F. J., Kachanoff, F., Kahn, J., Dangol, A. K., Keudel, O., Koppel, L., Koverola, M., Kubin, E., Kunnari, A., Kutiyski, Y., Laguna, O. M., Leota, J., Lermer, E., Levy, J., Levy, N., Li, C., Long, E. U., Maglić, M., McCashin, D., Metcalf, A. L., Mikloušić, I., El Mimouni, S., Miura, A., Molina-Paredes, J., Monroy-Fonseca, C., Morales-Marente, E., Moreau, D., Muda, R., Myer, A., Nash, K., Nesh-Nash, T., Nitschke, J. P., Nurse, M. S., Ohtsubo, Y., de Mello, V. O., O'Madagain, C., Onderco, M., Palacios-Galvez, M. S., Palomöki, J., Pan, Y., Papp, Z., Pärnamets, P., Paruzel-Czachura, M., Pavlović, Z., Payán-Gómez, C., Perander, S., Pitman, M. M., Prasad, R., Pyrkosz-Pacyna, J., Rathje, S., Raza, A., Rhee, K., Robertson, C. E., Rodríguez-Pascual, I., Saikkonen, T., Salvador-Ginez, O., Santi, G. C., Santiago-Tovar, N., Savage, D., Scheffer, J. A., Schultner, D. T., Schutte, E. M., Scott, A., Sharma, M., Sharma, P., Skali, A., Stadelmann, D., Stafford, C. A., Stanojević, D., Stefaniak, A., Sternisko, A., Stoica, A., Stoyanova, K. K., Strickland, B., Sundvall, J., Thomas, J. P., Tinghög, G., Torgler, B., Traast, I. J., Tucciarelli, R., Tyrala, M., Ungson, N. D., Uysal, M. S., Van Lange, P. A., van Prooijen, J. W., van Rooy, D., Västfjäll, D., Verkoeijen, P., Vieira, J. B., von Sikorski, C., Walker, A. C., Watermeyer, J., Wetter, E., Whillans, A., White, K., Habib, R., Willardt, R., Wohl, M. J., Wójcik, A. D., Wu, K., Yamada, Y., Yilmaz, O., Yogeeswaran, K., Ziemer, C. T., Zwaan, R. A., Boggio, P. S., Sampaio, W. M. 2023; 10 (1): 272

    Abstract

    The COVID-19 pandemic has affected all domains of human life, including the economic and social fabric of societies. One of the central strategies for managing public health throughout the pandemic has been through persuasive messaging and collective behaviour change. To help scholars better understand the social and moral psychology behind public health behaviour, we present a dataset comprising of 51,404 individuals from 69 countries. This dataset was collected for the International Collaboration on Social & Moral Psychology of COVID-19 project (ICSMP COVID-19). This social science survey invited participants around the world to complete a series of moral and psychological measures and public health attitudes about COVID-19 during an early phase of the COVID-19 pandemic (between April and June 2020). The survey included seven broad categories of questions: COVID-19 beliefs and compliance behaviours; identity and social attitudes; ideology; health and well-being; moral beliefs and motivation; personality traits; and demographic variables. We report both raw and cleaned data, along with all survey materials, data visualisations, and psychometric evaluations of key variables.

    View details for DOI 10.1038/s41597-023-02080-8

    View details for PubMedID 37169799

    View details for PubMedCentralID PMC10173241

  • COVID-19 non-pharmaceutical interventions: data annotation for rapidly changing local policy information. Scientific data Hurt, B., Hoque, O. B., Mokrzycki, F., Mathew, A., Xue, M., Gabitsinashvili, L., Mokrzycki, H., Fischer, R., Telesca, N., Xue, L. A., Ritchie, J., Zamfirescu-Pereira, J. D., Bernstein, M., Whiting, M., Marathe, M. 2023; 10 (1): 126

    Abstract

    Understanding the scope, prevalence, and impact of the COVID-19 pandemic response will be a rich ground for research for many years. Key to the response to COVID-19was the non-pharmaceutical intervention (NPI) measures, such as mask mandates or stay-in-place orders. For future pandemic preparedness, it is critical to understand the impact and scope of these interventions. Given the ongoing nature of the pandemic, existing NPI studies covering only the initial portion provide only a narrow view of the impact of NPI measures. This paper describes a dataset of NPI measures taken by counties in the U.S. state of Virginia that include measures taken over the first two years of the pandemic beginningin March 2020. This data enables analyses of NPI measures over a long time period that can produce impact analyses on both the individual NPI effectiveness in slowing the pandemic spread,and the impact of various NPI measures on the behavior and conditions of the different counties and state.

    View details for DOI 10.1038/s41597-023-01979-6

    View details for PubMedID 36894597

  • On Second Thought, Let′s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning Shaikh, O., Zhang, H., Held, W., Bernstein, M., Yang, D., Rogers, A., Boyd-Graber, J., Okazaki, N. ASSOC COMPUTATIONAL LINGUISTICS-ACL. 2023: 4454-4470
  • Breaking Out of the Ivory Tower: A Large-scale Analysis of Patent Citations to HCI Research Cao, H., Lu, Y., Deng, Y., McFarland, D. A., Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2023
  • Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design Lam, M. S., Ma, Z., Li, A., Freitas, I., Wang, D., Landay, J. A., Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2023
  • Architecting Novel Interactions With Generative AI Models Bernstein, M. S., Park, J., Morris, M., Amershi, S., Chilton, L., Gordon, M. L., ACM ASSOC COMPUTING MACHINERY. 2023
  • Socially situated artificial intelligence enables learning from human interaction. Proceedings of the National Academy of Sciences of the United States of America Krishna, R., Lee, D., Fei-Fei, L., Bernstein, M. S. 2022; 119 (39): e2115730119

    Abstract

    Regardless of how much data artificial intelligence agents have available, agents will inevitably encounter previously unseen situations in real-world deployments. Reacting to novel situations by acquiring new information from other people-socially situated learning-is a core faculty of human development. Unfortunately, socially situated learning remains an open challenge for artificial intelligence agents because they must learn how to interact with people to seek out the information that they lack. In this article, we formalize the task of socially situated artificial intelligence-agents that seek out new information through social interactions with people-as a reinforcement learning problem where the agent learns to identify meaningful and informative questions via rewards observed through social interaction. We manifest our framework as an interactive agent that learns how to ask natural language questions about photos as it broadens its visual intelligence on a large photo-sharing social network. Unlike active-learning methods, which implicitly assume that humans are oracles willing to answer any question, our agent adapts its behavior based on observed norms of which questions people are or are not interested to answer. Through an 8-mo deployment where our agent interacted with 236,000 social media users, our agent improved its performance at recognizing new visual information by 112%. A controlled field experiment confirmed that our agent outperformed an active-learning baseline by 25.6%. This work advances opportunities for continuously improving artificial intelligence (AI) agents that better respect norms in open social environments.

    View details for DOI 10.1073/pnas.2115730119

    View details for PubMedID 36122244

  • Social Simulacra: Creating Populated Prototypes for Social Computing Systems Park, J., Popowski, L., Cai, C. J., Morris, M., Liang, P., Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2022
  • Jury Learning: Integrating Dissenting Voices into Machine Learning Models Gordon, M. L., Lam, M. S., Park, J., Patel, K., Hancock, J. T., Hashimoto, T., Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2022
  • Ethics and society review: Ethics reflection as a precondition to research funding. Proceedings of the National Academy of Sciences of the United States of America Bernstein, M. S., Levi, M., Magnus, D., Rajala, B. A., Satz, D., Waeiss, C. 1800; 118 (52)

    Abstract

    Researchers in areas as diverse as computer science and political science must increasingly navigate the possible risks of their research to society. However, the history of medical experiments on vulnerable individuals influenced many research ethics reviews to focus exclusively on risks to human subjects rather than risks to human society. We describe an Ethics and Society Review board (ESR), which fills this moral gap by facilitating ethical and societal reflection as a requirement to access grant funding: Researchers cannot receive grant funding from participating programs until the researchers complete the ESR process for their proposal. Researchers author an initial statement describing their proposed research's risks to society, subgroups within society, and globally and commit to mitigation strategies for these risks. An interdisciplinary faculty panel iterates with the researchers to refine these risks and mitigation strategies. We describe a mixed-method evaluation of the ESR over 1 y, in partnership with a large artificial intelligence grant program at our university. Surveys and interviews of researchers who interacted with the ESR found 100% (95% CI: 87 to 100%) were willing to continue submitting future projects to the ESR, and 58% (95% CI: 37 to 77%) felt that it had influenced the design of their research project. The ESR panel most commonly identified issues of harms to minority groups, inclusion of diverse stakeholders in the research plan, dual use, and representation in datasets. These principles, paired with possible mitigation strategies, offer scaffolding for future research designs.

    View details for DOI 10.1073/pnas.2117261118

    View details for PubMedID 34934006

  • Understanding the Representation and Representativeness of Age in AI Data Sets Park, J., Bernstein, M. S., Brewer, R. N., Kamar, E., Morris, M., ASSOC COMP MACHINERY ASSOC COMPUTING MACHINERY. 2021: 834-842
  • Scene Graph Prediction with Limited Labels. Proceedings. IEEE International Conference on Computer Vision Chen, V. S., Varma, P. n., Krishna, R. n., Bernstein, M. n., Ré, C. n., Fei-Fei, L. n. 2020; 2019: 2580–90

    Abstract

    Visual knowledge bases such as Visual Genome power numerous applications in computer vision, including visual question answering and captioning, but suffer from sparse, incomplete relationships. All scene graph models to date are limited to training on a small set of visual relationships that have thousands of training labels each. Hiring human annotators is expensive, and using textual knowledge base completion methods are incompatible with visual data. In this paper, we introduce a semi-supervised method that assigns probabilistic relationship labels to a large number of unlabeled images using few' labeled examples. We analyze visual relationships to suggest two types of image-agnostic features that are used to generate noisy heuristics, whose outputs are aggregated using a factor graph-based generative model. With as few as 10 labeled examples per relationship, the generative model creates enough training data to train any existing state-of-the-art scene graph model. We demonstrate that our method outperforms all baseline approaches on scene graph prediction by 5.16 recall@ 100 for PREDCLS. In our limited label setting, we define a complexity metric for relationships that serves as an indicator (R2 = 0.778) for conditions under which our method succeeds over transfer learning, the de-facto approach for training with limited labels.

    View details for DOI 10.1109/iccv.2019.00267

    View details for PubMedID 32218709

    View details for PubMedCentralID PMC7098690

  • Information Maximizing Visual Question Generation Krishna, R., Bernstein, M., Li Fei-Fei, IEEE Comp Soc IEEE COMPUTER SOC. 2019: 2008–18
  • Eevee: Transforming Images by Bridging High-level Goals and Low-level Edit Operations Lam, M. S., Young, G. B., Xu, C. Y., Krishna, R., Bernstein, M. S., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2019
  • Scene Graph Prediction with Limited Labels Chen, V. S., Varma, P., Krishna, R., Bernstein, M., Re, C., Fei-Fei, L., IEEE IEEE COMPUTER SOC. 2019: 1772–82
  • Conservation of Procrastination: Do Productivity Interventions Save Time or Just Redistribute It? Kovacs, G., Gregory, D., Ma, Z., Wu, Z., Emami, G., Ray, J., Bernstein, M. S., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2019
  • Street-Level Algorithms: A Theory at the Gaps Between Policy and Decisions Alkhatib, A., Bernstein, M., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2019
  • HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models Zhou, S., Gordon, M. L., Krishna, R., Narcomey, A., Li Fei-Fei, Bernstein, M. S., Wallach, H., Larochelle, H., Beygelzimer, A., d'Alche-Buc, F., Fox, E., Garnett, R. NEURAL INFORMATION PROCESSING SYSTEMS (NIPS). 2019
  • Visual Relationships as Functions: Enabling Few-Shot Scene Graph Prediction Dornadula, A., Narcomey, A., Krishna, R., Bernstein, M., Li Fei-Fei, IEEE IEEE COMPUTER SOC. 2019: 1730–39
  • Ink: Increasing Worker Agency to Reduce Friction in Hiring Crowd Workers ACM TRANSACTIONS ON COMPUTER-HUMAN INTERACTION Salehi, N., Bernstein, M. S. 2018; 25 (2)

    View details for DOI 10.1145/3177882

    View details for Web of Science ID 000431849700003

  • Engagement Learning: Expanding Visual Knowledge by Engaging Online Participants Krishna, R., Lee, D., Li, F., Bernstein, M., ACM ASSOC COMPUTING MACHINERY. 2018: 87–89
  • In Search of the Dream Team: Temporally Constrained Multi-Armed Bandits for Identifying Effective Team Structures Zhou, S., Valentine, M., Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2018
  • Iris: A Conversational Agent for Complex Tasks Fast, E., Chen, B., Mendelsohn, J., Bassen, J., Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2018
  • Shared Autonomy for an Interactive AI System Zhou, S., Mu, T., Goel, K., Bernstein, M., Brunskill, E., ACM ASSOC COMPUTING MACHINERY. 2018: 20–22
  • Mechanical Novel: Crowdsourcing Complex Work Through Reflection and Revision DESIGN THINKING RESEARCH: MAKING DISTINCTIONS: COLLABORATION VERSUS COOPERATION Kim, J., Sterman, S., Cohen, A., Bernstein, M. S., Plattner, H., Meinel, C., Leifer, L. 2018: 79–104
  • Mosaic: Designing Online Creative Communities for Sharing Works-in-Progress DESIGN THINKING RESEARCH: MAKING DISTINCTIONS: COLLABORATION VERSUS COOPERATION Kim, J., Agrawala, M., Bernstein, M. S., Plattner, H., Meinel, C., Leifer, L. 2018: 105–29
  • Referring Relationships Krishna, R., Chami, I., Bernstein, M., Li Fei-Fei, IEEE IEEE. 2018: 6867–76
  • Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations INTERNATIONAL JOURNAL OF COMPUTER VISION Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L., Shamma, D. A., Bernstein, M. S., Li Fei-Fei, F. F. 2017; 123 (1): 32-73
  • Anyone Can Become a Troll AMERICAN SCIENTIST Cheng, J., Danescu-Niculescu-Mizil, C., Leskovec, J., Bernstein, M. 2017; 105 (3): 152–55
  • Anyone Can Become a Troll: Causes of Trolling Behavior in Online Discussions Cheng, J., Bernstein, M., Danescu-Niculescu-Mizil, C., Leskovec, J., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2017: 1217–30

    Abstract

    In online communities, antisocial behavior such as trolling disrupts constructive discussion. While prior work suggests that trolling behavior is confined to a vocal and antisocial minority, we demonstrate that ordinary people can engage in such behavior as well. We propose two primary trigger mechanisms: the individual's mood, and the surrounding context of a discussion (e.g., exposure to prior trolling behavior). Through an experiment simulating an online discussion, we find that both negative mood and seeing troll posts by others significantly increases the probability of a user trolling, and together double this probability. To support and extend these results, we study how these same mechanisms play out in the wild via a data-driven, longitudinal analysis of a large online news discussion community. This analysis reveals temporal mood effects, and explores long range patterns of repeated exposure to trolling. A predictive model of trolling behavior shows that mood and discussion context together can explain trolling behavior better than an individual's history of trolling. These results combine to suggest that ordinary people can, under the right circumstances, behave like trolls.

    View details for PubMedID 29399664

  • Huddler: Convening Stable and Familiar Crowd Teams Despite Unpredictable Availability Salehi, N., McCabe, A., Valentine, M., Bernstein, M. S., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2017: 1700-1713
  • The Daemo Crowdsourcing Marketplace Gaikwad, S. S., Whiting, M. E., Gamage, D., Mullings, C. A., Majeti, D., Goyal, S., Gilbee, A., Chhibber, N., Ginzberg, A., Richmond-Fuller, A., Matin, S., Sehgal, V., Sarma, T., Nasser, A., Ballav, A., Regino, J., Zhou, S., Mananova, K., Srinivas, P., Ziulkoski, K., Dhakal, D., Stolzoff, A., Niranga, S. S., Salih, M., Sinha, A., Vaish, R., Bernstein, M. S., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2017: 1-4
  • Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms Whiting, M. E., Gamage, D., Gaikwad, S. S., Gilbee, A., Goyal, S., Ballav, A., Majeti, D., Chhibber, N., Richmond-Fuller, A., Vargus, F., Sarma, T., Chandrakanthan, V., Moura, T., Salih, M., Kalejaiye, G., Ginzberg, A., Mullings, C. A., Dayan, Y., Milland, K., Orefice, H., Regino, J., Parsi, S., Mainali, K., Sehgal, V., Matin, S., Sinha, A., Vaish, R., Bernstein, M. S., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2017: 1902-1913
  • Founder Center: Enabling Access to Collective Social Capital Kokkalis, N., Fan, C., Breier, T., Bernstein, M. S., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2017: 2010-2022
  • Crowd Research: Open and Scalable University Laboratories Vaish, R., Gaikwad, S. S., Kovacs, G., Veit, A., Krishna, R., Ibarra, I., Simoiu, C., Wilber, M., Belongie, S., Goel, S., Davis, J., Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2017: 829-843
  • Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revision Kim, J., Sterman, S., Cohen, A., Bernstein, M. S., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2017: 233–45
  • Mosaic: Designing Online Creative Communities for Sharing Works-in-Progress Kim, J., Agrawala, M., Bernstein, M. S., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2017: 246–58
  • A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality Hata, K., Krishna, R., Fei-Fei, L., Bernstein, M. S., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2017: 889–901
  • Examining Crowd Work and Gig Work Through The Historical Lens of Piecework Alkhatib, A., Bernstein, M. S., Levi, M., ACM ASSOC COMPUTING MACHINERY. 2017: 4599–4616
  • MyriadHub: Efficiently Scaling Personalized Email Conversations with Valet Crowdsourcing Kokkalis, N., Fan, C., Roith, J., Bernstein, M. S., Klemmer, S., ACM ASSOC COMPUTING MACHINERY. 2017: 73–84
  • Visual7W: Grounded Question Answering in Images Zhu, Y., Groth, O., Bernstein, M., Li Fei-Fei, IEEE IEEE. 2016: 4995–5004
  • Talkabout: Making Distance Matter with Small Groups in Massive Classes DESIGN THINKING RESEARCH: MAKING DESIGN THINKING FOUNDATIONAL Kulkarni, C., Cambre, J., Kotturi, Y., Bernstein, M. S., Klemmer, S., Plattner, H., Meinel, C., Leifer, L. 2016: 67-92
  • Meta: Enabling Programming Languages to Learn from the Crowd Fast, E., Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2016: 259-270
  • Boomerang: Rebounding the Consequences of Reputation Feedback on Crowdsourcing Platforms Gaikwad, S. S., Morina, D., Ginzberg, A., Mullings, C., Goyal, S., Gamage, D., Diemert, C., Burton, M., Zhou, S., Whiting, M., Ziulkoski, K., Ballav, A., Gilbee, A., Niranga, S. S., Sehgal, V., Lin, J., Kristianto, L., Richmond-Fuller, A., Regino, J., Chhibber, N., Majeti, D., Sharma, S., Mananova, K., Dhaka, D., Dai, W., Purynova, V., Sandeep, S., Chandrakanthan, V., Sarma, T., Matin, S., Nasser, A., Nistala, R., Stolzoff, A., Milland, K., Mathur, V., Vaish, R., Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2016: 625-637
  • Augur: Mining Human Behaviors from Fiction to Power Interactive Systems Fast, E., McGrath, W., Rajpurkar, P., Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2016: 237-247
  • Pay It Backward: Per-Task Payments on Crowdsourcing Platforms Reduce Productivity Ikeda, K., Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2016: 4111-4121
  • Empath: Understanding Topic Signals in Large-Scale Text Fast, E., Chen, B., Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2016: 4647-4657
  • Designing Scalable and Sustainable Peer Interactions Online DESIGN THINKING RESEARCH: TAKING BREAKTHROUGH INNOVATION HOME Kulkarni, C., Kotturi, Y., Bernstein, M. S., Klemmer, S., Plattner, H., Meinel, C., Leifer, L. 2016: 237-273
  • Atelier: Repurposing Expert Crowdsourcing Tasks as Micro-internships Suzuki, R., Salehi, N., Lam, M. S., Marroquin, J. C., Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2016: 2645–56
  • Visual Relationship Detection with Language Priors Lu, C., Krishna, R., Bernstein, M., Li Fei-Fei, Leibe, B., Matas, J., Sebe, N., Welling, M. SPRINGER INTERNATIONAL PUBLISHING AG. 2016: 852-869
  • Embracing Error to Enable Rapid Crowdsourcing Krishna, R., Hata, K., Chen, S., Kravitz, J., Shamma, D. A., Li Fei-Fei, Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2016: 3167-3179
  • ImageNet Large Scale Visual Recognition Challenge INTERNATIONAL JOURNAL OF COMPUTER VISION Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., Fei-fei, L. 2015; 115 (3): 211-252
  • Soylent: A Word Processor with a Crowd Inside COMMUNICATIONS OF THE ACM Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., Panovich, K. 2015; 58 (8): 85-94
  • Image Retrieval using Scene Graphs Johnson, J., Krishna, R., Stark, M., Li, L., Shamma, D. A., Bernstein, M. S., Li Fei-Fei, IEEE IEEE. 2015: 3668–78
  • Human-Computer Interaction and Collective Intelligence HANDBOOK OF COLLECTIVE INTELLIGENCE Bigham, J. P., Bernstein, M. S., Adar, E., Malone, T. W., Bernstein, M. S. 2015: 57-83
  • Flock: Hybrid Crowd-Machine Learning Classifiers Cheng, J., Bernstein, M. S., ACM ASSOC COMPUTING MACHINERY. 2015: 600-611
  • Talkabout: Making Distance Matter with Small Groups in Massive Classes Kulkarni, C., Cambre, J., Kotturi, Y., Bernstein, M. S., Klemmer, S., ACM ASSOC COMPUTING MACHINERY. 2015: 1116-1128
  • Handbook of Collective Intelligence Introduction HANDBOOK OF COLLECTIVE INTELLIGENCE Malone, T. W., Bernstein, M. S., Malone, T. W., Bernstein, M. S. 2015: 1-13
  • Measuring Crowdsourcing Effort with Error-Time Curves Cheng, J., Teevan, J., Bernstein, M. S., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2015: 1365-1374
  • We Are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers Salehi, N., Irani, L. C., Bernstein, M. S., Alkhatib, A., Ogbe, E., Milland, K., Clickhappier, Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2015: 1621-1630
  • Break It Down: A Comparison of Macro- and Microtasks Cheng, J., Teevan, J., Iqbal, S. T., Bernstein, M. S., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2015: 4061-4064
  • Motif: Supporting Novice Creativity through Expert Patterns Kim, J., Dontcheva, M., Li, W., Bernstein, M. S., Steinsapir, D., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2015: 1211–20
  • Learning Perceptual Kernels for Visualization Design IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS Demiralp, C., Bernstein, M. S., Heer, J. 2014; 20 (12): 1933-1942

    Abstract

    Visualization design can benefit from careful consideration of perception, as different assignments of visual encoding variables such as color, shape and size affect how viewers interpret data. In this work, we introduce perceptual kernels: distance matrices derived from aggregate perceptual judgments. Perceptual kernels represent perceptual differences between and within visual variables in a reusable form that is directly applicable to visualization evaluation and automated design. We report results from crowd-sourced experiments to estimate kernels for color, shape, size and combinations thereof. We analyze kernels estimated using five different judgment types--including Likert ratings among pairs, ordinal triplet comparisons, and manual spatial arrangement--and compare them to existing perceptual models. We derive recommendations for collecting perceptual similarities, and then demonstrate how the resulting kernels can be applied to automate visualization design decisions.

    View details for DOI 10.1109/TVCG.2014.2346978

    View details for Web of Science ID 000344991700038

  • Catalyst: Triggering Collective Action with Thresholds Cheng, J., Bernstein, M. 2014
  • Designing and Deploying Online Field Experiments Bakshy, E., Eckles, D., Bernstein, M. S., Assoc Comp Machinery ASSOC COMPUTING MACHINERY. 2014: 283-292
  • Ensemble: Exploring Complementary Strengths of Leaders and Crowds in Creative Collaboration Kim, J., Cheng, J., Bernstein, M. 2014
  • Crowd-Powered Systems KUNSTLICHE INTELLIGENZ Bernstein, M. S. 2013; 27 (1): 69-73
  • Quantifying the Invisible Audience in Social Networks Bernstein, Michael, S., Bakshy, E., Burke, M., Karrer, B. 2013
  • EmailValet: Managing Email Overload through Private, Accountable Crowdsourcing Kokkalis, N., Köhn, T., Pfeiffer, C., Chornyi, D., Bernstein, Michael, S., Klemmer, Scott, R. 2013
  • Crowd-scale Interactive Formal Reasoning and Analytics Fast, E., Lee, C., Aiken, A., Bernstein, M., Koller, D., Smith, E. 2013
  • The Future of Crowd Work Kittur, A., Nickerson, Jeffrey, V., Bernstein, Michael, S., Gerber, Elizabeth, M., Shaw, A., Zimmerman, J. 2013
  • Leveraging Online Populations for Crowdsourcing IEEE INTERNET COMPUTING Chi, E. H., Bernstein, M. S. 2012; 16 (5): 10-12
  • Who Gives A Tweet? Evaluating Microblog Content Value Andre, P., Bernstein, M., Luther, K. 2012
  • Direct Answers for Search Queries in the Long Tail Bernstein, M., Teevan, J., Dumais, S., Liebling, D., Horvitz, E. 2012
  • Analytic Methods for Optimizing Realtime Crowdsourcing CI: Collective Intelligence 2012 Bernstein, M., Karger, D., Miller, R., Brandt, J. 2012
  • The Trouble with Social Computing Systems Research Bernstein, M., Ackerman, M., Chi, Ed, H., Miller, R. 2011
  • Crowds in Two Seconds: Enabling Realtime Crowd-Powered Interfaces Bernstein, M., Brandt, J., Miller, R., Karger, D. 2011
  • PingPong++: Community Customization in Games and Entertainment Xiao, X., Bernstein, M., Yao, L., Lakatos, D., Gust, L., Acquah, K. 2011
  • TwitInfo: Aggregating and Visualizing Microblogs for Event Exploration Marcus, A., Bernstein, M., Badar, O., Karger, D., Madden, S., Miller, R. 2011
  • 4chan and /b/: An Analysis of Anonymity and Ephemerality in a Large Online Community Bernstein, M., Monroy-Hernandez, A., Harry, D., Andre, P., Panovich, K., Vargas, G. 2011
  • Eddi: Interactive Topic-Based Browsing of Social Status Streams Bernstein, M., Suh, B., Hong, L., Chen, J., Kairam, S., Chi, Ed, H. 2010
  • Short and Tweet: Experiments on Recommending Content from Information Streams Chen, J., Nairn, R., Nelson, L., Bernstein, M., Chi, E. 2010
  • Personalization via Friendsourcing ACM Transactions on Computer-Human Interaction 2010 Bernstein, M., Tan, D., Smith, G., Czerwinski, M., Horvitz, E. 2010
  • Who Am I? Two-Four-Six-Oh-One! Bernstein, M., Marcus, A., Karger, D., Miller, R. 2010
  • Enhancing Directed Content Sharing on the Web Bernstein, M., Marcus, A., Karger, D., Miller, R. 2010
  • A Torrent of Tweets: Managing Information Overload in Online Social Streams Bernstein, M., Kairam, S., Suh, B., Hong, L., Chi, Ed, H. 2010
  • Soylent: A Word Processor with a Crowd Inside Bernstein, M., Little, G., Miller, R., Hartmann, B., Ackerman, M., Karger, D. 2010
  • Collabio: A Game for Annotating People within Social Networks Bernstein, M., Tan, D., Smith, G., Czerwinski, M., Horvitz, E. 2009
  • Note to Self: Examining Personal Information Keeping in a Lightweight Note-Taking Tool Van Kleek, M., Bernstein, M., Panovich, K., Vargas, G., Karger, D., schraefel, m. c. 2009
  • CHIstory Bernstein, M., Andre, P., Luther, K., Poole, E. S., Solovey, E., Paul, S. 2009
  • Taskpose: Exploring Fluid Boundaries in an Associative Window Visualization 21st Annual ACM Symposium on User Interface Software and Technology Bernstein, M., Shrager, J., Winograd, T. ASSOC COMPUTING MACHINERY. 2008: 231–234
  • Simplifying Knowledge Creation and Access for End-Users on the Semantic Web Van Kleek, M., Bernstein, M., Andre, P., Pertunnen, M., Karger, D., schraefel, m. c. 2008
  • Evolution and Evaluation of an Information Scrap Manager Bernstein, M., Van Kleek, M., schraefel, m. c., Karger, D. 2008
  • Inky: A Sloppy Command Line for the Web with Rich Visual Feedback Miller, R., Chou, V., Bernstein, M., Little, G., Van Kleek, M., Karger, D. 2008
  • Wicked Problems and Gnarly Results: Reflecting on Design and Evaluation Methods for Idiosyncratic Personal Information Management Tasks MIT-CSAIL-TR-2008-007 2008 Bernstein, M., Kleek, M. V., Khushraj, D., Nayak, R., Liu, C., Karger, D. 2008
  • Information Scraps: How and Why Information Eludes our Personal Information Management Tools ACM Transactions on Information Systems 2008 Bernstein, M., Kleek, M. V., Karger, D., schraefel, m. 2008
  • Management of Personal Information Scraps Bernstein, M., Van Kleek, M., schraefel, m. c., Karger, D. 2007
  • GUI — Phooey!: The Case for Text Input Van Kleek, M., Bernstein, M., Karger, D., schraefel, m. c. 2007
  • Personal Information Management, Personal Information Retrieval? Bernstein, M., Van Kleek, M., Karger, D., schraefel, m. c. 2007
  • Diamond's Edge: From Notebook to Table and Back Again Ubicomp: Posters 2006 Bernstein, M., Robinson-Mosher, A., Yeh, R., Klemmer, S. 2006
  • Reflective Physical Prototyping through Integrated Design, Test, and Analysis Hartmann, B., Klemmer, Scott, R., Bernstein, M., Abdulla, L., Burr, B., Robinson-Mosher, A. 2006
  • D.tools: Integrated prototyping for physical interaction design IEEE PERVASIVE COMPUTING Hartmann, B., Klemmer, S. R., Bernstein, M. 2005; 4 (4): 79-79
  • d.tools: Visually Prototyping Physical UIs through Statecharts UIST: Extended Abstracts 2005 Hartmann, B., Klemmer, S. R., Bernstein, M., Mehta, N. 2005
  • Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations Valentine, M., Retelny, D., To, A., Rahmati, N., Doshi, T., Bernstein, M. 2017

    View details for DOI 10.1145/3025453.3025811