Advancing AI through cognitive science
Instructor: Brenden Lake
Meeting time and location:
Thursday 4-5:50 PM
Meyer Room 465 (4 Washington Place)
PSYCH-GA 3405.001 (Psychology)
DS-GA 3001.014 (Data Science)
Wednesdays 10-11:00 am, or by appointment; 60 5th Ave., Room 610
Summary: Why are people smarter than machines? This course explores how the study of human intelligence can inform and improve artificial intelligence. We will look to cognitive science, with special focus on cognitive development, to help elucidate a set of “key ingredients” that are important components of human learning and thought, but are either underutilized or absent in contemporary artificial intelligence. Through readings and discussion, we will cover ingredients such as “intuitive physics,” “intuitive psychology,” “compositionality,” “causality,” and “learning-to-learn,” although students will be encouraged to contribute other ingredients. Each ingredient will be discussed and compared from the perspectives of both cognitive science and AI, with readings drawn from both fields with roughly a 50/50 proportion.
This is a small discussion-based seminar, so please come ready to participate in the discussion. Please note that this syllabus is not final and there may be further adjustments.
- This course is intended for graduate students in cognitive science or graduate students in data science / AI.
- Students are not expected to have a background in both cognitive science and AI. Instead, students may have experience in one field and the desire to learn about the other. Ideally, at the end of the course, students will have a deeper appreciation of contemporary issues in both fields and their potential for synergy.
- At minimum, it would be very good to have taken a graduate level psychology course, OR a graduate level machine learning / AI course. If you have taken neither, this probably not the right course for you.
- Programming is not a requirement for this course, although students may choose to incorporate programming in their final project.
The final grade is based on the final paper or project (50%), written reactions to the reading (25%), and participating in discussions (25%).
The final paper or project is done individually. For the final assignment, students may either write a final paper that proposes an additional ingredient of human intelligence that is underutilized in AI, or complete a project that implements one of the ingredients discussed in an algorithm.
The project will represent either an substantial extension of one of the homeworks (e.g., exploring some new aspect of one of the assignments), implementing and replicated an existing cognitive modeling paper, or a written paper discussing one of the core modeling topics. The final projects will need to be approved by the instructor at least 6 weeks before the end of the semester.
We will be using Piazza for reactions to readings and class discussion.
The signup link for our Piazza page is available here (https://piazza.com/nyu/spring2018/psychga3405001).
Once signed up, our class Piazza page is available here (https://piazza.com/nyu/spring2018/psychga3405001/home).
- The final paper or project is done individually (not as a group).
- Option 1: A final paper that proposes an additional ingredient of human intelligence that is underutilized in AI. The paper should summarize the psychological literature on the ingredient, and discuss the relevant AI literature or lack thereof (about 8 pages)
- Option 2: Complete a project that implements an important aspect of one of the ingredients discussed in class (intuitive physics, intuitive psychology, compositionality etc.) in an algorithm (with a 4 page writeup)
- If you can link the project to your research, that’s encouraged!
- The final assignment should be discussed with the instructor and approved with at least 6 weeks before the end of the semester.
Auditing: Unfortunately we have no additional spots for auditors due to the large number of previous requests. If I have replied to your request, you may audit pending available seats. Priority goes to registered students and then by date of audit request.
Overview of topics and schedule
- 1/25 Introduction and overview
- 2/1 Deep learning – Lecture
- 2/8 Deep learning – Discussion
- 2/15 Intuitive physics (part 1: humans)
- 2/22 Intuitive physics (part 2: machines)
- 3/1 Intuitive psychology
- 3/8 Compositionality
- 3/15 NO CLASS. Spring Recess
- 3/22 Causality
- 3/29 Learning-to-learn
- 4/5 Critiques of “Building machines that learn and think like people”
- 4/12 Response to critiques
- 4/19 Language and Culture
- 4/26 Emotion and Embodiment
- 5/3 Neuroscience
Detailed schedule and readings
Please see below for the assigned readings for each class (to be read before class). Before each class, students will be asked to submit a reaction to the readings (three paragraphs). Reaction posts are submitted via Piazza. Papers are available for download on NYU Classes in the “Resources” folder. Reactions are due by midnight the day before class (so I have time to read the reactions!)
1/25 Introduction and overview
2/1 Deep learning – Lecture
- LeCun, Y., Bengio, Y. & Hinton, G. (2015). Deep learning. Nature 521:436–44.
- Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (pp. 1097-1105).
- No reaction post is requried for these readings
2/8 Deep learning – Discussion
- Lake, B. M., Ullman, T. D., Tenenbaum, J. B., Gerhsman, S. J. (2017). Building machines that learn and think like people. Behavioral and Brain Sciences.
Only Sections 1-4 (pgs. 1-9)
- Mnih, V., Kavukcuoglu, K., Silver, D., …. & Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature 518(7540):529–33.
- Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., … & Badia, A. P. (2016). Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626), 471-476.
- Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., & Bengio, Y. (2015). Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning (pp. 2048-2057).
- Reaction post is requried for this class and the following classes (due midnight the night before class)
2/15 Intuitive physics (part 1: humans)
- Building machines that learn and think like people (Section 4 through 4.1, pg. 9-11)
- Spelke, E. S. (1990). Principles of object perception. Cognitive Science 14(1):29–56.
- Baillargeon, R. (2004). Infants’ physical world. Current Directions in Psychological Science 13:89–94
- Battaglia, P. W., Hamrick, J. B. & Tenenbaum, J. B. (2013). Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences 110(45):18327–32.
2/22 Intuitive physics (part 2: machines)
- Lerer, A., Gross, S. & Fergus, R. (2016). Learning physical intuition of block towers by example. Presented at the 33rd International Conference on Machine Learning (ICML).
- Battaglia, P., Pascanu, R., Lai, M. & Rezende, D. J. (2016). Interaction networks for learning about objects, relations and physics. Advances in Neural Information Processing Systems.
- Mottaghi, R., Bagherinezhad, H., Rastegari, M., & Farhadi, A. (2016). Newtonian scene understanding: Unfolding the dynamics of objects in static images. Computer Vision and Pattern Recognition (pp. 3521-3529).
3/1 Intuitive psychology
- Building machines that learn and think like people (Section 4.1.2, pg. 11-2)
- Baker, C. L., Jara-Ettinger, J., Saxe, R. & Tenenbaum, J. B. (2017). Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour.
- Csibra, G., Biro, S., Koos, O. & Gergely, G. (2003). One-year-old infants use teleological representations of actions productively. Cognitive Science 27:111–33
- Leslie, A. M., Friedman, O., & German, T. P. (2004). Core mechanisms in ‘theory of mind’. Trends in Cognitive Sciences, 8(12), 528-533.
- Building machines that learn and think like people (Section 4.2-4.2.1, pg. 12-15)
- Marcus, G. (1998) Rethinking eliminative connectionism. Cognitive Psychology 282 (37):243–82.
- Tversky, B. & Hemenway, K. (1984) Objects, parts, and categories. Journal of Experimental Psychology: General 113(2):169–91.
- Lake, B. M. and Baroni, M. (2017). Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks. Preprint available on arXiv:1711.00350.
- Optional: Eslami, S. M., Heess, N., Weber, T., Tassa, Y., Kavukcuoglu, K. & Hinton, G. E. (2016) Attend, infer, repeat: Fast scene understanding with generative models. Presented at the 2016 Neural Information Processing Systems conference, Barcelona, Spain.
3/15 NO CLASS. Spring Recess
- Building machines that learn and think like people (Section 4.2.2, pg. 15-16)
- Bever, T. G. & Poeppel, D. (2010) Analysis by synthesis: A (re-) emerging program of research for language and vision. Biolinguistics 4:174–200.
- Lake, B. M., Salakhutdinov, R. & Tenenbaum, J. B. (2015) Human-level concept learning through probabilistic program induction. Science 350(6266):1332–38.
- Rezende, D. J., Mohamed, S., Danihelka, I., Gregor, K. & Wierstra, D. (2016) One-shot generalization in deep generative models. Presented at the International Conference on Machine Learning,
- Optional: Murphy, G. L. & Medin, D. L. (1985) The role of theories in conceptual coherence. Psychological Review 92(3):289–316.
- Building machines that learn and think like people (Section 4.2.3-4.3, pg. 16-19)
- Smith, L. B., Jones, S. S., Landau, B., Gershkoff-Stowe, L. & Samuelson, L. (2002) Object name learning provides on-the-job training for attention. Psychological Science 13(1):13–19.
- Ritter, S., Barrett, D. G., Santoro, A., & Botvinick, M. M. (2017). Cognitive psychology for deep neural networks: A shape bias case study. International Conference on Machine Learning (ICML).
- Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., … & Botvinick, M. (2016). Learning to reinforcement learn. arXiv preprint arXiv:1611.05763.
4/5 Critiques of “Building machines that learn and think like people” (with special guest Ernie Davis)
- Building machines that learn and think like people (Section 5-end, pg. 19-25)
- Commentaries to read:
- Botvinick et al., “Building machines that learn and think for themselves”
- Chater and Oaksford, “Theories or fragments?”
- Clegg and Corriveu, “Children begin with the same start-up software, but their software updates are cultural “
- Davis and Marcus, “Causal generative models are just a start”
- Hanson, Lampinen, Suriv, McClelland, “Building on prior knowledge without building it in”
- MacLennan, “Benefits of embodiment”
- Moerman, “The argument for single-purpose robots”
- Spelke and Blass, “Intelligent machines and human minds”
- Tessler, Goodman, Frank, “Avoiding frostbite: It helps to learn from others”
4/12 Response to critiques
- Response, Lake, Ullman, Gershman, Tenenbaum, “Ingredients of intelligence: From classic debates to an engineering roadmap” (pg. 50-59)
4/19 Language and Culture
- Mikolov, T., Joulin, A. & Baroni, M. (2016) A roadmap towards machine intelligence. arXiv preprint 1511.08130.
- Lupyan, G. & Bergen, B. (2016) How language programs the mind. Topics in Cognitive Science 8(2):408–24.
4/26 Emotion and Embodiment
- Ong, D. C., Zaki, J., & Goodman, N. D. (2015). Affective cognition: Exploring lay theories of emotion. Cognition, 143, 141-162.
- Niedenthal, P. M. (2007). Embodying emotion. Science, 316(5827), 1002-1005.
- Hassabis, D., Kumaran, D., Summerfield, C., & Botvinick, M. (2017). Neuroscience-inspired artificial intelligence. Neuron, 95(2), 245-258.
- George, D., Lehrach, W., Kansky, K., Lázaro-Gredilla, M., Laan, C., Marthi, B., … & Lavin, A. (2017). A generative vision model that trains with high data efficiency and breaks text-based CAPTCHAs. Science, 358(6368).
- Marcus, G., Marblestone, A., & Dean, T. (2014). The atoms of neural computation. Science, 346(6209), 551-552.