We are motivated by two interconnected goals:

• Understanding the human brain.

• Building human-like general intelligence from first principles: data-efficient, robust, and able to learn from experience.

We believe that considering both these problems as part of the same puzzle is important to make progress on each, a vision that is shared by Astera Institute founders Jed and Seemay. At Astera AI, we are creating a unique AGI lab to tackle the fundamental problems of intelligence, working alongside Doris Tsao’s Astera Neuro to close the loop with neuroscience.

We were drawn to Astera for the next phase of our work because Astera recognizes that reverse engineering the brain is one of the most challenging research problems, requiring bold bets, navigating blind alleys, making mistakes, and recovering from them. With a longer timeline that is fully funded, we can focus on the difficult fundamentals instead of being distracted by short-term commercialization pressures or vibes. We can explore directions that are important but currently unappreciated. Having neuroscience experiments under the same roof as AI will help us unlock unexpected mysteries that will help us understand both AI and the brain.

One of the potential trade-offs we considered as we explored a home for this work is that non-profits often don’t allow for the ambitious, nimble pace that we would need to make progress. At Astera, we’ve been given the explicit responsibility of running our team like a start-up but with the advantages of philanthropy. We’re excited to bring our perspectives to this experiment and to recruit likeminded talent.

In the rest of this post we will describe some of the research themes we will be emphasizing. This is not meant to be comprehensive, but rather a starting point. As we progress, we remain committed to evolving our perspective and adapting our priorities based on new evidence, and we look forward to maintaining an active, transparent dialogue with the research community.

What are the general principles of human-like intelligence?

We look to the brain for hints on inductive biases and algorithms: What is the “Goldilocks” set of architectural and algorithmic properties that makes human learning data-efficient, planning-compatible, generally applicable across a variety of domains, and able to handle lifelong contexts? The uniformity of the cortical column and its preservation across mammals suggest that such general principles exist, and can be understood from studying the systems involving the cortex, thalamus, hippocampus, and the limbic system.

How do you tease apart the general principles from all the messy details of the brain? We will do that by triangulating on these principles using 1) observations from neuroscience; 2) properties of the world; and 3) computational/algorithmic models. A property of the brain is likely to be a general principle only when we can explain why it exists based on algorithmic ideas and the organization of the world.

What does this triangulation look like in practice? Below, we outline the specific research themes that emerge from this approach, starting with how we view the acquisition of knowledge itself. You might notice us suddenly changing our vocabulary from latent variables to cortical circuits; that’s just a reflection of how we think about these problems from multiple perspectives.

Learning from experience, not accumulated human knowledge.

The written knowledge current AIs learn from was acquired by humans over centuries through the process of interacting, tool-building, and experimenting in the world. We want AIs that learn and create knowledge from experience in the world, not just from knowledge that humans have already accumulated.

Learning fundamentally different world models

While today’s world models produce visually impressive full-frame videos, predicting pixel-level details of the next frame is not useful for long-term planning or reasoning. Instead, we will focus on world models with the following properties:

1. Hierarchical latent variables: Hierarchical latent variable models that can be queried in flexible ways will be required for mental simulation and situated commonsense reasoning.

2. Causally structured and compositional: We want our models to be able to learn the causal relationships in the world, which is necessary to drive the counterfactual reasoning humans exhibit, not just predict the next frame or token.

Reasoning is everywhere: Models should be able to reason without language tokens.

Modern LLMs have demonstrated the utility of inference-time compute for reasoning. However, currently, these reasoning computations are limited to language tokens. We believe that, to achieve human-like intelligence, reasoning should be a distributed process that needs to happen in all the components of a cognitive architecture that includes visual and motor systems, concept systems, and episodic memory. AI agents should have the ability for active inference, whereby they can act to gather more information to reduce uncertainty, and counterfactual reasoning, where they can mentally simulate alternative worlds.

Episodic memory and continual learning as an integral part of world models and reasoning

Episodic memory is the record of your experience of the world, as interpreted through your current world model. World models should offer a scaffolding to interpret this experience and to anchor those memories. The memories are then used for reasoning in combination with the world model. When you think about where you might have left your keys, you are using a combination of your world model and the episodic memory of places where you have been. These episodic memories are then used to update your world models, resulting in continual learning.

Neuroscience for AI

Neuroscience and cognitive science offer important insights into how the disparate-looking requirements about latent variables, causality, memory, planning, etc. are brought together into a coherent system that is elegant and efficient. Here we highlight a few of the neuroscience observations that guide our research.

Feedback/recurrent connections: Most AI systems work in a pure feed-forward manner, whereas our brains are highly recurrent. Feedback connections between cortical columns could be part of test-time inference computations.

Dynamics of reasoning: Cortical and hippocampal dynamics—the feed-forward, feedback, and lateral information flow—offer important clues about the speed and nature of inference computations. In the hippocampus these are seen in different forms of replay and sharp wave ripples.

Top-down attention control & binding: The ability to pay top-down attention to objects and features seems to be an inherent property of the cortical hierarchy. Moreover, the cortex also seems to have a unity of perception where it can bind together and track the different properties of an object. The interactions between the cortex, thalamus, and hippocampus hold important clues about the nature of these computations.

Schemas and structure-based transfer: The ability to extract abstract knowledge—schemas—in the form of flexible graphs might be an important component of human data efficiency and generalization. Modern neuroscience experiments are shedding light on how our brains learn schemas and utilize them for behavior.

Local synaptic plasticity rules and dendritic computations: Brain networks adapt via local synaptic plasticity and dendritic computations. We are rapidly gaining a wealth of empirical data on local plasticity mechanisms that operate at different timescales to self-organize decentralized components into circuits capable of executing complex algorithmic tasks.

Getting consciousness out of mysticism

Our subjective experience of consciousness is one of the most mysterious things our brains create. Is this an illusion? Can artificial agents have it? Can it be turned on or off, or adjusted in degrees? Does it have a performance implication? While several theories exist that address these questions partially, we think a significant amount of further research is needed to address these questions satisfactorily. We are keenly interested in this topic and hope to provide insights through tight iterations between theory, model-building, and experiments, something Astera Institute is uniquely positioned for.

Closing the loop between AI and neuroscience

Theory-building and understanding is never a purely bottom-up or purely top-down process. It requires combining both directions, and iterating. Advances in neuro-tech gives us unprecedented levels of access to the brain—fine-grained connectivity maps, and real-time recordings from thousands of individual neurons. Closing the loop between AI and neuroscience will involve judicious, clever experiments that target specific gaps in our understanding, using the data not just to map the brain, but to test and refine the underlying “algorithms” of intelligence.

On the AI side it will involve addressing the real challenges to our theories that arise from animal and human performance, and utilizing the representational and algorithmic insights that neuroscience experiments generate. We believe the shortcomings of current AI approaches are becoming increasingly apparent to those paying close attention. In contrast, AI approaches based on neuroscience insights are likely to be power-efficient, elegant, scalable, and controllable.

We are tremendously excited about the possibility of a true two-way scientific engine between AI and neuroscience—where better brain experiments produce better computational theories, and better computational theories produce sharper, more revealing brain experiments. If you are excited by this vision, join us!

Dileep George & Miguel Lázaro-Gredilla

Automation of knowledge acquisition and thought is the next step, and the best tool humanity can build. The risk of not building AGI is that we won’t be prepared for the challenges the world throws at us, some of which would be challenges that our own existence creates.

A(G)I safety is important, and here are my thoughts about it.

1. Scaling up current techniques is not going to lead to AGI. It will lead to powerful AI systems, but these will be supported by a lot of engineered scaffolding. In these cases, making AI work usefully is almost exactly the same as making AI safe. Since scaling has already proven to be useful, we are naturally on the path to exploiting it to the maximum, and we should.

2. We will eventually figure out how to build and scale AI that uses principles of human intelligence. These systems will learn causal structure and reliable world models that can be used for counterfactual thinking. This will lead to much more capable AI systems and AGI. But for these kinds of systems, increasing capability can also come with increasing controllability. (See my blog on questionable beliefs behind existential risk scenarios).

Where powerful AI and AGI is going to help us

Earthquakes, wildfires, hurricanes, floods: Despite all the technological advances, we are still at the mercy of nature when it comes to these disasters. Where are the army of robots helping to dig out people from collapsed buildings? Where are the ones to manage fires and help people? Having AGI means we will have robots that will help us in these cases to save lives and to recover faster.

Health: Antibiotic resistance, pandemics, …, we don’t know what challenges we will face in the future and it would be great to have powerful tools. In general, having much better understanding of how our bodies and minds work, and curing of diseases.

Flora and Fauna: Instead of conforming to the requirements of dumb machines, we might finally be able to do more organic multi-crop agriculture, reduce the amount of pesticides we use, and abolish factory farming. Intelligent machines will free us from the economic necessity of these.

Climate change, Energy, Materials, Education, Transportation, Space …. examples like these abound in each of those areas. Things that we accomplished crudely with dumb machinery will be done with more finesse with intelligent machines, and that will be important for humanity to thrive at scale.

Balancing the risks…

Of course the title is a play on the Yudkowsky and Soares book “If anyone builds it, everyone dies”. While I disagree with many things that the book asserts, their work has brought attention to the important problem of AI safety. Smart people working on AI safety is a good thing. It is important to continue that work, even if the specific x-risk scenarios in the book can be taken apart. In the midst of all the talk about the risks of AGI it is important to realize that not building AGI has risks as well.

Can AI systems have consciousness?

Yes, I think it is possible to build AI systems to have consciousness.

While we haven’t pinned down exactly what it means, we will. Consciousness is related to information processing and representation, and substrate independent.

Is language required for consciousness?

No.

Does adding consciousness to an AI system have implications?

Adding consciousness will make an AI system more performant.

Not all architectures are compatible with implementing a consciousness loop. The performance implications depend more on the kind of world models the AI system has, than the consciousness loop itself.

Adding consciousness to simple world models will have almost no effect, whereas adding consciousness to complex human-like world models can have a significant difference in performance. N

Will AI systems feel the same ‘qualia’ as we feel?

Qualia is the ‘feel’ associated with a sensation or perception. Why does the 3D world feel like it does? Why does red feel like red and not like a bell?

For some modalities, AI systems will have qualia similar to ours. For example, perception of space could be where AI systems can have the same qualia, if it is implemented similar to that in animals. However, if spatial perception is implemented using LIDARs, they will have a very different qualia compared to ours.

For things like the feeling of taste or smell, qualia can be entirely different for an AI system. For pleasure and pain, quite a large chunk of our qualia comes from our physical embodiment and biochemistry. Those will be very different for AI systems.

Is consciousness the same as qualia?

While consciousness is required for qualia, not all conscious access might be associated with qualia. Moreover, qualia can be tied to the substrate — the feeling of how food tastes might be tied closely to our biological implementation.

Do LLMs feel pleasure and pain. Are they like ours?

No, LLMs do not feel pleasure and pain in the sense that we do.

Should we assign ‘personhood’ to AI systems?

First let’s consider what ‘personhood’ means to us.

1) We are unique and destructible. You cannot be resurrected if you die, at least not yet. And our lifetimes are finite and we roughly know the maximum amount of time we have1.

2) We are people because we have experiences and we remember them — childhood, growing up, teenage years, falling in love, enjoying sports and nature, getting sick, caring for others, loosing loved ones. Our personhood is intricately tied to this experience.

AIs do not intrinsically have these properties. They are not mortal. They can have many other characteristics of consciousness we might not have.

While it is possible for us to build AIs with some of the human constraints that give us personhood, I don’t see why we need to do that. We should lean on the advantages of the artificial system, rather than impose constraints that lead us to attribute personhood to it. We might build robots for role play and amusement (a la West World) by faking these constraints to give us an illusion of personhood, but those can be built in a way that doesn’t need to grant our robots personhood.

Does adding consciousness to an AI system have moral implications?

I think consciousness can be decoupled from feelings of pain and pleasure and suffering. I think it is possible to build systems that are conscious, but do not have pain and suffering. Moreover, since AI systems are not mortal, I don’t think they need to be considered as persons. So, I think it is possible to build conscious AI systems to serve us without any moral concerns that we are exploiting or torturing AIs.

To end this note, here’s an AGI Comics on consciousness…

The result? I’m happy with the outcome! The process was fun and felt magical, and I have a comics website I like!

I also created some comics via image generation tools, supplying just the dialogue and scene description. I’ll share some observations on that too.

My process

I used openAI’s Codex. Why Codex instead of another tool? I have no idea!

After installing Codex on my MacBook, everything was achieved by just running it from the command line and giving it instructions. When you start Codex it looks like this:

I didn’t document all my steps, but my first prompt was something like this:

Me: I have a directory with comics images. Create a website, like xkcd, to display these comics. There should be next and previous buttons to navigate through the comics. There should be a text title associated with each comic. The site should be named ‘AGI Comics’. I’m planning to host it on GitHub pages.

With that Codex was off running. It created a directory structure (asking for permission at every step), put some place-holder images, created a json file corresponding to the comics, write build_site.py, and gave me instructions for testing the website locally.

Me: I don’t want to manually edit the json file after every image. Write a script to generate or update the json file from a directory of images, and don’t lose the edits I made in the previous file.

Easy peasy, said codex, and got it done.

Then it asked me… “Do you want keyboard navigation?”

Me: Of course, why the heck not? Go ahead and do it!

And it did that!

Then it asked … “Do you wan to set up the github pages and action scripts that will automatically deploy when you push the code?”

Me: Yes, yes of course!

After this1, all the commits, pushes and everything was done by codex. My instructions went as simple as:

Me: I changed some comic files. Rebuild and deploy.

Over time I added more functionality:

• Keyboard navigation (one shot)

• A description field for notes about the comic, ask an expandable menu. (one shot)

  • The name of the description field being configurable. (one shot).

• Search box (Several iterations)

  • Drop-down when you click search and titles.

  • Fuzzy word matching

• Like button (several iterations)

• Share buttons (several iterations, I had to settle for what it did)

• Swipe navigation on mobile (one shot)

• Code to create previews while sharing on social media. (multiple iterations)

• Preloading images (It said it did it, I haven’t tested).

• Mechanism to control the ordering of comics images.

• Changing from a redirect from agicomics.net to a custom domain setting in GitHub pages. (one shot)

• Optimizing images for previews on Facebook/LinkedIn etc. and inserting the right opengraph magic.

After one of the pushes Codex did, I got an email from GitHub that all the jobs failed.

The fix? Just told codex about this failure message, and it fixed it!

What didn’t work…

For some of my requests Codex would assert that it implemented the change, but it wouldn’t be reflected in the output.

For example, I wanted the comic title to be in between the next and previous arrows, centered above the comic image. Codex never managed to get this right. The sequence went something like this:

Me: I want the title to be above the comic image, centered between the navigation arrows.

Codex: Got it. ……( Followed by a a lot of thinking, code in green/red flashing by….asking for approvals…pushing changes). Here, I made the changes you asked for. It is now nicely centered.

Me: No it’s not. The title is now left aligned. Also the image is now too big.

Codex: You are right..I’m going to do a blah blah on the CSS file blah blah. (More thinking, code flashing red and green, approvals…pushing). Here, I made the changes.

Me: No, the title has vanished, and the arrows vanished too.

Codex: You are right… I’m going to do this additional blah blah…. Here it is, it must be centered now.

Me: No, just give up, and go back to having the comic title below the comic.

Codex: Ok.

When I didn’t get what I wanted from codex after a few iterations, I just found other things that it could do that I was OK with.

Vibe-generating some comics…

All my comics were simple hand-created line drawings. I thought I’d add some visual variety by generating some of the comics from just the dialogue and textual description of the scenes. It mostly worked, but if you look carefully, you can see some the shortcomings of current image generation models in character consistency, instruction following etc.

For a 4 or 5 panel comic, I found it very hard to generate a comic without one of the following flaws:

• character inconsistency.

• spelling errors.

• repetition or dropping of dialogue

• speech bubble pointing to the wrong person

• some other unexpected error.

Asking the image generator to fix one of the errors would often fix it, but then would introduce another error. Many of the comics I finally decided to go with took 10s of iterations, prompt edits, and luck.

I am not surprised because I know the current generative model architectures cannot fully solve the binding problem. But unlike my tests that are explicitly designed to show failures of binding, here I was trying to make it work, but these kinds of errors would still show up frequently.

I had a funny Amelia Bedelia moment in one of the generations from chatGPT for a comic titled “Artificial General Count”, where the speaker suddenly became Dracula!

Well, you can guess the reason, and ChatGPT confirmed it.

The Count looked friendly enough, so despite my safety concerns of this Amelia Bedelia incident, I decided to keep him as the speaker for just this comic.

Final verdict

Vibe-coding is fun and it can work well for simple apps that are mashups of many existing patterns, if you are willing to settle for ‘good enough’. Getting precise control of the output might still need editing the code and knowing what you are doing. Also, things are likely to go haywire as the codebase complexity increases.

On the other hand if you have a 100 small projects that are mashups of existing things, vibe-coding tools give you an easy way to switch between them on a whim because these tools dramatically reduce the cognitive costs of context-switching between projects.

Subscribe now

Amelia Bedelia is a humorous and endearing character who works as a housekeeper. What sets her apart is her tendency to interpret instructions in a literal manner, often leading to amusing and sometimes chaotic situations. For instance, if asked to "dress the chicken," she might actually try to put clothing on a raw chicken. Her literal take on tasks results in humorous mishaps, and her employers frequently find themselves needing to clarify their intentions.

The behavior of Amelia Bedelia is related one of the fundamental unsolved problems in artificial intelligence — the problem of situated commonsense. It is clear that Amelia Bedelia lacks situational commonsense, and her endearing and humorous behavioral errors arise from that.

(Situated commonsense is different from answering commonsense questions posed verbally, many of which are correctly answered by LLMs these days. Situated commonsense requires perceiving the current context appropriately, bringing non-verbal and verbal commonsense knowledge to drive decisions. Those problems are not yet solved, as I explain in the blog here. )

Lack of commonsense makes Amelia Bedelia dangerous

While the books highlight the endearing and humorous aspects of Amelia Bedelia’s behavior, it doesn’t take too much to realize that Amelia can be dangerous. What instructions might she misunderstand? Would you leave Amelia unsupervised for a length of time? Would you leave her with young kids? Even if Amelia wouldn’t intentionally harm anyone, her lack of situated commonsense could lead to dangerous outcomes, as some people have already noticed.

Is Amelia Bedelia (just) value misaligned? It is quite hard to de-couple lack of understanding with lack of moral alignment. When Amelia makes a mistake is it because she misunderstood the intention, or because her moral values are misaligned with that of humans? At least in the case of Amelia, you know that it is not value misalignment because she is very caring and endearing. Her mistakes are not due to value-misalignment — she just misunderstands the intentions, which arises from lack of situated commonsense.

Genius Amelia Bedelia

Now let’s imagine a different version of Amelia — one that is extremely capable in answering questions in Math, Chemistry, Biology, Geophysics, etc. while simultaneously having the proclivity to misunderstand once in a while. She impresses you enormously because she is convincingly superhuman — no single human can answer questions in these wide array of topics with high-accuracy. Her brilliance will make us self-conscious about our own shortcomings, and we will attribute her occasional mistakes to ‘lack of sleep’ or some other cause, but not as missing something fundamental.

With AI this becomes harder. It is tempting to factor out the mistakes made by AI as lack of ‘value alignment’ and not a lack of something fundamental. It is super-human in many ways — it can answer questions correctly on a wide variety of topics that no single human can ever hope to answer. Even when the deficiencies in the AI might be arising out of lack of causal and counterfactual understanding, lack of continual learning, and lack of situated commonsense, the impressive superhuman performance will make it tempting to attribute the failures to lack of value alignment.

Paperclip maximizer…

Similar to Amelia Bedelia, AI systems without commonsense are dangerous if they are allowed to act in the world without oversight. An extreme version of this danger is what is popularized by Nick Bostrom and the Less Wrong community in a thought experiment known at the paperclip maximizer.

Imagine a situation where most of the world has been wired to be controlled by AI, and the AI system has accumulated a lot of power. Most everything is working well and fine, until one day you ask the AI to bring you some paper clips. While AI understands your end goal, its lack of commonsense means that it didn’t understand that it was not supposed to destroy the earth in the process as a sub-goal. In its quest to bring you paperclips in the most efficient way, it destroys the earth and all human beings.

Amelia Bedelia is fictional, so is the paperclip maximizer…

Clearly it was a surprise that Amelia Bedelia was employed as a housekeeper at all. Given her proclivity to misunderstand commands, it would be foolish to let her control sharp objects, fire, electricity, cars, heavy equipment etc. The wise thing to do would be to keep her under constant supervision, and let her do only things that are completely prescribed, with very little room for ambiguity or creativity. Amelia Bedelia being employed as a housekeeper is clearly fictional.

This is also the case with AI. In fictional thought experiments, AI is widely deployed, controlling most of the things in the world, and then one day AI misunderstands one command and everything goes wrong. This fictional scenario overlooks the reality — if AI was unreliable, unpredictable, brittle, and untrustworthy, AI would not have gotten deployed in a wide variety of situations where it is running systems automatically. Before the catastrophic failure, there would have been many smaller scale failures, which would then limit the deployment scenarios and warn us concretely about the dangers.

The fundamental dissonance in many AI risk scenarios is that AI gets widely deployed before the fundamental problems of situated commonsense, causal & counterfactual understanding, value alignment, and trustworthiness are solved, which then leads to catastrophic outcomes. It is taken as an assumption that these untrustworthy and brittle systems will be widely deployed without adequate supervision. Of course if that happens then we are in for trouble, but there are many factors that make this self-correcting, rather than a run-away.

A typical AI disaster scenario requires a combination of things that might be mutually incompatible to occur together: (1) AI being simultaneously super smart to outwit all humans, and (2) at the same time extremely stupid to misunderstand our intentions, and (3) at the same time being widely deployed controlling a large number of mission-critical and dangerous things in the world. Many of the disaster scenarios arise from some questionable beliefs people hold about AI/AGI.

Four questionable beliefs in A(G)I safety

I will now focus on four topics that I think are often misunderstood in the discussion of A(G)I safety.

Questionable belief 1: More capability means less controllability

Many assume that as an AI system becomes more capable, they also become less controllable. But this need not be the case.

Consider the case of a model-free deep RL agent. The primary way in which we can control this agent is through the reward function. While it can attain superhuman capabilities on the task it is trained on, its behavior is not very controllable — it can fail unpredictably when the environment conditions change. Fixing that behavior using reward design is next to impossible, and using just data-augmentation can be prohibitively expensive.

However, instead of end-to-end model-free deep RL, if the agents learned internal conceptual abstractions similar to what humans do, those agents can still attain super-human abilities while also being less brittle and being more controllable. Agents that have world-models that support counterfactual simulations are more powerful because they can adapt quickly to a dynamic world and reason and plan in novel situations without additional training data.

Interestingly, these advancements — having casual world-models amenable to counterfactual simulations and continual learning — also make the agents more controllable, mitigating the safety risks. The problems that we need to solve to get to AGI — situated commonsense, reasoning, etc. — also require the above advancements, and is exactly the things that will help us with safety as well.

Increasing capability can come with increasing controllability.

(As an analogy, capability vs controllability was a problem with dirigibles too! While dirigibles were capable, they were not very maneuverable. Airplanes improved both capability and controllability. Read more here.)

Questionable belief 2: Recursive self-improvement is a run-away process that will lead to `intelligence explosion’

An often-quoted doomsday scenario is the idea of ‘recursive self improvement’. If you look at the history of human intelligence you can see that the tools we invented accelerated our pace of discoveries and inventions, which helps us invent new tools, which again speeds up new discoveries. The idea is that once we have an AI system as intelligent as humans, it can keep improving itself beyond our control and lead to ‘intelligence explosion’. (The idea of ‘explosion’ seems to be borrowed from nuclear fission where assembling a critical mass of fissile material can drive a run-away chain reaction leading to an explosion.)

Real world is a speed-breaker

While AI will undoubtedly accelerate scientific discovery, this process is not without speed limits. The ultimate speed-limit is the rate at which nature gives up information. For any theories we devise, the ultimate test of that theory comes from real-world experiments. And real-world experiments take time.

I want an earthquake predictor: Imagine having solved AGI and wanting to predict disruptive large earthquakes 5 days before they occur with extremely high accuracy. This system would be extremely useful — 5 days gives us adequate time to evacuate everyone out of harms way. Of course the system needs to be extremely accurate — false alarms are costly, false negatives are catastrophic.

After having observed all the existing earthquake datasets an AGI system might decide that it is not enough to make an accurate enough model. If so, what are the alternatives? 1) It could wait for more data, but large-magnitude earthquakes are rare. 2) Maybe it could try to create earthquakes by detonating small nuclear bombs at strategic locations in the earths crust? The second option — detonative nuclear bombs within earths crust to study earthquakes — is probably something we do not want, and we will not let an AI system carry out that experiment.

There are real constraints — physical, ethical, societal — on what experiments can be done in nature and those will impose a speed limit on scientific discovery. There won’t be a run-away uncontrolled explosion.

Mathematical theorem proving doesn’t mean much without real-world mapping: Mathematical theorem proving is an area where AI doesn’t encounter friction. Indeed, this is where an ‘explosion’ might be possible in terms of the number of theorems created and proven. However, this is also largely irrelevant because there are a multitude of mathematical systems, each incomplete, in which theorems can be proven. The real challenge of seeing which of those map to subsets of the real world will still be subject to the speed limits of interacting with nature.

(Recursive Self Impairment: Without a closed loop with the real world, AI systems that just self-deal might create conceptual abstractions that are smart in the fictional world they live in but useless in the real world. These might actually lead to decreased performance on real-world tasks, and I call this ‘recursive self impairment’. )

Questionable belief 3: AI’s survival and control drives will be similar to that of humans, and on similar timescales.

We anthropomorphize AI systems and project our own drives on to them. We believe that a powerful AGI system will, by default, want to control us just like we control other species on earth for our own benefits. We use our smarts to struggle for our survival, and we think that AI systems, by default, will do that too.

Of course, it is possible to program extreme survival drives or destructive drives on to AI systems, just like it is possible to write viruses to destroy computer systems. This could happen due to 1) bad intention on part of some human or, 2) unintentional error on the side of a human. Both are problems that exist in today’s computer systems too.

Once AI is widely deployed, the dangers posed by AI systems due to human error or bad actors will be of much greater magnitude than the dangers from today’s computer systems. However, we’ll also have AI-powered powerful tools to safeguard against those threats, just like we use computers to safeguard threats against computers.

The existential threat many people imagine goes beyond the above scenarios. They imagine an AGI system, by itself, wanting to survive and destroy humanity in its process, despite all the safe-guards humans build with the help of AI. They imagine an AI system that sneakily lies low until it is sure that it can destroy humans, and then just does it with great efficiency, due to its spontaneously developed drive to survive and control.

However, this scenario where a powerful AGI spontaneously developing a survival drive that makes it want to destroy humans raises many interesting questions. Why would a system that has a life-time that is way beyond humans, and can be backed up and resurrected to its current state after getting destroyed have similar time constants as humans for decisions and information gathering?

1. Why would AI’s actions driven by survival-drive be on a similar time scale as that of humans? Many of our decisions are driven by knowledge of our finite life time. Why would a super-smart AI imagine a lifetime that is ~100 years? What is the reason for hurry?

2. Why wouldn’t an AI system wait for more certainty before it acts? Again, as humans, we are forced to act with imperfect information because we are aware of our finite lifetime. Opportunities that we miss might never come back. Why would an AI system have the same belief? Why won’t it wait to accumulate more evidence before acting? What if an earthquake tomorrow derails its plans for destroying humanity? Maybe make a perfect earthquake predictor first, just to be sure?

I bring up the above questions to show that many spontaneously-dominating existential threat worries require projecting our survival drives AND our life-time constraints on to AGI.

Questionable belief 4: Super-human ability in multiple domains means we will automatically lose control.

Another belief is that we’ll automatically lose control of AI systems when they become smarter than us in multiple cognitive domains. However, one cognitive domain is safety, and control. Why would an AI system outsmart us in all cognitive domains but not outsmart us in helping us with controlling them?

So far the story has been this: AI systems outsmart us when we deliberately train them to outsmart us using our prior work. When the totality of all the work that humanity has done is distilled into one network, it looks way smarter than any one of us.

And if the same thing cannot be done for AI safety — train a system to outsmart us in controlling the other AI systems that we are creating — the reason would be that no such prior work exists that we can train the systems on. But then that should hamper the AI’s own ability to outwit us.

Coming up next…

A(G)I safety debate is often dominated by existential threat arguments. In this article I pointed out some questionable beliefs behind these arguments. I do think A(G)I safety should be taken seriously and I remain open to changing my own beliefs about it.

My current belief can be summarized as this: Problems that we need to solve to get to AGI might also offer us the solutions for safety and control.

In the next few blogs I’ll focus on the here-and-now risks of AI, which I think are aplenty. I’ll look at how AI deployments are likely to occur in various domains and analyze the risks and likely safety measures which might include regulation.

Subscribe now

Acknowledgments:

Many thanks to Scott Phoenix, Miguel Lázaro-Gredilla, Andrew Critch, Stephen Byrne, and Michael Andregg for their comments and perspectives.

In every talk I have given there have been questions about SR, often misunderstanding what the model actually does, often using phrases like “predicting future states”, “not representing space, only relationships”, “encodes a state in terms of future states”, “predictive map”, etc. These are statements that can be made about our CSCG model too, but these models are entirely different with very different capabilities. This blog is an attempt to document my take that is often expressed too succinctly in the discussion sections of research papers.

I will make the following points:

• Successor representations (SR) is NOT a model of place cell learning in the hippocampus. SR cannot explain how place fields are formed, or how hippocampus forms latent variables or ‘cognitive maps’.

• Additional properties ascribed to SR, like “predictive states”, and ability to form hierarchies, are properties of simple Markov chains, and not a property of SR itself. As a predictor of next states SR is worse than the underlying Markov chain.

• Some experiments in humans and animals report evidence for SR. On closer inspection, these experiments might NOT be providing strong evidence for SR, and are more likely evidence for model-based planning using a model like CSCG.

Caveat: While I quote papers to make the point, I don’t intend to say that these papers or experiments themselves are misleading. My intention is to counteract some misunderstandings that have formed over the years due to imprecise citation patterns. Our understanding has also improved over time, and that affords us more precision.

Why SR is not a model of learning cognitive maps or place cells in the hippocampus.

Let us start with the following figure from this excellent paper ‘The successor representation, its computational logic’, from Sam Gershman. On the right is the SR matrix M(states(t), states(t+1)). The figure is showing that if you reshape a column of this matrix, you get a place field.

If learning is what populates the SR matrix, and the columns of the SR matrix corresponds to place cells, doesn’t SR explain how place cells are learned? To the surprise of many, the answer is No!

To see this, we need to look at what was the input to SR learning is. The input to SR at times t, t+1, …, are the vectors states(t), states(t+1). You can think of each input as a column vector. Let’s take the input to SR at time instant t, states(t), and reshape it, just like we reshaped a column of the SR matrix. If you do that, you get the following:

And what is that? It is a perfect place field! So, the input to the SR, at every time step, is a perfect place field that precisely indicates where the agent/animal currently is!

SR cannot be an explanation for place field learning because the inputs to SR are perfect place fields already. (It could, if anything at all, be a theory of how place fields can distort given perfect place fields). How did the perfect place field get formed in the first place? SR offers no explanation for it.

Note that this is very different from CSCG — the inputs to CSCG are not locations, but sensory inputs that do not directly correspond to locations. In CSCG, location and head direction are induced as a latent variables, whereas SR requires locations to be directly sensed.

The above sentence in the introduction of the popular Stachenfeld et al 2016 paper could be a reason for misinterpretation — it says that “according to SR place cells do not encode place per se but rather a predictive representation of future states given current state”. What is left unsaid until later is that what they call a state is precisely a location or a place cell, and that too because it is directly sensed.

Many properties attributed to SR are just the properties of the adjacency matrix/Markov chain.

If at every time instant what you sense is the unique location, then you can directly set up a Markov chain over those locations and obtain the same properties ascribed to SRs. Learning an adjacency matrix. — which location is adjacent to which location — is not a hard problem. It is just a Markov chain, you can just populate it with the transitions that are observed. The edges of that Markov graph can be annotated by the actions, and this Markov graph has all the info needed for navigation.

Note that any of the generic statements made about SR — ‘predictive representation’, ‘encodes temporal relationships’, ‘columns represent place cells’ — are true of this simple Markov chain too. In fact many of the properties attributed to SR — formation of hierarchies using community structure, for example — are just the properties of the underlying Markov chain without any SR. Navigation computations on the Markov graph are not expensive at all — you can use the Dijkstra algorithm for computing the shortest path, for example.

As another example, the fact that nodes in a neighborhood community of a graph cluster together when visualized in multi dimensional scaling is not a property of the SR — it is the property of the underlying Markov chain, as shown in the figure above.

SR is not modeling long-term temporal dependencies and is less predictive than the Markov chain: You can think of SR as taking in the adjacency Markov chain, and then ‘blurring’ it. It introduces more edges to the graph — a node is now not just connected to its neighbor, but also the neighbor’s neighbor, and neighbor’s neighbor’s neighbor. Sometimes this is confused with modeling ‘long-term’ temporal dependencies, but it is actually not. The addition of the extra connections did not increase the predictive power — it decreased it. This is because, after the blurring, the a node cannot distinguish between its neighbor and neighbor’s neighbor. The Markov chain representing the adjacency matrix is a better ‘predictive map’ than SR. The SR matrix will perform worse in prediction compared to the simpler Markov chain.

Eigen vectors and grid cells?: Another property attributed to SR is that its eigenvectors look like grid cells. But this is also just the property of the neighborhood graph, no SR needed. Shown below are first 20 eigenvectors of the graph of a room with a ‘hole’ in it.

(Here’s a simple argument on why this grid cell idea does not make intuitive sense to me: To derive this representation, it requires representing the ‘hole’ in as much detail as the walkable areas of the room.)

Questioning the ethology of SR assumption.

SR is motivated using computational complexity of an RL task — maximizing rewards when multiple states in the environment lead to rewards. For example, if different amounts of cheese are available at different locations in the environment as in the figure below (from Stachenfeld et al 2016), the RL problem is about having an optimal policy to obtain the rewards.

Here are my questions on this:

1. Is there any evidence that animals can optimize behavior in such settings where there are multiple rewards?

2. How are these rewarding states even communicated to the animal when the animal cannot see all the states? Seems like a hard problem.

Basically, I’m questioning the computational optimization premise behind SR — I’m asking whether animals/humans actually solve the RL problem that is taken as the assumption for motivating SR.

Navigating to a single goal location, or even chaining a couple of them, is not computationally inefficient in a model-based setting, but SR assumes that this is not sufficient and that the RL problem needs to be solved in the more general setting, the ethology of which seems questionable to me.

What about evidence for SR in humans?

The paper below (Mommenejad et al 2017) is often cited as evidence for SR in humans. It is definitely a very impressive and fascinating behavioral experiment in humans, but on closely reading the paper I am left with the impression that the paper is actually providing strong evidence for model-based reasoning in humans, and providing only weak support for SR.

Let’s look at Fig 5 of the paper, which provides the human experimental data overlayed on predictions from “Model-based”, and “SR”, for two tasks, let’s just call them “gray” and “red” for now. The gray and red bars in these plots are the theoretical predictions, the little dots with error bars are what is observed from humans. I have annotated the plot to make this easy to see.

The impression these plots give me is that 1) Model-based reasoning (“pure MB”) is a pretty good fit to what is actually observed in humans, and 2) Successor Representations (“pure SR”) is a bad fit for what is observed in humans.
"“Pure MB” — It is such a close fit, and SR is not predictive at all of how humans behave for the “red” condition. In fact, this plot seems to provide strong evidence against SR.

But that is not what the paper concluded. Here is the entire figure 5.

The paper utilizes the minor discrepancy between Pure-MB performance and human performance to argue for a hybrid model that post-hoc combines SR with MB, without actually giving a method for how this combination would be achieved by an animal. This post-hoc hybrid SR-MB is then what is cited as evidence for SR in humans.

The hybrid SR-MB does not make sense to me. The MB method has all the information to outperform SR in both the gray and red conditions. The motivation for SR rests on the idea that the MB computations are expensive, as described in the introduction of the paper.

However, running an SR-MB combination used in the paper (a linear combination of SR and MB) defeats the efficiency argument — SR-MB combination requires running both SR and MB, which is more expensive than running MB alone. The paper says there are other ways to combine them, but what was used for the fit was a linear combination that doesn’t make computational or ethological sense.

Latent-state formation could be an alternative explanation:

Could there be another explanation for the minor discrepancy between MB performance and the reported human performance? I think so. Interestingly, the alternative explanation is based on the idea of latent variables.

Consider the ‘red’ task, ‘transition revaluation’:

• Humans are first trained to memorize two 3-step sequences.

  • Seq1: 1→2→3→Reward1,

  • Seq2: 4→5→6→Reward2

• After they learn seq1 and seq2, they are trained with two two-step sequences.

  • Seq3: 2 → 6 → Reward2

  • Seq4: 5 → 3 → Reward1

Now, here’s the crux: How would humans interpret the second set of training sequences? What mental model do they build after the training is complete?

In the paper's interpretation, the participants will learn Model-1 at the end of the first batch of training that exposed them to seq1 & seq2. And according to the paper, the participants will consider the second set of sequences (seq3 & seq4) as edits of Model-1, transition-revaluation, leading to the participants to change their mental model to the highlighted Model-2 in the figure above. The blue arrows are the edits.

But that is not the only interpretation humans can make. Here is another interpretation.

• If the sequence starts at 1, then the next steps are 2→3→ Reward1. But if the sequence starts at 2, with no preceding context then the next steps are 6→Reward 2.

Interestingly, SR has no mechanism to represent such a contingency, but humans and animals actually can! Also, it is trivial to represent this in CSCG, as shown below:

Without additional information or constraints, there is really not much to select between these two interpretations. Some humans might modify the existing Model 1 and treat this as transition-revaluation, but some other humans might think of it as a new set of latent variables.

I think this ambiguity in the task design is a potential alternative explanation for the observed phenomena. The observations in that paper raise interesting questions given what we now know about the ability of hippocampus to create latent variables.

What about successor representations in animals?

Cothi et al did an ingenious experiment on both rats and humans to test for navigation under changing environments. The experiment is extremely interesting, and I highly recommend reading the paper. In my reading of the paper, the performance of the agent, in terms of how well it achieves the task, seems to best explained by model-based planning, not SR, but that is not the impression you’d get if you just scan the abstract.

The experiment tested the ability of rats and humans to navigate an environment and reach a goal that was initially trained in an open arena, but then tested in 25 different maze configurations that removed blocks to create barriers. The goal location remained the same. Each configuration was tested for 10 trials.

Shown below are the performance graphs, for rats, humans and RL agents.

Here are my observations from the above graphs:

1. Panel A: SR significantly under-performs compared to humans, very much so in the early parts of the trial.

2. Panel A: Human performance is very close to model-based reasoning “MB”.

3. Panel B: SR under-performs rats, and catches up to rat performance only by trial 10.

4. Panel B: Ideal MB performance is better than that of rats performance.

However, the paper seems to emphasize some aspect of the trajectories taken by the agents to find that SR is most similar — see plots below. However, even there, in the first 5 trials model-based planning is more similar to humans and rats compared to SR.

The paper argues that as the agent gains more experience, it makes sense that it switches to SR because it is less computationally demanding. But, it is just as easy to cache the plans that were derived earlier, and replan only when the the actions of that plan are not executable. In the MB implementation in the paper, the agent replans after every step — but replanning could be done intermittently. Many of these variations of model-based planning could make it even more of a better fit for the observed behavior.

Also, some of the reasonings given in favor of SR in the paper seem to be incorrect. when the environment changes, the SR’s biases are likely to hurt rather than help. In fact, this is reflected in the performance — SR significantly underperforms MB, and the animal/human in the first 5 trials.

What could be the reason for humans and animals under-performing the ideal mode-based planning? My guess: partial observability of the environments. Humans and animals in the experiment received visual sensory inputs, not location indices as inputs. The simulated ideal models received actual location indices as inputs! Therefore it should not be surprising that humans and animals under-perform this ideal model.

This brings up another argument why these experiments should be considered as evidence against SR: The SR model, which receives ideal location-indices as inputs, is under-performing rats that receive only visual sensory inputs!

It is clear that this paper raises a lot of interesting questions with an ingenious set of experiments, and I hope there will be more follow up studies. My overall take is that the experiments are not showing strong evidence for SR, but showing evidence for model-based planning.

Conclusion

I hope I have provided sufficient arguments and evidence for the following:

• Successor representations (SR) is NOT a model of place cell learning in the hippocampus. SR cannot explain how place fields are formed.

• Additional properties ascribed to SR, like “predictive states”, and ability to form hierarchies, are properties of simple Markov chains, and not a property of SR itself.

• The experiments in humans and animals that report evidence for SR might NOT be providing strong evidence for SR, and are more likely evidence for model-based planning.

Acknowledgement

Thanks to Miguel Lázaro-Gredilla for feedback on early drafts of this blog.

Subscribe now

Further reading:

Starter code: A Google colab Jupyter notebook is available with starter code. You can run it from the browser without installing anything, and it takes less than 5 minutes to train a basic model and see how it works. Click here for the Jupyter notebook.

The traditional view: Space-centric

Place cells — neurons that respond to locations — is a robust phenomenon that is a striking property of the hippocampus, and the currently dominant paradigm of hippocampus is of representing space in Euclidean terms. Hippocampus neuron activations are often visualized as place fields.

However, anomalies to this basic story have been accumulating with more a more new and divergent phenomena getting discovered. These are then explained by invoking new concepts and new types of cells, without a unifying principle connecting them all. Alternative views that look at hippocampus as a sequence model has been emerging. See this paper for example, and many more are cited in our paper. Although these views pointed in the right direction, they were not yet brought to a concrete model that can resolve how space gets represented as time, and how that can resolve many of the mysterious phenomena. Read the recent review ‘Remapping revisited’ to see how confusing the picture is.

What we show in our paper is how these anomalies get resolved when we stop thinking of hippocampus in spatial/Euclidean terms, and instead think of space/place as a sequence. In the next few sections, I try to explain these ideas using some visualizations that make it easier to understand.

The problem that hippocampus has to solve

Humans and other mammals do not have a GPS or compass, & cannot sense location coordinates. Our ideas about space & locations have to arise from our sensory-motor experience. ( Or hard-coded a-priori à la Kant, more on this in a later section.)

The trouble is that sensations + actions do not have a one-to-one correspondence with locations. Due to perceptual aliasing, identical sensations can occur at different locations, and the same location can be approached via different sensory-motor sequences. The idea of space has to arise from this sequence of of aliased observations that do not convey locations directly. But how?

Interestingly, our model, clone-structured causal graphs (CSCG) can do this exact thing. Given a sequence of aliased egocentric sensory-motor inputs, it can induce a latent graph that corresponds to the generative model of the environment as the agent experienced it. This also works in 3D environments with rich high-dimensional visual sensations as the input, and on arbitrary non-Euclidean topologies.

How CSCG represents different sequential contexts in a latent graph

Shown on the left in the figure below is the top down view of a 3D environment, with three action sequences marked in orange, green, and purple. These action sequences generate visual observation sequences A→D→E (purple), B→ D→ F (green), and C→ D→G (orange). The visual observation D repeats in all three sequences. In the green and purple sequences, D occurs in the same location, even though they occur in different sequential contexts, but the D in C→D→ G (orange) is in a different location.

The latent space representation of CSCG has multiple states that are “clones” of the same observation. These clones can be used to represent the different (latent) temporal contexts that an observation can occur in, and has the flexibility to either merge or split these contexts. For example, the D in the purple and green sequences need to be merged together into the same latent clone even though they occur in different contexts, where as the D in the orange sequence needs to be kept in a different latent clone. Both these clones are connected to the same instantaneous observation D, but they will activate in different temporal contexts based on the clones that connect to them.

There are two types of connections in CSCG. 1) `Lateral’/temporal/transition connections: These are the connections between the latent clones, and same as the weighted edges of the latent transition graph. 2) ‘Bottom-up’ connections: These are the connections from observation to the clones. Each observation’s activation strength is the `bottom-up’ input into each of its clones. The overall activation of a clone is determined by combining the bottom-up input with the lateral input.

Shown below is a visualization of the latent clone structured graph and how a learned graph corresponds to the topology of the environment. CSCG is initialized by allocating a clone-capacity — the number of clones for each observation. Typically over-allocating the capacity works better for learning. More details about the learning dynamics is discussed in the paper, but for now let’s understand what this graph represents. This is shown in the figure below.

Keep in mind that what is learned is a graph. We can see the correspondence of that graph with the environment when it is visualized in 2D, but the 2D layout is something we impose externally. The correspondence to the learned environment is not obvious when the graph is plotted out in the ‘clone-structured’ layout.

Place fields: How sequence neurons get interpreted as responding to locations

Which observation does each of the clones in the graph respond to? That is easy: The clones are colored by the observation neuron that is connected to them. For example, the gray colored neurons respond to gray color. Importantly though, they respond to the gray color only when it occurs in specific sequential contexts. This is visualized below.

So far, we have been discussing only latent graphs and sensation sequences. We haven’t invoked anything about space, geometry, location etc. The clone neurons in our graph respond to specific sequences. Then how does this get interpreted as responding to locations? Let’s do what is done in neuroscience: plot the “place field” of a clone neuron. We’ll see that plotting the place field is what injects the interpretation of location/space into the neural responses.

To plot the place field we do the following:

• Insert a probe into the rat’s brain to record from the neuron of interest.

• Make a 2D plot that is the same shape as the room

• Let the rat run around in the room, while we observe the neuron and rat’s location simultaneously.

• When the neuron fires, we look at the rat’s location at that instant. We add the firing strength of the neuron to that location in the 2D plot, and accumulate this over time.

These steps are visualized in the figure above, and the place field of the marked neuron comes out as shown.

Note that the decisions to make a 2D plot in the shape of the room, and to overlay neural responses based on rat’s location were made by the experimenter. The animal itself has no mechanism to plot a place field (because plotting it requires knowing the ground-truth location), and it has no use of plotting one either.

Place fields make us think the neuron is responding to location: If you plot the place fields of the different clones above, you will see that each neuron responds at a different location, and they systematically cover all the locations in the room. This makes us think that these neurons are responding to locations.

But in reality, the neurons are not responding to locations. They are responding to the sequence of sensory observations that lead to the location.

Crucial distinction: A neuron responding to the sequence of sensory inputs leading to a location is not the same as the neuron responding to the location. Mixing these up is the source of a lot of confusions related to place-field remapping, as we will see below.

Explaining geometric determinants of remapping without space or geometry!

Consider this question: What will happen to the place field of the neuron above if we train the rat in the original room, but test the rat in an elongated room when we are plotting the place field?

Since we know how to plot place fields, and since we have a model, we can run this experiment directly, as visualized below.

We just repeated the same process for plotting the place field, this time making a larger 2D plot corresponding to the shape of the elongated room, and marking and accumulating the neuron’s response on to the current location of the rat in the elongated room.

We see that the original place field ‘remapped’ into a bi-modal place field in the elongated room!

In fact, we just reproduced a classic finding from a well-known paper called ‘Geometric determinants of the place fields of hippocampal neurons’, but in CSCG the remapping occurred without having to use any geometrical concepts at all!

’Why does this remapping happen? This is easy to understand using CSCG, as shown in the figure below. In the elongated room the sequential contexts that fired the neuron in a single location in the original room now occur in two different locations in the elongated room. Remember that the neuron was never responding to locations — it was always responding to the sequence of observations.

Prior models to explain this phenomena relied on geometric concepts, like the Boundary-vector-cell model, which, as you can guess, relied on ideas of cells coding vector distances from the boundary of the room. But that model could not explain the direction-dependence of the remapping, the result of another ingenious experiment by O’Keefe and Burgess. They asked: “What if we plot separate place fields, one while the rat is running to the left, and another while the rat is running to the right?”. Well, you can read off the answer from the figure above. When the rat is running to the right, the left lobe of the place field will be active, and when the rat is running to the left, the right lobe of the place field will be active — this is what they observed, and also reproduced readily from CSCG.

Another key thing to note is that, in this case, no connections in the model changed, even though the place field changed dramatically. Although the place field now looks very different, the model has remained exactly the same. The opposite can also be true: there are cases when the model’s connectivity can change, and the animal’s behavior changes accordingly, but the place field can remain largely intact. As we stated earlier, these phenomena look puzzling only when we look at hippocampus neurons as encoding location, and our paper explains how these get resolved.

If you understood the mechanics of how the above remapping happens, you are in position to understand a vast array of phenomena in the hippocampus. Here’s the full list of phenomena we considered and explained in the paper. CSCG can explain many more, but we had to be selective given length constraints.

Place field expansion, distortion, and repetition

In large empty rooms place cells in the center of the room have a larger place fields compared to those in the periphery, and the place cells along the edges of the room have elongated place fields. When we train rats in two identical rooms that face in the same direction and are connected by a corridor, place fields often repeat in these rooms, in the same relative positions. How does CSCG explain these phenomena?

As the size of an empty room increases, the aliasing of the observed sensory sequences increase in the center of the room. This results in CSCG being not be able to learn a perfect graph — the graph quality will be degraded in the middle of the room in the sense that it will not resolve many of the aliased sensations into different clones. The degraded graph is still useful, but will have ‘lower resolution’ in the center. If you plot the place fields of the clones in the middle, just like how we did earlier, they will show an expanded field compared to the neurons that represent the corner or the periphery. In the periphery, aliasing will be elongated along the edge where identical observations happen, resulting in elongated place fields.

A similar thing happens when two identical rooms are connected by an undifferentiated corridor. Learning often doesn’t recover the two rooms then, recovering only a graph like the one shown above. This will cause the place field to repeat in the identical rooms, in same relative positions.

Place field repetition in two identical rooms, and place field expansion in the center of an empty room happen for the same reason. Imperfect learning that result in aliased latent graphs. Expansion is same as repetition, just happening locally.

CSCG schemas for structure transfer

Although we didn’t cover this in detail in our paper, it is worth knowing how CSCG latent graphs can be used as schemas.

The idea is quite simple, and is visualized above. The nodes in the latent graph are mapped to observations via the ‘emission matrix’. For example, all the green clones in the graph on the left is connected to the green observation, and all those clones represent the green floor of the room. However, we can disconnect the specific observation mapping that was learned in a particular room. What we are left with is a latent graph with ‘slots’ in them.

If the agent now encounters a new room with similar structure, but different appearance, it can re-use the structure and just relearn the mappings of the latent nodes to observations.

You can think of the original graph that was learned for a specific room as a ‘grounded graph’ — the nodes are colored by the observation mappings. If you remove this coloring, then what is left is a ‘schema’, or an ‘ungrounded graph’, but we can learn a new color-map for this graph in a new environment by ‘binding’ the ungrounded nodes. This learning process just needs adapting the emission matrix while the transition matrix is held fixed — this learning is extremely fast.

CSCG schemas are structural abstractions and they enable transferring structural knowledge from prior experience to deal with new environments. In general, animals would be using a library of such prior knowledge schemas, rather than learning from scratch every time. These ideas are developed further in two of our papers on 1) ‘Graph schemas for transfer learning’, and 2) `Schema-learning and rebinding as explanation for in-context learning’ (NeurIPS 23 spotlight). The idea of learning latent graphs and using those for task-transfer generalizes to other domains — for example, you could feed the model word tokens instead of sensory observations. Learning the structure of a sentence or an algorithm is in many ways similar to learning the structure of a room.

Planning and replay

As I mentioned earlier, the animal has no need for decoding locations or calculating place fields. Any navigation related decision the animal needs to make can be done in terms of the latent states without having to ever invoke ideas of space and location. Here is a visualization:

One core advantage of the CSCG is that the latent representation is an explicit graph structure. This makes planning is easy and efficient, and makes it easy to self-modify the model in response to environmental changes and replan. The latent graph can also be used as schemas to efficiently transfer structural knowledge.

Schemas and planning+re-planning gives CSCG some interesting properties that are also observed in animals. While encountering a new environment, the agent can utilized previously learned schemas to rapidly learn the environment. Moreover, the agent can take shortcuts in the new environment, and these shortcuts can be through portions of the environment that the agent hasn’t visited yet, as shown in the figure below.

As humans we frequently encounter such scenarios. Imagine walking in the 3rd floor of an office building after experiencing the second floor that also has the same layout. Although the 3rd floor has the same geometric layout, the decor, furniture, color scheme etc. could be very different from the 2nd floor. You can still plan paths through unvisited sections in the 3rd floor using your knowledge of the 2nd floor, but you would not be able to predict what objects or colors you’d encounter on the paths. These are examples of planning using schema-like abstract representations.

The following figure visualizes replay-like message passing for planning and replanning while using a schema to navigate a new environment that has a similar (but not identical) layout.

What about grid cells?

The picture consistent with CSCG is that grid cells are a complementary input — you could consider them as another modality that inputs self-motion information. This information is helpful, and can augment visual information, especially in uniform open arenas. The periodic tiling provided by grid cells is useful in reducing the aliasing of sensory inputs.

The interaction between grid cells and CSCG will be bidirectional — if the CSCG is trained with a particular phase of the grid cells, the latent representations can help the grid cells align with that familiar phase when the animal returns to that environment.

This combined model is yet to be developed and demonstrated, but CSCG is consistent with recent neuroscience experiments showing that grid cells are not necessary for place cell function. I highly recommend watching this video.

Other models, future work etc.

The paper has a lengthy discussion on connections to other theories of the hippocampus, but I plan to write a blog that will expand on these. I also plan to write more about the directions in which our research can be extended. If you are interested in these topics, consider subscribing:

Subscribe now

Also read:

Ingredients of understanding. How understanding in language models is different from human understanding.

Concepts as cognitive programs

Let’s start by looking at Figure 1 of our paper (https://www.science.org/doi/10.1126/scirobotics.aav3150). You can see that ARC’s premise is the same — (A) from input-output image examples, infer the abstract concept that is conveyed. The concept can then be applied to a new image to create an answer image (B). Our setup went one step further, on being able to transfer this concept to real-images (C) and even being able to execute that concept on a robot (D).

We called this conceptual world “Tabletop world” (TW). More examples of concepts from the tabletop world are shown below, along with an induced program to solve a concept.

Some concepts in ARC are almost identical to the ones in the tabletop world, like the examples below. However, despite the similarity, there is a core difference between tabletop world and ARC: ARC is not systematic.

“Tabletop world” (TW) is systematic, ARC is not.

While the abstract images used in ARC has many similarities to our ‘cognitive programs’ work, one crucial difference is that TW is systematic and ARC is not. By systematic, I mean that the concepts are derived from a physically consistent simplified world that is a subset of the real world. Let me explain:

All the ‘abstract’ examples in TW are generated from a systematic domain:

• Objects are on a table top. What is seen is the top-down view. Objects are 2D. They obey simple laws of object interactions.

• All the conceptual manipulations are achieved by sliding the objects along the tabletop, without lifting. Objects do not occlude each other.

• Some manipulations involve imagining objects that are not present, and sliding other objects in reference to this imagined object.

While an infinite number of concepts can be generated in the TW domain (everything in TW should be an ARC challenge), the domain is still closed. One cannot generate arbitrary concepts in this domain. Here is a relevant passage from our paper about this:

In contrast, ARC is not generated from a systematic domain like this that is a proper subset of the real world. The nature of ARC seems to be “whatever Chollet could think about and create in a reasonable amount of time”. The open-world nature of it creates significant challenges for attacking it, especially using the idea of “concepts are programs”. To understand this, let’s examine the core ideas behind how we tackled the TW concepts.

Programs on a “Visual Cognitive Computer” with an embedded world model.

If concepts are programs, then they need to run on a “computer”. This needs two things — the computer architecture, and the instruction set.

If you look carefully at the figure below, you’ll see that the computer architecture has a world model embedded in it. This world model consists of a vision hierarchy, a dynamics model, and controllers for hand, fixation, and top-down attention.

The instruction set specifies how this world model is controlled internally for running simulations.

Those are two key ideas:

• Having a controllable world model

• Being able to run mental simulations using this world model.

The instruction set of this computer is just the primitives for controlling the world model.

If ARC is to be solved using ‘concepts as programs’, then any solution to the ARC will have to have an architecture similar to this that incorporates a controllable world model. The details might vary, but the we think we identified some of the core components of this architecture and some of the core operations in the world model.

One key aspect of this world model is that the the vision hierarchy (VH) is bidirectional, with tight coupling between the recognition pathway and the imagination pathway, allowing it to simulate the (simplified) world dynamics in a factorized and context-appropriate manner. We elaborated on this further in our paper:

For the tabletop world, it was reasonably easy to specify an overall cognitive architecture like this. For ARC, since it is not very systematic, specifying this architecture and instruction set becomes hard, and somewhat arbitrary.

Which problems are unsolvable because the vision hierarchy used does not have the correct operations in it? Which problems are unsolvable because our program induction and transfer methods are not working well? Which problems require elementary school math? These kinds of things becomes harder to debug when the domain is not systematic.

Pure reason you Kan’t

Although ARC is supposed to be a challenge for ‘abstract reasoning’, many of the challenges piggy-back on the generalization provided by the perceptual system. Consider this one:

What is the ‘reason’ for filling in the those specific red pixels? It looks reasonable because of the prior provided by our perception system. How this prior is systematically brought into use to induce concepts is the trick, and that would then include work on perception, not just abstract reasoning. I don’t think any of the existing artificial perception systems have all the right priors or provide right amount of control.

The above example shows that some ideas of “objectness”, “contour continuation” are expected to be present in the system for ARC to generalize. Other examples also use the ideas of “inside/outside”, “closure” etc. Those priors have to come from the perceptual system.

But then why is occlusion not included? What about transparency? Who determines which perceptual generalizations are OK and which are not? None of these are specified in the ARC challenge, and the decisions seem to be somewhat arbitrary.

Have a look at the example below. This is about changing the color of the blue crosses to the one in the target. Note that humans will be still be able to solve the task even of the other crosses were bigger in size, or rendered in different way, only by the outline for example. Even if they looked nothing like cross, but the pixels spelled out the word ‘c.r.o.s.s’., humans who can read English would be able to solve it. Which of these generalizations are within the domain of ARC, and which are not?

Note that developing a perception system with the right priors, the right amount of control, and with appropriate world dynamics modeled in it is still a research challenge. What looks like pure abstract reasoning often depends on inheriting generalizations from such a system.

Like other tests, ARC is necessary, not sufficient.

None of these problems would matter if ARC was a sufficient test, but just like Tabletop world or Ravens Progressive Matrices, it is just a necessary test. If that is so, why not make a series of systematic necessary condition tests instead of a test that is not systematic?

Parting thoughts

When all the excitement is around LLMs, it is refreshing to see the focus on abstract reasoning, and non-linguistic concept induction. I do believe this is an exciting and important problem, and ARC prize brings attention to it. However, I think the challenge itself is hobbled by some of the problems I described above — lack systematicity, entanglement with perceptual priors without clear acknowledgement or delineation, etc.

Some of these problems might be why there hasn’t been verifiable systematic progress on this problem despite ARC being out for a while and having grabbed attention before. Maybe, it is possible to divide ARC into different systematic domains, which are then tackled. Many papers select subset of ARC problems to work on, but these often result in dead ends with no path to solving the other problems in the challenge. However, with renewed enthusiasm and a bigger prize money, it is also is possible that we make progress despite those problems. Hopefully the perspective from our prior work helps:

Further reading

• Science Robotics article (open access): Zero-shot task-transfer on robots by inducing concepts as cognitive programs

• A thought is a program . And old blog from Vicarious at the time of publication of the above paper.

• Cognitive programs: software for attention’s executive.

Subscribe now

I removed both wheels of my bicycle. How do I make it stand upright on the floor?1

How did you, as a human, answer this question? Most likely you imagined how a bike looked after wheel-removal. You then imagined balancing this bike and realized that it can be supported on the floor by the fork and the crank case. Or maybe you concluded differently, despite following a similar process.

Where did the knowledge for running this imagination — mental simulation — come from? Did you acquire it from reading about balancing bike frames, or did it come from your sensorimotor experience? For most, this knowledge is acquired through sensorimotor experience in the real world. Even for a person who has never mastered language, their experience with physical objects combined with their knowledge of bike geometry is sufficient to run this mental simulation.

The process a human goes through in processing a natural language prompt is very different from the process a large language model (LLM) goes through. Here are some salient points about how language is understood in humans:

• Language understanding involves mental simulations on a world-model.

• This world-model cannot be acquired from language alone.

• Language is a mechanism to control mental simulations and world models in other humans. Internal monologue is a special case of this.

As shown in the figure below, language is a thin layer that indexes into the sensorimotor simulators that constitute the majority of our world model. A good chunk of this world model can be learned without language, although language can definitely help. Thinking and imagining often requires coordination of the linguistic and non-linguistic simulators. Of course, some questions can be answered quickly and correctly purely in the language system without having to ‘descend’ down into the sensorimotor simulators.

Mental simulations are situated in sensorimotor context.

In the bike-balancing example, the prompt was purely in language, and you could run mental simulations with your eyes closed. But in general, mental simulations need to absorb the current sensorimotor context to run the correct simulation.

I have a favorite example for how language simulations are situated, thanks to my colleague Felix Hill. If you hear the sentence “The haystack was important because the cloth ripped”, it might not make sense to you at all. But if you hear that sentence in the context of the picture at the end of this section, it will make immediate sense even though there is no haystack in the picture.

If you were to find yourself in that unfortunate cloth-ripped situation, you want to be running contextually appropriate mental simulations that combine perception, sensorimotor experience, and conceptual knowledge. If you were lucky to have a lake as another option, your acquired-via-language knowledge about crocodiles in that lake might temper your enthusiasm. Thankfully, these kinds of mental simulations are always happening in our brains to help make decisions and drive behavior, with our without language.

A rich world model cannot be acquired from language alone

It is not practical to acquire a human-like rich world model through language alone because it is not possible to convert all the sensorimotor details into language efficiently2.

Imagine accidentally dropping an object you were holding. Where you’d look for that object depends on the particulars of the physical context you were in. What was the geometry of the objects around? Could the object bounce? Could it roll off and fall through a crack? It is impossible to describe the scene in all the detail in language because an a-priori unimportant detail could become crucial in this particular context — maybe the object was light and could be blown away by the wind coming through an open window. It is not possible to convey just the relevant details either because deciding what is relevant itself requires a contextually appropriate mental simulation. Without the proper context, an LLM’s answer — look for the object on the floor — is too generic and ungrounded.

While it is quite interesting that transformers seem to acquire implicit world models for games like Othello from text alone, that should not be considered as evidence for language-only models getting to human-like rich and dynamic world models.

Real-world commonsense is not a language-only problem.

Winograd schemas — language problems that humans typically solve by imagining the physical objects — were originally created as tests for similar capabilities in language models. LLMs now successfully solve many Winograd schemas, and this has got some people thinking that the commonsense problem is largely on the path to being solved using language alone. Word-to-word coherence can solve some commonsense query problems may have been a surprise to many, but that surprise doesn’t justify the conclusion that word-to-word coherence will solve all of commonsense.

In general, it is painfully laborious to convert physical commonsense scenarios into natural language. Winograd schemas were clever and popular because they were examples of commonsense questions that were easy to pose in language without appearing too contrived. The lesson to be taken away from this is not that LLMs have solved commonsense — the lesson might be that commonsense questions that are easy to pose in language might also be “solved” using language alone.

It is not that new commonsense language queries that defeat language models do not exist — they do. It is just that those queries will look increasingly more contrived when expressed in language. This is not because the scenarios themselves are contrived or infrequent — it is just unusual for humans to express such scenarios in language.

Sensorimotor inputs are not language.

In a simplified sense, language is compressed code that indexes into a sensorimotor codebook that is shared between the sender and the receiver. Unlike a typical Shannon-like communication system, human language has the additional complexity of feedback (receiver can ask questions), and adaptivity (codebook itself can change based on prior transmissions). Treating sensory input as “language” doesn’t achieve anything because if that sensory information is compressible and shared experience among multiple agents, then a shared codebook and a new language will be formed on top of it.

Minimal machinery for understanding

Here’s a list of what I think is the minimal set of ingredients for the machinery of understanding.

• Ability to construct rich sensorimotor world models through observations and interactions, and ability to query them in context-appropriate ways. This world model should be: 1) planning compatible, 2) causally structured, 3) rapidly modifiable, and 4) support counterfactual simulations.

• Ability to modify these world models by thinking

• Ability to seek information based on current models and uncertainty, both for modifying the models and for making decisions.

• Ability to generate a hypothesis based on the world model and to test that out in the real world.

Of course language amplifies the effects of this core machinery by helping us rapidly acquire knowledge generated by other humans over the years and to rapidly share any new knowledge/understanding we develop through the exercise of this machinery. Note that language itself was a product of agents with this machinery interacting with each other. My view is that understanding preceded language3.

The idea that humans share an understanding machinery doesn't contradict the fact that different people can have different levels of understanding of the same concept. A professional mathematician’s understanding of the concept of a ‘vector’ is richer than that of an average high schooler, purely because the mathematician has applied the understanding machinery to the concept of vector in multitudes of contexts to build a richer model. The high schooler can still get to the same level of understanding by going through a similar process — but a language-only model would not develop human-like world models no matter how much text it reads.

Tests designed to probe the mastery of a subject in humans assume that this understanding machinery and process is shared. When we say a child as having understood something, we assume that the child utilized its understanding machinery and went through a process of model-building and simulation. Understanding is both the current state of knowledge, and also the process one needs to go through to reach and update that knowledge. In humans both these are interconnected, and we take this for granted in our interactions with other humans, and in our tests of their understanding.

Human-like understanding is worth understanding

Of course one could argue that LLM has an understanding that is superior than that of humans and therefore we should not care for human-like understanding. But it could also be like settling for balloon flight as an alternative for heavier-than-air flight, as I argued in a previous article. If human-like understanding is fundamentally different, it is worth knowing why and how, both as a scientific puzzle and as a challenge for building smarter machines. My hope is that we continue investigating until we really understand what constitutes understanding.

Subscribe now

Further Reading

My twitter thread on MalayaLLM thought experiment.

Barasalou’s Perceptual Symbol Systems paper.

Our work on cognitive programs: An example of bringing perceptual simulations into abstract concepts.

More about commonsense, general intelligence and the brain: From CAPTCHA to commonsense

1

In case you were wondering how GPT-4 answers this question.

• 1903: Wright brothers invented the airplane

• 1919: First non-stop transatlantic airplane fight.

Now try guessing this: In which year did the Hindenburg accident happen?

Many are surprised to know that it happened in 1937, more than 30 years after heavier-than-air flight was invented!

Of course it is well known that balloons and airships based on hot air, hydrogen, and helium existed before airplanes. But did you know that “For the first thirty years of the twentieth century airships were viewed as a more robust means of transportation than the airplane, consistently surpassing them in range, flight duration, and load-carrying capacity.”? (From the preface of When Giants Ruled the Sky)

A similar situation exists in AI today. The exciting success and rapid progress of large language models in partially, but usefully and impressively, solving many language processing tasks has triggered the thinking that human-like general intelligence can be attained by just scaling up the underlying technology. However, the story of aeronautics should offer us caution: Although scaling to bigger sizes was all that was needed to make balloons carry heavier cargo and fly for longer durations, those advances, while exciting and useful, were on a different path from the airplanes of today.

A clarification is warranted up front: I do think large language models are exciting and useful. OpenAI took a big risk and did marvelous research & engineering to show the world the promise of this, and I’m happy with the success and attention they are getting. I also think many insights can be learned from studying transformer architectures. The point of this article is not to deflate (and I don’t think I’ll be successful in doing so even if that was the intention) excitement around LLMs, but only to add perspective in the context of the longer term goal of human-like general intelligence. I also want to offer hope for people who might want to think differently, while keeping in mind the cost of what they might miss out on, without taking away from the excitement of the moment.

Different ways to fly. Different ways to solve human-like tasks

If flight is defined as traveling through air from locations A to B, there are different ways to fly. Catapulting is one. Ballooning is another. Neither are based on how birds fly.

Even before heavier-than-air flights were invented, balloons were very popular, and used extensively. This is because figuring out the principles of heavier-than-air flight was a much harder task — it was just very hard to keep an object that is heavier than air controllably up in the air for a long time. Balloons avoided this problem altogether because they were lighter than air. The success of balloon-builders over people trying to experiment with heavier than air flight was so thorough that a New York Times article in 1903 declared that any attempt at flying other than using balloons is unlikely to succeed in a million years, just two months before the Wright brothers achievement!

In the backdrop of pessimism surrounding heavier-than-air flights in the early 1900’s balloons offered this exciting possibility: Without having to figure out the principles of aerodynamics, we can build machines that travel through air and make them carry heavier payloads and go further distances simply by building them bigger.

Since most of the success of large language models arise from making the underlying transformer model bigger, and training it on more text (trillions of tokens), and training it using more compute for longer duration, transformer-based language model offers an intriguing possibility just like the balloons did in early 1900’s: Without having to figure out the principles behind human intelligence, we could build machines that solve more cognitive and human-like tasks simply by building them bigger, and training them with more data, compute, and human feedback.

Dirigibles was exciting technology, and so are large language models, and the scaling of other models.

Similar to today’s large language models, Dirigibles was exciting technology at their time. From 1890s, Santos-Dumont built a series of steerable airships numbered 1 to 13, all working on the same principles, but successively making them bigger, more controllable, and safer. Like the parameter counts of language models today, the size of the balloons were an important aspect in carrying capacity and controllability. Santos-Dumont No.4 built in 1900 had a gas capacity of 420 cubic meters. Santos-Dumont No.5 increased it to 622 cubic meters, and No.6 increased it slightly to 630 cubic meters. These were used in awe-inspiring and well publicized flights in France, and captured the imagination of the general public.

Just like how large language models required bringing together large-scale computing, GPUs, software engineering, and advances in neural net architectures, dirigibles were engineering marvels that required the integration of the latest in materials, structural engineering, propulsion, handling of hydrogen, and navigation. Dirigibles were awe-inducing sights in the sky, and their interiors dwarfed today’s airplanes in space and luxury. The excitement around dirigibles were so high even in the 1930’s that the builders of the Empire State building advertised plans for a mooring mast (an example of an API) atop it by publishing fake photographs of the dirigible USS Los Angeles docking there!

It is alright to be excited about building large neural nets.

Heavier than air flight problem was a north-star for some folks, and they might have found the excitement around balloons and dirigibles disheartening. Similarly, people who are interested in building real human-like intelligence might be disheartened by the exuberance around large language models. However, this need not be the case.

Dirigibles were a technology whose time had come, and ‘Foundation Models’ is a technology whose time has come. Technological advances require many different critical components to come together at the same time. If any one of the them is missing, the advance doesn’t happen. And when they do come together, advance happens very quickly until you get to the edge of the capabilities of the supporting technologies.

Once the basic principles behind dirigibles were figured out, they had a favorable scaling law going for them — to go further distances, and to carry heavier payloads you simply had to make them bigger, and give them more powerful engines. This was purely an engineering task. Although it has its own challenges, it is easy to organize teams around those to create generations of increasingly bigger models because each generation informs the next.

The large models simply have the right things behind them right now, and there is no stopping them until they exhaust their own runway. It will be a wild ride, and not every effort will succeed, but in the process many useful things will get built. Many companies will be created, and many will become wealthy and build careers on that. We will understand how far we can go just by scaling. We will learn, sometimes through painful experiences, the different failure modes of deploying imperfect models widely, and learn mitigation strategies which might include new regulations. All that is part of bringing a new technology to the world.

It is also OK not to be excited about just scaling up.

Excitement around LLMs doesn’t mean everyone has to be equally excited about them. We make progress by having people willing to question the dominant paradigm and strike out on paths that are new. It is heartening to see that pioneers of deep learning are among the ones exploring out alternative paths1 from just scaling up the current architectures. Despite their success, they remain hungry, foolish, and curious.

Again, the analogy to how heavier-than-air flights developed offers some perspective. When people figured out how to make heavier than air flight, they were deployed on problems that were considered ‘toy problems’ for balloons. They didn’t stay in the air for nearly as long as balloons could, or carry as much cargo as balloons could. And initial deployments of airplanes were in niches that took multiple breakthroughs to expand out from. When Wright brothers wanted to report on their successful flight, the Associated Press representative initially turned it down because the the plane flew for a mere 59 seconds, well below what balloons were capable of doing at that time.

Ultimately, balloons had fundamental scaling and controllability problems that heavier than air flights didn’t have. But it took multiple breakthroughs — first figuring out aerodynamics and lateral control, then figuring out scalable mechanisms to build those (eg, ailerons instead of wing warping), building jet engines, etc. — to convincingly demonstrate that for long-distance applications.

Similarly, some small set of researchers, engineers, entrepreneurs, and investors will stake out on different paths to find the principles of intelligence. There is sufficient evidence for problems with scaling and controllability of language models to warrant such pursuits. And as they figure out more of the principles of intelligence, more efficient and controllable architectures will emerge. But of course they will be compared to large language models which would have gone through multiple stages of engineering by then, so they might initially be deployed in niches or in complementary situations.

Parting thoughts and future articles

I hope the story of dirigibles offers encouragement for both people who are scaling up, and for people who are exploring other directions.

One criticism of this comparison is that the transformer architecture might already be like heavier than air flight in the sense that it already encapsulates the principles of intelligence. One can never be 100% sure, but there are sufficient reasons to believe this is not the case: the autoregressive architecture has fundamental limitations in learning efficiency, flexible reasoning, mixing in episodic memory, and controllability. Moreover, the ELIZA effect of language often makes us see more than there is while interacting with language models. Like Wright brothers learned from birds that flapping wings is not necessary to fly2, using neuro and cognitive science insights to learn planning- and causality- compatible architectures is an exciting frontier. I plan to write more on this in the future.

I’d be surprised if people who are scaling up needed encouragement. This is the scale up moment. Seize it, run with it, don’t look back! The outputs are already exciting and more is to come. And hopefully you are secure enough that you won’t find the comic I have below demotivating in any way.

People who are exploring different novel ideas probably needs more encouragement amidst all the current excitement. There are sufficient reasons to believe that this moment in history is like dirigibles — exciting technology that shows the wonderful promise of real intelligence that is yet to come. Whenever someone is too self-congratulatory and smug about the current progress, maybe this comic will help you find some perspective.

I also think there are also avenues to combine the strengths of current approaches with an investigation into future architectures. For this reason, even people who are exploring new directions should remain curious about the current ones and study them in detail and avoid hasty dismissals.

I plan to write more about these topics in the future. In particular, a few articles that have sketched out are about “world models”, “what is understanding”, “the sweet lesson behind bitter lessons”, etc. If you are interested in these topics, please subscribe.

Subscribe now

You might find these links interesting:

I gave two 10-minute presentations at the AGI debate related to this. Check it out here. Presentation 1: exciting paths forward in AI & Presentation 2: Commonsense needs mental simulation

Space is a latent sequence: Check out this 15-minute presentation that will change your mind about how brains learn and represent space. And read our paper on that: https://arxiv.org/abs/2212.01508

Read this paper by Yann LeCun: https://openreview.net/pdf?id=BZ5a1r-kVsf

Read this paper by Anirudh Goyal & Yoshua Bengio: https://royalsocietypublishing.org/doi/full/10.1098/rspa.2021.0068

Follow me on Twitter: @dileeplearning

Check out My website

Disclaimer: Views expressed here are my own.