>there are also avenues to combine the strengths of current approaches with an investigation into future architectures.
Yes, hybrids should be powerful. Especially when new architectures can read from the intermediate layers of LLMs (and, perhaps, write to them).
This is where this really beautiful analogy with LLMs as dirigibles stops working. A hybrid between an airplane and a dirigible is not a promising idea. A hybrid between a shiny new architecture and an LLM might be what one needs to progress from excelling in "toy problems" to convincingly beating stand-alone LLMs across the board.
>there are also avenues to combine the strengths of current approaches with an investigation into future architectures.
Yes, hybrids should be powerful. Especially when new architectures can read from the intermediate layers of LLMs (and, perhaps, write to them).
This is where this really beautiful analogy with LLMs as dirigibles stops working. A hybrid between an airplane and a dirigible is not a promising idea. A hybrid between a shiny new architecture and an LLM might be what one needs to progress from excelling in "toy problems" to convincingly beating stand-alone LLMs across the board.