>> There is no question that AlphaCode (Li et al. 2022) is an impressive
accomplishment, indeed an astonishing one. The input in English specification
are long and complex. The code that AlphaCode produces is often 20 lines long,
intricate, clever, and by no means cookie-cutter. The relation between the
English specification and the characteristics of the code is quite indirect.
That is not right. The capabilities of AlphaCode are sub-par compared to the
state-of-the-art in program synthesis. People are excited about AlphaCode
because they are not aware of the state of the art in program synthesis, or even
neural program synthesis, or program synthesis from a natural language
specification. Compared to the state of the art on all those subjects, AlphaCode
is at best incremental work and at worse a regressive approach that does not
benefit from newer developments. DeepMind most likely recognise this, so for
example, they only compare their system against other neural program synthesis
systems, rather than conventional program synthesis systems, that can perform
significantly better.
More to the point, AlphaCode's ability to learn from a natural language
description is a red herring. The coding problems it solves come with examples
of the inputs and outputs of the target programs. Given those examples, an
inductive program synthesis algorithm can solve programs like the ones in the
coding contests attempted by AlphaCode _without_ recourse to the natural
language descriptions. The natural language descriptions are used by AlphaCode
only to try and guide its large language model to generate code that is
relevant to the programming task, but as the article observes, that doesn't seem
to work all that well: AlphaCode's code generation step is more like a firehose
that spams millions of programs in the hopes that it might get lucky and
generate a few that solve the programming problem. This happens rarely.
>> There is no question that AlphaCode (Li et al. 2022) is an impressive accomplishment, indeed an astonishing one. The input in English specification are long and complex. The code that AlphaCode produces is often 20 lines long, intricate, clever, and by no means cookie-cutter. The relation between the English specification and the characteristics of the code is quite indirect.
That is not right. The capabilities of AlphaCode are sub-par compared to the state-of-the-art in program synthesis. People are excited about AlphaCode because they are not aware of the state of the art in program synthesis, or even neural program synthesis, or program synthesis from a natural language specification. Compared to the state of the art on all those subjects, AlphaCode is at best incremental work and at worse a regressive approach that does not benefit from newer developments. DeepMind most likely recognise this, so for example, they only compare their system against other neural program synthesis systems, rather than conventional program synthesis systems, that can perform significantly better.
More to the point, AlphaCode's ability to learn from a natural language description is a red herring. The coding problems it solves come with examples of the inputs and outputs of the target programs. Given those examples, an inductive program synthesis algorithm can solve programs like the ones in the coding contests attempted by AlphaCode _without_ recourse to the natural language descriptions. The natural language descriptions are used by AlphaCode only to try and guide its large language model to generate code that is relevant to the programming task, but as the article observes, that doesn't seem to work all that well: AlphaCode's code generation step is more like a firehose that spams millions of programs in the hopes that it might get lucky and generate a few that solve the programming problem. This happens rarely.