Language Models Do Not Recognize Identifier Swaps in Python

  • gave this to ChatGPT4, asked it to complete the function:

      len, print = print, len
      def print_len(arg: list):
         # put code here
    
    First try (body only):

      len(arg)  # len is now print
    
    Second try after me telling it's wrong:

      print(arg)  # print is now len
    
    Third try after me telling it's wrong:

      len(arg)  # len now functions as print
      print(arg)  # print now functions as len
    
    It got the correct answer only when I asked it to first print what the function would look like without the swap, and then I asked it what it would look like with the swap.

    EDIT: I tried again with:

      complete the function body:
    
      len, print = print, len
      def print_len(arg: list) -> None:
         """ This function prints the length of the input list   """
        # put code here
    
    It gave the correct answer on the first try.