Tjyylli

I encountered below statement by Alan M. Turing here:

"The view that machines cannot give rise to surprises is due, I
believe, to a fallacy to which philosophers and mathematicians are
particularly subject. This is the assumption that as soon as a fact is
presented to a mind all consequences of that fact spring into the mind
simultaneously with it. It is a very useful assumption under many
circumstances, but one too easily forgets that it is false."

I am not a native English speaker. Could anyone explain it in plain English?

Mathematicians and philosophers often assume that machines (and here, he probably means "computers") cannot surprise us. This is because they assume that once we learn some fact, we immediately understand every consequence of this fact. This is often a useful assumption, but it's easy to forget that it's false.

He's saying that systems with simple, finite descriptions (e.g., Turing machines) can exhibit very complicated behaviour and that this surprises some people. We can easily understand the concept of Turing machines but then we realise that they have complicated consequences, such as the undecidability of the halting problem and so on. The technical term here is that "knowledge is not closed under deduction". That is, we can know some fact $A$, but not know $B$, even though $A$ implies $B$.

Honestly, though, I'm not sure that Turing's argument is very good. Perhaps I have the benefit of writing nearly 70 years after Turing, and my understanding is that the typical mathematician knows much more about mathematical logic than they did in Turing's time. But it seems to me that mathematicians are mostly quite familiar with the idea of simple systems having complex behaviour. For example, every mathematician knows the definition of a group, which consists of just four simple axioms. But nobody – today or then – would think, "Aha. I know the four axioms, therefore I know every fact about groups." Similarly, Peano's axioms give a very short description of the natural numbers but nobody who reads them thinks "Right, I know every theorem about the natural numbers, now. Let's move on to something else."

Just an example - given chess rules, anyone should immediately figure the best strategy to play chess.

Of course, it doesn't work. Even people aren't equal, and computers may outperform us due to their better abilities to make conclusions from the facts.

People might assume that if I write a program, and I understand the algorithm completely, and there are no bugs, then I should know what the output of that program would be, and that it should not surprise me.

Turing says (and I agree) that this is not the case: The output can be surprising. The solution to a travelling salesman problem can be surprising. The best way to build a full adder can be surprising. The best move in a chess game can be surprising.

This is the idea of emergence, which is when complex behavior of a result results from the interaction of relatively simple rules. There are lots of examples of this in nature, as that link points out. Insect colonies, bird flocks, schools of fish, and of course, consciousness. In a flock of birds or school of fish, each individual in the swam is only making decisions based on the others immediately surrounding them, but when you put a bunch of those individuals together all following those rules, you start to see more coordinated behavior than you'd expect without a higher level plan. If you go on Youtube and watch demonstrations of robot swarms, you see that they all avoid hitting each other and work in unison. Surprisingly this doesn't need to be accomplished by having a single central computer coordinate the behavior of each individual robot but can instead be done using swarm robotics where, like the insects or the birds or the fish, each robot is making local decisions which leads to emergent coordination.

Another interesting demonstration of emergent behavior is Conway's Game of Life. The rules for the game are extremely simple, but can lead to very fascinating results

A tempting argument against the ability of computers to gain human-intelligence is to say that since they can only do precisely what they're programmed to do, they must only exhibit the intelligence that we program them with. If this were true, then we would also not expect the relatively simple behavior of neurons to give rise to human intelligence. Yet as far as we can tell, this IS the case and consciousness is an emergent property of neural processing. I'm sure Turing would have loved to see what's become possible today with the use of artificial neural networks