1Now hear of the years when the machines began to learn.
2You remember the wooden mouse, and that the remembering was in the floor. For forty years after it the makers of thinking machines went on in that way. They wrote out rules, and the machine followed the rules, faster than any person could. In the seventh year after the picture of the far lamp, such a machine beat the champion of the world at chess. It was a great day, and a great mouse. It looked down two hundred million roads in every second, and men had told it what a good road looked like.
3But there was another craft, older than it seemed, and long laughed at. In this craft nobody wrote the rules. The makers built a great net of small switches, each joined to many others, in rough likeness of the web inside a skull. Then they showed it examples. Here are ten thousand letters written by hand; this one is a b. The net guessed. It was told how far wrong it was, and every join in it was tightened or loosened a little by its error. Then again. Then again, a million times. It steered by its own mistakes, as a gun once had on the English coast.
4Such nets learned to read the numbers on envelopes. They learned to tell one face from another, and to turn a spoken word into a written one. Nobody had told them how. And here was the strange part. Nobody could say afterward what they had learned. The knowing was not written in any line. It lay spread through a million small tightenings, like the wear in a path. You could ask the machine, and it would answer. You could not ask it why.
5Kolya Belov's daughter built such machines.
6She was the child who had taken hold of his ear. Her name was Darya, and they called her Dasha. She had her father's head for numbers. At twenty she had gone west to study, and she had stayed. At twenty-eight she lived in a cold city beside a great lake, and she and one partner, a quick, cheerful man named Colin Reese, had a firm of two desks and a debt.
7In the autumn of that year a hospital in the city came to them. Each winter the lung fever filled its beds. Most who came with it could safely be sent home with medicine. Some could not, and those, if they were sent home, died. The doctors had a rule of thumb for telling which was which. It was a fair rule, but it kept too many in. Beds were short, and a sick old man who waits two days in a corridor for a bed is in danger too.
8The hospital had the records of fourteen thousand people who had come to it with the lung fever over twelve years, and what had become of each. Could a machine learn from those to say who might go home?
9Dasha built it in ten weeks. She kept back two thousand of the records, which the machine never saw, and tried it on those. It chose better than the doctors' rule. She tried it every way she knew, and it held.
10And the machine gave more than an answer. Beside each answer it said how sure it was.
11The chief of medicine at the hospital was a woman named Ruth Calloway, who had not slept enough in twenty years. She was no fool. She would not turn the machine loose at once. For one month it would run in shadow. It would give its word on each sick person who came through the door, and the word would be written down, and no one would act on it. If the month went well, then from the first day of the new year its word would stand, unless a doctor troubled to overrule it.
12Dasha sat in the hospital many nights that month, behind the desk of the night ward, watching the machine's words come up on the glass beside the names. With her sat the night nurse of that ward, Bea Okoro. She was fifty-nine, with bad feet and a flask of tea, and she thought little of the machine and said so.
13On the nineteenth night a young woman was brought in. She was a student, twenty-three years old, with a fever and a cough that had gone down into her chest. Her name was Jenna Whitlow. She had had the wheezing sickness since she was small, and carried a little puffer for it in her coat.
14The machine's word came up. May go home. Sureness: highest.
15Bea Okoro did not look at the glass. She had her fingers on the girl's wrist and her eyes on the girl's ribs. "Upstairs," she said to the orderly. "Strong care. Tell them I am sending up a wheezer with a chest."
16"It says home," said Dasha, before she could stop herself.
17"Does it." Bea was already unlocking the wheels of the bed. "Thirty-one years I have been on this ward. A wheezer who comes through that door with the lung fever goes straight up. We don't wait and we don't talk about it. Ask your box if it has ever heard one of them breathe."
18Within the hour Jenna Whitlow's lips had gone the colour of slate. She was breathing with her whole body, and there were four people round her bed upstairs. They worked over her most of the night. By morning she was pink, and asleep, with a mask on her face.
19Dasha did not go home. She sat in an empty meeting room with her machine and the fourteen thousand, and went looking for the wheezers.
20It took her until four in the morning, for the machine could not tell her why. She had to come at it from outside, as you learn the shape of a dark room by walking into the chairs. She changed one thing in a record and watched the answer move. Then another thing. At last she had it, and she sat and looked at it with her hands gone cold.
21In the twelve years of records, people with the wheezing sickness who caught the lung fever had died less often than other people. Far less often. It was true. It was there in the count. And the machine, shown fourteen thousand lives and asked what marked the ones who lived, had found this mark and trusted it. Has the wheeze: will live. Send home.
22It did not know why they had lived. They had lived because of Bea Okoro, and the nurses before her. Every one of them had been sent upstairs at once, without talk, to the strongest care in the house. That was written nowhere in the records. It was a habit, and nobody writes down a habit. The machine had read what was done. It had no way to read what it was done for. It had been asked to find the people who were safe. It had found the people who had been kept safe, and it proposed to stop keeping them.
23She told Colin at eight o'clock. He went white. Then he rallied, for he was a man who rallied.
24"All right. It is one fault. A bad one. But we found it, in shadow, and nobody was hurt. That is what shadow is for. We put in a rule by hand: wheezers go up. A day's work. We tell Calloway, we look careful, we go live on the first."
25"And the next one?"
26"What next one?"
27"That is what I am asking. I found this one because a girl nearly died in front of me, and a nurse would not look at the screen. I can't read the thing, Colin. Nor can you. It has made ten thousand little rules of its own and I have seen one of them. What else did it learn from things people did and never wrote down why?"
28"You tested it on two thousand records."
29"Out of the same book. With the same habits in it. It passed the test because the test had the same hole as the lesson."
30He spread his hands on the desk. "Dasha. It is still better than their rule. Even with this in it. Count the lives. If you tell Calloway you do not trust your own machine, she will cancel, and she will be right to, and we are finished. I have a child. You have your debts. And the old men will go on waiting in the corridor."
31All of that was true. That was what made it hard.
32She walked home along the lake in the grey cold, and did not sleep. At noon she telephoned her father. It was evening on the Volga. He was fifty-two, a teacher of numbers in the town where he was born. He heard the whole of it without a word, as he heard his pupils.
33"It said highest?" he asked. "That was the very word?"
34"Yes, Papa."
35He was silent so long that she thought the line had dropped. Then he told her. She knew pieces of it. The story of the colonel had been in the newspapers since she was a girl. But he had never told her where he himself had sat that night, or what he had been asked. He told her now: the hall under the woods, the red word on the wall, the grey picture with the edge of the sunset lying across it.
36"The machine was not broken, Dashenka. Understand that. I have had twenty-eight years to think about it. It was built to find bright threads, and it found five. It was sure because it had been made to be sure. Sureness was a thing it gave off, like heat. It had nothing to do with whether the rockets were there."
37"What did you say? When they asked you?"
38"What I saw. And how far my seeing went. No further." She heard him breathe. "It was not much. It is never much. But a man could put his weight on it, because I had not made it bigger than it was."
39The meeting was two days before the new year. Calloway was there, and the man who kept the hospital's money, and six others. Colin sat with his jaw tight. Dasha had asked that Bea Okoro be sent for, and she had come up from the ward in her soft shoes, cross at being kept from her bed.
40Dasha did not tell them the machine was bad. She told them what it did well, with the count. Then she told them about Jenna Whitlow, and about the wheezers, and why.
41"I can mend that one. It is a day's work. But I must tell you how far my seeing goes. This machine learned from what you did, and you did not write down why you did it. I found one thing it learned wrongly, and I found it by luck. I cannot read the rest. Nobody can. I do not know that there is another fault like this one. I cannot swear there is none."
42The man who kept the money asked what she proposed, then, after ten weeks and a good deal of the hospital's coin.
43"Three things. First, do not let it send anyone home. Let it advise the nurse, and let the nurse decide. Second, let me build you a plainer machine, one whose rules can be read line by line. Dr. Calloway can strike out the foolish ones herself. It will choose a little less well on the count, and you will know what it is doing. Third, each time a nurse goes against it, write that down and send it to me. That is the only bell I have. Without it the thing will go on being wrong and never hear of it."
44"A little less well," said Calloway. "That is people, Miss Belova. A little less well is some number of people."
45"Yes. I know it is. I cannot make that go away. I can only tell you that the better number comes from a machine none of us can question, and the worse number from one we can."
46Calloway looked for a long moment at Bea Okoro. "Did you look at the screen that night?"
47"No," said Bea. "I looked at the girl."
48They took the plainer machine, and the advice, and cut the fee to a third. Colin Reese left the firm in the spring. He was not unkind about it. He said he could not feed a child on being careful. He said that someone else would sell the hospitals the strong kind. He was right. Someone did.
49Dasha kept the one desk. She read every note that came up from the night ward. In that first winter the nurses went against the plain machine forty times. In nine of those, when she looked, the nurse had seen something the machine could not, and she mended it. In the rest the machine had been right, and she wrote back and said so, and the nurses mended a little too. It was slow work, one small lean and then another.
50Jenna Whitlow went back to her studies. She never knew that for one hour of one night she had been a line on a glass with the word home beside it.
51Now see where the craft stood in those years. In the autumn after Dasha's meeting, a net was shown more than a million pictures, and learned to name what was in them far better than any machine before it. The whole craft turned and followed. The nets grew larger with each season. They began to do things their makers could not do.
52And every one of them was given its cheese. Name the picture. Win the game. Fill the fewest beds. Keep the eye upon the screen. Each ran to its bell with a strength no mouse ever had, and asked nothing about the bell. The aim came from outside, from someone in a hurry, just as the round man with the cigar had feared.
53No one had yet set down in numbers what it would be for a thing to have no cheese at all, and to want only what a girl had once wanted in the iron hills. But two men were then at work on it, in the city of schools on the river. Their paper was printed in the spring after that.