How did Boo know Jem and Scout were in trouble?

The most likely explanation is that Boo heard the children screaming (Jem yells "Run!"). Since the children had been playing games, pretending to be the Radley's, Boo had taken an interest in them. He put gifts in the tree until his father put an end to that. So, there is the distinct possibility that Boo was looking out for them, from his home. 

It is likely that he heard them because they would have been...

The most likely explanation is that Boo heard the children screaming (Jem yells "Run!"). Since the children had been playing games, pretending to be the Radley's, Boo had taken an interest in them. He put gifts in the tree until his father put an end to that. So, there is the distinct possibility that Boo was looking out for them, from his home. 

It is likely that he heard them because they would have been close to the Radley house when Bob Ewell attacked. The high school and elementary school are at the south end of town. Going north, coming home from the pageant, the children would go from the schoolyard, make a turn in the road, and this leads to the Radley house and eventually the Finch's home. 

In Chapter 28, when the children are on their way to the pageant, they pass the Radley's house. It is so quiet that they can (symbolically) hear a mockingbird. So, it might have been just as quiet when they were returning. Atticus didn't hear anything because he had the radio on. But the Radley house was quiet. On a quiet night, with only the birds to be heard, it is likely that Boo heard Jem tell Scout to run. They are attacked when they are near one of the two oaks near the Radley's home. This offers further proof that they were near Boo's house and makes it more likely that he would have heard their screams.