How do you know that it is a hound which is frightening the family in The Hound of the Baskervilles?

Although Dr. Mortimer believes Sir Charles may have been killed by the fantastical hellhound of the "Baskerville curse," Sherlock Holmes, being a scientific and rational investigator, is not sucked in by that supernatural explanation. He believes a physical hound is involved.

First of all, by Dr. Mortimer's own account, the footprints of a gigantic hound were visible "some little distance off" from Sir Charles's fallen body. Dr. Mortimer also reports that prior to Sir Charles's death, neighbors in the area said they had seen a "huge creature, luminous, ghastly, and spectral" that reminded them of the dog in the Baskerville legend. 

Although readers may not pick up on the significance of the clue as soon as Holmes does, the fact that Sir Henry has a single boot stolen from two pairs of shoes also indicates the presence of a real, living dog and a human agent behind the creature. The brown boot was taken to give the scent of Sir Henry, the next victim, to the hound, but since it was new, it held no scent, so an old black boot was taken next. 

Finally, Watson hears "a strange cry" on the moor that sounds like "the cry of a hound."