What is the poem implying about who lives in the house in the poem "The Listeners," by Walter de la Mare?

The poem is implying that the people in the house are dead and thus ghosts or that only ghostly presences remain from prior inhabitants.  We have some clues to this in the text of the poem.  We are told that when the traveler knocks there were,

But only a host of phantom listeners   

    That dwelt in the lone house then (lines 13-14)

We know that these phantoms actually did listen, "to the voice from...

The poem is implying that the people in the house are dead and thus ghosts or that only ghostly presences remain from prior inhabitants.  We have some clues to this in the text of the poem.  We are told that when the traveler knocks there were,

But only a host of phantom listeners   

    That dwelt in the lone house then (lines 13-14)

We know that these phantoms actually did listen, "to the voice from the world of men" (line 16).  So these listeners are not humans, certainly. The traveler felt "their strangeness" (line 21), and while he could not see them, he felt this in his heart.   He believes that whatever is in the house will hear his message, that he had come as promised, had kept his word. 

This poem is so mysterious, left to the reader's imagination to make meaning of the scene, the traveler arriving to find only ghosts or phantoms, too late to have helped in some way that had been expected of him.  The inhabitants may have deserted the house, or they may all have starved or been slaughtered.  We sense how upset he is as he bangs loudly on the door, pounding and crying out his anguish at what seems to have been his failure to come when needed.