The question I need to answer is: What other factors (besides the witches) contribute to Macbeth's and Banquo's increasing distrust of one another?...

After the witches' prophesy, Macbeth welcomes Duncan and his entourage (including Banquo) to his castle for the night. Macbeth and his wife have plotted to kill Duncan, but before that happens, Banquo and Fleance are out after midnight because Banquo can't sleep. He says to Fleance, who is guiding him with a torch: 

A heavy summons lies like lead upon me,
And yet I would not sleep: merciful powers,
Restrain in me the cursed thoughts that nature
Gives way to in repose!

He has a nasty feeling about what is about to happen. He's having dreams that disturb him. He feels like he should do something, but isn't sure what. He and Fleance encounter Macbeth and are surprised that he is still awake. Banquo tells him he dreamt of the weird sisters, for "To you they have show'd some truth." Macbeth lies and says he hasn't given them any thought. So we know what woke Banquo and what sneaking suspicions he has. They agree to talk later--presumably about the prophesies--when they have the time. 

After the murder, Banquo is alone and thinks: 

Thou hast it now: king, Cawdor, Glamis, all,
As the weird women promised, and, I fear,
Thou play'dst most foully for't: yet it was said
It should not stand in thy posterity,
But that myself should be the root and father
Of many kings. 

That is, Macbeth now has everything the weird sisters promised him, but Banquo is afraid he murdered Duncan to make it happen. At the same time, he remembers that the witches predicted he himself would sire many kings, while Macbeth would have no line of kings. This, no doubt, makes him nervous, particularly now that Macbeth is king. 

Shortly thereafter, Banquo and Fleance ride because they presumably have business a day's ride away, but they promise to return for Macbeth's feast. After they leave, Macbeth tells us (the audience): 

To be thus is nothing;
But to be safely thus.--Our fears in Banquo
Stick deep; and in his royalty of nature
Reigns that which would be fear'd: 'tis much he dares;
And, to that dauntless temper of his mind,
He hath a wisdom that doth guide his valour
To act in safety. There is none but he
Whose being I do fear...

He is afraid that Banquo knows the truth and will avenge the king, and do so stealthily. But further, Macbeth is angry that he has murdered "the gracious Duncan" for Banquo's sons, since no son of Macbeth's will succeed him.