Why does Jim have a "peculiar expression" on his face when he first comes home in "The Gift of the Magi"?

Jim’s expression comes from the fact that he realizes that the combs he bought Della for Christmas will be useless now.  

When Jim and Della do not have enough money to buy each other presents, each sells the one thing valued most to get the other a present.  When Jim realizes what Della has done, he is not angry at her. He is shocked. 

It was not anger, nor surprise, nor disapproval, nor horror,...

Jim’s expression comes from the fact that he realizes that the combs he bought Della for Christmas will be useless now.  

When Jim and Della do not have enough money to buy each other presents, each sells the one thing valued most to get the other a present.  When Jim realizes what Della has done, he is not angry at her. He is shocked. 

It was not anger, nor surprise, nor disapproval, nor horror, nor any of the sentiments that she had been prepared for. He simply stared at her fixedly with that peculiar expression on his face.

Della cut her hair.  He sold his watch to buy her a special gift that requires hair. He knew how much she valued her hair.  He was thinking about how much she would love the combs.  The fact that she can’t use his gift leaves him stumped and horrified.  He is not upset with her, he is just baffled by the fact that he has no gift for her now.

Della’s reaction is telling.  She is afraid he is angry at her for cutting her hair.  She tries to read his expression, which too her is “peculiar.”  She begins to wonder if she knew him as well as she thought.  Fortunately, Jem proves to be the person she thought he was.  Once it sinks in, he hugs her.

Out of his trance Jim seemed quickly to wake. He enfolded his Della. For ten seconds let us regard with discreet scrutiny some inconsequential object in the other direction. Eight dollars a week or a million a year—what is the difference?

Jim and Della’s presents show that they both know each other well and care about each other.  They also show that they love each other enough to give up what matters most.  Possessions and beauty are shallow.  The desire to make someone a gesture is what matters.  This is why O’Henry says that they were wise.  They each gave the gift of sacrifice.