Is it weird of me to be jealous over this?

I have an issue. My boyfriend has dated my brothers wife. They were 15 years old, but they have kissed and it's just awkward for me. He is almost 29 and I am 26. But I am still having an issue getting over it when they talk and laugh together. She is totally not the type to come onto him or anthing, in fact, she is a real sweetheart. I love her alot, but still I have that jealousy. So here is my issue...My brother (her husband) is a hunter - as well as my boyfriend. Well, my boyfriend is going to take off work on friday and potentially go over to their house (while my brother is at work) and hunt on their property. My issue is, that she is always the type of girl to be very "hospitable". I know that she is going to invite him in the house after his morning hunt to have coffee or something with her. I have a problem with that and I think that it would be inapporpriate if he proceeded to go inside and hang out. (mind you, my brother would be at work). I know it sounds silly because all of this might not even happen. But i am the type that would like to be prepared. Is it weird that this would bother me? Even though I know that nothing would happen if he went inside? It's just the fact that it would bother me and be innapropriate. Please help!