There is a saying in my native language that goes like this "respect last longer between friends by paying visit to each other rather than leaving together.for this reason ,it is better the way he rejected staying with you in oder to keep that relationship healthy.