Oh, I really understand the feeling of insecurity. I remember the first time I wore something yellow (which wasn't bright at all) I felt totally exposed. But now I just love bold colour combinations :)
I'd say this colour matching thing is like a muscle if you exercise it, it gets stronger - don't stop, just go for it!

A classic box pleated skirt would look lovely made out of that yellow tartan. Just looked up some images on google - there are so many variations! The key is to choose the right one for you :DDD