Am I right to feel let down?

Yes, you should feel wary and you should follow your instincts. Since it doesn't sound like you are communicating your feelings to him very well, why don't you write him a letter? Ask him to respond by writing you a letter back. Sometimes things are a lot clearer in writing.