Wouldn't "one's" be clearer than "your"? Somehow I think this could be read as "I find it hard to live up to your reputation".
This sounds natural. We use "your" very often this way.
People "live up" to their own reputation, not someone else's.
"live up to your expectations" is used to refer to the other person's expectations, though.