Duplicates of this sentence have been deleted:
x #1147129

My first thought was that [お爺さん」was the own grandfather, but as the original sentence included 「あなたの」it was clear that it referred to someone else's grandfather. Is there a non-context wise way to know if talking about your or my grandfather without using possessives?

No, but there are some clues to narrow down the implicit possessive.
Without context, the most definite one is "your." The speaker is asking the listener a question, so it's most natural to think that it's about the listener's grandfather.
The next possible candidates are anything other than "my" and "your."
"My" is the least likely. About the situation: you're supposed to know your grandfather's age. About the word choice: when referring to your own grandfather, "おじいちゃん"(casual) or "祖父"(formal) is more common.

Plus, "おじいさん" can also mean "old man" in general, so to avoid confusion, an explicit possessive is usually given if it's not "your."

>About the word choice: when referring to your own grandfather, "おじいちゃん"(casual) or "祖父"(formal) is more common.

That's what I was expecting, thank you.

Actually, I just imagined a little kid asking about his granpa's age (which isn't always straightforward when we are young), but he would use "じいちゃん", as you said, so everything is now clear for me, great!