Think about it: if you break down all the lyrics, they could be related to a friendship or a romantic relationship, it's actually not clear either way once you remove any assumptions (cough). The only line that seems to point more towards a romantic situation is 'we let the life we had unwind' just because one would normally talk about a marriage or love relationship as a life you build together.
But it could be about a friendship. I am old enough to have friendships that go back 40 years, and I can relate to the idea that friends build a life together, too. There are people who weave in and out of your life over time, some who have a more prominent thread color throughout, and some who only skirt the edges of your tapestry with a neutral tone that wouldn't be much missed should it fade out...wow that sounds harsh, but you get the analogy ;-) Your friends are a part of the life you build, and in the same way as I wrote before, if you find yourself wanting different things you could easily diverge and realize you have very little in common anymore.
If we put the friendship filter on these lyrics, then the lines 'Silently turning to others / We looked for the smallest sign' describe what could be quite common, I suppose; you realize your once dear friend has drifted away from you, and on top of that you realize you're actually ok with it because you have something else pulling you in another direction. So yes, at that point in time you would look around for others who you have an affinity with, you hope for someone to show a sign that they want to fill that space in your life and that you can fill that space for them, too.
But then there are still these lines:
I close my eyes and I hear you
Telling me all the things I am
It takes me back to the start
Refilling the hourglass with sand
In a friendship context, this still works: someone who has known you for many years will have reflected back to you, explicitly or otherwise, your goodness and your talents and your - for lack of any one word that says what I want it to say - worth. And a distance or divergence between you later on in life doesn't take away that gift of understanding from before, you can always call on it and be comforted by it, be challenged by it (this person knows you at your best and at your worst), be grounded by it, even if that person is no longer an active part of your life. I like the idea that this knowledge is still there, bringing a kind of renewal despite the fact that these two people have nothing holding them together anymore. Which by the way basically means everything they had isn't gone, there will always be some kind of shared history between them, and this comfort in being known is still there. The relationship may be over, but it certainly wasn't a waste.
[There is also the possibility that the first verse is about the old relationship and the second verse is about the new one, with the refrain focusing on the end of the first relationship. This makes my little brain hurt, so that's all I'll say about that ;-) ]
It is still very sad to consider the notion of 'all we ever had is gone' in either a romantic or friendship situation. Walking away from someone who knows you so well, with whom you share a substantial history, must be difficult regardless. As I wrote before, the lyrics don't hint at any injury or betrayal or flasehood or any real 'root cause' for this ending, just that they want different things. And these things are different enough that 'nothing' remains. Heavy stuff!