You’re probably thinking that placing more value on the things we already have is harmless behavior, but it cuts both ways. For example, it doesn’t help when you hold onto a losing investment.

This happens all the time. Investors think about what price they paid for a stock or where it was six months ago, and then try to figure out their chances of getting back to even. In reality, investments don’t care what price you paid.

And the other factor working against investors is an aversion to loss. It’s been shown we feel losses about two to two-and-a-half times more than we feel gains. To say that another way, we like gains, but hate losses.

(This phenomenon has been used to explain why golfers do significantly better when putting to save par than when putting to make birdie. In short, golfers like birdies, but hate bogies.)

So investors have these two powerful forces working against them: They fall in love with the investments they own, because they own them, and they hate taking losses. This is what keeps investors hanging on to terrible investments for years, losing out on potential gains.

To avoid this trap, here’s the one question successful always ask themselves about each investment they own: “If I didn’t own this stock or bond today, would I still buy it now?

If the answer is no, sell. And if the answer is yes, don’t sell. It’s as simple as that.

Warren Buffett echoed this very idea when he wrote in his 1996 annual letter to shareholders, “If you aren’t willing to own a stock for 10 years, don’t even think about owning it for 10 minutes.”

In his 1998 annual letter he wrote: “[…] we made major purchases of Federal Home Loan Mortgage Pfd. (‘Freddie Mac’) and Coca Cola. We expect to hold these securities for a long time. In fact, when we own portions of outstanding businesses with outstanding managements, our favorite holding period is forever.”

Yet, in 2000, he sold his Freddie Mac shares. “We try, therefore, to keep our estimates conservative and to focus on industries where business surprises are unlikely to wreak havoc on owners,” Buffett wrote in his annual letter that year. “Even so, we make many mistakes: I’m the fellow, remember, who thought he understood the future economics of trading stamps, textiles, shoes and second-tier department stores.”

When you sell, you should be refining how you invest and improving your ability to make better decisions. As a result, your portfolio should improve.

It’s so easy for investors to get paralyzed when selling, especially when looking at a loss, but all that really matters is that your money is invested as best as it possibly can — and that means constantly assessing your thinking about past investments. Like billionaire Sam Zell once said, “Anytime you don’t sell, you buy.”

Research shows it’s not natural for us to think this way. We have a tendency to focus on what we’re losing instead of what we’re gaining. It’s why we stay in losing investments and toxic jobs and bad relationships.

We think, “I’ll do it differently next time.” The thing is, there is no next time. You don’t get a chance to do it over.

Chris Reining is an investor and writer covering investing and financial independence. He has been profiled in The New York Times, and his advice has been featured on Business Insider, CNN, Forbes and more. Chris currently lives in Madison, WI. Follow him on Twitter @ ChrisJReining .

Like this story? Subscribe to CNBC Make It on YouTube!

Don’t miss: