For one thing, the Dreamcast was NOT the absolute failure many claim it to be. The system had a huge and successful launch ($96 million launch day), had a healthy buy ration (something like 6-8 titles per console) and was generally a well made machine with a fantastic selection of titles.
What did the system in was the PS2 hype. While Sega managed to establish some 10 million users, they would have needed a lot more to survive the PS2, and when Microsoft threw their towel in the industry, Sega just couldn't compete with two gigantic corporations with the disposable income to establish themselves in the market.
Even Nintendo has more or less stopped in straight competition with Sony/MS, they've contented themselves with a niche market, like Apple or BMW (though, without the high prices )
That said, I think the PS3 will have a hard time the same reason the 3DO did: cost. The 3DO machine was $700 (the single most expensive home console in history), and it has long been established that new consoles should be $250-$300, with $400 as the absolute maximum.
Now, Sony is launching a minimum pricepoint of $500 with a $600 upper unit.
I seriously doubt the Playstation brand name to be enough, especially in the US where the XBOX has a strong grasp and the 360 will be going into a second generation of titles. What will likely happen is that Japan will go gangbusters for the console while it will be slow going in the west where the XBOX360 is established with more games better suited to the markets.