I have found that there is sort of a gray area between conventional publishing and the dreaded "vanity press". There are a few upstart companies that are trying a new approach. Basically, they sign authors and distribute their books exclusively online through online retailers, printing nearly all of their hard-copies "on demand". They claim that "mainstream" publishers print on demand as well, with respect to book demand. Some of them distribute through e-books as well, but I'm not talking about e-book exclusive publishers.
No, I am not talking about vanity press (or subsidiary press). I'll just repeate that. But I am talking about a strangely savage deal. One example, PublishAmerica, takes exclusive rights to your work for 7 years. I did a little research, and quite a few of their novels (if not all) are available for purchase online. Really, I've even found quite a number of reviews for the books published through them. Supposedly one of their authors hit #10 on the NYT Best Seller List.
From what I understand, with these kind of deals, the royalties aren't great. But there are successes/semi-successes. I am thinking of trying it as a last-resort, but I can't help wondering if it wouldn't be better to keep my work unpublished for now, while I plow on with my next book.
With these types of deals, unless you make out wildly successfully, I have a hard time finding that traditional publishers would take you seriously having been published in this manner. Sure, you've always got a shot, but I can't help feeling that its only a small step above vanity . . . at least you aren't paying them anything.
The other is that it is impossible to order books that they publish through a regular book shop, you have to buy them online. This is because they will not supply books on a sale-or-return basis.
There are, apparently, people out there who offer a similar service without these problems. Don't ask me who they are, though, 'cause I don't know.
They've got a number of links on Preditors and Editors-- scroll down to their listing.
Theresa Nielsen Hayden, an editor at Tor Books, discusses them as a new scam publishing format here -- while PA isn't explicitly named, it is their MO. (They also mention some honest POD publishers in the comments thread. Places like PA having their authors special-order non-returnable books in shops have seriously hurt the orders of non-scam POD publishers.)
And a discussion, again on TNH's weblog, of the contract issues with PA, here.