The whole "unconference" movement was supposed to be a much more informal, low-cost sort of gathering, where people with information to share could share. Even when I organized WordCamp Dallas, the $20 we asked for is worth about $17 today, and we provided a free lunch, a t-shirt, wifi. and a small party the night before. I paid for one of the speaker's hotel room out of my own pocket. That was all because the sponsors ponied up the money and even then, we lost a little money. So, all in all, it was really just one, big gift to the WordPress community,
If you're doing a conference right, and you mean to make money, first I'd say, "Good luck!" I'd imagine conferences are to speakers what concerts are to most major musicians: You probably won't make much money from ticket sales, but you'll expect to recoup your money on book sales and/or new clients.
So to those who balk at $50, if they don't buy at that price, it's probably better that way. In the end, the free market will bear what the free market will bear. If you're not delivering at least $50 worth of value, no one will be willing to attend your next conference. If they're not willing or able to spend $50 today on something that's worth $50 or more today, then they're not in your target market.
Free markets for the win!