DISCLAIMER
You are about to read a long story on how I got burnt with MongoDB and depressed with it. I am not blaming MongoDB, anyone using, advocating or developing it. I am blaming myself for this. MongoDB is a good tool. You can use it but just make sure it is what you need and it handles your requirements very well. This is not specific to MongoDB but applies to every tool we use.
I think it's a bit harsh for people to criticise either him or MongoDB too much.
At the end of the day, it was a big rush job, no time was allowed for testing for scaling, and it fell over.
My only real surprise is that it wasn't Apache that fell over first.
True, but it's defaults are not really "battle ready" and since they didn't tweak MongoDB in advance, I can only guess that Apache was already pre-configured for that load - perhaps they used it for their existing web servers?
I would guess Apache httpd is a tool they are familiar with and comfortable managing. Trying to building a high performance MongoDB system in just a couple days was probably a bad first project with the tool.
They really should have just stuck to what they know, until familiar with the new shiny.
26 webservers and only two mongodb instances? Seems stacked.. Why not set up more instances and split the load across them (sharding, splitting collections across replica sets, etc) or similar?
Also - big question for me: were you using a ridiculously old version of mongo? I have yet to run into anyone who is on 2.0 or 1.8 and is still running master/slave (as opposed to replica sets). Sounds like your writes weren't working when you moved to replica sets because your client wasn't properly configured.
DISCLAIMER You are about to read a long story on how I got burnt with MongoDB and depressed with it. I am not blaming MongoDB, anyone using, advocating or developing it. I am blaming myself for this. MongoDB is a good tool. You can use it but just make sure it is what you need and it handles your requirements very well. This is not specific to MongoDB but applies to every tool we use.
I think it's a bit harsh for people to criticise either him or MongoDB too much.
At the end of the day, it was a big rush job, no time was allowed for testing for scaling, and it fell over.
My only real surprise is that it wasn't Apache that fell over first.