Downsides of working with Deloitte are typical of the public accounting industry in general and of working for a large, buearucratic company. It is true that in public accounting, you either move up or out. Some people can't handle the hours, and if you let them, they will work you like a dog. Work/life balance is a really cute recruiting (external and internal) marketing tool Deloitte uses, but work/life balance only exists where you make it happen. And it feels like a total joke when you sit in meetings with the entire audit team of your office and have the lead audit partner tell you that we need to work my hours and we are all expected to work 300 to 500 overtime hours. Utilzation, overtime, and how many chargeable hours you book are really big metrics for partners and managers, and they expect certain results, but the ability for staff and seniors to control it is minimal. Also, there is a constant pull between keeping your charegable hours up but not exceeding budget. It's really frustrating. Flextime schedules do exist, but are hard to come by and not as flexible as you might think.
Just like any large company, Deloitte has a bunch of stupid policies and procedures that are a pain in the ass. Job evaluations are subjective and hit and miss, but promotions operate on a standard schedule so it really only affects your ranking and raise (but not by that much).
Auditing is a stressful job and takes a lot out of you. Some people just handle it better than others.