Most managers are jokes and if you are not their pet employee will barely know you exist. This mean that on quarterly and annual review your manager will inflate the work done by his favorite employees and strongly back their deserving for a bonus, promotion etc, while having little to say to the committees about the value of your work. If promotions are rare and reserved for a select few, raises are almost non-existent and far below inflation.
20% time mean you have to sleep less and work more during the week and forget about weekends. The running joke is that 20% percent means you work 20% more time than the current 140% time you spend on a fictional 40h work week.
Taking vacation lower your performance ratings resulting in lower bonus and less chance for a promotion or raise. Same goes for sick days, parental leave, or even for being unreachable on a weekend (while not on call).
Some of my coworkers had to complain to HR about their managers and got little support. HR would not recognize any wrongdoing and downplay issues as personality mismatch and misunderstandings. I've witnessed situations unblock with HR only after a VP was contacted for help by the employee.
