Putting an "innocent" tracking links and "pixels" in your email is like taking photos of people without asking or even putting a security camera in your guest's bathroom. Just because most people won't notice or complain and you pinky promise not to use it for anything nefarious doesn't make it less rude.
Pretty much everyone are doing it now. Almost all companies, recruiters, salespeople, even official US government emails. Since I use NextDNS and block most tracking on DNS level, I am acutely aware that I can't just click any link from an email without hitting a blocked tracking page.
And if you're doing it, then I think you, your company or your institution are rude.