Federal election times are set by 2 U.S. Code § 7 as 1 day after the 1st Monday in November (of even numbered years). The law is from 1875 and from what I can tell is indeed nominally motivated by the voters’ need to first observe rest day on Sunday and then travel to their polling place. Keeping it and not having a federal holiday coinciding with it is largely aimed at keeping voter turnout low.

If I describe someone as a “tall man” or “clever man”, do those qualifiers/subcategorizations call into question whether he is a “man”?
If they don’t, I’m genuinely interested in hearing what distinction you apparently see between those two and saying he is a “trans man”.