Validator Set limitation
Certainly there is additional network overhead and longer times to finality the more validators there are. What exactly a good practical limit is will need to be determined using data, but order thousands probably isn't a bad guess.
There are a variety of factors that determine what validators there are and how many there are. Failing to follow the protocol gets you slashed and failing to meet performance requirements gets you ejected, both of which decrease the number of validators. On the other hand bonding adds new validators and we are considering adding some economic pressures to regulate bonding.
Overall, how these things balance out will be something we need to measure and tweak to get right.