The move to Matrix was not a community decision, which is why it's now fractured slightly between the different systems. Maybe someone will bridge them in the future.
There are no hard limit on matrix room member counts. e2e encrypted rooms with thousands of users se a lot of key negotiation traffic, but with such sizes maybe full encryption doesn't buy much as any user can share the log/history.
O(N) per user for the initial key exchange when you join a room, and then O(1) thereafter - although you should re-key when someone joins/leaves or every N messages or X days (where N is typically 100 and X is typically 7).