use swap_remove instead of remove in a few places

`swap_remove` is faster if we don't care about the order (O(1))

Signed-off-by: strawberry <strawberry@puppygock.gay>

src/service/rooms/timeline/mod.rs

@@ -1078,7 +1078,7 @@ impl Service {
 			.into_iter()
 			.position(|server_name| server_is_ours(&server_name))
 		{
-			servers.remove(server_index);
+			servers.swap_remove(server_index);
 		}
 
 		servers.sort_unstable();