Files
element-call/src/room/InCallView.tsx

852 lines
27 KiB
TypeScript
Raw Normal View History

2022-05-04 17:09:48 +01:00
/*
Copyright 2022-2024 New Vector Ltd.
2022-05-04 17:09:48 +01:00
SPDX-License-Identifier: AGPL-3.0-only OR LicenseRef-Element-Commercial
Please see LICENSE in the repository root for full details.
2022-05-04 17:09:48 +01:00
*/
import { IconButton, Text, Tooltip } from "@vector-im/compound-web";
import { type MatrixClient, type Room as MatrixRoom } from "matrix-js-sdk";
import {
type FC,
type PointerEvent,
type TouchEvent,
useCallback,
useEffect,
useMemo,
useRef,
useState,
type JSX,
type ReactNode,
} from "react";
2023-06-05 20:51:01 +02:00
import useMeasure from "react-use-measure";
2025-03-13 13:58:43 +01:00
import { type MatrixRTCSession } from "matrix-js-sdk/lib/matrixrtc";
import classNames from "classnames";
2025-05-16 11:32:32 +02:00
import { BehaviorSubject, map } from "rxjs";
2025-10-07 16:00:59 +02:00
import { useObservable } from "observable-hooks";
2025-03-13 13:58:43 +01:00
import { logger } from "matrix-js-sdk/lib/logger";
import {
VoiceCallSolidIcon,
VolumeOnSolidIcon,
} from "@vector-im/compound-design-tokens/assets/web/icons";
import { useTranslation } from "react-i18next";
2022-08-02 00:46:16 +02:00
2023-09-27 19:06:10 -04:00
import LogoMark from "../icons/LogoMark.svg?react";
import LogoType from "../icons/LogoType.svg?react";
2025-03-13 18:15:58 +01:00
import type { IWidgetApiRequest } from "matrix-widget-api";
2022-01-05 15:06:51 -08:00
import {
EndCallButton,
2022-01-05 15:06:51 -08:00
MicButton,
VideoButton,
ShareScreenButton,
SettingsButton,
Support for generic reactions (#2708) * Initial support for Hand Raise feature Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Refactored to use reaction and redaction events Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Replacing button svg with raised hand emoji Signed-off-by: Milton Moura <miltonmoura@gmail.com> * SpotlightTile should not duplicate the raised hand Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Update src/room/useRaisedHands.tsx Element Call recently changed to AGPL-3.0 * Use relations to load existing reactions when joining the call Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Links to sha commit of matrix-js-sdk that exposes the call membership event id and refactors some async code Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Removing RaiseHand.svg * Check for reaction & redaction capabilities in widget mode Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Fix failing GridTile test Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Center align hand raise. * Add support for displaying the duration of a raised hand. * Add a sound for when a hand is raised. * Refactor raised hand indicator and add tests. * lint * Refactor into own files. * Redact the right thing. * Tidy up useEffect * Lint tests * Remove extra layer * Add better sound. (woosh) * Add a small mode for spotlight * Fix timestamp calculation on relaod. * Fix call border resizing video * lint * Fix and update tests * Allow timer to be configurable. * Add preferences tab for choosing to enable timer. * Drop border from raised hand icon * Handle cases when a new member event happens. * Prevent infinite loop * Major refactor to support various state problems. * Tidy up and finish test rewrites * Add some explanation comments. * Even more comments. * Use proper duration formatter * Remove rerender * Fix redactions not working because they pick up events in transit. * More tidying * Use deferred value * linting * Add tests for cases where we got a reaction from someone else. * Be even less brittle. * Transpose border to GridTile. * First PoC for reactions * hide menu by default * Add lightbulb. * Add reaction indicator. * Add sounds. * Tidy up + add support for floating emoji. * Linting and general stability improvements. * Subscribe to the ecall reaction event type. * fix import * Center emoji picker * Overflow buttons when screen is too narrow * lint * Add settings for disabling animations / sounds. * Make vertical divider more visually distinct. * Make event listener more resillient. * lint * Fix some tests. * Remove old raised hand component * Add new icon * Update text * Update compound hand raised icon. * Add deer. * Fix case where you could send larger strings as emoji * Const the active time. * Document time in css. * Add rock emoji * Add licence file. * Add type def for custom reaction type. * better reaction description * Factor out reactions test structure to utils file. * Add tests for ReactionToggleButton * Add keyboard shortcuts for reaction sending. * type tidyups * lint * Add tests for ReactionAudioRenderer * lint * prettier * i18n sort * final lint? * Preload reaction sounds to prevent delays. * Update rock sounds * add onclick back * Fix test * lint * simplify * Tweak line height * modal impl * Modal refactor attempts. * Remove closed menu test since we're using Modal. * Swap icon, make mobile view better. * Fix mobile view for emoji picker. * Use Intl.Segmenter * Clear timeouts on component close. * Remove useless useCallback * Use prefers-reduced-motion * Add toggle for raise hand. * Add lower hand text * Add lower motion mode. * Decomplicate className system for Modal * Add error for failured to send reaction. * i18n * Spacing for emoji buttons search * Remove unrequired media query * Fix generic sound not playing. * Clear reactions if we're clearing timeouts. * Fix tests * Relabel lower hand * More translations * Add comments on reaction interface * Move polyfill. * lint * Replace deer sound * Another attempt to fix the sizing of the reactions * cleanup * fix button * fix --------- Signed-off-by: Milton Moura <miltonmoura@gmail.com> Co-authored-by: Milton Moura <miltonmoura@gmail.com> Co-authored-by: fkwp <fkwp@users.noreply.github.com>
2024-11-08 17:36:40 +00:00
ReactionToggleButton,
2022-01-05 15:06:51 -08:00
} from "../button";
import { Header, LeftNav, RightNav, RoomHeaderInfo } from "../Header";
import { type HeaderStyle, useUrlParams } from "../UrlParams";
2023-06-05 20:51:01 +02:00
import { useCallViewKeyboardShortcuts } from "../useCallViewKeyboardShortcuts";
import { ElementWidgetActions, widget } from "../widget";
import styles from "./InCallView.module.css";
import { GridTile } from "../tile/GridTile";
import { type OTelGroupCallMembership } from "../otel/OTelGroupCallMembership";
2023-12-01 17:43:09 -05:00
import { SettingsModal, defaultSettingsTab } from "../settings/SettingsModal";
2023-05-22 15:30:29 -04:00
import { useRageshakeRequestModal } from "../settings/submit-rageshake";
import { RageshakeRequestModal } from "./RageshakeRequestModal";
import { useWakeLock } from "../useWakeLock";
import { useMergedRefs } from "../useMergedRefs";
2025-08-29 18:46:24 +02:00
import { type MuteStates } from "../state/MuteStates";
import { type MatrixInfo } from "./VideoPreview";
import { InviteButton } from "../button/InviteButton";
import { LayoutToggle } from "./LayoutToggle";
import { CallViewModel, type GridMode } from "../state/CallViewModel";
import { Grid, type TileProps } from "../grid/Grid";
import { useInitial } from "../useInitial";
import { SpotlightTile } from "../tile/SpotlightTile";
import { type EncryptionSystem } from "../e2ee/sharedKeyManagement";
Knocking support (#2281) * Add joining with knock room creation flow. Also add `WaitForInviteView` after knocking. And appropriate error views when knock failed or gets rejected. Signed-off-by: Timo K <toger5@hotmail.de> * Refactor encryption information. We had lots of enums and booleans to describe the encryption situation. Now we only use the `EncryptionSystem` "enum" which contains the additional information like sharedKey. (and we don't use the isRoomE2EE function that is somewhat confusing since it checks `return widget === null && !room.getCanonicalAlias();` which is only indirectly related to e2ee) Signed-off-by: Timo K <toger5@hotmail.de> * Update recent list. - Don't use deprecated `groupCallEventHander` anymore (it used the old `m.call` state event.) - make the recent list reactive (getting removed from a call removes the item from the list) - support having rooms without shared secret but actual matrix encryption in the recent list - change the share link creation button so that we create a link with pwd for sharedKey rooms and with `perParticipantE2EE=true` for matrix encrypted rooms. Signed-off-by: Timo K <toger5@hotmail.de> * fix types Signed-off-by: Timo K <toger5@hotmail.de> * patch js-sdk for linter Signed-off-by: Timo K <toger5@hotmail.de> * ignore ts expect error Signed-off-by: Timo K <toger5@hotmail.de> * Fix error in widget mode. We cannot call client.getRoomSummary in widget mode. The code path needs to throw before reaching this call. (In general we should never call getRoomSummary if getRoom returns a room) Signed-off-by: Timo K <toger5@hotmail.de> * tempDemo Signed-off-by: Timo K <toger5@hotmail.de> * remove wait for invite view Signed-off-by: Timo K <toger5@hotmail.de> * yarn i18n Signed-off-by: Timo K <toger5@hotmail.de> * reset back mute participant count * add logic to show error view when getting removed * include reason whenever someone gets removed from a call. * fix activeRoom not beeing early enough * fix lints * add comment about encryption situation Signed-off-by: Timo K <toger5@hotmail.de> * Fix lockfile * Use (unmerged!) RoomSummary type from the js-sdk Temporarily change the js-sdk dependency to the PR branch that provides that type * review Signed-off-by: Timo K <toger5@hotmail.de> * review (remove participant count unknown) Signed-off-by: Timo K <toger5@hotmail.de> * remove error for unencrypted calls (allow intentional unencrypted calls) Signed-off-by: Timo K <toger5@hotmail.de> * update js-sdk Signed-off-by: Timo K <toger5@hotmail.de> --------- Signed-off-by: Timo K <toger5@hotmail.de> Co-authored-by: Andrew Ferrazzutti <andrewf@element.io>
2024-04-23 15:15:13 +02:00
import { E2eeType } from "../e2ee/e2eeType";
2024-05-17 16:38:00 -04:00
import { makeGridLayout } from "../grid/GridLayout";
2024-06-07 16:59:56 -04:00
import {
type CallLayoutOutputs,
2024-06-07 16:59:56 -04:00
defaultPipAlignment,
defaultSpotlightAlignment,
} from "../grid/CallLayout";
import { makeOneOnOneLayout } from "../grid/OneOnOneLayout";
import { makeSpotlightExpandedLayout } from "../grid/SpotlightExpandedLayout";
import { makeSpotlightLandscapeLayout } from "../grid/SpotlightLandscapeLayout";
import { makeSpotlightPortraitLayout } from "../grid/SpotlightPortraitLayout";
import { GridTileViewModel, type TileViewModel } from "../state/TileViewModel";
import {
ReactionsSenderProvider,
useReactionsSender,
} from "../reactions/useReactionsSender";
Support for generic reactions (#2708) * Initial support for Hand Raise feature Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Refactored to use reaction and redaction events Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Replacing button svg with raised hand emoji Signed-off-by: Milton Moura <miltonmoura@gmail.com> * SpotlightTile should not duplicate the raised hand Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Update src/room/useRaisedHands.tsx Element Call recently changed to AGPL-3.0 * Use relations to load existing reactions when joining the call Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Links to sha commit of matrix-js-sdk that exposes the call membership event id and refactors some async code Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Removing RaiseHand.svg * Check for reaction & redaction capabilities in widget mode Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Fix failing GridTile test Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Center align hand raise. * Add support for displaying the duration of a raised hand. * Add a sound for when a hand is raised. * Refactor raised hand indicator and add tests. * lint * Refactor into own files. * Redact the right thing. * Tidy up useEffect * Lint tests * Remove extra layer * Add better sound. (woosh) * Add a small mode for spotlight * Fix timestamp calculation on relaod. * Fix call border resizing video * lint * Fix and update tests * Allow timer to be configurable. * Add preferences tab for choosing to enable timer. * Drop border from raised hand icon * Handle cases when a new member event happens. * Prevent infinite loop * Major refactor to support various state problems. * Tidy up and finish test rewrites * Add some explanation comments. * Even more comments. * Use proper duration formatter * Remove rerender * Fix redactions not working because they pick up events in transit. * More tidying * Use deferred value * linting * Add tests for cases where we got a reaction from someone else. * Be even less brittle. * Transpose border to GridTile. * First PoC for reactions * hide menu by default * Add lightbulb. * Add reaction indicator. * Add sounds. * Tidy up + add support for floating emoji. * Linting and general stability improvements. * Subscribe to the ecall reaction event type. * fix import * Center emoji picker * Overflow buttons when screen is too narrow * lint * Add settings for disabling animations / sounds. * Make vertical divider more visually distinct. * Make event listener more resillient. * lint * Fix some tests. * Remove old raised hand component * Add new icon * Update text * Update compound hand raised icon. * Add deer. * Fix case where you could send larger strings as emoji * Const the active time. * Document time in css. * Add rock emoji * Add licence file. * Add type def for custom reaction type. * better reaction description * Factor out reactions test structure to utils file. * Add tests for ReactionToggleButton * Add keyboard shortcuts for reaction sending. * type tidyups * lint * Add tests for ReactionAudioRenderer * lint * prettier * i18n sort * final lint? * Preload reaction sounds to prevent delays. * Update rock sounds * add onclick back * Fix test * lint * simplify * Tweak line height * modal impl * Modal refactor attempts. * Remove closed menu test since we're using Modal. * Swap icon, make mobile view better. * Fix mobile view for emoji picker. * Use Intl.Segmenter * Clear timeouts on component close. * Remove useless useCallback * Use prefers-reduced-motion * Add toggle for raise hand. * Add lower hand text * Add lower motion mode. * Decomplicate className system for Modal * Add error for failured to send reaction. * i18n * Spacing for emoji buttons search * Remove unrequired media query * Fix generic sound not playing. * Clear reactions if we're clearing timeouts. * Fix tests * Relabel lower hand * More translations * Add comments on reaction interface * Move polyfill. * lint * Replace deer sound * Another attempt to fix the sizing of the reactions * cleanup * fix button * fix --------- Signed-off-by: Milton Moura <miltonmoura@gmail.com> Co-authored-by: Milton Moura <miltonmoura@gmail.com> Co-authored-by: fkwp <fkwp@users.noreply.github.com>
2024-11-08 17:36:40 +00:00
import { ReactionsAudioRenderer } from "./ReactionAudioRenderer";
2024-11-11 12:07:02 +00:00
import { ReactionsOverlay } from "./ReactionsOverlay";
import { CallEventAudioRenderer } from "./CallEventAudioRenderer";
import {
debugTileLayout as debugTileLayoutSetting,
useSetting,
} from "../settings/settings";
import { ReactionsReader } from "../reactions/ReactionsReader";
import { LivekitRoomAudioRenderer } from "../livekit/MatrixAudioRenderer.tsx";
2025-05-16 11:32:32 +02:00
import { muteAllAudio$ } from "../state/MuteAllAudioModel.ts";
import { useMediaDevices } from "../MediaDevicesContext.ts";
import { EarpieceOverlay } from "./EarpieceOverlay.tsx";
import { useAppBarHidden, useAppBarSecondaryButton } from "../AppBar.tsx";
import { useBehavior } from "../useBehavior.ts";
import { Toast } from "../Toast.tsx";
Fix the interactivity of buttons while reconnecting or in earpiece mode (#3486) * Fix the interactivity of buttons while reconnecting or in earpiece mode When we're in one of these modes, we need to ensure that everything above the overlay (the header and footer buttons) is interactive, while everything obscured by the overlay (the media tiles) is non-interactive and removed from the accessibility tree. It's not a very easy task to trap focus *outside* an element, so the best solution I could come up with is to set tabindex="-1" manually on all interactive elements belonging to the media tiles. * Write a Playwright test for reconnecting * fix lints Signed-off-by: Timo K <toger5@hotmail.de> * fix test Signed-off-by: Timo K <toger5@hotmail.de> * enable http2 for matrx-rtc host to allow the jwt service to talk to the SFU * remove rate limit for delayed events * more time to connect to livekit SFU * Due to a Firefox issue we set the start anchor for the tab test to the Mute microphone button * adapt to most recent Element Web version * Use the "End call" button as proofe for a started call * Currrenty disabled due to recent Element Web - not indicating the number of participants - bypassing Lobby * linting * disable 'can only interact with header and footer while reconnecting' for firefox --------- Signed-off-by: Timo K <toger5@hotmail.de> Co-authored-by: Timo <16718859+toger5@users.noreply.github.com> Co-authored-by: Timo K <toger5@hotmail.de> Co-authored-by: fkwp <github-fkwp@w4ve.de>
2025-09-18 12:58:47 +02:00
import overlayStyles from "../Overlay.module.css";
import { Avatar, Size as AvatarSize } from "../Avatar";
import waitingStyles from "./WaitingForJoin.module.css";
import { prefetchSounds } from "../soundUtils";
import { useAudioContext } from "../useAudioContext";
import ringtoneMp3 from "../sound/ringtone.mp3?url";
import ringtoneOgg from "../sound/ringtone.ogg?url";
import { useTrackProcessorObservable$ } from "../livekit/TrackProcessorContext.tsx";
import { type Layout } from "../state/layout-types.ts";
import { ObservableScope } from "../state/ObservableScope.ts";
const maxTapDurationMs = 400;
export interface ActiveCallProps
Add simple global controls to put the call in picture-in-picture mode (#2573) * Stop sharing state observables when the view model is destroyed By default, observables running with shareReplay will continue running forever even if there are no subscribers. We need to stop them when the view model is destroyed to avoid memory leaks and other unintuitive behavior. * Hydrate the call view model in a less hacky way This ensures that only a single view model is created per call, unlike the previous solution which would create extra view models in strict mode which it was unable to dispose of. The other way was invalid because React gives us no way to reliably dispose of a resource created in the render phase. This is essentially a memory leak fix. * Add simple global controls to put the call in picture-in-picture mode Our web and mobile apps (will) all support putting calls into a picture-in-picture mode. However, it'd be nice to have a way of doing this that's more explicit than a breakpoint, because PiP views could in theory get fairly large. Specifically, on mobile, we want a way to do this that can tell you whether the call is ongoing, and that works even without the widget API (because we support SPA calls in the Element X apps…) To this end, I've created a simple global "controls" API on the window. Right now it only has methods for controlling the picture-in-picture state, but in theory we can expand it to also control mute states, which is current possible via the widget API only. * Fix footer appearing in large PiP views * Add a method for whether you can enter picture-in-picture mode * Have the controls emit booleans directly
2024-08-27 07:47:20 -04:00
extends Omit<InCallViewProps, "vm" | "livekitRoom" | "connState"> {
Knocking support (#2281) * Add joining with knock room creation flow. Also add `WaitForInviteView` after knocking. And appropriate error views when knock failed or gets rejected. Signed-off-by: Timo K <toger5@hotmail.de> * Refactor encryption information. We had lots of enums and booleans to describe the encryption situation. Now we only use the `EncryptionSystem` "enum" which contains the additional information like sharedKey. (and we don't use the isRoomE2EE function that is somewhat confusing since it checks `return widget === null && !room.getCanonicalAlias();` which is only indirectly related to e2ee) Signed-off-by: Timo K <toger5@hotmail.de> * Update recent list. - Don't use deprecated `groupCallEventHander` anymore (it used the old `m.call` state event.) - make the recent list reactive (getting removed from a call removes the item from the list) - support having rooms without shared secret but actual matrix encryption in the recent list - change the share link creation button so that we create a link with pwd for sharedKey rooms and with `perParticipantE2EE=true` for matrix encrypted rooms. Signed-off-by: Timo K <toger5@hotmail.de> * fix types Signed-off-by: Timo K <toger5@hotmail.de> * patch js-sdk for linter Signed-off-by: Timo K <toger5@hotmail.de> * ignore ts expect error Signed-off-by: Timo K <toger5@hotmail.de> * Fix error in widget mode. We cannot call client.getRoomSummary in widget mode. The code path needs to throw before reaching this call. (In general we should never call getRoomSummary if getRoom returns a room) Signed-off-by: Timo K <toger5@hotmail.de> * tempDemo Signed-off-by: Timo K <toger5@hotmail.de> * remove wait for invite view Signed-off-by: Timo K <toger5@hotmail.de> * yarn i18n Signed-off-by: Timo K <toger5@hotmail.de> * reset back mute participant count * add logic to show error view when getting removed * include reason whenever someone gets removed from a call. * fix activeRoom not beeing early enough * fix lints * add comment about encryption situation Signed-off-by: Timo K <toger5@hotmail.de> * Fix lockfile * Use (unmerged!) RoomSummary type from the js-sdk Temporarily change the js-sdk dependency to the PR branch that provides that type * review Signed-off-by: Timo K <toger5@hotmail.de> * review (remove participant count unknown) Signed-off-by: Timo K <toger5@hotmail.de> * remove error for unencrypted calls (allow intentional unencrypted calls) Signed-off-by: Timo K <toger5@hotmail.de> * update js-sdk Signed-off-by: Timo K <toger5@hotmail.de> --------- Signed-off-by: Timo K <toger5@hotmail.de> Co-authored-by: Andrew Ferrazzutti <andrewf@element.io>
2024-04-23 15:15:13 +02:00
e2eeSystem: EncryptionSystem;
// TODO refactor those reasons into an enum
onLeft: (reason: "user" | "timeout" | "decline" | "allOthersLeft") => void;
}
export const ActiveCall: FC<ActiveCallProps> = (props) => {
const mediaDevices = useMediaDevices();
Add simple global controls to put the call in picture-in-picture mode (#2573) * Stop sharing state observables when the view model is destroyed By default, observables running with shareReplay will continue running forever even if there are no subscribers. We need to stop them when the view model is destroyed to avoid memory leaks and other unintuitive behavior. * Hydrate the call view model in a less hacky way This ensures that only a single view model is created per call, unlike the previous solution which would create extra view models in strict mode which it was unable to dispose of. The other way was invalid because React gives us no way to reliably dispose of a resource created in the render phase. This is essentially a memory leak fix. * Add simple global controls to put the call in picture-in-picture mode Our web and mobile apps (will) all support putting calls into a picture-in-picture mode. However, it'd be nice to have a way of doing this that's more explicit than a breakpoint, because PiP views could in theory get fairly large. Specifically, on mobile, we want a way to do this that can tell you whether the call is ongoing, and that works even without the widget API (because we support SPA calls in the Element X apps…) To this end, I've created a simple global "controls" API on the window. Right now it only has methods for controlling the picture-in-picture state, but in theory we can expand it to also control mute states, which is current possible via the widget API only. * Fix footer appearing in large PiP views * Add a method for whether you can enter picture-in-picture mode * Have the controls emit booleans directly
2024-08-27 07:47:20 -04:00
const [vm, setVm] = useState<CallViewModel | null>(null);
const { autoLeaveWhenOthersLeft, waitForCallPickup, sendNotificationType } =
useUrlParams();
const trackProcessorState$ = useTrackProcessorObservable$();
useEffect(() => {
const scope = new ObservableScope();
const reactionsReader = new ReactionsReader(scope, props.rtcSession);
const vm = new CallViewModel(
scope,
props.rtcSession,
props.matrixRoom,
mediaDevices,
2025-08-29 18:46:24 +02:00
props.muteStates,
{
encryptionSystem: props.e2eeSystem,
autoLeaveWhenOthersLeft,
waitForCallPickup: waitForCallPickup && sendNotificationType === "ring",
},
reactionsReader.raisedHands$,
reactionsReader.reactions$,
scope.behavior(trackProcessorState$),
);
setVm(vm);
vm.leave$.pipe(scope.bind()).subscribe(props.onLeft);
2024-06-04 11:20:25 -04:00
return (): void => {
scope.end();
};
}, [
props.rtcSession,
props.matrixRoom,
mediaDevices,
2025-08-29 18:46:24 +02:00
props.muteStates,
props.e2eeSystem,
autoLeaveWhenOthersLeft,
sendNotificationType,
waitForCallPickup,
props.onLeft,
trackProcessorState$,
]);
Add simple global controls to put the call in picture-in-picture mode (#2573) * Stop sharing state observables when the view model is destroyed By default, observables running with shareReplay will continue running forever even if there are no subscribers. We need to stop them when the view model is destroyed to avoid memory leaks and other unintuitive behavior. * Hydrate the call view model in a less hacky way This ensures that only a single view model is created per call, unlike the previous solution which would create extra view models in strict mode which it was unable to dispose of. The other way was invalid because React gives us no way to reliably dispose of a resource created in the render phase. This is essentially a memory leak fix. * Add simple global controls to put the call in picture-in-picture mode Our web and mobile apps (will) all support putting calls into a picture-in-picture mode. However, it'd be nice to have a way of doing this that's more explicit than a breakpoint, because PiP views could in theory get fairly large. Specifically, on mobile, we want a way to do this that can tell you whether the call is ongoing, and that works even without the widget API (because we support SPA calls in the Element X apps…) To this end, I've created a simple global "controls" API on the window. Right now it only has methods for controlling the picture-in-picture state, but in theory we can expand it to also control mute states, which is current possible via the widget API only. * Fix footer appearing in large PiP views * Add a method for whether you can enter picture-in-picture mode * Have the controls emit booleans directly
2024-08-27 07:47:20 -04:00
if (vm === null) return null;
return (
<ReactionsSenderProvider vm={vm} rtcSession={props.rtcSession}>
<InCallView {...props} vm={vm} />
</ReactionsSenderProvider>
);
};
export interface InCallViewProps {
2022-08-02 00:46:16 +02:00
client: MatrixClient;
Add simple global controls to put the call in picture-in-picture mode (#2573) * Stop sharing state observables when the view model is destroyed By default, observables running with shareReplay will continue running forever even if there are no subscribers. We need to stop them when the view model is destroyed to avoid memory leaks and other unintuitive behavior. * Hydrate the call view model in a less hacky way This ensures that only a single view model is created per call, unlike the previous solution which would create extra view models in strict mode which it was unable to dispose of. The other way was invalid because React gives us no way to reliably dispose of a resource created in the render phase. This is essentially a memory leak fix. * Add simple global controls to put the call in picture-in-picture mode Our web and mobile apps (will) all support putting calls into a picture-in-picture mode. However, it'd be nice to have a way of doing this that's more explicit than a breakpoint, because PiP views could in theory get fairly large. Specifically, on mobile, we want a way to do this that can tell you whether the call is ongoing, and that works even without the widget API (because we support SPA calls in the Element X apps…) To this end, I've created a simple global "controls" API on the window. Right now it only has methods for controlling the picture-in-picture state, but in theory we can expand it to also control mute states, which is current possible via the widget API only. * Fix footer appearing in large PiP views * Add a method for whether you can enter picture-in-picture mode * Have the controls emit booleans directly
2024-08-27 07:47:20 -04:00
vm: CallViewModel;
matrixInfo: MatrixInfo;
rtcSession: MatrixRTCSession;
matrixRoom: MatrixRoom;
muteStates: MuteStates;
header: HeaderStyle;
otelGroupCallMembership?: OTelGroupCallMembership;
onShareClick: (() => void) | null;
2022-08-02 00:46:16 +02:00
}
export const InCallView: FC<InCallViewProps> = ({
client,
Add simple global controls to put the call in picture-in-picture mode (#2573) * Stop sharing state observables when the view model is destroyed By default, observables running with shareReplay will continue running forever even if there are no subscribers. We need to stop them when the view model is destroyed to avoid memory leaks and other unintuitive behavior. * Hydrate the call view model in a less hacky way This ensures that only a single view model is created per call, unlike the previous solution which would create extra view models in strict mode which it was unable to dispose of. The other way was invalid because React gives us no way to reliably dispose of a resource created in the render phase. This is essentially a memory leak fix. * Add simple global controls to put the call in picture-in-picture mode Our web and mobile apps (will) all support putting calls into a picture-in-picture mode. However, it'd be nice to have a way of doing this that's more explicit than a breakpoint, because PiP views could in theory get fairly large. Specifically, on mobile, we want a way to do this that can tell you whether the call is ongoing, and that works even without the widget API (because we support SPA calls in the Element X apps…) To this end, I've created a simple global "controls" API on the window. Right now it only has methods for controlling the picture-in-picture state, but in theory we can expand it to also control mute states, which is current possible via the widget API only. * Fix footer appearing in large PiP views * Add a method for whether you can enter picture-in-picture mode * Have the controls emit booleans directly
2024-08-27 07:47:20 -04:00
vm,
matrixInfo,
matrixRoom,
muteStates,
header: headerStyle,
onShareClick,
}) => {
const { t } = useTranslation();
const { supportsReactions, sendReaction, toggleRaisedHand } =
useReactionsSender();
Hand raise feature (#2542) * Initial support for Hand Raise feature Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Refactored to use reaction and redaction events Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Replacing button svg with raised hand emoji Signed-off-by: Milton Moura <miltonmoura@gmail.com> * SpotlightTile should not duplicate the raised hand Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Update src/room/useRaisedHands.tsx Element Call recently changed to AGPL-3.0 * Use relations to load existing reactions when joining the call Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Links to sha commit of matrix-js-sdk that exposes the call membership event id and refactors some async code Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Removing RaiseHand.svg * Check for reaction & redaction capabilities in widget mode Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Fix failing GridTile test Signed-off-by: Milton Moura <miltonmoura@gmail.com> * Center align hand raise. * Add support for displaying the duration of a raised hand. * Add a sound for when a hand is raised. * Refactor raised hand indicator and add tests. * lint * Refactor into own files. * Redact the right thing. * Tidy up useEffect * Lint tests * Remove extra layer * Add better sound. (woosh) * Add a small mode for spotlight * Fix timestamp calculation on relaod. * Fix call border resizing video * lint * Fix and update tests * Allow timer to be configurable. * Add preferences tab for choosing to enable timer. * Drop border from raised hand icon * Handle cases when a new member event happens. * Prevent infinite loop * Major refactor to support various state problems. * Tidy up and finish test rewrites * Add some explanation comments. * Even more comments. * Use proper duration formatter * Remove rerender * Fix redactions not working because they pick up events in transit. * More tidying * Use deferred value * linting * Add tests for cases where we got a reaction from someone else. * Be even less brittle. * Transpose border to GridTile. * lint --------- Signed-off-by: Milton Moura <miltonmoura@gmail.com> Co-authored-by: fkwp <fkwp@users.noreply.github.com> Co-authored-by: Half-Shot <will@half-shot.uk> Co-authored-by: Will Hunt <github@half-shot.uk>
2024-11-04 08:54:13 -01:00
useWakeLock();
// TODO-MULTI-SFU This is unused now??
2025-10-07 16:00:59 +02:00
// const connectionState = useObservableEagerState(vm.livekitConnectionState$);
// annoyingly we don't get the disconnection reason this way,
// only by listening for the emitted event
// This needs to be done differential. with the vm connection state we start with Disconnected.
// TODO-MULTI-SFU decide how to handle this properly
// @BillCarsonFr
// if (connectionState === ConnectionState.Disconnected)
// throw new ConnectionLostError();
const containerRef1 = useRef<HTMLDivElement | null>(null);
const [containerRef2, bounds] = useMeasure();
// Merge the refs so they can attach to the same element
const containerRef = useMergedRefs(containerRef1, containerRef2);
2025-09-24 13:54:54 -04:00
const { showControls } = useUrlParams();
const muteAllAudio = useBehavior(muteAllAudio$);
// Call pickup state and display names are needed for waiting overlay/sounds
const callPickupState = useBehavior(vm.callPickupState$);
// Preload a waiting and decline sounds
const pickupPhaseSoundCache = useInitial(async () => {
return prefetchSounds({
waiting: { mp3: ringtoneMp3, ogg: ringtoneOgg },
});
});
const pickupPhaseAudio = useAudioContext({
sounds: pickupPhaseSoundCache,
latencyHint: "interactive",
muted: muteAllAudio,
});
2025-08-29 18:46:24 +02:00
const audioEnabled = useBehavior(muteStates.audio.enabled$);
const videoEnabled = useBehavior(muteStates.video.enabled$);
const toggleAudio = useBehavior(muteStates.audio.toggle$);
const toggleVideo = useBehavior(muteStates.video.toggle$);
const setAudioEnabled = useBehavior(muteStates.audio.setEnabled$);
// This function incorrectly assumes that there is a camera and microphone, which is not always the case.
// TODO: Make sure that this module is resilient when it comes to camera/microphone availability!
useCallViewKeyboardShortcuts(
containerRef1,
2025-08-29 18:46:24 +02:00
toggleAudio,
toggleVideo,
setAudioEnabled,
(reaction) => void sendReaction(reaction),
() => void toggleRaisedHand(),
);
// const allLivekitRooms = useBehavior(vm.allLivekitRooms$);
const audioParticipants = useBehavior(vm.audioParticipants$);
const participantCount = useBehavior(vm.participantCount$);
const reconnecting = useBehavior(vm.reconnecting$);
const windowMode = useBehavior(vm.windowMode$);
const layout = useBehavior(vm.layout$);
const tileStoreGeneration = useBehavior(vm.tileStoreGeneration$);
const [debugTileLayout] = useSetting(debugTileLayoutSetting);
const gridMode = useBehavior(vm.gridMode$);
const showHeader = useBehavior(vm.showHeader$);
const showFooter = useBehavior(vm.showFooter$);
const earpieceMode = useBehavior(vm.earpieceMode$);
const audioOutputSwitcher = useBehavior(vm.audioOutputSwitcher$);
2025-09-24 13:54:54 -04:00
const sharingScreen = useBehavior(vm.sharingScreen$);
const fatalCallError = useBehavior(vm.configError$);
// Stop the rendering and throw for the error boundary
if (fatalCallError) throw fatalCallError;
// We need to set the proper timings on the animation based upon the sound length.
const ringDuration = pickupPhaseAudio?.soundDuration["waiting"] ?? 1;
useEffect((): (() => void) => {
// The CSS animation includes the delay, so we must double the length of the sound.
window.document.body.style.setProperty(
"--call-ring-duration-s",
`${ringDuration * 2}s`,
);
window.document.body.style.setProperty(
"--call-ring-delay-s",
`${ringDuration}s`,
);
// Remove properties when we unload.
return () => {
window.document.body.style.removeProperty("--call-ring-duration-s");
window.document.body.style.removeProperty("--call-ring-delay-s");
};
}, [pickupPhaseAudio?.soundDuration, ringDuration]);
// When waiting for pickup, loop a waiting sound
useEffect((): void | (() => void) => {
if (callPickupState !== "ringing" || !pickupPhaseAudio) return;
const endSound = pickupPhaseAudio.playSoundLooping("waiting", ringDuration);
return () => {
void endSound().catch((e) => {
logger.error("Failed to stop ringing sound", e);
});
};
}, [callPickupState, pickupPhaseAudio, ringDuration]);
// Waiting UI overlay
const waitingOverlay: JSX.Element | null = useMemo(() => {
// No overlay if not in ringing state
if (callPickupState !== "ringing") return null;
// Use room state for other participants data (the one that we likely want to reach)
// TODO: this screams it wants to be a behavior in the vm.
const roomOthers = [
...matrixRoom.getMembersWithMembership("join"),
...matrixRoom.getMembersWithMembership("invite"),
].filter((m) => m.userId !== client.getUserId());
// Yield if there are not other members in the room.
if (roomOthers.length === 0) return null;
const otherMember = roomOthers.length > 0 ? roomOthers[0] : undefined;
const isOneOnOne = roomOthers.length === 1 && otherMember;
const text = isOneOnOne
? `Waiting for ${otherMember.name ?? otherMember.userId} to join…`
: "Waiting for other participants…";
const avatarMxc = isOneOnOne
? (otherMember.getMxcAvatarUrl?.() ?? undefined)
: (matrixRoom.getMxcAvatarUrl() ?? undefined);
return (
<div className={classNames(overlayStyles.bg, waitingStyles.overlay)}>
<div
className={classNames(overlayStyles.content, waitingStyles.content)}
>
<div className={waitingStyles.pulse}>
<Avatar
id={isOneOnOne ? otherMember.userId : matrixRoom.roomId}
name={isOneOnOne ? otherMember.name : matrixRoom.name}
src={avatarMxc}
size={AvatarSize.XL}
/>
</div>
<Text size="md" className={waitingStyles.text}>
{text}
</Text>
</div>
</div>
);
}, [callPickupState, client, matrixRoom]);
// Ideally we could detect taps by listening for click events and checking
// that the pointerType of the event is "touch", but this isn't yet supported
// in Safari: https://developer.mozilla.org/en-US/docs/Web/API/Element/click_event#browser_compatibility
// Instead we have to watch for sufficiently fast touch events.
const touchStart = useRef<number | null>(null);
const onTouchStart = useCallback(() => (touchStart.current = Date.now()), []);
const onTouchEnd = useCallback(() => {
const start = touchStart.current;
if (start !== null && Date.now() - start <= maxTapDurationMs)
vm.tapScreen();
touchStart.current = null;
}, [vm]);
const onTouchCancel = useCallback(() => (touchStart.current = null), []);
// We also need to tell the footer controls to prevent touch events from
// bubbling up, or else the footer will be dismissed before a click/change
// event can be registered on the control
const onControlsTouchEnd = useCallback(
(e: TouchEvent) => {
// Somehow applying pointer-events: none to the controls when the footer
// is hidden is not enough to stop clicks from happening as the footer
// becomes visible, so we check manually whether the footer is shown
if (showFooter) {
e.stopPropagation();
vm.tapControls();
} else {
e.preventDefault();
}
},
[vm, showFooter],
);
const onPointerMove = useCallback(
(e: PointerEvent) => {
if (e.pointerType === "mouse") vm.hoverScreen();
},
[vm],
);
const onPointerOut = useCallback(() => vm.unhoverScreen(), [vm]);
const [settingsModalOpen, setSettingsModalOpen] = useState(false);
const [settingsTab, setSettingsTab] = useState(defaultSettingsTab);
const openSettings = useCallback(
() => setSettingsModalOpen(true),
[setSettingsModalOpen],
);
const closeSettings = useCallback(
() => setSettingsModalOpen(false),
[setSettingsModalOpen],
);
const openProfile = useMemo(
() =>
// Profile settings are unavailable in widget mode
widget === null
? (): void => {
setSettingsTab("profile");
setSettingsModalOpen(true);
}
: null,
[setSettingsTab, setSettingsModalOpen],
);
const [headerRef, headerBounds] = useMeasure();
const [footerRef, footerBounds] = useMeasure();
2024-05-17 16:38:00 -04:00
const gridBounds = useMemo(
() => ({
width: bounds.width,
height:
bounds.height -
headerBounds.height -
(windowMode === "flat" ? 0 : footerBounds.height),
}),
[
bounds.width,
bounds.height,
headerBounds.height,
footerBounds.height,
windowMode,
],
);
const gridBoundsObservable$ = useObservable(
(inputs$) => inputs$.pipe(map(([gridBounds]) => gridBounds)),
[gridBounds],
);
2024-05-17 16:38:00 -04:00
const spotlightAlignment$ = useInitial(
2024-06-07 16:59:56 -04:00
() => new BehaviorSubject(defaultSpotlightAlignment),
);
const pipAlignment$ = useInitial(
2024-06-07 16:59:56 -04:00
() => new BehaviorSubject(defaultPipAlignment),
);
2024-05-17 16:38:00 -04:00
const setGridMode = useCallback(
2024-06-07 17:29:48 -04:00
(mode: GridMode) => vm.setGridMode(mode),
[vm],
);
2024-06-07 17:29:48 -04:00
useEffect(() => {
widget?.api.transport
.send(
gridMode === "grid"
? ElementWidgetActions.TileLayout
: ElementWidgetActions.SpotlightLayout,
{},
)
.catch((e) => {
logger.error("Failed to send layout change to widget API", e);
});
2024-06-07 17:29:48 -04:00
}, [gridMode]);
useEffect(() => {
if (widget) {
const onTileLayout = (ev: CustomEvent<IWidgetApiRequest>): void => {
setGridMode("grid");
widget!.api.transport.reply(ev.detail, {});
};
const onSpotlightLayout = (ev: CustomEvent<IWidgetApiRequest>): void => {
setGridMode("spotlight");
widget!.api.transport.reply(ev.detail, {});
};
widget.lazyActions.on(ElementWidgetActions.TileLayout, onTileLayout);
widget.lazyActions.on(
ElementWidgetActions.SpotlightLayout,
onSpotlightLayout,
);
return (): void => {
widget!.lazyActions.off(ElementWidgetActions.TileLayout, onTileLayout);
widget!.lazyActions.off(
ElementWidgetActions.SpotlightLayout,
onSpotlightLayout,
);
};
}
}, [setGridMode]);
useAppBarSecondaryButton(
useMemo(() => {
if (audioOutputSwitcher === null) return null;
const isEarpieceTarget = audioOutputSwitcher.targetOutput === "earpiece";
const Icon = isEarpieceTarget ? VoiceCallSolidIcon : VolumeOnSolidIcon;
const label = isEarpieceTarget
? t("settings.devices.handset")
: t("settings.devices.loudspeaker");
return (
<Tooltip label={label}>
<IconButton
onClick={(e) => {
e.preventDefault();
audioOutputSwitcher.switch();
}}
>
<Icon />
</IconButton>
</Tooltip>
);
}, [t, audioOutputSwitcher]),
);
useAppBarHidden(!showHeader);
let header: ReactNode = null;
if (showHeader) {
switch (headerStyle) {
case "none":
// Cosmetic header to fill out space while still affecting the bounds of
// the grid
header = (
<div
className={classNames(styles.header, styles.filler)}
ref={headerRef}
/>
);
break;
case "standard":
header = (
<Header
className={styles.header}
ref={headerRef}
disconnectedBanner={false} // This screen has its own 'reconnecting' toast
>
<LeftNav>
<RoomHeaderInfo
id={matrixInfo.roomId}
name={matrixInfo.roomName}
avatarUrl={matrixInfo.roomAvatar}
encrypted={matrixInfo.e2eeSystem.kind !== E2eeType.NONE}
participantCount={participantCount}
/>
</LeftNav>
<RightNav>
{showControls && onShareClick !== null && (
<InviteButton
className={styles.invite}
onClick={onShareClick}
/>
)}
</RightNav>
</Header>
);
}
}
Fix the interactivity of buttons while reconnecting or in earpiece mode (#3486) * Fix the interactivity of buttons while reconnecting or in earpiece mode When we're in one of these modes, we need to ensure that everything above the overlay (the header and footer buttons) is interactive, while everything obscured by the overlay (the media tiles) is non-interactive and removed from the accessibility tree. It's not a very easy task to trap focus *outside* an element, so the best solution I could come up with is to set tabindex="-1" manually on all interactive elements belonging to the media tiles. * Write a Playwright test for reconnecting * fix lints Signed-off-by: Timo K <toger5@hotmail.de> * fix test Signed-off-by: Timo K <toger5@hotmail.de> * enable http2 for matrx-rtc host to allow the jwt service to talk to the SFU * remove rate limit for delayed events * more time to connect to livekit SFU * Due to a Firefox issue we set the start anchor for the tab test to the Mute microphone button * adapt to most recent Element Web version * Use the "End call" button as proofe for a started call * Currrenty disabled due to recent Element Web - not indicating the number of participants - bypassing Lobby * linting * disable 'can only interact with header and footer while reconnecting' for firefox --------- Signed-off-by: Timo K <toger5@hotmail.de> Co-authored-by: Timo <16718859+toger5@users.noreply.github.com> Co-authored-by: Timo K <toger5@hotmail.de> Co-authored-by: fkwp <github-fkwp@w4ve.de>
2025-09-18 12:58:47 +02:00
// The reconnecting toast cannot be dismissed
const onDismissReconnectingToast = useCallback(() => {}, []);
// We need to use a non-modal toast to avoid trapping focus within the toast.
// However, a non-modal toast will not render any background overlay on its
// own, so we must render one manually.
const reconnectingToast = (
<>
<div
className={classNames(overlayStyles.bg, overlayStyles.animate)}
data-state={reconnecting ? "open" : "closed"}
/>
<Toast
onDismiss={onDismissReconnectingToast}
open={reconnecting}
modal={false}
>
{t("common.reconnecting")}
</Toast>
</>
);
const earpieceOverlay = (
<EarpieceOverlay
show={earpieceMode && !reconnecting}
onBackToVideoPressed={audioOutputSwitcher?.switch}
/>
);
// If the reconnecting toast or earpiece overlay obscures the media tiles, we
// need to remove them from the accessibility tree and block focus.
const contentObscured = reconnecting || earpieceMode;
const Tile = useMemo(
() =>
function Tile({
ref,
className,
style,
targetWidth,
targetHeight,
model,
}: TileProps<TileViewModel, HTMLDivElement>): ReactNode {
const spotlightExpanded = useBehavior(vm.spotlightExpanded$);
const onToggleExpanded = useBehavior(vm.toggleSpotlightExpanded$);
const showSpeakingIndicatorsValue = useBehavior(
vm.showSpeakingIndicators$,
);
const showSpotlightIndicatorsValue = useBehavior(
vm.showSpotlightIndicators$,
);
Keep tiles in a stable order (#2670) * Keep tiles in a stable order This introduces a new layer of abstraction on top of MediaViewModel: TileViewModel, which gives us a place to store data relating to tiles rather than their media, and also generally makes it easier to reason about tiles as they move about the call layout. I have created a class called TileStore to keep track of these tiles. This allows us to swap out the media shown on a tile as the spotlight speaker changes, and avoid moving tiles around unless they really need to jump between the visible/invisible regions of the layout. * Don't throttle spotlight updates Since we now assume that the spotlight and grid will be in sync (i.e. an active speaker in one will behave as an active speaker in the other), we don't want the spotlight to ever lag behind due to throttling. If this causes usability issues we should maybe look into making LiveKit's 'speaking' indicators less erratic first. * Make layout shifts due to a change in speaker less surprising Although we try now to avoid layout shifts due to the spotlight speaker changing wherever possible, a spotlight speaker coming from off screen can still trigger one. Let's shift the layout a bit more gracefully in this case. * Improve the tile ordering tests * Maximize the spotlight tile in portrait layout * Tell tiles whether they're actually visible in a more timely manner * Fix test * Fix speaking indicators logic * Improve readability of marbles * Fix test case --------- Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
2024-11-06 04:36:48 -05:00
return model instanceof GridTileViewModel ? (
<GridTile
ref={ref}
Keep tiles in a stable order (#2670) * Keep tiles in a stable order This introduces a new layer of abstraction on top of MediaViewModel: TileViewModel, which gives us a place to store data relating to tiles rather than their media, and also generally makes it easier to reason about tiles as they move about the call layout. I have created a class called TileStore to keep track of these tiles. This allows us to swap out the media shown on a tile as the spotlight speaker changes, and avoid moving tiles around unless they really need to jump between the visible/invisible regions of the layout. * Don't throttle spotlight updates Since we now assume that the spotlight and grid will be in sync (i.e. an active speaker in one will behave as an active speaker in the other), we don't want the spotlight to ever lag behind due to throttling. If this causes usability issues we should maybe look into making LiveKit's 'speaking' indicators less erratic first. * Make layout shifts due to a change in speaker less surprising Although we try now to avoid layout shifts due to the spotlight speaker changing wherever possible, a spotlight speaker coming from off screen can still trigger one. Let's shift the layout a bit more gracefully in this case. * Improve the tile ordering tests * Maximize the spotlight tile in portrait layout * Tell tiles whether they're actually visible in a more timely manner * Fix test * Fix speaking indicators logic * Improve readability of marbles * Fix test case --------- Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
2024-11-06 04:36:48 -05:00
vm={model}
onOpenProfile={openProfile}
targetWidth={targetWidth}
targetHeight={targetHeight}
className={classNames(className, styles.tile)}
style={style}
showSpeakingIndicators={showSpeakingIndicatorsValue}
Fix the interactivity of buttons while reconnecting or in earpiece mode (#3486) * Fix the interactivity of buttons while reconnecting or in earpiece mode When we're in one of these modes, we need to ensure that everything above the overlay (the header and footer buttons) is interactive, while everything obscured by the overlay (the media tiles) is non-interactive and removed from the accessibility tree. It's not a very easy task to trap focus *outside* an element, so the best solution I could come up with is to set tabindex="-1" manually on all interactive elements belonging to the media tiles. * Write a Playwright test for reconnecting * fix lints Signed-off-by: Timo K <toger5@hotmail.de> * fix test Signed-off-by: Timo K <toger5@hotmail.de> * enable http2 for matrx-rtc host to allow the jwt service to talk to the SFU * remove rate limit for delayed events * more time to connect to livekit SFU * Due to a Firefox issue we set the start anchor for the tab test to the Mute microphone button * adapt to most recent Element Web version * Use the "End call" button as proofe for a started call * Currrenty disabled due to recent Element Web - not indicating the number of participants - bypassing Lobby * linting * disable 'can only interact with header and footer while reconnecting' for firefox --------- Signed-off-by: Timo K <toger5@hotmail.de> Co-authored-by: Timo <16718859+toger5@users.noreply.github.com> Co-authored-by: Timo K <toger5@hotmail.de> Co-authored-by: fkwp <github-fkwp@w4ve.de>
2025-09-18 12:58:47 +02:00
focusable={!contentObscured}
/>
) : (
<SpotlightTile
ref={ref}
Keep tiles in a stable order (#2670) * Keep tiles in a stable order This introduces a new layer of abstraction on top of MediaViewModel: TileViewModel, which gives us a place to store data relating to tiles rather than their media, and also generally makes it easier to reason about tiles as they move about the call layout. I have created a class called TileStore to keep track of these tiles. This allows us to swap out the media shown on a tile as the spotlight speaker changes, and avoid moving tiles around unless they really need to jump between the visible/invisible regions of the layout. * Don't throttle spotlight updates Since we now assume that the spotlight and grid will be in sync (i.e. an active speaker in one will behave as an active speaker in the other), we don't want the spotlight to ever lag behind due to throttling. If this causes usability issues we should maybe look into making LiveKit's 'speaking' indicators less erratic first. * Make layout shifts due to a change in speaker less surprising Although we try now to avoid layout shifts due to the spotlight speaker changing wherever possible, a spotlight speaker coming from off screen can still trigger one. Let's shift the layout a bit more gracefully in this case. * Improve the tile ordering tests * Maximize the spotlight tile in portrait layout * Tell tiles whether they're actually visible in a more timely manner * Fix test * Fix speaking indicators logic * Improve readability of marbles * Fix test case --------- Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
2024-11-06 04:36:48 -05:00
vm={model}
expanded={spotlightExpanded}
onToggleExpanded={onToggleExpanded}
targetWidth={targetWidth}
targetHeight={targetHeight}
showIndicators={showSpotlightIndicatorsValue}
Fix the interactivity of buttons while reconnecting or in earpiece mode (#3486) * Fix the interactivity of buttons while reconnecting or in earpiece mode When we're in one of these modes, we need to ensure that everything above the overlay (the header and footer buttons) is interactive, while everything obscured by the overlay (the media tiles) is non-interactive and removed from the accessibility tree. It's not a very easy task to trap focus *outside* an element, so the best solution I could come up with is to set tabindex="-1" manually on all interactive elements belonging to the media tiles. * Write a Playwright test for reconnecting * fix lints Signed-off-by: Timo K <toger5@hotmail.de> * fix test Signed-off-by: Timo K <toger5@hotmail.de> * enable http2 for matrx-rtc host to allow the jwt service to talk to the SFU * remove rate limit for delayed events * more time to connect to livekit SFU * Due to a Firefox issue we set the start anchor for the tab test to the Mute microphone button * adapt to most recent Element Web version * Use the "End call" button as proofe for a started call * Currrenty disabled due to recent Element Web - not indicating the number of participants - bypassing Lobby * linting * disable 'can only interact with header and footer while reconnecting' for firefox --------- Signed-off-by: Timo K <toger5@hotmail.de> Co-authored-by: Timo <16718859+toger5@users.noreply.github.com> Co-authored-by: Timo K <toger5@hotmail.de> Co-authored-by: fkwp <github-fkwp@w4ve.de>
2025-09-18 12:58:47 +02:00
focusable={!contentObscured}
className={classNames(className, styles.tile)}
style={style}
/>
);
},
Fix the interactivity of buttons while reconnecting or in earpiece mode (#3486) * Fix the interactivity of buttons while reconnecting or in earpiece mode When we're in one of these modes, we need to ensure that everything above the overlay (the header and footer buttons) is interactive, while everything obscured by the overlay (the media tiles) is non-interactive and removed from the accessibility tree. It's not a very easy task to trap focus *outside* an element, so the best solution I could come up with is to set tabindex="-1" manually on all interactive elements belonging to the media tiles. * Write a Playwright test for reconnecting * fix lints Signed-off-by: Timo K <toger5@hotmail.de> * fix test Signed-off-by: Timo K <toger5@hotmail.de> * enable http2 for matrx-rtc host to allow the jwt service to talk to the SFU * remove rate limit for delayed events * more time to connect to livekit SFU * Due to a Firefox issue we set the start anchor for the tab test to the Mute microphone button * adapt to most recent Element Web version * Use the "End call" button as proofe for a started call * Currrenty disabled due to recent Element Web - not indicating the number of participants - bypassing Lobby * linting * disable 'can only interact with header and footer while reconnecting' for firefox --------- Signed-off-by: Timo K <toger5@hotmail.de> Co-authored-by: Timo <16718859+toger5@users.noreply.github.com> Co-authored-by: Timo K <toger5@hotmail.de> Co-authored-by: fkwp <github-fkwp@w4ve.de>
2025-09-18 12:58:47 +02:00
[vm, openProfile, contentObscured],
);
const layouts = useMemo(() => {
const inputs = {
minBounds$: gridBoundsObservable$,
spotlightAlignment$,
pipAlignment$,
};
return {
grid: makeGridLayout(inputs),
"spotlight-landscape": makeSpotlightLandscapeLayout(inputs),
"spotlight-portrait": makeSpotlightPortraitLayout(inputs),
"spotlight-expanded": makeSpotlightExpandedLayout(inputs),
"one-on-one": makeOneOnOneLayout(inputs),
};
}, [gridBoundsObservable$, spotlightAlignment$, pipAlignment$]);
const renderContent = (): JSX.Element => {
if (layout.type === "pip") {
return (
<SpotlightTile
className={classNames(styles.tile, styles.maximised)}
Keep tiles in a stable order (#2670) * Keep tiles in a stable order This introduces a new layer of abstraction on top of MediaViewModel: TileViewModel, which gives us a place to store data relating to tiles rather than their media, and also generally makes it easier to reason about tiles as they move about the call layout. I have created a class called TileStore to keep track of these tiles. This allows us to swap out the media shown on a tile as the spotlight speaker changes, and avoid moving tiles around unless they really need to jump between the visible/invisible regions of the layout. * Don't throttle spotlight updates Since we now assume that the spotlight and grid will be in sync (i.e. an active speaker in one will behave as an active speaker in the other), we don't want the spotlight to ever lag behind due to throttling. If this causes usability issues we should maybe look into making LiveKit's 'speaking' indicators less erratic first. * Make layout shifts due to a change in speaker less surprising Although we try now to avoid layout shifts due to the spotlight speaker changing wherever possible, a spotlight speaker coming from off screen can still trigger one. Let's shift the layout a bit more gracefully in this case. * Improve the tile ordering tests * Maximize the spotlight tile in portrait layout * Tell tiles whether they're actually visible in a more timely manner * Fix test * Fix speaking indicators logic * Improve readability of marbles * Fix test case --------- Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
2024-11-06 04:36:48 -05:00
vm={layout.spotlight}
expanded
onToggleExpanded={null}
targetWidth={gridBounds.height}
targetHeight={gridBounds.width}
showIndicators={false}
Fix the interactivity of buttons while reconnecting or in earpiece mode (#3486) * Fix the interactivity of buttons while reconnecting or in earpiece mode When we're in one of these modes, we need to ensure that everything above the overlay (the header and footer buttons) is interactive, while everything obscured by the overlay (the media tiles) is non-interactive and removed from the accessibility tree. It's not a very easy task to trap focus *outside* an element, so the best solution I could come up with is to set tabindex="-1" manually on all interactive elements belonging to the media tiles. * Write a Playwright test for reconnecting * fix lints Signed-off-by: Timo K <toger5@hotmail.de> * fix test Signed-off-by: Timo K <toger5@hotmail.de> * enable http2 for matrx-rtc host to allow the jwt service to talk to the SFU * remove rate limit for delayed events * more time to connect to livekit SFU * Due to a Firefox issue we set the start anchor for the tab test to the Mute microphone button * adapt to most recent Element Web version * Use the "End call" button as proofe for a started call * Currrenty disabled due to recent Element Web - not indicating the number of participants - bypassing Lobby * linting * disable 'can only interact with header and footer while reconnecting' for firefox --------- Signed-off-by: Timo K <toger5@hotmail.de> Co-authored-by: Timo <16718859+toger5@users.noreply.github.com> Co-authored-by: Timo K <toger5@hotmail.de> Co-authored-by: fkwp <github-fkwp@w4ve.de>
2025-09-18 12:58:47 +02:00
focusable={!contentObscured}
aria-hidden={contentObscured}
/>
);
}
const layers = layouts[layout.type] as CallLayoutOutputs<Layout>;
const fixedGrid = (
<Grid
key="fixed"
className={styles.fixedGrid}
style={{
insetBlockStart:
headerBounds.height > 0 ? headerBounds.bottom : bounds.top,
height: gridBounds.height,
}}
model={layout}
Layout={layers.fixed}
Tile={Tile}
Fix the interactivity of buttons while reconnecting or in earpiece mode (#3486) * Fix the interactivity of buttons while reconnecting or in earpiece mode When we're in one of these modes, we need to ensure that everything above the overlay (the header and footer buttons) is interactive, while everything obscured by the overlay (the media tiles) is non-interactive and removed from the accessibility tree. It's not a very easy task to trap focus *outside* an element, so the best solution I could come up with is to set tabindex="-1" manually on all interactive elements belonging to the media tiles. * Write a Playwright test for reconnecting * fix lints Signed-off-by: Timo K <toger5@hotmail.de> * fix test Signed-off-by: Timo K <toger5@hotmail.de> * enable http2 for matrx-rtc host to allow the jwt service to talk to the SFU * remove rate limit for delayed events * more time to connect to livekit SFU * Due to a Firefox issue we set the start anchor for the tab test to the Mute microphone button * adapt to most recent Element Web version * Use the "End call" button as proofe for a started call * Currrenty disabled due to recent Element Web - not indicating the number of participants - bypassing Lobby * linting * disable 'can only interact with header and footer while reconnecting' for firefox --------- Signed-off-by: Timo K <toger5@hotmail.de> Co-authored-by: Timo <16718859+toger5@users.noreply.github.com> Co-authored-by: Timo K <toger5@hotmail.de> Co-authored-by: fkwp <github-fkwp@w4ve.de>
2025-09-18 12:58:47 +02:00
aria-hidden={contentObscured}
/>
);
const scrollingGrid = (
<Grid
key="scrolling"
className={styles.scrollingGrid}
model={layout}
Layout={layers.scrolling}
Tile={Tile}
Fix the interactivity of buttons while reconnecting or in earpiece mode (#3486) * Fix the interactivity of buttons while reconnecting or in earpiece mode When we're in one of these modes, we need to ensure that everything above the overlay (the header and footer buttons) is interactive, while everything obscured by the overlay (the media tiles) is non-interactive and removed from the accessibility tree. It's not a very easy task to trap focus *outside* an element, so the best solution I could come up with is to set tabindex="-1" manually on all interactive elements belonging to the media tiles. * Write a Playwright test for reconnecting * fix lints Signed-off-by: Timo K <toger5@hotmail.de> * fix test Signed-off-by: Timo K <toger5@hotmail.de> * enable http2 for matrx-rtc host to allow the jwt service to talk to the SFU * remove rate limit for delayed events * more time to connect to livekit SFU * Due to a Firefox issue we set the start anchor for the tab test to the Mute microphone button * adapt to most recent Element Web version * Use the "End call" button as proofe for a started call * Currrenty disabled due to recent Element Web - not indicating the number of participants - bypassing Lobby * linting * disable 'can only interact with header and footer while reconnecting' for firefox --------- Signed-off-by: Timo K <toger5@hotmail.de> Co-authored-by: Timo <16718859+toger5@users.noreply.github.com> Co-authored-by: Timo K <toger5@hotmail.de> Co-authored-by: fkwp <github-fkwp@w4ve.de>
2025-09-18 12:58:47 +02:00
aria-hidden={contentObscured}
/>
);
// The grid tiles go *under* the spotlight in the portrait layout, but
// *over* the spotlight in the expanded layout
return layout.type === "spotlight-expanded" ? (
2024-06-07 17:29:48 -04:00
<>
{fixedGrid}
{scrollingGrid}
</>
) : (
<>
{scrollingGrid}
{fixedGrid}
2024-06-07 17:29:48 -04:00
</>
);
};
const rageshakeRequestModalProps = useRageshakeRequestModal(
matrixRoom.roomId,
);
const buttons: JSX.Element[] = [];
buttons.push(
<MicButton
key="audio"
2025-08-29 18:46:24 +02:00
muted={!audioEnabled}
onClick={toggleAudio ?? undefined}
onTouchEnd={onControlsTouchEnd}
2025-08-29 18:46:24 +02:00
disabled={toggleAudio === null}
data-testid="incall_mute"
/>,
<VideoButton
key="video"
2025-08-29 18:46:24 +02:00
muted={!videoEnabled}
onClick={toggleVideo ?? undefined}
onTouchEnd={onControlsTouchEnd}
2025-08-29 18:46:24 +02:00
disabled={toggleVideo === null}
data-testid="incall_videomute"
/>,
);
2025-09-24 13:54:54 -04:00
if (vm.toggleScreenSharing !== null) {
buttons.push(
<ShareScreenButton
key="share_screen"
className={styles.shareScreen}
2025-09-24 13:54:54 -04:00
enabled={sharingScreen}
onClick={vm.toggleScreenSharing}
onTouchEnd={onControlsTouchEnd}
data-testid="incall_screenshare"
/>,
);
}
if (supportsReactions) {
buttons.push(
<ReactionToggleButton
vm={vm}
key="raise_hand"
className={styles.raiseHand}
identifier={`${client.getUserId()}:${client.getDeviceId()}`}
onTouchEnd={onControlsTouchEnd}
/>,
);
}
if (layout.type !== "pip")
buttons.push(
<SettingsButton
key="settings"
onClick={openSettings}
onTouchEnd={onControlsTouchEnd}
/>,
);
buttons.push(
<EndCallButton
key="end_call"
onClick={function (): void {
vm.hangup();
}}
onTouchEnd={onControlsTouchEnd}
data-testid="incall_leave"
/>,
);
const footer = (
<div
ref={footerRef}
className={classNames(styles.footer, {
[styles.overlay]: windowMode === "flat",
[styles.hidden]:
!showFooter || (!showControls && headerStyle === "none"),
})}
>
{headerStyle !== "none" && (
<div className={styles.logo}>
<LogoMark width={24} height={24} aria-hidden />
<LogoType
width={80}
height={11}
aria-label={import.meta.env.VITE_PRODUCT_NAME || "Element Call"}
/>
{/* Don't mind this odd placement, it's just a little debug label */}
{debugTileLayout
? `Tiles generation: ${tileStoreGeneration}`
: undefined}
</div>
)}
{showControls && <div className={styles.buttons}>{buttons}</div>}
{showControls && (
<LayoutToggle
className={styles.layout}
layout={gridMode}
setLayout={setGridMode}
onTouchEnd={onControlsTouchEnd}
/>
)}
</div>
);
return (
<div
className={styles.inRoom}
ref={containerRef}
onTouchStart={onTouchStart}
onTouchEnd={onTouchEnd}
onTouchCancel={onTouchCancel}
onPointerMove={onPointerMove}
onPointerOut={onPointerOut}
>
{header}
{audioParticipants.map(({ livekitRoom, url, participants }) => (
<LivekitRoomAudioRenderer
key={url}
url={url}
livekitRoom={livekitRoom}
validIdentities={participants.map((p) => p.identity)}
muted={muteAllAudio}
/>
))}
{renderContent()}
<CallEventAudioRenderer vm={vm} muted={muteAllAudio} />
<ReactionsAudioRenderer vm={vm} muted={muteAllAudio} />
Fix the interactivity of buttons while reconnecting or in earpiece mode (#3486) * Fix the interactivity of buttons while reconnecting or in earpiece mode When we're in one of these modes, we need to ensure that everything above the overlay (the header and footer buttons) is interactive, while everything obscured by the overlay (the media tiles) is non-interactive and removed from the accessibility tree. It's not a very easy task to trap focus *outside* an element, so the best solution I could come up with is to set tabindex="-1" manually on all interactive elements belonging to the media tiles. * Write a Playwright test for reconnecting * fix lints Signed-off-by: Timo K <toger5@hotmail.de> * fix test Signed-off-by: Timo K <toger5@hotmail.de> * enable http2 for matrx-rtc host to allow the jwt service to talk to the SFU * remove rate limit for delayed events * more time to connect to livekit SFU * Due to a Firefox issue we set the start anchor for the tab test to the Mute microphone button * adapt to most recent Element Web version * Use the "End call" button as proofe for a started call * Currrenty disabled due to recent Element Web - not indicating the number of participants - bypassing Lobby * linting * disable 'can only interact with header and footer while reconnecting' for firefox --------- Signed-off-by: Timo K <toger5@hotmail.de> Co-authored-by: Timo <16718859+toger5@users.noreply.github.com> Co-authored-by: Timo K <toger5@hotmail.de> Co-authored-by: fkwp <github-fkwp@w4ve.de>
2025-09-18 12:58:47 +02:00
{reconnectingToast}
{earpieceOverlay}
<ReactionsOverlay vm={vm} />
{waitingOverlay}
{footer}
{layout.type !== "pip" && (
<>
<RageshakeRequestModal {...rageshakeRequestModalProps} />
<SettingsModal
client={client}
roomId={matrixRoom.roomId}
open={settingsModalOpen}
onDismiss={closeSettings}
tab={settingsTab}
onTabChange={setSettingsTab}
// TODO expose correct data to setttings modal
livekitRooms={[]}
/>
</>
)}
</div>
);
};