Assist remote canvas control (#3287)

* refactor(searchStore): reformat filterMap function parameters (#3166)

- Reformat the parameters of the filterMap function for better readability.
- Comment out the fetchSessions call in clearSearch method to avoid unnecessary session fetch.

* Increment frontend chart version (#3167)

Co-authored-by: GitHub Action <action@github.com>

* refactor(chalice): cleaned code
fix(chalice): fixed session-search-pg sortKey issue
fix(chalice): fixed CH-query-formatter to handle special chars
fix(chalice): fixed /ids response

* feat(auth): implement withCaptcha HOC for consistent reCAPTCHA (#3177)

* feat(auth): implement withCaptcha HOC for consistent reCAPTCHA

This commit refactors the reCAPTCHA implementation across the application
by introducing a Higher Order Component (withCaptcha) that encapsulates
captcha verification logic. The changes:

- Create a reusable withCaptcha HOC in withRecaptcha.tsx
- Refactor Login, ResetPasswordRequest, and CreatePassword components
- Extract SSOLogin into a separate component
- Improve error handling and user feedback
- Standardize loading and verification states across forms
- Make captcha implementation more maintainable and consistent

* feat(auth): support msaas edition for enterprise features

Add msaas to the isEnterprise check alongside ee edition to properly
display enterprise features. Use userStore.isEnterprise in SSOLogin
component instead of directly checking authDetails.edition for
consistent
enterprise status detection.

* Increment frontend chart version (#3179)

Co-authored-by: GitHub Action <action@github.com>

* feat(assist): improved caching mechanism for cluster mode (#3180)

* Increment assist chart version (#3181)

Co-authored-by: GitHub Action <action@github.com>

* ui: fix table column export

* Increment frontend chart version

* fix(auth): remove unnecessary captcha token validation (#3188)

The token validation checks were redundant as the validation is already
handled by the captcha wrapper component. This change simplifies the
password reset flow while maintaining security.

* Increment frontend chart version (#3189)

Co-authored-by: GitHub Action <action@github.com>

* ui: onboarding fixes

* ui: fixes for onboarding ui

* Increment frontend chart version

* feat(helm): add TOKEN_SECRET environment variable

Add TOKEN_SECRET environment variable to HTTP service deployment and
generate a random value for it in vars.yaml.

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(GraphQL): remove unused useTranslation hook (#3200) (#3206)

Co-authored-by: PiRDub <pirddeveloppeur@gmail.com>

* Increment frontend chart version

* chore(http): remove default token_string

scripts/helmcharts/openreplay/charts/http/scripts/entrypoint.sh

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(dashboard): update filter condition in MetricsList

Change the filter type comparison from checking against 'all' to
checking against an empty string. This ensures proper filtering
behavior when filtering metrics in the dashboard component.

* Increment frontend chart version

* ui: shrink icons when no space, adjust player area for events export … (#3217)

* ui: shrink icons when no space, adjust player area for events export panel, fix panel size

* ui: rm log

* Increment frontend chart version

* refactor(chalice): changed user-journey

* Increment chalice chart version

* refactor(auth): separate SSO support from enterprise edition

Add dedicated isSSOSupported property to correctly identify when SSO
authentication is available, properly handling the 'msaas' edition
case separately from enterprise edition checks. This fixes SSO
visibility in the login interface.

* Increment frontend chart version

* UI patches (28.03) (#3231)

* ui: force getting url for location in tabmanagers

* Assist add turn servers (#3229)

* fixed conflicts

* add offers

* add config to sicket query

* add config to sicket query

* add config init

* removed console logs

* removed wrong updates

* fixed conflicts

* add offers

* add config to sicket query

* add config to sicket query

* add config init

* removed console logs

* removed wrong updates

* ui: fix chat draggable, fix default params

---------

Co-authored-by: nick-delirium <nikita@openreplay.com>

* ui: fix spritemap generation for assist sessions

* ui: fix yarnlock

* fix errors

* updated widget link

* resolved conflicts

* updated widget url

---------

Co-authored-by: Andrey Babushkin <55714097+reyand43@users.noreply.github.com>
Co-authored-by: Андрей Бабушкин <andreybabushkin2000@gmail.com>

* fix(init): remove duplicate clone

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* Increment assist chart version

* Increment frontend chart version

* ui: add old devtool filters

* ui: filter keys

* Increment frontend chart version

* ui: fix modules mapper

* ui: fix modules label

* Increment frontend chart version

* ui: fix double fetches for sessions

* Increment frontend chart version

* pulled updates (#3254)

* Increment frontend chart version (#3255)

Co-authored-by: GitHub Action <action@github.com>

* Increment assist chart version (#3256)

Co-authored-by: GitHub Action <action@github.com>

* feat(chalice): added for_spot=True for authenticate_sso (#3259)

* Increment chalice chart version (#3260)

Co-authored-by: GitHub Action <action@github.com>

* Assist patch canvas (#3265)

* add agent info to assist and tracker

* removed AGENTS_CONNECTED event

* Increment frontend chart version (#3266)

Co-authored-by: GitHub Action <action@github.com>

* Increment assist chart version (#3267)

Co-authored-by: GitHub Action <action@github.com>

* resolved conflict

* removed comments

* add global method support

* fix errors

* remove wrong updates

* remove wrong updates

* add onDrag as option

---------

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>
Co-authored-by: Shekar Siri <sshekarsiri@gmail.com>
Co-authored-by: Mehdi Osman <estradino@users.noreply.github.com>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: Taha Yassine Kraiem <tahayk2@gmail.com>
Co-authored-by: Alexander <zavorotynskiy@pm.me>
Co-authored-by: nick-delirium <nikita@openreplay.com>
Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
Co-authored-by: PiRDub <pirddeveloppeur@gmail.com>

frontend/app/player/web/assist/RemoteControl.ts

@@ -17,6 +17,9 @@ export interface State {
 
 export default class RemoteControl {
   private assistVersion = 1;
+  private isDragging = false;
+  private dragStart: any | null = null;
+  private readonly dragThreshold = 3;
 
   static readonly INITIAL_STATE: Readonly<State> = {
     remoteControl: RemoteControlStatus.Disabled,
@@ -81,6 +84,7 @@ export default class RemoteControl {
   }
 
   private onMouseMove = (e: MouseEvent): void => {
+    if (this.isDragging) return;
     const data = this.screen.getInternalCoordinates(e);
     this.emitData('move', [data.x, data.y]);
   };
@@ -154,16 +158,61 @@ export default class RemoteControl {
     this.emitData('click', [data.x, data.y]);
   };
 
+  private onMouseDown = (e: MouseEvent): void => {
+    if (this.store.get().annotating) return;
+
+    const { x, y } = this.screen.getInternalViewportCoordinates(e);
+    this.dragStart = [x, y];
+    this.isDragging = false;
+
+    const handleMove = (moveEvent: MouseEvent) => {
+      const { x: mx, y: my } =
+        this.screen.getInternalViewportCoordinates(moveEvent);
+      const [sx, sy] = this.dragStart!;
+      const dx = Math.abs(mx - sx);
+      const dy = Math.abs(my - sy);
+
+      if (
+        !this.isDragging &&
+        (dx > this.dragThreshold || dy > this.dragThreshold)
+      ) {
+        this.emitData('startDrag', [sx, sy]);
+        this.isDragging = true;
+      }
+
+      if (this.isDragging) {
+        this.emitData('drag', [mx, my, mx - sx, my - sy]);
+      }
+    };
+
+    const handleUp = () => {
+      if (this.isDragging) {
+        this.emitData('stopDrag');
+      }
+
+      this.dragStart = null;
+      this.isDragging = false;
+
+      window.removeEventListener('mousemove', handleMove);
+      window.removeEventListener('mouseup', handleUp);
+    };
+
+    window.addEventListener('mousemove', handleMove);
+    window.addEventListener('mouseup', handleUp);
+  };
+
   private toggleRemoteControl(enable: boolean) {
     if (enable) {
       this.screen.overlay.addEventListener('mousemove', this.onMouseMove);
       this.screen.overlay.addEventListener('click', this.onMouseClick);
       this.screen.overlay.addEventListener('wheel', this.onWheel);
+      this.screen.overlay.addEventListener('mousedown', this.onMouseDown);
       this.store.update({ remoteControl: RemoteControlStatus.Enabled });
     } else {
       this.screen.overlay.removeEventListener('mousemove', this.onMouseMove);
       this.screen.overlay.removeEventListener('click', this.onMouseClick);
       this.screen.overlay.removeEventListener('wheel', this.onWheel);
+      this.screen.overlay.removeEventListener('mousedown', this.onMouseDown);
       this.store.update({ remoteControl: RemoteControlStatus.Disabled });
       this.toggleAnnotation(false);
     }

scripts/helmcharts/openreplay/charts/assist/Chart.yaml

@@ -1,7 +1,6 @@
 apiVersion: v2
 name: assist
 description: A Helm chart for Kubernetes
-
 # A chart can be either an 'application' or a 'library' chart.
 #
 # Application charts are a collection of templates that can be packaged into versioned archives
@@ -11,12 +10,10 @@ description: A Helm chart for Kubernetes
 # a dependency of application charts to inject those assist and functions into the rendering
 # pipeline. Library charts do not define any templates and therefore cannot be deployed.
 type: application
-
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
 version: 0.1.1
-
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to
 # follow Semantic Versioning. They should reflect the version the application is using.

tracker/tracker-assist/src/Assist.ts

@@ -37,6 +37,7 @@ export interface Options {
   onCallDeny?: () => any;
   onRemoteControlDeny?: (agentInfo: Record<string, any>) => any;
   onRecordingDeny?: (agentInfo: Record<string, any>) => any;
+  onDragCamera?: (dx: number, dy: number) => void;
   session_calling_peer_key: string;
   session_control_peer_key: string;
   callConfirm: ConfirmOptions;
@@ -106,6 +107,7 @@ export default class Assist {
         onCallStart: () => {},
         onAgentConnect: () => {},
         onRemoteControlStart: () => {},
+        onDragCamera: () => {},
         callConfirm: {},
         controlConfirm: {}, // TODO: clear options passing/merging/overwriting
         recordingConfirm: {},
@@ -379,6 +381,15 @@ export default class Assist {
       socket.on("move", (id, event) =>
         processEvent(id, event, this.remoteControl?.move)
       );
+      socket.on("startDrag", (id, event) =>
+        processEvent(id, event, this.remoteControl?.startDrag)
+      );
+      socket.on("drag", (id, event) =>
+        processEvent(id, event, this.remoteControl?.drag)
+      );
+      socket.on("stopDrag", (id, event) =>
+        processEvent(id, event, this.remoteControl?.stopDrag)
+      );
       socket.on("focus", (id, event) =>
         processEvent(id, event, (clientID, nodeID) => {
           const el = app.nodes.getNode(nodeID);
@@ -755,6 +766,7 @@ export default class Assist {
           app.debug.error("Error requesting local stream", e);
           // if something didn't work out, we terminate the call
           initiateCallEnd();
+          this.options.onCallDeny?.();
           return;
         }
 

tracker/tracker-assist/src/Mouse.ts

@@ -1,10 +1,15 @@
+import { hasOpenreplayAttribute } from "./utils.js"
+
 type XY = [number, number]
+type XYDXDY = [number, number, number, number]
 
 
 export default class Mouse {
   private readonly mouse: HTMLDivElement
   private position: [number,number] = [0,0,]
-  constructor(private readonly agentName?: string) {
+  private isDragging = false
+
+  constructor(private readonly agentName?: string, private onDragCamera?: (dx: number, dy: number) => void) {
     this.mouse = document.createElement('div')
     const agentBubble = document.createElement('div')
     const svg ='<svg version="1.1" width="20" height="20" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px" xml:space="" viewBox="8.2 4.9 11.6 18.2"><polygon fill="#FFFFFF" points="8.2,20.9 8.2,4.9 19.8,16.5 13,16.5 12.6,16.6 "></polygon><polygon fill="#FFFFFF" points="17.3,21.6 13.7,23.1 9,12 12.7,10.5 "></polygon><rect x="12.5" y="13.6" transform="matrix(0.9221 -0.3871 0.3871 0.9221 -5.7605 6.5909)" width="2" height="8"></rect><polygon points="9.2,7.3 9.2,18.5 12.2,15.6 12.6,15.5 17.4,15.5 "></polygon></svg>'
@@ -23,6 +28,8 @@ export default class Mouse {
       whiteSpace: 'nowrap',
     })
 
+    this.onDragCamera = onDragCamera
+
     const agentNameStr = this.agentName ? this.agentName.length > 10 ? this.agentName.slice(0, 9) + '...' : this.agentName : 'Agent'
     agentBubble.innerHTML = `<span>${agentNameStr}</span>`
 
@@ -84,8 +91,63 @@ export default class Mouse {
     return null
   }
 
-  private readonly pScrEl = document.scrollingElement || document.documentElement // Is it always correct
-  private lastScrEl: Element | 'window' | null = null
+  startDrag(pos: XY) {
+    this.move(pos)
+    const el = document.elementFromPoint(pos[0], pos[1])
+    if (el) {
+      const downEvt = new MouseEvent("mousedown", {
+        bubbles: true,
+        cancelable: true,
+        clientX: pos[0],
+        clientY: pos[1],
+        buttons: 1,
+      });
+      el.dispatchEvent(downEvt);
+      this.isDragging = true;
+    }
+  }
+
+  drag(pos: XYDXDY) {
+    const [x, y, dx, dy] = pos
+    this.move([x, y]);
+
+    if (!this.isDragging) return;
+
+    const el = document.elementFromPoint(x, y);
+    if (el) {
+      const moveEvt = new MouseEvent("mousemove", {
+        bubbles: true,
+        cancelable: true,
+        clientX: x,
+        clientY: y,
+        buttons: 1,
+      });
+      el.dispatchEvent(moveEvt);
+      if (hasOpenreplayAttribute(el, 'draggable') && this.onDragCamera) {
+        this.onDragCamera(dx, dy);
+      }
+    }
+  }
+
+  stopDrag() {
+    if (!this.isDragging) return;
+    const [x, y] = this.position;
+    const el = document.elementFromPoint(x, y);
+    if (el) {
+      const upEvt = new MouseEvent("mouseup", {
+        bubbles: true,
+        cancelable: true,
+        clientX: x,
+        clientY: y,
+        buttons: 0,
+      });
+      el.dispatchEvent(upEvt);
+    }
+    this.isDragging = false;
+  }
+
+  private readonly pScrEl = document.scrollingElement || document.documentElement; // Is it always correct
+  private lastScrEl: Element | "window" | null = null;
   private readonly resetLastScrEl = () => { this.lastScrEl = null }
   private readonly handleWScroll = e => {
     if (e.target !== this.lastScrEl &&

tracker/tracker-assist/src/RemoteControl.ts

@@ -94,7 +94,7 @@ export default class RemoteControl {
     if (this.mouse) {
       this.resetMouse()
     }
-    this.mouse = new Mouse(agentName)
+    this.mouse = new Mouse(agentName, this.options.onDragCamera)
     this.mouse.mount()
     document.addEventListener('visibilitychange', () => {
       if (document.hidden) this.releaseControl(false, true)
@@ -123,6 +123,15 @@ export default class RemoteControl {
   focus = (id, el: HTMLElement) => {
     this.focused = el
   }
+  startDrag = (id, xy) => {
+    this.mouse?.startDrag(xy)
+  }
+  drag = (id, xydxdy) => {
+    this.mouse?.drag(xydxdy);
+  }
+  stopDrag = (id) => {
+    this.mouse?.stopDrag();
+  }
   input = (id, value: string) => {
     if (id !== this.agentID || !this.mouse || !this.focused) { return }
     if (this.focused instanceof HTMLTextAreaElement

tracker/tracker-assist/src/utils.ts

@@ -0,0 +1,33 @@
+export const DOCS_HOST = 'https://docs.openreplay.com'
+
+const warnedFeatures: { [key: string]: boolean } = {}
+
+export function deprecationWarn(nameOfFeature: string, useInstead: string, docsPath = '/'): void {
+    if (warnedFeatures[nameOfFeature]) {
+      return
+    }
+    console.warn(
+      `OpenReplay: ${nameOfFeature} is deprecated. ${
+        useInstead ? `Please, use ${useInstead} instead.` : ''
+      } Visit ${DOCS_HOST}${docsPath} for more information.`,
+    )
+    warnedFeatures[nameOfFeature] = true
+  }
+
+export function hasOpenreplayAttribute(e: Element, attr: string): boolean {
+    const newName = `data-openreplay-${attr}`
+    if (e.hasAttribute(newName)) {
+      // @ts-ignore
+      if (DEPRECATED_ATTRS[attr]) {
+        deprecationWarn(
+          `"${newName}" attribute`,
+          // @ts-ignore
+          `"${DEPRECATED_ATTRS[attr] as string}" attribute`,
+          '/en/sdk/sanitize-data',
+        )
+      }
+      return true
+    }
+  
+    return false
+  }