Commit graph

93 commits

Author SHA1 Message Date
Alexander
2bf9298836
Mobile renaming (protocol + backend) (#2078)
* feat(backend): renamed ios to mobile

* feat(backend): added missing changes to iterator

* feat(backend): added missing changes to ender

* feat(backend): as usual auto deleted import in ch connector

* feat(backend): updated ee/conditions func calls
2024-04-12 09:30:24 +02:00
Alexander
daddedebcc feat(backend): fixed an issue with no mobile replays for android tracker 2024-04-05 17:00:45 +02:00
Alexander
36b6dfb5bd feat(backend): fixed no sessID in storage service 2024-03-14 17:17:50 +01:00
Alexander
ee75b2795a feat(backend): fixed broken logs in failover module 2024-03-14 16:40:14 +01:00
Alexander
45c956c489
Json logs format (#1952)
* feat(backend): try a new approach for logs formatting (http)

* feat(backend): added logger module

* feat(backend): added project/session info to /i endpoint

* feat(backend): found a solution for correct caller information

* feat(backend): finished logs for http handlers

* feat(backend): finished logs for mobile http handlers

* feat(backend): finished ender

* feat(backend): finished assets

* feat(backend): finished heuristics

* feat(backend): finished image-storage

* feat(backend): finished sink

* feat(backend): finished storage

* feat(backend): formatted logs in all services

* feat(backend): finished foss part

* feat(backend): added missed foss part

* feat(backend): fixed panic in memory manager and sink service

* feat(backend): connectors
2024-03-14 12:51:14 +01:00
Alexander
89752b2acc
Canvas archives (#1938)
* feat(api): added second pre-signed urls for canvases

* feat(api): fixed old key format

* feat(backend): draft version for new approach for canvases

* feat(deploy): fixed some issues

* feat(backend): debug logs for bash command

* feat(backend): removed wrong parameter from bash pipeline

* feat(backend): removed canvas-maker service + small refactoring
2024-03-11 17:51:49 +01:00
Alexander
7c58ece936 feat(backend): added separate config for canvas-handler 2024-02-28 13:21:19 +01:00
Alexander
a073ce498d
No ffmpeg solution (#1905)
* feat(video-storage): added zstd library to the machine

* feat(video-storage): added new method to pack screenshots into compressed tar arch

* feat(video-storage): try to split command into 2

* feat(video-storage): try a new approad to avoid no file error

* feat(api): added support to a new pre-signed url for screenshots archive

* feat(video-storage): fixed an issue in extension check

* feat(video-storage): correct file name

* feat(backend): removed video-storage and splitted logic

* feat(backend): removed video-storage from helm charts

* feat(backend): split canvas and screenshot handlers

* feat(canvas): clean up canvas-handler

* feat(api): changed mobile replay url (screenshots instead of video)

* feat(backend): removed msg.SessID() call

* feat(backend): clean up code in imagestorage main
2024-02-26 14:16:43 +01:00
Alexander
4bc7c2294d feat(backend): fixed zero duration issue in mobile sessions 2024-01-25 16:11:52 +01:00
Alexander
255f879476
Mobile replay (#1853)
* feat(backend): added new trigger topic for mobile video replay maker

* feat(backend): mobile video maker with mix list support

* feat(backend): fixed panic

* feat(backend): removed commented code
2024-01-25 14:30:40 +01:00
Alexander
acc21fe82d feat(backend): fixed typo in canvas group name 2024-01-23 11:37:47 +01:00
Alexander
78a9146144 feat(backend): added new consumer groups for new canvas pipeline 2024-01-22 17:03:15 +01:00
Alexander
68a87b2743
Video replay pipeline split (#1848)
* feat(backend): split

* feat(docker): added ffmpeg to canvas-maker service
2024-01-22 09:25:58 +01:00
Alexander
622b189d39
Tag and watch backend support (#1838)
* feat(ui/tracker): start tag n watch

* fix(tracker): test coverage, fix some watcher api

* fix(tracker): add intersectionobserver, adjust tests

* feat(tracker): relay + apollo plugins

* feat(ui): tags search

* feat(ui): tags name edit

* feat(ui): tags search icon

* feat(ui): icons for tabs in player

* feat(ui): save and find button

* feat(tracker): save tags in session storage (just in case)

* feat(ui): improve loading

* feat(ui): fix icon names gen

* feat(ui): fix typo

* feat(backend): added TagAndWatch backend support

* feat(backend): use msgIndex as a seqIndex

* feat(backend): try to use truncSqlIdx for seq_index

* feat(backend): added tags cache and tags filter to DB service

* feat(backend): removed debug logs

---------

Co-authored-by: nick-delirium <nikita@openreplay.com>
2024-01-19 15:21:10 +01:00
Alexander
b4b6ceb025
Canvas refactoring (#1836)
* feat(backend): added new topic for canvasToVideo communication, moved some logic

* feat(backend): enabled canvas recording

* feat(backend): fixed canvas service main logic

* feat(backend): fixed sessionEnd detector

* feat(backend): send canvas mix lists instead of just a sessID

* feat(backend): enabled canvas recording

* feat(backend): removed old logs from video-storage

* feat(backend): default low setting for canvas recording
2024-01-17 14:39:12 +01:00
Alexander
f8a8dfa459
[Integrations] small refactoring (#1752)
* feat(backend): small refactoring in integrations

* feat(backend): ignore context timeout error
2023-12-14 11:28:05 +01:00
Alexander
552c7e77e1 feat(backend): removed debug logs from videostorage 2023-12-06 12:22:29 +01:00
ⵄⵎⵉⵔⵓⵛ
6ae3ee1927
Canvas support (#1711)
* feat(http): added new parameters to start response and new endpoint for canvas screenshorts

* fix(http): added new topic to dockerfile

* feat(http): try different multipart parser

* feat(image-storage): reused the same workflow for canvas topic handler

* feat(video-storage): new canvas parser and ffmpeg script

* feat(video-storage): use correct replay name for canvas

* feat(backend): added new message (CanvasNode)

* feat(backend): add canvas record events to db

* feat(backend): implemented missing new method for azure

---------

Co-authored-by: Alexander <zavorotynskiy@pm.me>
2023-11-27 16:22:03 +01:00
Alexander
dd8050aaa2 feat(backend): fixed possible panic in ender 2023-10-30 17:40:41 +01:00
Delirium
35461acaf3
[WIP] Mobile replayer (#1452)
* fix(ui): fix up mobile recordings display

* fix(ui): some messages

* fix(ui): some messages

* fix(player): fix msg generation for ios messages

* feat(player): add generic mmanager interface for ios player impl

* feat(player): mobile player and message manager; touch manager; videoplayer

* feat(player): add iphone shells, add log panel,

* feat(player): detect ios sessions and inject correct player

* feat(player): move screen mapper to utils

* feat(player): events panel for mobile, map shell sizes to device type data,

* feat(player): added network tab to mobile player; unify network message (ios and web)

* feat(player): resize canvas up to phone screen size, fix capitalize util

* feat(player): some general changes to support mobile events and network entries

* feat(player): remove swipes from timeline

* feat(player): more stuff for list walker

* fix(ui): performance tab, mobile project typings and form

* fix(ui):some ui touches for ios replayer shell

* fix(ui): more fixes for ui, new onboarding screen for mobile projects

* feat(ui): mobile overview panel (xray)

* feat(ui): fixes for phone shell and tap events

* fix(tracker): change phone shells and sizes

* fix(tracker): fix border on replay screen

* feat(ui): use crashes from db to show in session

* feat(ui): use event name for xray

* feat(ui): some overall ui fixes

* feat(ui): IOS -> iOS

* feat(ui): change tags to ant d

* fix(ui): fast fix

* fix(ui): fix for capitalizer

* fix(ui): fix for browser display

* fix(ui): fix for note popup

* fix(ui): change exceptions display

* fix(ui): add click rage to ios xray

* fix(ui): some icons and resizing

* fix(ui): fix ios context menu overlay, fix console logs creation for ios

* feat(ui): added icons

* feat(ui): performance warnings

* feat(ui): performance warnings

* feat(ui): different styles

* feat(ui): rm debug true

* feat(ui): fix warnings display

* feat(ui): some styles for animation

* feat(ui): add some animations to warnings

* feat(ui): move perf warnings to performance graph

* feat(ui): hide/show warns dynamically

* feat(ui): new mobile touch animation

* fix(tracker): update msg for ios

* fix(ui): taprage fixes

* fix(ui): regenerate icons and messages

* fix(ui): fix msgs

* fix(backend): fix events gen

* fix(backend): fix userid msg
2023-10-27 12:12:09 +02:00
Alexander Zavorotynskiy
d06192907c fix(backend): removed unused message types from iterator filter 2023-10-09 15:05:14 +02:00
Alexander
d22e4e138e
Backend for mobile tracker support (#1456)
* feat(backend): handlers for mobile messages

* feat(backend): new service template

* feat(backend): save mobile session start and send batches to kafka

* feat(backend): added mobile support to sink, ender and storage

* helm(videostorage): added helm chart for a new service videostorage

* fix(backend): added pointer to streing for userBrowser (because it's null for mobile sessions)

* feat(backend): added MsgIOSBatchMeta handler to message iterator's logic

* feat(backend): added ios ts parser to ender

* feat(backend): enabled producing batch of messages to queue

* feat(backend): removed iosstart from mob files

* feat(backend): added new ios message types

* feat(backend): added iosStart and iosEnd

* fix(backend): fixed log issue

* feat(backend): send tar.gz archives to special queue topic

* feat(backend): read raw archives from kafka

* fix(backend): added missing file

* fix(backend): removed the second file reading

* fix(backend): fixed wrong queue topic name

* fix(backend): fixed mobile trigger topic name

* feat(backend): added tar.gz extractor and iOSSessionEnd handler

* feat(backend): debug logs on message uploading

* fix(backend): added raw-images topic consumption

* feat(backend): now sink send iosSessionEnd to video-storage

* feat(backend): added dir creator for new sessions

* feat(backend): added correct command to execute

* feat(backend): added overwrite option

* feat(backend): added s3 uploader for video session replays

* feat(backend): new consumer group for mobile sessions

* feat(backend): debug logs for uploader

* feat(backend): removed unused log

* feat(backend): fixed s3 key for video replays

* feat(backend): removed debug code

* feat(backend): fixed video-storage message filter

* fix(backend): added mobileSessionEnd to SessionEnd converter

* feat(backend): added first version if db support for mobile events

* fix(backend): added swipe events to mob file

* feat(backend): added swipe event to pg

* feat(backend): split logic into 2 services: image-storage and video-storage

* feat(backend): added helm chart for image-storage service

* fix(backend): fixed table name for mobile taps

* feat(backend): added metadata handler for mobile message parser + fix message filters

* feat(backend): added iosRawTopic to DB message consumer

* fix(backend): removed value from mobile inputs

* feat(backend): removed debug log from iterator

* feat(backend): added new apple devices to iOS device parser

* fix(backend): added real projectID instead of 0

* feat(backend): extended a list of simulators for device detector

* feat(backend): updated networkCall mobile message

* fix(backend): added new way to define is network call successed or not

* feat(backend): added timezone support for mobile start request

* feat(backend): added 2 mobile events Input and Click to mob file

* feat(backend): refactored image storage

* feat(backend): video storage with 2 workers

* feat(backend): added project's platform support

* feat(backend): added memory size field for mobile start request

* feat(backend): changed video preset for ultrafast

* feat(backend): added debug log to http /late handler

* feat(backend): added debug log to db service for iosCrash messages

* feat(backend): added tapRage event handler to heuristics

* feat(backend): changed table and field names for ios crashes

* feat(backend): added payload for tapRage events

* feat(backend): added TapRage events insert to DB

* feat(backend): added fps value to /mobile/start response

* feat(backend): added image quality parameter to /mobile/start response

* feat(backend): added ScreenLeave handler

* feat(backend): removed screenEnter and screenLeave events, added new viewComponent event

---------

Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-10-09 15:02:20 +02:00
Alexander
bb7c086c5b
Performance patch (#1508)
* feat(backend): removed legacy InputEvent builder from heuristics service

* feat(backend): removed InputEvent handler from DB service

* feat(backend): removed InputEvent from ClickHouse connector

* feat(backend): added extra env variable for brotli compression algorithm
2023-10-09 13:54:53 +02:00
Alexander Zavorotynskiy
9892e63a51 feat(backend/ender): comsuming 2023-09-11 11:30:21 +02:00
Alexander Zavorotynskiy
f4883f481c feat(backend/ender): no comsuming 2023-09-11 11:29:22 +02:00
Alexander Zavorotynskiy
7bffd2f321 feat(backend): added app crash detector to heuristics service 2023-09-05 16:57:34 +02:00
Alexander
676041d90b
Ender logic update (#1435)
* feat(backend): new session end detection logic + several fixes

* feat(backend): support partitions managment in ender

* feat(backend): added new consumer support to redis

* feat(backend): added support for new consumer in kafka

* feat(backend): added new consumer support to redis (ee)

* feat(backend): small refactoring in ender
2023-08-03 14:57:15 +04:00
Alexander Zavorotynskiy
900a25993f fix(backend): removed mini ddos of sessions with negative duration 2023-07-31 15:48:25 +04:00
Alexander
4b8f3bee25
Sessions refactoring (#1371)
* feat(backend): moved sql requests related to sessions table to one place

* feat(backend): refactoring in db.Saver handler

* feat(backend): hude refactoring in db/postgres module

* fix(backend): workable feature flags

* fix(backend): workable integrations

* fix(backend): workable sessions and projects modules

* fix(backend): added missed projects module to sessions

* feat(backend): renaming

* feat(backend): moved session struct to sessions module and split methods into interface, cache and storage levels

* feat(backend): moved project struct to projects module

* feat(backend): added projects model

* feat(backend): implemented new in memory cache for sessions and projects

* feat(backend): implemented new cache in projects

* feat(backend): there are 2 methods in cache module now: Get() and GetAndRefresh()

* feat(backend): added cache update operations

* fix(backend): fixed import cycle

* fix(backend): fixed panic in db message handler

* fix(backend): fixed panic in projects module

* fix(backend): fixed panic in sessions.GetDuration

* feat(backend): added direct call to get session duration if session is already in cache

* feat(backend): used pg pool everywhere except db service

* fix(backend): added missing part after rebase

* fix(backend): removed old sessions file

* feat(backend): added refactored redis client with produce/consume options

* feat(backend): added cache layer for projects

* fix(backend): added missing redis config

* fix(backend): added missing method for producer

* feat(backend): cache integration for sessions

* feat(backend): temporary method to get session directly from db

* feat(backend): adapt EE version of message handler

* fix(backend): fixed issue in fts realisation

* fix(backend): added redis cache to sessions module

* fix(backend): set 0 duration or hesitation time for inputs without focus event

* feat(backend): added cache for session updates and failover mechanism for batch.Insert() operation

* feat(backend): debug log

* feat(backend): more debug log

* feat(backend): removed debug log

* fix(backend): fixed an issue of tracking input events with empty label

* fix(backend): disabled debug log in projects cache

* fix(backend): renamed session updater

* fix(backend): fixed closed pool issue in DB service

* fix(backend): fixed dead lock in db Stop() method

* fix(backend): fixed panic in heuristics service

* feat(backend): enabled redis cache in projects

* feat(backend): clear cache on each update operation

* feat(backend): fully integrated cache layer with auto switch

* feat(backend): small refactoring in session updates

* fix(backend): fixed wrong events counter issue

* feat(backend): enabled full cache support in ender and http services

* fix(backend/ee): added missed import

* feat(backend): added second cache layer for db to speed up the service

* feat(backend): disable redis cache

* feat(backend): moved redis cache to ee
2023-07-06 10:55:43 +02:00
Alexander
8e58c367dd
Fixed issue with wrong expiration timeout for projects (#1300)
* fix(backend): fixed issue with wrong expiration timeout for projects in cache layer

* fix(backend): removed direct call db.GetProject()

* feat(backend): set default PROJECT_EXPIRATION to 10 minutes
2023-06-08 18:08:53 +02:00
Alexander
111d6a1b54
Azure support (#1270)
* feat(backend): implemented azure blob storage support

* feat(backend): added azure implementation to backend services

* feat(backend): added azure blob storage support to chalice service

* fix(backend): removed prev version of s3

* feat(backend): moved azure support to ee subfolder

* feat(backend): prepared ee code for new utils.objects package

* feat(backend): added missed modules to go.mod

* feat(backend): added missed modules to go.sum

* feat(backend): go mod tidy

* feat(backend): temporary made s3 vars are not required

* feat(backend): added azure lib to ee chalice

* feat(api): changed azure env var name

* feat(api): added new object store and extra methods to chalice ee

* fix(api): added azure blob lib to alerts

* fix(api): fixed incorrect call in sessions_devtool

* fix(crons): added azure blob storage library to requirements list

* chore(build): Error message for not providing flag.

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(backend): removed ios headers and object store for ios messages

* feat(backend): object config refactoring

* chore(helm): Update BUCKET_NAME

* fix(backend): removed object storage usage in http

* feat(backend): added debug logs to azure upload method

* fix(backend): fixed empty folder issue

* fix(backend): removed extra debug log

* chore(helm): Adding global variables for crons

* chore(helm): Remove clickhouse resource limit

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(backend): removed assets debug log

* feat(api): use ABC class instead of empty interface

* feat(api): renamed helpers to generators

* feat(api): changed prep/clean dev scripts

* feat(api): changed name obj_store -> StorageClient

* feat(api): some changes after code review

* fix(api): removed unnecesery packages in oss api

* feat(backend): moved azure implementation to ee folder

---------

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>
Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-06-08 13:31:54 +02:00
Delirium
2ed4bba33e
feat(tracker/ui): support for multi tab sessions (#1236)
* feat(tracker): add support for multi tab sessions

* feat(backend): added support of multitabs

* fix(backend): added support of deprecated batch meta message to pre-decoder

* fix(backend): fixed nil meta issue for TabData messages in sink

* feat(player): add tabmanager

* feat(player): basic tabchange event support

* feat(player): pick tabstate for console panel and timeline

* fix(player): only display tabs that are created

* feat(player): connect performance, xray and events to tab state

* feat(player): merge all tabs data for overview

* feat(backend/tracker): extract tabdata into separate message from batchmeta

* fix(tracker): fix new session check

* fix(backend): remove batchmetadeprecated

* fix(backend): fix switch case

* fix(player): fix for tab message size

* feat(tracker): check for active tabs with broadcast channel

* feat(tracker): prevent multiple messages

* fix(tracker): ignore beacons from same tab, only ask if token isnt present yet, add small delay before start to wait for answer

* feat(player): support new msg struct in assist player

* fix(player): fix some livepl components for multi tab states

* feat(tracker): add option to disable multitab

* feat(tracker): add multitab to assist plugin

* feat(player): back compat for tab id

* fix(ui): fix missing list in controls

* fix(ui): optional list update

* feat(ui): fix visuals for multitab; use window focus event for tabs

* fix(tracker): fix for dying tests (added tabid to writer, refactored other tests)

* feat(ui): update LivePlayerSubHeader.tsx to support tabs

* feat(backend): added tabs support to devtools mob files

* feat(ui): connect state to current tab properly

* feat(backend): added multitab support to assits

* feat(backend): removed data check in agent message

* feat(backend): debug on

* fix(backend): fixed typo in message broadcast

* feat(backend): fixed issue in connect method

* fix(assist): fixed typo

* feat(assist): added more debug logs

* feat(assist): removed one log

* feat(assist): more logs

* feat(assist): use query.peerId

* feat(assist): more logs

* feat(assist): fixed session update

* fix(assist): fixed getSessions

* fix(assist): fixed request_control broadcast

* fix(assist): fixed typo

* fix(assist): added missed line

* fix(assist): fix typo

* feat(tracker): multitab support for assist sessions

* fix(tracker): fix dead tests (tabid prop)

* fix(tracker): fix yaml

* fix(tracker): timers issue

* fix(ui): fix ui E2E tests with magic?

* feat(assist): multitabs support for ee version

* fix(assist): added missed method import

* fix(tracker): fix fix events in assist

* feat(assist): added back compatibility for sessions without tabId

* fix(assist): apply message's top layer structure before broadcast call

* fix(assist): added random tabID for prev version

* fix(assist): added random tabID for prev version (ee)

* feat(assist): added debug logs

* fix(assist): fix typo in sessions_agents_count method

* fix(assist): fixed more typos in copy-pastes

* fix(tracker): fix restart timings

* feat(backend): added tabIDs for some events

* feat(ui): add tab change event to the user steps bar

* Revert "feat(backend): added tabIDs for some events"

This reverts commit 1467ad7f9f.

* feat(ui): revert timeline and xray to grab events from all tabs

* fix(ui): fix typo

---------

Co-authored-by: Alexander Zavorotynskiy <zavorotynskiy@pm.me>
2023-06-07 10:40:32 +02:00
Alexander
742c038ecd
New env variable CLOUD (aws by default) (#1232)
* feat(backend): added new env variable CLOUD (aws by default)

* chore(backend): Adding env variable for CLOUD

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

---------

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>
Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-05-04 16:48:48 +02:00
Alexander
d74ed000fa
Enable session encryption (#1121)
* feat(backend): enable session encryption

* feat(backend): fixed updated method name in failover algo

* feat(backend): disable encryption by default
2023-04-11 16:44:26 +02:00
Alexander
f561593b1a
Memory control manager (#1067)
* feat(backend): added a mechanism to control memory consumption

* feat(backend): extra log for system allocation

* feat(backend): implemented new memory manager in db and heuristics service
2023-04-11 16:01:37 +02:00
Alexander Zavorotynskiy
181756ae59 feat(backend): added unbindNodes event to db filter 2023-03-17 12:17:14 +01:00
nick-delirium
fcf4d1bc7e Merge branch 'dev' into new-frustrations 2023-03-16 17:27:19 +01:00
Alexander Zavorotynskiy
9e59d5e1ab feat(backend): added heuristics metric 2023-03-14 11:12:46 +01:00
Alexander Zavorotynskiy
3d7ab8b31d feat(backend): added new message types to db message filter 2023-03-10 12:19:32 +01:00
Alexander
c6aac11cbf
Heuristics refactoring (#987)
* feat(backend): refactored heuristics service

* feat(backend): refactored db service (moved several events to heuristics)
2023-03-09 09:54:12 +01:00
Alexander
fdd28dbc4a
Draft: New metrics module (#982)
* feat(backend): created new metrics module
2023-02-20 16:37:55 +01:00
Alexander
a40ce7c376
[DB] async db inserts for PG and CH (#967)
* feat(backend): added async workers for db service
2023-02-13 13:58:03 +01:00
Kraiem Taha Yassine
767d2f6f0c
V1.8.2 workers app name (#772)
* feat(backend): added application name to postgres connection

Co-authored-by: Alexander Zavorotynskiy <zavorotynskiy@pm.me>
2023-01-23 09:08:14 +01:00
Alexander
bfe14db353
[DB] added last quickwit changes (#950)
* feat(backend): added last quickwit changes
2023-01-20 16:18:00 +01:00
Alexander
43b042aef8
[Sink] new message iterator (#945)
* feat(backend): implemented new version of sink message iterator
2023-01-20 09:49:49 +01:00
Alexander
861302bba7
[Ender] new message iterator (#929)
* feat(backend): added new message iterator especially for ender
2023-01-18 14:43:24 +01:00
Alexander Zavorotynskiy
efe1ae1dd7 feat(backend): removed old queue logger 2022-12-20 13:51:53 +01:00
Alexander
09adccf4be
feat(backend): added env variable for enabling pprof (#880) 2022-12-13 17:48:00 +01:00
Alexander Zavorotynskiy
19b6c896b9 feat(backend): fixed big file log in storage 2022-12-13 17:28:52 +01:00
Alex K
b3cbf74deb
Merge branch 'dev' into network-request-messahe 2022-12-13 17:12:15 +01:00