Commit graph

281 commits

Author SHA1 Message Date
Alexander
f7744a0c2c
Mob split (#2015)
* feat(backend): try to split mob files by ts

* feat(backend): removed debug code and used worker pool
2024-04-03 17:10:44 +02:00
Alexander
2ac3d38078 feat(backend): support s3 connection without creds 2024-03-28 12:30:00 +01:00
Alexander
d2cabcdb54 feat(backend): moved file tagging feature to EE 2024-03-22 14:10:13 +01:00
Alexander
36b6dfb5bd feat(backend): fixed no sessID in storage service 2024-03-14 17:17:50 +01:00
Alexander
51a6d87e68 feat(backend): fixed failover mock 2024-03-14 17:06:43 +01:00
Alexander
45c956c489
Json logs format (#1952)
* feat(backend): try a new approach for logs formatting (http)

* feat(backend): added logger module

* feat(backend): added project/session info to /i endpoint

* feat(backend): found a solution for correct caller information

* feat(backend): finished logs for http handlers

* feat(backend): finished logs for mobile http handlers

* feat(backend): finished ender

* feat(backend): finished assets

* feat(backend): finished heuristics

* feat(backend): finished image-storage

* feat(backend): finished sink

* feat(backend): finished storage

* feat(backend): formatted logs in all services

* feat(backend): finished foss part

* feat(backend): added missed foss part

* feat(backend): fixed panic in memory manager and sink service

* feat(backend): connectors
2024-03-14 12:51:14 +01:00
Alexander
36fb1a07f7
Worker pool for image processing services (#1912)
* feat(backend): added worker pool for screenshot handler

* feat(backend): added worker pool to canvas handler

* feat(backend): added worker pool to canvas maker

* feat(backend): added missing method to canvas-maker service
2024-02-27 13:48:27 +01:00
Alexander
88ea925263 feat(backend): added new flag USE_S3_TAGS to disable s3 tags 2024-01-29 16:50:08 +01:00
Eng Zer Jun
76f0f3ff3d
refactor(backend): move from io/ioutil to io and os packages (#1784)
The io/ioutil package has been deprecated as of Go 1.16 [1]. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.

[1]: https://golang.org/doc/go1.16#ioutil
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2024-01-29 16:37:59 +01:00
Alexander
16af8ee09f feat(backend): fixed nil tags response 2024-01-26 15:25:40 +01:00
Alexander
df2ee71bbb feat(backend): fixed issue in start request 2024-01-26 14:25:44 +01:00
Alexander
354f071fb0
Conditions (ee version) (#1858)
* feat(backend): moved conditions to ee folder

* feat(backend): fixed typo
2024-01-26 14:03:29 +01:00
Alexander
5674f5b9b7 feat(backend): fixed getConditions request 2024-01-22 16:33:44 +01:00
Alexander
b6fe561242 feat(backend): send empty list instead of null (conditions) 2024-01-22 15:12:58 +01:00
Alexander
9e3548e9ce
Conditional recording (#1849)
* feat(backend): added mocked endpoint for conditional recordings

* feat(backend): disabled project check

* feat(backend): adapt getConditions endpoint for API implementation

* feat(backend): added condition option to start request

* feat(backend): added missing file

* feat(backend): debug log
2024-01-22 14:52:02 +01:00
Alexander
622b189d39
Tag and watch backend support (#1838)
* feat(ui/tracker): start tag n watch

* fix(tracker): test coverage, fix some watcher api

* fix(tracker): add intersectionobserver, adjust tests

* feat(tracker): relay + apollo plugins

* feat(ui): tags search

* feat(ui): tags name edit

* feat(ui): tags search icon

* feat(ui): icons for tabs in player

* feat(ui): save and find button

* feat(tracker): save tags in session storage (just in case)

* feat(ui): improve loading

* feat(ui): fix icon names gen

* feat(ui): fix typo

* feat(backend): added TagAndWatch backend support

* feat(backend): use msgIndex as a seqIndex

* feat(backend): try to use truncSqlIdx for seq_index

* feat(backend): added tags cache and tags filter to DB service

* feat(backend): removed debug logs

---------

Co-authored-by: nick-delirium <nikita@openreplay.com>
2024-01-19 15:21:10 +01:00
Delirium
309a9fd970
feat: tag and watch (UI/Tracker) (#1822)
* feat(ui/tracker): start tag n watch

* fix(tracker): test coverage, fix some watcher api

* fix(tracker): add intersectionobserver, adjust tests

* feat(tracker): relay + apollo plugins

* feat(ui): tags search

* feat(ui): tags name edit

* feat(ui): tags search icon

* feat(ui): icons for tabs in player

* feat(ui): save and find button

* feat(tracker): save tags in session storage (just in case)

* feat(ui): improve loading

* feat(ui): fix icon names gen

* feat(ui): fix typo
2024-01-19 11:11:27 +01:00
Alexander
0dea805366 feat(backend): added 2 more events to deadClick handler 2024-01-16 08:29:46 +01:00
Delirium
9e1add4ad9
feat(tracker/ui): add websocket support (#1733)
* feat(tracker/ui): add websocket support

* feat(tracker): expose ws tracker method

* fix(ui): add docs, fix types for ws methods

* fix(ui): some style fixes, rename field in mob

* fix(ui): change ws modal

* fix(ui): change ws modal

* fix(ui): rm mock data
2024-01-03 15:54:18 +01:00
Alexander
f8a8dfa459
[Integrations] small refactoring (#1752)
* feat(backend): small refactoring in integrations

* feat(backend): ignore context timeout error
2023-12-14 11:28:05 +01:00
Alexander
1c6290a312
Small http fixes (#1767)
* fix(backend): removed issue with beta versions of tracker

* feat(backend): added filter for getUxTest request
2023-12-11 15:19:25 +01:00
Alexander
68ead7292e feat(backend): removed ua insert from session module 2023-12-07 11:12:00 +01:00
Alexander
746222d729 feat(backend): insert null comment to ux_test_signal is string is empty 2023-12-04 17:28:03 +01:00
Alexander
a68351d509
[Backend] Zstd support for mob files (#1732)
* feat(backend): added zstd compression support to storage service

* feat(backend): try to ignore content-encoding for zstd compression
2023-12-04 16:00:59 +01:00
Alexander
d2eda3a54e feat(backend/uxtesting): insert NULL when duration <= 0 2023-12-01 17:38:11 +01:00
Alexander
ea782e76a7 feat(backend): added projectID check for getUXTest endpoint (add missing file) 2023-12-01 14:36:22 +01:00
Alexander
d49a377110 feat(backend): returned ua to sessions table because of sql constraints 2023-11-28 12:31:20 +01:00
Alexander
6379f053ec feat(backend): removed unused UA field from session insert 2023-11-28 11:49:08 +01:00
Alexander
636df78a20 feat(backend): removed duplicate code in sink message iterator 2023-11-28 11:23:36 +01:00
ⵄⵎⵉⵔⵓⵛ
6ae3ee1927
Canvas support (#1711)
* feat(http): added new parameters to start response and new endpoint for canvas screenshorts

* fix(http): added new topic to dockerfile

* feat(http): try different multipart parser

* feat(image-storage): reused the same workflow for canvas topic handler

* feat(video-storage): new canvas parser and ffmpeg script

* feat(video-storage): use correct replay name for canvas

* feat(backend): added new message (CanvasNode)

* feat(backend): add canvas record events to db

* feat(backend): implemented missing new method for azure

---------

Co-authored-by: Alexander <zavorotynskiy@pm.me>
2023-11-27 16:22:03 +01:00
ⵄⵎⵉⵔⵓⵛ
f321fccc11
Ux testing backend (#1709)
* feat(backend): added ux-testing support

* feat(backend): added ux-testing module

* feat(http): added bucket name for http service

* feat(backend): fixed small typos in http router

---------

Co-authored-by: Alexander <zavorotynskiy@pm.me>
2023-11-27 15:58:36 +01:00
Alexander
5040bf9480 feat(backend): removed userAgent from sessions and unstarted-sessions tables 2023-11-24 10:25:17 +01:00
Delirium
07046cc2fb
feat: canvas support [assist] (#1641)
* feat(tracker/ui): start canvas support

* feat(tracker): slpeer -> peerjs for canvas streams

* fix(ui): fix agent canvas peer id

* fix(ui): fix agent canvas peer id

* fix(ui): fix peer removal

* feat(tracker): canvas recorder

* feat(tracker): canvas recorder

* feat(tracker): canvas recorder

* feat(tracker): canvas recorder

* feat(ui): canvas support for ui

* fix(tracker): fix falling tests

* feat(ui): replay canvas in video

* feat(ui): refactor video streaming to draw on canvas

* feat(ui): 10hz check for canvas replay

* feat(ui): fix for tests

* feat(ui): fix for tests

* feat(ui): fix for tests

* feat(ui): fix for tests cov

* feat(ui): mroe test coverage

* fix(ui): styling

* fix(tracker): support backend settings for canvas
2023-11-21 11:22:54 +01:00
Alexander
bd434ae7a9 fix(backend): added missing Timezone to getSession method 2023-11-08 10:00:39 +01:00
Alexander
a93848a40a fix(backend): removed debug log from heuristics 2023-11-06 11:12:17 +01:00
Alexander
f569ed3a54 feat(backend): downgrade pg library version to fix feature flags 500 error 2023-10-30 14:00:46 +01:00
Delirium
35461acaf3
[WIP] Mobile replayer (#1452)
* fix(ui): fix up mobile recordings display

* fix(ui): some messages

* fix(ui): some messages

* fix(player): fix msg generation for ios messages

* feat(player): add generic mmanager interface for ios player impl

* feat(player): mobile player and message manager; touch manager; videoplayer

* feat(player): add iphone shells, add log panel,

* feat(player): detect ios sessions and inject correct player

* feat(player): move screen mapper to utils

* feat(player): events panel for mobile, map shell sizes to device type data,

* feat(player): added network tab to mobile player; unify network message (ios and web)

* feat(player): resize canvas up to phone screen size, fix capitalize util

* feat(player): some general changes to support mobile events and network entries

* feat(player): remove swipes from timeline

* feat(player): more stuff for list walker

* fix(ui): performance tab, mobile project typings and form

* fix(ui):some ui touches for ios replayer shell

* fix(ui): more fixes for ui, new onboarding screen for mobile projects

* feat(ui): mobile overview panel (xray)

* feat(ui): fixes for phone shell and tap events

* fix(tracker): change phone shells and sizes

* fix(tracker): fix border on replay screen

* feat(ui): use crashes from db to show in session

* feat(ui): use event name for xray

* feat(ui): some overall ui fixes

* feat(ui): IOS -> iOS

* feat(ui): change tags to ant d

* fix(ui): fast fix

* fix(ui): fix for capitalizer

* fix(ui): fix for browser display

* fix(ui): fix for note popup

* fix(ui): change exceptions display

* fix(ui): add click rage to ios xray

* fix(ui): some icons and resizing

* fix(ui): fix ios context menu overlay, fix console logs creation for ios

* feat(ui): added icons

* feat(ui): performance warnings

* feat(ui): performance warnings

* feat(ui): different styles

* feat(ui): rm debug true

* feat(ui): fix warnings display

* feat(ui): some styles for animation

* feat(ui): add some animations to warnings

* feat(ui): move perf warnings to performance graph

* feat(ui): hide/show warns dynamically

* feat(ui): new mobile touch animation

* fix(tracker): update msg for ios

* fix(ui): taprage fixes

* fix(ui): regenerate icons and messages

* fix(ui): fix msgs

* fix(backend): fix events gen

* fix(backend): fix userid msg
2023-10-27 12:12:09 +02:00
Alexander
3a2ed49385 feat(backend): removed unused ObjectStorage method from interface definition 2023-10-26 14:37:23 +02:00
Delirium
c7e5145282
feat(tracker): add beacon proxy and request body size (#1389)
* feat(tracker): add beacon proxy and body size

* feat(ui): remove unused components

* feat(ui): generate new messages, add body size to resource parser

* feat(ui): fix tooltip text, fix size detection (ts safe)

* feat(ui): cover resource with tests

* feat(ui): enable test coverage for player, utils and mstore

* fix(tracker): adjust test to support new message

* fix(tracker): fix tracker version for back compat

* feat(backend): added new column to network requests

* fix(tracker): fix unit tests

* fix(backend): fix msg gen

* fix(tracker): ci fun

* fix(tracker): changelog

* fix(tracker): fix some test

---------

Co-authored-by: Alexander Zavorotynskiy <zavorotynskiy@pm.me>
2023-10-17 12:25:20 +02:00
Alexander Zavorotynskiy
cdfa53ba6e feat(backend): added empty template for new mobile event 2023-10-09 17:19:53 +02:00
Alexander Zavorotynskiy
e88bd5a3bf fix(backend): removed unused mobile events 2023-10-09 15:24:37 +02:00
Alexander
d22e4e138e
Backend for mobile tracker support (#1456)
* feat(backend): handlers for mobile messages

* feat(backend): new service template

* feat(backend): save mobile session start and send batches to kafka

* feat(backend): added mobile support to sink, ender and storage

* helm(videostorage): added helm chart for a new service videostorage

* fix(backend): added pointer to streing for userBrowser (because it's null for mobile sessions)

* feat(backend): added MsgIOSBatchMeta handler to message iterator's logic

* feat(backend): added ios ts parser to ender

* feat(backend): enabled producing batch of messages to queue

* feat(backend): removed iosstart from mob files

* feat(backend): added new ios message types

* feat(backend): added iosStart and iosEnd

* fix(backend): fixed log issue

* feat(backend): send tar.gz archives to special queue topic

* feat(backend): read raw archives from kafka

* fix(backend): added missing file

* fix(backend): removed the second file reading

* fix(backend): fixed wrong queue topic name

* fix(backend): fixed mobile trigger topic name

* feat(backend): added tar.gz extractor and iOSSessionEnd handler

* feat(backend): debug logs on message uploading

* fix(backend): added raw-images topic consumption

* feat(backend): now sink send iosSessionEnd to video-storage

* feat(backend): added dir creator for new sessions

* feat(backend): added correct command to execute

* feat(backend): added overwrite option

* feat(backend): added s3 uploader for video session replays

* feat(backend): new consumer group for mobile sessions

* feat(backend): debug logs for uploader

* feat(backend): removed unused log

* feat(backend): fixed s3 key for video replays

* feat(backend): removed debug code

* feat(backend): fixed video-storage message filter

* fix(backend): added mobileSessionEnd to SessionEnd converter

* feat(backend): added first version if db support for mobile events

* fix(backend): added swipe events to mob file

* feat(backend): added swipe event to pg

* feat(backend): split logic into 2 services: image-storage and video-storage

* feat(backend): added helm chart for image-storage service

* fix(backend): fixed table name for mobile taps

* feat(backend): added metadata handler for mobile message parser + fix message filters

* feat(backend): added iosRawTopic to DB message consumer

* fix(backend): removed value from mobile inputs

* feat(backend): removed debug log from iterator

* feat(backend): added new apple devices to iOS device parser

* fix(backend): added real projectID instead of 0

* feat(backend): extended a list of simulators for device detector

* feat(backend): updated networkCall mobile message

* fix(backend): added new way to define is network call successed or not

* feat(backend): added timezone support for mobile start request

* feat(backend): added 2 mobile events Input and Click to mob file

* feat(backend): refactored image storage

* feat(backend): video storage with 2 workers

* feat(backend): added project's platform support

* feat(backend): added memory size field for mobile start request

* feat(backend): changed video preset for ultrafast

* feat(backend): added debug log to http /late handler

* feat(backend): added debug log to db service for iosCrash messages

* feat(backend): added tapRage event handler to heuristics

* feat(backend): changed table and field names for ios crashes

* feat(backend): added payload for tapRage events

* feat(backend): added TapRage events insert to DB

* feat(backend): added fps value to /mobile/start response

* feat(backend): added image quality parameter to /mobile/start response

* feat(backend): added ScreenLeave handler

* feat(backend): removed screenEnter and screenLeave events, added new viewComponent event

---------

Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-10-09 15:02:20 +02:00
Alexander
bb7c086c5b
Performance patch (#1508)
* feat(backend): removed legacy InputEvent builder from heuristics service

* feat(backend): removed InputEvent handler from DB service

* feat(backend): removed InputEvent from ClickHouse connector

* feat(backend): added extra env variable for brotli compression algorithm
2023-10-09 13:54:53 +02:00
Alexander
5eaed7e351
Connectors improvements (#1469) 2023-09-12 13:12:23 +02:00
Alexander Zavorotynskiy
5618d104f1 fix(backend): added return statement to avoid wrong event inserts 2023-09-05 16:58:40 +02:00
Alexander Zavorotynskiy
7bffd2f321 feat(backend): added app crash detector to heuristics service 2023-09-05 16:57:34 +02:00
Alexander
481db19dbe
Go redshift connector (#1457)
* feat(backend): redshift connector draft

* fix(backend): fixed memory leak, empty string ddos

* feat(backend): draft for sessions part

* feat(backend): session handler

* fix(backend): fixed wrong columns list in sessionToCSV convertor

* feat(backend): load session info from db/cache if there is nothing in memory when sessionEnd event recieved

* feat(backend): added filters for connector

* feat(backend): memory leak fix + extra cache for sessions

* feat(backend): moved table names as an env variable

* fix(backend): added timeout for last session messages to avoid memory leak

* fix(backend): fixed last memory leak

* feat(backend): moved redshift connector to ee folder
2023-09-05 12:18:47 +02:00
Alexander Zavorotynskiy
3f858132c3 fix(backend): found and fixed an issue in the method for generating sessionID using startTimestamp and extraction this timestamp back from sessionID 2023-08-30 15:39:49 +02:00
Alexander Zavorotynskiy
fe2e935012 feat(backend): added new column timezone to PG and CH session tables 2023-08-21 10:27:31 +02:00
Alexander Zavorotynskiy
5391b63c45 feat(backend): force session update in cache after sessionEnd event 2023-08-10 10:28:45 +04:00