Commit graph

94 commits

Author SHA1 Message Date
Alexander
f569ed3a54 feat(backend): downgrade pg library version to fix feature flags 500 error 2023-10-30 14:00:46 +01:00
Delirium
35461acaf3
[WIP] Mobile replayer (#1452)
* fix(ui): fix up mobile recordings display

* fix(ui): some messages

* fix(ui): some messages

* fix(player): fix msg generation for ios messages

* feat(player): add generic mmanager interface for ios player impl

* feat(player): mobile player and message manager; touch manager; videoplayer

* feat(player): add iphone shells, add log panel,

* feat(player): detect ios sessions and inject correct player

* feat(player): move screen mapper to utils

* feat(player): events panel for mobile, map shell sizes to device type data,

* feat(player): added network tab to mobile player; unify network message (ios and web)

* feat(player): resize canvas up to phone screen size, fix capitalize util

* feat(player): some general changes to support mobile events and network entries

* feat(player): remove swipes from timeline

* feat(player): more stuff for list walker

* fix(ui): performance tab, mobile project typings and form

* fix(ui):some ui touches for ios replayer shell

* fix(ui): more fixes for ui, new onboarding screen for mobile projects

* feat(ui): mobile overview panel (xray)

* feat(ui): fixes for phone shell and tap events

* fix(tracker): change phone shells and sizes

* fix(tracker): fix border on replay screen

* feat(ui): use crashes from db to show in session

* feat(ui): use event name for xray

* feat(ui): some overall ui fixes

* feat(ui): IOS -> iOS

* feat(ui): change tags to ant d

* fix(ui): fast fix

* fix(ui): fix for capitalizer

* fix(ui): fix for browser display

* fix(ui): fix for note popup

* fix(ui): change exceptions display

* fix(ui): add click rage to ios xray

* fix(ui): some icons and resizing

* fix(ui): fix ios context menu overlay, fix console logs creation for ios

* feat(ui): added icons

* feat(ui): performance warnings

* feat(ui): performance warnings

* feat(ui): different styles

* feat(ui): rm debug true

* feat(ui): fix warnings display

* feat(ui): some styles for animation

* feat(ui): add some animations to warnings

* feat(ui): move perf warnings to performance graph

* feat(ui): hide/show warns dynamically

* feat(ui): new mobile touch animation

* fix(tracker): update msg for ios

* fix(ui): taprage fixes

* fix(ui): regenerate icons and messages

* fix(ui): fix msgs

* fix(backend): fix events gen

* fix(backend): fix userid msg
2023-10-27 12:12:09 +02:00
Delirium
c7e5145282
feat(tracker): add beacon proxy and request body size (#1389)
* feat(tracker): add beacon proxy and body size

* feat(ui): remove unused components

* feat(ui): generate new messages, add body size to resource parser

* feat(ui): fix tooltip text, fix size detection (ts safe)

* feat(ui): cover resource with tests

* feat(ui): enable test coverage for player, utils and mstore

* fix(tracker): adjust test to support new message

* fix(tracker): fix tracker version for back compat

* feat(backend): added new column to network requests

* fix(tracker): fix unit tests

* fix(backend): fix msg gen

* fix(tracker): ci fun

* fix(tracker): changelog

* fix(tracker): fix some test

---------

Co-authored-by: Alexander Zavorotynskiy <zavorotynskiy@pm.me>
2023-10-17 12:25:20 +02:00
Alexander Zavorotynskiy
e88bd5a3bf fix(backend): removed unused mobile events 2023-10-09 15:24:37 +02:00
Alexander
d22e4e138e
Backend for mobile tracker support (#1456)
* feat(backend): handlers for mobile messages

* feat(backend): new service template

* feat(backend): save mobile session start and send batches to kafka

* feat(backend): added mobile support to sink, ender and storage

* helm(videostorage): added helm chart for a new service videostorage

* fix(backend): added pointer to streing for userBrowser (because it's null for mobile sessions)

* feat(backend): added MsgIOSBatchMeta handler to message iterator's logic

* feat(backend): added ios ts parser to ender

* feat(backend): enabled producing batch of messages to queue

* feat(backend): removed iosstart from mob files

* feat(backend): added new ios message types

* feat(backend): added iosStart and iosEnd

* fix(backend): fixed log issue

* feat(backend): send tar.gz archives to special queue topic

* feat(backend): read raw archives from kafka

* fix(backend): added missing file

* fix(backend): removed the second file reading

* fix(backend): fixed wrong queue topic name

* fix(backend): fixed mobile trigger topic name

* feat(backend): added tar.gz extractor and iOSSessionEnd handler

* feat(backend): debug logs on message uploading

* fix(backend): added raw-images topic consumption

* feat(backend): now sink send iosSessionEnd to video-storage

* feat(backend): added dir creator for new sessions

* feat(backend): added correct command to execute

* feat(backend): added overwrite option

* feat(backend): added s3 uploader for video session replays

* feat(backend): new consumer group for mobile sessions

* feat(backend): debug logs for uploader

* feat(backend): removed unused log

* feat(backend): fixed s3 key for video replays

* feat(backend): removed debug code

* feat(backend): fixed video-storage message filter

* fix(backend): added mobileSessionEnd to SessionEnd converter

* feat(backend): added first version if db support for mobile events

* fix(backend): added swipe events to mob file

* feat(backend): added swipe event to pg

* feat(backend): split logic into 2 services: image-storage and video-storage

* feat(backend): added helm chart for image-storage service

* fix(backend): fixed table name for mobile taps

* feat(backend): added metadata handler for mobile message parser + fix message filters

* feat(backend): added iosRawTopic to DB message consumer

* fix(backend): removed value from mobile inputs

* feat(backend): removed debug log from iterator

* feat(backend): added new apple devices to iOS device parser

* fix(backend): added real projectID instead of 0

* feat(backend): extended a list of simulators for device detector

* feat(backend): updated networkCall mobile message

* fix(backend): added new way to define is network call successed or not

* feat(backend): added timezone support for mobile start request

* feat(backend): added 2 mobile events Input and Click to mob file

* feat(backend): refactored image storage

* feat(backend): video storage with 2 workers

* feat(backend): added project's platform support

* feat(backend): added memory size field for mobile start request

* feat(backend): changed video preset for ultrafast

* feat(backend): added debug log to http /late handler

* feat(backend): added debug log to db service for iosCrash messages

* feat(backend): added tapRage event handler to heuristics

* feat(backend): changed table and field names for ios crashes

* feat(backend): added payload for tapRage events

* feat(backend): added TapRage events insert to DB

* feat(backend): added fps value to /mobile/start response

* feat(backend): added image quality parameter to /mobile/start response

* feat(backend): added ScreenLeave handler

* feat(backend): removed screenEnter and screenLeave events, added new viewComponent event

---------

Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-10-09 15:02:20 +02:00
Alexander
bb7c086c5b
Performance patch (#1508)
* feat(backend): removed legacy InputEvent builder from heuristics service

* feat(backend): removed InputEvent handler from DB service

* feat(backend): removed InputEvent from ClickHouse connector

* feat(backend): added extra env variable for brotli compression algorithm
2023-10-09 13:54:53 +02:00
Alexander
5eaed7e351
Connectors improvements (#1469) 2023-09-12 13:12:23 +02:00
Alexander Zavorotynskiy
5618d104f1 fix(backend): added return statement to avoid wrong event inserts 2023-09-05 16:58:40 +02:00
Alexander Zavorotynskiy
7bffd2f321 feat(backend): added app crash detector to heuristics service 2023-09-05 16:57:34 +02:00
Alexander
4b8f3bee25
Sessions refactoring (#1371)
* feat(backend): moved sql requests related to sessions table to one place

* feat(backend): refactoring in db.Saver handler

* feat(backend): hude refactoring in db/postgres module

* fix(backend): workable feature flags

* fix(backend): workable integrations

* fix(backend): workable sessions and projects modules

* fix(backend): added missed projects module to sessions

* feat(backend): renaming

* feat(backend): moved session struct to sessions module and split methods into interface, cache and storage levels

* feat(backend): moved project struct to projects module

* feat(backend): added projects model

* feat(backend): implemented new in memory cache for sessions and projects

* feat(backend): implemented new cache in projects

* feat(backend): there are 2 methods in cache module now: Get() and GetAndRefresh()

* feat(backend): added cache update operations

* fix(backend): fixed import cycle

* fix(backend): fixed panic in db message handler

* fix(backend): fixed panic in projects module

* fix(backend): fixed panic in sessions.GetDuration

* feat(backend): added direct call to get session duration if session is already in cache

* feat(backend): used pg pool everywhere except db service

* fix(backend): added missing part after rebase

* fix(backend): removed old sessions file

* feat(backend): added refactored redis client with produce/consume options

* feat(backend): added cache layer for projects

* fix(backend): added missing redis config

* fix(backend): added missing method for producer

* feat(backend): cache integration for sessions

* feat(backend): temporary method to get session directly from db

* feat(backend): adapt EE version of message handler

* fix(backend): fixed issue in fts realisation

* fix(backend): added redis cache to sessions module

* fix(backend): set 0 duration or hesitation time for inputs without focus event

* feat(backend): added cache for session updates and failover mechanism for batch.Insert() operation

* feat(backend): debug log

* feat(backend): more debug log

* feat(backend): removed debug log

* fix(backend): fixed an issue of tracking input events with empty label

* fix(backend): disabled debug log in projects cache

* fix(backend): renamed session updater

* fix(backend): fixed closed pool issue in DB service

* fix(backend): fixed dead lock in db Stop() method

* fix(backend): fixed panic in heuristics service

* feat(backend): enabled redis cache in projects

* feat(backend): clear cache on each update operation

* feat(backend): fully integrated cache layer with auto switch

* feat(backend): small refactoring in session updates

* fix(backend): fixed wrong events counter issue

* feat(backend): enabled full cache support in ender and http services

* fix(backend/ee): added missed import

* feat(backend): added second cache layer for db to speed up the service

* feat(backend): disable redis cache

* feat(backend): moved redis cache to ee
2023-07-06 10:55:43 +02:00
Alexander Zavorotynskiy
00ca3d6184 feat(backend): added user's state and city to CH 2023-06-22 14:26:31 +02:00
Alexander
db7d624b3b
[Backend] feature flags (#1354)
* feat(backend): added ff mock

* feat(backend): added feature flag pg method

* fix(backend): fixed ff request field

* feat(backend): added multivariant support

* feat(backend): added logic handler for feature flag filters

* fix(backend): correct sql request for multivariants flags

* feat(backend): added new fields to sessionStart response

* feat(backend): removed unused fields from getFeatureFlags request

* fix(backend): removed comments

* feat(backend): added debug logs

* feat(backend): added single type case for arrayToNum parser

* feat(backend): added unit tests for feature flags
2023-06-21 13:41:06 +02:00
Alexander Zavorotynskiy
d4b8d3a7f9 refactor(backend): deleted old unused code 2023-06-21 12:39:29 +02:00
Alexander
f89d84232a
[DB] Added support for city/state autocomplete (#1317)
* feat(backend): added support for city/state autocomplete

* fix(backend): removed unused http handlers
2023-06-08 15:59:53 +02:00
Alexander
691c84d5f4
[HTTP] added city and state support for geo extractor (#1312)
* feat(backend/http): added city and state support for geo extractor

* feat(backend): use new GeoLite db in Dockerfile

* feat(backend/http): added unit test
2023-06-08 09:07:08 +02:00
Alexander
a0b842f13c
feat(backend): insert referrer to sessions table (#1237) 2023-05-05 18:22:49 +02:00
Alexander
1125d0c15d
feat(backend): stopped inserting input values into the database (#1068) 2023-04-11 16:04:24 +02:00
Alexander Zavorotynskiy
9110e9cd6d feat(backend): removed debug logs 2023-03-17 12:17:14 +01:00
Alexander Zavorotynskiy
a7ca8ac54f feat(backend): implemented new events for CH 2023-03-15 16:21:44 +01:00
Alexander Zavorotynskiy
907ae9a131 feat(backend): added hesitation time for click events 2023-03-13 12:54:34 +01:00
Alexander Zavorotynskiy
4480a5d93a fix(backend): fixed typo in issue type 2023-03-13 11:11:21 +01:00
Alexander Zavorotynskiy
ded9a88fe9 feat(backend): implemented db inserts for InputDuration and MouseThrashing 2023-03-10 09:56:31 +01:00
Alexander
c6aac11cbf
Heuristics refactoring (#987)
* feat(backend): refactored heuristics service

* feat(backend): refactored db service (moved several events to heuristics)
2023-03-09 09:54:12 +01:00
Alexander Zavorotynskiy
3d9ea580a2 fix(backend): removed wrong line from batchSet 2023-02-20 16:45:00 +01:00
Alexander
fdd28dbc4a
Draft: New metrics module (#982)
* feat(backend): created new metrics module
2023-02-20 16:37:55 +01:00
Alexander
a40ce7c376
[DB] async db inserts for PG and CH (#967)
* feat(backend): added async workers for db service
2023-02-13 13:58:03 +01:00
Kraiem Taha Yassine
a68d2f2a58
Merge pull request #905 from openreplay/db_limits
[DB] limits for some columns
2023-01-20 17:47:58 +01:00
Alexander
bfe14db353
[DB] added last quickwit changes (#950)
* feat(backend): added last quickwit changes
2023-01-20 16:18:00 +01:00
Alexander
a6e4779652
[HTTP] added custom project beacon size (#926)
* feat(backend): added project beacon size select on each sessionStart request
2023-01-18 12:36:34 +01:00
Alexander Zavorotynskiy
22fac6d5ea feat(backend): added pg limits for some columns 2022-12-30 10:52:15 +01:00
Alexander Zavorotynskiy
ee64d91a7e feat(backend): added metrics for postgres bulks (size and insert duration) 2022-12-27 18:58:46 +01:00
Alexander
f1d852abb4
[DB] moved click events to bulks (#892)
* feat(backend): moved click events to bulks
* feat(backend): insert click event with host + path
2022-12-26 15:59:32 +01:00
Alexander
c54e23386a
[DB] removed pg transactions (#889)
* feat(backend): removed pg transactions for webErrors and issueEvents
2022-12-20 13:32:32 +01:00
Alex Kaminskii
ff80386aff feat(backend,mobs):replace Fetch with NetworkRequest 2022-12-09 15:48:47 +01:00
Alex K
e861bd9641
refactor(backend):remove legacy message wrapping (#854)
* refactor(backend):remove legacy message wrapping & few backend-only messages
2022-12-06 11:54:36 +01:00
Alexander Zavorotynskiy
df8aed3464 feat(backend): get fresh session from db only on SessionEnd message 2022-11-04 09:40:11 +01:00
Alexander Zavorotynskiy
920c2523d6 fix(backend): fixed bug in sessions cache layer and in sessionEnd handler 2022-11-03 17:41:43 +01:00
Alex Kaminskii
e1cd349cb9 fix(backend/db): truncate messageIndex to Int32 before insertion 2022-10-31 11:12:27 +01:00
Alexander Zavorotynskiy
7e53bcf17c feat(backend/db): removed unnecessary logs 2022-10-26 11:52:36 +02:00
Alex K
974afa454b
feat(backend): insert error tags+ (#768)
* feat(backend): transform legacy messages

* refactor(backend/postgres):remove seqIndex transformation

* feat(backend/db): parse & insert errors_tags

* chore(backend): remove building folder after build

* fix(backend/db):remove message types from filter that do not go through kafka 

* fix(backend/assets):use JSException instead of ErrorEvent

* refactor(backend/db):ErrorEvent is no longer a message

* feat(backend/db):insert error tags in clickhouse

* refactor(backend):remove unused LongTask code
2022-10-19 10:31:13 +02:00
Alexander Zavorotynskiy
35a58edec2 feat(backend): removed context from issue event 2022-10-18 14:12:19 +02:00
Alexander
a166482227
Session mod files encryption (#766)
* feat(backend): added session mod files encryption
2022-10-18 12:50:36 +02:00
Alexander Zavorotynskiy
c526d611f5 fix(backend): fixed some issues in db service 2022-09-02 18:16:10 +02:00
Alexander Zavorotynskiy
a621dd39b4 feat(backend): added message_id to events table in CH 2022-09-02 11:59:13 +02:00
Alexander
3904509f18
ClickHouse bugs fix (#707)
* fix(backend): fixed issues in CH connector
2022-09-01 17:45:28 +02:00
Alexander Zavorotynskiy
6a956da100 fix(backend): fixed import error in db pg connector 2022-08-31 19:13:47 +02:00
Alexander
7421154939
Ch improvements (#705)
* feat(backend): in-memory batches for click house
* feat(backend): new scheme for ClickHouse tables
2022-08-31 19:00:26 +02:00
Alexander
1997d326d6
Fix duplicates (#628)
* feat(backend/ender): added check for sessionEnd duplicates
2022-07-21 10:55:20 +02:00
Alexander Zavorotynskiy
56f68273d5 fix(backend/db): fixed arguments order for autocomplete inserts 2022-07-19 17:45:22 +02:00
Alexander
4dc82294e9
feat(backend/db): implemented bulk inserts for db service (#619) 2022-07-19 15:02:15 +02:00