Commit graph

119 commits

Author SHA1 Message Date
Alexander Zavorotynskiy
f4883f481c feat(backend/ender): no comsuming 2023-09-11 11:29:22 +02:00
Alexander Zavorotynskiy
7bffd2f321 feat(backend): added app crash detector to heuristics service 2023-09-05 16:57:34 +02:00
Alexander
676041d90b
Ender logic update (#1435)
* feat(backend): new session end detection logic + several fixes

* feat(backend): support partitions managment in ender

* feat(backend): added new consumer support to redis

* feat(backend): added support for new consumer in kafka

* feat(backend): added new consumer support to redis (ee)

* feat(backend): small refactoring in ender
2023-08-03 14:57:15 +04:00
Alexander Zavorotynskiy
900a25993f fix(backend): removed mini ddos of sessions with negative duration 2023-07-31 15:48:25 +04:00
Alexander
4b8f3bee25
Sessions refactoring (#1371)
* feat(backend): moved sql requests related to sessions table to one place

* feat(backend): refactoring in db.Saver handler

* feat(backend): hude refactoring in db/postgres module

* fix(backend): workable feature flags

* fix(backend): workable integrations

* fix(backend): workable sessions and projects modules

* fix(backend): added missed projects module to sessions

* feat(backend): renaming

* feat(backend): moved session struct to sessions module and split methods into interface, cache and storage levels

* feat(backend): moved project struct to projects module

* feat(backend): added projects model

* feat(backend): implemented new in memory cache for sessions and projects

* feat(backend): implemented new cache in projects

* feat(backend): there are 2 methods in cache module now: Get() and GetAndRefresh()

* feat(backend): added cache update operations

* fix(backend): fixed import cycle

* fix(backend): fixed panic in db message handler

* fix(backend): fixed panic in projects module

* fix(backend): fixed panic in sessions.GetDuration

* feat(backend): added direct call to get session duration if session is already in cache

* feat(backend): used pg pool everywhere except db service

* fix(backend): added missing part after rebase

* fix(backend): removed old sessions file

* feat(backend): added refactored redis client with produce/consume options

* feat(backend): added cache layer for projects

* fix(backend): added missing redis config

* fix(backend): added missing method for producer

* feat(backend): cache integration for sessions

* feat(backend): temporary method to get session directly from db

* feat(backend): adapt EE version of message handler

* fix(backend): fixed issue in fts realisation

* fix(backend): added redis cache to sessions module

* fix(backend): set 0 duration or hesitation time for inputs without focus event

* feat(backend): added cache for session updates and failover mechanism for batch.Insert() operation

* feat(backend): debug log

* feat(backend): more debug log

* feat(backend): removed debug log

* fix(backend): fixed an issue of tracking input events with empty label

* fix(backend): disabled debug log in projects cache

* fix(backend): renamed session updater

* fix(backend): fixed closed pool issue in DB service

* fix(backend): fixed dead lock in db Stop() method

* fix(backend): fixed panic in heuristics service

* feat(backend): enabled redis cache in projects

* feat(backend): clear cache on each update operation

* feat(backend): fully integrated cache layer with auto switch

* feat(backend): small refactoring in session updates

* fix(backend): fixed wrong events counter issue

* feat(backend): enabled full cache support in ender and http services

* fix(backend/ee): added missed import

* feat(backend): added second cache layer for db to speed up the service

* feat(backend): disable redis cache

* feat(backend): moved redis cache to ee
2023-07-06 10:55:43 +02:00
Alexander
8e58c367dd
Fixed issue with wrong expiration timeout for projects (#1300)
* fix(backend): fixed issue with wrong expiration timeout for projects in cache layer

* fix(backend): removed direct call db.GetProject()

* feat(backend): set default PROJECT_EXPIRATION to 10 minutes
2023-06-08 18:08:53 +02:00
Alexander
111d6a1b54
Azure support (#1270)
* feat(backend): implemented azure blob storage support

* feat(backend): added azure implementation to backend services

* feat(backend): added azure blob storage support to chalice service

* fix(backend): removed prev version of s3

* feat(backend): moved azure support to ee subfolder

* feat(backend): prepared ee code for new utils.objects package

* feat(backend): added missed modules to go.mod

* feat(backend): added missed modules to go.sum

* feat(backend): go mod tidy

* feat(backend): temporary made s3 vars are not required

* feat(backend): added azure lib to ee chalice

* feat(api): changed azure env var name

* feat(api): added new object store and extra methods to chalice ee

* fix(api): added azure blob lib to alerts

* fix(api): fixed incorrect call in sessions_devtool

* fix(crons): added azure blob storage library to requirements list

* chore(build): Error message for not providing flag.

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(backend): removed ios headers and object store for ios messages

* feat(backend): object config refactoring

* chore(helm): Update BUCKET_NAME

* fix(backend): removed object storage usage in http

* feat(backend): added debug logs to azure upload method

* fix(backend): fixed empty folder issue

* fix(backend): removed extra debug log

* chore(helm): Adding global variables for crons

* chore(helm): Remove clickhouse resource limit

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(backend): removed assets debug log

* feat(api): use ABC class instead of empty interface

* feat(api): renamed helpers to generators

* feat(api): changed prep/clean dev scripts

* feat(api): changed name obj_store -> StorageClient

* feat(api): some changes after code review

* fix(api): removed unnecesery packages in oss api

* feat(backend): moved azure implementation to ee folder

---------

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>
Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-06-08 13:31:54 +02:00
Delirium
2ed4bba33e
feat(tracker/ui): support for multi tab sessions (#1236)
* feat(tracker): add support for multi tab sessions

* feat(backend): added support of multitabs

* fix(backend): added support of deprecated batch meta message to pre-decoder

* fix(backend): fixed nil meta issue for TabData messages in sink

* feat(player): add tabmanager

* feat(player): basic tabchange event support

* feat(player): pick tabstate for console panel and timeline

* fix(player): only display tabs that are created

* feat(player): connect performance, xray and events to tab state

* feat(player): merge all tabs data for overview

* feat(backend/tracker): extract tabdata into separate message from batchmeta

* fix(tracker): fix new session check

* fix(backend): remove batchmetadeprecated

* fix(backend): fix switch case

* fix(player): fix for tab message size

* feat(tracker): check for active tabs with broadcast channel

* feat(tracker): prevent multiple messages

* fix(tracker): ignore beacons from same tab, only ask if token isnt present yet, add small delay before start to wait for answer

* feat(player): support new msg struct in assist player

* fix(player): fix some livepl components for multi tab states

* feat(tracker): add option to disable multitab

* feat(tracker): add multitab to assist plugin

* feat(player): back compat for tab id

* fix(ui): fix missing list in controls

* fix(ui): optional list update

* feat(ui): fix visuals for multitab; use window focus event for tabs

* fix(tracker): fix for dying tests (added tabid to writer, refactored other tests)

* feat(ui): update LivePlayerSubHeader.tsx to support tabs

* feat(backend): added tabs support to devtools mob files

* feat(ui): connect state to current tab properly

* feat(backend): added multitab support to assits

* feat(backend): removed data check in agent message

* feat(backend): debug on

* fix(backend): fixed typo in message broadcast

* feat(backend): fixed issue in connect method

* fix(assist): fixed typo

* feat(assist): added more debug logs

* feat(assist): removed one log

* feat(assist): more logs

* feat(assist): use query.peerId

* feat(assist): more logs

* feat(assist): fixed session update

* fix(assist): fixed getSessions

* fix(assist): fixed request_control broadcast

* fix(assist): fixed typo

* fix(assist): added missed line

* fix(assist): fix typo

* feat(tracker): multitab support for assist sessions

* fix(tracker): fix dead tests (tabid prop)

* fix(tracker): fix yaml

* fix(tracker): timers issue

* fix(ui): fix ui E2E tests with magic?

* feat(assist): multitabs support for ee version

* fix(assist): added missed method import

* fix(tracker): fix fix events in assist

* feat(assist): added back compatibility for sessions without tabId

* fix(assist): apply message's top layer structure before broadcast call

* fix(assist): added random tabID for prev version

* fix(assist): added random tabID for prev version (ee)

* feat(assist): added debug logs

* fix(assist): fix typo in sessions_agents_count method

* fix(assist): fixed more typos in copy-pastes

* fix(tracker): fix restart timings

* feat(backend): added tabIDs for some events

* feat(ui): add tab change event to the user steps bar

* Revert "feat(backend): added tabIDs for some events"

This reverts commit 1467ad7f9f.

* feat(ui): revert timeline and xray to grab events from all tabs

* fix(ui): fix typo

---------

Co-authored-by: Alexander Zavorotynskiy <zavorotynskiy@pm.me>
2023-06-07 10:40:32 +02:00
Alexander
742c038ecd
New env variable CLOUD (aws by default) (#1232)
* feat(backend): added new env variable CLOUD (aws by default)

* chore(backend): Adding env variable for CLOUD

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

---------

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>
Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-05-04 16:48:48 +02:00
Alexander
d74ed000fa
Enable session encryption (#1121)
* feat(backend): enable session encryption

* feat(backend): fixed updated method name in failover algo

* feat(backend): disable encryption by default
2023-04-11 16:44:26 +02:00
Alexander
f561593b1a
Memory control manager (#1067)
* feat(backend): added a mechanism to control memory consumption

* feat(backend): extra log for system allocation

* feat(backend): implemented new memory manager in db and heuristics service
2023-04-11 16:01:37 +02:00
Alexander Zavorotynskiy
181756ae59 feat(backend): added unbindNodes event to db filter 2023-03-17 12:17:14 +01:00
nick-delirium
fcf4d1bc7e Merge branch 'dev' into new-frustrations 2023-03-16 17:27:19 +01:00
Alexander Zavorotynskiy
9e59d5e1ab feat(backend): added heuristics metric 2023-03-14 11:12:46 +01:00
Alexander Zavorotynskiy
3d7ab8b31d feat(backend): added new message types to db message filter 2023-03-10 12:19:32 +01:00
Alexander
c6aac11cbf
Heuristics refactoring (#987)
* feat(backend): refactored heuristics service

* feat(backend): refactored db service (moved several events to heuristics)
2023-03-09 09:54:12 +01:00
Alexander
fdd28dbc4a
Draft: New metrics module (#982)
* feat(backend): created new metrics module
2023-02-20 16:37:55 +01:00
Alexander
a40ce7c376
[DB] async db inserts for PG and CH (#967)
* feat(backend): added async workers for db service
2023-02-13 13:58:03 +01:00
Kraiem Taha Yassine
767d2f6f0c
V1.8.2 workers app name (#772)
* feat(backend): added application name to postgres connection

Co-authored-by: Alexander Zavorotynskiy <zavorotynskiy@pm.me>
2023-01-23 09:08:14 +01:00
Alexander
bfe14db353
[DB] added last quickwit changes (#950)
* feat(backend): added last quickwit changes
2023-01-20 16:18:00 +01:00
Alexander
43b042aef8
[Sink] new message iterator (#945)
* feat(backend): implemented new version of sink message iterator
2023-01-20 09:49:49 +01:00
Alexander
861302bba7
[Ender] new message iterator (#929)
* feat(backend): added new message iterator especially for ender
2023-01-18 14:43:24 +01:00
Alexander Zavorotynskiy
efe1ae1dd7 feat(backend): removed old queue logger 2022-12-20 13:51:53 +01:00
Alexander
09adccf4be
feat(backend): added env variable for enabling pprof (#880) 2022-12-13 17:48:00 +01:00
Alexander Zavorotynskiy
19b6c896b9 feat(backend): fixed big file log in storage 2022-12-13 17:28:52 +01:00
Alex K
b3cbf74deb
Merge branch 'dev' into network-request-messahe 2022-12-13 17:12:15 +01:00
Alexander
bab5a81959
[Storage] added workers perf improvements (#877)
* feat(backend): added workers for storage service
2022-12-13 14:35:41 +01:00
Alex Kaminskii
ff80386aff feat(backend,mobs):replace Fetch with NetworkRequest 2022-12-09 15:48:47 +01:00
Alex K
e861bd9641
refactor(backend):remove legacy message wrapping (#854)
* refactor(backend):remove legacy message wrapping & few backend-only messages
2022-12-06 11:54:36 +01:00
Alexander Zavorotynskiy
065ecf9e03 feat(backend/ender): logs improvements 2022-12-01 12:47:08 +01:00
Alexander Zavorotynskiy
37c31db69d feat(backend/ender): logs clean up 2022-12-01 12:33:28 +01:00
Alexander Zavorotynskiy
2dc8af998c feat(backend/ender): remove from memory sessions with wrong duration 2022-11-30 17:44:35 +01:00
Alexander Zavorotynskiy
5e8111e5d4 feat(backend): added rebalancing handler to all services 2022-11-29 13:21:16 +01:00
Alexander Zavorotynskiy
90f88abdb4 feat(backend): fix vulnerability in /x/net 2022-11-29 11:44:58 +01:00
Alexander
8a8cf4d938
feat(backend/sink): manually sync all files on partitions rebalanced event (#847) 2022-11-29 11:34:39 +01:00
Alexander
9e319ed27c
[Sink] Improved files sync algo (#831)
* feat(backend): use channel of changed sessions instead of sync.Map

* feat(backend): avoid memory alloc for message body in message iterator

* feat(backend): removed unnecessary locks in file syncer

* feat(backend): sync.Map with prev updates

* feat(backend): improved write algorith (added bufio.Writer)

* feat(backend): session writer refactoring

* feat(backend): removed unnecessary type definition

* feat(backend): added write retrier to avoid data losing

* feat(backend): refactoring

* feat(backend): added session file implementation
2022-11-25 17:25:55 +01:00
Rajesh Rajendran
4698075aa0
Make helm installation namespaces customizable (#836)
* chore(helm): Better configurability.

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* chore(init): Adding sleep 10 for resources to be up

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(helm): Updated env vars

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* chore(helm): Derive namespace from minio endpoint

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(helm): Update namespace for chalice env vars

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* chore(init): random secret for assist jwt secret

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* chore(backend): Removing unnecessary files

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>
2022-11-25 10:27:12 +00:00
Alexander
e67c3ec876
[Sink] Zombie session killer feature (#829)
* feat(backend): added zombie session killer feature
2022-11-22 11:53:21 +01:00
Alexander
5a1cd27ebc
[Sink] Async session writer (#826)
* feat(backend): implemented async session writer module
2022-11-21 19:22:10 +01:00
Alexander
91709ac909
[Sink] splitting mob files without folders (#824)
* feat(backend): split mob file into 2 without folders
2022-11-17 16:15:25 +01:00
Alexander
9afc95d894
Mob file backward compatibility (#804)
* feat(backend): added mob file backward compatibility
2022-11-09 10:52:47 +01:00
Alexander Zavorotynskiy
df8aed3464 feat(backend): get fresh session from db only on SessionEnd message 2022-11-04 09:40:11 +01:00
Alexander Zavorotynskiy
920c2523d6 fix(backend): fixed bug in sessions cache layer and in sessionEnd handler 2022-11-03 17:41:43 +01:00
Alexander
2abf063ba2
Sessions cache layer (#789)
* feat(backend/db): cache layer for sessions with auto deleting logic
2022-11-02 17:00:41 +01:00
Alexander
c4f0252677
feat(backend/sink): added cache layer for assets url rewriter (#785) 2022-10-25 12:16:45 +02:00
Alex K
974afa454b
feat(backend): insert error tags+ (#768)
* feat(backend): transform legacy messages

* refactor(backend/postgres):remove seqIndex transformation

* feat(backend/db): parse & insert errors_tags

* chore(backend): remove building folder after build

* fix(backend/db):remove message types from filter that do not go through kafka 

* fix(backend/assets):use JSException instead of ErrorEvent

* refactor(backend/db):ErrorEvent is no longer a message

* feat(backend/db):insert error tags in clickhouse

* refactor(backend):remove unused LongTask code
2022-10-19 10:31:13 +02:00
Alexander
a166482227
Session mod files encryption (#766)
* feat(backend): added session mod files encryption
2022-10-18 12:50:36 +02:00
Alex K
c3fcda45d4
Devtools separation (#752)
* feat (backend+frontend/player): writing  devtools-related messages into a separate file
2022-10-07 16:20:48 +02:00
Alexander
8bf1dcba18
Workers pool for assets (#763)
* feat(backend): added workers pool to assets
2022-10-06 14:10:25 +02:00
Alexander
617c78f224
Speed up Ender (#762)
* feat(backend): process only Timestamp messages in ender's messageHandler

* feat(backend): moved stats logger to session ender
2022-10-05 13:30:43 +02:00