Commit graph

676 commits

Author SHA1 Message Date
Alexander
402cf8f631 feat(backend): added more transparent way to use different compression algorithms in storage 2023-10-27 10:56:35 +02:00
Alexander
1cd8d311e4 feat(backend): updated go.mod file 2023-10-27 09:58:31 +02:00
rjshrjndrn
de797f614b refactor(build): clear tmp build directory
Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-10-26 18:06:16 +02:00
Alexander
14694876dd fix(backend): added missing ConnectorType field to connector config 2023-10-26 17:53:29 +02:00
Alexander
7ed2967361 feat(backend): added mobile event handlers for CH db 2023-10-26 17:11:19 +02:00
Alexander
aa2e84a674 fix(backend): added missing Database field to CH connector config 2023-10-26 15:24:23 +02:00
Alexander
f65f5e1b47 fix(backend): added missing ch config to connector service 2023-10-26 15:18:34 +02:00
Alexander
3a2ed49385 feat(backend): removed unused ObjectStorage method from interface definition 2023-10-26 14:37:23 +02:00
Delirium
c7e5145282
feat(tracker): add beacon proxy and request body size (#1389)
* feat(tracker): add beacon proxy and body size

* feat(ui): remove unused components

* feat(ui): generate new messages, add body size to resource parser

* feat(ui): fix tooltip text, fix size detection (ts safe)

* feat(ui): cover resource with tests

* feat(ui): enable test coverage for player, utils and mstore

* fix(tracker): adjust test to support new message

* fix(tracker): fix tracker version for back compat

* feat(backend): added new column to network requests

* fix(tracker): fix unit tests

* fix(backend): fix msg gen

* fix(tracker): ci fun

* fix(tracker): changelog

* fix(tracker): fix some test

---------

Co-authored-by: Alexander Zavorotynskiy <zavorotynskiy@pm.me>
2023-10-17 12:25:20 +02:00
Alexander
7d4bbf733e
Added CH connector (#1476)
* feat(backend): added CH connector

* feat(backend): added fixed clickhouse connector
2023-10-16 14:50:51 +02:00
Alexander Zavorotynskiy
cdfa53ba6e feat(backend): added empty template for new mobile event 2023-10-09 17:19:53 +02:00
Alexander Zavorotynskiy
e88bd5a3bf fix(backend): removed unused mobile events 2023-10-09 15:24:37 +02:00
Alexander Zavorotynskiy
d06192907c fix(backend): removed unused message types from iterator filter 2023-10-09 15:05:14 +02:00
Alexander
d22e4e138e
Backend for mobile tracker support (#1456)
* feat(backend): handlers for mobile messages

* feat(backend): new service template

* feat(backend): save mobile session start and send batches to kafka

* feat(backend): added mobile support to sink, ender and storage

* helm(videostorage): added helm chart for a new service videostorage

* fix(backend): added pointer to streing for userBrowser (because it's null for mobile sessions)

* feat(backend): added MsgIOSBatchMeta handler to message iterator's logic

* feat(backend): added ios ts parser to ender

* feat(backend): enabled producing batch of messages to queue

* feat(backend): removed iosstart from mob files

* feat(backend): added new ios message types

* feat(backend): added iosStart and iosEnd

* fix(backend): fixed log issue

* feat(backend): send tar.gz archives to special queue topic

* feat(backend): read raw archives from kafka

* fix(backend): added missing file

* fix(backend): removed the second file reading

* fix(backend): fixed wrong queue topic name

* fix(backend): fixed mobile trigger topic name

* feat(backend): added tar.gz extractor and iOSSessionEnd handler

* feat(backend): debug logs on message uploading

* fix(backend): added raw-images topic consumption

* feat(backend): now sink send iosSessionEnd to video-storage

* feat(backend): added dir creator for new sessions

* feat(backend): added correct command to execute

* feat(backend): added overwrite option

* feat(backend): added s3 uploader for video session replays

* feat(backend): new consumer group for mobile sessions

* feat(backend): debug logs for uploader

* feat(backend): removed unused log

* feat(backend): fixed s3 key for video replays

* feat(backend): removed debug code

* feat(backend): fixed video-storage message filter

* fix(backend): added mobileSessionEnd to SessionEnd converter

* feat(backend): added first version if db support for mobile events

* fix(backend): added swipe events to mob file

* feat(backend): added swipe event to pg

* feat(backend): split logic into 2 services: image-storage and video-storage

* feat(backend): added helm chart for image-storage service

* fix(backend): fixed table name for mobile taps

* feat(backend): added metadata handler for mobile message parser + fix message filters

* feat(backend): added iosRawTopic to DB message consumer

* fix(backend): removed value from mobile inputs

* feat(backend): removed debug log from iterator

* feat(backend): added new apple devices to iOS device parser

* fix(backend): added real projectID instead of 0

* feat(backend): extended a list of simulators for device detector

* feat(backend): updated networkCall mobile message

* fix(backend): added new way to define is network call successed or not

* feat(backend): added timezone support for mobile start request

* feat(backend): added 2 mobile events Input and Click to mob file

* feat(backend): refactored image storage

* feat(backend): video storage with 2 workers

* feat(backend): added project's platform support

* feat(backend): added memory size field for mobile start request

* feat(backend): changed video preset for ultrafast

* feat(backend): added debug log to http /late handler

* feat(backend): added debug log to db service for iosCrash messages

* feat(backend): added tapRage event handler to heuristics

* feat(backend): changed table and field names for ios crashes

* feat(backend): added payload for tapRage events

* feat(backend): added TapRage events insert to DB

* feat(backend): added fps value to /mobile/start response

* feat(backend): added image quality parameter to /mobile/start response

* feat(backend): added ScreenLeave handler

* feat(backend): removed screenEnter and screenLeave events, added new viewComponent event

---------

Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-10-09 15:02:20 +02:00
Alexander
bb7c086c5b
Performance patch (#1508)
* feat(backend): removed legacy InputEvent builder from heuristics service

* feat(backend): removed InputEvent handler from DB service

* feat(backend): removed InputEvent from ClickHouse connector

* feat(backend): added extra env variable for brotli compression algorithm
2023-10-09 13:54:53 +02:00
Alexander Zavorotynskiy
8ca58f764c feat(backend): fixed go.mod file issue 2023-09-12 14:13:30 +02:00
Alexander Zavorotynskiy
ff3084f441 feat(backend): go version upgrage 2023-09-12 13:26:09 +02:00
Alexander
5eaed7e351
Connectors improvements (#1469) 2023-09-12 13:12:23 +02:00
Alexander Zavorotynskiy
9892e63a51 feat(backend/ender): comsuming 2023-09-11 11:30:21 +02:00
Alexander Zavorotynskiy
f4883f481c feat(backend/ender): no comsuming 2023-09-11 11:29:22 +02:00
Alexander Zavorotynskiy
5618d104f1 fix(backend): added return statement to avoid wrong event inserts 2023-09-05 16:58:40 +02:00
Alexander Zavorotynskiy
7bffd2f321 feat(backend): added app crash detector to heuristics service 2023-09-05 16:57:34 +02:00
Alexander
481db19dbe
Go redshift connector (#1457)
* feat(backend): redshift connector draft

* fix(backend): fixed memory leak, empty string ddos

* feat(backend): draft for sessions part

* feat(backend): session handler

* fix(backend): fixed wrong columns list in sessionToCSV convertor

* feat(backend): load session info from db/cache if there is nothing in memory when sessionEnd event recieved

* feat(backend): added filters for connector

* feat(backend): memory leak fix + extra cache for sessions

* feat(backend): moved table names as an env variable

* fix(backend): added timeout for last session messages to avoid memory leak

* fix(backend): fixed last memory leak

* feat(backend): moved redshift connector to ee folder
2023-09-05 12:18:47 +02:00
Alexander Zavorotynskiy
3f858132c3 fix(backend): found and fixed an issue in the method for generating sessionID using startTimestamp and extraction this timestamp back from sessionID 2023-08-30 15:39:49 +02:00
Alexander Zavorotynskiy
fe2e935012 feat(backend): added new column timezone to PG and CH session tables 2023-08-21 10:27:31 +02:00
Alexander Zavorotynskiy
5391b63c45 feat(backend): force session update in cache after sessionEnd event 2023-08-10 10:28:45 +04:00
Alexander Zavorotynskiy
efd4039ec4 feat(backend): improved projects cache layer 2023-08-09 17:35:58 +04:00
Alexander Zavorotynskiy
5a59b7a1bb fix(backend): fixed an issue in projects cache layer 2023-08-09 10:50:53 +04:00
Alexander Zavorotynskiy
a01a66afeb feat(backend): fixed issue in sessions cache layer 2023-08-09 10:34:38 +04:00
Alexander
676041d90b
Ender logic update (#1435)
* feat(backend): new session end detection logic + several fixes

* feat(backend): support partitions managment in ender

* feat(backend): added new consumer support to redis

* feat(backend): added support for new consumer in kafka

* feat(backend): added new consumer support to redis (ee)

* feat(backend): small refactoring in ender
2023-08-03 14:57:15 +04:00
Alexander Zavorotynskiy
900a25993f fix(backend): removed mini ddos of sessions with negative duration 2023-07-31 15:48:25 +04:00
Alexander Zavorotynskiy
9d7cc4935d fix(backend): fixed missing js_exception issue type in sessions table 2023-07-31 10:34:46 +04:00
rjshrjndrn
62c6e06b1e fix(patch): Get correct chart name
Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-07-13 11:53:52 +02:00
Alexander Zavorotynskiy
ae0e9a1167 feat(backend): added comment about TLS for redis 2023-07-06 14:45:53 +02:00
Alexander Zavorotynskiy
02ef8dbf38 fix(backend): upgraded imports to fix critical vulnerability 2023-07-06 11:27:22 +02:00
Alexander
4b8f3bee25
Sessions refactoring (#1371)
* feat(backend): moved sql requests related to sessions table to one place

* feat(backend): refactoring in db.Saver handler

* feat(backend): hude refactoring in db/postgres module

* fix(backend): workable feature flags

* fix(backend): workable integrations

* fix(backend): workable sessions and projects modules

* fix(backend): added missed projects module to sessions

* feat(backend): renaming

* feat(backend): moved session struct to sessions module and split methods into interface, cache and storage levels

* feat(backend): moved project struct to projects module

* feat(backend): added projects model

* feat(backend): implemented new in memory cache for sessions and projects

* feat(backend): implemented new cache in projects

* feat(backend): there are 2 methods in cache module now: Get() and GetAndRefresh()

* feat(backend): added cache update operations

* fix(backend): fixed import cycle

* fix(backend): fixed panic in db message handler

* fix(backend): fixed panic in projects module

* fix(backend): fixed panic in sessions.GetDuration

* feat(backend): added direct call to get session duration if session is already in cache

* feat(backend): used pg pool everywhere except db service

* fix(backend): added missing part after rebase

* fix(backend): removed old sessions file

* feat(backend): added refactored redis client with produce/consume options

* feat(backend): added cache layer for projects

* fix(backend): added missing redis config

* fix(backend): added missing method for producer

* feat(backend): cache integration for sessions

* feat(backend): temporary method to get session directly from db

* feat(backend): adapt EE version of message handler

* fix(backend): fixed issue in fts realisation

* fix(backend): added redis cache to sessions module

* fix(backend): set 0 duration or hesitation time for inputs without focus event

* feat(backend): added cache for session updates and failover mechanism for batch.Insert() operation

* feat(backend): debug log

* feat(backend): more debug log

* feat(backend): removed debug log

* fix(backend): fixed an issue of tracking input events with empty label

* fix(backend): disabled debug log in projects cache

* fix(backend): renamed session updater

* fix(backend): fixed closed pool issue in DB service

* fix(backend): fixed dead lock in db Stop() method

* fix(backend): fixed panic in heuristics service

* feat(backend): enabled redis cache in projects

* feat(backend): clear cache on each update operation

* feat(backend): fully integrated cache layer with auto switch

* feat(backend): small refactoring in session updates

* fix(backend): fixed wrong events counter issue

* feat(backend): enabled full cache support in ender and http services

* fix(backend/ee): added missed import

* feat(backend): added second cache layer for db to speed up the service

* feat(backend): disable redis cache

* feat(backend): moved redis cache to ee
2023-07-06 10:55:43 +02:00
Alexander Zavorotynskiy
4e7b03581c feat(backend): added customIssue to issueEvent convertor 2023-06-26 14:35:41 +02:00
Alexander Zavorotynskiy
00ca3d6184 feat(backend): added user's state and city to CH 2023-06-22 14:26:31 +02:00
Alexander
db7d624b3b
[Backend] feature flags (#1354)
* feat(backend): added ff mock

* feat(backend): added feature flag pg method

* fix(backend): fixed ff request field

* feat(backend): added multivariant support

* feat(backend): added logic handler for feature flag filters

* fix(backend): correct sql request for multivariants flags

* feat(backend): added new fields to sessionStart response

* feat(backend): removed unused fields from getFeatureFlags request

* fix(backend): removed comments

* feat(backend): added debug logs

* feat(backend): added single type case for arrayToNum parser

* feat(backend): added unit tests for feature flags
2023-06-21 13:41:06 +02:00
Alexander Zavorotynskiy
d4b8d3a7f9 refactor(backend): deleted old unused code 2023-06-21 12:39:29 +02:00
Alexander Zavorotynskiy
73a4339db2 fix(backend): added toUpper() method for network requests to avoid ignoring http methods in low case 2023-06-12 11:45:26 +02:00
Alexander
8e58c367dd
Fixed issue with wrong expiration timeout for projects (#1300)
* fix(backend): fixed issue with wrong expiration timeout for projects in cache layer

* fix(backend): removed direct call db.GetProject()

* feat(backend): set default PROJECT_EXPIRATION to 10 minutes
2023-06-08 18:08:53 +02:00
Alexander
f89d84232a
[DB] Added support for city/state autocomplete (#1317)
* feat(backend): added support for city/state autocomplete

* fix(backend): removed unused http handlers
2023-06-08 15:59:53 +02:00
Alexander
111d6a1b54
Azure support (#1270)
* feat(backend): implemented azure blob storage support

* feat(backend): added azure implementation to backend services

* feat(backend): added azure blob storage support to chalice service

* fix(backend): removed prev version of s3

* feat(backend): moved azure support to ee subfolder

* feat(backend): prepared ee code for new utils.objects package

* feat(backend): added missed modules to go.mod

* feat(backend): added missed modules to go.sum

* feat(backend): go mod tidy

* feat(backend): temporary made s3 vars are not required

* feat(backend): added azure lib to ee chalice

* feat(api): changed azure env var name

* feat(api): added new object store and extra methods to chalice ee

* fix(api): added azure blob lib to alerts

* fix(api): fixed incorrect call in sessions_devtool

* fix(crons): added azure blob storage library to requirements list

* chore(build): Error message for not providing flag.

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(backend): removed ios headers and object store for ios messages

* feat(backend): object config refactoring

* chore(helm): Update BUCKET_NAME

* fix(backend): removed object storage usage in http

* feat(backend): added debug logs to azure upload method

* fix(backend): fixed empty folder issue

* fix(backend): removed extra debug log

* chore(helm): Adding global variables for crons

* chore(helm): Remove clickhouse resource limit

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(backend): removed assets debug log

* feat(api): use ABC class instead of empty interface

* feat(api): renamed helpers to generators

* feat(api): changed prep/clean dev scripts

* feat(api): changed name obj_store -> StorageClient

* feat(api): some changes after code review

* fix(api): removed unnecesery packages in oss api

* feat(backend): moved azure implementation to ee folder

---------

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>
Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-06-08 13:31:54 +02:00
Alexander
691c84d5f4
[HTTP] added city and state support for geo extractor (#1312)
* feat(backend/http): added city and state support for geo extractor

* feat(backend): use new GeoLite db in Dockerfile

* feat(backend/http): added unit test
2023-06-08 09:07:08 +02:00
Delirium
2ed4bba33e
feat(tracker/ui): support for multi tab sessions (#1236)
* feat(tracker): add support for multi tab sessions

* feat(backend): added support of multitabs

* fix(backend): added support of deprecated batch meta message to pre-decoder

* fix(backend): fixed nil meta issue for TabData messages in sink

* feat(player): add tabmanager

* feat(player): basic tabchange event support

* feat(player): pick tabstate for console panel and timeline

* fix(player): only display tabs that are created

* feat(player): connect performance, xray and events to tab state

* feat(player): merge all tabs data for overview

* feat(backend/tracker): extract tabdata into separate message from batchmeta

* fix(tracker): fix new session check

* fix(backend): remove batchmetadeprecated

* fix(backend): fix switch case

* fix(player): fix for tab message size

* feat(tracker): check for active tabs with broadcast channel

* feat(tracker): prevent multiple messages

* fix(tracker): ignore beacons from same tab, only ask if token isnt present yet, add small delay before start to wait for answer

* feat(player): support new msg struct in assist player

* fix(player): fix some livepl components for multi tab states

* feat(tracker): add option to disable multitab

* feat(tracker): add multitab to assist plugin

* feat(player): back compat for tab id

* fix(ui): fix missing list in controls

* fix(ui): optional list update

* feat(ui): fix visuals for multitab; use window focus event for tabs

* fix(tracker): fix for dying tests (added tabid to writer, refactored other tests)

* feat(ui): update LivePlayerSubHeader.tsx to support tabs

* feat(backend): added tabs support to devtools mob files

* feat(ui): connect state to current tab properly

* feat(backend): added multitab support to assits

* feat(backend): removed data check in agent message

* feat(backend): debug on

* fix(backend): fixed typo in message broadcast

* feat(backend): fixed issue in connect method

* fix(assist): fixed typo

* feat(assist): added more debug logs

* feat(assist): removed one log

* feat(assist): more logs

* feat(assist): use query.peerId

* feat(assist): more logs

* feat(assist): fixed session update

* fix(assist): fixed getSessions

* fix(assist): fixed request_control broadcast

* fix(assist): fixed typo

* fix(assist): added missed line

* fix(assist): fix typo

* feat(tracker): multitab support for assist sessions

* fix(tracker): fix dead tests (tabid prop)

* fix(tracker): fix yaml

* fix(tracker): timers issue

* fix(ui): fix ui E2E tests with magic?

* feat(assist): multitabs support for ee version

* fix(assist): added missed method import

* fix(tracker): fix fix events in assist

* feat(assist): added back compatibility for sessions without tabId

* fix(assist): apply message's top layer structure before broadcast call

* fix(assist): added random tabID for prev version

* fix(assist): added random tabID for prev version (ee)

* feat(assist): added debug logs

* fix(assist): fix typo in sessions_agents_count method

* fix(assist): fixed more typos in copy-pastes

* fix(tracker): fix restart timings

* feat(backend): added tabIDs for some events

* feat(ui): add tab change event to the user steps bar

* Revert "feat(backend): added tabIDs for some events"

This reverts commit 1467ad7f9f.

* feat(ui): revert timeline and xray to grab events from all tabs

* fix(ui): fix typo

---------

Co-authored-by: Alexander Zavorotynskiy <zavorotynskiy@pm.me>
2023-06-07 10:40:32 +02:00
Alexander
fe0840ee84
feat(backend): added support for new env variable to enable/disable Access-Control-* headers (#1308) 2023-06-06 16:54:56 +02:00
Alexander Zavorotynskiy
72ccac1aed fix(backend): added COMPRESSION_THRESHOLD env variable to Dockerfile 2023-05-09 17:50:16 +02:00
Alexander Zavorotynskiy
2c6f9a15ab fix(backend): added Content-Encoding to CORS for http service 2023-05-09 17:35:17 +02:00
Alexander
a0b842f13c
feat(backend): insert referrer to sessions table (#1237) 2023-05-05 18:22:49 +02:00