Commit graph

113 commits

Author SHA1 Message Date
Alexander
e84bdb5eef
Patch/patch ch clicks with url (#2698)
* added url and url_path to click events

* added app_crash support

* fixed an url in click event
2024-10-28 18:47:31 +01:00
Alexander
d9fe2b5bb8
added url and url_path to click events (#2697)
* added url and url_path to click events

* added app_crash support
2024-10-28 18:36:22 +01:00
Alexander
90d13c69ab
Revert "feat(clickhouse): added host/url to click events (#2680)" (#2693)
This reverts commit a56b94ba92.
2024-10-28 16:38:56 +01:00
Alexander
a56b94ba92
feat(clickhouse): added host/url to click events (#2680) 2024-10-25 14:25:20 +02:00
Alexander
99af6ce45f feat(spot): added tenant select for public key check in ee version 2024-09-11 09:57:31 +02:00
Alexander
47c33655de feat(spot): added spot jwt support to authorizer in ee version 2024-09-11 09:26:50 +02:00
Alexander
de691e0f8d feat(spot): added correct methods filter for api tracer 2024-09-11 09:18:58 +02:00
Alexander
345f316b27
Spots (#2305)
* feat(spot): first version to test http endpoints

* fix(helm): changed nginx path prefix

* fix(spots): added missing BUCKET_NAME env var

* fix(spots): added services init check

* feat(spots): removed geo module

* feat(spots): removed uaparser

* feat(spots): added more detailed authorization error log

* feat(spots): changed the authorization middleware

* feat(spots): extended http body size limit to 128kb

* feat(spots): added s3 error log

* feat(spots): added new handler for uploaded event

* feat(backend): small api changes in spot service

* feat(backend): rewrote request parameters grabber for getSpot handler

* feat(backend): added tenantID to auth struct

* feat(backend): added pre-signed download urls for preview, mob et video files

* feat(backend): added user's email to spots table, and getSpot responses

* feat(backend): returning spotID as a string

* feat(spot): added transcoder pipeline

* fix(spot): return spotID as a string

* feat(spot): added volume mount to spot service

* feat(spot): fixed volume mounting

* feat(spot): helm fix

* feat(spot): helm another fix

* fix(spot): correct video.webm path

* fix(spot): correct pre-signed url for download original video

* feat(spot): added PATCH and DELETE methods to CORS

* feat(spot): use string format for spotIDs in delete method

* feat(spot): added public key implemented

* fix(spot): correct public-key parser

* fix(spot): fixed query params issue + user's tenantID

* fix(spot): use 1 as a default tenant

* feat(spot): added correct total spots calculation

* fix(spot): fixed offset calculation

* feat(spot): added extra check in auth method

* fix(spot): removed / from video file name

* fix(spot): devided codec flag into 2 parts

* feat(spot): use fixed tenantID = 1 for oss users

* feat(spot): return 404 for public key not found issue

* feat(spots): added spots folder to minio path rule

* feat(spot): added spot video streaming support

* fix(spot): fixed an sql request for spot streams

* feat(spot): return playlist file in getSpot responce

* feat(spot): try to use aac audio codec

* feat(spot): added permissions support (oss/ee)

* feat(spot): added authorizer method

* feat(spot): added license check

* feat(spot): added spot preview for get response

* fix(spot): fixed a problem with permissions

* feat(spot): added crop feature

* feat(spot): upload cropped video back to s3

* feat(spot): manage expired modified playlist file

* feat(backend): hack with video formats

* feat(backend): removed space

* feat(spot): req tracing

* feat(spot): manual method's name mapping

* feat(spot): added a second method to public key auth support

* feat(spot): metrics

* feat(spot): added rate limiter per user

* feat(spot): added ping endpoint for spot jwt token check

* feat(spot): getStatus endpoint

* feat(spot): added missing import

* feat(spot): transcoding issue fix

* feat(spot): temp remove tasks

* feat(spot): better error log message

* feat(spot): set default jwt_secret value

* feat(spot): debug auth

* feat(spot): 2 diff jwt tokens support

* feat(spot): pg tasks with process status

* feat(spot): more logs

* feat(spot): improved defer for GetTask method

* feat(spot): keep only failed tasks

* feat(spot): removing temp dir with spot files

* feat(spot): added several workers for transcoding module

* feat(spot): fixed spot path for temp video files

* feat(spot): use custom statusWriter to track response code in middleware

* feat(spot): added body and parameter parser for auditrail feature

* feat(spot): fixed IsAuth method signature

* feat(spot): fixed ee service builder

* feat(spot): added import

* feat(spot): fix data type for payload and parameters jsonb fields

* feat(spot): typo fix

* feat(spot): moved out consts

* feat(spot): new table's name

* feat(spot): added missing imports in go.mod

* feat(spot): added a check for the number of comments (20 by default)
2024-08-29 16:08:33 +02:00
Alexander
b3d04d76d4
Heatmaps fix (float coordinates) (#2403)
* feat(spot): use float click coordinates instead of ints in PG

* feat(db): added support for float clicks in CH

* feat(db): fix float instead of uint8

* feat(mobile): new naming for mobile autocomplete types
2024-07-18 12:32:44 +02:00
Alexander
dfe527b412 feat(backend): added missing imports 2024-07-05 16:27:27 +02:00
Alexander
75eb6924c1 feat(connector): added s3 batches support for CH connector 2024-07-05 16:16:02 +02:00
Alexander
c6c46c6486 fix(backend): fixed naming issue in CH connector 2024-07-03 16:48:13 +02:00
Alexander
b9dc87884c fix(backend): fixed an empty url_path for request events in CH 2024-07-03 16:40:44 +02:00
Alexander
939d661cdc fix(backend): changed normalized_x/y from uint32 to uint8 2024-07-03 16:34:39 +02:00
Alexander
2cd210fbaa fix(backend): fixed an issue in url_path builder 2024-06-27 13:00:23 +02:00
Delirium
960da9f037
Tracker 14.x.x changes (#2240)
* feat tracker: add document titles to tabs

* feat: titles for tabs

* feat tracker: send initial title, parse titles better

* feat ui: tab name styles

* feat tracker: update changelogs

* fix tracker: fix tests

* fix tracker: fix failing tests, add some coverage

* fix tracker: fix failing tests, add some coverage

* Heatmaps  (#2264)

* feat ui: start heatmaps ui and tracker update

* fix ui: drop clickmap from session

* fix ui: refactor heatmap painter

* fix ui: store click coords as int percent

* feat(backend): insert normalized x and y to PG

* feat(backend): insert normalized x and y to CH

* feat(connector): added missing import

* feat(backend): fixed different uint type issue

* fix tracker: use max scrollable size for doc

* fix gen files

* fix ui: fix random crash, remove demo data generator

* fix ui: rm some dead code

---------

Co-authored-by: Alexander <zavorotynskiy@pm.me>

* fix tracker: add heatmap changelog to tracker CHANGELOG.md

* fix tracker: fix peerjs version to 1.5.4 (was 1.5.1)

* fix document height calculation

* Crossdomain tracking (#2277)

* feat tracker: crossdomain tracking (start commit)

* catch crossdomain messages, add nodeid placeholder

* click scroll

* frame placeholder number -> dynamic

* click rewriter, fix scroll prop

* some docs

* some docs

* fix options merging

* CHANGELOG.md update

* checking that crossdomain will not fire automatically

* simplify func declaration

* update test data

* change clickmap document height calculation to scrollheight (which should be true height)

---------

Co-authored-by: Alexander <zavorotynskiy@pm.me>
2024-06-24 13:49:26 +02:00
Alexander
3c3da5b584 feat(connector): added missing import 2024-05-30 16:38:13 +02:00
Alexander
400b3d2e61 feat(backend): added url_path inserts into CH 2024-05-29 13:25:33 +02:00
Alexander
b92177a151
Assist-only feature (#2214)
* feat(http): small prep for task implementation

* feat(http): added assist-only check to ee

* feat(backend): using a new license check url

* feat(backend): packages upgrade
2024-05-28 09:41:16 +02:00
Alexander
55b6cd7dcd feat(db): insert full url to pages table 2024-05-24 09:42:27 +02:00
Alexander
17d7f127bb Revert "feat(db): insert url path to CH pages table"
This reverts commit dee1ef1618.
2024-05-24 09:40:04 +02:00
Alexander
dee1ef1618 feat(db): insert url path to CH pages table 2024-05-24 09:27:42 +02:00
Alexander
993b6b58ab
feat(backend): added utm data to PG (#2152)
* feat(backend): added utm data to PG

* feat(backend): added utm data to CH

* feat(db): removed nullableString() calls
2024-05-24 08:33:28 +02:00
Alexander
4c96cbcef3 feat(connector): added new events + projectID filter 2024-05-23 12:09:59 +02:00
Alexander
de9ffeb6a9 feat(connector): added s3 connector + small improvements 2024-05-22 15:23:45 +02:00
Alexander
d1c4ba496d feat(backend): changed licence verification url 2024-04-25 13:56:29 +02:00
Alexander
035d116fb2 feat(backend): insert click's selector into CH 2024-04-18 13:08:33 +02:00
Alexander
2bf9298836
Mobile renaming (protocol + backend) (#2078)
* feat(backend): renamed ios to mobile

* feat(backend): added missing changes to iterator

* feat(backend): added missing changes to ender

* feat(backend): as usual auto deleted import in ch connector

* feat(backend): updated ee/conditions func calls
2024-04-12 09:30:24 +02:00
Alexander
d2cabcdb54 feat(backend): moved file tagging feature to EE 2024-03-22 14:10:13 +01:00
Alexander
b6d71194d7 feat(backend): autodeleted include 2024-03-19 14:48:05 +01:00
Alexander
6c12af6125 feat(backend): fixed an issue in CH connector 2024-03-19 12:20:46 +01:00
Alexander
36b6dfb5bd feat(backend): fixed no sessID in storage service 2024-03-14 17:17:50 +01:00
Alexander
ee75b2795a feat(backend): fixed broken logs in failover module 2024-03-14 16:40:14 +01:00
Alexander
45c956c489
Json logs format (#1952)
* feat(backend): try a new approach for logs formatting (http)

* feat(backend): added logger module

* feat(backend): added project/session info to /i endpoint

* feat(backend): found a solution for correct caller information

* feat(backend): finished logs for http handlers

* feat(backend): finished logs for mobile http handlers

* feat(backend): finished ender

* feat(backend): finished assets

* feat(backend): finished heuristics

* feat(backend): finished image-storage

* feat(backend): finished sink

* feat(backend): finished storage

* feat(backend): formatted logs in all services

* feat(backend): finished foss part

* feat(backend): added missed foss part

* feat(backend): fixed panic in memory manager and sink service

* feat(backend): connectors
2024-03-14 12:51:14 +01:00
Alexander
a73bf81902 feat(backend): added support for conditional recording feature (mobile sessions) 2024-02-26 15:13:51 +01:00
Alexander
df2ee71bbb feat(backend): fixed issue in start request 2024-01-26 14:25:44 +01:00
Alexander
354f071fb0
Conditions (ee version) (#1858)
* feat(backend): moved conditions to ee folder

* feat(backend): fixed typo
2024-01-26 14:03:29 +01:00
Alexander
4711bf28f0 feat(backend): small improvements in CH client 2024-01-26 13:48:03 +01:00
ⵄⵎⵉⵔⵓⵛ
6ae3ee1927
Canvas support (#1711)
* feat(http): added new parameters to start response and new endpoint for canvas screenshorts

* fix(http): added new topic to dockerfile

* feat(http): try different multipart parser

* feat(image-storage): reused the same workflow for canvas topic handler

* feat(video-storage): new canvas parser and ffmpeg script

* feat(video-storage): use correct replay name for canvas

* feat(backend): added new message (CanvasNode)

* feat(backend): add canvas record events to db

* feat(backend): implemented missing new method for azure

---------

Co-authored-by: Alexander <zavorotynskiy@pm.me>
2023-11-27 16:22:03 +01:00
Alexander
07d8b4e30f fix(backend): added type convertor for TransferredBodySize field in CH connector 2023-11-08 10:57:33 +01:00
nick-delirium
91f5c16e28 fix(backend): msg name 2023-10-27 12:17:11 +02:00
Alexander
7ed2967361 feat(backend): added mobile event handlers for CH db 2023-10-26 17:11:19 +02:00
Delirium
c7e5145282
feat(tracker): add beacon proxy and request body size (#1389)
* feat(tracker): add beacon proxy and body size

* feat(ui): remove unused components

* feat(ui): generate new messages, add body size to resource parser

* feat(ui): fix tooltip text, fix size detection (ts safe)

* feat(ui): cover resource with tests

* feat(ui): enable test coverage for player, utils and mstore

* fix(tracker): adjust test to support new message

* fix(tracker): fix tracker version for back compat

* feat(backend): added new column to network requests

* fix(tracker): fix unit tests

* fix(backend): fix msg gen

* fix(tracker): ci fun

* fix(tracker): changelog

* fix(tracker): fix some test

---------

Co-authored-by: Alexander Zavorotynskiy <zavorotynskiy@pm.me>
2023-10-17 12:25:20 +02:00
Alexander
7d4bbf733e
Added CH connector (#1476)
* feat(backend): added CH connector

* feat(backend): added fixed clickhouse connector
2023-10-16 14:50:51 +02:00
Alexander Zavorotynskiy
15b55c837b fix(backend): added missed import for clickhouse connector 2023-10-10 11:51:59 +02:00
Alexander
d22e4e138e
Backend for mobile tracker support (#1456)
* feat(backend): handlers for mobile messages

* feat(backend): new service template

* feat(backend): save mobile session start and send batches to kafka

* feat(backend): added mobile support to sink, ender and storage

* helm(videostorage): added helm chart for a new service videostorage

* fix(backend): added pointer to streing for userBrowser (because it's null for mobile sessions)

* feat(backend): added MsgIOSBatchMeta handler to message iterator's logic

* feat(backend): added ios ts parser to ender

* feat(backend): enabled producing batch of messages to queue

* feat(backend): removed iosstart from mob files

* feat(backend): added new ios message types

* feat(backend): added iosStart and iosEnd

* fix(backend): fixed log issue

* feat(backend): send tar.gz archives to special queue topic

* feat(backend): read raw archives from kafka

* fix(backend): added missing file

* fix(backend): removed the second file reading

* fix(backend): fixed wrong queue topic name

* fix(backend): fixed mobile trigger topic name

* feat(backend): added tar.gz extractor and iOSSessionEnd handler

* feat(backend): debug logs on message uploading

* fix(backend): added raw-images topic consumption

* feat(backend): now sink send iosSessionEnd to video-storage

* feat(backend): added dir creator for new sessions

* feat(backend): added correct command to execute

* feat(backend): added overwrite option

* feat(backend): added s3 uploader for video session replays

* feat(backend): new consumer group for mobile sessions

* feat(backend): debug logs for uploader

* feat(backend): removed unused log

* feat(backend): fixed s3 key for video replays

* feat(backend): removed debug code

* feat(backend): fixed video-storage message filter

* fix(backend): added mobileSessionEnd to SessionEnd converter

* feat(backend): added first version if db support for mobile events

* fix(backend): added swipe events to mob file

* feat(backend): added swipe event to pg

* feat(backend): split logic into 2 services: image-storage and video-storage

* feat(backend): added helm chart for image-storage service

* fix(backend): fixed table name for mobile taps

* feat(backend): added metadata handler for mobile message parser + fix message filters

* feat(backend): added iosRawTopic to DB message consumer

* fix(backend): removed value from mobile inputs

* feat(backend): removed debug log from iterator

* feat(backend): added new apple devices to iOS device parser

* fix(backend): added real projectID instead of 0

* feat(backend): extended a list of simulators for device detector

* feat(backend): updated networkCall mobile message

* fix(backend): added new way to define is network call successed or not

* feat(backend): added timezone support for mobile start request

* feat(backend): added 2 mobile events Input and Click to mob file

* feat(backend): refactored image storage

* feat(backend): video storage with 2 workers

* feat(backend): added project's platform support

* feat(backend): added memory size field for mobile start request

* feat(backend): changed video preset for ultrafast

* feat(backend): added debug log to http /late handler

* feat(backend): added debug log to db service for iosCrash messages

* feat(backend): added tapRage event handler to heuristics

* feat(backend): changed table and field names for ios crashes

* feat(backend): added payload for tapRage events

* feat(backend): added TapRage events insert to DB

* feat(backend): added fps value to /mobile/start response

* feat(backend): added image quality parameter to /mobile/start response

* feat(backend): added ScreenLeave handler

* feat(backend): removed screenEnter and screenLeave events, added new viewComponent event

---------

Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-10-09 15:02:20 +02:00
Alexander
bb7c086c5b
Performance patch (#1508)
* feat(backend): removed legacy InputEvent builder from heuristics service

* feat(backend): removed InputEvent handler from DB service

* feat(backend): removed InputEvent from ClickHouse connector

* feat(backend): added extra env variable for brotli compression algorithm
2023-10-09 13:54:53 +02:00
Alexander
5eaed7e351
Connectors improvements (#1469) 2023-09-12 13:12:23 +02:00
Alexander
481db19dbe
Go redshift connector (#1457)
* feat(backend): redshift connector draft

* fix(backend): fixed memory leak, empty string ddos

* feat(backend): draft for sessions part

* feat(backend): session handler

* fix(backend): fixed wrong columns list in sessionToCSV convertor

* feat(backend): load session info from db/cache if there is nothing in memory when sessionEnd event recieved

* feat(backend): added filters for connector

* feat(backend): memory leak fix + extra cache for sessions

* feat(backend): moved table names as an env variable

* fix(backend): added timeout for last session messages to avoid memory leak

* fix(backend): fixed last memory leak

* feat(backend): moved redshift connector to ee folder
2023-09-05 12:18:47 +02:00
Alexander Zavorotynskiy
fe2e935012 feat(backend): added new column timezone to PG and CH session tables 2023-08-21 10:27:31 +02:00