Commit graph

116 commits

Author SHA1 Message Date
Alexander
3b3e95a413
Observability upgrade (#3146)
* feat(metrics): grand update

* feat(metrics): fixed missing part in ee tracer

* feat(assets): added missing arg

* feat(metrics): fixed naming problems
2025-03-13 08:09:29 +01:00
Alexander
4b9be69719 feat(spot): added missing prefix support to the auth middleware 2025-03-10 15:12:19 +01:00
Alexander
4b09213448 feat(images): added a proper observability 2025-03-10 14:14:43 +01:00
Alexander
5f3b3bb2ef feat(canvases): added a proper canvas observability 2025-03-07 15:46:27 +01:00
Alexander
07ffb06db1 feat(images): renamed + small improvements 2025-03-07 11:22:55 +01:00
Alexander
4ebbfd3501 feat(canvases): improved performance 2025-03-06 14:12:49 +01:00
Alexander
6dc3dcfd4e feat(proto): removed a part of deprecated messages (min supported tracker version is 6.0.0) 2025-03-06 13:49:18 +01:00
Alexander
fedc48bd0e feat(backend): small changes from saas repo 2025-02-28 14:39:54 +01:00
Alexander
f1614b6626 feat(canvas): removed unnecessary log 2025-02-17 15:21:37 +01:00
Alexander
eb11f4bf58 feat(ender): optimized logs 2025-01-20 15:38:53 +01:00
Alexander
763aed14a1 feat(analytics): moved charts data to the separate module 2024-12-23 15:32:30 +01:00
Alexander
f05e84777b feat(analytics): small refactoring of the service's architecture 2024-12-23 14:24:06 +01:00
Alexander
d35d201a10 feat(go imports): total imports upgrade 2024-12-20 11:05:28 +01:00
Alexander
d7f810809e feat(analytics): removed unnecessary keys import 2024-12-18 11:10:38 +01:00
Shekar Siri
566d6c2fdb
feat(analytics): dashboards (#2788) 2024-12-11 09:55:07 +01:00
Alexander
9b75e4502f
ClickHouse support (#2830)
* feat(db): added CH support to db service

* feat(db): removed license check for CH client

* feat(db): removed fts integration

* feat(clickhouse): added config instead of direct env parsing

* feat(clickhouse): removed prev extraHandlers

* feat(clickhouse): an unified approach for data insertion to dbs

* feat(clickhouse): removed unused imports
2024-12-10 12:41:52 +01:00
Alexander
6830c8879f
web module refactoring (#2725)
* feat(server): moved an http server object into a pkg subdir to be reusable for http, spots, and integrations

* feat(web): isolated web module (server, router, middleware, utils) used in spots and new integrations

* feat(web): removed possible panic

* feat(web): split all handlers from http service into different packages for better management.

* feat(web): changed router's method signature

* feat(web): added missing handlers interface

* feat(web): added health middleware to remove unnecessary checks

* feat(web): customizable middleware set for web servers

* feat(web): simplified the handler's structure

* feat(web): created an unified server.Run method for all web services (http, spot, integrations)

* feat(web): fixed a json size limit issue

* feat(web): removed Keys and PG connection from router

* feat(web): simplified integration's main file

* feat(web): simplified spot's main file

* feat(web): simplified http's main file (builder)

* feat(web): refactored audit trail functionality

* feat(web): added ee version of audit trail

* feat(web): added ee version of conditions module

* feat(web): moved ee version of some web session structs

* feat(web): new format of web metrics

* feat(web): added new web metrics to all handlers

* feat(web): added justExpired feature to web ingest handler

* feat(web): added small integrations improvements
2024-11-21 17:48:04 +01:00
Alexander
30a69893bb
New backend logs integrations (#2717)
* feat(integrations): new version of backend integrations

* feat(integrations): added ingress rule

* feat(integrations): fixed a port number

* feat(integrations): enabled ingress in values.yaml

* feat(integrations): added startup log

* feat(integrations): added extra logger for 3 of 4 backend logs integrations.

* feat(integrations): removed a logs loop call

* feat(integrations): fixed a table name

* feat(integrations): disabled extra logger

* feat(integrations): made extra logger as an option

* feat(integrations): changed contentType for logs file

* feat(integrations): bug fix

* feat(integrations): struct/string config support for datadog provider

* feat(integrations): map config support for datadog provider

* feat(integrations): removed unnecessary transformation

* feat(integrations): fixed datadog and sentry response format

* feat(integrations): added correct creds parser for sentry provider

* feat(integrations): removed unnecessary return statement

* feat(integrations): added correct creds parser for elastic search

* feat(integrations): changed elastic to elasticsearch

* feat(integrations): added correct creds parser for dynatrace

* feat(integrations): fixed an issue in query request for elasticsearch provider

* feat(integrations): made extra logger configurable by env var

* feat(integrations): removed debug logs
2024-10-31 15:28:38 +01:00
Alexander
a17796e811 feat(backend): removed insert operations for session's resources 2024-10-30 13:46:13 +01:00
Delirium
97a08853e8
Webvitals for replays (#2627)
* adding new web vitals track

* adding new web vitals track

* update vitals message

* feat(heuristics): added web vitals support to the page event builder

* update mtype

* feat(heuristics): applied a new value type

* feat(heuristics): fixed if err case

* feat(heuristics): fixed the sql issue

* new event display

* tracker v 15.0.0 start

---------

Co-authored-by: Alexander <zavorotynskiy@pm.me>
2024-09-30 16:08:42 +02:00
Alexander
345f316b27
Spots (#2305)
* feat(spot): first version to test http endpoints

* fix(helm): changed nginx path prefix

* fix(spots): added missing BUCKET_NAME env var

* fix(spots): added services init check

* feat(spots): removed geo module

* feat(spots): removed uaparser

* feat(spots): added more detailed authorization error log

* feat(spots): changed the authorization middleware

* feat(spots): extended http body size limit to 128kb

* feat(spots): added s3 error log

* feat(spots): added new handler for uploaded event

* feat(backend): small api changes in spot service

* feat(backend): rewrote request parameters grabber for getSpot handler

* feat(backend): added tenantID to auth struct

* feat(backend): added pre-signed download urls for preview, mob et video files

* feat(backend): added user's email to spots table, and getSpot responses

* feat(backend): returning spotID as a string

* feat(spot): added transcoder pipeline

* fix(spot): return spotID as a string

* feat(spot): added volume mount to spot service

* feat(spot): fixed volume mounting

* feat(spot): helm fix

* feat(spot): helm another fix

* fix(spot): correct video.webm path

* fix(spot): correct pre-signed url for download original video

* feat(spot): added PATCH and DELETE methods to CORS

* feat(spot): use string format for spotIDs in delete method

* feat(spot): added public key implemented

* fix(spot): correct public-key parser

* fix(spot): fixed query params issue + user's tenantID

* fix(spot): use 1 as a default tenant

* feat(spot): added correct total spots calculation

* fix(spot): fixed offset calculation

* feat(spot): added extra check in auth method

* fix(spot): removed / from video file name

* fix(spot): devided codec flag into 2 parts

* feat(spot): use fixed tenantID = 1 for oss users

* feat(spot): return 404 for public key not found issue

* feat(spots): added spots folder to minio path rule

* feat(spot): added spot video streaming support

* fix(spot): fixed an sql request for spot streams

* feat(spot): return playlist file in getSpot responce

* feat(spot): try to use aac audio codec

* feat(spot): added permissions support (oss/ee)

* feat(spot): added authorizer method

* feat(spot): added license check

* feat(spot): added spot preview for get response

* fix(spot): fixed a problem with permissions

* feat(spot): added crop feature

* feat(spot): upload cropped video back to s3

* feat(spot): manage expired modified playlist file

* feat(backend): hack with video formats

* feat(backend): removed space

* feat(spot): req tracing

* feat(spot): manual method's name mapping

* feat(spot): added a second method to public key auth support

* feat(spot): metrics

* feat(spot): added rate limiter per user

* feat(spot): added ping endpoint for spot jwt token check

* feat(spot): getStatus endpoint

* feat(spot): added missing import

* feat(spot): transcoding issue fix

* feat(spot): temp remove tasks

* feat(spot): better error log message

* feat(spot): set default jwt_secret value

* feat(spot): debug auth

* feat(spot): 2 diff jwt tokens support

* feat(spot): pg tasks with process status

* feat(spot): more logs

* feat(spot): improved defer for GetTask method

* feat(spot): keep only failed tasks

* feat(spot): removing temp dir with spot files

* feat(spot): added several workers for transcoding module

* feat(spot): fixed spot path for temp video files

* feat(spot): use custom statusWriter to track response code in middleware

* feat(spot): added body and parameter parser for auditrail feature

* feat(spot): fixed IsAuth method signature

* feat(spot): fixed ee service builder

* feat(spot): added import

* feat(spot): fix data type for payload and parameters jsonb fields

* feat(spot): typo fix

* feat(spot): moved out consts

* feat(spot): new table's name

* feat(spot): added missing imports in go.mod

* feat(spot): added a check for the number of comments (20 by default)
2024-08-29 16:08:33 +02:00
Alexander
212b412838 fix(backend): added SetPage transform handler to support old message type 2024-07-05 10:27:43 +02:00
Alexander
cac458f5fa fix(backend): added old mouseClick event to message filter 2024-07-04 10:30:10 +02:00
Alexander
2bf9298836
Mobile renaming (protocol + backend) (#2078)
* feat(backend): renamed ios to mobile

* feat(backend): added missing changes to iterator

* feat(backend): added missing changes to ender

* feat(backend): as usual auto deleted import in ch connector

* feat(backend): updated ee/conditions func calls
2024-04-12 09:30:24 +02:00
Alexander
daddedebcc feat(backend): fixed an issue with no mobile replays for android tracker 2024-04-05 17:00:45 +02:00
Alexander
36b6dfb5bd feat(backend): fixed no sessID in storage service 2024-03-14 17:17:50 +01:00
Alexander
ee75b2795a feat(backend): fixed broken logs in failover module 2024-03-14 16:40:14 +01:00
Alexander
45c956c489
Json logs format (#1952)
* feat(backend): try a new approach for logs formatting (http)

* feat(backend): added logger module

* feat(backend): added project/session info to /i endpoint

* feat(backend): found a solution for correct caller information

* feat(backend): finished logs for http handlers

* feat(backend): finished logs for mobile http handlers

* feat(backend): finished ender

* feat(backend): finished assets

* feat(backend): finished heuristics

* feat(backend): finished image-storage

* feat(backend): finished sink

* feat(backend): finished storage

* feat(backend): formatted logs in all services

* feat(backend): finished foss part

* feat(backend): added missed foss part

* feat(backend): fixed panic in memory manager and sink service

* feat(backend): connectors
2024-03-14 12:51:14 +01:00
Alexander
89752b2acc
Canvas archives (#1938)
* feat(api): added second pre-signed urls for canvases

* feat(api): fixed old key format

* feat(backend): draft version for new approach for canvases

* feat(deploy): fixed some issues

* feat(backend): debug logs for bash command

* feat(backend): removed wrong parameter from bash pipeline

* feat(backend): removed canvas-maker service + small refactoring
2024-03-11 17:51:49 +01:00
Alexander
7c58ece936 feat(backend): added separate config for canvas-handler 2024-02-28 13:21:19 +01:00
Alexander
a073ce498d
No ffmpeg solution (#1905)
* feat(video-storage): added zstd library to the machine

* feat(video-storage): added new method to pack screenshots into compressed tar arch

* feat(video-storage): try to split command into 2

* feat(video-storage): try a new approad to avoid no file error

* feat(api): added support to a new pre-signed url for screenshots archive

* feat(video-storage): fixed an issue in extension check

* feat(video-storage): correct file name

* feat(backend): removed video-storage and splitted logic

* feat(backend): removed video-storage from helm charts

* feat(backend): split canvas and screenshot handlers

* feat(canvas): clean up canvas-handler

* feat(api): changed mobile replay url (screenshots instead of video)

* feat(backend): removed msg.SessID() call

* feat(backend): clean up code in imagestorage main
2024-02-26 14:16:43 +01:00
Alexander
4bc7c2294d feat(backend): fixed zero duration issue in mobile sessions 2024-01-25 16:11:52 +01:00
Alexander
255f879476
Mobile replay (#1853)
* feat(backend): added new trigger topic for mobile video replay maker

* feat(backend): mobile video maker with mix list support

* feat(backend): fixed panic

* feat(backend): removed commented code
2024-01-25 14:30:40 +01:00
Alexander
acc21fe82d feat(backend): fixed typo in canvas group name 2024-01-23 11:37:47 +01:00
Alexander
78a9146144 feat(backend): added new consumer groups for new canvas pipeline 2024-01-22 17:03:15 +01:00
Alexander
68a87b2743
Video replay pipeline split (#1848)
* feat(backend): split

* feat(docker): added ffmpeg to canvas-maker service
2024-01-22 09:25:58 +01:00
Alexander
622b189d39
Tag and watch backend support (#1838)
* feat(ui/tracker): start tag n watch

* fix(tracker): test coverage, fix some watcher api

* fix(tracker): add intersectionobserver, adjust tests

* feat(tracker): relay + apollo plugins

* feat(ui): tags search

* feat(ui): tags name edit

* feat(ui): tags search icon

* feat(ui): icons for tabs in player

* feat(ui): save and find button

* feat(tracker): save tags in session storage (just in case)

* feat(ui): improve loading

* feat(ui): fix icon names gen

* feat(ui): fix typo

* feat(backend): added TagAndWatch backend support

* feat(backend): use msgIndex as a seqIndex

* feat(backend): try to use truncSqlIdx for seq_index

* feat(backend): added tags cache and tags filter to DB service

* feat(backend): removed debug logs

---------

Co-authored-by: nick-delirium <nikita@openreplay.com>
2024-01-19 15:21:10 +01:00
Alexander
b4b6ceb025
Canvas refactoring (#1836)
* feat(backend): added new topic for canvasToVideo communication, moved some logic

* feat(backend): enabled canvas recording

* feat(backend): fixed canvas service main logic

* feat(backend): fixed sessionEnd detector

* feat(backend): send canvas mix lists instead of just a sessID

* feat(backend): enabled canvas recording

* feat(backend): removed old logs from video-storage

* feat(backend): default low setting for canvas recording
2024-01-17 14:39:12 +01:00
Alexander
f8a8dfa459
[Integrations] small refactoring (#1752)
* feat(backend): small refactoring in integrations

* feat(backend): ignore context timeout error
2023-12-14 11:28:05 +01:00
Alexander
552c7e77e1 feat(backend): removed debug logs from videostorage 2023-12-06 12:22:29 +01:00
ⵄⵎⵉⵔⵓⵛ
6ae3ee1927
Canvas support (#1711)
* feat(http): added new parameters to start response and new endpoint for canvas screenshorts

* fix(http): added new topic to dockerfile

* feat(http): try different multipart parser

* feat(image-storage): reused the same workflow for canvas topic handler

* feat(video-storage): new canvas parser and ffmpeg script

* feat(video-storage): use correct replay name for canvas

* feat(backend): added new message (CanvasNode)

* feat(backend): add canvas record events to db

* feat(backend): implemented missing new method for azure

---------

Co-authored-by: Alexander <zavorotynskiy@pm.me>
2023-11-27 16:22:03 +01:00
Alexander
dd8050aaa2 feat(backend): fixed possible panic in ender 2023-10-30 17:40:41 +01:00
Delirium
35461acaf3
[WIP] Mobile replayer (#1452)
* fix(ui): fix up mobile recordings display

* fix(ui): some messages

* fix(ui): some messages

* fix(player): fix msg generation for ios messages

* feat(player): add generic mmanager interface for ios player impl

* feat(player): mobile player and message manager; touch manager; videoplayer

* feat(player): add iphone shells, add log panel,

* feat(player): detect ios sessions and inject correct player

* feat(player): move screen mapper to utils

* feat(player): events panel for mobile, map shell sizes to device type data,

* feat(player): added network tab to mobile player; unify network message (ios and web)

* feat(player): resize canvas up to phone screen size, fix capitalize util

* feat(player): some general changes to support mobile events and network entries

* feat(player): remove swipes from timeline

* feat(player): more stuff for list walker

* fix(ui): performance tab, mobile project typings and form

* fix(ui):some ui touches for ios replayer shell

* fix(ui): more fixes for ui, new onboarding screen for mobile projects

* feat(ui): mobile overview panel (xray)

* feat(ui): fixes for phone shell and tap events

* fix(tracker): change phone shells and sizes

* fix(tracker): fix border on replay screen

* feat(ui): use crashes from db to show in session

* feat(ui): use event name for xray

* feat(ui): some overall ui fixes

* feat(ui): IOS -> iOS

* feat(ui): change tags to ant d

* fix(ui): fast fix

* fix(ui): fix for capitalizer

* fix(ui): fix for browser display

* fix(ui): fix for note popup

* fix(ui): change exceptions display

* fix(ui): add click rage to ios xray

* fix(ui): some icons and resizing

* fix(ui): fix ios context menu overlay, fix console logs creation for ios

* feat(ui): added icons

* feat(ui): performance warnings

* feat(ui): performance warnings

* feat(ui): different styles

* feat(ui): rm debug true

* feat(ui): fix warnings display

* feat(ui): some styles for animation

* feat(ui): add some animations to warnings

* feat(ui): move perf warnings to performance graph

* feat(ui): hide/show warns dynamically

* feat(ui): new mobile touch animation

* fix(tracker): update msg for ios

* fix(ui): taprage fixes

* fix(ui): regenerate icons and messages

* fix(ui): fix msgs

* fix(backend): fix events gen

* fix(backend): fix userid msg
2023-10-27 12:12:09 +02:00
Alexander Zavorotynskiy
d06192907c fix(backend): removed unused message types from iterator filter 2023-10-09 15:05:14 +02:00
Alexander
d22e4e138e
Backend for mobile tracker support (#1456)
* feat(backend): handlers for mobile messages

* feat(backend): new service template

* feat(backend): save mobile session start and send batches to kafka

* feat(backend): added mobile support to sink, ender and storage

* helm(videostorage): added helm chart for a new service videostorage

* fix(backend): added pointer to streing for userBrowser (because it's null for mobile sessions)

* feat(backend): added MsgIOSBatchMeta handler to message iterator's logic

* feat(backend): added ios ts parser to ender

* feat(backend): enabled producing batch of messages to queue

* feat(backend): removed iosstart from mob files

* feat(backend): added new ios message types

* feat(backend): added iosStart and iosEnd

* fix(backend): fixed log issue

* feat(backend): send tar.gz archives to special queue topic

* feat(backend): read raw archives from kafka

* fix(backend): added missing file

* fix(backend): removed the second file reading

* fix(backend): fixed wrong queue topic name

* fix(backend): fixed mobile trigger topic name

* feat(backend): added tar.gz extractor and iOSSessionEnd handler

* feat(backend): debug logs on message uploading

* fix(backend): added raw-images topic consumption

* feat(backend): now sink send iosSessionEnd to video-storage

* feat(backend): added dir creator for new sessions

* feat(backend): added correct command to execute

* feat(backend): added overwrite option

* feat(backend): added s3 uploader for video session replays

* feat(backend): new consumer group for mobile sessions

* feat(backend): debug logs for uploader

* feat(backend): removed unused log

* feat(backend): fixed s3 key for video replays

* feat(backend): removed debug code

* feat(backend): fixed video-storage message filter

* fix(backend): added mobileSessionEnd to SessionEnd converter

* feat(backend): added first version if db support for mobile events

* fix(backend): added swipe events to mob file

* feat(backend): added swipe event to pg

* feat(backend): split logic into 2 services: image-storage and video-storage

* feat(backend): added helm chart for image-storage service

* fix(backend): fixed table name for mobile taps

* feat(backend): added metadata handler for mobile message parser + fix message filters

* feat(backend): added iosRawTopic to DB message consumer

* fix(backend): removed value from mobile inputs

* feat(backend): removed debug log from iterator

* feat(backend): added new apple devices to iOS device parser

* fix(backend): added real projectID instead of 0

* feat(backend): extended a list of simulators for device detector

* feat(backend): updated networkCall mobile message

* fix(backend): added new way to define is network call successed or not

* feat(backend): added timezone support for mobile start request

* feat(backend): added 2 mobile events Input and Click to mob file

* feat(backend): refactored image storage

* feat(backend): video storage with 2 workers

* feat(backend): added project's platform support

* feat(backend): added memory size field for mobile start request

* feat(backend): changed video preset for ultrafast

* feat(backend): added debug log to http /late handler

* feat(backend): added debug log to db service for iosCrash messages

* feat(backend): added tapRage event handler to heuristics

* feat(backend): changed table and field names for ios crashes

* feat(backend): added payload for tapRage events

* feat(backend): added TapRage events insert to DB

* feat(backend): added fps value to /mobile/start response

* feat(backend): added image quality parameter to /mobile/start response

* feat(backend): added ScreenLeave handler

* feat(backend): removed screenEnter and screenLeave events, added new viewComponent event

---------

Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-10-09 15:02:20 +02:00
Alexander
bb7c086c5b
Performance patch (#1508)
* feat(backend): removed legacy InputEvent builder from heuristics service

* feat(backend): removed InputEvent handler from DB service

* feat(backend): removed InputEvent from ClickHouse connector

* feat(backend): added extra env variable for brotli compression algorithm
2023-10-09 13:54:53 +02:00
Alexander Zavorotynskiy
9892e63a51 feat(backend/ender): comsuming 2023-09-11 11:30:21 +02:00
Alexander Zavorotynskiy
f4883f481c feat(backend/ender): no comsuming 2023-09-11 11:29:22 +02:00
Alexander Zavorotynskiy
7bffd2f321 feat(backend): added app crash detector to heuristics service 2023-09-05 16:57:34 +02:00
Alexander
676041d90b
Ender logic update (#1435)
* feat(backend): new session end detection logic + several fixes

* feat(backend): support partitions managment in ender

* feat(backend): added new consumer support to redis

* feat(backend): added support for new consumer in kafka

* feat(backend): added new consumer support to redis (ee)

* feat(backend): small refactoring in ender
2023-08-03 14:57:15 +04:00