Commit graph

24 commits

Author SHA1 Message Date
Alexander
45c956c489
Json logs format (#1952)
* feat(backend): try a new approach for logs formatting (http)

* feat(backend): added logger module

* feat(backend): added project/session info to /i endpoint

* feat(backend): found a solution for correct caller information

* feat(backend): finished logs for http handlers

* feat(backend): finished logs for mobile http handlers

* feat(backend): finished ender

* feat(backend): finished assets

* feat(backend): finished heuristics

* feat(backend): finished image-storage

* feat(backend): finished sink

* feat(backend): finished storage

* feat(backend): formatted logs in all services

* feat(backend): finished foss part

* feat(backend): added missed foss part

* feat(backend): fixed panic in memory manager and sink service

* feat(backend): connectors
2024-03-14 12:51:14 +01:00
Alexander
d22e4e138e
Backend for mobile tracker support (#1456)
* feat(backend): handlers for mobile messages

* feat(backend): new service template

* feat(backend): save mobile session start and send batches to kafka

* feat(backend): added mobile support to sink, ender and storage

* helm(videostorage): added helm chart for a new service videostorage

* fix(backend): added pointer to streing for userBrowser (because it's null for mobile sessions)

* feat(backend): added MsgIOSBatchMeta handler to message iterator's logic

* feat(backend): added ios ts parser to ender

* feat(backend): enabled producing batch of messages to queue

* feat(backend): removed iosstart from mob files

* feat(backend): added new ios message types

* feat(backend): added iosStart and iosEnd

* fix(backend): fixed log issue

* feat(backend): send tar.gz archives to special queue topic

* feat(backend): read raw archives from kafka

* fix(backend): added missing file

* fix(backend): removed the second file reading

* fix(backend): fixed wrong queue topic name

* fix(backend): fixed mobile trigger topic name

* feat(backend): added tar.gz extractor and iOSSessionEnd handler

* feat(backend): debug logs on message uploading

* fix(backend): added raw-images topic consumption

* feat(backend): now sink send iosSessionEnd to video-storage

* feat(backend): added dir creator for new sessions

* feat(backend): added correct command to execute

* feat(backend): added overwrite option

* feat(backend): added s3 uploader for video session replays

* feat(backend): new consumer group for mobile sessions

* feat(backend): debug logs for uploader

* feat(backend): removed unused log

* feat(backend): fixed s3 key for video replays

* feat(backend): removed debug code

* feat(backend): fixed video-storage message filter

* fix(backend): added mobileSessionEnd to SessionEnd converter

* feat(backend): added first version if db support for mobile events

* fix(backend): added swipe events to mob file

* feat(backend): added swipe event to pg

* feat(backend): split logic into 2 services: image-storage and video-storage

* feat(backend): added helm chart for image-storage service

* fix(backend): fixed table name for mobile taps

* feat(backend): added metadata handler for mobile message parser + fix message filters

* feat(backend): added iosRawTopic to DB message consumer

* fix(backend): removed value from mobile inputs

* feat(backend): removed debug log from iterator

* feat(backend): added new apple devices to iOS device parser

* fix(backend): added real projectID instead of 0

* feat(backend): extended a list of simulators for device detector

* feat(backend): updated networkCall mobile message

* fix(backend): added new way to define is network call successed or not

* feat(backend): added timezone support for mobile start request

* feat(backend): added 2 mobile events Input and Click to mob file

* feat(backend): refactored image storage

* feat(backend): video storage with 2 workers

* feat(backend): added project's platform support

* feat(backend): added memory size field for mobile start request

* feat(backend): changed video preset for ultrafast

* feat(backend): added debug log to http /late handler

* feat(backend): added debug log to db service for iosCrash messages

* feat(backend): added tapRage event handler to heuristics

* feat(backend): changed table and field names for ios crashes

* feat(backend): added payload for tapRage events

* feat(backend): added TapRage events insert to DB

* feat(backend): added fps value to /mobile/start response

* feat(backend): added image quality parameter to /mobile/start response

* feat(backend): added ScreenLeave handler

* feat(backend): removed screenEnter and screenLeave events, added new viewComponent event

---------

Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-10-09 15:02:20 +02:00
Delirium
2ed4bba33e
feat(tracker/ui): support for multi tab sessions (#1236)
* feat(tracker): add support for multi tab sessions

* feat(backend): added support of multitabs

* fix(backend): added support of deprecated batch meta message to pre-decoder

* fix(backend): fixed nil meta issue for TabData messages in sink

* feat(player): add tabmanager

* feat(player): basic tabchange event support

* feat(player): pick tabstate for console panel and timeline

* fix(player): only display tabs that are created

* feat(player): connect performance, xray and events to tab state

* feat(player): merge all tabs data for overview

* feat(backend/tracker): extract tabdata into separate message from batchmeta

* fix(tracker): fix new session check

* fix(backend): remove batchmetadeprecated

* fix(backend): fix switch case

* fix(player): fix for tab message size

* feat(tracker): check for active tabs with broadcast channel

* feat(tracker): prevent multiple messages

* fix(tracker): ignore beacons from same tab, only ask if token isnt present yet, add small delay before start to wait for answer

* feat(player): support new msg struct in assist player

* fix(player): fix some livepl components for multi tab states

* feat(tracker): add option to disable multitab

* feat(tracker): add multitab to assist plugin

* feat(player): back compat for tab id

* fix(ui): fix missing list in controls

* fix(ui): optional list update

* feat(ui): fix visuals for multitab; use window focus event for tabs

* fix(tracker): fix for dying tests (added tabid to writer, refactored other tests)

* feat(ui): update LivePlayerSubHeader.tsx to support tabs

* feat(backend): added tabs support to devtools mob files

* feat(ui): connect state to current tab properly

* feat(backend): added multitab support to assits

* feat(backend): removed data check in agent message

* feat(backend): debug on

* fix(backend): fixed typo in message broadcast

* feat(backend): fixed issue in connect method

* fix(assist): fixed typo

* feat(assist): added more debug logs

* feat(assist): removed one log

* feat(assist): more logs

* feat(assist): use query.peerId

* feat(assist): more logs

* feat(assist): fixed session update

* fix(assist): fixed getSessions

* fix(assist): fixed request_control broadcast

* fix(assist): fixed typo

* fix(assist): added missed line

* fix(assist): fix typo

* feat(tracker): multitab support for assist sessions

* fix(tracker): fix dead tests (tabid prop)

* fix(tracker): fix yaml

* fix(tracker): timers issue

* fix(ui): fix ui E2E tests with magic?

* feat(assist): multitabs support for ee version

* fix(assist): added missed method import

* fix(tracker): fix fix events in assist

* feat(assist): added back compatibility for sessions without tabId

* fix(assist): apply message's top layer structure before broadcast call

* fix(assist): added random tabID for prev version

* fix(assist): added random tabID for prev version (ee)

* feat(assist): added debug logs

* fix(assist): fix typo in sessions_agents_count method

* fix(assist): fixed more typos in copy-pastes

* fix(tracker): fix restart timings

* feat(backend): added tabIDs for some events

* feat(ui): add tab change event to the user steps bar

* Revert "feat(backend): added tabIDs for some events"

This reverts commit 1467ad7f9f.

* feat(ui): revert timeline and xray to grab events from all tabs

* fix(ui): fix typo

---------

Co-authored-by: Alexander Zavorotynskiy <zavorotynskiy@pm.me>
2023-06-07 10:40:32 +02:00
Alexander
c6aac11cbf
Heuristics refactoring (#987)
* feat(backend): refactored heuristics service

* feat(backend): refactored db service (moved several events to heuristics)
2023-03-09 09:54:12 +01:00
Alexander
fdd28dbc4a
Draft: New metrics module (#982)
* feat(backend): created new metrics module
2023-02-20 16:37:55 +01:00
Alexander
43b042aef8
[Sink] new message iterator (#945)
* feat(backend): implemented new version of sink message iterator
2023-01-20 09:49:49 +01:00
Alexander
09adccf4be
feat(backend): added env variable for enabling pprof (#880) 2022-12-13 17:48:00 +01:00
Alexander Zavorotynskiy
90f88abdb4 feat(backend): fix vulnerability in /x/net 2022-11-29 11:44:58 +01:00
Alexander
8a8cf4d938
feat(backend/sink): manually sync all files on partitions rebalanced event (#847) 2022-11-29 11:34:39 +01:00
Alexander
9e319ed27c
[Sink] Improved files sync algo (#831)
* feat(backend): use channel of changed sessions instead of sync.Map

* feat(backend): avoid memory alloc for message body in message iterator

* feat(backend): removed unnecessary locks in file syncer

* feat(backend): sync.Map with prev updates

* feat(backend): improved write algorith (added bufio.Writer)

* feat(backend): session writer refactoring

* feat(backend): removed unnecessary type definition

* feat(backend): added write retrier to avoid data losing

* feat(backend): refactoring

* feat(backend): added session file implementation
2022-11-25 17:25:55 +01:00
Alexander
e67c3ec876
[Sink] Zombie session killer feature (#829)
* feat(backend): added zombie session killer feature
2022-11-22 11:53:21 +01:00
Alexander
5a1cd27ebc
[Sink] Async session writer (#826)
* feat(backend): implemented async session writer module
2022-11-21 19:22:10 +01:00
Alexander
91709ac909
[Sink] splitting mob files without folders (#824)
* feat(backend): split mob file into 2 without folders
2022-11-17 16:15:25 +01:00
Alexander
9afc95d894
Mob file backward compatibility (#804)
* feat(backend): added mob file backward compatibility
2022-11-09 10:52:47 +01:00
Alexander
c4f0252677
feat(backend/sink): added cache layer for assets url rewriter (#785) 2022-10-25 12:16:45 +02:00
Alex K
c3fcda45d4
Devtools separation (#752)
* feat (backend+frontend/player): writing  devtools-related messages into a separate file
2022-10-07 16:20:48 +02:00
Alexander Zavorotynskiy
43434f892a feat(backend): fixes for graceful shutdown in services 2022-09-28 17:29:09 +02:00
Alexander
56ed06ed17
Message processing refactoring (#743)
* feat(backend): refactored message processing logic and cleaned up previous changes
2022-09-28 13:41:50 +02:00
Alexander Zavorotynskiy
9ff555cc64 feat(backend): set seek pointer at the end of batch for avoiding memory leaks 2022-09-19 10:59:00 +02:00
Alexander Zavorotynskiy
0eed5b58d4 feat(backend): extra checks to avoid nil cast panic 2022-09-08 16:21:41 +02:00
Alexander Zavorotynskiy
513aa89267 fix(backend/sink): fixed messages types 2022-08-30 10:43:33 +02:00
Alex Kaminskii
15c6370153 fix(backend): fix rewrite-message filter 2022-08-29 10:19:06 +02:00
Alexander
5887ab5ddb
Message protocol update (message size) (#663)
* feat(tracker): update message schema with BatchMetadata; separate message-related responsibilities; add message size
* chore(docker): removing edge busybox, as the main repo updated
* feat(backend): updated message protocol templates
* feat(backend): added support of message size
* feat(backend): implemented iterator for new message protocol (with message size)

Co-authored-by: Alex Kaminskii <alex@openreplay.com>
2022-08-22 12:32:48 +02:00
Kraiem Taha Yassine
16c70044fb
Chore(release): v1.7.0 (#578)
* change(ui) - redirect to the landing url on SSO login

* fix(ui): fix share popup styles

* change(ui) - non admin user preference restrictions

* fix(ui) - redirect fix

* change(ui) - show installation btn without mouse hover

* feat(api): api-v1 handle wrong projectKey
feat(api): api-v1 get live sessions

* change(ui) - show role edit on hover

* change(ui) - audit trail count with comma

* fix(ui) - audit trail date range custom picker alignment

* change(ui) - show a message when mob file not found

* feat(api): api-v1 fixed search live sessions

* feat(api): api-v1 handle wrong projectKey

* feat(api): fixed assist error response

* fix(tracker): check node scrolls only on start

* fixup! fix(tracker): check node scrolls only on start

* feat(ui/player): scroll view in click map

* feat(ui/player): rm unused check

* New configuration module (#558)

* ci(dbmigrate): Create db migrate when there is change

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui): fix login error/button margins

* fix(ui) - checkbox click

* fix(ui) - search rename and save fixes

* change(ui) - text changes

* fix(ui) - button text nowrap

* fix(ui): fix slowestdomains widget height

* change(ui) - ignore clicks while annotating

* change(ui) - if block with braces

* change(ui) - capitalize first letter in breadcrumb

* feat(db): remove errors from permissions
feat(api): remove errors from permissions

* feat(api): changed reset password response

* fix(ui) - assist active tab list, broken after with new api changes (pagination)

* fix(ui) - assist active tab list, broken after with new api changes (pagination)

* change(ui) - search compare

* fix(ui): last fixes for 1.7

* fix(ui): fix timeline

* fix(ui): small code fixes

* fix(ui): remove unused

* feat(frontend/assist): show when client tab is inactive + fix reconnection status update

* fix(ui) - visibility settings

* feat(assist): refactored extractSessionInfo
feat(assist): hardcoded session's attributes

* Added snabbdom (JS)

* fix(tracker): version check works with x.x.x-beta versions

* fix(backend): keep the highest user's timestamp instead of the latest message timestamp for correct session duration value

* feat(backend/s3): added file tag RETENTION (#561)

* change(ui) - search optimization and autocomplete improvements

* feat(backend/assets): added new metrics assets_downloaded

* change(ui) - show back the date range in bookmarks since the api is filtering by daterange

* feat(backend-assets): custom headers for cacher requests

* chore(backend): no tidy in dockerfile (local build speed up)

* feat(backend/assets): added proxy support for cacher module

* feat(backend/storage): set retention env variable as not required

* fix(ui): fix jira issues

* ci(helm): use kubectl for deployment

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(tracker):3.5.13: performance improvements for a case of extensive dom

* fix(backend): added missed err var and continue statement

* ci(helm): forcing namespace

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(api): fixed slowest_domains query

* ci(helm): update helm deployment method

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* change(ui) - filter dropdown colros

* fix(ui) - speed index location avg attribute changed to value

* ci(api): enable kubectl apply

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui) - widget y axis label

* feat(api): fixed slowest_domains query

* chore(helm): Adding namespaces to all templates (#565)

* feat(api): assist type-autocomplete

* feat(api): assist global-autocomplete

* feat(sourcemaps): include wasm file in build

* feat(sourcemaps-reader): refactored

* fix(ui): fix data for funnels

* fix(ui): fix all sessions section margin

* fix(ui) - assist loader flag

* fix(ui) - assist loader flag

* fix(ui): fix weird check

* feat(api): autocomplete accept unsupported types

* feat(ui): migrate to yarn v3

* feat(ui): minor fixes for installment

* feat(ui): add typescript plugin to yarn

* chore(helm): Ability to override image registry

* chore(helm): Overriding openreplay docker registry

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui): fix control arrows on firefox

* feat(crons): EE crons

* feat(api): fixed build script

* feat(alerts): fixed build script

* feat(crons): fixed build script

* chore(helm): Updating cron version

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(crons): changes

* chore(helm): optional minio ingress

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(crons): fix build script
feat(alerts): fix build script

* Revert "chore(helm): Updating cron version"

This reverts commit 3ca190ea2f.

* feat(crons): fix build script

* feat(crons): fix Dockerfile

* feat(api): fixed metadata change-case

* change(ui) - remove capitalize for the meta value

* change(ui) - autocomplete improvements with custom textfield

* fix(tracker):3.5.13+:reuse metadata on internal-caused restarts

* fix(tracker-assist):3.5.13:send active:true on start; scroll behavior fix

* change(ui) - filters autocomplete blur on pressing Enter key

* fix(tracker): fix node v to lower

* fix(tracker): fix deps

* fix(tracker): fix deps

* fix(ui) - dashboard modal width

* change(ui) - filter dropdown overflow

* chore(helm): clickhouse reclaim polity to retain

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(tracker): fix engine max v

* fix(ui): load metadata in assist tab for sorting

* fix(ui): rm unneeded api call

* fix(tracker): build script to cjs

* change(ui) - removed sample data

* chore(tracker): remove upper node version limit

* Updating Beacon size

Beacon size should be <= QUEUE_MESSAGE_SIZE_LIMIT

* feat(crons): run 24/7
feat(alerts): support env-file override

* feat(api): changed EE env handler

* fix(ui): fix sessions search modal

* change(ui) - margin for error message

* change(ui) - disable assist sort when there are no meta options to choose

* chore(helm): Adding utilities service namespace

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui) - dashboard date range selection reload, metric not found message

* change(ui) - disable clearsearch in assist when there are no filters\

* feat(api): fixed EE env handler

* chore(helm): Adding migration namespaces

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui) - report logo path

* chore(helm): Removing unnecessary SA

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(api): changed EE env handler

* feat(api): changed EE env handler

* feat(api): changed EE env handler

* feat(api): changed EE env handler

* feat(crons): changed crons

* feat(api): accept wrong metric_id

* feat(crons): changed env handler
feat(api): changed env handler
feat(alerts): changed env handler

* feat(utilities): support old version of nodejs

* feat(crons): changed env handler
feat(api): changed env handler
feat(alerts): changed env handler

* fix(tracker): fix srcset tracking

* chore(build): Adding frontent

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(assist): changed general helper

* feat(assist): changed general helper

* fix(ui): fix widget pagination (#570)

* feat(crons): changed entrypoint

* feat(player): dev-log on skipping message

* fix(tracker): removeNode mutation priority over attributes

* fix(tracker): capture relative img timing;use startsWith instead of substr; codestyle fix

* chore(build): fixing api build script

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* chore(ci): faster deployment

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* change(ui) - assist list show active status

* chore(actions): option to build all/specific services in GH

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui) - slowest domain metric data as per the api changes

* ci(helm): updated variable name

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* ci(backend): cherrypick changes to ee

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(backend): disabled pprof in http service

* fix(ui) - TimeToRender avg value as per the API change

* fix(ui) - ResponseTimeDistribution avg value as per the API change

* fix(ui) - MemoryConsumption avg value as per the API change

* fix(ui) - ResponseTime avg value as per the API change

* fix(ui) - DomBuildTime avg value as per the API change

* fix(ui) - FrameRate avg value as per the API change

* chore(helm): proper default tag

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(backend): removed sensitive information from http logs

* ci(backend): adding default parameter value for workflow dispatch

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(backend): deleted empty file

* fix(actions): creating image source file prior

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(helm): variable substitution

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* change(ui) - project list item installation button text change

* fix(ui) - project create validation

* fix(backend): removed unsafe string logs in http service

* chore(kafka): Adding new topic

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(efs-cron): variable name

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui) - developer tools - hint links

* fix(ui) - session filters - country and platform dropdown values

* chore(helm): updating version

* chore(kafka): Update kafka default message size while provisioning

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(tracker): fix dependency security

* change(ui) - webhook delete confirmation

* change(ui) - assist url to handle when empty

* feat(api): autocomplete replace console with errors
feat(DB): clean extra files

* chore(helm): Adding cron jobs

* change(ui) - set changed flag to false after the metric delete to avoid prompt

* chore(helm): enbaling cron only for ee

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(api): autocomplete remove console

* change(ui) - removed Console filter type

* fix(ui) - timeline position

* fix(helm): RFC naming

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui): let user change project in dashboards and select default dashboard

* chore(helm): update registry url

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(DB): return pages_count to DB

* fix(ui) - account settings opt out checkbox

* fix(ui): fix modal width

* fix(ui) - explore circle bg

* fix(ui) - user name overlap

* fix(ui) - empty dashboards create button

* fix(ui): fix timeline position cursor for safari

* fix(ui) - custom metrics errors modal url reset on close

* fix(ui) - onboarding check for siteId

* change(ui) - tracker version

* Update local_deploy.sh

* fix(ui) - drilldown timestamp

* fix(tracker): fix deps for assist

* fix(tracker): update peerjs library

* fix(tracker): update assist v

* fix(tracker): fix type error

* fix(backend): no missing resource relying on resource zero-timing

* Update tracker to v3.5.15

* chore(helm): Adding CSP override variable.

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(backend): added pem file support for kafka ssl setup

* feat(backend): added useBatch setup for kafka producer

* ci(backend): set verbose logging

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(backend): using setKey instead of direct writes

* ci(backend): fix error code

* ci(deploy): Updating the image registry

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(api): changed get user id alias

* ci(frontent): removing depricated steps

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* ci(fix): variable replace

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* ci(helm): creating image image_override

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui): fix timezone settings

* Added failover mechanism for storage service (#576)

* fix(ui): fix typescript config to remove array iterator error

* fix(ui): refactor timezone settings store/comp

* feat(snippet): opensource snippet

* feat(assist): support multiple IPs

* fix(ui): fix type errors in select /timezones fix

* feat(backend): set size of first part of sessions at 500kb

* change(ui) - removed logs

* fix(ui) - custom metric errors reset url on modal close

* feat(DB): no funnel migration

* fix(ui): fix screensize bug

* feat(DB): migrate super old funnels support

* changed db-migration workflow

Co-authored-by: Shekar Siri <sshekarsiri@gmail.com>
Co-authored-by: sylenien <nikita@openreplay.com>
Co-authored-by: Alex Kaminskii <alex@openreplay.com>
Co-authored-by: Alexander <zavorotynskiy@pm.me>
Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
Co-authored-by: Mehdi Osman <estradino@users.noreply.github.com>
Co-authored-by: Alexander <alexander@openreplay.com>
Co-authored-by: Rajesh Rajendran <rjshrjndrn@users.noreply.github.com>
Co-authored-by: Delirium <sylenien@gmail.com>
2022-07-07 18:44:43 +02:00