Commit graph

9 commits

Author SHA1 Message Date
Alexander
45c956c489
Json logs format (#1952)
* feat(backend): try a new approach for logs formatting (http)

* feat(backend): added logger module

* feat(backend): added project/session info to /i endpoint

* feat(backend): found a solution for correct caller information

* feat(backend): finished logs for http handlers

* feat(backend): finished logs for mobile http handlers

* feat(backend): finished ender

* feat(backend): finished assets

* feat(backend): finished heuristics

* feat(backend): finished image-storage

* feat(backend): finished sink

* feat(backend): finished storage

* feat(backend): formatted logs in all services

* feat(backend): finished foss part

* feat(backend): added missed foss part

* feat(backend): fixed panic in memory manager and sink service

* feat(backend): connectors
2024-03-14 12:51:14 +01:00
Alexander
d22e4e138e
Backend for mobile tracker support (#1456)
* feat(backend): handlers for mobile messages

* feat(backend): new service template

* feat(backend): save mobile session start and send batches to kafka

* feat(backend): added mobile support to sink, ender and storage

* helm(videostorage): added helm chart for a new service videostorage

* fix(backend): added pointer to streing for userBrowser (because it's null for mobile sessions)

* feat(backend): added MsgIOSBatchMeta handler to message iterator's logic

* feat(backend): added ios ts parser to ender

* feat(backend): enabled producing batch of messages to queue

* feat(backend): removed iosstart from mob files

* feat(backend): added new ios message types

* feat(backend): added iosStart and iosEnd

* fix(backend): fixed log issue

* feat(backend): send tar.gz archives to special queue topic

* feat(backend): read raw archives from kafka

* fix(backend): added missing file

* fix(backend): removed the second file reading

* fix(backend): fixed wrong queue topic name

* fix(backend): fixed mobile trigger topic name

* feat(backend): added tar.gz extractor and iOSSessionEnd handler

* feat(backend): debug logs on message uploading

* fix(backend): added raw-images topic consumption

* feat(backend): now sink send iosSessionEnd to video-storage

* feat(backend): added dir creator for new sessions

* feat(backend): added correct command to execute

* feat(backend): added overwrite option

* feat(backend): added s3 uploader for video session replays

* feat(backend): new consumer group for mobile sessions

* feat(backend): debug logs for uploader

* feat(backend): removed unused log

* feat(backend): fixed s3 key for video replays

* feat(backend): removed debug code

* feat(backend): fixed video-storage message filter

* fix(backend): added mobileSessionEnd to SessionEnd converter

* feat(backend): added first version if db support for mobile events

* fix(backend): added swipe events to mob file

* feat(backend): added swipe event to pg

* feat(backend): split logic into 2 services: image-storage and video-storage

* feat(backend): added helm chart for image-storage service

* fix(backend): fixed table name for mobile taps

* feat(backend): added metadata handler for mobile message parser + fix message filters

* feat(backend): added iosRawTopic to DB message consumer

* fix(backend): removed value from mobile inputs

* feat(backend): removed debug log from iterator

* feat(backend): added new apple devices to iOS device parser

* fix(backend): added real projectID instead of 0

* feat(backend): extended a list of simulators for device detector

* feat(backend): updated networkCall mobile message

* fix(backend): added new way to define is network call successed or not

* feat(backend): added timezone support for mobile start request

* feat(backend): added 2 mobile events Input and Click to mob file

* feat(backend): refactored image storage

* feat(backend): video storage with 2 workers

* feat(backend): added project's platform support

* feat(backend): added memory size field for mobile start request

* feat(backend): changed video preset for ultrafast

* feat(backend): added debug log to http /late handler

* feat(backend): added debug log to db service for iosCrash messages

* feat(backend): added tapRage event handler to heuristics

* feat(backend): changed table and field names for ios crashes

* feat(backend): added payload for tapRage events

* feat(backend): added TapRage events insert to DB

* feat(backend): added fps value to /mobile/start response

* feat(backend): added image quality parameter to /mobile/start response

* feat(backend): added ScreenLeave handler

* feat(backend): removed screenEnter and screenLeave events, added new viewComponent event

---------

Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-10-09 15:02:20 +02:00
Alexander
4b8f3bee25
Sessions refactoring (#1371)
* feat(backend): moved sql requests related to sessions table to one place

* feat(backend): refactoring in db.Saver handler

* feat(backend): hude refactoring in db/postgres module

* fix(backend): workable feature flags

* fix(backend): workable integrations

* fix(backend): workable sessions and projects modules

* fix(backend): added missed projects module to sessions

* feat(backend): renaming

* feat(backend): moved session struct to sessions module and split methods into interface, cache and storage levels

* feat(backend): moved project struct to projects module

* feat(backend): added projects model

* feat(backend): implemented new in memory cache for sessions and projects

* feat(backend): implemented new cache in projects

* feat(backend): there are 2 methods in cache module now: Get() and GetAndRefresh()

* feat(backend): added cache update operations

* fix(backend): fixed import cycle

* fix(backend): fixed panic in db message handler

* fix(backend): fixed panic in projects module

* fix(backend): fixed panic in sessions.GetDuration

* feat(backend): added direct call to get session duration if session is already in cache

* feat(backend): used pg pool everywhere except db service

* fix(backend): added missing part after rebase

* fix(backend): removed old sessions file

* feat(backend): added refactored redis client with produce/consume options

* feat(backend): added cache layer for projects

* fix(backend): added missing redis config

* fix(backend): added missing method for producer

* feat(backend): cache integration for sessions

* feat(backend): temporary method to get session directly from db

* feat(backend): adapt EE version of message handler

* fix(backend): fixed issue in fts realisation

* fix(backend): added redis cache to sessions module

* fix(backend): set 0 duration or hesitation time for inputs without focus event

* feat(backend): added cache for session updates and failover mechanism for batch.Insert() operation

* feat(backend): debug log

* feat(backend): more debug log

* feat(backend): removed debug log

* fix(backend): fixed an issue of tracking input events with empty label

* fix(backend): disabled debug log in projects cache

* fix(backend): renamed session updater

* fix(backend): fixed closed pool issue in DB service

* fix(backend): fixed dead lock in db Stop() method

* fix(backend): fixed panic in heuristics service

* feat(backend): enabled redis cache in projects

* feat(backend): clear cache on each update operation

* feat(backend): fully integrated cache layer with auto switch

* feat(backend): small refactoring in session updates

* fix(backend): fixed wrong events counter issue

* feat(backend): enabled full cache support in ender and http services

* fix(backend/ee): added missed import

* feat(backend): added second cache layer for db to speed up the service

* feat(backend): disable redis cache

* feat(backend): moved redis cache to ee
2023-07-06 10:55:43 +02:00
Alexander
8e58c367dd
Fixed issue with wrong expiration timeout for projects (#1300)
* fix(backend): fixed issue with wrong expiration timeout for projects in cache layer

* fix(backend): removed direct call db.GetProject()

* feat(backend): set default PROJECT_EXPIRATION to 10 minutes
2023-06-08 18:08:53 +02:00
Kraiem Taha Yassine
767d2f6f0c
V1.8.2 workers app name (#772)
* feat(backend): added application name to postgres connection

Co-authored-by: Alexander Zavorotynskiy <zavorotynskiy@pm.me>
2023-01-23 09:08:14 +01:00
Alexander
bfe14db353
[DB] added last quickwit changes (#950)
* feat(backend): added last quickwit changes
2023-01-20 16:18:00 +01:00
Alexander
09adccf4be
feat(backend): added env variable for enabling pprof (#880) 2022-12-13 17:48:00 +01:00
Alexander
b9c30e256d
FTS support (quickwit) (#703)
* feat(backend): sending fetch, page and graphQL events to quickwit topic
2022-08-30 14:13:55 +02:00
Kraiem Taha Yassine
16c70044fb
Chore(release): v1.7.0 (#578)
* change(ui) - redirect to the landing url on SSO login

* fix(ui): fix share popup styles

* change(ui) - non admin user preference restrictions

* fix(ui) - redirect fix

* change(ui) - show installation btn without mouse hover

* feat(api): api-v1 handle wrong projectKey
feat(api): api-v1 get live sessions

* change(ui) - show role edit on hover

* change(ui) - audit trail count with comma

* fix(ui) - audit trail date range custom picker alignment

* change(ui) - show a message when mob file not found

* feat(api): api-v1 fixed search live sessions

* feat(api): api-v1 handle wrong projectKey

* feat(api): fixed assist error response

* fix(tracker): check node scrolls only on start

* fixup! fix(tracker): check node scrolls only on start

* feat(ui/player): scroll view in click map

* feat(ui/player): rm unused check

* New configuration module (#558)

* ci(dbmigrate): Create db migrate when there is change

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui): fix login error/button margins

* fix(ui) - checkbox click

* fix(ui) - search rename and save fixes

* change(ui) - text changes

* fix(ui) - button text nowrap

* fix(ui): fix slowestdomains widget height

* change(ui) - ignore clicks while annotating

* change(ui) - if block with braces

* change(ui) - capitalize first letter in breadcrumb

* feat(db): remove errors from permissions
feat(api): remove errors from permissions

* feat(api): changed reset password response

* fix(ui) - assist active tab list, broken after with new api changes (pagination)

* fix(ui) - assist active tab list, broken after with new api changes (pagination)

* change(ui) - search compare

* fix(ui): last fixes for 1.7

* fix(ui): fix timeline

* fix(ui): small code fixes

* fix(ui): remove unused

* feat(frontend/assist): show when client tab is inactive + fix reconnection status update

* fix(ui) - visibility settings

* feat(assist): refactored extractSessionInfo
feat(assist): hardcoded session's attributes

* Added snabbdom (JS)

* fix(tracker): version check works with x.x.x-beta versions

* fix(backend): keep the highest user's timestamp instead of the latest message timestamp for correct session duration value

* feat(backend/s3): added file tag RETENTION (#561)

* change(ui) - search optimization and autocomplete improvements

* feat(backend/assets): added new metrics assets_downloaded

* change(ui) - show back the date range in bookmarks since the api is filtering by daterange

* feat(backend-assets): custom headers for cacher requests

* chore(backend): no tidy in dockerfile (local build speed up)

* feat(backend/assets): added proxy support for cacher module

* feat(backend/storage): set retention env variable as not required

* fix(ui): fix jira issues

* ci(helm): use kubectl for deployment

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(tracker):3.5.13: performance improvements for a case of extensive dom

* fix(backend): added missed err var and continue statement

* ci(helm): forcing namespace

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(api): fixed slowest_domains query

* ci(helm): update helm deployment method

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* change(ui) - filter dropdown colros

* fix(ui) - speed index location avg attribute changed to value

* ci(api): enable kubectl apply

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui) - widget y axis label

* feat(api): fixed slowest_domains query

* chore(helm): Adding namespaces to all templates (#565)

* feat(api): assist type-autocomplete

* feat(api): assist global-autocomplete

* feat(sourcemaps): include wasm file in build

* feat(sourcemaps-reader): refactored

* fix(ui): fix data for funnels

* fix(ui): fix all sessions section margin

* fix(ui) - assist loader flag

* fix(ui) - assist loader flag

* fix(ui): fix weird check

* feat(api): autocomplete accept unsupported types

* feat(ui): migrate to yarn v3

* feat(ui): minor fixes for installment

* feat(ui): add typescript plugin to yarn

* chore(helm): Ability to override image registry

* chore(helm): Overriding openreplay docker registry

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui): fix control arrows on firefox

* feat(crons): EE crons

* feat(api): fixed build script

* feat(alerts): fixed build script

* feat(crons): fixed build script

* chore(helm): Updating cron version

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(crons): changes

* chore(helm): optional minio ingress

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(crons): fix build script
feat(alerts): fix build script

* Revert "chore(helm): Updating cron version"

This reverts commit 3ca190ea2f.

* feat(crons): fix build script

* feat(crons): fix Dockerfile

* feat(api): fixed metadata change-case

* change(ui) - remove capitalize for the meta value

* change(ui) - autocomplete improvements with custom textfield

* fix(tracker):3.5.13+:reuse metadata on internal-caused restarts

* fix(tracker-assist):3.5.13:send active:true on start; scroll behavior fix

* change(ui) - filters autocomplete blur on pressing Enter key

* fix(tracker): fix node v to lower

* fix(tracker): fix deps

* fix(tracker): fix deps

* fix(ui) - dashboard modal width

* change(ui) - filter dropdown overflow

* chore(helm): clickhouse reclaim polity to retain

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(tracker): fix engine max v

* fix(ui): load metadata in assist tab for sorting

* fix(ui): rm unneeded api call

* fix(tracker): build script to cjs

* change(ui) - removed sample data

* chore(tracker): remove upper node version limit

* Updating Beacon size

Beacon size should be <= QUEUE_MESSAGE_SIZE_LIMIT

* feat(crons): run 24/7
feat(alerts): support env-file override

* feat(api): changed EE env handler

* fix(ui): fix sessions search modal

* change(ui) - margin for error message

* change(ui) - disable assist sort when there are no meta options to choose

* chore(helm): Adding utilities service namespace

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui) - dashboard date range selection reload, metric not found message

* change(ui) - disable clearsearch in assist when there are no filters\

* feat(api): fixed EE env handler

* chore(helm): Adding migration namespaces

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui) - report logo path

* chore(helm): Removing unnecessary SA

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(api): changed EE env handler

* feat(api): changed EE env handler

* feat(api): changed EE env handler

* feat(api): changed EE env handler

* feat(crons): changed crons

* feat(api): accept wrong metric_id

* feat(crons): changed env handler
feat(api): changed env handler
feat(alerts): changed env handler

* feat(utilities): support old version of nodejs

* feat(crons): changed env handler
feat(api): changed env handler
feat(alerts): changed env handler

* fix(tracker): fix srcset tracking

* chore(build): Adding frontent

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(assist): changed general helper

* feat(assist): changed general helper

* fix(ui): fix widget pagination (#570)

* feat(crons): changed entrypoint

* feat(player): dev-log on skipping message

* fix(tracker): removeNode mutation priority over attributes

* fix(tracker): capture relative img timing;use startsWith instead of substr; codestyle fix

* chore(build): fixing api build script

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* chore(ci): faster deployment

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* change(ui) - assist list show active status

* chore(actions): option to build all/specific services in GH

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui) - slowest domain metric data as per the api changes

* ci(helm): updated variable name

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* ci(backend): cherrypick changes to ee

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(backend): disabled pprof in http service

* fix(ui) - TimeToRender avg value as per the API change

* fix(ui) - ResponseTimeDistribution avg value as per the API change

* fix(ui) - MemoryConsumption avg value as per the API change

* fix(ui) - ResponseTime avg value as per the API change

* fix(ui) - DomBuildTime avg value as per the API change

* fix(ui) - FrameRate avg value as per the API change

* chore(helm): proper default tag

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(backend): removed sensitive information from http logs

* ci(backend): adding default parameter value for workflow dispatch

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(backend): deleted empty file

* fix(actions): creating image source file prior

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(helm): variable substitution

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* change(ui) - project list item installation button text change

* fix(ui) - project create validation

* fix(backend): removed unsafe string logs in http service

* chore(kafka): Adding new topic

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(efs-cron): variable name

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui) - developer tools - hint links

* fix(ui) - session filters - country and platform dropdown values

* chore(helm): updating version

* chore(kafka): Update kafka default message size while provisioning

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(tracker): fix dependency security

* change(ui) - webhook delete confirmation

* change(ui) - assist url to handle when empty

* feat(api): autocomplete replace console with errors
feat(DB): clean extra files

* chore(helm): Adding cron jobs

* change(ui) - set changed flag to false after the metric delete to avoid prompt

* chore(helm): enbaling cron only for ee

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(api): autocomplete remove console

* change(ui) - removed Console filter type

* fix(ui) - timeline position

* fix(helm): RFC naming

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui): let user change project in dashboards and select default dashboard

* chore(helm): update registry url

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(DB): return pages_count to DB

* fix(ui) - account settings opt out checkbox

* fix(ui): fix modal width

* fix(ui) - explore circle bg

* fix(ui) - user name overlap

* fix(ui) - empty dashboards create button

* fix(ui): fix timeline position cursor for safari

* fix(ui) - custom metrics errors modal url reset on close

* fix(ui) - onboarding check for siteId

* change(ui) - tracker version

* Update local_deploy.sh

* fix(ui) - drilldown timestamp

* fix(tracker): fix deps for assist

* fix(tracker): update peerjs library

* fix(tracker): update assist v

* fix(tracker): fix type error

* fix(backend): no missing resource relying on resource zero-timing

* Update tracker to v3.5.15

* chore(helm): Adding CSP override variable.

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(backend): added pem file support for kafka ssl setup

* feat(backend): added useBatch setup for kafka producer

* ci(backend): set verbose logging

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(backend): using setKey instead of direct writes

* ci(backend): fix error code

* ci(deploy): Updating the image registry

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* feat(api): changed get user id alias

* ci(frontent): removing depricated steps

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* ci(fix): variable replace

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* ci(helm): creating image image_override

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* fix(ui): fix timezone settings

* Added failover mechanism for storage service (#576)

* fix(ui): fix typescript config to remove array iterator error

* fix(ui): refactor timezone settings store/comp

* feat(snippet): opensource snippet

* feat(assist): support multiple IPs

* fix(ui): fix type errors in select /timezones fix

* feat(backend): set size of first part of sessions at 500kb

* change(ui) - removed logs

* fix(ui) - custom metric errors reset url on modal close

* feat(DB): no funnel migration

* fix(ui): fix screensize bug

* feat(DB): migrate super old funnels support

* changed db-migration workflow

Co-authored-by: Shekar Siri <sshekarsiri@gmail.com>
Co-authored-by: sylenien <nikita@openreplay.com>
Co-authored-by: Alex Kaminskii <alex@openreplay.com>
Co-authored-by: Alexander <zavorotynskiy@pm.me>
Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
Co-authored-by: Mehdi Osman <estradino@users.noreply.github.com>
Co-authored-by: Alexander <alexander@openreplay.com>
Co-authored-by: Rajesh Rajendran <rjshrjndrn@users.noreply.github.com>
Co-authored-by: Delirium <sylenien@gmail.com>
2022-07-07 18:44:43 +02:00