Commit graph

84 commits

Author SHA1 Message Date
MauricioGarciaS
ecbc42a03f
fix(connectors): Updated sql to varchar 8000 (#1402)
* fix(connectors): Changed sql from varchar to varchar 8000

* fix(connectors): changed cropping to size 8000 for str in worker task
2023-07-11 13:59:46 +02:00
MauricioGarciaS
b9b187d684
Redshift pool (#1397)
* fix(connectors): added try method for redshift replace in fill

* fix(connectors): adding userId only if not empty string

* fix(connectors): Added fix as a worker
2023-07-10 14:29:40 +02:00
MauricioGarciaS
efd562321e
fix(connectors): added try method for redshift replace in fill (#1396) 2023-07-10 11:10:42 +02:00
MauricioGarciaS
4cf50b3066
fix(connector): Redshift pool fixes (#1393)
* fix(connector): set env variable for replace and set it to one minute default

* style(connector): different log for redshift and pg

* style(connector): different log for redshift select and replace

* fix(connector): replacing from oldest to newest to avoid blocking

* fix(connector): empty string changed to NN and fixed str issue for cron job

* Changing methods in connector

* fix(connectors): solved issues when replacing with null, reduced number of queries to redshift

* fix(connectors): fixed save method

* fix(connectors): fixed issue while saving event object
2023-07-07 15:33:22 +02:00
MauricioGarciaS
3479060d02
style(connectors): different prints for each script (fill, worker) (#1388) 2023-06-30 12:41:36 +02:00
Rajesh Rajendran
53081a9822
Improvements (#1387)
* fix for memory cleansing and added setup files for msgcodec build

* Kafka reader now calling only once to PG to verify sessionIds, updated fill missing user_ids script

* fix(connector): small error while reading from pg

---------

Co-authored-by: MauricioGarciaS <47052044+MauricioGarciaS@users.noreply.github.com>
2023-06-30 12:04:23 +02:00
rjshrjndrn
a5cf104ddc fix(docker): Base image name
Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-06-23 16:07:37 +02:00
MauricioGarciaS
9108dd7f04
fix(connectors): Changed to consumer pool (#1368) 2023-06-23 15:33:43 +02:00
MauricioGarciaS
91921d622e
fix(connectors): Created module in Cython to speed up message decoding (#1338)
* Updated pg connector

* fix(player): fix first 8 byte checker

* fix(player): fix commit conflict

* Added pylint

* Removed pylint for incompatible license

* change(ui): check for sessions records state

* Patch/api v1.12.0 (#1299)

* fix(chalice): include metadata in sessions exp search

* fix(chalice): fixed sessions exp search wrong col name

* fix(chalice): removed cookies

* fix(chalice): changed base image to support SSO/xmlsec

* fix(chalice): changed Dockerfile to support SSO/xmlsec

* fix(chalice): changed Dockerfile to support SSO/xmlsec

(cherry picked from commit 4b8cf9742c)

* fix(ui): project fallback to recorded variable

* Patch/api v1.12.0 (#1301)

* fix(chalice): changed base image to support SSO/xmlsec

* fix(chalice): fixed exp search null metadata

(cherry picked from commit ab000751d2)

* change(ui): assist no content message styles and icons

* change(ui): revert menu disable

* fix(connector): Added method to save state in s3 for redshift if sigterm arise

* Rewriting python code in cython

* Added pyx module for messages

* Auto create pyx files

* Updated and fixed msgcodec.pyx

* Added new module to connector code

* Updated kafka lib for base image

* cleaned Docker and updated base image version for pandas

* cleaned prints

* Added code to fetch data from db and add it into redshift

* Updated consumer reading method. Async multithreading over sessionId

* Added split for country (Country,State,City)

* Fixed decoding issue for uint

* Created service able to fix data from redshift by reading from db

* Handle when process ended because of lost connection to pg, country set to country value only
2023-06-23 14:49:39 +02:00
Kraiem Taha Yassine
678bc33820
Api v1.13.0 release (#1336)
* fix(connectors): upgraded dependencies

* refactor(chalice): changed update account
2023-06-12 16:38:22 +02:00
MauricioGarciaS
1ac262b984
style(Insights): updated API to show error details (#1316)
* style(sessions-insights): Added error message into Error Insights

* Added messages file
2023-06-08 15:46:04 +02:00
Delirium
2ed4bba33e
feat(tracker/ui): support for multi tab sessions (#1236)
* feat(tracker): add support for multi tab sessions

* feat(backend): added support of multitabs

* fix(backend): added support of deprecated batch meta message to pre-decoder

* fix(backend): fixed nil meta issue for TabData messages in sink

* feat(player): add tabmanager

* feat(player): basic tabchange event support

* feat(player): pick tabstate for console panel and timeline

* fix(player): only display tabs that are created

* feat(player): connect performance, xray and events to tab state

* feat(player): merge all tabs data for overview

* feat(backend/tracker): extract tabdata into separate message from batchmeta

* fix(tracker): fix new session check

* fix(backend): remove batchmetadeprecated

* fix(backend): fix switch case

* fix(player): fix for tab message size

* feat(tracker): check for active tabs with broadcast channel

* feat(tracker): prevent multiple messages

* fix(tracker): ignore beacons from same tab, only ask if token isnt present yet, add small delay before start to wait for answer

* feat(player): support new msg struct in assist player

* fix(player): fix some livepl components for multi tab states

* feat(tracker): add option to disable multitab

* feat(tracker): add multitab to assist plugin

* feat(player): back compat for tab id

* fix(ui): fix missing list in controls

* fix(ui): optional list update

* feat(ui): fix visuals for multitab; use window focus event for tabs

* fix(tracker): fix for dying tests (added tabid to writer, refactored other tests)

* feat(ui): update LivePlayerSubHeader.tsx to support tabs

* feat(backend): added tabs support to devtools mob files

* feat(ui): connect state to current tab properly

* feat(backend): added multitab support to assits

* feat(backend): removed data check in agent message

* feat(backend): debug on

* fix(backend): fixed typo in message broadcast

* feat(backend): fixed issue in connect method

* fix(assist): fixed typo

* feat(assist): added more debug logs

* feat(assist): removed one log

* feat(assist): more logs

* feat(assist): use query.peerId

* feat(assist): more logs

* feat(assist): fixed session update

* fix(assist): fixed getSessions

* fix(assist): fixed request_control broadcast

* fix(assist): fixed typo

* fix(assist): added missed line

* fix(assist): fix typo

* feat(tracker): multitab support for assist sessions

* fix(tracker): fix dead tests (tabid prop)

* fix(tracker): fix yaml

* fix(tracker): timers issue

* fix(ui): fix ui E2E tests with magic?

* feat(assist): multitabs support for ee version

* fix(assist): added missed method import

* fix(tracker): fix fix events in assist

* feat(assist): added back compatibility for sessions without tabId

* fix(assist): apply message's top layer structure before broadcast call

* fix(assist): added random tabID for prev version

* fix(assist): added random tabID for prev version (ee)

* feat(assist): added debug logs

* fix(assist): fix typo in sessions_agents_count method

* fix(assist): fixed more typos in copy-pastes

* fix(tracker): fix restart timings

* feat(backend): added tabIDs for some events

* feat(ui): add tab change event to the user steps bar

* Revert "feat(backend): added tabIDs for some events"

This reverts commit 1467ad7f9f.

* feat(ui): revert timeline and xray to grab events from all tabs

* fix(ui): fix typo

---------

Co-authored-by: Alexander Zavorotynskiy <zavorotynskiy@pm.me>
2023-06-07 10:40:32 +02:00
MauricioGarciaS
164232377d
fix(connector): Fixed small issues and added checkpoint method (#1251)
* fix(connector): fixed bug of cache dict size error

* fix(connector): Added method to save state in s3 for redshift if sigterm arise

* fix(connector): Added exit signal handler and checkpoint method

* Added sslmode selection for connection to database, added use_ssl parameter for S3 connection

* fix(connector): Handle error when broken session_id
2023-05-31 11:42:49 +02:00
MauricioGarciaS
1a414d5764
fix(connector): Added checkpoints and sigterm handler (#1234)
* fix(connector): fixed bug of cache dict size error

* fix(connector): Added method to save state in s3 for redshift if sigterm arise

* fix(connector): Added exit signal handler and checkpoint method

* Added sslmode selection for connection to database, added use_ssl parameter for S3 connection
2023-05-09 11:35:50 +02:00
MauricioGarciaS
41c0f19c62
fix(connector): fixed bug of cache dict size error (#1226) 2023-05-04 13:49:10 +02:00
MauricioGarciaS
298288301a
fix(connector): handle db connection drop (#1223)
* Added compatibility with SaaS, added reboot of connection if connection droped

* Small fix
2023-05-02 16:31:20 +02:00
MauricioGarciaS
28182b951e
Redshift connector (#1170)
* Updated dependancies for redshift connector, changed os module for python-decouple module

* Updated service and images

* Updated message protocol, added exception for BatchMetadata when version is 0 (we apply old read method)

* fixed load error from s3 to redshift. null values for string columns are now empty strings ("")

* Added file test consumer_async.py: reads every 3 minutes kafka raw and send task in background to upload to cloud

* Added method to skip messages that are not inserted to cloud

* Added logs into consumer_async. Changed urls and issues in sessions table from list to string

* Split between messages for sessions table and for events table

* Updated redshift tables

* Fixed small issue in query redshift_sessions.sql

* Updated Dockerfiles. Cleaned logs of consumer_async. Updated/Fixed tables. Transformed Nan as NULL for VARCHAR columns

* Added error handler for sql dropped connection

* chore(docker): Optimize docker builds

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* Variables renamed

* Adding compression libraries

* Set default value of count events to 0 (instead of NULL) when event did not occur

* Added support specific  project tracking. Added PG handler to connect to sessions table

* Added method to update values in db connection for sessions ended and restarted

* Removing intelligent file copying

* chore(connector): Build file

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>

* Adding connection pool for pg

* Renaming and optimizing

* Fixed issue of missing information of sessions

---------

Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>
Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
2023-05-02 14:02:57 +02:00
nick-delirium
7156fa0fc4 change(tracker): fix msgs format 2023-03-22 10:00:06 +01:00
nick-delirium
d65e789b5b change(ui): regen messages; wait for player initialization; insert event lists instead of creating new ones 2023-03-22 10:00:06 +01:00
nick-delirium
89ec4b67f1 change(tracker): regen messages; add trans size and isCache to resource timing message, remove x-cached header detection 2023-03-22 10:00:06 +01:00
nick-delirium
2f1b233070 change(tracker/player): create new network msg for cached req 2023-03-22 10:00:06 +01:00
nick-delirium
fbab981614 change(tracker/backend): regenerate messages 2023-03-17 12:17:14 +01:00
nick-delirium
a34d557f7f change(tracker): send percent of removed nodes instead of int 2023-03-17 12:17:14 +01:00
nick-delirium
532d9b82f8 change(tracker): remove dom drop detection on tracker side 2023-03-17 12:17:14 +01:00
nick-delirium
01e4412815 change(tracker): add nodes drop message 2023-03-17 12:17:14 +01:00
nick-delirium
d9ab7b6853 change(tracker): add nodes drop message 2023-03-17 12:17:14 +01:00
Alexander
a3c7681dfa
Merge branch 'dev' into new-frustrations 2023-03-09 15:33:58 +01:00
Alexander
c6aac11cbf
Heuristics refactoring (#987)
* feat(backend): refactored heuristics service

* feat(backend): refactored db service (moved several events to heuristics)
2023-03-09 09:54:12 +01:00
nick-delirium
4a96d87cb7 change(tracker): add mouse thrashing event message 2023-02-17 13:04:00 +01:00
nick-delirium
9628bcdfad change(tracker): add more events to input, fix typo and timestamp calc 2023-02-17 13:04:00 +01:00
nick-delirium
c2501530a0 fix(tracker): capture selection event 2023-02-17 13:04:00 +01:00
nick-delirium
3266f46440 fix(tracker): some events changes... 2023-02-17 13:04:00 +01:00
nick-delirium
018f618e12 change(tracker): add mouse shake and update input hesitation message 2023-02-17 13:03:59 +01:00
Alex Kaminskii
1c89afa4e2 fix(message-protocol;tracker;player): use uint for StringDict keys 2023-02-06 12:30:48 +01:00
Alex Kaminskii
e8d2379943 feat(mobs): new attribute string-dictionary messages 2023-02-06 11:01:02 +01:00
Alex Kaminskii
d310ca7bfe refactor(mobs):update comments, remove deprecated since one release back 2023-02-03 17:10:05 +01:00
MauricioGarciaS
4d442d045c Changed bigquery module version to latest 2023-01-20 15:50:37 +01:00
MauricioGarciaS
f4e7321b86 Changed python version to 3.11 and fixed confluent-kafka import error 2023-01-20 15:19:27 +01:00
MauricioGarciaS
e85933e96d Changed python-kafka to confluent-kafka 2023-01-20 12:25:22 +01:00
Alexander
da7d2038ef
[DB] added url field for click rage issues (#925)
* feat(backend): added url field for click rage issues
2023-01-18 13:38:10 +01:00
Alex Kaminskii
ff80386aff feat(backend,mobs):replace Fetch with NetworkRequest 2022-12-09 15:48:47 +01:00
Alex K
e861bd9641
refactor(backend):remove legacy message wrapping (#854)
* refactor(backend):remove legacy message wrapping & few backend-only messages
2022-12-06 11:54:36 +01:00
MauricioGarciaS
806b67d6ca Updated snowflake modules 2022-11-21 18:17:33 +01:00
MauricioGarciaS
f9a638f02b Updated modules versions for each connector 2022-11-07 17:14:07 +01:00
Alex K
822ca53980
Dev aleksk (#795)
*feat(tracker;backend;frontend/player): focus in elem
*feat(tracker;frontend/player): FontFace load
2022-11-04 11:33:29 +01:00
Alexander
51b6414cb6
feat(backend): added convertor for old sessionEnd message (#786) 2022-10-25 14:50:02 +02:00
Alex K
974afa454b
feat(backend): insert error tags+ (#768)
* feat(backend): transform legacy messages

* refactor(backend/postgres):remove seqIndex transformation

* feat(backend/db): parse & insert errors_tags

* chore(backend): remove building folder after build

* fix(backend/db):remove message types from filter that do not go through kafka 

* fix(backend/assets):use JSException instead of ErrorEvent

* refactor(backend/db):ErrorEvent is no longer a message

* feat(backend/db):insert error tags in clickhouse

* refactor(backend):remove unused LongTask code
2022-10-19 10:31:13 +02:00
Alexander
a166482227
Session mod files encryption (#766)
* feat(backend): added session mod files encryption
2022-10-18 12:50:36 +02:00
Alex Kaminskii
d48a580d62 feat(mobs;backend): negative filtering for mob-file messages (no backend involved in plugin creation) 2022-10-17 10:29:47 +02:00
sylenien
117b1aca96 change(tracker/backend): change msg format for exceptions 2022-10-12 12:44:05 +02:00