Commit graph

179 commits

Author SHA1 Message Date
Alexander
2dcbfe2ef9 feat(integrations): removed 'req' from jwt_spot_secret 2025-02-28 15:01:03 +01:00
Alexander
f1614b6626 feat(canvas): removed unnecessary log 2025-02-17 15:21:37 +01:00
Alexander
16182031e1 feat(s3): added the content encoding support 2025-01-27 15:37:04 +01:00
Shekar Siri
566d6c2fdb
feat(analytics): dashboards (#2788) 2024-12-11 09:55:07 +01:00
Alexander
9b75e4502f
ClickHouse support (#2830)
* feat(db): added CH support to db service

* feat(db): removed license check for CH client

* feat(db): removed fts integration

* feat(clickhouse): added config instead of direct env parsing

* feat(clickhouse): removed prev extraHandlers

* feat(clickhouse): an unified approach for data insertion to dbs

* feat(clickhouse): removed unused imports
2024-12-10 12:41:52 +01:00
Alexander
6830c8879f
web module refactoring (#2725)
* feat(server): moved an http server object into a pkg subdir to be reusable for http, spots, and integrations

* feat(web): isolated web module (server, router, middleware, utils) used in spots and new integrations

* feat(web): removed possible panic

* feat(web): split all handlers from http service into different packages for better management.

* feat(web): changed router's method signature

* feat(web): added missing handlers interface

* feat(web): added health middleware to remove unnecessary checks

* feat(web): customizable middleware set for web servers

* feat(web): simplified the handler's structure

* feat(web): created an unified server.Run method for all web services (http, spot, integrations)

* feat(web): fixed a json size limit issue

* feat(web): removed Keys and PG connection from router

* feat(web): simplified integration's main file

* feat(web): simplified spot's main file

* feat(web): simplified http's main file (builder)

* feat(web): refactored audit trail functionality

* feat(web): added ee version of audit trail

* feat(web): added ee version of conditions module

* feat(web): moved ee version of some web session structs

* feat(web): new format of web metrics

* feat(web): added new web metrics to all handlers

* feat(web): added justExpired feature to web ingest handler

* feat(web): added small integrations improvements
2024-11-21 17:48:04 +01:00
Alexander
03c18def0a feat(http): return 401 even after successfully saved batch for JustExpired case 2024-11-15 11:21:18 +01:00
Alexander
eb7e7de463 feat(backend): fix to save the latest message tracker just after the token has been expired 2024-11-15 10:56:58 +01:00
Alexander
30a69893bb
New backend logs integrations (#2717)
* feat(integrations): new version of backend integrations

* feat(integrations): added ingress rule

* feat(integrations): fixed a port number

* feat(integrations): enabled ingress in values.yaml

* feat(integrations): added startup log

* feat(integrations): added extra logger for 3 of 4 backend logs integrations.

* feat(integrations): removed a logs loop call

* feat(integrations): fixed a table name

* feat(integrations): disabled extra logger

* feat(integrations): made extra logger as an option

* feat(integrations): changed contentType for logs file

* feat(integrations): bug fix

* feat(integrations): struct/string config support for datadog provider

* feat(integrations): map config support for datadog provider

* feat(integrations): removed unnecessary transformation

* feat(integrations): fixed datadog and sentry response format

* feat(integrations): added correct creds parser for sentry provider

* feat(integrations): removed unnecessary return statement

* feat(integrations): added correct creds parser for elastic search

* feat(integrations): changed elastic to elasticsearch

* feat(integrations): added correct creds parser for dynatrace

* feat(integrations): fixed an issue in query request for elasticsearch provider

* feat(integrations): made extra logger configurable by env var

* feat(integrations): removed debug logs
2024-10-31 15:28:38 +01:00
Alexander
a17796e811 feat(backend): removed insert operations for session's resources 2024-10-30 13:46:13 +01:00
Alexander
bbf1dbcc23 feat(http): ff and uxt are disabled by default 2024-09-03 11:41:24 +02:00
Alexander
345f316b27
Spots (#2305)
* feat(spot): first version to test http endpoints

* fix(helm): changed nginx path prefix

* fix(spots): added missing BUCKET_NAME env var

* fix(spots): added services init check

* feat(spots): removed geo module

* feat(spots): removed uaparser

* feat(spots): added more detailed authorization error log

* feat(spots): changed the authorization middleware

* feat(spots): extended http body size limit to 128kb

* feat(spots): added s3 error log

* feat(spots): added new handler for uploaded event

* feat(backend): small api changes in spot service

* feat(backend): rewrote request parameters grabber for getSpot handler

* feat(backend): added tenantID to auth struct

* feat(backend): added pre-signed download urls for preview, mob et video files

* feat(backend): added user's email to spots table, and getSpot responses

* feat(backend): returning spotID as a string

* feat(spot): added transcoder pipeline

* fix(spot): return spotID as a string

* feat(spot): added volume mount to spot service

* feat(spot): fixed volume mounting

* feat(spot): helm fix

* feat(spot): helm another fix

* fix(spot): correct video.webm path

* fix(spot): correct pre-signed url for download original video

* feat(spot): added PATCH and DELETE methods to CORS

* feat(spot): use string format for spotIDs in delete method

* feat(spot): added public key implemented

* fix(spot): correct public-key parser

* fix(spot): fixed query params issue + user's tenantID

* fix(spot): use 1 as a default tenant

* feat(spot): added correct total spots calculation

* fix(spot): fixed offset calculation

* feat(spot): added extra check in auth method

* fix(spot): removed / from video file name

* fix(spot): devided codec flag into 2 parts

* feat(spot): use fixed tenantID = 1 for oss users

* feat(spot): return 404 for public key not found issue

* feat(spots): added spots folder to minio path rule

* feat(spot): added spot video streaming support

* fix(spot): fixed an sql request for spot streams

* feat(spot): return playlist file in getSpot responce

* feat(spot): try to use aac audio codec

* feat(spot): added permissions support (oss/ee)

* feat(spot): added authorizer method

* feat(spot): added license check

* feat(spot): added spot preview for get response

* fix(spot): fixed a problem with permissions

* feat(spot): added crop feature

* feat(spot): upload cropped video back to s3

* feat(spot): manage expired modified playlist file

* feat(backend): hack with video formats

* feat(backend): removed space

* feat(spot): req tracing

* feat(spot): manual method's name mapping

* feat(spot): added a second method to public key auth support

* feat(spot): metrics

* feat(spot): added rate limiter per user

* feat(spot): added ping endpoint for spot jwt token check

* feat(spot): getStatus endpoint

* feat(spot): added missing import

* feat(spot): transcoding issue fix

* feat(spot): temp remove tasks

* feat(spot): better error log message

* feat(spot): set default jwt_secret value

* feat(spot): debug auth

* feat(spot): 2 diff jwt tokens support

* feat(spot): pg tasks with process status

* feat(spot): more logs

* feat(spot): improved defer for GetTask method

* feat(spot): keep only failed tasks

* feat(spot): removing temp dir with spot files

* feat(spot): added several workers for transcoding module

* feat(spot): fixed spot path for temp video files

* feat(spot): use custom statusWriter to track response code in middleware

* feat(spot): added body and parameter parser for auditrail feature

* feat(spot): fixed IsAuth method signature

* feat(spot): fixed ee service builder

* feat(spot): added import

* feat(spot): fix data type for payload and parameters jsonb fields

* feat(spot): typo fix

* feat(spot): moved out consts

* feat(spot): new table's name

* feat(spot): added missing imports in go.mod

* feat(spot): added a check for the number of comments (20 by default)
2024-08-29 16:08:33 +02:00
Alexander
1637e5b5ec feat(backend): changed mobile autocomplete naming 2024-07-25 10:19:36 +02:00
Alexander
b3d04d76d4
Heatmaps fix (float coordinates) (#2403)
* feat(spot): use float click coordinates instead of ints in PG

* feat(db): added support for float clicks in CH

* feat(db): fix float instead of uint8

* feat(mobile): new naming for mobile autocomplete types
2024-07-18 12:32:44 +02:00
Alexander
75eb6924c1 feat(connector): added s3 batches support for CH connector 2024-07-05 16:16:02 +02:00
Alexander
16fe6674f3 feat(assets): use UA header every get call 2024-07-02 12:20:07 +02:00
Alexander
b92177a151
Assist-only feature (#2214)
* feat(http): small prep for task implementation

* feat(http): added assist-only check to ee

* feat(backend): using a new license check url

* feat(backend): packages upgrade
2024-05-28 09:41:16 +02:00
Alexander
993b6b58ab
feat(backend): added utm data to PG (#2152)
* feat(backend): added utm data to PG

* feat(backend): added utm data to CH

* feat(db): removed nullableString() calls
2024-05-24 08:33:28 +02:00
Alexander
4c96cbcef3 feat(connector): added new events + projectID filter 2024-05-23 12:09:59 +02:00
Alexander
de9ffeb6a9 feat(connector): added s3 connector + small improvements 2024-05-22 15:23:45 +02:00
Alexander
2bf9298836
Mobile renaming (protocol + backend) (#2078)
* feat(backend): renamed ios to mobile

* feat(backend): added missing changes to iterator

* feat(backend): added missing changes to ender

* feat(backend): as usual auto deleted import in ch connector

* feat(backend): updated ee/conditions func calls
2024-04-12 09:30:24 +02:00
Alexander
bdb3f37c56 feat(backend): added es connector 2024-04-09 14:58:47 +02:00
Alexander
486a7115d7 feat(backend): added session's screen width/height to PG 2024-04-08 13:49:29 +02:00
Alexander
daddedebcc feat(backend): fixed an issue with no mobile replays for android tracker 2024-04-05 17:00:45 +02:00
Alexander
54156fc81d feat(backend): moved batch info to query param 2024-04-05 16:46:02 +02:00
Alexander
37276006fb feat(backend): added missing mobile log 2024-04-05 14:54:48 +02:00
Alexander
75ccacf576 feat(backend): added missing sessionID to log in start request + batch info 2024-04-05 09:42:55 +02:00
Alexander
f7744a0c2c
Mob split (#2015)
* feat(backend): try to split mob files by ts

* feat(backend): removed debug code and used worker pool
2024-04-03 17:10:44 +02:00
Alexander
5356d72417 feat(backend): tmp print screen size in logs 2024-04-02 17:43:13 +02:00
Alexander
eb3952666e feat(backend): moved mobile screen size to browser version 2024-04-02 17:35:37 +02:00
Alexander
8c84fd6550 feat(backend): added new fields to mobile start request 2024-03-25 09:14:00 +01:00
Alexander
fa24030f0b feat(backend): added more details to http logs 2024-03-14 15:43:57 +01:00
Alexander
a453256f23 feat(backend): moved log to prevent error code loss 2024-03-14 14:30:46 +01:00
Alexander
45c956c489
Json logs format (#1952)
* feat(backend): try a new approach for logs formatting (http)

* feat(backend): added logger module

* feat(backend): added project/session info to /i endpoint

* feat(backend): found a solution for correct caller information

* feat(backend): finished logs for http handlers

* feat(backend): finished logs for mobile http handlers

* feat(backend): finished ender

* feat(backend): finished assets

* feat(backend): finished heuristics

* feat(backend): finished image-storage

* feat(backend): finished sink

* feat(backend): finished storage

* feat(backend): formatted logs in all services

* feat(backend): finished foss part

* feat(backend): added missed foss part

* feat(backend): fixed panic in memory manager and sink service

* feat(backend): connectors
2024-03-14 12:51:14 +01:00
Alexander
89752b2acc
Canvas archives (#1938)
* feat(api): added second pre-signed urls for canvases

* feat(api): fixed old key format

* feat(backend): draft version for new approach for canvases

* feat(deploy): fixed some issues

* feat(backend): debug logs for bash command

* feat(backend): removed wrong parameter from bash pipeline

* feat(backend): removed canvas-maker service + small refactoring
2024-03-11 17:51:49 +01:00
Alexander
68d4090e15 feat(backend): added projectID to mobile session start responce 2024-03-01 16:42:49 +01:00
Alexander
7c58ece936 feat(backend): added separate config for canvas-handler 2024-02-28 13:21:19 +01:00
Alexander
21c4ebd6e3 feat(backend): upgraded user-agent for assist service 2024-02-27 13:54:45 +01:00
Alexander
36fb1a07f7
Worker pool for image processing services (#1912)
* feat(backend): added worker pool for screenshot handler

* feat(backend): added worker pool to canvas handler

* feat(backend): added worker pool to canvas maker

* feat(backend): added missing method to canvas-maker service
2024-02-27 13:48:27 +01:00
Alexander
a73bf81902 feat(backend): added support for conditional recording feature (mobile sessions) 2024-02-26 15:13:51 +01:00
Alexander
a073ce498d
No ffmpeg solution (#1905)
* feat(video-storage): added zstd library to the machine

* feat(video-storage): added new method to pack screenshots into compressed tar arch

* feat(video-storage): try to split command into 2

* feat(video-storage): try a new approad to avoid no file error

* feat(api): added support to a new pre-signed url for screenshots archive

* feat(video-storage): fixed an issue in extension check

* feat(video-storage): correct file name

* feat(backend): removed video-storage and splitted logic

* feat(backend): removed video-storage from helm charts

* feat(backend): split canvas and screenshot handlers

* feat(canvas): clean up canvas-handler

* feat(api): changed mobile replay url (screenshots instead of video)

* feat(backend): removed msg.SessID() call

* feat(backend): clean up code in imagestorage main
2024-02-26 14:16:43 +01:00
Alexander
88ea925263 feat(backend): added new flag USE_S3_TAGS to disable s3 tags 2024-01-29 16:50:08 +01:00
Eng Zer Jun
76f0f3ff3d
refactor(backend): move from io/ioutil to io and os packages (#1784)
The io/ioutil package has been deprecated as of Go 1.16 [1]. This commit
replaces the existing io/ioutil functions with their new definitions in
io and os packages.

[1]: https://golang.org/doc/go1.16#ioutil
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2024-01-29 16:37:59 +01:00
Alexander
df2ee71bbb feat(backend): fixed issue in start request 2024-01-26 14:25:44 +01:00
Alexander
354f071fb0
Conditions (ee version) (#1858)
* feat(backend): moved conditions to ee folder

* feat(backend): fixed typo
2024-01-26 14:03:29 +01:00
Alexander
db17366aae feat(backend): fixed no mobile sessions issue in CH 2024-01-26 13:46:17 +01:00
Alexander
255f879476
Mobile replay (#1853)
* feat(backend): added new trigger topic for mobile video replay maker

* feat(backend): mobile video maker with mix list support

* feat(backend): fixed panic

* feat(backend): removed commented code
2024-01-25 14:30:40 +01:00
Alexander
acc21fe82d feat(backend): fixed typo in canvas group name 2024-01-23 11:37:47 +01:00
Alexander
78a9146144 feat(backend): added new consumer groups for new canvas pipeline 2024-01-22 17:03:15 +01:00
Alexander
9e3548e9ce
Conditional recording (#1849)
* feat(backend): added mocked endpoint for conditional recordings

* feat(backend): disabled project check

* feat(backend): adapt getConditions endpoint for API implementation

* feat(backend): added condition option to start request

* feat(backend): added missing file

* feat(backend): debug log
2024-01-22 14:52:02 +01:00