* feat(db): added CH support to db service
* feat(db): removed license check for CH client
* feat(db): removed fts integration
* feat(clickhouse): added config instead of direct env parsing
* feat(clickhouse): removed prev extraHandlers
* feat(clickhouse): an unified approach for data insertion to dbs
* feat(clickhouse): removed unused imports
* feat(server): moved an http server object into a pkg subdir to be reusable for http, spots, and integrations
* feat(web): isolated web module (server, router, middleware, utils) used in spots and new integrations
* feat(web): removed possible panic
* feat(web): split all handlers from http service into different packages for better management.
* feat(web): changed router's method signature
* feat(web): added missing handlers interface
* feat(web): added health middleware to remove unnecessary checks
* feat(web): customizable middleware set for web servers
* feat(web): simplified the handler's structure
* feat(web): created an unified server.Run method for all web services (http, spot, integrations)
* feat(web): fixed a json size limit issue
* feat(web): removed Keys and PG connection from router
* feat(web): simplified integration's main file
* feat(web): simplified spot's main file
* feat(web): simplified http's main file (builder)
* feat(web): refactored audit trail functionality
* feat(web): added ee version of audit trail
* feat(web): added ee version of conditions module
* feat(web): moved ee version of some web session structs
* feat(web): new format of web metrics
* feat(web): added new web metrics to all handlers
* feat(web): added justExpired feature to web ingest handler
* feat(web): added small integrations improvements
* feat(integrations): new version of backend integrations
* feat(integrations): added ingress rule
* feat(integrations): fixed a port number
* feat(integrations): enabled ingress in values.yaml
* feat(integrations): added startup log
* feat(integrations): added extra logger for 3 of 4 backend logs integrations.
* feat(integrations): removed a logs loop call
* feat(integrations): fixed a table name
* feat(integrations): disabled extra logger
* feat(integrations): made extra logger as an option
* feat(integrations): changed contentType for logs file
* feat(integrations): bug fix
* feat(integrations): struct/string config support for datadog provider
* feat(integrations): map config support for datadog provider
* feat(integrations): removed unnecessary transformation
* feat(integrations): fixed datadog and sentry response format
* feat(integrations): added correct creds parser for sentry provider
* feat(integrations): removed unnecessary return statement
* feat(integrations): added correct creds parser for elastic search
* feat(integrations): changed elastic to elasticsearch
* feat(integrations): added correct creds parser for dynatrace
* feat(integrations): fixed an issue in query request for elasticsearch provider
* feat(integrations): made extra logger configurable by env var
* feat(integrations): removed debug logs
* feat(spot): first version to test http endpoints
* fix(helm): changed nginx path prefix
* fix(spots): added missing BUCKET_NAME env var
* fix(spots): added services init check
* feat(spots): removed geo module
* feat(spots): removed uaparser
* feat(spots): added more detailed authorization error log
* feat(spots): changed the authorization middleware
* feat(spots): extended http body size limit to 128kb
* feat(spots): added s3 error log
* feat(spots): added new handler for uploaded event
* feat(backend): small api changes in spot service
* feat(backend): rewrote request parameters grabber for getSpot handler
* feat(backend): added tenantID to auth struct
* feat(backend): added pre-signed download urls for preview, mob et video files
* feat(backend): added user's email to spots table, and getSpot responses
* feat(backend): returning spotID as a string
* feat(spot): added transcoder pipeline
* fix(spot): return spotID as a string
* feat(spot): added volume mount to spot service
* feat(spot): fixed volume mounting
* feat(spot): helm fix
* feat(spot): helm another fix
* fix(spot): correct video.webm path
* fix(spot): correct pre-signed url for download original video
* feat(spot): added PATCH and DELETE methods to CORS
* feat(spot): use string format for spotIDs in delete method
* feat(spot): added public key implemented
* fix(spot): correct public-key parser
* fix(spot): fixed query params issue + user's tenantID
* fix(spot): use 1 as a default tenant
* feat(spot): added correct total spots calculation
* fix(spot): fixed offset calculation
* feat(spot): added extra check in auth method
* fix(spot): removed / from video file name
* fix(spot): devided codec flag into 2 parts
* feat(spot): use fixed tenantID = 1 for oss users
* feat(spot): return 404 for public key not found issue
* feat(spots): added spots folder to minio path rule
* feat(spot): added spot video streaming support
* fix(spot): fixed an sql request for spot streams
* feat(spot): return playlist file in getSpot responce
* feat(spot): try to use aac audio codec
* feat(spot): added permissions support (oss/ee)
* feat(spot): added authorizer method
* feat(spot): added license check
* feat(spot): added spot preview for get response
* fix(spot): fixed a problem with permissions
* feat(spot): added crop feature
* feat(spot): upload cropped video back to s3
* feat(spot): manage expired modified playlist file
* feat(backend): hack with video formats
* feat(backend): removed space
* feat(spot): req tracing
* feat(spot): manual method's name mapping
* feat(spot): added a second method to public key auth support
* feat(spot): metrics
* feat(spot): added rate limiter per user
* feat(spot): added ping endpoint for spot jwt token check
* feat(spot): getStatus endpoint
* feat(spot): added missing import
* feat(spot): transcoding issue fix
* feat(spot): temp remove tasks
* feat(spot): better error log message
* feat(spot): set default jwt_secret value
* feat(spot): debug auth
* feat(spot): 2 diff jwt tokens support
* feat(spot): pg tasks with process status
* feat(spot): more logs
* feat(spot): improved defer for GetTask method
* feat(spot): keep only failed tasks
* feat(spot): removing temp dir with spot files
* feat(spot): added several workers for transcoding module
* feat(spot): fixed spot path for temp video files
* feat(spot): use custom statusWriter to track response code in middleware
* feat(spot): added body and parameter parser for auditrail feature
* feat(spot): fixed IsAuth method signature
* feat(spot): fixed ee service builder
* feat(spot): added import
* feat(spot): fix data type for payload and parameters jsonb fields
* feat(spot): typo fix
* feat(spot): moved out consts
* feat(spot): new table's name
* feat(spot): added missing imports in go.mod
* feat(spot): added a check for the number of comments (20 by default)
* feat(api): added second pre-signed urls for canvases
* feat(api): fixed old key format
* feat(backend): draft version for new approach for canvases
* feat(deploy): fixed some issues
* feat(backend): debug logs for bash command
* feat(backend): removed wrong parameter from bash pipeline
* feat(backend): removed canvas-maker service + small refactoring
* feat(video-storage): added zstd library to the machine
* feat(video-storage): added new method to pack screenshots into compressed tar arch
* feat(video-storage): try to split command into 2
* feat(video-storage): try a new approad to avoid no file error
* feat(api): added support to a new pre-signed url for screenshots archive
* feat(video-storage): fixed an issue in extension check
* feat(video-storage): correct file name
* feat(backend): removed video-storage and splitted logic
* feat(backend): removed video-storage from helm charts
* feat(backend): split canvas and screenshot handlers
* feat(canvas): clean up canvas-handler
* feat(api): changed mobile replay url (screenshots instead of video)
* feat(backend): removed msg.SessID() call
* feat(backend): clean up code in imagestorage main
* feat(backend): added new trigger topic for mobile video replay maker
* feat(backend): mobile video maker with mix list support
* feat(backend): fixed panic
* feat(backend): removed commented code
* feat(backend): added new topic for canvasToVideo communication, moved some logic
* feat(backend): enabled canvas recording
* feat(backend): fixed canvas service main logic
* feat(backend): fixed sessionEnd detector
* feat(backend): send canvas mix lists instead of just a sessID
* feat(backend): enabled canvas recording
* feat(backend): removed old logs from video-storage
* feat(backend): default low setting for canvas recording
* feat(http): added new parameters to start response and new endpoint for canvas screenshorts
* fix(http): added new topic to dockerfile
* feat(http): try different multipart parser
* feat(image-storage): reused the same workflow for canvas topic handler
* feat(video-storage): new canvas parser and ffmpeg script
* feat(video-storage): use correct replay name for canvas
* feat(backend): added new message (CanvasNode)
* feat(backend): add canvas record events to db
* feat(backend): implemented missing new method for azure
---------
Co-authored-by: Alexander <zavorotynskiy@pm.me>
* feat(backend): added ux-testing support
* feat(backend): added ux-testing module
* feat(http): added bucket name for http service
* feat(backend): fixed small typos in http router
---------
Co-authored-by: Alexander <zavorotynskiy@pm.me>
* feat(backend): handlers for mobile messages
* feat(backend): new service template
* feat(backend): save mobile session start and send batches to kafka
* feat(backend): added mobile support to sink, ender and storage
* helm(videostorage): added helm chart for a new service videostorage
* fix(backend): added pointer to streing for userBrowser (because it's null for mobile sessions)
* feat(backend): added MsgIOSBatchMeta handler to message iterator's logic
* feat(backend): added ios ts parser to ender
* feat(backend): enabled producing batch of messages to queue
* feat(backend): removed iosstart from mob files
* feat(backend): added new ios message types
* feat(backend): added iosStart and iosEnd
* fix(backend): fixed log issue
* feat(backend): send tar.gz archives to special queue topic
* feat(backend): read raw archives from kafka
* fix(backend): added missing file
* fix(backend): removed the second file reading
* fix(backend): fixed wrong queue topic name
* fix(backend): fixed mobile trigger topic name
* feat(backend): added tar.gz extractor and iOSSessionEnd handler
* feat(backend): debug logs on message uploading
* fix(backend): added raw-images topic consumption
* feat(backend): now sink send iosSessionEnd to video-storage
* feat(backend): added dir creator for new sessions
* feat(backend): added correct command to execute
* feat(backend): added overwrite option
* feat(backend): added s3 uploader for video session replays
* feat(backend): new consumer group for mobile sessions
* feat(backend): debug logs for uploader
* feat(backend): removed unused log
* feat(backend): fixed s3 key for video replays
* feat(backend): removed debug code
* feat(backend): fixed video-storage message filter
* fix(backend): added mobileSessionEnd to SessionEnd converter
* feat(backend): added first version if db support for mobile events
* fix(backend): added swipe events to mob file
* feat(backend): added swipe event to pg
* feat(backend): split logic into 2 services: image-storage and video-storage
* feat(backend): added helm chart for image-storage service
* fix(backend): fixed table name for mobile taps
* feat(backend): added metadata handler for mobile message parser + fix message filters
* feat(backend): added iosRawTopic to DB message consumer
* fix(backend): removed value from mobile inputs
* feat(backend): removed debug log from iterator
* feat(backend): added new apple devices to iOS device parser
* fix(backend): added real projectID instead of 0
* feat(backend): extended a list of simulators for device detector
* feat(backend): updated networkCall mobile message
* fix(backend): added new way to define is network call successed or not
* feat(backend): added timezone support for mobile start request
* feat(backend): added 2 mobile events Input and Click to mob file
* feat(backend): refactored image storage
* feat(backend): video storage with 2 workers
* feat(backend): added project's platform support
* feat(backend): added memory size field for mobile start request
* feat(backend): changed video preset for ultrafast
* feat(backend): added debug log to http /late handler
* feat(backend): added debug log to db service for iosCrash messages
* feat(backend): added tapRage event handler to heuristics
* feat(backend): changed table and field names for ios crashes
* feat(backend): added payload for tapRage events
* feat(backend): added TapRage events insert to DB
* feat(backend): added fps value to /mobile/start response
* feat(backend): added image quality parameter to /mobile/start response
* feat(backend): added ScreenLeave handler
* feat(backend): removed screenEnter and screenLeave events, added new viewComponent event
---------
Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
* feat(backend): removed legacy InputEvent builder from heuristics service
* feat(backend): removed InputEvent handler from DB service
* feat(backend): removed InputEvent from ClickHouse connector
* feat(backend): added extra env variable for brotli compression algorithm
* feat(backend): redshift connector draft
* fix(backend): fixed memory leak, empty string ddos
* feat(backend): draft for sessions part
* feat(backend): session handler
* fix(backend): fixed wrong columns list in sessionToCSV convertor
* feat(backend): load session info from db/cache if there is nothing in memory when sessionEnd event recieved
* feat(backend): added filters for connector
* feat(backend): memory leak fix + extra cache for sessions
* feat(backend): moved table names as an env variable
* fix(backend): added timeout for last session messages to avoid memory leak
* fix(backend): fixed last memory leak
* feat(backend): moved redshift connector to ee folder
* feat(backend): moved sql requests related to sessions table to one place
* feat(backend): refactoring in db.Saver handler
* feat(backend): hude refactoring in db/postgres module
* fix(backend): workable feature flags
* fix(backend): workable integrations
* fix(backend): workable sessions and projects modules
* fix(backend): added missed projects module to sessions
* feat(backend): renaming
* feat(backend): moved session struct to sessions module and split methods into interface, cache and storage levels
* feat(backend): moved project struct to projects module
* feat(backend): added projects model
* feat(backend): implemented new in memory cache for sessions and projects
* feat(backend): implemented new cache in projects
* feat(backend): there are 2 methods in cache module now: Get() and GetAndRefresh()
* feat(backend): added cache update operations
* fix(backend): fixed import cycle
* fix(backend): fixed panic in db message handler
* fix(backend): fixed panic in projects module
* fix(backend): fixed panic in sessions.GetDuration
* feat(backend): added direct call to get session duration if session is already in cache
* feat(backend): used pg pool everywhere except db service
* fix(backend): added missing part after rebase
* fix(backend): removed old sessions file
* feat(backend): added refactored redis client with produce/consume options
* feat(backend): added cache layer for projects
* fix(backend): added missing redis config
* fix(backend): added missing method for producer
* feat(backend): cache integration for sessions
* feat(backend): temporary method to get session directly from db
* feat(backend): adapt EE version of message handler
* fix(backend): fixed issue in fts realisation
* fix(backend): added redis cache to sessions module
* fix(backend): set 0 duration or hesitation time for inputs without focus event
* feat(backend): added cache for session updates and failover mechanism for batch.Insert() operation
* feat(backend): debug log
* feat(backend): more debug log
* feat(backend): removed debug log
* fix(backend): fixed an issue of tracking input events with empty label
* fix(backend): disabled debug log in projects cache
* fix(backend): renamed session updater
* fix(backend): fixed closed pool issue in DB service
* fix(backend): fixed dead lock in db Stop() method
* fix(backend): fixed panic in heuristics service
* feat(backend): enabled redis cache in projects
* feat(backend): clear cache on each update operation
* feat(backend): fully integrated cache layer with auto switch
* feat(backend): small refactoring in session updates
* fix(backend): fixed wrong events counter issue
* feat(backend): enabled full cache support in ender and http services
* fix(backend/ee): added missed import
* feat(backend): added second cache layer for db to speed up the service
* feat(backend): disable redis cache
* feat(backend): moved redis cache to ee
* fix(backend): fixed issue with wrong expiration timeout for projects in cache layer
* fix(backend): removed direct call db.GetProject()
* feat(backend): set default PROJECT_EXPIRATION to 10 minutes
* feat(backend): implemented azure blob storage support
* feat(backend): added azure implementation to backend services
* feat(backend): added azure blob storage support to chalice service
* fix(backend): removed prev version of s3
* feat(backend): moved azure support to ee subfolder
* feat(backend): prepared ee code for new utils.objects package
* feat(backend): added missed modules to go.mod
* feat(backend): added missed modules to go.sum
* feat(backend): go mod tidy
* feat(backend): temporary made s3 vars are not required
* feat(backend): added azure lib to ee chalice
* feat(api): changed azure env var name
* feat(api): added new object store and extra methods to chalice ee
* fix(api): added azure blob lib to alerts
* fix(api): fixed incorrect call in sessions_devtool
* fix(crons): added azure blob storage library to requirements list
* chore(build): Error message for not providing flag.
Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>
* feat(backend): removed ios headers and object store for ios messages
* feat(backend): object config refactoring
* chore(helm): Update BUCKET_NAME
* fix(backend): removed object storage usage in http
* feat(backend): added debug logs to azure upload method
* fix(backend): fixed empty folder issue
* fix(backend): removed extra debug log
* chore(helm): Adding global variables for crons
* chore(helm): Remove clickhouse resource limit
Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>
* fix(backend): removed assets debug log
* feat(api): use ABC class instead of empty interface
* feat(api): renamed helpers to generators
* feat(api): changed prep/clean dev scripts
* feat(api): changed name obj_store -> StorageClient
* feat(api): some changes after code review
* fix(api): removed unnecesery packages in oss api
* feat(backend): moved azure implementation to ee folder
---------
Signed-off-by: rjshrjndrn <rjshrjndrn@gmail.com>
Co-authored-by: rjshrjndrn <rjshrjndrn@gmail.com>
* feat(backend): added a mechanism to control memory consumption
* feat(backend): extra log for system allocation
* feat(backend): implemented new memory manager in db and heuristics service
