Commit graph

10 commits

Author SHA1 Message Date
MauricioGarciaS
5a5df5f218
fix(redshift-connector): Solved issue while inserting from s3 to redshift (events table) (#1543) 2023-10-24 14:06:26 +02:00
MauricioGarciaS
852ce7b324
fix(redshift-connector): Redshift connector update (#1538)
* Updated sql events table

* Updated and fixed redshift connector python
2023-10-23 17:21:05 +02:00
MauricioGarciaS
a1e12ed276
fix(connectors): fixed issue while inserting string with quotation mark (#1410) 2023-07-12 15:01:06 +02:00
MauricioGarciaS
c9b4644350
style(connectors): Added query log (#1409)
* style(connectors): Added query log

* fix(connectors): changed info log to error log
2023-07-12 12:53:27 +02:00
MauricioGarciaS
ecbc42a03f
fix(connectors): Updated sql to varchar 8000 (#1402)
* fix(connectors): Changed sql from varchar to varchar 8000

* fix(connectors): changed cropping to size 8000 for str in worker task
2023-07-11 13:59:46 +02:00
MauricioGarciaS
b9b187d684
Redshift pool (#1397)
* fix(connectors): added try method for redshift replace in fill

* fix(connectors): adding userId only if not empty string

* fix(connectors): Added fix as a worker
2023-07-10 14:29:40 +02:00
MauricioGarciaS
4cf50b3066
fix(connector): Redshift pool fixes (#1393)
* fix(connector): set env variable for replace and set it to one minute default

* style(connector): different log for redshift and pg

* style(connector): different log for redshift select and replace

* fix(connector): replacing from oldest to newest to avoid blocking

* fix(connector): empty string changed to NN and fixed str issue for cron job

* Changing methods in connector

* fix(connectors): solved issues when replacing with null, reduced number of queries to redshift

* fix(connectors): fixed save method

* fix(connectors): fixed issue while saving event object
2023-07-07 15:33:22 +02:00
MauricioGarciaS
3479060d02
style(connectors): different prints for each script (fill, worker) (#1388) 2023-06-30 12:41:36 +02:00
Rajesh Rajendran
53081a9822
Improvements (#1387)
* fix for memory cleansing and added setup files for msgcodec build

* Kafka reader now calling only once to PG to verify sessionIds, updated fill missing user_ids script

* fix(connector): small error while reading from pg

---------

Co-authored-by: MauricioGarciaS <47052044+MauricioGarciaS@users.noreply.github.com>
2023-06-30 12:04:23 +02:00
MauricioGarciaS
91921d622e
fix(connectors): Created module in Cython to speed up message decoding (#1338)
* Updated pg connector

* fix(player): fix first 8 byte checker

* fix(player): fix commit conflict

* Added pylint

* Removed pylint for incompatible license

* change(ui): check for sessions records state

* Patch/api v1.12.0 (#1299)

* fix(chalice): include metadata in sessions exp search

* fix(chalice): fixed sessions exp search wrong col name

* fix(chalice): removed cookies

* fix(chalice): changed base image to support SSO/xmlsec

* fix(chalice): changed Dockerfile to support SSO/xmlsec

* fix(chalice): changed Dockerfile to support SSO/xmlsec

(cherry picked from commit 4b8cf9742c)

* fix(ui): project fallback to recorded variable

* Patch/api v1.12.0 (#1301)

* fix(chalice): changed base image to support SSO/xmlsec

* fix(chalice): fixed exp search null metadata

(cherry picked from commit ab000751d2)

* change(ui): assist no content message styles and icons

* change(ui): revert menu disable

* fix(connector): Added method to save state in s3 for redshift if sigterm arise

* Rewriting python code in cython

* Added pyx module for messages

* Auto create pyx files

* Updated and fixed msgcodec.pyx

* Added new module to connector code

* Updated kafka lib for base image

* cleaned Docker and updated base image version for pandas

* cleaned prints

* Added code to fetch data from db and add it into redshift

* Updated consumer reading method. Async multithreading over sessionId

* Added split for country (Country,State,City)

* Fixed decoding issue for uint

* Created service able to fix data from redshift by reading from db

* Handle when process ended because of lost connection to pg, country set to country value only
2023-06-23 14:49:39 +02:00