55 Commits

Author SHA1 Message Date
Andrey Antukh
db689d151e ♻️ Refactor profile and session handling
- makes the profile access more efficient (replace in-app joins to a
  simple select query on profile table
- add partial support for access-tokens (still missing some RPC methods)
- move router definitions to specific modules and simplify the main http
  module definitions to simple includes
- simplifiy authentication code related to access-tokens and sessions
- normalize db parameters with proper namespaced props
- more work on convert all modules initialization to use proper specs
  with fully-qualified keyword config props
2023-01-18 10:51:58 +01:00
Andrey Antukh
226afe98e0 💄 Change worker logging 2022-12-15 11:33:10 +01:00
Andrey Antukh
d7459db292 🎉 Add task deduplication by label 2022-12-13 23:13:11 +01:00
Andrey Antukh
c0a4b7dc76 Improve worker queue management
and add specific worker instance for webhooks
2022-12-13 16:17:31 +01:00
Andrey Antukh
8bad9d8340 ♻️ Refactor loggers/audit, auth/oidc, and http/clent modules 2022-12-05 08:53:00 +01:00
Andrey Antukh
7f7efc5760 Improve exception formating on backend 2022-12-05 08:51:54 +01:00
Andrey Antukh
c451c7bb9d 🐛 Fix regression on worker task handling 2022-11-30 13:13:11 +01:00
Andrey Antukh
0600b2abe4 ♻️ Make the worker abstraction more scalable
Start using redis for dispatcher to worker communication
and add the ability to start multiple threads to worker
for increase the concurrency.
2022-11-28 12:21:44 +01:00
Andrey Antukh
13a092b192 ♻️ Normalize internal naming on the worker module 2022-11-28 12:20:58 +01:00
luz paz
e30bea0b6f 🔧 Fix typos in source code
Found via `codespell -q 3 -S *.po,./frontend/yarn.lock -L childs,clen,fpr,inflight,ody,ot,ro,te,trys,ue`
2022-10-04 10:40:34 +02:00
Andrey Antukh
41134f22e9 📎 Update license header 2022-09-20 23:23:22 +02:00
Andrey Antukh
9bfdcc6277 Make the task retry algorithm use better backoff values 2022-09-20 23:04:37 +02:00
Andrey Antukh
6f42f4ec45 ♻️ Refactor semaphore and executors 2022-09-20 11:32:45 +02:00
Andrey Antukh
7fa609d5f4 Allow disable worker 2022-08-12 08:52:36 +02:00
Andrey Antukh
f2140a1421 🐛 Fix cron scheduler locking mechanism
And add improved logging to the worker/cron code
2022-07-04 11:32:36 +02:00
Andrey Antukh
9e4a50fb15 ♻️ Refactor backend to be more async friendly 2022-03-03 16:05:52 +01:00
Andrey Antukh
d1c834e647 🐛 Fix minor issue on executors monitor 2022-03-01 14:34:13 +01:00
Andrey Antukh
e60b8a7aef 🐛 Minor fix on worker executors monitor 2022-02-28 17:21:36 +01:00
Andrey Antukh
5d2715dd32 Improve monitors monitor 2022-02-28 15:29:30 +01:00
Andrey Antukh
7cf27ac86d ♻️ Refactor general resource and concurrency model on backend 2022-02-22 13:05:41 +01:00
Andrey Antukh
7eed8c5ee5 Merge remote-tracking branch 'origin/main' into develop 2022-02-16 11:23:26 +01:00
Andrey Antukh
9b78b2a432 Improve error reporting on background tasks 2022-02-15 12:26:36 +01:00
Andrey Antukh
321b2c7c23 🐛 Fix error handling on s3 delete-in-bulk operation 2022-02-15 12:26:36 +01:00
Andrey Antukh
741d2b3f3c Merge remote-tracking branch 'origin/main' into develop 2022-02-12 17:33:28 +01:00
Andrey Antukh
827c2140b7 ♻️ Refactor error reporting and logging context formatting
The prev approach uses clojure.pprint without any limit extensivelly
for format error context data and all this is done on the calling
thread. The clojure.pprint seems very inneficient in cpu and memory
usage on pprinting large data structures.

This is improved in the following way:

- All formatting and pretty printing is moved to logging thread,
  reducing unnecesary blocking and load on jetty http threads.
- Replace the clojure.pprint with fipp.edn that looks considerably
  faster than the first one.
- Add some safe limits on pretty printer for strip printing some
  data when the data structure is very large, very deep or both.
2022-02-10 15:12:35 +01:00
Andrey Antukh
bc2a0432b9 Allow connect to read-only databases. 2022-01-27 16:11:32 +01:00
Andrey Antukh
2d118ecc65 Merge remote-tracking branch 'origin/staging' into develop 2021-12-27 09:50:15 +01:00
Andrey Antukh
1839397ebc Minor enhacements on log processing. 2021-12-23 18:36:58 +01:00
Josh Soref
cd2d3d5fa3 🐛 Fix typos in backend 2021-11-15 09:53:10 -05:00
Andrey Antukh
b64d5ef357 🎉 Add unified logging api. 2021-09-30 11:48:14 +02:00
Andrey Antukh
d7eec3b92b Merge remote-tracking branch 'origin/main' into develop 2021-06-03 12:56:37 +02:00
Andrey Antukh
bae709df5b 🐛 Fix custom font deletion task. 2021-06-03 12:55:31 +02:00
Andrey Antukh
548664f6ce ♻️ Internal directory refactor.
Make common as first-class module.
2021-05-31 11:04:32 +02:00
Andrey Antukh
f545e41d10 📎 Fix license header. 2021-04-12 16:49:43 +02:00
Andrey Antukh
8daf6e822e 🎉 Add profile activity registry logger. 2021-04-09 15:28:18 +02:00
Andrey Antukh
e12a6e65a6 ♻️ Refactor logging. 2021-04-09 15:28:18 +02:00
Andrey Antukh
0926fbcbc6 ♻️ Minor code reorganization.
Improves modularity and reusability and allows usage
of backend code as a library.
2021-03-31 09:28:15 +02:00
Andrey Antukh
fbe2e2a285 Improve tasks metrics. 2021-02-22 14:50:01 +01:00
Andrey Antukh
a63f28a2e5 Normalize logging messages on backend. 2021-02-22 14:50:01 +01:00
Andrey Antukh
c1476d0397 🎉 Add optional loki integration.
And refactor internal error reporting.
2021-02-16 11:31:48 +01:00
Andrey Antukh
6e08c6bc35 📎 Fix linter issues. 2021-02-04 11:48:47 +01:00
Andrey Antukh
80309cbff3 Improve error reporting of tasks. 2021-02-04 11:48:47 +01:00
Andrey Antukh
b80295a21c Fix all linter issues on backend code. 2021-02-01 09:48:28 +01:00
Andrey Antukh
ff6482fa29 🎉 Add telemetry client. 2021-01-07 11:29:57 +01:00
Andrey Antukh
707fa160e8 🎉 Add simple telemetry server module. 2021-01-07 11:29:57 +01:00
Andrey Antukh
9f12456456 ♻️ Replace mount with integrant. 2021-01-07 11:29:57 +01:00
Andrey Antukh
f84d0f34e6 ♻️ Minor task naming and directory structure refactor. 2020-12-21 12:15:53 +01:00
Andrey Antukh
4849904b0b ♻️ Refactor file-media-gc task (mainly add more traces). 2020-12-21 12:15:53 +01:00
Andrey Antukh
4d7a34a998 🎉 Add better error reporting. 2020-12-09 14:38:33 +01:00
Andrey Antukh
0da16c7219 Make all background task execute daily (instead of hourly). 2020-12-04 16:11:26 +01:00