190 Commits

Author SHA1 Message Date
Andrey Antukh
746d898245 Improve the db api efficiency
Mainly setup proper defaults and reduce unnecesary allocations
on every db api call.
2024-01-04 12:41:16 +01:00
Andrey Antukh
addb392ecc Add safety mechanism for direct object deletion
The main objective is prevent deletion of objects that can leave
unreachable orphan objects which we are unable to correctly track.

Additionally, this commit includes:

1. Properly implement safe cascade deletion of all participating
   tables on soft deletion in the objects-gc task;

2. Make the file thumbnail related tables also participate in the
   touch/refcount mechanism applyign to the same safety checks;

3. Add helper for db query lazy iteration using PostgreSQL support
   for server side cursors;

4. Fix efficiency issues on gc related task using server side
   cursors instead of custom chunked iteration for processing data.

   The problem resided when a large chunk of rows that has identical
   value on the deleted_at column and the chunk size is small (the
   default); when the custom chunked iteration only reads a first N
   items and skip the rest of the set to the next run.

   This has caused many objects to remain pending to be eliminated,
   taking up space for longer than expected. The server side cursor
   based iteration does not has this problem and iterates correctly
   over all objects.

5. Fix refcount issues on font variant deletion RPC methods
2024-01-03 10:56:57 +01:00
Andrey Antukh
87615ce221 💄 Fix format issues on backend module 2023-11-29 12:55:58 +01:00
Andrey Antukh
2295d085d3 Improve performance on error formating and reporting 2023-11-27 14:25:12 +01:00
Andrey Antukh
bb5a4c0fa5 Update yetti and adapt for ring-2.0 2023-11-27 14:25:12 +01:00
Andrey Antukh
d1fba8982e Adjust error reporting thresholds 2023-11-21 13:35:19 +01:00
Andrey Antukh
ac3d7f00d5 Make file schema validation configurable using flags 2023-11-20 19:42:46 +01:00
Andrey Antukh
0081db4770 Improve error reporting 2023-11-20 19:42:46 +01:00
Andrey Antukh
4a016dce14 📎 Add minor improvements on params formating on error logger 2023-08-08 14:53:55 +02:00
Andrey Antukh
7afaa9d31f 🐛 Fix incorrect implementation on error reporting context collection 2023-08-04 18:40:47 +02:00
Andrey Antukh
259b05db51 Add more improvements to error reporting 2023-08-04 13:10:36 +02:00
Andrey Antukh
9334f935eb Add better error reporting on response encoding middleware 2023-08-03 16:10:41 +02:00
Andrey Antukh
046f501152 Improve error reporting context 2023-08-02 14:51:12 +02:00
Andrey Antukh
00f7c94377 Improve database error reporter 2023-08-02 13:43:53 +02:00
Andrey Antukh
5ca3d01ea1 🎉 Add malli based validation and coersion subsystem 2023-05-17 16:05:29 +02:00
Andrey Antukh
4e7f32aa88 Improve retry mechanism and macros 2023-05-17 15:47:21 +02:00
Alejandro Alonso
890583a13a Add mvp access-token support 2023-05-04 22:14:55 +02:00
Andrey Antukh
b37a92aaf7 Merge remote-tracking branch 'origin/staging' into develop 2023-04-25 13:34:28 +02:00
Alejandro Alonso
408de63ea3 Merge remote-tracking branch 'origin/staging' into develop 2023-04-05 07:35:36 +02:00
Andrey Antukh
cdaf63afa0 🐛 Enable by default mattermost webhook error reporter 2023-04-04 08:38:05 +02:00
Andrey Antukh
aafbf6bc15 ♻️ Refactor cocurrency model on backend
Mainly the followin changes:

- Pass majority of code to the old and plain synchronous style
  and start using virtual threads for the RPC (and partially some
  HTTP server middlewares).
- Make some improvements on how CLIMIT is handled, simplifying code
- Improve considerably performance reducing the reflection and
  unnecesary funcion calls on the whole stack-trace of an RPC call.
- Improve efficiency reducing considerably the total threads number.
2023-03-14 12:30:27 +01:00
Andrey Antukh
d76baa3266 ⬆️ Update promesa dependency
And adapt all code for breaking changes
2023-03-14 12:30:27 +01:00
Andrey Antukh
1d21ee7089 Merge remote-tracking branch 'origin/staging' into develop 2023-02-24 18:30:05 +01:00
Andrey Antukh
f0c0e5e43a Add proper audit log for invitations 2023-02-24 10:28:07 +01:00
Andrey Antukh
78aea0f24e 🐛 Fix incorrect props cleaning on auditlog 2023-02-08 10:35:57 +01:00
Andrey Antukh
8161d3ae09 🐛 Fix incorrect props cleaning on auditlog 2023-02-07 18:38:54 +01:00
Andrey Antukh
bb055a3c84 ♻️ Refactor logging subsystem and error reporting 2023-02-02 13:38:04 +01:00
Andrey Antukh
50ee0ad3fd Merge remote-tracking branch 'origin/staging' into develop 2023-02-01 23:25:25 +01:00
Andrey Antukh
86712f977d 🐛 Skip unnecesary mutation events from audit log 2023-02-01 18:06:12 +01:00
Alejandro Alonso
11db7590eb Merge remote-tracking branch 'origin/staging' into develop 2023-01-30 15:39:17 +01:00
Alejandro Alonso
2c38f31aa9 🐛 Fix clean archived auditlog 2023-01-30 13:11:50 +01:00
Andrey Antukh
db689d151e ♻️ Refactor profile and session handling
- makes the profile access more efficient (replace in-app joins to a
  simple select query on profile table
- add partial support for access-tokens (still missing some RPC methods)
- move router definitions to specific modules and simplify the main http
  module definitions to simple includes
- simplifiy authentication code related to access-tokens and sessions
- normalize db parameters with proper namespaced props
- more work on convert all modules initialization to use proper specs
  with fully-qualified keyword config props
2023-01-18 10:51:58 +01:00
Andrey Antukh
853be27780 🐛 Fix issues on database logger 2023-01-09 11:43:26 +01:00
Andrey Antukh
b235d3f0f2 Improve update-file webhook batching
make it per user
2023-01-09 11:43:02 +01:00
Andrey Antukh
970dc04bc6 🐛 Fix incorrect output on webhook internal queries 2022-12-16 09:41:32 +01:00
Andrey Antukh
609fa87fe2 Add font events to webhooks 2022-12-16 09:33:52 +01:00
Andrey Antukh
bdd00be5e4 🐛 Fix incorrect deduplication of tasks on webhooks 2022-12-15 11:27:43 +01:00
Andrey Antukh
6ea0279c9e Improve webhook URI validation 2022-12-14 16:22:13 +01:00
Andrey Antukh
6a7a25121e Improve default update-file webhook batch timeout 2022-12-14 16:22:13 +01:00
Andrey Antukh
c820c49fc5 Add generic (blocking) retry macro
And use it on audit handling
2022-12-14 16:22:13 +01:00
Andrey Antukh
be5053ce22 Use the same value for created_at and tracked_at on audit 2022-12-14 16:22:13 +01:00
Andrey Antukh
d56082307b 🎉 Add update-file (batched) to webhooks 2022-12-13 23:14:55 +01:00
Andrey Antukh
f2b60261f8 🎉 Add tests for webhooks rpc and logger 2022-12-13 16:17:31 +01:00
Andrey Antukh
5b9f0ed0b1 🎉 Add webhook processing worker 2022-12-13 16:17:31 +01:00
Andrey Antukh
d584ae5a0f Improve json encode/decode api 2022-12-13 16:17:31 +01:00
Andrey Antukh
7f589b09ca ♻️ Move audit http handler to RPC 2022-12-13 16:17:31 +01:00
Andrey Antukh
861328af3e ⬆️ Update promesa library to 10.0.570 2022-12-05 09:06:24 +01:00
Andrey Antukh
8bad9d8340 ♻️ Refactor loggers/audit, auth/oidc, and http/clent modules 2022-12-05 08:53:00 +01:00
Andrey Antukh
57e7691e66 Merge remote-tracking branch 'origin/staging' into develop 2022-11-03 14:14:27 +01:00
Pablo Alba
8b988e0f1f 📎 Fix linter (unused line on loki) 2022-11-03 08:57:14 +01:00