entrouvert
/
wcs-olap
16
0
Fork 0
Code Issues Pull Requests Releases Activity
231 Commits 6 Branches 81 Tags 385 KiB
Commit Graph

162 Commits

Author SHA1 Message Date
Frédéric Péters d9609f739e tests: adapt to wcs switch to timestamptz (#86887)
gitea/wcs-olap/pipeline/head This commit looks good Details
2024-02-12 18:34:30 +01:00
Frédéric Péters 694e4044d2 misc: add explicit flag to get anonymised forms (#86606)
gitea/wcs-olap/pipeline/head This commit looks good Details
2024-02-06 14:30:12 +01:00
Frédéric Péters b6d00fab97 misc: add support for the new "file-import" channel (#81074)
gitea/wcs-olap/pipeline/head This commit looks good Details
2023-09-14 07:57:43 +02:00
Valentin Deniaud 4627cf8207 misc: apply double-quote-string-fixer (#79788) 2023-08-16 10:31:39 +02:00
Frédéric Péters 461254e986 trivial: apply pre-commit 2022-07-16 08:36:18 +02:00
Benjamin Dauvergne 74eb8aa08c feeder: do not use UNLOGGED tables for persistent data (#65220) 2022-05-13 14:57:47 +02:00
Frédéric Péters d888b4152f only consider bool/item/items fields if explicitely marked for stats (#63432) 2022-04-01 09:17:25 +02:00
Benjamin Dauvergne 420c724be7 feeder: ignore non string items (#60243) 2022-01-21 11:07:26 +01:00
Benjamin Dauvergne 642b7efa3e feeder: create UNLOGGED tables (#59656) 2021-12-22 10:56:22 +01:00
Benjamin Dauvergne a08c629f75 feeder: create an unique slug if necessary (#42911) 2021-08-19 11:59:03 +02:00
Benjamin Dauvergne 05c4031776 feeder: prevent situation of half-dropped schema (#54658) 
To prevent loosing currently loaded data wcs-olap, failing ro rename the
temporary schema to its final name, wcs-olap will:

- first, inside a transaction, rename the current schema instead of
  dropping it, then rename the new schema to the current schema's name;
  in case of failure it will retry 33 times sleeping 1 second between
  each attempt;

- if successfull, drop the renamed old schema, again in a retry loop, if
  it fails to drop it logs an error, without aborting the current
  feeding.
 2021-08-17 23:07:54 +02:00
Benjamin Dauvergne e0a28c2f85 feeder: set synchronous_commit to off during feed (#56039) 2021-08-17 19:27:48 +02:00
Benjamin Dauvergne 47a85d8531 feeder: add option post-sync-commands (#56164) 
Exemple:

  [wcs-olap]
  post-sync-commands = ALTER DEFAULT PRIVILEGES IN SCHEMA {schema} GRANT SELECT ON TABLES TO user;
    GRANT USAGE ON SCHEMA {schema} TO user;
    GRANT SELECT ON ALL TABLES IN SCHEMA {schema}  TO user;

Those commands will be launched after the synchronization, interpolation
variables are usable (like {schema} for the target schema name).
 2021-08-14 23:41:16 +02:00
Benjamin Dauvergne 5baffc8156 cmd: use Logger.setLevel() (#56038) 
Directly setting logger.level should never used as it does not clear the
isEnabledFor cache of the logger which breaks some tests.
 2021-08-14 23:35:48 +02:00
Benjamin Dauvergne 9476ad799b misc: integrate with sentry (#56038) 
To use it add the following lines to /etc/wcs-olap/config.ini:

[sentry]
dsn = https://apikey@sentry.example.com/1
environment = prod
 2021-08-07 15:35:30 +02:00
Benjamin Dauvergne b3a59edb54 Revert "feeder: set deadlock_timeout to 10 seconds (#54658)" 
This reverts commit c74766bf2c.
 2021-06-14 09:44:41 +02:00
Frédéric Péters 9bd160e0dd do not store digits value if outside integer range (#54702) 2021-06-11 08:51:45 +02:00
Benjamin Dauvergne c74766bf2c feeder: set deadlock_timeout to 10 seconds (#54658) 
It makes wcs-olap always win in a deadlock with another client.
 2021-06-08 17:26:43 +02:00
Benjamin Dauvergne ee4bcadcc0 feeder: log failing SQL statements (#45314) 2021-05-28 08:19:34 +02:00
Benjamin Dauvergne d44fafb844 misc: handle invisible statuses using real_status (#40643) 
The formdata API returns the visible status in data.workflow.status,
if we want to index the current status we must use
data.workflow.real_status.
 2021-05-18 19:50:11 +02:00
Benjamin Dauvergne f7f1a6d26f feeder: filter empty formdefs as early as possible (#52015) 2021-04-01 13:19:20 +02:00
Frédéric Péters 1640abbfdc add index on dates (#51681) 2021-03-16 17:20:01 +01:00
Frédéric Péters 1dc7fb5833 trivial: remove debugging print 2021-03-13 11:05:51 +01:00
Frédéric Péters 30e56df41a misc: add social network submission channel (#51816) 2021-03-09 10:35:18 +01:00
Benjamin Dauvergne dc36901dde feeder: store truncated temp schema's name (#50835) 2021-02-04 10:35:24 +01:00
Benjamin Dauvergne 1d35d39f6f misc: remove SQL objects in order to use less locks (#43108) 2021-02-03 18:05:25 +01:00
Benjamin Dauvergne fe2f5b5d89 do not use count to compute emptyness of formdef (#50368) 2021-01-21 11:37:46 +01:00
Serghei Mihai 30f6873d27 feeder: write new model file after schema renaming (#42320)
gitea/wcs-olap/pipeline/head There was a failure building this commit Details
2021-01-11 14:28:51 +01:00
Benjamin Dauvergne ea72fe5d7e use truncate_pg_identifier on fields and table names (#48684) 2020-12-17 14:41:25 +01:00
Serghei Mihai fc92793e49 feeder: try to cast digit string fields with numeric value only (#48426) 2020-11-26 10:37:47 +01:00
Benjamin Dauvergne a16fb4ee98 misc: accept duplicate fields with the same type (#42429) 2020-09-25 20:39:41 +02:00
Nicolas Roche 3eb714f93a wcs_api: provide new include-disabled parameter (#45898) 2020-08-31 17:24:06 +02:00
Benjamin Dauvergne 9f5ff07675 wcs_api: use WcsApi.batch_size for FormDatas batch size (#44970) 2020-07-09 14:26:43 +02:00
Benjamin Dauvergne 909712e15a feeder: remove import six (#43546) 2020-06-26 11:42:54 +02:00
Benjamin Dauvergne afb70fe986 feeder: create measure for integer fields (#43546) 2020-06-26 11:42:54 +02:00
Benjamin Dauvergne 6e4cef6d5c feeder: update public.dates atomically (#44244) 2020-06-26 11:38:42 +02:00
Benjamin Dauvergne d9409b15bd misc: truncate schema_temp to 63 characters (#43165) 2020-05-20 17:37:10 +02:00
Benjamin Dauvergne 44d11e2b02 misc: raise ValueError is schema name is too long (#43165) 2020-05-20 17:30:49 +02:00
Benjamin Dauvergne f45373eb51 misc: quote table names when droping (#43063) 2020-05-19 10:00:31 +02:00
Serghei Mihai 799d799565 feeder: fix debug message on schema renaming (#42323) 2020-04-30 16:42:41 +02:00
Serghei Mihai c68304bf96 feeder: drop old schema sequentially (#42322) 2020-04-30 16:42:41 +02:00
Serghei Mihai d11d196fa8 feeder: preseve categories and form names order (#36930) 2020-03-02 22:07:03 +01:00
Benjamin Dauvergne 3afc79b42e wcs_api: handle empty dict in FormData.workflow (#39373) 2020-01-31 18:26:21 +01:00
Benjamin Dauvergne 8fc092f4d7 feeder: add prefix to join's names of fields (#39373) 2020-01-31 17:51:12 +01:00
Benjamin Dauvergne 04ad4ee2d2 feeder: factorize values in field's schemas (#39373) 2020-01-31 17:51:12 +01:00
Benjamin Dauvergne a6af6fd665 remove dead code (#39430) 2020-01-31 17:49:42 +01:00
Benjamin Dauvergne dce418d99d migrate to python3 (#39430) 2020-01-30 23:33:16 +01:00
Valentin Deniaud c304e0f8f8 misc: add slug to cubes (#38597) 
This also reverts commit 084b0d06b4,
which has never been actually used.
 2019-12-18 10:35:34 +01:00
Benjamin Dauvergne f806b9d675 feeder: define all joins (#38066) 
* join with the dates table must be inner because we do not want to
see results for forms outside the dates table (every form must have a
receipt_time),
* all other joins must be left because form's schemas are dynamic and
dimension can be absent from older forms but we still want to count them.
 2019-12-04 10:17:24 +01:00
Frédéric Péters ba95cc7f28 ignore internal/unknown roles assigned to functions (#37015) 2019-10-17 09:46:21 +02:00