tests: fix PytestCollectionWarning (#75521)

(only if max_items == 1)

data/themes/django/wcs.css

@@ -1,12 +0,0 @@
-@import url(../../qo/css/sofresh.css);
-
-#page {
-	-webkit-transform: rotate(2deg);
-	-webkit-transition: all 200ms ease-out;
-	-webkit-filter: grayscale(100%);
-}
-
-#page:hover {
-	-webkit-transform: rotate(0deg);
-	-webkit-filter: none;
-}

data/web/css/wcs.css

@@ -1,2 +0,0 @@
-@import url(sofresh.css);
-

debian/control

@@ -22,6 +22,7 @@ Depends: graphviz,
          python3-django-ckeditor,
          python3-django-ratelimit,
          python3-dnspython,
+         python3-emoji,
          python3-hobo,
          python3-lasso,
          python3-lxml,

help/fr/api-auth.page

@@ -1,317 +0,0 @@
-<page xmlns="http://projectmallard.org/1.0/"
-      type="topic" id="api-auth" xml:lang="fr">
-
-<info>
-  <link type="guide" xref="index#api" />
-  <revision docversion="0.1" date="2013-01-04" status="draft"/>
-  <revision docversion="0.2" date="2015-12-18" status="draft"/>
-  <credit type="author">
-    <name>Frédéric Péters</name>
-    <email>fpeters@entrouvert.com</email>
-  </credit>
-  <desc>Accès aux API, identifiants et clé d’utilisation, utilisateurs, signature, etc.</desc>
-
-</info>
-
-<title>Authentification</title>
-
-<section>
-<title>Gestion des accès aux API</title>
-
-<p>
-La création d’accès aux API se fait dans l’espace de paramétrage, dans la
-section « Sécurité », en suivant le lien « Accès aux API ». Le bouton « Nouvel
-accès aux API » permet d’ajouter un accès, et il faut indiquer :
-</p>
-
-<list>
-
-<item><p>Nom : le nom choisi pour l’accès, qui sera affiché dans la page des
-accès ;</p></item>
-
-<item><p>Description : pour se rappeler de l’usage prévu pour cet
-accès ;</p></item>
-
-<item><p>Identifiant d’accès : le nom de l’utilisateur à utiliser pour
-l’authentification HTTP Basic, ou le paramètre <code>orig</code> pour
-l’authentification par signature ;</p></item>
-
-<item><p>Clé d’accès : le mot de passe pour l’authentification HTTP Basic de
-cet utilisateur, ou la clé partagée à utiliser pour l’authentification par
-signature ;</p></item>
-
-<item><p>Rôles : liste des rôles automatiquement obtenus lorsque cet accès est
-utilisé. Par exemple, s’il s’agit de permettre de lister des demandes d’une
-certaine démarche, il faut indiquer un rôle qui permet de voir les demandes.
-Ce rôle est à déterminer selon le paramétrage du formulaire de la démarche
-et/ou de son workflow.</p></item>
-
-</list>
-
-<note><p>Il est conseillé de créer des «rôles techniques» dédiés aux API. Ces
-rôles ne sont jamais donnés à des utilisateurs de la plateforme, ils sont
-utilisés uniquement dans le paramétrage des accès. Typiquement une démarche est
-paramétrée pour qu’un certain rôle technique ait accès à ses demandes, et ce
-même rôle est indiqué dans l’accès aux API dédié.</p></note>
-
-<section>
-<title>Définitions de l’usager concerné</title>
-
-<p>
-Si l’authentification utilisée passe par un accès aux API qui ne donne pas de
-rôle spécifique, alors il faut indiquer dans la query-string un usager qui aura
-les rôles nécessaires.
-</p>
-
-<p>
-C'est aussi nécessaire pour les appels concernant un usager particulier, tel
-que la récupération de la liste de ses formulaires en cours.
-</p>
-
-<p>L’usager est précisé en ajoutant dans la query string :</p>
-
-<list>
-<item><p>soit un paramètre <code>email</code> pour trouver l’usager selon son
-adresse électronique</p></item>
-<item><p>soit un paramètre <code>NameID</code> pour trouver l’usager selon son
-NameID SAML.</p></item>
-</list>
-
-</section>
-</section>
-
-<section>
-<title>Authentification simple HTTP Basic</title>
-
-<p>
-Pour accéder aux API avec l’authentification HTTP Basic classique, il faut
-utiliser le nom d’utilisateur (identifiant d’accès) et le mot de passe (clé
-d’accès) de l’accès configuré ci-dessus.
-</p>
-
-</section>
-
-
-<section>
-<title>Authentification par signature de l’URL</title>
-
-<p>
-Un système d’authentification basé sur une signature ajoutée à la fin de l’URL
-est également disponible. Il peut être jugé plus sécurisé que
-l’authentification HTTP Basic, par ailleurs il assure une protection plus
-explicite contre le rejeu. Ce système est spécifique à Publik/w.c.s.
-</p>
-
-<section id="req-security-shared-secret">
-<title>Signature des requêtes</title>
-
-<p>
-La signature d’un appel aux API passe par une clé partagée à
-configurer des deux cotés de la liaison, la signature est du type HMAC;
-l’algorithme de hash à employer est passé en paramètre.
-</p>
-
-<note><p>En ce qui concerne l’algorithme de hash, il est préconisé d’utiliser
-SHA-256 par respect du <link
-href="http://references.modernisation.gouv.fr/securite">Référentiel Général
-de Sécurité (RGS)</link>.</p></note>
-
-<p>
-La signature est à calculer sur la query string encodée complète, en
-enlevant les paramètres terminaux <code>algo</code>, <code>timestamp</code>,
-<code>nonce</code>, <code>orig</code> et <code>signature</code> s’ils sont
-présents.</p>
-
-<p>La formule de calcul de la signature est la suivante :</p>
-
-<code>
-BASE64(HMAC-HASH(query_string+'algo=HASH&amp;timestamp=' + timestamp + '&amp;nonce=' + nonce '&amp;orig=" + orig, clé))
-</code>
-
-<list>
-
-<item><p><code>timestamp</code> est la date dans la zone GMT au format ISO8601
-en se limitant à la précision des secondes (ex : 2012-04-04T12:34:00Z),
-</p></item>
-
-<item><p><code>nonce</code> est un aléa, typiquement la réprésentation hexa
-d’un nombre pseudo-aléatoire de 128 bits,</p></item>
-
-<item><p><code>orig</code> est une chaîne précisant l’émetteur de la
-requête,</p></item>
-
-<item><p>algo est une chaîne représentant l’algorithme de hachage utilisé, sont
-définis : sha1, sha256, sha512 pour les trois algorithmes correspondant.
-L’utilisation d’une valeur différente n’est pas définie. L’algorithme sha256
-est préconisé.</p></item>
-
-</list>
-
-<p>
-La query string définitive est ainsi :
-</p>
-
-<code>
-<var>qs_initial</var>&amp;algo=<var>algo</var>&amp;timestamp=<var>timestamp</var>&amp;nonce=<var>nonce</var>&amp;orig=<var>orig</var>&amp;signature=<var>signature</var>
-</code>
-
-</section>
-
-<section>
-<title>Configuration des clés partagées</title>
-
-<p>
-La définition des clés se fait lors de la création des accès aux API (voir
-plus haut). Lors de la création d'un nouvel accès, l’identifiant d'accès
-correspond au «orig» et la clé d'accès est la clé de signature. Les rôles sont
-ceux qui seront obtenus lors de l’usage de l’accès.
-</p>
-
-<p>
-Les clés partagées peuvent aussi être définies dans le fichier
-<code>site-options.cfg</code>, dans une section <code>[api-secrets]</code>, par
-exemple :
-</p>
-
-<code>
-[api-secrets]
-intranet = 12345
-</code>
-
-</section>
-
-<section>
-<title>Exemples d'implémentation de l’algorithme de signature</title>
-
-<p>
-Voici des exemples de code pour créer des URLs signées selon l’algorithme
-expliqué ci-dessus.
-</p>
-
-<listing>
-<title>Python</title>
-<code mime="text/x-python">
-#! /usr/bin/env python3
-
-import base64
-import datetime
-import hashlib
-import hmac
-import random
-import urllib.parse
-
-
-def sign_url(url, key, algo='sha256', orig=None, timestamp=None, nonce=None):
-    parsed = urllib.parse.urlparse(url)
-    new_query = sign_query(parsed.query, key, algo, orig, timestamp, nonce)
-    return urllib.parse.urlunparse(parsed[:4] + (new_query,) + parsed[5:])
-
-
-def sign_query(query, key, algo='sha256', orig=None, timestamp=None, nonce=None):
-    if timestamp is None:
-        timestamp = datetime.datetime.utcnow()
-    timestamp = timestamp.strftime('%Y-%m-%dT%H:%M:%SZ')
-    if nonce is None:
-        nonce = hex(random.getrandbits(128))[2:].rstrip('L')
-    new_query = query
-    if new_query:
-        new_query += '&amp;'
-    new_query += urllib.parse.urlencode((('algo', algo), ('timestamp', timestamp), ('nonce', nonce)))
-    if orig is not None:
-        new_query += '&amp;' + urllib.parse.urlencode({'orig': orig})
-    signature = base64.b64encode(sign_string(new_query, key, algo=algo))
-    new_query += '&amp;' + urllib.parse.urlencode({'signature': signature})
-    return new_query
-
-
-def sign_string(s, key, algo='sha256', timedelta=30):
-    if not isinstance(key, bytes):
-        key = key.encode('utf-8')
-    if not isinstance(s, bytes):
-        s = s.encode('utf-8')
-    digestmod = getattr(hashlib, algo)
-    hash = hmac.HMAC(key, digestmod=digestmod, msg=s)
-    return hash.digest()
-
-
-# usage:
-url = sign_url('https://www.example.net/uri/?arg=val&amp;arg2=val2', 'user-key', orig='user')
-</code>
-</listing>
-
-<listing>
-<title>PHP</title>
-<code mime="application/x-php">
-&lt;?php
-
-function sign_url(string $url, string $orig, string $key) {
-	$parsed_url = parse_url($url);
-	$timestamp =  gmstrftime("%Y-%m-%dT%H:%M:%SZ");
-	$nonce = bin2hex(random_bytes(16));
-	$new_query = '';
-	if (isset($parsed_url['query'])) {
-		$new_query .= $parsed_url['query'] . '&amp;';
-	}
-	$new_query .= http_build_query(array(
-		'algo' => 'sha256',
-		'timestamp' => $timestamp,
-		'nonce' => $nonce,
-		'orig' => $orig));
-	$signature = base64_encode(hash_hmac('sha256', $new_query, $key, $raw_output = true));
-	$new_query .= '&amp;' . http_build_query(array('signature' => $signature));
-	$scheme   = isset($parsed_url['scheme']) ? $parsed_url['scheme'] . '://' : '';
-	$host     = isset($parsed_url['host']) ? $parsed_url['host'] : '';
-	$port     = isset($parsed_url['port']) ? ':' . $parsed_url['port'] : '';
-	$user     = isset($parsed_url['user']) ? $parsed_url['user'] : '';
-	$pass     = isset($parsed_url['pass']) ? ':' . $parsed_url['pass']  : '';
-	$pass     = ($user || $pass) ? "$pass@" : '';
-	$path     = isset($parsed_url['path']) ? $parsed_url['path'] : '';
-	$fragment = isset($parsed_url['fragment']) ? '#' . $parsed_url['fragment'] : '';
-	return "$scheme$user$pass$host$port$path?$new_query$fragment";
-}
-
-# usage:
-url = sign_url("http://www.example.net/uri/?arg=val&amp;arg2=val2", "user", "user-key");
-
-?&gt;
-</code>
-</listing>
-
-<listing>
-<title>Shell (bash)</title>
-<code mime="application/x-shellscript">
-#!/bin/bash
-
-url="http://www.example.net/uri/?arg=val&amp;arg2=val2"
-orig="user"
-key="user-key"
-
-function rawurlencode() {
-  local string="${1}"
-  local strlen=${#string}
-  local encoded=""
-  local pos c o
-  for ((pos=0; pos&lt;strlen; pos++)); do
-     c=${string:$pos:1}
-     case "$c" in
-        [-_.~a-zA-Z0-9] ) o="${c}" ;;
-        * )               printf -v o '%%%02x' "'$c"
-     esac
-     encoded+="${o}"
-  done
-  echo "${encoded}"
-}
-
-now=$(date -u +%FT%TZ);
-nonce=$(od -N16 -txL /dev/urandom | cut -c8- | tr -d " ")
-qs="algo=sha256&amp;timestamp=$now&amp;nonce=$nonce&amp;orig=$orig"
-sig=$(rawurlencode $(echo -n "$qs" | openssl dgst -binary -sha256 -hmac "$key" | base64))
-signed="${url}?$qs&amp;signature=$sig"
-echo "$signed"
-</code>
-</listing>
-
-</section>
-</section>
-
-</page>

help/fr/api-cards.page

@@ -86,16 +86,12 @@ schémas de données associés.
   </p>
 
   <screen>
-<output style="prompt">$ </output><input>curl -H "Content-type: application/json" \
-       -H "Accept: application/json" \
-       -d@donnees.json \
-       https://www.example.net/api/cards/parkings/submit<var>?signature…</var></input>
+<input>POST https://www.example.net/api/cards/parkings/submit</input>
 <output>{"err": 0, "data": {"id": "5"}}</output>
   </screen>
 
   <p>
-   Le fichier de données utilisé (<file>donnees.json</file>) contient le
-   dictionnaire JSON suivant :
+   Avec les données suivantes en entrée :
   </p>
 
   <code mime="application/json">
@@ -134,10 +130,7 @@ le workflow correspondant.
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -X PUT -H "Content-Type: text/csv" \
-       -H "Accept: application/json" \
-       https://www.example.net/api/cards/<var>slug</var>/import-csv<var>?signature…</var>
-       --data-binary @fichier.csv</input>
+<input>PUT https://www.example.net/api/cards/<var>slug</var>/import-csv</input>
 <output>{"err": 0}</output>
 </screen>
 
@@ -160,10 +153,7 @@ paramètres de l’URL :
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -X PUT -H "Content-Type: text/csv" \
-       -H "Accept: application/json" \
-       https://www.example.net/api/cards/<var>slug</var>/import-csv<var>?async=on</var>
-       --data-binary @fichier.csv</input>
+<input>PUT https://www.example.net/api/cards/<var>slug</var>/import-csv<var>?async=on</var></input>
 <output>{
     "err": 0,
     "data": {
@@ -182,8 +172,7 @@ suivre la progression en appellant son URL indiquée en retour de l’appel PUT.
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/jobs/1234/</input>
+<input>GET https://www.example.net/api/jobs/1234/</input>
 <output>{
     "err": 0,
     "data": {
@@ -215,8 +204,7 @@ de fiche a comme identifiant « parkings ».
 </p>
 
   <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/cards/parkings/5/<var>?signature…</var></input>
+<input>GET https://www.example.net/api/cards/parkings/5/</input>
 </screen>
 
 <p>
@@ -270,11 +258,8 @@ Les données attendues sont similaires à la création d’une nouvelle fiche
 (<link xref="#create"/>), seuls les champs présents seront pris en compte.
 </p>
 
-  <screen>
-<output style="prompt">$ </output><input>curl -H "Content-Type: application/json" \
-       -H "Accept: application/json" \
-       -d@donnees.json \
-       https://www.example.net/api/cards/parkings/5/<var>?signature…</var></input>
+<screen>
+<input>POST https://www.example.net/api/cards/parkings/5/</input>
 </screen>
 
 </section>
@@ -296,8 +281,7 @@ effacées, puis effectuera pour chacune les actions nécessaires.
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/cards/parkings/list<var>?signature…</var></input>
+<input>GET https://www.example.net/api/cards/parkings/list</input>
 <output>{
     "data": [
         {
@@ -346,8 +330,7 @@ vue personnalisée, en ajoutant l’identifiant de celle-ci à l’adresse, ex
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/cards/parkings/list/vue-personnalisee<var>?signature…</var></input>
+<input>GET https://www.example.net/api/cards/parkings/list/vue-personnalisee</input>
 <output>{
     "data": [
         {
@@ -382,8 +365,7 @@ Une API existe pour récupérer le schéma de données d’un modèle de fiches.
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/cards/parkings/@schema<var>?signature…</var></input>
+<input>GET https://www.example.net/api/cards/parkings/@schema</input>
 <output>{
    "always_advertise" : false,
    "appearance_keywords" : null,
@@ -440,8 +422,7 @@ Une API existe pour récupérer le schéma de données d’un modèle de fiches.
 <p>Une API permet de récupérer la liste des modèles de fiches.</p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/cards/@list<var>?signature…</var></input>
+<input>GET https://www.example.net/api/cards/@list</input>
 <output>{
    "data" : [
       {

help/fr/api-datasource.page

@@ -32,7 +32,7 @@ L’adresse appelée doit répondre aux exigences suivantes :
 <example>
 <title>Exemple JSON</title>
 <screen>
-<output style="prompt">$ </output><input>curl https://www.example.net/data/fruits</input>
+<input>GET https://www.example.net/data/fruits</input>
 <output>{
     "data": [
         {
@@ -63,7 +63,7 @@ doit respecter les exigences supplémentaires suivantes :
 <example>
 <title>Exemple JSON d’un élément unique désigné par son identifiant</title>
   <screen>
-<output style="prompt">$ </output><input>curl https://www.example.net/data/fruits?id=1</input>
+<input>GET https://www.example.net/data/fruits?id=1</input>
 <output>{
     "data": [
         {
@@ -77,7 +77,7 @@ doit respecter les exigences supplémentaires suivantes :
 <example>
 <title>Exemple JSON filtré par contenu</title>
   <screen>
-<output style="prompt">$ </output><input>curl https://www.example.net/data/fruits?q=pom</input>
+<input>GET https://www.example.net/data/fruits?q=pom</input>
 <output>{
     "data": [
         {
@@ -98,7 +98,7 @@ de contexte du formulaire.
 <example>
 <title>Exemple JSON enrichi</title>
 <screen>
-<output style="prompt">$ </output><input>curl https://www.example.net/data/fruits</input>
+<input>GET https://www.example.net/data/fruits</input>
 <output>{
     "data": [
         {

help/fr/api-fill.page

@@ -128,16 +128,12 @@ formulaire existant.
   </p>
 
   <screen>
-<output style="prompt">$ </output><input>curl -H "Content-type: application/json" \
-       -H "Accept: application/json" \
-       -d@donnees.json \
-       https://www.example.net/api/formdefs/newsletter/submit<var>?signature…</var></input>
+<input>POST https://www.example.net/api/formdefs/newsletter/submit</input>
 <output>{"err": 0, "data": {"id": "1"}}</output>
   </screen>
 
   <p>
-   Le fichier de données utilisé (<file>donnees.json</file>) contient le
-   dictionnaire JSON suivant :
+   Avec les données suivantes en entrée :
   </p>
 
   <code mime="application/json">
@@ -174,10 +170,7 @@ formulaire existant.
   </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Content-type: application/json" \
-       -H "Accept: application/json" \
-       -d@donnees.json \
-       https://www.example.net/api/forms/newsletter/1/<var>?signature…</var></input>
+<input>POST https://www.example.net/api/forms/newsletter/1/</input>
 <output>{"err": 0}</output>
   </screen>
 

help/fr/api-get.page

@@ -30,8 +30,7 @@ newsletter.
 </p>
 
   <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/forms/newsletter/16/<var>?signature…</var></input>
+<input>GET https://www.example.net/api/forms/newsletter/16/</input>
 </screen>
 
 <p>
@@ -181,8 +180,7 @@ comme un commentaire ou une erreur lors de l’appel d’un <em>web service</em>
 <note>
  <p>
   Il est bien sûr nécessaire de disposer des autorisations nécessaires pour
-  accéder ainsi aux données d’un formulaire. (cf <link
-  xref="api-auth"/> pour les explications sur le sujet)
+  accéder ainsi aux données d’un formulaire.
  </p>
 </note>
 
@@ -270,8 +268,7 @@ etc.).
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/forms/inscriptions/list<var>?signature…</var></input>
+<input>GET https://www.example.net/api/forms/inscriptions/list</input>
 </screen>
 
 <code mime="application/json">
@@ -299,8 +296,7 @@ demandes non terminées (pending) :
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/forms/inscriptions/list?filter=pending<var>&amp;signature…</var></input>
+<input>GET https://www.example.net/api/forms/inscriptions/list?filter=pending</input>
 </screen>
 
 <p>
@@ -311,8 +307,7 @@ possibles est « gratuit » :
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/forms/inscriptions/list?filter-type=gratuit<var>&amp;signature…</var></input>
+<input>GET https://www.example.net/api/forms/inscriptions/list?filter-type=gratuit</input>
 </screen>
 
 <p>
@@ -340,8 +335,7 @@ l’adresse.
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/forms/inscriptions/list?full=on<var>&amp;signature…</var></input>
+<input>GET https://www.example.net/api/forms/inscriptions/list?full=on</input>
 </screen>
 
 <p>
@@ -362,10 +356,8 @@ n’est pas nécessaire de préciser l’identifiant d’un utilisateur.
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/forms/inscriptions/list?full=on&amp;anonymise<var>&amp;signature…</var></input>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/forms/inscriptions/10/?anonymise<var>&amp;signature…</var></input>
+<input>GET https://www.example.net/api/forms/inscriptions/list?full=on&amp;anonymise</input>
+<input>GET https://www.example.net/api/forms/inscriptions/10/?anonymise</input>
 </screen>
 
 </section>
@@ -380,8 +372,7 @@ n’est pas nécessaire de préciser l’identifiant d’un utilisateur.
   </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/forms/<var>?signature…</var></input>
+<input>GET https://www.example.net/api/forms/</input>
 </screen>
 
 <code mime="application/json">
@@ -416,8 +407,7 @@ Par exemple, pour avoir une liste limitée aux demandes terminées :
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/forms/?status=done<var>&amp;signature…</var></input>
+<input>GET https://www.example.net/api/forms/?status=done</input>
 </screen>
 
 <note><p>
@@ -437,8 +427,7 @@ webservice <code>/geojson</code>.
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/forms/inscriptions/geojson<var>?signature…</var></input>
+<input>GET https://www.example.net/api/forms/inscriptions/geojson</input>
 <output>{
   "type": "FeatureCollection",
   "features": [
@@ -474,8 +463,7 @@ l’ensemble des demandes :
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/forms/geojson<var>?signature…</var></input>
+<input>GET https://www.example.net/api/forms/geojson</input>
 </screen>
 
 </section>
@@ -489,8 +477,7 @@ Une API existe pour déterminer l’existence d’un code de suivi et, le cas
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/code/QRFPTSLR<var>?signature…</var></input>
+<input>GET https://www.example.net/api/code/QRFPTSLR</input>
 <output>{"url": "...",
  "load_url": "...",
  "err": 0}</output>

help/fr/api-intro.page

@@ -36,14 +36,6 @@ d’exemples.  Les différentes pages détaillent les points d’accès à
 utiliser pour réaliser les différentes opérations.
 </p>
 
-<note>
-<p>
-Les exemples donnés dans ce document utilisent pour la plupart l’outil en
-ligne de commande <app>curl</app> qui permet de manière simple l’envoi de
-requêtes HTTP à un serveur.
-</p>
-</note>
-
 </section>
 
 </page>

help/fr/api-schema.page

@@ -28,8 +28,7 @@ l’URL <code>/api/formdefs/</code>.
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/formdefs/<var>?signature…</var></input>
+<input>GET https://www.example.net/api/formdefs/</input>
 <output>
 [{"url": "https://www.example.net/inscriptions/newsletter",
   "title": "Newsletter",
@@ -95,8 +94,7 @@ La liste des catégories est disponible à l’URL <code>/api/categories/</code>
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/categories/<var>?signature…</var></input>
+<input>GET https://www.example.net/api/categories/</input>
 <output>
 {"data":
   [
@@ -124,8 +122,7 @@ Les formulaires d’une catégorie précise sont disponibles à l’URL
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/categories/inscriptions/formdefs/<var>?signature…</var></input>
+<input>GET https://www.example.net/api/categories/inscriptions/formdefs/</input>
 </screen>
 
 <p>
@@ -146,8 +143,7 @@ La liste des rôles est disponible à l’URL <code>/api/roles</code>.
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" \
-       https://www.example.net/api/roles<var>?signature…</var></input>
+<input>GET https://www.example.net/api/roles</input>
 <output>
 {"data":
   [

help/fr/api-user.page

@@ -29,7 +29,7 @@ associées aux usagers enregistrés.
     </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl --user <var>…</var> https://www.example.net/api/users/<var>uuid</var>/forms</input>
+<input>GET https://www.example.net/api/users/<var>uuid</var>/forms</input>
 <output>{
     "err": 0,
     "data": [
@@ -124,7 +124,7 @@ particulier.
     </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl --user <var>…</var> https://www.example.net/api/user/<var>uuid</var>/drafts</input>
+<input>GET https://www.example.net/api/user/<var>uuid</var>/drafts</input>
 <output>{
     "err": 0,
     "data": [

help/fr/api-workflow.page

@@ -45,9 +45,13 @@ la référence à l’identifiant de déclencheur (<code>validate</code> dans
 l’exemple qui suit).
 </p>
 
+<p>
+Lors de cette requête, il est nécessaire d’inclure l’entête
+<code>Accept: application/json</code>.
+</p>
+
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" -X POST \
-     https://www.example.net/inscriptions/newsletter/14/jump/trigger/validate/<var>?signature…</var></input>
+<input>POST https://www.example.net/inscriptions/newsletter/14/jump/trigger/validate/</input>
 <output>{"url": null, "err": 0}</output>
 </screen>
 
@@ -57,13 +61,6 @@ de statut d’une série de données, qui seront enregistrées dans les données
 workflow du formulaire.
 </p>
 
-<screen>
-<output style="prompt">$ </output><input>curl -H "Content-type: application/json" -H "Accept: application/json" \
-     -X POST -d@donnes.json \
-     https://www.example.net/inscriptions/newsletter/14/jump/trigger/validate/<var>?signature…</var></input>
-<output>{"url": null, "err": 0}</output>
-</screen>
-
 <p>
 Il est également possible de définir des déclencheurs au niveau des actions
 globales du workflow, ils pourront alors être appelés quel que soit le statut
@@ -76,8 +73,7 @@ ferait ainsi :
 </p>
 
 <screen>
-<output style="prompt">$ </output><input>curl -H "Accept: application/json" -X POST \
-     https://www.example.net/api/forms/newsletter/14/hooks/urgent/<var>?signature…</var></input>
+<input>POST https://www.example.net/api/forms/newsletter/14/hooks/urgent/</input>
 <output>{"err": 0}</output>
 </screen>
 

setup.py

@@ -203,6 +203,7 @@ setup(
         'requests',
         'setproctitle',
         'phonenumbers',
+        'emoji',
     ],
     package_dir={'wcs': 'wcs'},
     packages=find_packages(),

tests/admin_pages/test_datasource.py

@@ -57,6 +57,7 @@ def test_data_sources_from_carddefs(pub):
     create_superuser(pub)
     CardDef.wipe()
     pub.custom_view_class.wipe()
+    NamedDataSource.wipe()
 
     app = login(get_app(pub))
     resp = app.get('/backoffice/settings/data-sources/')
@@ -80,10 +81,15 @@ def test_data_sources_from_carddefs(pub):
     resp = app.get('/backoffice/settings/data-sources/')
     assert 'Data Sources from Card Models' in resp.text
     assert 'There are no data sources from card models.' not in resp.text
-    assert '<li><a href="http://example.net/backoffice/data/foo/">foo</a></li>' in resp.text
+    assert resp.pyquery('.section .objects-list li:first-child a').text() == 'foo'
     assert (
-        '<li><a href="http://example.net/backoffice/data/foo/datasource-card-view/">foo - datasource card view</a></li>'
-        in resp.text
+        resp.pyquery('.section .objects-list li:first-child a').attr['href']
+        == 'http://example.net/backoffice/data/foo/'
+    )
+    assert resp.pyquery('.section .objects-list li:last-child a').text() == 'foo - datasource card view'
+    assert (
+        resp.pyquery('.section .objects-list li:last-child a').attr['href']
+        == 'http://example.net/backoffice/data/foo/datasource-card-view/'
     )
 
 
@@ -106,6 +112,8 @@ def test_data_sources_agenda_without_chrono(pub):
 def test_data_sources_agenda(pub, chrono_url):
     create_superuser(pub)
     NamedDataSource.wipe()
+    CardDef.wipe()
+    pub.custom_view_class.wipe()
 
     app = login(get_app(pub))
     resp = app.get('/backoffice/settings/data-sources/')
@@ -125,21 +133,22 @@ def test_data_sources_agenda(pub, chrono_url):
     resp = app.get('/backoffice/settings/data-sources/')
     assert 'Agendas' in resp.text
     assert 'There are no agendas.' not in resp.text
-    assert '<li><a href="%s/">foobar (foobar)</a></li>' % data_source.id in resp.text
+    assert resp.pyquery('.section .objects-list li:first-child a').text() == 'foobar (foobar)'
+    assert resp.pyquery('.section .objects-list li:first-child a').attr['href'] == data_source.get_admin_url()
 
     data_source.external_status = 'not-found'
     data_source.store()
     resp = app.get('/backoffice/settings/data-sources/')
-    assert (
-        '<li><a href="%s/">foobar (foobar) - <span class="extra-info">not found</span></a></li>'
-        % data_source.id
-        in resp.text
-    )
+    assert resp.pyquery('.section .objects-list li:first-child a').text() == 'foobar (foobar) - not found'
+    assert resp.pyquery('.section .objects-list li:first-child a').attr['href'] == data_source.get_admin_url()
+    assert resp.pyquery('.section .objects-list li:first-child a span.extra-info').text() == 'not found'
 
 
 def test_data_sources_users(pub):
     create_superuser(pub)
     NamedDataSource.wipe()
+    CardDef.wipe()
+    pub.custom_view_class.wipe()
 
     app = login(get_app(pub))
     resp = app.get('/backoffice/settings/data-sources/')
@@ -153,7 +162,8 @@ def test_data_sources_users(pub):
     data_source.store()
     resp = app.get('/backoffice/settings/data-sources/')
     assert 'There are no users data sources defined.' not in resp
-    assert '<li><a href="%s/">foobar (foobar)</a></li>' % data_source.id in resp
+    assert resp.pyquery('.section .objects-list li:first-child a').text() == 'foobar (foobar)'
+    assert resp.pyquery('.section .objects-list li:first-child a').attr['href'] == data_source.get_admin_url()
 
 
 def test_data_sources_new(pub):

tests/admin_pages/test_form.py

@@ -18,7 +18,7 @@ from wcs.data_sources import NamedDataSource
 from wcs.formdef import FormDef
 from wcs.qommon.errors import ConnectionError
 from wcs.qommon.http_request import HTTPRequest
-from wcs.testdef import TestResult
+from wcs.testdef import TestDef, TestResult
 from wcs.workflows import Workflow, WorkflowBackofficeFieldsFormDef, WorkflowVariablesFieldsFormDef
 from wcs.wscalls import NamedWsCall
 
@@ -85,6 +85,13 @@ def test_forms_new(pub):
     assert formdef.fields == []
     assert formdef.disabled is True
 
+    # check max title length
+    resp = app.get('/backoffice/forms/')
+    resp = resp.click('New Form')
+    resp.forms[0]['name'] = 'form title ' * 30
+    resp = resp.forms[0].submit()
+    assert resp.pyquery('#form_error_name').text() == 'Too long, value must be at most 250 characters.'
+
     # check workflow selection is available when there are workflows
     Workflow.wipe()
     workflow = Workflow(name='Workflow One')
@@ -2001,7 +2008,7 @@ def test_form_edit_string_field_validation(pub):
     assert '1st field' in resp.text
 
     resp = resp.click('Edit', href='1/')
-    resp.form['validation$type'] = 'Regular Expression'
+    resp.form['validation$type'] = 'regex'
     resp.form['validation$value_regex'] = r'\d+'
     resp.form['validation$error_message'] = 'Foo Error'
     resp = resp.form.submit('submit').follow()
@@ -2012,12 +2019,12 @@ def test_form_edit_string_field_validation(pub):
     }
 
     resp = resp.click('Edit', href='1/')
-    resp.form['validation$type'] = 'None'
+    resp.form['validation$type'] = ''
     resp = resp.form.submit('submit').follow()
     assert FormDef.get(formdef.id).fields[0].validation is None
 
     resp = resp.click('Edit', href='1/')
-    resp.form['validation$type'] = 'Django Condition'
+    resp.form['validation$type'] = 'django'
     resp.form['validation$value_django'] = 'value|decimal < 20'
     resp.form['validation$error_message'] = 'Bar Error'
     resp = resp.form.submit('submit').follow()
@@ -2028,11 +2035,30 @@ def test_form_edit_string_field_validation(pub):
     }
 
     resp = resp.click('Edit', href='1/')
-    resp.form['validation$type'] = 'Django Condition'
+    resp.form['validation$type'] = 'django'
     resp.form['validation$value_django'] = '{{ value|decimal < 20 }}'
     resp = resp.form.submit('submit')
     assert 'syntax error' in resp.text
 
+    # check default error message is not saved
+    resp.form['validation$value_django'] = ''
+    resp.form['validation$type'] = 'time'
+    resp.form['validation$error_message'] = 'Invalid time'
+    resp = resp.form.submit('submit').follow()
+    assert FormDef.get(formdef.id).fields[0].validation == {
+        'type': 'time',
+    }
+
+    # but custom message is saved
+    resp = resp.click('Edit', href='1/')
+    resp.form['validation$type'] = 'time'
+    resp.form['validation$error_message'] = 'Invalid time, it must be in hh:mm format.'
+    resp = resp.form.submit('submit')
+    assert FormDef.get(formdef.id).fields[0].validation == {
+        'type': 'time',
+        'error_message': 'Invalid time, it must be in hh:mm format.',
+    }
+
 
 def test_form_edit_text_field(pub):
     create_superuser(pub)
@@ -3798,9 +3824,15 @@ def test_form_field_statistics_data_update(pub):
 
 def test_forms_last_test_result(pub, formdef):
     TestResult.wipe()
+    TestDef.wipe()
     create_superuser(pub)
     create_role(pub)
 
+    testdef = TestDef()
+    testdef.object_type = formdef.get_table_name()
+    testdef.object_id = formdef.id
+    testdef.store()
+
     app = login(get_app(pub))
     resp = app.get('/backoffice/forms/1/')
     assert '<h2>form title</h2>' in resp.text
@@ -3844,3 +3876,8 @@ def test_forms_last_test_result(pub, formdef):
         resp = app.get(url)
         assert 'test-success' in resp.text
         assert 'test-failure' not in resp.text
+
+    TestDef.remove_object(testdef.id)
+    for url in ('/backoffice/forms/1/', '/backoffice/forms/1/fields/'):
+        resp = app.get(url)
+        assert '<h2>form title</h2>' in resp.text

tests/admin_pages/test_i18n.py

@@ -52,6 +52,8 @@ def test_i18n_link_on_studio_page(pub):
 
 
 def test_i18n_page(pub):
+    create_superuser(pub)
+
     workflow = Workflow(name='workflow')
     st = workflow.add_status('First Status')
     sendmail = st.add_action('sendmail')
@@ -69,7 +71,12 @@ def test_i18n_page(pub):
     formdef.name = 'test title'
     formdef.fields = [
         StringField(id='1', label='text field', type='string'),
-        StringField(id='2', label='text field', type='string'),
+        StringField(
+            id='2',
+            label='text field',
+            type='string',
+            validation={'type': 'django', 'value': 'False', 'error_message': 'Custom Error'},
+        ),
         ItemField(id='3', label='list field', type='item', items=['first', 'second', 'third']),
     ]
     formdef.workflow = workflow
@@ -115,6 +122,9 @@ def test_i18n_page(pub):
     # check edit button label
     assert TranslatableMessage.count([Equal('string', 'Edit Button')]) == 1
 
+    # check custom validation message
+    assert TranslatableMessage.count([Equal('string', 'Custom Error')]) == 1
+
     # check table
     assert resp.pyquery('tr').length == TranslatableMessage.count()
 
@@ -318,9 +328,9 @@ def test_i18n_pagination(pub):
     resp = resp.click('Go to multilinguism page')
 
     # check page limit
-    assert resp.pyquery('#page-links a').text() == '1 2 3 4 5  10 50 100'
+    assert resp.pyquery('#page-links a').text() == '1 2 3 4 5 6  10 50 100'
     resp = resp.click('50')
-    assert resp.pyquery('#page-links a').text() == '1 2  10 20 100'
+    assert resp.pyquery('#page-links a').text() == '1 2 3  10 20 100'
     resp = resp.click('20')
     resp = resp.click('3')
     assert 'offset=40' in resp.request.url

tests/admin_pages/test_tests.py

@@ -7,6 +7,7 @@ from django.utils.html import escape
 from webtest import Upload
 
 from wcs import fields
+from wcs.blocks import BlockDef
 from wcs.formdef import FormDef
 from wcs.qommon.http_request import HTTPRequest
 from wcs.qommon.upload_storage import PicklableUpload
@@ -61,7 +62,7 @@ def test_tests_link_on_formdef_page(pub):
 
 
 def test_tests_page(pub):
-    user = create_superuser(pub)
+    create_superuser(pub)
 
     formdef = FormDef()
     formdef.name = 'test title'
@@ -71,77 +72,42 @@ def test_tests_page(pub):
     formdef.store()
 
     app = login(get_app(pub))
-
     resp = app.get(formdef.get_admin_url())
     resp = resp.click('Tests')
     assert 'There are no tests yet.' in resp.text
-    assert 'Tests cannot be created because there are no completed forms.' in resp.text
-    assert 'New' not in resp.text
-
-    formdata = formdef.data_class()()
-    formdata.just_created()
-    formdata.receipt_time = datetime.datetime(2021, 1, 1, 0, 0).timetuple()
-    formdata.data['1'] = 'a'
-    formdata.user_id = user.id
-    formdata.store()
-
-    resp = app.get(formdef.get_admin_url())
-    resp = resp.click('Tests')
-    assert 'There are no tests yet.' in resp.text
-    assert 'New tests cannot be created' not in resp.text
 
     resp = resp.click('New')
     resp.form['name'] = 'First test'
-    resp.form['formdata_id'].select(text='1-1 - admin')
-    assert len(resp.form['formdata_id'].options) == 1
-
     resp = resp.form.submit().follow()
+    assert 'Edit test data' in resp.text
+
+    resp.form['f1'] = 'abcdefg'
+    resp = resp.form.submit('submit')
+    assert resp.location == 'http://example.net/backoffice/forms/1/tests/1/'
+
+    resp = resp.follow()
+    assert 'First test' in resp.text
+    assert 'abcdefg' in resp.text
+    assert 'This test is empty' not in resp.text
+
+    resp = app.get('/backoffice/forms/1/tests/')
     assert 'First test' in resp.text
     assert 'no tests yet' not in resp.text
 
-    formdata = formdef.data_class()()
-    formdata.just_created()
-    formdata.receipt_time = datetime.datetime(2021, 1, 1, 0, 0).timetuple()
-    formdata.data['1'] = 'b'
-    formdata.store()
-
-    resp = app.get('/backoffice/forms/1/tests/new')
+    resp = resp.click('New')
     resp.form['name'] = 'Second test'
-    resp.form['formdata_id'].select(text='1-2 - Unknown User')
-    assert len(resp.form['formdata_id'].options) == 2
+    # submit but skip redirection to edit page
+    resp.form.submit()
 
-    resp = resp.form.submit().follow()
+    resp = app.get('/backoffice/forms/1/tests/')
     assert 'First test' in resp.text
     assert 'Second test' in resp.text
 
+    resp = resp.click('Second test')
+    assert 'This test is empty' in resp.text
 
-def test_tests_page_formdefs_isolation(pub):
-    formdef = FormDef()
-    formdef.name = 'dummy'
-    formdef.store()
-
-    formdata = formdef.data_class()()
-    formdata.just_created()
-    formdata.store()
-
-    other_formdef = FormDef()
-    other_formdef.name = 'dummy2'
-    other_formdef.store()
-
-    formdata2 = other_formdef.data_class()()
-    formdata2.just_created()
-    formdata2.store()
-
-    create_superuser(pub)
-    app = login(get_app(pub))
-
-    resp = app.get('/backoffice/forms/1/tests/new')
-    assert len(resp.form['formdata_id'].options) == 1
-    assert resp.form['formdata_id'].options[0][2] == '1-1 - Unknown User'
-
-    resp = app.get('/backoffice/forms/2/tests/new')
-    assert len(resp.form['formdata_id'].options) == 1
-    assert resp.form['formdata_id'].options[0][2] == '2-1 - Unknown User'
+    # test run with empty test is allowed
+    app.get('/backoffice/forms/1/tests/results/run').follow()
 
 
 def test_tests_page_deprecated_fields(pub):
@@ -251,6 +217,37 @@ def test_tests_status_page(pub):
     assert 'result-true' in resp.text
 
 
+def test_tests_status_page_block_field(pub):
+    create_superuser(pub)
+
+    block = BlockDef()
+    block.name = 'foobar'
+    block.fields = [fields.ItemField(id='1', label='Test item', type='item', items=['foo', 'bar', 'baz'])]
+    block.store()
+
+    formdef = FormDef()
+    formdef.name = 'test title'
+    formdef.fields = [
+        fields.BlockField(id='1', label='Block Data', varname='blockdata', type='block:foobar', max_items=3),
+    ]
+    formdef.store()
+
+    formdata = formdef.data_class()()
+    formdata.just_created()
+    formdata.receipt_time = datetime.datetime(2021, 1, 1, 0, 0).timetuple()
+    formdata.data['1'] = {'data': [{'1': 'foo'}]}
+
+    testdef = TestDef.create_from_formdata(formdef, formdata)
+    testdef.name = 'First test'
+    testdef.store()
+
+    app = login(get_app(pub))
+
+    resp = app.get('/backoffice/forms/1/tests/%s/' % testdef.id)
+    assert resp.pyquery('div.field-type-block div.field-type-item p.label').text() == 'Test item'
+    assert resp.pyquery('div.field-type-block div.field-type-item div.value').text() == 'foo'
+
+
 def test_tests_status_page_image_field(pub):
     create_superuser(pub)
 
@@ -294,11 +291,13 @@ def test_tests_edit(pub):
 
     formdef = FormDef()
     formdef.name = 'test title'
+    formdef.fields = [fields.StringField(id='1', label='Text', varname='text')]
     formdef.store()
 
     formdata = formdef.data_class()()
     formdata.receipt_time = datetime.datetime(2021, 1, 1, 0, 0).timetuple()
     formdata.user_id = user.id
+    formdata.data = {'1': 'xxx'}
     formdata.store()
 
     testdef = TestDef.create_from_formdata(formdef, formdata)
@@ -365,12 +364,90 @@ def test_tests_edit_data(pub):
     resp = resp.click('Edit data')
     resp.form['f1'] = 'test 3'
     resp = resp.form.submit('submit')
+    assert 'Save data' in resp.text
     resp = resp.form.submit('submit').follow()  # change nothing on second page
     assert 'test 1' not in resp.text
     assert 'test 3' in resp.text
     assert 'test 2' in resp.text
 
 
+def test_tests_edit_data_mark_as_failing(pub):
+    create_superuser(pub)
+
+    formdef = FormDef()
+    formdef.name = 'test title'
+    formdef.fields = [
+        fields.PageField(
+            id='0',
+            label='1st page',
+            type='page',
+            post_conditions=[
+                {
+                    'condition': {'type': 'django', 'value': 'form_var_text|length > 5'},
+                    'error_message': 'Not enough chars.',
+                }
+            ],
+        ),
+        fields.StringField(id='1', label='Text', varname='text', validation={'type': 'digits'}),
+    ]
+    formdef.store()
+
+    formdata = formdef.data_class()()
+    formdata.just_created()
+    formdata.receipt_time = datetime.datetime(2021, 1, 1, 0, 0).timetuple()
+    formdata.data['1'] = '12345'
+    formdata.store()
+
+    testdef = TestDef.create_from_formdata(formdef, formdata)
+    testdef.name = 'First test'
+    testdef.store()
+
+    app = login(get_app(pub))
+
+    resp = app.get('/backoffice/forms/1/tests/%s/' % testdef.id)
+    resp = resp.click('Edit data')
+    assert 'Mark as failing' not in resp.text
+
+    resp.form['f1'] = '123456'
+    resp = resp.form.submit('submit').follow()
+    assert '123456' in resp.text
+
+    resp = resp.click('Edit data')
+    assert 'Mark as failing' not in resp.text
+
+    # two errors on page
+    resp.form['f1'] = '123a'
+    resp = resp.form.submit('submit')
+    assert 'Mark as failing' not in resp.text
+
+    # one error
+    resp.form['f1'] = '1234'
+    resp = resp.form.submit('submit')
+    assert 'If test should fail on error "Not enough chars.", click button below.' in resp.text
+    assert 'Mark as failing' in resp.text
+
+    # other error
+    resp.forms[0]['f1'] = 'abcdefg'
+    resp = resp.forms[0].submit('submit')
+    assert 'If test should fail on error "Only digits are allowed", click button below.' in resp.text
+    assert 'Mark as failing' in resp.text
+
+    # click mark as failing button
+    resp = resp.forms[1].submit().follow()
+    assert 'abcdefg' in resp.text
+    assert escape('This test is expected to fail on error "Only digits are allowed".') in resp.text
+
+    resp = resp.click('Edit data')
+    assert 'This test is expected to fail on error "Only digits are allowed".' in resp.text
+
+    resp.form['f1'] = '1234567'
+    resp = resp.form.submit('submit').follow()
+    assert 'This test is expected to fail' not in resp.text
+
+    # only post is allowed
+    app.get('/backoffice/forms/1/tests/%s/edit-data/mark-as-failing' % testdef.id, status=404)
+
+
 def test_tests_manual_run(pub):
     user = create_superuser(pub)
 
@@ -404,12 +481,14 @@ def test_tests_manual_run(pub):
     assert 'No test results yet.' in resp.text
 
     resp = resp.click('Run tests')
-    assert resp.location == 'http://example.net/backoffice/forms/1/tests/results/1/'
+    result = TestResult.select()[-1]
+    assert resp.location == 'http://example.net/backoffice/forms/1/tests/results/%s/' % result.id
 
     resp = resp.follow()
     assert 'Started by: Manual run.' in resp.text
     assert len(resp.pyquery('tr')) == 1
     assert 'Success!' in resp.text
+    assert 'Missing required fields: 0' in resp.text
 
     resp = resp.click('First test')
     assert 'Edit data' in resp.text
@@ -424,18 +503,38 @@ def test_tests_manual_run(pub):
     formdef.fields.append(fields.StringField(id='2', label='String', varname='string'))
     formdef.store()
 
+    resp = app.get('/backoffice/forms/1/tests/')  # run from test listing page
     resp = resp.click('Run tests')
-    assert resp.location == 'http://example.net/backoffice/forms/1/tests/results/2/'
+    result = TestResult.select()[-1]
+    assert resp.location == 'http://example.net/backoffice/forms/1/tests/results/%s/' % result.id
 
     resp = app.get('/backoffice/forms/1/tests/results/')
     assert len(resp.pyquery('tr')) == 2
-    assert len(resp.pyquery('span.test-success')) == 1
+    assert len(resp.pyquery('span.test-success')) == 2
+
+    resp = resp.click('#%s' % result.id)
+    assert 'Started by: Manual run.' in resp.text
+    assert 'Success!' in resp.text
+    assert 'Missing required fields: 1' in resp.text
+
+    # add validation to first field
+    formdef.fields[0].validation = {'type': 'digits'}
+    formdef.store()
+
+    resp = app.get('/backoffice/forms/1/tests/results/')
+    resp = resp.click('Run tests')
+    result = TestResult.select()[-1]
+    assert resp.location == 'http://example.net/backoffice/forms/1/tests/results/%s/' % result.id
+
+    resp = app.get('/backoffice/forms/1/tests/results/')
+    assert len(resp.pyquery('tr')) == 3
+    assert len(resp.pyquery('span.test-success')) == 2
     assert len(resp.pyquery('span.test-failure')) == 1
 
-    resp = resp.click('#2')
+    resp = resp.click('#%s' % result.id)
     assert 'Started by: Manual run.' in resp.text
     assert 'Success!' not in resp.text
-    assert 'Empty value for field' in resp.text
+    assert 'Only digits are allowed.' in resp.text
     assert 'disabled' not in resp.text
 
     TestDef.remove_object(testdef.id)
@@ -451,7 +550,9 @@ def test_tests_run_order(pub):
 
     formdef = FormDef()
     formdef.name = 'test title'
-    formdef.fields = [fields.StringField(id='1', label='String', varname='string')]
+    formdef.fields = [
+        fields.StringField(id='1', label='String', varname='string', validation={'type': 'digits'})
+    ]
     formdef.store()
 
     app = login(get_app(pub))
@@ -459,11 +560,12 @@ def test_tests_run_order(pub):
     formdata = formdef.data_class()()
     formdata.just_created()
 
+    formdata.data['1'] = 'a'
     testdef = TestDef.create_from_formdata(formdef, formdata)
     testdef.name = 'Failing test'
     testdef.store()
 
-    formdata.data['1'] = 'a'
+    formdata.data['1'] = '1'
     testdef = TestDef.create_from_formdata(formdef, formdata)
     testdef.name = 'Passing test'
     testdef.store()
@@ -471,5 +573,45 @@ def test_tests_run_order(pub):
     resp = app.get('/backoffice/forms/1/tests/results/')
     resp = resp.click('Run tests').follow()
     assert resp.text.count('Success!') == 1
-    assert resp.text.count('Empty value for field') == 1
+    assert resp.text.count('Only digits are allowed') == 1
     assert resp.text.index('Failing test') < resp.text.index('Passing test')
+
+
+def test_tests_duplicate(pub):
+    user = create_superuser(pub)
+
+    formdef = FormDef()
+    formdef.name = 'test title'
+    formdef.fields = [fields.StringField(id='1', varname='test field', label='Test', type='string')]
+    formdef.store()
+
+    formdata = formdef.data_class()()
+    formdata.just_created()
+    formdata.receipt_time = datetime.datetime(2021, 1, 1, 0, 0).timetuple()
+    formdata.data['1'] = 'abcdefg'
+    formdata.user_id = user.id
+    formdata.store()
+
+    testdef = TestDef.create_from_formdata(formdef, formdata)
+    testdef.name = 'First test'
+    testdef.store()
+
+    app = login(get_app(pub))
+
+    assert TestDef.count() == 1
+
+    resp = app.get('/backoffice/forms/1/tests/%s/' % testdef.id)
+    resp = resp.click('Duplicate')
+    resp = resp.form.submit().follow()
+
+    assert 'First test (copy)' in resp.text
+    assert 'abcdefg' in resp.text
+    assert TestDef.count() == 2
+
+    resp = app.get('/backoffice/forms/1/tests/%s/' % testdef.id)
+    resp = resp.click('Duplicate')
+    resp = resp.form.submit().follow()
+
+    assert 'First test (copy 2)' in resp.text
+    assert 'abcdefg' in resp.text
+    assert TestDef.count() == 3

tests/admin_pages/test_workflow.py

@@ -59,7 +59,7 @@ def test_workflows_default(pub):
     app = login(get_app(pub))
     resp = app.get('/backoffice/workflows/')
     assert 'Default' in resp.text
-    resp = resp.click(href=r'^_default/')
+    resp = resp.click(href=r'/backoffice/workflows/_default/$')
     assert 'Just Submitted' in resp.text
     assert 'This is the default workflow' in resp.text
     # makes sure it cannot be edited
@@ -216,6 +216,15 @@ def test_workflows_category(pub):
     workflow.store()
 
     app = login(get_app(pub))
+
+    resp = app.get('/backoffice/workflows/')
+    assert [x.attrib['href'] for x in resp.pyquery('.single-links a')] == [
+        'http://example.net/backoffice/workflows/_default/',
+        'http://example.net/backoffice/workflows/_carddef_default/',
+        'http://example.net/backoffice/workflows/1/',
+    ]
+    assert 'Uncategorised' not in resp.text
+
     resp = app.get('/backoffice/workflows/categories/')
     resp = resp.click('New Category')
     resp.forms[0]['name'] = 'a new category'
@@ -223,6 +232,15 @@ def test_workflows_category(pub):
     resp = resp.forms[0].submit('submit')
     assert WorkflowCategory.get(1).name == 'a new category'
 
+    # a category is defined -> an implicit "Uncategorised" section is displayed.
+    resp = app.get('/backoffice/workflows/')
+    assert [x.attrib['href'] for x in resp.pyquery('.single-links a')] == [
+        'http://example.net/backoffice/workflows/_default/',
+        'http://example.net/backoffice/workflows/_carddef_default/',
+        'http://example.net/backoffice/workflows/1/',
+    ]
+    assert 'Uncategorised' in resp.text
+
     resp = app.get('/backoffice/workflows/1/')
     resp = resp.click(href='category')
     resp.forms[0]['category_id'] = '1'
@@ -236,6 +254,14 @@ def test_workflows_category(pub):
     resp = resp.forms[0].submit('submit').follow()
     workflow.refresh_from_storage()
     assert str(workflow.category_id) == '1'
+    resp = app.get('/backoffice/workflows/')
+    assert '<h2>a new category' in resp.text
+    assert [x.attrib['href'] for x in resp.pyquery('.single-links a')] == [
+        'http://example.net/backoffice/workflows/_default/',
+        'http://example.net/backoffice/workflows/_carddef_default/',
+        'http://example.net/backoffice/workflows/1/',
+    ]
+    assert 'Uncategorised' not in resp.text
 
     resp = app.get('/backoffice/workflows/categories/')
     resp = resp.click('New Category')
@@ -2311,6 +2337,8 @@ def test_workflows_global_actions_edit(pub):
 
     Workflow.wipe()
     workflow = Workflow(name='foo')
+    st1 = workflow.add_status('Status1')
+    workflow.add_status('Status2')
     workflow.store()
 
     app = login(get_app(pub))
@@ -2337,14 +2365,22 @@ def test_workflows_global_actions_edit(pub):
     # test modifying a trigger
     resp = app.get('/backoffice/workflows/%s/' % workflow.id)
     resp = resp.click('Global Action')
+    assert resp.pyquery('#triggers-list li > a').text() == 'Manual, not assigned'
     assert len(Workflow.get(workflow.id).global_actions[0].triggers) == 1
     resp = resp.click(
         href='triggers/%s/' % Workflow.get(workflow.id).global_actions[0].triggers[0].id, index=0
     )
     assert resp.form['roles$element0'].value == 'None'
     resp.form['roles$element0'].value = '_receiver'
-    resp = resp.form.submit('submit')
+    resp = resp.form.submit('submit').follow()
+    assert resp.pyquery('#triggers-list li > a').text() == 'Manual, by Recipient'
     assert Workflow.get(workflow.id).global_actions[0].triggers[0].roles == ['_receiver']
+    # limit to some statuses
+    resp = resp.click(href=resp.pyquery('#triggers-list li > a').attr.href, index=0)
+    resp.form['statuses$element0'].value = st1.id
+    resp = resp.form.submit('submit').follow()
+    assert Workflow.get(workflow.id).global_actions[0].triggers[0].statuses == [st1.id]
+    assert resp.pyquery('#triggers-list li > a').text() == 'Manual, from status "Status1", by Recipient'
 
     resp = app.get('/backoffice/workflows/%s/' % workflow.id)
     resp = resp.click('Global Action')

tests/api/test_carddata.py

@@ -339,3 +339,101 @@ def test_card_get_file(pub):
     assert os.path.exists(thumb_filepath) is True
     # again, thumbs_dir already exists
     get_app(pub).get(sign_uri(file_url + '&thumbnail=1'), status=200)
+
+
+def test_api_list_formdata_phone_order_by_rank(pub):
+    pub.role_class.wipe()
+    role = pub.role_class(name='test')
+    role.store()
+
+    ApiAccess.wipe()
+    access = ApiAccess()
+    access.name = 'test'
+    access.access_identifier = 'test'
+    access.access_key = '12345'
+    access.store()
+
+    CardDef.wipe()
+    carddef = CardDef()
+    carddef.name = 'test'
+    carddef.workflow_roles = {'_receiver': role.id}
+    carddef.fields = [
+        fields.StringField(id='0', label='a', type='string', display_locations=['listings']),
+        fields.StringField(id='1', label='b', type='string'),
+    ]
+    carddef.store()
+
+    data_class = carddef.data_class()
+    data_class.wipe()
+
+    # 1st carddata, with phone number
+    carddata1 = data_class()
+    carddata1.data = {'1': '+33623456789'}
+    carddata1.just_created()
+    carddata1.jump_status('new')
+    carddata1.store()
+
+    # 2nd carddata, with no value
+    carddata2 = data_class()
+    carddata2.data = {}
+    carddata2.just_created()
+    carddata2.jump_status('new')
+    carddata2.store()
+
+    # check fts
+    resp = get_app(pub).get(
+        sign_uri(
+            '/api/cards/test/list?full=on&q=0623456789', orig=access.access_identifier, key=access.access_key
+        )
+    )
+    assert len(resp.json['data']) == 1
+    assert [int(x['id']) for x in resp.json['data']] == [carddata1.id]
+
+
+def test_carddata_list_skip_evolutions(pub, sql_queries):
+    pub.role_class.wipe()
+    role = pub.role_class(name='test')
+    role.id = '123'
+    role.store()
+
+    CardDef.wipe()
+    carddef = CardDef()
+    carddef.name = 'test'
+    carddef.workflow_roles = {'_viewer': role.id}
+    carddef.fields = [
+        fields.StringField(id='1', label='foobar', varname='foobar'),
+    ]
+    carddef.store()
+
+    carddef.data_class().wipe()
+
+    for i in range(10):
+        carddata = carddef.data_class()()
+        carddata.data = {'1': 'FOO BAR %s' % i}
+        carddata.just_created()
+        carddata.store()
+
+    ApiAccess.wipe()
+    access = ApiAccess()
+    access.name = 'test'
+    access.access_identifier = 'test'
+    access.access_key = '12345'
+    access.roles = [role]
+    access.store()
+
+    app = get_app(pub)
+
+    def get_url(url, **kwargs):
+        app.set_authorization(('Basic', ('test', '12345')))
+        return app.get(url, **kwargs)
+
+    sql_queries.clear()
+    get_url('/api/cards/test/list')
+    assert not [x for x in sql_queries if '%s_evolution' % carddef.table_name in x]
+    sql_queries.clear()
+    get_url('/api/cards/test/list?include-workflow=on')
+    assert [x for x in sql_queries if '%s_evolution' % carddef.table_name in x]
+    sql_queries.clear()
+    get_url('/api/cards/test/list?include-evolution=on')
+    assert [x for x in sql_queries if '%s_evolution' % carddef.table_name in x]
+    sql_queries.clear()

tests/api/test_export_import.py

@@ -6,6 +6,7 @@ import xml.etree.ElementTree as ET
 
 import pytest
 
+from wcs.applications import Application, ApplicationElement
 from wcs.blocks import BlockDef
 from wcs.carddef import CardDef
 from wcs.categories import (
@@ -19,9 +20,10 @@ from wcs.categories import (
 )
 from wcs.comment_templates import CommentTemplate
 from wcs.data_sources import NamedDataSource
-from wcs.fields import BlockField, CommentField, PageField, StringField
+from wcs.fields import BlockField, CommentField, ComputedField, PageField, StringField
 from wcs.formdef import FormDef
 from wcs.mail_templates import MailTemplate
+from wcs.sql import Equal
 from wcs.wf.form import WorkflowFormFieldsFormDef
 from wcs.workflows import Workflow, WorkflowBackofficeFieldsFormDef, WorkflowVariablesFieldsFormDef
 from wcs.wscalls import NamedWsCall
@@ -41,6 +43,8 @@ coucou = 1234
 '''
         )
 
+    Application.wipe()
+    ApplicationElement.wipe()
     Category.wipe()
     FormDef.wipe()
     CardDefCategory.wipe()
@@ -56,6 +60,7 @@ coucou = 1234
     DataSourceCategory.wipe()
     NamedDataSource.wipe()
     NamedWsCall.wipe()
+    pub.custom_view_class.wipe()
 
     return pub
 
@@ -138,6 +143,8 @@ def test_export_import_dependencies(pub):
     wscall.store()
     wscall = NamedWsCall(name='Test sexies')
     wscall.store()
+    wscall = NamedWsCall(name='Test in computed field')
+    wscall.store()
 
     carddef = CardDef()
     carddef.name = 'Test'
@@ -291,6 +298,12 @@ def test_export_import_dependencies(pub):
             label='X {{ webservice.test }} X {{ cards|objects:"test" }} X {{ forms|objects:"test-ter" }} X',
             type='comment',
         ),
+        ComputedField(
+            id='3',
+            label='computed field',
+            varname='computed_field',
+            value_template='{{ webservice.test_in_computed_field.xxx }}',
+        ),
     ]
     formdef.workflow = workflow
     formdef.store()
@@ -304,6 +317,7 @@ def test_export_import_dependencies(pub):
         ('test', 'wscalls'),
         ('test_quater', 'wscalls'),
         ('test_quinquies', 'wscalls'),
+        ('test_in_computed_field', 'wscalls'),
         ('test', 'cards'),
         ('test-bis', 'cards'),
         ('test-bis', 'forms'),
@@ -472,6 +486,12 @@ def test_export_import_redirect_url(pub):
     mail_template_category = MailTemplateCategory(name='Test')
     mail_template_category.store()
 
+    comment_template = CommentTemplate(name='Test')
+    comment_template.store()
+
+    comment_template_category = CommentTemplateCategory(name='Test')
+    comment_template_category.store()
+
     elements = [
         ('forms', '/backoffice/forms/%s/' % formdef.id),
         ('cards', '/backoffice/cards/%s/' % carddef.id),
@@ -488,6 +508,11 @@ def test_export_import_redirect_url(pub):
             'mail-templates-categories',
             '/backoffice/workflows/mail-templates/categories/%s/' % mail_template_category.id,
         ),
+        ('comment-templates', '/backoffice/workflows/comment-templates/%s/' % comment_template.id),
+        (
+            'comment-templates-categories',
+            '/backoffice/workflows/comment-templates/categories/%s/' % comment_template_category.id,
+        ),
     ]
     for object_type, obj_url in elements:
         resp = get_app(pub).get(sign_uri('/api/export-import/%s/' % object_type))
@@ -506,13 +531,19 @@ def test_export_import_redirect_url(pub):
     get_app(pub).get('/api/export-import/roles/test/redirect/', status=404)
 
 
-def create_bundle(elements, *args):
+def create_bundle(elements, *args, **kwargs):
+    visible = kwargs.get('visible', True)
     tar_io = io.BytesIO()
     with tarfile.open(mode='w', fileobj=tar_io) as tar:
         manifest_json = {
             'application': 'Test',
             'slug': 'test',
-            'description': '',
+            'icon': 'foo.png',
+            'description': 'Foo Bar',
+            'documentation_url': 'http://foo.bar',
+            'visible': visible,
+            'version_number': '42.0',
+            'version_notes': 'foo bar blah',
             'elements': elements,
         }
         manifest_fd = io.BytesIO(json.dumps(manifest_json, indent=2).encode())
@@ -520,6 +551,13 @@ def create_bundle(elements, *args):
         tarinfo.size = len(manifest_fd.getvalue())
         tar.addfile(tarinfo, fileobj=manifest_fd)
 
+        icon_fd = io.BytesIO(
+            b'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABAQAAAAA3bvkkAAAACklEQVQI12NoAAAAggCB3UNq9AAAAABJRU5ErkJggg=='
+        )
+        tarinfo = tarfile.TarInfo('foo.png')
+        tarinfo.size = len(icon_fd.getvalue())
+        tar.addfile(tarinfo, fileobj=icon_fd)
+
         for path, obj in args:
             tarinfo = tarfile.TarInfo(path)
             if hasattr(obj, 'export_for_application'):
@@ -594,6 +632,12 @@ def test_export_import_bundle_import(pub):
     mail_template.category = mail_template_category
     mail_template.store()
 
+    comment_template_category = CommentTemplateCategory(name='Test')
+    comment_template_category.store()
+    comment_template = CommentTemplate(name='Test')
+    comment_template.category = comment_template_category
+    comment_template.store()
+
     wscall = NamedWsCall(name='Test')
     wscall.store()
 
@@ -610,6 +654,8 @@ def test_export_import_bundle_import(pub):
             {'type': 'workflows', 'slug': 'test', 'name': 'test'},
             {'type': 'mail-templates-categories', 'slug': 'test', 'name': 'test'},
             {'type': 'mail-templates', 'slug': 'test', 'name': 'test'},
+            {'type': 'comment-templates-categories', 'slug': 'test', 'name': 'test'},
+            {'type': 'comment-templates', 'slug': 'test', 'name': 'test'},
             {'type': 'data-sources-categories', 'slug': 'test', 'name': 'test'},
             {'type': 'data-sources', 'slug': 'test', 'name': 'test'},
             {'type': 'wscalls', 'slug': 'test', 'name': 'test'},
@@ -626,6 +672,8 @@ def test_export_import_bundle_import(pub):
         ('data-sources/test', data_source),
         ('mail-templates-categories/test', mail_template_category),
         ('mail-templates/test', mail_template),
+        ('comment-templates-categories/test', comment_template_category),
+        ('comment-templates/test', comment_template),
         ('roles/test', role),
         ('wscalls/test', wscall),
     )
@@ -639,6 +687,8 @@ def test_export_import_bundle_import(pub):
     Workflow.wipe()
     MailTemplateCategory.wipe()
     MailTemplate.wipe()
+    CommentTemplateCategory.wipe()
+    CommentTemplate.wipe()
     DataSourceCategory.wipe()
     NamedDataSource.wipe()
     pub.role_class.wipe()
@@ -655,7 +705,7 @@ def test_export_import_bundle_import(pub):
     afterjob_url = resp.json['url']
     resp = get_app(pub).put(sign_uri(afterjob_url))
     assert resp.json['data']['status'] == 'completed'
-    assert resp.json['data']['completion_status'] == '17/17 (100%)'
+    assert resp.json['data']['completion_status'] == '19/19 (100%)'
 
     assert Category.count() == 1
     assert FormDef.count() == 1
@@ -674,11 +724,42 @@ def test_export_import_bundle_import(pub):
     assert MailTemplateCategory.count() == 1
     assert MailTemplate.count() == 1
     assert MailTemplate.select()[0].category_id == MailTemplateCategory.select()[0].id
+    assert CommentTemplateCategory.count() == 1
+    assert CommentTemplate.count() == 1
+    assert CommentTemplate.select()[0].category_id == CommentTemplateCategory.select()[0].id
     assert DataSourceCategory.count() == 1
     assert NamedDataSource.count() == 1
     assert NamedDataSource.select()[0].category_id == DataSourceCategory.select()[0].id
     assert NamedWsCall.count() == 1
     assert pub.custom_view_class().count() == 1
+    assert Application.count() == 1
+    application = Application.select()[0]
+    assert application.slug == 'test'
+    assert application.name == 'Test'
+    assert application.description == 'Foo Bar'
+    assert application.documentation_url == 'http://foo.bar'
+    assert application.version_number == '42.0'
+    assert application.version_notes == 'foo bar blah'
+    assert application.icon.base_filename == 'foo.png'
+    assert application.editable is False
+    assert application.visible is True
+    assert ApplicationElement.count() == 15
+
+    # check editable flag is kept on install
+    application.editable = False
+    application.store()
+
+    # create some links to elements not present in manifest: they should be unlinked
+    element1 = ApplicationElement()
+    element1.application_id = application.id
+    element1.object_type = 'foobar'
+    element1.object_id = '42'
+    element1.store()
+    element2 = ApplicationElement()
+    element2.application_id = application.id
+    element2.object_type = 'foobarblah'
+    element2.object_id = '35'
+    element2.store()
 
     # run new import to check it doesn't duplicate objects
     resp = get_app(pub).put(sign_uri('/api/export-import/bundle-import/'), bundle)
@@ -696,10 +777,36 @@ def test_export_import_bundle_import(pub):
     assert Workflow.count() == 1
     assert MailTemplateCategory.count() == 1
     assert MailTemplate.count() == 1
+    assert CommentTemplateCategory.count() == 1
+    assert CommentTemplate.count() == 1
     assert DataSourceCategory.count() == 1
     assert NamedDataSource.count() == 1
     assert pub.custom_view_class().count() == 1
     assert NamedWsCall.count() == 1
+    assert Application.count() == 1
+    assert ApplicationElement.count() == 15
+    assert (
+        ApplicationElement.select(
+            [
+                Equal('application_id', application.id),
+                Equal('object_type', element1.object_type),
+                Equal('object_id', element1.object_id),
+            ]
+        )
+        == []
+    )
+    assert (
+        ApplicationElement.select(
+            [
+                Equal('application_id', application.id),
+                Equal('object_type', element2.object_type),
+                Equal('object_id', element2.object_id),
+            ]
+        )
+        == []
+    )
+    application = Application.select()[0]
+    assert application.editable is False
 
     # change immutable attributes and check they are not reset
     formdef = FormDef.select()[0]
@@ -747,3 +854,275 @@ def test_export_import_formdef_do_not_overwrite_table_name(pub):
     formdef = FormDef.select()[0]
     assert formdef.table_name == 'formdata_%s_test2' % formdef.id
     assert formdef.data_class().count() == 1
+
+
+def test_export_import_bundle_declare(pub):
+    workflow_category = WorkflowCategory(name='test')
+    workflow_category.store()
+
+    workflow = Workflow(name='test')
+    workflow.store()
+
+    block_category = BlockCategory(name='test')
+    block_category.store()
+
+    block = BlockDef(name='test')
+    block.store()
+
+    role = pub.role_class(name='test')
+    role.store()
+
+    category = Category(name='Test')
+    category.store()
+
+    formdef = FormDef()
+    formdef.name = 'Test'
+    formdef.store()
+
+    card_category = CardDefCategory(name='Test')
+    card_category.store()
+
+    carddef = CardDef()
+    carddef.name = 'Test'
+    carddef.store()
+
+    ds_category = DataSourceCategory(name='Test')
+    ds_category.store()
+    data_source = NamedDataSource(name='Test')
+    data_source.store()
+
+    mail_template_category = MailTemplateCategory(name='Test')
+    mail_template_category.store()
+    mail_template = MailTemplate(name='Test')
+    mail_template.store()
+
+    comment_template_category = CommentTemplateCategory(name='Test')
+    comment_template_category.store()
+    comment_template = CommentTemplate(name='Test')
+    comment_template.store()
+
+    wscall = NamedWsCall(name='Test')
+    wscall.store()
+
+    bundle = create_bundle(
+        [
+            {'type': 'forms-categories', 'slug': 'test', 'name': 'test'},
+            {'type': 'forms', 'slug': 'test', 'name': 'test'},
+            {'type': 'cards-categories', 'slug': 'test', 'name': 'test'},
+            {'type': 'cards', 'slug': 'test', 'name': 'test'},
+            {'type': 'blocks-categories', 'slug': 'test', 'name': 'test'},
+            {'type': 'blocks', 'slug': 'test', 'name': 'test'},
+            {'type': 'roles', 'slug': 'test', 'name': 'test'},
+            {'type': 'workflows-categories', 'slug': 'test', 'name': 'test'},
+            {'type': 'workflows', 'slug': 'test', 'name': 'test'},
+            {'type': 'mail-templates-categories', 'slug': 'test', 'name': 'test'},
+            {'type': 'mail-templates', 'slug': 'test', 'name': 'test'},
+            {'type': 'comment-templates-categories', 'slug': 'test', 'name': 'test'},
+            {'type': 'comment-templates', 'slug': 'test', 'name': 'test'},
+            {'type': 'data-sources-categories', 'slug': 'test', 'name': 'test'},
+            {'type': 'data-sources', 'slug': 'test', 'name': 'test'},
+            {'type': 'wscalls', 'slug': 'test', 'name': 'test'},
+        ],
+        ('forms-categories/test', category),
+        ('forms/test', formdef),
+        ('cards-categories/test', card_category),
+        ('cards/test', carddef),
+        ('blocks-categories/test', block_category),
+        ('blocks/test', block),
+        ('workflows-categories/test', workflow_category),
+        ('workflows/test', workflow),
+        ('data-sources-categories/test', ds_category),
+        ('data-sources/test', data_source),
+        ('mail-templates-categories/test', mail_template_category),
+        ('mail-templates/test', mail_template),
+        ('comment-templates-categories/test', comment_template_category),
+        ('comment-templates/test', comment_template),
+        ('roles/test', role),
+        ('wscalls/test', wscall),
+        visible=False,
+    )
+
+    resp = get_app(pub).put(sign_uri('/api/export-import/bundle-declare/'), bundle)
+    afterjob_url = resp.json['url']
+    resp = get_app(pub).put(sign_uri(afterjob_url))
+    assert resp.json['data']['status'] == 'completed'
+    assert resp.json['data']['completion_status'] == '15/15 (100%)'
+
+    assert Application.count() == 1
+    application = Application.select()[0]
+    assert application.slug == 'test'
+    assert application.name == 'Test'
+    assert application.description == 'Foo Bar'
+    assert application.documentation_url == 'http://foo.bar'
+    assert application.version_number == '42.0'
+    assert application.version_notes == 'foo bar blah'
+    assert application.icon.base_filename == 'foo.png'
+    assert application.editable is True
+    assert application.visible is False
+    assert ApplicationElement.count() == 15
+
+    # create some links to elements not present in manifest: they should be unlinked
+    element1 = ApplicationElement()
+    element1.application_id = application.id
+    element1.object_type = 'foobar'
+    element1.object_id = '42'
+    element1.store()
+    element2 = ApplicationElement()
+    element2.application_id = application.id
+    element2.object_type = 'foobarblah'
+    element2.object_id = '35'
+    element2.store()
+    # and remove an object to have an unkown reference in manifest
+    MailTemplate.wipe()
+
+    resp = get_app(pub).put(sign_uri('/api/export-import/bundle-declare/'), bundle)
+    afterjob_url = resp.json['url']
+    resp = get_app(pub).put(sign_uri(afterjob_url))
+    assert resp.json['data']['status'] == 'completed'
+
+    assert Application.count() == 1
+    assert ApplicationElement.count() == 14
+    assert (
+        ApplicationElement.select(
+            [
+                Equal('application_id', application.id),
+                Equal('object_type', element1.object_type),
+                Equal('object_id', element1.object_id),
+            ]
+        )
+        == []
+    )
+    assert (
+        ApplicationElement.select(
+            [
+                Equal('application_id', application.id),
+                Equal('object_type', element2.object_type),
+                Equal('object_id', element2.object_id),
+            ]
+        )
+        == []
+    )
+    assert (
+        ApplicationElement.select(
+            [
+                Equal('application_id', application.id),
+                Equal('object_type', MailTemplate.xml_root_node),
+            ]
+        )
+        == []
+    )
+
+
+def test_export_import_bundle_unlink(pub):
+    application = Application()
+    application.slug = 'test'
+    application.name = 'Test'
+    application.version_number = 'foo'
+    application.store()
+
+    other_application = Application()
+    other_application.slug = 'other-test'
+    other_application.name = 'Other Test'
+    other_application.version_number = 'foo'
+    other_application.store()
+
+    workflow_category = WorkflowCategory(name='test')
+    workflow_category.store()
+    ApplicationElement.update_or_create_for_object(application, workflow_category)
+
+    workflow = Workflow(name='test')
+    workflow.store()
+    ApplicationElement.update_or_create_for_object(application, workflow)
+
+    block_category = BlockCategory(name='test')
+    block_category.store()
+    ApplicationElement.update_or_create_for_object(application, block_category)
+
+    block = BlockDef(name='test')
+    block.store()
+    ApplicationElement.update_or_create_for_object(application, block)
+
+    category = Category(name='Test')
+    category.store()
+    ApplicationElement.update_or_create_for_object(application, category)
+
+    formdef = FormDef()
+    formdef.name = 'Test'
+    formdef.store()
+    ApplicationElement.update_or_create_for_object(application, formdef)
+
+    card_category = CardDefCategory(name='Test')
+    card_category.store()
+    ApplicationElement.update_or_create_for_object(application, card_category)
+
+    carddef = CardDef()
+    carddef.name = 'Test'
+    carddef.store()
+    ApplicationElement.update_or_create_for_object(application, carddef)
+
+    ds_category = DataSourceCategory(name='Test')
+    ds_category.store()
+    ApplicationElement.update_or_create_for_object(application, ds_category)
+    data_source = NamedDataSource(name='Test')
+    data_source.store()
+    ApplicationElement.update_or_create_for_object(application, data_source)
+
+    mail_template_category = MailTemplateCategory(name='Test')
+    mail_template_category.store()
+    ApplicationElement.update_or_create_for_object(application, mail_template_category)
+    mail_template = MailTemplate(name='Test')
+    mail_template.store()
+    ApplicationElement.update_or_create_for_object(application, mail_template)
+
+    comment_template_category = CommentTemplateCategory(name='Test')
+    comment_template_category.store()
+    ApplicationElement.update_or_create_for_object(application, comment_template_category)
+    comment_template = CommentTemplate(name='Test')
+    comment_template.store()
+    ApplicationElement.update_or_create_for_object(application, comment_template)
+
+    wscall = NamedWsCall(name='Test')
+    wscall.store()
+    ApplicationElement.update_or_create_for_object(application, wscall)
+
+    element = ApplicationElement()
+    element.application_id = application.id
+    element.object_type = 'foobar'
+    element.object_id = '42'
+    element.store()
+
+    other_element = ApplicationElement()
+    other_element.application_id = other_application.id
+    other_element.object_type = 'foobar'
+    other_element.object_id = '42'
+    other_element.store()
+
+    assert Application.count() == 2
+    assert ApplicationElement.count() == 17
+
+    get_app(pub).post(sign_uri('/api/export-import/unlink/'), {'application': 'test'})
+
+    assert Application.count() == 1
+    assert ApplicationElement.count() == 1
+
+    assert (
+        Application.count(
+            [
+                Equal('id', other_application.id),
+            ]
+        )
+        == 1
+    )
+    assert (
+        ApplicationElement.count(
+            [
+                Equal('application_id', other_application.id),
+            ]
+        )
+        == 1
+    )
+
+    # again
+    get_app(pub).post(sign_uri('/api/export-import/unlink/'), {'application': 'test'})
+    assert Application.count() == 1
+    assert ApplicationElement.count() == 1

tests/api/test_formdata.py

@@ -106,8 +106,39 @@ def ics_data(local_user):
     formdata.store()
 
 
-def test_formdata(pub, local_user):
+@pytest.mark.parametrize('user', ['query-email', 'api-access'])
+@pytest.mark.parametrize('auth', ['signature', 'http-basic'])
+def test_formdata(pub, local_user, user, auth):
     NamedDataSource.wipe()
+
+    app = get_app(pub)
+
+    if user == 'api-access':
+        ApiAccess.wipe()
+        access = ApiAccess()
+        access.name = 'test'
+        access.access_identifier = 'test'
+        access.access_key = '12345'
+        access.store()
+
+        if auth == 'http-basic':
+
+            def get_url(url, **kwargs):
+                app.set_authorization(('Basic', ('test', '12345')))
+                return app.get(url, **kwargs)
+
+        else:
+
+            def get_url(url, **kwargs):
+                return app.get(sign_uri(url, orig=access.access_identifier, key=access.access_key), **kwargs)
+
+    else:
+        if auth == 'http-basic':
+            pytest.skip('http basic authentication requires ApiAccess')
+
+        def get_url(url, **kwargs):
+            return app.get(sign_uri(url, user=local_user), **kwargs)
+
     data_source = NamedDataSource(name='foobar')
     data_source.data_source = {
         'type': 'jsonvalue',
@@ -189,11 +220,16 @@ def test_formdata(pub, local_user):
     formdata.geolocations = {'base': {'lon': 10, 'lat': -12}}
     formdata.store()
 
-    resp = get_app(pub).get(sign_uri('/api/forms/test/%s/' % formdata.id, user=local_user), status=403)
+    resp = get_url('/api/forms/test/%s/' % formdata.id, status=403)
 
-    local_user.roles = [role.id]
-    local_user.store()
-    resp = get_app(pub).get(sign_uri('/api/forms/test/%s/' % formdata.id, user=local_user), status=200)
+    if user == 'api-access':
+        access.roles = [role]
+        access.store()
+    else:
+        local_user.roles = [role.id]
+        local_user.store()
+
+    resp = get_url('/api/forms/test/%s/' % formdata.id, status=200)
 
     assert datetime.datetime.strptime(resp.json['last_update_time'], '%Y-%m-%dT%H:%M:%S')
     assert datetime.datetime.strptime(resp.json['receipt_time'], '%Y-%m-%dT%H:%M:%S')
@@ -210,6 +246,7 @@ def test_formdata(pub, local_user):
     assert resp.json['fields']['file']['content_type'] == 'text/plain'
     assert resp.json['fields']['file']['url'].startswith('http://example.net/test/1/download?hash=')
     assert 'thumbnail_url' not in resp.json['fields']['file']
+    get_url(resp.json['fields']['file']['url'], status=200)
     assert resp.json['fields']['item'] == 'foo'
     assert resp.json['fields']['item_raw'] == '1'
     assert resp.json['fields']['item_structured'] == {'id': '1', 'text': 'foo', 'more': 'XXX'}
@@ -248,17 +285,16 @@ def test_formdata(pub, local_user):
     workflow.add_status('Status1', 'st1')
     workflow.possible_status[-1].visibility = ['unknown']
     workflow.store()
+    formdef.refresh_from_storage()  # also update cached workflow
     formdata.jump_status('st1')
     assert formdata.status == 'wf-st1'
 
-    resp = get_app(pub).get(sign_uri('/api/forms/test/%s/' % formdata.id, user=local_user), status=200)
+    resp = get_url('/api/forms/test/%s/' % formdata.id, status=200)
     assert resp.json['workflow']['status'] == {'id': 'new', 'name': 'New'}
     assert resp.json['workflow']['real_status'] == {'id': 'st1', 'name': 'Status1'}
 
     # check ?include-files-content=off
-    resp = get_app(pub).get(
-        sign_uri('/api/forms/test/%s/?include-files-content=off' % formdata.id, user=local_user), status=200
-    )
+    resp = get_url('/api/forms/test/%s/?include-files-content=off' % formdata.id, status=200)
     assert 'content' not in resp.json['fields']['file']
     assert resp.json['fields']['file']['url']
     assert resp.json['fields']['file']['filename'] == 'test.txt'
@@ -391,7 +427,7 @@ def test_formdata_submission_fields(pub, local_user):
 
 
 def test_formdata_backoffice_fields(pub, local_user):
-    test_formdata(pub, local_user)
+    test_formdata(pub, local_user, 'query-email', 'signature')
     Workflow.wipe()
     workflow = Workflow(name='foo')
     workflow.backoffice_fields_formdef = WorkflowBackofficeFieldsFormDef(workflow)

tests/api/test_statistics.py

@@ -321,55 +321,37 @@ def test_statistics_forms_count(pub):
     formdata.store()
 
     resp = get_app(pub).get(sign_uri('/api/statistics/forms/count/'))
-    assert resp.json == {
-        'data': {
-            'series': [{'data': [20, 0, 30], 'label': 'Forms Count'}],
-            'x_labels': ['2021-01', '2021-02', '2021-03'],
-            'subfilters': [],
-        },
-        'err': 0,
-    }
+    assert resp.json['data']['series'] == [{'data': [20, 0, 30], 'label': 'Forms Count'}]
+    assert resp.json['data']['x_labels'] == ['2021-01', '2021-02', '2021-03']
 
     resp = get_app(pub).get(sign_uri('/api/statistics/forms/count/?time_interval=year'))
-    assert resp.json == {
-        'data': {
-            'series': [{'data': [50], 'label': 'Forms Count'}],
-            'x_labels': ['2021'],
-            'subfilters': [],
-        },
-        'err': 0,
-    }
+    assert resp.json['data']['series'] == [{'data': [50], 'label': 'Forms Count'}]
+    assert resp.json['data']['x_labels'] == ['2021']
 
     resp = get_app(pub).get(sign_uri('/api/statistics/forms/count/?time_interval=weekday'))
-    assert resp.json == {
-        'data': {
-            'series': [{'data': [30, 0, 0, 0, 20, 0, 0], 'label': 'Forms Count'}],
-            'x_labels': ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday'],
-            'subfilters': [],
-        },
-        'err': 0,
-    }
+    assert resp.json['data']['series'] == [{'data': [30, 0, 0, 0, 20, 0, 0], 'label': 'Forms Count'}]
+    assert resp.json['data']['x_labels'] == [
+        'Monday',
+        'Tuesday',
+        'Wednesday',
+        'Thursday',
+        'Friday',
+        'Saturday',
+        'Sunday',
+    ]
 
     resp = get_app(pub).get(sign_uri('/api/statistics/forms/count/?time_interval=hour'))
-    assert resp.json == {
-        'data': {
-            'series': [
-                {
-                    'label': 'Forms Count',
-                    'data': [20, 0, 30, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
-                }
-            ],
-            'x_labels': list(range(24)),
-            'subfilters': [],
-        },
-        'err': 0,
-    }
+    assert resp.json['data']['series'] == [
+        {
+            'data': [20, 0, 30, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+            'label': 'Forms Count',
+        }
+    ]
+    assert resp.json['data']['x_labels'] == list(range(24))
 
     resp = get_app(pub).get(sign_uri('/api/statistics/forms/count/?time_interval=none'))
-    assert resp.json == {
-        'data': {'series': [{'label': 'Forms Count', 'data': [50]}], 'x_labels': [''], 'subfilters': []},
-        'err': 0,
-    }
+    assert resp.json['data']['series'] == [{'data': [50], 'label': 'Forms Count'}]
+    assert resp.json['data']['x_labels'] == ['']
 
     # time_interval=day is not supported
     get_app(pub).get(sign_uri('/api/statistics/forms/count/?time_interval=day'), status=400)
@@ -388,14 +370,8 @@ def test_statistics_forms_count(pub):
 
     # apply period filter
     resp = get_app(pub).get(sign_uri('/api/statistics/forms/count/?end=2021-02-01'))
-    assert resp.json == {
-        'data': {
-            'series': [{'data': [20], 'label': 'Forms Count'}],
-            'x_labels': ['2021-01'],
-            'subfilters': [],
-        },
-        'err': 0,
-    }
+    assert resp.json['data']['series'] == [{'data': [20], 'label': 'Forms Count'}]
+    assert resp.json['data']['x_labels'] == ['2021-01']
 
     # apply channel filter
     resp = get_app(pub).get(sign_uri('/api/statistics/forms/count/?channel=mail'))
@@ -936,6 +912,51 @@ def test_statistics_forms_count_group_by_same_varname(pub, formdef):
     assert resp.json['data']['series'] == [{'data': [2], 'label': 'foo'}]
 
 
+def test_statistics_forms_count_group_by_form(pub):
+    formdef = FormDef()
+    formdef.name = 'A'
+    formdef.store()
+
+    for i in range(10):
+        formdata = formdef.data_class()()
+        formdata.just_created()
+        formdata.receipt_time = datetime.datetime(2022, 1, 1, 0, 0).timetuple()
+        formdata.store()
+
+    formdef = FormDef()
+    formdef.name = 'B'
+    formdef.store()
+
+    for i in range(5):
+        formdata = formdef.data_class()()
+        formdata.just_created()
+        formdata.receipt_time = datetime.datetime(2021, 1, 1, 0, 0).timetuple()
+        formdata.store()
+
+    resp = get_app(pub).get(sign_uri('/api/statistics/forms/count/'))
+    assert len(resp.json['data']['subfilters']) == 1
+    assert resp.json['data']['subfilters'][0] == {
+        'id': 'group-by',
+        'label': 'Group by',
+        'options': [{'id': 'form', 'label': 'Form'}],
+    }
+
+    resp = get_app(pub).get(sign_uri('/api/statistics/forms/count/?time_interval=year'))
+    assert resp.json['data']['x_labels'] == ['2021', '2022']
+    assert resp.json['data']['series'] == [{'data': [5, 10], 'label': 'Forms Count'}]
+
+    resp = get_app(pub).get(sign_uri('/api/statistics/forms/count/?time_interval=year&group-by=form'))
+    assert resp.json['data']['x_labels'] == ['2021', '2022']
+    assert resp.json['data']['series'] == [
+        {'data': [None, 10], 'label': 'A'},
+        {'data': [5, None], 'label': 'B'},
+    ]
+
+    resp = get_app(pub).get(sign_uri('/api/statistics/forms/count/?time_interval=none&group-by=form'))
+    assert resp.json['data']['x_labels'] == ['A', 'B']
+    assert resp.json['data']['series'] == [{'data': [10, 5], 'label': 'Forms Count'}]
+
+
 def test_statistics_cards_count(pub):
     carddef = CardDef()
     carddef.name = 'test 1'
@@ -1010,7 +1031,7 @@ def test_statistics_resolution_time(pub, freezer):
         'series': [
             {
                 'data': [86400.0, 172800.0, 129600.0, 129600.0],
-                'label': 'Time between two statuses',
+                'label': 'Time between "New status" and any final status',
             }
         ],
         'subfilters': [
@@ -1051,6 +1072,7 @@ def test_statistics_resolution_time(pub, freezer):
 
     # specify end status
     resp = get_app(pub).get(sign_uri('/api/statistics/resolution-time/?form=test&end_status=3'))
+    assert resp.json['data']['series'][0]['label'] == 'Time between "New status" and "End status"'
     assert get_humanized_duration_serie(resp.json) == [
         '1 day(s) and 0 hour(s)',
         '1 day(s) and 0 hour(s)',
@@ -1071,6 +1093,7 @@ def test_statistics_resolution_time(pub, freezer):
     resp = get_app(pub).get(
         sign_uri('/api/statistics/resolution-time/?form=test&start_status=2&end_status=4')
     )
+    assert resp.json['data']['series'][0]['label'] == 'Time between "Middle status" and "End status 2"'
     assert get_humanized_duration_serie(resp.json) == [
         '1 day(s) and 0 hour(s)',
         '1 day(s) and 0 hour(s)',

tests/api/test_workflow.py

@@ -79,7 +79,10 @@ def test_workflow_trigger(pub, local_user):
 
     # verify trigger presence (not-404 response)
     formdata.store()  # reset
-    get_app(pub).get(sign_uri(formdata.get_url() + 'jump/trigger/XXX'), status=403)  # not 404: ok
+    resp = get_app(pub).get(
+        sign_uri(formdata.get_url() + 'jump/trigger/XXX'), headers={'accept': 'application/json'}, status=403
+    )  # not 404: ok
+    assert resp.json['err_desc'] == 'wrong HTTP method (must be POST)'
     assert formdef.data_class().get(formdata.id).status == 'wf-st1'
     get_app(pub).get(sign_uri(formdata.get_url() + 'jump/trigger/ABC'), status=404)
     # jump, and then test trigger is not available
@@ -385,8 +388,17 @@ def test_workflow_trigger_http_auth_access(pub, local_user):
     access.store()
 
     app = get_app(pub)
+    app.set_authorization(('Basic', ('test', 'wrong')))
+    resp = app.post(
+        formdata.get_url() + 'jump/trigger/XXX/', headers={'accept': 'application/json'}, status=403
+    )
+    assert resp.json['err_desc'] == 'user not authenticated'
+
     app.set_authorization(('Basic', ('test', '12345')))
-    app.post(formdata.get_url() + 'jump/trigger/XXX/', status=403)
+    resp = app.post(
+        formdata.get_url() + 'jump/trigger/XXX/', headers={'accept': 'application/json'}, status=403
+    )
+    assert resp.json['err_desc'] == 'unsufficient roles'
     assert formdef.data_class().get(formdata.id).status == 'wf-st1'  # no change
 
     access.roles = [role]

tests/backoffice_pages/test_all.py

@@ -917,6 +917,58 @@ def test_backoffice_multi_actions_jump_condition(pub):
             assert formdef.data_class().get(id).status == 'wf-new'
 
 
+@pytest.mark.parametrize('target', [None, 'xxx'])
+def test_backoffice_multi_actions_mistarget_jump(pub, target):
+    create_superuser(pub)
+    FormDef.wipe()
+    Workflow.wipe()
+
+    # add extra jump, with no target
+    workflow = Workflow.get_default_workflow()
+    workflow.id = '2'
+    jump = workflow.get_status('new').add_action('choice', id='_test')
+    jump.label = 'Test'
+    jump.identifier = 'test'
+    jump.by = ['_receiver']
+    jump.status = target
+    workflow.store()
+
+    formdef = FormDef()
+    formdef.name = 'form title'
+    formdef.workflow_roles = {'_receiver': 1}
+    formdef.workflow_id = workflow.id
+    formdef.store()
+    formdef.data_class().wipe()
+
+    formdata = formdef.data_class()()
+    formdata.just_created()
+    formdata.jump_status('new')
+
+    app = login(get_app(pub))
+    resp = app.get('/backoffice/management/form-title/?filter=all')
+    assert 'select[]' in resp.forms['multi-actions'].fields
+    assert len(resp.pyquery.find('#multi-actions div.buttons button')) == 1
+    assert resp.pyquery.find('#multi-actions div.buttons button').text() == 'Test'
+
+    ids = []
+    for checkbox in resp.forms[0].fields['select[]']:
+        if checkbox._value == '_all':
+            continue
+        # check them all
+        ids.append(checkbox._value)
+        checkbox.checked = True
+
+    resp = resp.forms['multi-actions'].submit('button-action-st-new-test-_test')
+    assert '?job=' in resp.location
+    resp = resp.follow()
+    assert 'Executing task "Test" on forms' in resp.text
+    assert '>completed<' in resp.text
+
+    # check status didn't change
+    formdata = formdef.data_class().get(formdata.id)
+    assert formdata.get_status().id == 'new'
+
+
 def test_backoffice_multi_actions_oldest_form(pub):
     create_superuser(pub)
     create_environment(pub)
@@ -2663,17 +2715,25 @@ def test_global_listing_parameters_from_query_string(pub):
     assert resp.forms['listing-settings']['q'].value == 'test'
 
 
-def test_global_listing_user_label(pub):
+@pytest.mark.parametrize('settings_mode', ['new', 'legacy'])
+def test_global_listing_user_label(pub, settings_mode):
     create_user(pub)
     FormDef.wipe()
 
     from wcs.admin.settings import UserFieldsFormDef
 
     user_formdef = UserFieldsFormDef(pub)
-    user_formdef.fields.append(fields.StringField(id='3', label='first_name', type='string'))
-    user_formdef.fields.append(fields.StringField(id='4', label='last_name', type='string'))
+    user_formdef.fields.append(
+        fields.StringField(id='3', label='first_name', type='string', varname='first_name')
+    )
+    user_formdef.fields.append(
+        fields.StringField(id='4', label='last_name', type='string', varname='last_name')
+    )
     user_formdef.store()
-    pub.cfg['users']['field_name'] = ['3', '4']
+    if settings_mode == 'new':
+        pub.cfg['users']['fullname_template'] = '{{ user_var_first_name }} {{ user_var_last_name }}'
+    else:
+        pub.cfg['users']['field_name'] = ['3', '4']
     pub.write_cfg()
 
     formdef = FormDef()

tests/backoffice_pages/test_carddata.py

@@ -617,6 +617,123 @@ def test_backoffice_cards_import_data_csv_no_backoffice_fields(pub):
     assert carddef.data_class().count() == 2
 
 
+def test_backoffice_cards_import_data_csv_blockfield(pub):
+    user = create_user(pub)
+
+    BlockDef.wipe()
+    block = BlockDef()
+    block.name = 'foobar'
+    block.fields = [
+        fields.StringField(id='1', label='Foo', varname='foo'),
+        fields.StringField(id='2', label='Bar', varname='bar'),
+    ]
+    block.digest_template = '{{ block_var_foo }}'
+    block.store()
+
+    CardDef.wipe()
+    carddef = CardDef()
+    carddef.backoffice_submission_roles = user.roles
+    carddef.name = 'test'
+    carddef.fields = [
+        fields.StringField(id='1', label='String'),
+        fields.BlockField(id='2', label='Block', varname='blockdata', type='block:foobar', max_items=2),
+    ]
+    carddef.store()
+
+    app = login(get_app(pub))
+    sample_resp = app.get('/backoffice/data/test/data-sample-csv')
+    assert sample_resp.text.splitlines()[0] == '"String","Block"'
+    assert (
+        sample_resp.text.splitlines()[1]
+        == '"value","will be ignored - type Field Block (foobar) not supported"'
+    )
+
+    # block is required, error
+    data = b'''\
+"String","Block"
+"test","item1"
+"test","item2"
+'''
+    resp = app.get('/backoffice/data/test/import-file')
+    resp.forms[0]['file'] = Upload('test.csv', data, 'text/csv')
+    resp = resp.forms[0].submit()
+    assert 'Block is required but cannot be filled from CSV.' in resp
+
+    # block is not required
+    carddef.fields[1].required = False
+    carddef.store()
+    resp = app.get('/backoffice/data/test/import-file')
+    resp.forms[0]['file'] = Upload('test.csv', data, 'text/csv')
+    resp = resp.forms[0].submit().follow()
+    assert carddef.data_class().count() == 2
+    carddata1, carddata2 = carddef.data_class().select(order_by='id')
+    assert carddata1.data == {'1': 'test', '2': None, '2_display': None}
+    assert carddata2.data == {'1': 'test', '2': None, '2_display': None}
+
+    # required, but max_items = 1
+    carddef.fields[1].required = True
+    carddef.fields[1].max_items = 1
+    carddef.store()
+    sample_resp = app.get('/backoffice/data/test/data-sample-csv')
+    assert sample_resp.text.splitlines()[0] == '"String","Block - Foo","Block - Bar"'
+    assert sample_resp.text.splitlines()[1] == '"value","value","value"'
+    data = b'''\
+"String","Block - Foo","Block - Bar"
+"test","foo1","bar1"
+'''
+    resp = app.get('/backoffice/data/test/import-file')
+    resp.forms[0]['file'] = Upload('test.csv', data, 'text/csv')
+    resp = resp.forms[0].submit().follow()
+    assert carddef.data_class().count() == 3
+    carddata1, carddata2, carddata3 = carddef.data_class().select(order_by='id')
+    assert carddata1.data == {'1': 'test', '2': None, '2_display': None}
+    assert carddata2.data == {'1': 'test', '2': None, '2_display': None}
+    assert carddata3.data == {
+        '1': 'test',
+        '2': {'data': [{'1': 'foo1', '2': 'bar1'}], 'schema': {'1': 'string', '2': 'string'}},
+        '2_display': 'foo1',
+    }
+
+    # not required, with another BlockField
+    carddef.fields[1].required = False
+    carddef.fields.append(
+        fields.BlockField(id='3', label='Block2', varname='blockdata2', type='block:foobar', max_items=1)
+    )
+    carddef.store()
+
+    sample_resp = app.get('/backoffice/data/test/data-sample-csv')
+    assert (
+        sample_resp.text.splitlines()[0]
+        == '"String","Block - Foo","Block - Bar","Block2 - Foo","Block2 - Bar"'
+    )
+    assert sample_resp.text.splitlines()[1] == '"value","value","value","value","value"'
+    data = b'''\
+"String","Block - Foo","Block - Bar","Block2 - Foo","Block2 - Bar"
+"test","foo2","","foo","bar"
+'''
+    resp = app.get('/backoffice/data/test/import-file')
+    resp.forms[0]['file'] = Upload('test.csv', data, 'text/csv')
+    resp = resp.forms[0].submit().follow()
+    assert carddef.data_class().count() == 4
+    carddata1, carddata2, carddata3, carddata4 = carddef.data_class().select(order_by='id')
+    assert carddata1.data == {'1': 'test', '2': None, '2_display': None, '3': None, '3_display': None}
+    assert carddata2.data == {'1': 'test', '2': None, '2_display': None, '3': None, '3_display': None}
+    assert carddata3.data == {
+        '1': 'test',
+        '2': {'data': [{'1': 'foo1', '2': 'bar1'}], 'schema': {'1': 'string', '2': 'string'}},
+        '2_display': 'foo1',
+        '3': None,
+        '3_display': None,
+    }
+    assert carddata4.data == {
+        '1': 'test',
+        '2': {'data': [{'1': 'foo2'}], 'schema': {'1': 'string'}},
+        '2_display': 'foo2',
+        '3': {'data': [{'1': 'foo', '2': 'bar'}], 'schema': {'1': 'string', '2': 'string'}},
+        '3_display': 'foo',
+    }
+
+
 def test_backoffice_cards_import_data_from_json(pub):
     user = create_user(pub)
 

tests/backoffice_pages/test_columns.py

@@ -263,7 +263,8 @@ def test_backoffice_file_column(pub):
     assert '<span>a filename that is too(…).txt</span>' in resp
 
 
-def test_backoffice_user_columns(pub):
+@pytest.mark.parametrize('settings_mode', ['new', 'legacy'])
+def test_backoffice_user_columns(pub, settings_mode):
     pub.user_class.wipe()
     create_superuser(pub)
     pub.role_class.wipe()
@@ -271,10 +272,15 @@ def test_backoffice_user_columns(pub):
     role.store()
 
     user_formdef = UserFieldsFormDef(pub)
-    user_formdef.fields.append(fields.StringField(id='_first_name', label='name', type='string'))
-    user_formdef.fields.append(fields.StringField(id='3', label='test', type='string'))
+    user_formdef.fields.append(
+        fields.StringField(id='_first_name', label='name', type='string', varname='first_name')
+    )
+    user_formdef.fields.append(fields.StringField(id='3', label='test', type='string', varname='last_name'))
     user_formdef.store()
-    pub.cfg['users']['field_name'] = ['3', '4']
+    if settings_mode == 'new':
+        pub.cfg['users']['fullname_template'] = '{{ user_var_last_name }}'
+    else:
+        pub.cfg['users']['field_name'] = ['3', '4']
     pub.write_cfg()
 
     user1 = pub.user_class(name='userA')

tests/backoffice_pages/test_statistics.py

@@ -50,6 +50,27 @@ def test_backoffice_statistics_with_no_formdefs(pub):
     assert 'This site is currently empty.' in resp
 
 
+def test_backoffice_statistics_feature_flag(pub):
+    create_superuser(pub)
+    FormDef.wipe()
+    formdef = FormDef()
+    formdef.name = 'form title'
+    formdef.store()
+
+    app = login(get_app(pub))
+    resp = app.get('/backoffice/management/form-title/')
+    assert 'Statistics' in resp.text
+
+    if not pub.site_options.has_section('options'):
+        pub.site_options.add_section('options')
+
+    pub.site_options.set('options', 'disable-internal-statistics', 'true')
+    with open(os.path.join(pub.app_dir, 'site-options.cfg'), 'w') as fd:
+        pub.site_options.write(fd)
+    resp = app.get('/backoffice/management/form-title/')
+    assert 'Statistics' not in resp.text
+
+
 def test_backoffice_statistics_status_filter(pub):
     create_superuser(pub)
     create_environment(pub)
@@ -156,6 +177,15 @@ def test_global_statistics(pub):
     resp = resp.forms[0].submit()
     assert 'Total count: 20' in resp.text
 
+    if not pub.site_options.has_section('options'):
+        pub.site_options.add_section('options')
+
+    pub.site_options.set('options', 'disable-internal-statistics', 'true')
+    with open(os.path.join(pub.app_dir, 'site-options.cfg'), 'w') as fd:
+        pub.site_options.write(fd)
+    resp = app.get('/backoffice/management/forms')
+    assert 'Global statistics' not in resp
+
 
 def test_backoffice_statistics(pub):
     create_superuser(pub)

tests/form_pages/test_file_field.py

@@ -1,5 +1,6 @@
 import os
 import re
+import urllib.parse
 
 import pytest
 import responses
@@ -332,6 +333,44 @@ def test_form_file_field_submit_wrong_mimetype(pub):
     assert resp.text == '%PDF-1.4 ...'
 
 
+def test_form_file_field_submit_garbage_pdf(pub):
+    FormDef.wipe()
+    formdef = FormDef()
+    formdef.name = 'test'
+    formdef.fields = [
+        fields.FileField(
+            id='0',
+            label='file',
+            document_type={
+                'id': 1,
+                'mimetypes': ['application/pdf'],
+                'label': 'PDF files',
+            },
+        )
+    ]
+    formdef.store()
+    formdef.data_class().wipe()
+
+    upload = Upload('test.pdf', b'x' * 500, 'application/pdf')
+    resp = get_app(pub).get('/test/')
+    resp.forms[0]['f0$file'] = upload
+    resp = resp.forms[0].submit('submit')
+    assert resp.pyquery('#form_error_f0').text() == 'invalid file type'
+
+    upload = Upload('test.pdf', b'x' * 500 + b'%PDF-1.4 ...', 'application/pdf')
+    resp = get_app(pub).get('/test/')
+    resp.forms[0]['f0$file'] = upload
+    resp = resp.forms[0].submit('submit')
+    assert 'Check values then click submit.' in resp.text
+    resp = resp.forms[0].submit('submit').follow()
+    assert 'The form has been recorded' in resp.text
+    resp = resp.click('test.pdf')
+    assert resp.location.endswith('/test.pdf')
+    resp = resp.follow()
+    assert resp.content_type == 'application/pdf'
+    assert '%PDF-1.4' in resp.text
+
+
 def test_form_file_field_submit_blacklist(pub):
     FormDef.wipe()
     formdef = FormDef()
@@ -413,3 +452,37 @@ def test_form_file_field_prefill(pub):
     formdata = formdef.data_class().select()[0]
     assert formdata.data['0'].base_filename == 'qrcode.png'
     assert formdata.data['0'].get_content().startswith(b'\x89PNG')
+
+
+SVG_CONTENT = b'''<?xml version="1.0" encoding="utf-8"?>
+<svg version="1.1" id="Calque_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+viewBox="0 0 63.72 64.25" style="enable-background:new 0 0 63.72 64.25;" xml:space="preserve"> <g> </g> </svg>'''
+
+
+def test_form_file_svg_thumbnail(pub):
+    FormDef.wipe()
+    formdef = FormDef()
+    formdef.name = 'test'
+    formdef.fields = [fields.FileField(id='0', label='file')]
+    formdef.store()
+    formdef.data_class().wipe()
+
+    upload = Upload('test.svg', SVG_CONTENT, 'image/svg+xml')
+
+    app = get_app(pub)
+    resp = app.get('/test/')
+    resp.forms[0]['f0$file'] = upload
+    resp = resp.forms[0].submit('submit')
+    thumbnail_url = resp.pyquery('.fileinfo.thumbnail img')[0].attrib['src']
+    svg_resp = app.get(urllib.parse.urljoin(resp.request.url, thumbnail_url))
+    assert svg_resp.body == SVG_CONTENT
+    assert svg_resp.headers['Content-Type'] == 'image/svg+xml'
+    resp = resp.forms[0].submit('submit')
+    assert resp.status_int == 302
+    resp = resp.follow()
+    assert 'The form has been recorded' in resp.text
+    thumbnail_url = resp.pyquery('.file-field img').attr('src')
+    svg_resp = app.get(urllib.parse.urljoin(resp.request.url, thumbnail_url))
+    svg_resp = svg_resp.follow()
+    assert svg_resp.body == SVG_CONTENT
+    assert svg_resp.headers['Content-Type'] == 'image/svg+xml'

tests/form_pages/test_i18n.py

@@ -5,7 +5,7 @@ import os
 import pytest
 
 from wcs.carddef import CardDef
-from wcs.fields import ItemField, PageField, StringField
+from wcs.fields import ItemField, ItemsField, PageField, StringField
 from wcs.formdef import FormDef
 from wcs.i18n import TranslatableMessage
 from wcs.qommon.http_request import HTTPRequest
@@ -96,6 +96,12 @@ def test_i18n_form(pub, user, emails, http_requests):
             items=['first', 'second', 'third'],
             hint='a second hint text',
         ),
+        ItemsField(
+            id='3',
+            label='mutiple list field',
+            type='items',
+            items=['first', 'second', 'third'],
+        ),
     ]
     formdef.workflow = workflow
     formdef.store()
@@ -108,6 +114,7 @@ def test_i18n_form(pub, user, emails, http_requests):
         ('page field', 'champ page'),
         ('text field', 'champ texte'),
         ('list field', 'champ liste'),
+        ('multiple list field', 'champ liste multiple'),
         ('first', 'premier'),
         ('second', 'deuxième'),
         ('third', 'troisième'),
@@ -133,12 +140,14 @@ def test_i18n_form(pub, user, emails, http_requests):
     assert resp.pyquery('#form_label_f1').text() == 'text field *'
     assert resp.pyquery('[data-field-id="1"] .hint').text() == 'an hint text'
     assert resp.pyquery('select [value=""]').text() == 'a second hint text'
+    assert resp.pyquery('[data-field-id="3"] li:first-child span').text() == 'first'
 
     resp = app.get(formdef.get_url(), headers={'Accept-Language': 'fr'})
     assert resp.pyquery('#steps li.first .label').text() == 'champ page'
     assert resp.pyquery('#form_label_f1').text() == 'champ texte*'
     assert resp.pyquery('[data-field-id="1"] .hint').text() == 'un texte d’aide'
     assert resp.pyquery('select [value=""]').text() == 'un deuxième texte d’aide'
+    assert resp.pyquery('[data-field-id="3"] li:first-child span').text() == 'premier'
 
     resp = app.get(formdef.get_url(), headers={'Accept-Language': 'fr,en;q=0.7,es;q=0.3'})
     assert resp.pyquery('h1').text() == 'formulaire test'
@@ -147,6 +156,7 @@ def test_i18n_form(pub, user, emails, http_requests):
 
     resp.form['f1'] = 'test'
     resp.form['f2'] = 'second'
+    resp.form['f3$element0'] = True
     resp = resp.form.submit('submit', headers={'Accept-Language': 'fr'})
     assert resp.pyquery('#form_label_f1').text() == 'champ texte'
     assert resp.form['f2'].value == 'second'
@@ -389,3 +399,36 @@ def test_form_titles(pub):
     resp = resp.form.submit('submit').follow()
     assert resp.pyquery('title').text().startswith('formulaire de test')
     assert resp.pyquery('h1').text() == 'formulaire de test'
+
+
+def test_form_validation_message(pub):
+    FormDef.wipe()
+    formdef = FormDef()
+    formdef.name = 'test form'
+    formdef.fields = [
+        StringField(
+            id='1',
+            label='text field',
+            type='string',
+            required=False,
+            validation={'type': 'django', 'value': 'False', 'error_message': 'validation error'},
+        ),
+    ]
+    formdef.store()
+    formdef.data_class().wipe()
+
+    TranslatableMessage.wipe()
+    msg = TranslatableMessage()
+    msg.string = 'validation error'
+    msg.string_fr = 'erreur de validation'
+    msg.store()
+
+    resp = get_app(pub).get(formdef.get_url(language='en'))
+    resp.form['f1'] = 'test'
+    resp = resp.form.submit('submit')  # -> validation error
+    assert resp.pyquery('.error').text() == 'validation error'
+
+    resp = get_app(pub).get(formdef.get_url(language='fr'))
+    resp.form['f1'] = 'test'
+    resp = resp.form.submit('submit')  # -> validation error
+    assert resp.pyquery('.error').text() == 'erreur de validation'

tests/test_content_snapshots.py

@@ -19,7 +19,7 @@ from wcs.qommon.ident.password_accounts import PasswordAccount
 from wcs.qommon.upload_storage import PicklableUpload
 from wcs.wf.backoffice_fields import SetBackofficeFieldsWorkflowStatusItem
 from wcs.wf.create_formdata import Mapping
-from wcs.workflows import ContentSnapshotPart, Workflow, WorkflowBackofficeFieldsFormDef
+from wcs.workflows import ContentSnapshotPart, EvolutionPart, Workflow, WorkflowBackofficeFieldsFormDef
 
 from .utilities import clean_temporary_pub, create_temporary_pub, get_app, login
 
@@ -905,6 +905,38 @@ def test_workflow_carddata_edit(pub):
     dt2 = carddata.evolution[1].parts[0].datetime
     assert dt2 > dt1
 
+    # no changes; no new evolution
+    formdata.store()
+    formdata.perform_workflow()
+    carddata.refresh_from_storage()
+    assert len(carddata.evolution) == 2
+
+    # but last evolution has a comment, so add a new evolution
+    carddata.evolution[-1].comment = 'foobar'
+    carddata.store()
+    carddata.refresh_from_storage()
+    formdata.store()
+    formdata.perform_workflow()
+    carddata.refresh_from_storage()
+    assert len(carddata.evolution) == 3
+
+    # last evolution is not empty, but contains only a ContentSnapshotPart; no new evolution
+    part = ContentSnapshotPart(formdata=formdata, old_data={})
+    carddata.evolution[-1].add_part(part)
+    carddata.store()
+    formdata.store()
+    formdata.perform_workflow()
+    carddata.refresh_from_storage()
+    assert len(carddata.evolution) == 3
+
+    # last evolution is not empty, add a new evolution
+    carddata.evolution[-1].add_part(EvolutionPart())
+    carddata.store()
+    formdata.store()
+    formdata.perform_workflow()
+    carddata.refresh_from_storage()
+    assert len(carddata.evolution) == 4
+
 
 def test_workflow_set_backoffice_field(http_requests, pub):
     Workflow.wipe()

tests/test_datasource.py

@@ -185,7 +185,7 @@ def test_python_datasource_errors(pub, error_email, http_requests, emails, caplo
     with open(os.path.join(pub.app_dir, 'site-options.cfg'), 'w') as fd:
         pub.site_options.write(fd)
 
-    # running with disabled python expressions
+    # running with forbidden python expressions
     pub.loggederror_class.wipe()
     datasource = {
         'type': 'formula',
@@ -202,6 +202,25 @@ def test_python_datasource_errors(pub, error_email, http_requests, emails, caplo
     assert logged_error.workflow_id is None
     assert logged_error.summary == 'Unauthorized Python Usage'
 
+    # exception list
+    with open(os.path.join(pub.app_dir, 'allowed-python.txt'), 'w') as fd:
+        fd.write('blah\n')
+
+    pub.loggederror_class.wipe()
+    assert data_sources.get_items(datasource) == []
+    assert pub.loggederror_class.count() == 1
+    assert pub.loggederror_class.select()[0].summary == 'Unauthorized Python Usage'
+
+    with open(os.path.join(pub.app_dir, 'allowed-python.txt'), 'a') as fd:
+        fd.write('%r\n' % ['foo', 'bar'])
+
+    pub.loggederror_class.wipe()
+    assert data_sources.get_items(datasource) == [
+        ('foo', 'foo', 'foo', {'id': 'foo', 'text': 'foo'}),
+        ('bar', 'bar', 'bar', {'id': 'bar', 'text': 'bar'}),
+    ]
+    assert pub.loggederror_class.count() == 0
+
 
 def test_python_datasource_with_evalutils(pub):
     plain_list = [
@@ -359,6 +378,14 @@ def test_jsonvalue_datasource_errors(pub):
         == "[DATASOURCE] JSON data source ('[{\"id\": \"1\", \"text\": \"foo\"}, {\"id\": \"2\", \"text\": \"\"}]') gave a non usable result"
     )
 
+    pub.loggederror_class.wipe()
+    # value not configured
+    datasource = {'type': 'jsonvalue', 'record_on_errors': True}
+    assert data_sources.get_items(datasource) == []
+    assert pub.loggederror_class.count() == 1
+    logged_error = pub.loggederror_class.select()[0]
+    assert logged_error.summary == "[DATASOURCE] JSON data source (None) gave a non usable result"
+
 
 def test_json_datasource(pub, requests_pub):
     get_request().datasources_cache = {}

tests/test_fields.py

@@ -1,5 +1,7 @@
+import json
 import os
 import re
+import time
 
 import pytest
 import responses
@@ -126,6 +128,27 @@ def test_text(pub):
     assert 'rows="12"' in str(form.render())
 
 
+def test_text_anonymise(pub):
+    formdef = FormDef()
+    formdef.name = 'title'
+    formdef.fields = [fields.TextField(id='0', label='comment', type='text', varname='comment')]
+    formdef.store()
+
+    formdata = formdef.data_class()()
+    formdata.just_created()
+    formdata.data = {'0': 'bar'}
+    formdata.anonymise()
+    assert not formdata.data.get('0')
+
+    formdef.fields[0].anonymise = False
+    formdef.store()
+    formdata = formdef.data_class()()
+    formdata.just_created()
+    formdata.data = {'0': 'bar'}
+    formdata.anonymise()
+    assert formdata.data.get('0') == 'bar'
+
+
 def test_email():
     assert (
         fields.EmailField().get_view_value('foo@localhost')
@@ -155,14 +178,14 @@ def test_bool_stats():
     assert re.findall('Yes.*75.*No.*25', str(stats))
 
 
-def test_items():
+def test_items(pub):
     assert fields.ItemsField(items=['a', 'b', 'c']).get_view_value(['a', 'b']) == 'a, b'
     assert fields.ItemsField(items=['a', 'b', 'c']).get_csv_value(['a', 'b']) == ['a', 'b', '']
     assert len(fields.ItemsField(items=['a', 'b', 'c']).get_csv_heading()) == 3
     assert fields.ItemsField(items=['a', 'b', 'c'], max_choices=2).get_csv_value(['a', 'b']) == ['a', 'b']
     assert len(fields.ItemsField(items=['a', 'b', 'c'], max_choices=2).get_csv_heading()) == 2
 
-    field = fields.ItemsField()
+    field = fields.ItemsField(label='plop')
     field.data_source = {
         'type': 'jsonvalue',
         'value': '[{"id": "1", "text": "foo"}, {"id": "2", "text": "bar"}]',
@@ -170,6 +193,45 @@ def test_items():
     assert field.get_options() == [('1', 'foo', '1'), ('2', 'bar', '2')]
     assert field.get_options() == [('1', 'foo', '1'), ('2', 'bar', '2')]  # twice for cached behaviour
 
+    assert field.get_csv_heading() == ['plop (identifier)', 'plop (label)', '(continued)', '(continued)']
+    assert field.get_csv_value(['a', 'b'], structured_value=json.loads(field.data_source['value'])) == [
+        '1',
+        'foo',
+        '2',
+        'bar',
+    ]
+
+    # check values is cut on max choices
+    field.max_choices = 1
+    assert field.get_csv_heading() == ['plop (identifier)', 'plop (label)']
+    assert field.get_csv_value(['a', 'b'], structured_value=json.loads(field.data_source['value'])) == [
+        '1',
+        'foo',
+    ]
+
+    # check empty columns are added if necessary
+    field._cached_data_source = None
+    field.max_choices = None
+    field.data_source[
+        'value'
+    ] = '[{"id": "1", "text": "foo"}, {"id": "2", "text": "bar"}, {"id": "3", "text": "baz"}]'
+    assert field.get_csv_heading() == [
+        'plop (identifier)',
+        'plop (label)',
+        '(continued)',
+        '(continued)',
+        '(continued)',
+        '(continued)',
+    ]
+    assert field.get_csv_value(['a', 'b'], structured_value=json.loads(field.data_source['value'])[:2]) == [
+        '1',
+        'foo',
+        '2',
+        'bar',
+        '',
+        '',
+    ]
+
     # if labels are available, display using <ul>
     field = fields.ItemsField()
     view_value = field.get_view_value('a, b', value_id=['1', '2'], labels=['a', 'b'])
@@ -709,6 +771,27 @@ def test_date():
     assert fields.DateField().convert_value_from_str('not a date') is None
 
 
+def test_date_anonymise(pub):
+    formdef = FormDef()
+    formdef.name = 'title'
+    formdef.fields = [fields.DateField(id='0', label='date', type='date')]
+    formdef.store()
+
+    formdata = formdef.data_class()()
+    formdata.just_created()
+    formdata.data = {'0': time.strptime('2023-03-28', '%Y-%m-%d')}
+    formdata.anonymise()
+    assert not formdata.data.get('0')
+
+    formdef.fields[0].anonymise = False
+    formdef.store()
+    formdata = formdef.data_class()()
+    formdata.just_created()
+    formdata.data = {'0': time.strptime('2023-03-28', '%Y-%m-%d')}
+    formdata.anonymise()
+    assert formdata.data.get('0') == time.strptime('2023-03-28', '%Y-%m-%d')
+
+
 def test_file_convert_from_anything():
     assert fields.FileField().convert_value_from_anything(None) is None
 

tests/test_formdata.py

@@ -3324,14 +3324,24 @@ def test_string_filters(pub, variable_test_data):
         assert tmpl.render(context) == ''
 
 
-def test_user_label(pub):
+@pytest.mark.parametrize('settings_mode', ['new', 'legacy'])
+def test_user_label(pub, settings_mode):
     from wcs.admin.settings import UserFieldsFormDef
 
     user_formdef = UserFieldsFormDef(pub)
-    user_formdef.fields.append(fields.StringField(id='3', label='first_name', type='string'))
-    user_formdef.fields.append(fields.StringField(id='4', label='last_name', type='string'))
+    user_formdef.fields.append(
+        fields.StringField(id='3', label='first_name', type='string', varname='first_name')
+    )
+    user_formdef.fields.append(
+        fields.StringField(id='4', label='last_name', type='string', varname='last_name')
+    )
     user_formdef.store()
-    pub.cfg['users']['field_name'] = ['3', '4']
+    if settings_mode == 'new':
+        pub.cfg['users'][
+            'fullname_template'
+        ] = '{{ user_var_first_name|default:"" }} {{ user_var_last_name|default:"" }}'
+    else:
+        pub.cfg['users']['field_name'] = ['3', '4']
     pub.write_cfg()
 
     FormDef.wipe()
@@ -3382,14 +3392,24 @@ def test_user_label(pub):
     assert formdef.data_class().get(formdata.id).get_user_label() == 'blah xxx'
 
 
-def test_user_label_from_block(pub):
+@pytest.mark.parametrize('settings_mode', ['new', 'legacy'])
+def test_user_label_from_block(pub, settings_mode):
     from wcs.admin.settings import UserFieldsFormDef
 
     user_formdef = UserFieldsFormDef(pub)
-    user_formdef.fields.append(fields.StringField(id='3', label='first_name', type='string'))
-    user_formdef.fields.append(fields.StringField(id='4', label='last_name', type='string'))
+    user_formdef.fields.append(
+        fields.StringField(id='3', label='first_name', type='string', varname='first_name')
+    )
+    user_formdef.fields.append(
+        fields.StringField(id='4', label='last_name', type='string', varname='last_name')
+    )
     user_formdef.store()
-    pub.cfg['users']['field_name'] = ['3', '4']
+    if settings_mode == 'new':
+        pub.cfg['users'][
+            'fullname_template'
+        ] = '{{ user_var_first_name|default:"" }} {{ user_var_last_name|default:"" }}'
+    else:
+        pub.cfg['users']['field_name'] = ['3', '4']
     pub.write_cfg()
 
     BlockDef.wipe()
@@ -4442,7 +4462,7 @@ def test_fts_phone(pub):
     assert formdef.data_class().count([FtsMatch('+33(0)123456789 bar')]) == 0
     assert formdef.data_class().count([FtsMatch('foo +33(0)123456789')]) == 1
     assert formdef.data_class().count([FtsMatch('bar +33(0)123456789')]) == 0
-    assert formdef.data_class().count([FtsMatch('123456789')]) == 2
+    assert formdef.data_class().count([FtsMatch('123456789')]) == 1
 
     formdata.data = {'1': '+32 2 345 67 89', '2': 'foo'}
     formdata.store()
@@ -4470,3 +4490,64 @@ def test_fts_display_id(pub):
     formdata.store()
 
     assert formdef.data_class().count([FtsMatch('123-4567')]) == 1
+
+
+def test_get_visible_status(pub, local_user):
+    Workflow.wipe()
+    workflow = Workflow(name='test')
+    st_new = workflow.add_status('New')
+    st_finished = workflow.add_status('Finished')
+    workflow.store()
+
+    formdef = FormDef()
+    formdef.workflow_id = workflow.id
+    formdef.name = 'foo'
+    formdef.fields = []
+    formdef.store()
+    formdef.data_class().wipe()
+
+    formdata = formdef.data_class()()
+    formdata.evolution = []
+
+    # create evolution [new, empty, finished, empty]
+    for status in (st_new, None, st_finished, None):
+        evo = Evolution()
+        evo.time = time.localtime()
+        if status:
+            evo.status = 'wf-%s' % status.id
+        formdata.evolution.append(evo)
+
+    formdata.store()
+
+    # check visible status is "Finished"
+    formdata = FormDef.get(formdef.id).data_class().get(formdata.id)
+    assert formdata.get_visible_status(user=None).name == 'Finished'
+
+    # mark finished as not visible
+    st_finished.visibility = ['unknown']
+    workflow.store()
+
+    # check "New" is now returned as visible status
+    formdata = FormDef.get(formdef.id).data_class().get(formdata.id)
+    assert formdata.get_visible_status(user=None).name == 'New'
+    assert formdata.get_visible_status(user=local_user).name == 'New'
+
+    # check with user from session
+    pub._request._user = local_user
+    assert formdata.get_visible_status().name == 'New'
+
+    # check from backoffice
+    pub._request.environ['PATH_INFO'] = 'backoffice/test/'
+    assert formdata.get_visible_status().name == 'New'
+
+    # check admin in backoffice gets the real status
+    local_user.is_admin = True
+    local_user.store()
+    assert formdata.get_visible_status().name == 'Finished'
+
+    # check another user in backoffice gets "New"
+    assert formdata.get_visible_status(user=None).name == 'New'
+
+    # check admin in front also gets "New"
+    pub._request.environ['PATH_INFO'] = ''
+    assert formdata.get_visible_status().name == 'New'

tests/test_formdef.py

@@ -12,6 +12,7 @@ from django.utils.encoding import force_bytes
 
 from wcs import fields
 from wcs.admin.settings import UserFieldsFormDef
+from wcs.applications import Application
 from wcs.blocks import BlockDef
 from wcs.carddef import CardDef
 from wcs.fields import DateField, ItemField, StringField
@@ -500,6 +501,21 @@ def test_unused_file_removal_job(pub):
             # 1 attachment
             assert len(glob.glob(os.path.join(pub.app_dir, 'unused-files/attachments/*/*'))) == 1
 
+    application = Application()
+    application.name = 'App 1'
+    application.slug = 'app-1'
+    application.icon = PicklableUpload('icon.png', 'image/png')
+    application.icon.receive([b'foobar'])
+    application.version_number = '1'
+    application.store()
+    assert application.icon.qfilename in os.listdir(os.path.join(pub.app_dir, 'uploads'))
+    clean_unused_files(pub)
+    assert application.icon.qfilename in os.listdir(os.path.join(pub.app_dir, 'uploads'))
+
+    Application.remove_object(application.id)
+    clean_unused_files(pub)
+    assert application.icon.qfilename not in os.listdir(os.path.join(pub.app_dir, 'uploads'))
+
     # unknown unused-files-behaviour: do nothing
     pub.site_options.set('options', 'unused-files-behaviour', 'foo')
     formdata = formdef.data_class()()

tests/test_formdef_import.py

@@ -275,6 +275,14 @@ def test_workflow_options_with_date(pub):
     assert formdef.workflow_options['foo'] == fd2.workflow_options['foo']
 
 
+def test_workflow_options_with_boolean(pub):
+    formdef = FormDef()
+    formdef.name = 'foo'
+    formdef.workflow_options = {'foo': True, 'bar': False}
+    fd2 = assert_xml_import_export_works(formdef)
+    assert formdef.workflow_options['foo'] == fd2.workflow_options['foo']
+
+
 def test_workflow_reference(pub):
     Workflow.wipe()
     FormDef.wipe()

tests/test_publisher.py

@@ -63,34 +63,30 @@ def test_plaintext_error():
     except Exception:
         exc_type, exc_value, tb = sys.exc_info()
     req.form = {'foo': 'bar'}
-    assert pub.USE_LONG_TRACES is True  # use long traces by default
     s = pub._generate_plaintext_error(req, None, exc_type, exc_value, tb)
     assert re.findall('^foo.*bar', s, re.MULTILINE)
     assert re.findall('^SERVER_NAME.*www.example.net', s, re.MULTILINE)
     assert re.findall('File.*?line.*?in test_plaintext_error', s)
     assert re.findall(r'^>.*\d+.*s = pub._generate_plaintext_error', s, re.MULTILINE)
 
-    pub.USE_LONG_TRACES = False
-    s = pub._generate_plaintext_error(req, None, exc_type, exc_value, tb)
-    assert re.findall('^foo.*bar', s, re.MULTILINE)
-    assert re.findall('^SERVER_NAME.*www.example.net', s, re.MULTILINE)
-    assert re.findall('File.*?line.*?in test_plaintext_error', s)
-    assert not re.findall(r'^>.*\d+.*s = pub._generate_plaintext_error', s, re.MULTILINE)
-
 
 def test_finish_failed_request():
-    pub.USE_LONG_TRACES = False
-
     req = get_request()
     pub._set_request(req)
-    body = pub.finish_failed_request()
-    assert '<h1>Internal Server Error</h1>' in str(body)
+    try:
+        raise Exception()
+    except Exception:
+        body = pub.finish_failed_request()
+        assert '<h1>Internal Server Error</h1>' in str(body)
 
     req = get_request()
     pub._set_request(req)
     req.form = {'format': 'json'}
-    body = pub.finish_failed_request()
-    assert body == '{"err": 1}'
+    try:
+        raise Exception('test')
+    except Exception:
+        body = pub.finish_failed_request()
+        assert body == '{"err": 1}'
 
     req = get_request()
     pub.cfg['debug'] = {'debug_mode': True}
@@ -98,11 +94,15 @@ def test_finish_failed_request():
     pub.set_config(request=req)
     pub._set_request(req)
     try:
+        secret = 'toto'  # noqa pylint: disable=unused-variable
         raise Exception()
     except Exception:
         body = pub.finish_failed_request()
-        assert 'Traceback (most recent call last)' in str(body)
-        assert '<div class="error-page">' not in str(body)
+        assert 'Stack trace (most recent call first)' in str(body)
+        # split looked up string so its occurence in the stacktrace doesn't count
+        assert str('to' + 'to') not in str(body)
+        assert str('secret = ' + "'********************'") in str(body)
+        assert str('<div ' + 'class="error-page">') not in str(body)
 
 
 def test_finish_interrupted_request():

tests/test_snapshots.py

@@ -1239,25 +1239,22 @@ def test_snapshots_test_results(pub):
     testdef.name = 'First test'
     testdef.store()
 
-    # add field
-    resp = app.get('/backoffice/forms/1/fields/')
-    resp.forms[0]['label'] = 'Foobar'
-    resp.forms[0]['type'] = 'string'
-    resp.forms[0].submit().follow()
+    # add field validation
+    resp = app.get('/backoffice/forms/1/fields/1/')
+    resp.form['validation$type'] = 'digits'
+    resp.form.submit('submit').follow()
 
-    # field is required, tests failed
+    # test failed
     resp = app.get('/backoffice/forms/1/history/')
-    assert 'Foobar' in resp.text
     assert '/tests/results/' in resp.text
     assert len(resp.pyquery('span.test-failure')) == 1
     assert len(resp.pyquery('span.test-success')) == 0
 
-    # make field optional
-    resp = app.get('/backoffice/forms/1/fields/2/')
-    resp.form['required'] = False
+    # remove validation
+    resp = app.get('/backoffice/forms/1/fields/1/')
+    resp.form['validation$type'] = ''
     resp.form.submit('submit').follow()
 
     resp = app.get('/backoffice/forms/1/history/')
-    assert 'Foobar' in resp.text
     assert len(resp.pyquery('span.test-failure')) == 1
     assert len(resp.pyquery('span.test-success')) == 1

tests/test_sql.py

@@ -457,14 +457,22 @@ def test_sql_fts_index_with_missing_block(formdef):
     formdata.store()
 
 
-def test_sql_fts_index_with_missing_block_and_user_fields_config(pub, formdef):
+@pytest.mark.parametrize('settings_mode', ['new', 'legacy'])
+def test_sql_fts_index_with_missing_block_and_user_fields_config(pub, formdef, settings_mode):
     from wcs.admin.settings import UserFieldsFormDef
 
     user_formdef = UserFieldsFormDef(pub)
-    user_formdef.fields.append(fields.StringField(id='3', label='first_name', type='string'))
-    user_formdef.fields.append(fields.StringField(id='4', label='last_name', type='string'))
+    user_formdef.fields.append(
+        fields.StringField(id='3', label='first_name', type='string', varname='first_name')
+    )
+    user_formdef.fields.append(
+        fields.StringField(id='4', label='last_name', type='string', varname='last_name')
+    )
     user_formdef.store()
-    pub.cfg['users']['field_name'] = ['3', '4']
+    if settings_mode == 'new':
+        pub.cfg['users']['fullname_template'] = '{{ user_var_first_name }} {{ user_var_last_name }}'
+    else:
+        pub.cfg['users']['field_name'] = ['3', '4']
     pub.write_cfg()
 
     data_class = formdef.data_class(mode='sql')

tests/test_testdef.py

@@ -245,11 +245,11 @@ def test_validation_required_field(pub):
 
     testdef = TestDef.create_from_formdata(formdef, formdata)
     testdef.run(formdef)
+    assert testdef.missing_required_fields == []
 
     formdef.fields[0].required = True
-    with pytest.raises(TestError) as excinfo:
-        testdef.run(formdef)
-    assert str(excinfo.value) == 'Empty value for field "Text": required field.'
+    testdef.run(formdef)
+    assert testdef.missing_required_fields == ['Text']
 
 
 def test_validation_item_field(pub):
@@ -273,9 +273,8 @@ def test_validation_item_field(pub):
 
     formdata.data['1'] = None
     testdef = TestDef.create_from_formdata(formdef, formdata)
-    with pytest.raises(TestError) as excinfo:
-        testdef.run(formdef)
-    assert str(excinfo.value) == 'Empty value for field "Test": required field.'
+    testdef.run(formdef)
+    assert testdef.missing_required_fields == ['Test']
 
 
 def test_validation_item_field_inside_block(pub):
@@ -502,9 +501,8 @@ def test_validation_items_field(pub):
 
     formdata.data['1'] = []
     testdef = TestDef.create_from_formdata(formdef, formdata)
-    with pytest.raises(TestError) as excinfo:
-        testdef.run(formdef)
-    assert str(excinfo.value) == 'Empty value for field "Test": required field.'
+    testdef.run(formdef)
+    assert testdef.missing_required_fields == ['Test']
 
 
 def test_validation_email_field(pub):
@@ -632,17 +630,24 @@ def test_validation_file_field(pub):
     testdef = TestDef.create_from_formdata(formdef, formdata)
     testdef.run(formdef)
 
+    # test against empty value
+    formdata.data['1'] = None
+    testdef = TestDef.create_from_formdata(formdef, formdata)
+    testdef.run(formdef)
+    assert testdef.missing_required_fields == ['File']
+
+    # test with filename that will negate next field condition
     upload = PicklableUpload('hop.pdf', 'application/pdf', 'ascii')
     upload.receive([b'first line', b'second line'])
     formdata.data['1'] = upload
     testdef = TestDef.create_from_formdata(formdef, formdata)
-    with pytest.raises(TestError) as excinfo:
-        testdef.run(formdef)
-    assert str(excinfo.value) == 'Empty value for field "Text": required field.'
+    testdef.run(formdef)
+    assert testdef.missing_required_fields == ['Text']
 
     formdata.data['2'] = 'xxx'
     testdef = TestDef.create_from_formdata(formdef, formdata)
     testdef.run(formdef)
+    assert testdef.missing_required_fields == []
 
     pub.site_options.set('options', 'blacklisted-file-types', 'application/pdf')
     testdef = TestDef.create_from_formdata(formdef, formdata)
@@ -868,3 +873,52 @@ def test_computed_field_value_too_long(pub):
 
     testdef = TestDef.create_from_formdata(formdef, formdata)
     testdef.run(formdef)
+
+
+def test_expected_error(pub):
+    formdef = FormDef()
+    formdef.name = 'test title'
+    formdef.fields = [
+        fields.PageField(
+            id='0',
+            label='1st page',
+            type='page',
+            post_conditions=[
+                {
+                    'condition': {'type': 'django', 'value': 'form_var_text|length > 5'},
+                    'error_message': 'Not enough chars.',
+                }
+            ],
+        ),
+        fields.StringField(id='1', label='Text', varname='text', validation={'type': 'digits'}),
+    ]
+    formdef.store()
+
+    formdata = formdef.data_class()()
+    formdata.just_created()
+    formdata.receipt_time = datetime.datetime(2021, 1, 1, 0, 0).timetuple()
+    formdata.data['1'] = '123456'
+
+    testdef = TestDef.create_from_formdata(formdef, formdata)
+    testdef.run(formdef)
+
+    formdata.data['1'] = '1'
+    testdef = TestDef.create_from_formdata(formdef, formdata)
+    with pytest.raises(TestError) as excinfo:
+        testdef.run(formdef)
+    assert str(excinfo.value) == 'Page 1 post condition was not met (form_var_text|length > 5).'
+
+    testdef = TestDef.create_from_formdata(formdef, formdata)
+    testdef.data['expected_error'] = 'Not enough chars.'
+    testdef.run(formdef)
+
+    testdef = TestDef.create_from_formdata(formdef, formdata)
+    testdef.data['expected_error'] = 'Other error.'
+    with pytest.raises(TestError) as excinfo:
+        testdef.run(formdef)
+    assert str(excinfo.value) == 'Expected error "Other error." but got error "Not enough chars." instead.'
+
+    formdata.data['1'] = 'abcdef'
+    testdef = TestDef.create_from_formdata(formdef, formdata)
+    testdef.data['expected_error'] = 'Only digits are allowed'
+    testdef.run(formdef)

tests/test_widgets.py

@@ -4,6 +4,7 @@ import shutil
 
 import mechanize
 import pytest
+from pyquery import PyQuery
 from quixote import cleanup, get_response
 from quixote.http_request import parse_query
 
@@ -1409,3 +1410,26 @@ def test_condition_widget_no_python():
     assert widget.get_error() == "syntax error: Could not parse the remainder: '{{' from '{{'"
 
     pub.site_options.set('options', 'disable-python-expressions', 'false')
+
+
+def test_emoji_button():
+    # textual button
+    form = Form(use_tokens=False)
+    form.add_submit('submit', 'Submit')
+    assert PyQuery(str(form.render()))('button').attr.name == 'submit'
+    assert not PyQuery(str(form.render()))('button').attr['aria-label']
+    assert PyQuery(str(form.render()))('button').text() == 'Submit'
+
+    # emoji + text
+    form = Form(use_tokens=False)
+    form.add_submit('submit', '✅ Submit')
+    assert PyQuery(str(form.render()))('button').attr.name == 'submit'
+    assert PyQuery(str(form.render()))('button').attr['aria-label'] == 'Submit'
+    assert PyQuery(str(form.render()))('button').text() == '✅ Submit'
+
+    # single emoji (do not do this) (no empty aria-label)
+    form = Form(use_tokens=False)
+    form.add_submit('submit', '✅')
+    assert PyQuery(str(form.render()))('button').attr.name == 'submit'
+    assert not PyQuery(str(form.render()))('button').attr['aria-label']
+    assert PyQuery(str(form.render()))('button').text() == '✅'

tests/utilities.py

@@ -169,6 +169,8 @@ def create_temporary_pub(pickle_mode=False, lazy_mode=False):
         TestDef.do_table()
         TestResult.do_table()
         sql.WorkflowTrace.do_table()
+        sql.Application.do_table()
+        sql.ApplicationElement.do_table()
         sql.init_global_table()
 
         conn.close()

tests/workflow/test_all.py

@@ -63,6 +63,7 @@ from wcs.wf.register_comment import JournalEvolutionPart, RegisterCommenterWorkf
 from wcs.wf.remove import RemoveWorkflowStatusItem
 from wcs.wf.remove_tracking_code import RemoveTrackingCodeWorkflowStatusItem
 from wcs.wf.roles import AddRoleWorkflowStatusItem, RemoveRoleWorkflowStatusItem
+from wcs.wf.sendmail import EmailEvolutionPart
 from wcs.wf.sms import SendSMSWorkflowStatusItem
 from wcs.wf.wscall import WebserviceCallStatusItem
 from wcs.workflows import (
@@ -1018,6 +1019,61 @@ def test_anonymise(pub):
     assert formdata.evolution[0].parts is None
     assert formdata.evolution[1].parts is None
 
+    assert item.render_as_line() == 'Anonymisation'
+    item.unlink_user = True
+    assert item.render_as_line() == 'Anonymisation (only user unlinking)'
+
+
+def test_anonymise_custom_view_user_filtered(pub):
+    CardDef.wipe()
+    FormDef.wipe()
+    Workflow.wipe()
+    pub.custom_view_class.wipe()
+
+    carddef = CardDef()
+    carddef.name = 'foo'
+    carddef.store()
+
+    carddata = carddef.data_class()()
+    carddata.data = {'0': 'FOO BAR 0'}
+    carddata.just_created()
+    carddata.jump_status('new')
+    carddata.store()
+
+    custom_view = pub.custom_view_class()
+    custom_view.title = 'card view'
+    custom_view.formdef = carddef
+    custom_view.columns = {'list': [{'id': '0'}]}
+    custom_view.filters = {'filter-user': 'on', 'filter-user-value': '__current__'}
+    custom_view.visibility = 'datasource'
+    custom_view.store()
+
+    formdef = FormDef()
+    formdef.name = 'baz'
+    formdef.fields = [
+        ItemsField(
+            id='1', label='list', type='items', data_source={'type': 'carddef:foo:card-view'}, anonymise=True
+        ),
+    ]
+    formdef.store()
+
+    formdata = formdef.data_class()()
+    formdata.just_created()
+    formdata.data = {
+        '1': ['foo', 'bar'],
+        '1_display': 'foo, bar',
+    }
+    formdata.store()
+
+    pub._set_request(None)  # must run without request
+    item = AnonymiseWorkflowStatusItem()
+    item.perform(formdata)
+    formdata.refresh_from_storage()
+    assert formdata.data == {
+        '1': None,
+        '1_display': None,
+    }
+
 
 def test_remove(pub):
     formdef = FormDef()
@@ -2326,6 +2382,12 @@ def test_webservice_with_complex_data(http_requests, pub):
     formdata.just_created()
     formdata.store()
 
+    attachment_content = b'hello'
+    formdata.evolution[-1].parts = [
+        AttachmentEvolutionPart('hello.txt', fp=io.BytesIO(attachment_content), varname='testfile')
+    ]
+    formdata.store()
+
     item = WebserviceCallStatusItem()
     item.method = 'POST'
     item.url = 'http://remote.example.net'
@@ -2348,6 +2410,8 @@ def test_webservice_with_complex_data(http_requests, pub):
         'empty_string': '{{ form_var_empty }}',
         'none': '{{ form_var_none }}',
         'bool': '{{ form_var_bool_raw }}',
+        'attachment': '{{ form_attachments_testfile }}',
+        'time': '{{ "13:12"|time }}',
     }
     pub.substitutions.feed(formdata)
     with get_publisher().complex_data():
@@ -2374,6 +2438,12 @@ def test_webservice_with_complex_data(http_requests, pub):
         'empty_string': '',
         'none': None,
         'bool': False,
+        'attachment': {
+            'filename': 'hello.txt',
+            'content_type': 'application/octet-stream',
+            'content': base64.b64encode(attachment_content).decode(),
+        },
+        'time': '13:12:00',
     }
 
     # check an empty boolean field is sent as False
@@ -5598,3 +5668,50 @@ def test_removal_of_obsolete_action_classes(pub):
 
     workflow = Workflow.get(1)
     assert workflow.possible_status[0].items == []
+
+
+def test_parts_are_saved_on_each_action(pub):
+    workflow = Workflow(name='register comment to')
+    workflow.backoffice_fields_formdef = WorkflowBackofficeFieldsFormDef(workflow)
+    workflow.backoffice_fields_formdef.fields = [
+        StringField(id='bo0', varname='bo', type='string', label='bo variable'),
+    ]
+    st0 = workflow.add_status('Status0', 'st0')
+    workflow.add_status('Status1', 'st1')
+
+    item = st0.add_action('set-backoffice-fields', id='set-bo')
+    item.fields = [{'field_id': 'bo0', 'value': 'foobar'}]
+
+    item = st0.add_action('sendmail')
+    item.to = ['_submitter']
+    item.subject = 'Foobar'
+    item.body = 'Hello'
+    item.varname = 'foobar'
+
+    item = st0.add_action('jump')
+    item.status = 'st1'
+
+    workflow.store()
+
+    user = pub.user_class(name='baruser')
+    user.email = 'foo@bar.com'
+    user.store()
+
+    FormDef.wipe()
+    formdef = FormDef()
+    formdef.name = 'foobar'
+    formdef.url_name = 'foobar'
+    formdef._workflow = workflow
+    formdef.store()
+
+    formdata = formdef.data_class()()
+    formdata.user = user
+    formdata.just_created()
+    formdata.store()
+    formdata.perform_workflow()
+    formdata.store()
+
+    formdata = formdef.data_class().get(formdata.id)
+    parts = list(formdata.iter_evolution_parts())
+    assert parts
+    assert any(isinstance(part, EmailEvolutionPart) for part in parts)

tests/workflow/test_carddata.py

@@ -653,6 +653,7 @@ def test_edit_carddata_with_data_sourced_object(pub):
     carddata.data = {'0': 'Foo', '1': 'Bar', '2': 'l'}
     carddata.data['2_display'] = carddef.fields[2].store_display_value(carddata.data, '2')
     carddata.data['2_structured'] = carddef.fields[2].store_structured_value(carddata.data, '2')
+    carddata.just_created()
     carddata.store()
 
     wf = Workflow(name='Card update')
@@ -677,7 +678,6 @@ def test_edit_carddata_with_data_sourced_object(pub):
 
     formdata = formdef.data_class()()
     formdata.data = {'0': '1', '1': 'c'}
-    formdata.store()
     formdata.just_created()
     formdata.store()
     formdata.perform_workflow()
@@ -808,6 +808,7 @@ def test_edit_carddata_manual_targeting(pub):
             '1': 'Last name %s' % i,
             '2': '0',
         }
+        carddata.just_created()
         carddata.store()
 
     # formdef workflow that will update carddata
@@ -1741,3 +1742,39 @@ def test_anonymise_action_unlink_user(pub, submitter_is_triggerer):
         pub.get_request().session.is_anonymous_submitter(carddef.data_class().select()[0])
         is submitter_is_triggerer
     )
+
+
+def test_anonymise_action_unlink_user_no_request(pub):
+    pub._request = None
+
+    CardDef.wipe()
+    FormDef.wipe()
+    pub.user_class.wipe()
+
+    user = pub.user_class()
+    user.email = 'test@example.net'
+    user.name_identifiers = ['xyz']
+    user.store()
+
+    wf = Workflow(name='test-unlink-user')
+    wf.possible_status = Workflow.get_default_workflow().possible_status[:]
+    anonymise = wf.possible_status[1].add_action('anonymise', id='_anonymise', prepend=True)
+    anonymise.label = 'Unlink User'
+    anonymise.varname = 'mycard'
+    anonymise.unlink_user = True
+    wf.store()
+
+    carddef = CardDef()
+    carddef.name = 'Person'
+    carddef.workflow_id = wf.id
+    carddef.store()
+    carddef.data_class().wipe()
+
+    carddata = carddef.data_class()()
+    carddata.data = {'0': 'Foo', '1': 'Bar'}
+    carddata.user_id = user.id
+    carddata.just_created()
+    carddata.store()
+
+    carddata.perform_workflow()
+    assert carddef.data_class().select()[0].user is None

wcs/Defaults.py

@@ -18,4 +18,3 @@ APP_DIR = "/var/lib/wcs"
 DATA_DIR = "/usr/share/wcs"
 ERROR_LOG = None
 REDIRECT_ON_UNKNOWN_VHOST = None
-USE_LONG_TRACES = True

wcs/admin/api_access.py

@@ -41,7 +41,7 @@ class ApiAccessUI:
             self.api_access = ApiAccess()
 
     def get_form(self):
-        form = Form(enctype='multipart/form-data', advanced_label=_('Additional options'))
+        form = Form(enctype='multipart/form-data')
         form.add(StringWidget, 'name', title=_('Name'), required=True, size=30, value=self.api_access.name)
         form.add(
             TextWidget,

wcs/admin/blocks.py

@@ -21,6 +21,7 @@ from quixote.html import TemplateIO, htmltext
 from wcs.admin import utils
 from wcs.admin.categories import BlockCategoriesDirectory, get_categories
 from wcs.admin.fields import FieldDefPage, FieldsDirectory
+from wcs.backoffice.applications import ApplicationsDirectory
 from wcs.backoffice.snapshots import SnapshotsDirectory
 from wcs.blocks import BlockDef, BlockdefImportError
 from wcs.categories import BlockCategory
@@ -125,7 +126,7 @@ class BlockDirectory(FieldsDirectory):
             form.widgets.append(
                 HtmlWidget('<p>%s</p>' % _('You are about to irrevocably delete this block.'))
             )
-            form.add_submit('delete', _('Submit'))
+            form.add_submit('delete', _('Delete'))
         else:
             form.widgets.append(
                 HtmlWidget('<p>%s</p>' % _('This block is still used, it cannot be deleted.'))
@@ -262,13 +263,14 @@ class BlockDirectory(FieldsDirectory):
 
 
 class BlocksDirectory(Directory):
-    _q_exports = ['', 'new', 'categories', ('import', 'p_import')]
+    _q_exports = ['', 'new', 'categories', ('import', 'p_import'), ('application', 'applications_dir')]
     do_not_call_in_templates = True
     categories = BlockCategoriesDirectory()
 
     def __init__(self, section):
         super().__init__()
         self.section = section
+        self.applications_dir = ApplicationsDirectory(BlockDef.xml_root_node)
 
     def _q_traverse(self, path):
         if not get_publisher().get_backoffice_root().is_global_accessible('forms'):
@@ -284,19 +286,35 @@ class BlocksDirectory(Directory):
         return BlockDirectory(self.section, block)
 
     def _q_index(self):
+        from wcs.applications import Application
+
         html_top(self.section, title=_('Fields Blocks'))
+        get_response().add_javascript(['popup.js'])
+        context = {
+            'view': self,
+            'applications': Application.select_for_object_type(BlockDef.xml_root_node),
+            'has_sidebar': True,
+        }
+        blocks = BlockDef.select(order_by='name')
+        Application.populate_objects(blocks)
+        context.update(self.get_list_context(blocks))
+        return template.QommonTemplateResponse(
+            templates=['wcs/backoffice/blocks.html'],
+            context=context,
+            is_django_native=True,
+        )
+
+    def get_list_context(self, blocks):
         categories = BlockCategory.select()
         BlockCategory.sort_by_position(categories)
-        blocks = BlockDef.select(order_by='name')
         if categories:
             categories.append(BlockCategory(_('Misc')))
             for category in categories:
                 category.blocks = [x for x in blocks if x.category_id == category.id]
-
-        return template.QommonTemplateResponse(
-            templates=['wcs/backoffice/blocks.html'],
-            context={'view': self, 'blocks': blocks, 'categories': categories},
-        )
+        return {
+            'blocks': blocks,
+            'categories': categories,
+        }
 
     def new(self):
         form = Form(enctype='multipart/form-data')

wcs/admin/categories.py

@@ -19,6 +19,7 @@ from quixote.directory import Directory
 from quixote.html import TemplateIO, htmltext
 
 from wcs.admin import utils
+from wcs.backoffice.applications import ApplicationsDirectory
 from wcs.backoffice.snapshots import SnapshotsDirectory
 from wcs.blocks import BlockDef
 from wcs.carddef import CardDef, get_cards_graph
@@ -358,7 +359,7 @@ class DataSourceCategoryPage(CategoryPage):
 
 
 class CategoriesDirectory(Directory):
-    _q_exports = ['', 'new', 'update_order']
+    _q_exports = ['', 'new', 'update_order', ('application', 'applications_dir')]
 
     base_section = 'forms'
     category_class = Category
@@ -366,30 +367,30 @@ class CategoriesDirectory(Directory):
     category_page_class = CategoryPage
     category_explanation = _('Categories are used to sort the different forms.')
 
-    def _q_index(self):
-        get_response().add_javascript(['jquery.js', 'jquery-ui.js', 'biglist.js', 'qommon.wysiwyg.js'])
-        html_top('categories', title=_('Categories'))
-        r = TemplateIO(html=True)
+    do_not_call_in_templates = True
 
-        r += htmltext('<div id="appbar">')
-        r += htmltext('<h2>%s</h2>') % _('Categories')
-        r += htmltext('<span class="actions">')
-        r += htmltext('<a class="new-item" href="new" rel="popup">%s</a>') % _('New Category')
-        r += htmltext('</span>')
-        r += htmltext('</div>')
-        r += htmltext('<div class="explanation bo-block"><p>%s</p></div>') % self.category_explanation
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.applications_dir = ApplicationsDirectory(self.category_class.xml_root_node)
+
+    def _q_index(self):
+        from wcs.applications import Application
+
+        get_response().add_javascript(['biglist.js', 'qommon.wysiwyg.js', 'popup.js'])
+        html_top('categories', title=_('Categories'))
         categories = self.category_class.select()
-        r += htmltext('<ul class="biglist sortable" id="category-list">')
         self.category_class.sort_by_position(categories)
-        for category in categories:
-            r += htmltext('<li class="biglistitem" id="itemId_%s">') % category.id
-            r += htmltext('<strong class="label"><a href="%s/">%s</a></strong>') % (
-                category.id,
-                category.name,
-            )
-            r += htmltext('</li>')
-        r += htmltext('</ul>')
-        return r.getvalue()
+        Application.populate_objects(categories)
+        return template.QommonTemplateResponse(
+            templates=['wcs/backoffice/categories.html'],
+            context={
+                'view': self,
+                'categories': categories,
+                'applications': Application.select_for_object_type(self.category_class.xml_root_node),
+                'has_sidebar': True,
+            },
+            is_django_native=True,
+        )
 
     def update_order(self):
         request = get_request()

wcs/admin/comment_templates.py

@@ -20,6 +20,7 @@ from quixote.html import TemplateIO, htmltext
 
 from wcs.admin import utils
 from wcs.admin.categories import CommentTemplateCategoriesDirectory, get_categories
+from wcs.backoffice.applications import ApplicationsDirectory
 from wcs.backoffice.snapshots import SnapshotsDirectory
 from wcs.categories import CommentTemplateCategory
 from wcs.comment_templates import CommentTemplate
@@ -40,10 +41,14 @@ from wcs.qommon.form import (
 
 
 class CommentTemplatesDirectory(Directory):
-    _q_exports = ['', 'new', 'categories', ('import', 'p_import')]
+    _q_exports = ['', 'new', 'categories', ('import', 'p_import'), ('application', 'applications_dir')]
     do_not_call_in_templates = True
     categories = CommentTemplateCategoriesDirectory()
 
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.applications_dir = ApplicationsDirectory(CommentTemplate.xml_root_node)
+
     def _q_traverse(self, path):
         if not get_publisher().get_backoffice_root().is_global_accessible('workflows'):
             raise errors.AccessForbiddenError()
@@ -54,18 +59,35 @@ class CommentTemplatesDirectory(Directory):
         return CommentTemplatePage(component)
 
     def _q_index(self):
+        from wcs.applications import Application
+
         html_top('comment_templates', title=_('Comment Templates'))
+        get_response().add_javascript(['popup.js'])
+        comment_templates = CommentTemplate.select(order_by='name')
+        Application.populate_objects(comment_templates)
+        context = {
+            'view': self,
+            'applications': Application.select_for_object_type(CommentTemplate.xml_root_node),
+            'has_sidebar': True,
+        }
+        context.update(self.get_list_context(comment_templates))
+        return template.QommonTemplateResponse(
+            templates=['wcs/backoffice/comment-templates.html'],
+            context=context,
+            is_django_native=True,
+        )
+
+    def get_list_context(self, comment_templates):
         categories = CommentTemplateCategory.select()
         CommentTemplateCategory.sort_by_position(categories)
-        comment_templates = CommentTemplate.select(order_by='name')
         if categories:
             categories.append(CommentTemplateCategory(_('Misc')))
             for category in categories:
                 category.comment_templates = [x for x in comment_templates if x.category_id == category.id]
-        return template.QommonTemplateResponse(
-            templates=['wcs/backoffice/comment-templates.html'],
-            context={'view': self, 'comment_templates': comment_templates, 'categories': categories},
-        )
+        return {
+            'comment_templates': comment_templates,
+            'categories': categories,
+        }
 
     def new(self):
         form = Form(enctype='multipart/form-data')
@@ -301,7 +323,7 @@ class CommentTemplatePage(Directory):
             form.widgets.append(
                 HtmlWidget('<p>%s</p>' % _('You are about to irrevocably delete this comment template.'))
             )
-            form.add_submit('delete', _('Submit'))
+            form.add_submit('delete', _('Delete'))
         else:
             form.widgets.append(
                 HtmlWidget('<p>%s</p>' % _('This comment template is still used, it cannot be deleted.'))

wcs/admin/data_sources.py

@@ -20,6 +20,7 @@ from quixote.html import TemplateIO, htmltext
 
 from wcs.admin import utils
 from wcs.admin.categories import DataSourceCategoriesDirectory, get_categories
+from wcs.backoffice.applications import ApplicationsDirectory
 from wcs.backoffice.snapshots import SnapshotsDirectory
 from wcs.carddef import CardDef
 from wcs.categories import DataSourceCategory
@@ -437,7 +438,7 @@ class NamedDataSourcePage(Directory):
             form.widgets.append(
                 HtmlWidget('<p>%s</p>' % _('You are about to irrevocably delete this data source.'))
             )
-            form.add_submit('delete', _('Submit'))
+            form.add_submit('delete', _('Delete'))
         else:
             form.widgets.append(
                 HtmlWidget('<p>%s</p>' % _('This datasource is still used, it cannot be deleted.'))
@@ -498,9 +499,14 @@ class NamedDataSourcesDirectory(Directory):
         'categories',
         ('import', 'p_import'),
         ('sync-agendas', 'sync_agendas'),
+        ('application', 'applications_dir'),
     ]
     categories = DataSourceCategoriesDirectory()
 
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.applications_dir = ApplicationsDirectory(NamedDataSource.xml_root_node)
+
     def _q_traverse(self, path):
         if (
             not get_publisher().get_backoffice_root().is_global_accessible('forms')
@@ -512,11 +518,33 @@ class NamedDataSourcesDirectory(Directory):
         return super()._q_traverse(path)
 
     def _q_index(self):
+        from wcs.applications import Application
+
         html_top('datasources', title=_('Data Sources'))
+        get_response().add_javascript(['popup.js'])
+        context = {
+            'view': self,
+            'has_chrono': has_chrono(get_publisher()),
+            'has_users': True,
+            'applications': Application.select_for_object_type(NamedDataSource.xml_root_node),
+            'has_sidebar': True,
+        }
+        data_sources = NamedDataSource.select(order_by='name')
+        Application.populate_objects(data_sources)
+        context.update(self.get_list_context(data_sources))
+        return template.QommonTemplateResponse(
+            templates=['wcs/backoffice/data-sources.html'],
+            context=context,
+            is_django_native=True,
+        )
+
+    def get_list_context(self, objects, application=None):
+        from wcs.applications import Application
+
         data_sources = []
         user_data_sources = []
         agenda_data_sources = []
-        for ds in NamedDataSource.select(order_by='name'):
+        for ds in objects:
             if ds.external == 'agenda':
                 agenda_data_sources.append(ds)
             elif ds.type == 'wcs:users':
@@ -531,18 +559,18 @@ class NamedDataSourcesDirectory(Directory):
                 category.data_sources = [x for x in data_sources if x.category_id == category.id]
         generated_data_sources = list(CardDef.get_carddefs_as_data_source())
         generated_data_sources.sort(key=lambda x: misc.simplify(x[1]))
-        return template.QommonTemplateResponse(
-            templates=['wcs/backoffice/data-sources.html'],
-            context={
-                'data_sources': data_sources,
-                'categories': categories,
-                'user_data_sources': user_data_sources,
-                'has_chrono': has_chrono(get_publisher()),
-                'has_users': True,
-                'agenda_data_sources': agenda_data_sources,
-                'generated_data_sources': generated_data_sources,
-            },
-        )
+        if application:
+            carddefs = application.get_objects_for_object_type(CardDef.xml_root_node, lightweight=True)
+            generated_data_sources = [g for g in generated_data_sources if g[0] in carddefs]
+        else:
+            Application.populate_objects([g[0] for g in generated_data_sources])
+        return {
+            'data_sources': data_sources,
+            'categories': categories,
+            'user_data_sources': user_data_sources,
+            'agenda_data_sources': agenda_data_sources,
+            'generated_data_sources': generated_data_sources,
+        }
 
     def _new(self, url, breadcrumb, title, ds_type=None):
         get_response().breadcrumb.append((url, breadcrumb))

wcs/admin/fields.py

@@ -188,7 +188,7 @@ class FieldDefPage(Directory):
             # add delete_fields checkbox only if the page has fields
             if to_be_deleted:
                 form.add(CheckboxWidget, 'delete_fields', title=_('Also remove all fields from the page'))
-        form.add_submit('delete', _('Submit'))
+        form.add_submit('delete', _('Delete'))
         form.add_submit("cancel", _("Cancel"))
         if form.get_widget('cancel').parse():
             return self.redirect_field_anchor(self.field)

wcs/admin/forms.py

@@ -23,6 +23,7 @@ from quixote import get_publisher, get_request, get_response, get_session, redir
 from quixote.directory import AccessControlled, Directory
 from quixote.html import TemplateIO, htmlescape, htmltext
 
+from wcs.backoffice.applications import ApplicationsDirectory
 from wcs.backoffice.snapshots import SnapshotsDirectory
 from wcs.carddef import CardDef
 from wcs.categories import Category
@@ -130,7 +131,9 @@ class FormDefUI:
             formdef = self.formdef
         else:
             formdef = self.formdef_class()
-        form.add(StringWidget, 'name', title=_('Name'), required=True, size=40, value=formdef.name)
+        form.add(
+            StringWidget, 'name', title=_('Name'), required=True, size=40, value=formdef.name, maxlength=250
+        )
         categories = self.get_categories()
         if categories:
             if is_global_accessible(self.section):
@@ -1073,7 +1076,14 @@ class FormDefPage(Directory):
             # if name and url name are in sync, keep them that way
             kwargs['data-slug-sync'] = 'url_name'
         form.add(
-            StringWidget, 'name', title=_('Name'), required=True, size=40, value=self.formdef.name, **kwargs
+            StringWidget,
+            'name',
+            title=_('Name'),
+            required=True,
+            size=40,
+            value=self.formdef.name,
+            maxlength=250,
+            **kwargs,
         )
 
         disabled_url_name = bool(self.formdef.data_class().count())
@@ -1260,7 +1270,7 @@ class FormDefPage(Directory):
 
     def duplicate(self):
         form = Form(enctype='multipart/form-data')
-        name_widget = form.add(StringWidget, 'name', title=_('Name'), required=True, size=30)
+        name_widget = form.add(StringWidget, 'name', title=_('Name'), required=True, size=30, maxlength=250)
         form.add_submit('submit', _('Submit'))
         form.add_submit('cancel', _('Cancel'))
         if form.get_widget('cancel').parse():
@@ -1703,7 +1713,15 @@ class NamedDataSourcesDirectoryInForms(NamedDataSourcesDirectory):
 
 
 class FormsDirectory(AccessControlled, Directory):
-    _q_exports = ['', 'new', ('import', 'p_import'), 'blocks', 'categories', ('data-sources', 'data_sources')]
+    _q_exports = [
+        '',
+        'new',
+        ('import', 'p_import'),
+        'blocks',
+        'categories',
+        ('data-sources', 'data_sources'),
+        ('application', 'applications_dir'),
+    ]
 
     category_class = Category
     categories = CategoriesDirectory()
@@ -1727,6 +1745,10 @@ class FormsDirectory(AccessControlled, Directory):
         'Do note it is disabled by default.'
     )
 
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.applications_dir = ApplicationsDirectory(self.formdef_class.xml_root_node)
+
     def html_top(self, title):
         return html_top(self.section, title)
 
@@ -1748,16 +1770,33 @@ class FormsDirectory(AccessControlled, Directory):
         return False
 
     def _q_index(self):
-        self.html_top(title=self.top_title)
-        get_response().add_javascript(['widget_list.js', 'select2.js'])
+        from wcs.applications import Application
 
+        self.html_top(title=self.top_title)
+        get_response().add_javascript(['widget_list.js', 'select2.js', 'popup.js'])
+
+        context = {
+            'view': self,
+            'has_roles': bool(get_publisher().role_class.count()),
+            'applications': Application.select_for_object_type(self.formdef_class.xml_root_node),
+            'has_sidebar': True,
+        }
+        formdefs = self.formdef_class.select(order_by='name', ignore_errors=True, lightweight=True)
+        Application.populate_objects(formdefs)
+        context.update(self.get_list_context(formdefs))
+        context.update(self.get_extra_index_context_data())
+
+        return template.QommonTemplateResponse(
+            templates=[self.index_template_name], context=context, is_django_native=True
+        )
+
+    def get_list_context(self, formdefs):
         global_access = is_global_accessible(self.section)
 
         categories = self.category_class.select()
         self.category_class.sort_by_position(categories)
         categories.append(self.category_class(_('Misc')))
 
-        formdefs = self.formdef_class.select(order_by='name', ignore_errors=True, lightweight=True)
         has_form_with_category_set = False
         for category in categories:
             if not global_access:
@@ -1777,15 +1816,10 @@ class FormsDirectory(AccessControlled, Directory):
             # no form with a category set, do not display "Misc" title
             categories[-1].name = None
 
-        context = {
-            'view': self,
+        return {
             'objects': formdefs,
             'categories': categories,
-            'has_roles': bool(get_publisher().role_class.count()),
         }
-        context.update(self.get_extra_index_context_data())
-
-        return template.QommonTemplateResponse(templates=[self.index_template_name], context=context)
 
     def get_extra_index_context_data(self):
         return {

wcs/admin/mail_templates.py

@@ -20,6 +20,7 @@ from quixote.html import TemplateIO, htmltext
 
 from wcs.admin import utils
 from wcs.admin.categories import MailTemplateCategoriesDirectory, get_categories
+from wcs.backoffice.applications import ApplicationsDirectory
 from wcs.backoffice.snapshots import SnapshotsDirectory
 from wcs.categories import MailTemplateCategory
 from wcs.mail_templates import MailTemplate
@@ -40,10 +41,14 @@ from wcs.qommon.form import (
 
 
 class MailTemplatesDirectory(Directory):
-    _q_exports = ['', 'new', 'categories', ('import', 'p_import')]
+    _q_exports = ['', 'new', 'categories', ('import', 'p_import'), ('application', 'applications_dir')]
     do_not_call_in_templates = True
     categories = MailTemplateCategoriesDirectory()
 
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.applications_dir = ApplicationsDirectory(MailTemplate.xml_root_node)
+
     def _q_traverse(self, path):
         if not get_publisher().get_backoffice_root().is_global_accessible('workflows'):
             raise errors.AccessForbiddenError()
@@ -54,18 +59,35 @@ class MailTemplatesDirectory(Directory):
         return MailTemplatePage(component)
 
     def _q_index(self):
+        from wcs.applications import Application
+
         html_top('mail_templates', title=_('Mail Templates'))
+        get_response().add_javascript(['popup.js'])
+        mail_templates = MailTemplate.select(order_by='name')
+        Application.populate_objects(mail_templates)
+        context = {
+            'view': self,
+            'applications': Application.select_for_object_type(MailTemplate.xml_root_node),
+            'has_sidebar': True,
+        }
+        context.update(self.get_list_context(mail_templates))
+        return template.QommonTemplateResponse(
+            templates=['wcs/backoffice/mail-templates.html'],
+            context=context,
+            is_django_native=True,
+        )
+
+    def get_list_context(self, mail_templates):
         categories = MailTemplateCategory.select()
         MailTemplateCategory.sort_by_position(categories)
-        mail_templates = MailTemplate.select(order_by='name')
         if categories:
             categories.append(MailTemplateCategory(_('Misc')))
             for category in categories:
                 category.mail_templates = [x for x in mail_templates if x.category_id == category.id]
-        return template.QommonTemplateResponse(
-            templates=['wcs/backoffice/mail-templates.html'],
-            context={'view': self, 'mail_templates': mail_templates, 'categories': categories},
-        )
+        return {
+            'mail_templates': mail_templates,
+            'categories': categories,
+        }
 
     def new(self):
         form = Form(enctype='multipart/form-data')
@@ -307,7 +329,7 @@ class MailTemplatePage(Directory):
             form.widgets.append(
                 HtmlWidget('<p>%s</p>' % _('You are about to irrevocably delete this mail template.'))
             )
-            form.add_submit('delete', _('Submit'))
+            form.add_submit('delete', _('Delete'))
         else:
             form.widgets.append(
                 HtmlWidget('<p>%s</p>' % _('This mail template is still used, it cannot be deleted.'))

wcs/admin/tests.py

@@ -30,12 +30,79 @@ from wcs.qommon.backoffice.listing import pagination_links
 from wcs.qommon.backoffice.menu import html_top
 from wcs.qommon.errors import TraversalError
 from wcs.qommon.form import FileWidget, Form, SingleSelectWidget, StringWidget
-from wcs.qommon.storage import Equal, StrictNotEqual
+from wcs.qommon.storage import Equal
 from wcs.testdef import TestDef, TestError, TestResult
 
 
+class TestEditPage(FormBackofficeEditPage):
+    filling_templates = ['wcs/backoffice/testdata_filling.html']
+    edit_mode_submit_label = _('Save data')
+
+    def __init__(self, *args, testdef, filled, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.testdef = testdef
+        self.edited_data = filled
+        self.edited_data.data['edited_testdef_id'] = self.testdef.id
+        self._q_exports.append(('mark-as-failing', 'mark_as_failing'))
+
+    def _q_index(self):
+        get_response().breadcrumb.append(('edit-data/', _('Edit data')))
+        return super()._q_index()
+
+    def modify_filling_context(self, context, *args, **kwargs):
+        super().modify_filling_context(context, *args, **kwargs)
+
+        form = context['html_form']
+        if form.get_submit() == 'submit':
+            self.testdef.data['expected_error'] = None
+
+        get_response().filter['sidebar'] = self.get_test_sidebar(form)
+
+    def get_test_sidebar(self, form):
+        context = {'testdef': self.testdef, 'mark_as_failing_form': self.get_mark_as_failing_form(form)}
+        return render_to_string('wcs/backoffice/test_edit_sidebar.html', context=context)
+
+    def get_mark_as_failing_form(self, form):
+        errors = form.global_error_messages or []
+
+        if not errors and not form.has_errors():
+            return
+
+        for widget in form.widgets:
+            widget = TestDef.get_error_widget(widget)
+            if widget:
+                errors.append(widget.error)
+
+        if len(errors) != 1:
+            return
+
+        form = Form(enctype='multipart/form-data', action='mark-as-failing', use_tokens=False)
+        form.add_hidden('error', errors[0])
+        form.test_error = errors[0]
+
+        magictoken = get_request().form.get('magictoken')
+        form.add_hidden('magictoken', magictoken)
+
+        form.add_submit('submit', _('Mark as failing'))
+        return form
+
+    def mark_as_failing(self):
+        if not get_request().get_method() == 'POST':
+            raise TraversalError()
+
+        magictoken = get_request().form.get('magictoken')
+        edited_data = self.get_transient_formdata(magictoken)
+
+        testdef = TestDef.create_from_formdata(self.formdef, edited_data)
+        self.testdef.data = testdef.data
+
+        self.testdef.data['expected_error'] = get_request().form.get('error')
+        self.testdef.store()
+        return redirect('..')
+
+
 class TestPage(FormBackOfficeStatusPage):
-    _q_extra_exports = ['delete', 'export', 'edit', ('edit-data', 'edit_data')]
+    _q_extra_exports = ['delete', 'export', 'edit', ('edit-data', 'edit_data'), 'duplicate']
 
     def __init__(self, component, objectdef):
         try:
@@ -46,6 +113,12 @@ class TestPage(FormBackOfficeStatusPage):
         filled = self.testdef.build_formdata(objectdef, include_fields=True)
         super().__init__(objectdef, filled)
 
+    @property
+    def edit_data(self):
+        return TestEditPage(
+            self.formdef.url_name, update_breadcrumbs=False, testdef=self.testdef, filled=self.filled
+        )
+
     def _q_index(self):
         get_response().add_javascript(['select2.js'])
         return super()._q_index()
@@ -68,15 +141,22 @@ class TestPage(FormBackOfficeStatusPage):
         r += htmltext('<div id="appbar">')
         r += htmltext('<h2>%s</h2>') % self.testdef
         r += htmltext('<span class="actions">')
-        r += htmltext('<a href="edit-data">%s</a>') % _('Edit data')
+        r += htmltext('<a href="edit-data/">%s</a>') % _('Edit data')
         r += htmltext('</span>')
         r += htmltext('</div>')
-        r += self.receipt(always_include_user=True, mine=False)
+        if self.testdef.data.get('expected_error'):
+            r += htmltext('<div class="infonotice"><p>%s</p></div>') % _(
+                'This test is expected to fail on error "%s".' % self.testdef.data['expected_error']
+            )
+        if self.testdef.data['fields']:
+            r += self.receipt(always_include_user=True, mine=False)
+        else:
+            r += htmltext('<div class="infonotice"><p>%s</p></div>') % _('This test is empty.')
         return r.getvalue()
 
     def delete(self):
         form = Form(enctype='multipart/form-data')
-        form.add_submit('delete', _('Submit'))
+        form.add_submit('delete', _('Delete'))
         form.add_submit('cancel', _('Cancel'))
 
         if form.get_widget('cancel').parse():
@@ -98,16 +178,6 @@ class TestPage(FormBackOfficeStatusPage):
         )
         return json.dumps(self.testdef.export_to_json())
 
-    def edit_data(self):
-        self.filled.data['edited_testdef_id'] = self.testdef.id
-
-        f = FormBackofficeEditPage(self.formdef.url_name)
-        f.testdef = self.testdef
-        f.edited_data = self.filled
-        f.action_url = 'edit-data'
-
-        return f._q_index()
-
     def edit(self):
         form = Form(enctype='multipart/form-data')
         form.add(StringWidget, 'name', title=_('Name'), required=True, size=50, value=self.testdef.name)
@@ -148,6 +218,38 @@ class TestPage(FormBackOfficeStatusPage):
             self.testdef.store()
             return redirect('.')
 
+    def duplicate(self):
+        form = Form(enctype='multipart/form-data')
+        name_widget = form.add(StringWidget, 'name', title=_('Name'), required=True, size=30)
+        form.add_submit('submit', _('Submit'))
+        form.add_submit('cancel', _('Cancel'))
+        if form.get_widget('cancel').parse():
+            return redirect('.')
+
+        if not form.is_submitted():
+            original_name = self.testdef.name
+            new_name = '%s %s' % (original_name, _('(copy)'))
+            names = [x.name for x in TestDef.select_for_objectdef(self.formdef)]
+            no = 2
+            while new_name in names:
+                new_name = _('%(name)s (copy %(no)d)') % {'name': original_name, 'no': no}
+                no += 1
+            name_widget.set_value(new_name)
+
+        if not form.is_submitted() or form.has_errors():
+            html_top('duplicate_test', title=_('Duplicate test'))
+            r = TemplateIO(html=True)
+            get_response().breadcrumb.append(('duplicate', _('Duplicate')))
+            r += htmltext('<h2>%s</h2>') % _('Duplicate test')
+            r += form.render()
+            return r.getvalue()
+
+        self.testdef.id = None
+        self.testdef.slug = None
+        self.testdef.name = form.get_widget('name').parse()
+        self.testdef.store()
+        return redirect(self.testdef.get_admin_url())
+
 
 class TestsDirectory(Directory):
     _q_exports = ['', 'new', ('import', 'p_import'), 'results']
@@ -167,14 +269,12 @@ class TestsDirectory(Directory):
 
     def _q_index(self):
         context = {
-            'testdefs': TestDef.select(
-                [Equal('object_type', self.objectdef.get_table_name()), Equal('object_id', self.objectdef.id)]
-            ),
-            'formdata': self.objectdef.data_class().select([StrictNotEqual('status', 'draft')]),
+            'testdefs': TestDef.select_for_objectdef(self.objectdef),
             'has_deprecated_fields': any(
                 x.type in ('table', 'table-select', 'tablerows', 'ranked-items')
                 for x in self.objectdef.fields
             ),
+            'has_sidebar': True,
         }
         get_response().add_javascript(['popup.js'])
         return template.QommonTemplateResponse(
@@ -184,13 +284,6 @@ class TestsDirectory(Directory):
     def new(self):
         form = Form(enctype='multipart/form-data')
         form.add(StringWidget, 'name', title=_('Name'), required=True, size=50)
-        formadata_options = [
-            (x.id, '%s - %s' % (x.id_display, x.user or _('Unknown User')))
-            for x in self.objectdef.data_class().select(
-                [StrictNotEqual('status', 'draft')], order_by='-receipt_time'
-            )
-        ]
-        form.add(SingleSelectWidget, 'formdata_id', title=_('Form'), required=True, options=formadata_options)
         form.add_submit('submit', _('Submit'))
         form.add_submit('cancel', _('Cancel'))
 
@@ -203,15 +296,13 @@ class TestsDirectory(Directory):
             r += htmltext('<h2>%s</h2>') % _('New test')
             r += form.render()
             return r.getvalue()
-        else:
-            formdata_id = form.get_widget('formdata_id').parse()
-            formdata = self.objectdef.data_class().get(formdata_id)
 
-            testdef = TestDef.create_from_formdata(self.objectdef, formdata)
-            testdef.name = form.get_widget('name').parse()
-            testdef.store()
+        # create empty test
+        testdef = TestDef.create_from_formdata(self.objectdef, self.objectdef.data_class()())
+        testdef.name = form.get_widget('name').parse()
+        testdef.store()
 
-        return redirect('.')
+        return redirect(testdef.get_admin_url() + 'edit-data/')
 
     def p_import(self):
         form = Form(enctype='multipart/form-data')
@@ -269,9 +360,7 @@ class TestResultPage(Directory):
         return super()._q_traverse(path)
 
     def _q_index(self):
-        testdefs = TestDef.select(
-            [Equal('object_type', self.objectdef.get_table_name()), Equal('object_id', self.objectdef.id)]
-        )
+        testdefs = TestDef.select_for_objectdef(self.objectdef)
         testdefs_by_id = {x.id: x for x in testdefs}
         for test in self.test_result.results:
             if test['id'] in testdefs_by_id:
@@ -346,9 +435,7 @@ class TestsAfterJob(AfterJob):
 
     @staticmethod
     def run_tests(objectdef, reason):
-        testdefs = TestDef.select(
-            [Equal('object_type', objectdef.get_table_name()), Equal('object_id', objectdef.id)]
-        )
+        testdefs = TestDef.select_for_objectdef(objectdef)
         if not testdefs:
             return
 
@@ -369,6 +456,7 @@ class TestsAfterJob(AfterJob):
                 'id': test.id,
                 'name': str(test),
                 'error': getattr(test, 'error', None),
+                'missing_required_fields': test.missing_required_fields,
             }
             for test in testdefs
         ]

wcs/admin/users.py

@@ -48,7 +48,7 @@ class UserUI:
         # them filled by SAML requests
         if not is_idp_managing_user_attributes():
             formdef = get_publisher().user_class.get_formdef()
-            if not formdef or not users_cfg.get('field_name'):
+            if not formdef or not get_publisher().has_user_fullname_config():
                 form.add(StringWidget, 'name', title=_('Name'), required=True, size=30, value=self.user.name)
             if not formdef or not users_cfg.get('field_email'):
                 form.add(
@@ -147,7 +147,7 @@ class UserPage(Directory):
 
         r += htmltext('<div class="form">')
 
-        if not users_cfg.get('field_name'):
+        if not get_publisher().has_user_fullname_config():
             r += htmltext('<div class="title">%s</div>') % _('Name')
             r += htmltext('<div class="StringWidget content">%s</div>') % self.user.name
 

wcs/admin/utils.py

@@ -93,18 +93,17 @@ def snapshot_info_block(snapshot, url_prefix='../../', url_suffix=''):
 
 
 def last_test_result_block(objectdef):
-    from wcs.testdef import TestResult
+    from wcs.testdef import TestDef, TestResult
 
     if not get_publisher().has_site_option('enable-tests'):
         return ''
 
-    test_results = TestResult.select(
-        [
-            Equal('object_type', objectdef.get_table_name()),
-            Equal('object_id', objectdef.id),
-        ],
-        order_by='-id',
-    )
+    criterias = [Equal('object_type', objectdef.get_table_name()), Equal('object_id', objectdef.id)]
+
+    if not TestDef.count(criterias):
+        return ''
+
+    test_results = TestResult.select(criterias, order_by='-id')
 
     if not test_results:
         return ''

wcs/admin/workflows.py

@@ -28,6 +28,7 @@ from quixote.directory import Directory
 from quixote.html import TemplateIO, htmltext
 
 from wcs.admin.categories import WorkflowCategoriesDirectory, get_categories
+from wcs.backoffice.applications import ApplicationsDirectory
 from wcs.backoffice.snapshots import SnapshotsDirectory
 from wcs.carddef import CardDef
 from wcs.categories import WorkflowCategory
@@ -1421,7 +1422,7 @@ class GlobalActionPage(WorkflowStatusPage):
         r += htmltext('<div class="bo-block">')
         r += htmltext('<h2>%s</h2>') % _('Triggers')
         r += (
-            htmltext('<ul id="items-list" class="biglist %s" data-order-function="update_triggers_order">')
+            htmltext('<ul id="triggers-list" class="biglist %s" data-order-function="update_triggers_order">')
             % sortable
         )
         for trigger in self.action.triggers:
@@ -1954,6 +1955,7 @@ class WorkflowsDirectory(Directory):
         ('data-sources', 'data_sources'),
         ('mail-templates', 'mail_templates'),
         ('comment-templates', 'comment_templates'),
+        ('application', 'applications_dir'),
     ]
 
     data_sources = NamedDataSourcesDirectoryInWorkflows()
@@ -1962,6 +1964,10 @@ class WorkflowsDirectory(Directory):
     category_class = WorkflowCategory
     categories = WorkflowCategoriesDirectory()
 
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.applications_dir = ApplicationsDirectory(Workflow.xml_root_node)
+
     def html_top(self, title):
         return html_top('workflows', title)
 
@@ -1983,22 +1989,26 @@ class WorkflowsDirectory(Directory):
         return False
 
     def _q_index(self):
+        from wcs.applications import Application
+
         self.html_top(title=_('Workflows'))
-        r = TemplateIO(html=True)
+        get_response().add_javascript(['popup.js'])
 
-        r += htmltext('<div id="appbar">')
-        r += htmltext('<h2>%s</h2>') % _('Workflows')
-        r += htmltext('<span class="actions">')
-        if is_global_accessible():
-            r += htmltext('<a href="comment-templates/">%s</a>') % _('Comment Templates')
-            r += htmltext('<a href="mail-templates/">%s</a>') % _('Mail Templates')
-            r += htmltext('<a href="data-sources/">%s</a>') % _('Data sources')
-            r += htmltext('<a href="categories/">%s</a>') % _('Categories')
-        r += htmltext('<a href="import" rel="popup">%s</a>') % _('Import')
-        r += htmltext('<a class="new-item" rel="popup" href="new">%s</a>') % _('New Workflow')
-        r += htmltext('</span>')
-        r += htmltext('</div>')
+        context = {
+            'view': self,
+            'is_global_accessible': is_global_accessible(),
+            'applications': Application.select_for_object_type(Workflow.xml_root_node),
+            'has_sidebar': True,
+        }
+        workflows = Workflow.select(order_by='name')
+        Application.populate_objects(workflows)
+        context.update(self.get_list_context(workflows))
 
+        return template.QommonTemplateResponse(
+            templates=['wcs/backoffice/workflows.html'], context=context, is_django_native=True
+        )
+
+    def get_list_context(self, workflow_qs, application=False):
         formdef_workflows = [Workflow.get_default_workflow()]
         workflows_in_formdef_use = set(formdef_workflows[0].id)
         for formdef in FormDef.select(lightweight=True):
@@ -2011,9 +2021,12 @@ class WorkflowsDirectory(Directory):
 
         shared_workflows = []
         unused_workflows = []
-        workflows = formdef_workflows + carddef_workflows
+        if application:
+            workflows = []
+        else:
+            workflows = formdef_workflows + carddef_workflows
 
-        for workflow in Workflow.select(order_by='name'):
+        for workflow in workflow_qs:
             if str(workflow.id) in workflows_in_formdef_use and str(workflow.id) in workflows_in_carddef_use:
                 shared_workflows.append(workflow)
             elif str(workflow.id) in workflows_in_formdef_use:
@@ -2037,63 +2050,51 @@ class WorkflowsDirectory(Directory):
 
         self.category_class.sort_by_position(categories)
 
-        if categories:
-            default_category = WorkflowCategory('Default')
-            default_category.id = '_default_category'
-            for workflow in workflows:
-                if workflow.id in ('_default', '_carddef_default'):
-                    workflow.category_id = default_category.id
-            categories = [default_category] + categories
+        default_category = WorkflowCategory()
+        default_category.id = '_default_category'
+        for workflow in workflows:
+            if workflow.id in ('_default', '_carddef_default'):
+                workflow.category_id = default_category.id
+        categories = [default_category] + categories
 
         if is_global_accessible():
-            categories = categories + [None]
+            if len(categories) > 1:
+                # if there are categorised workflows, add an explicit uncategorised
+                # category
+                uncategorised_category = WorkflowCategory(_('Uncategorised'))
+            else:
+                # otherwise just add a "silent" category
+                uncategorised_category = WorkflowCategory('')
+            uncategorised_category.id = '_uncategorised'
+            categories = categories + [uncategorised_category]
 
-        def workflow_section(r, workflows):
-            r += htmltext('<ul class="objects-list single-links">')
-            for workflow in workflows:
-                if workflow in shared_workflows:
-                    css_class = 'shared-workflow'
-                    usage_label = _('Forms and card models')
-                elif workflow in formdef_workflows:
-                    css_class = 'formdef-workflow'
-                    usage_label = _('Forms')
-                elif workflow in carddef_workflows:
-                    css_class = 'carddef-workflow'
-                    usage_label = _('Card models')
-                else:
-                    css_class = 'unused-workflow'
-                    usage_label = _('Unused')
-                r += htmltext('<li class="%s">' % css_class)
-                r += htmltext('<a href="%s/">%s</a>') % (
-                    workflow.id,
-                    workflow.name,
-                )
-                if usage_label and carddef_workflows:
-                    r += htmltext('<span class="badge">%s</span>') % usage_label
-                r += htmltext('</li>')
-            r += htmltext('</ul>')
+        for workflow in workflows:
+            if workflow in shared_workflows:
+                workflow.css_class = 'shared-workflow'
+                workflow.usage_label = _('Forms and card models')
+            elif workflow in formdef_workflows:
+                workflow.css_class = 'formdef-workflow'
+                workflow.usage_label = _('Forms')
+            elif workflow in carddef_workflows:
+                workflow.css_class = 'carddef-workflow'
+                workflow.usage_label = _('Card models')
+
+        for workflow in unused_workflows:
+            workflow.css_class = 'unused-workflow'
+            if carddef_workflows:
+                workflow.usage_label = _('Unused')
 
         for category in categories:
-            if category is None:
-                category_workflows = [x for x in workflows + unused_workflows if not x.category_id]
+            if category.id == '_uncategorised':
+                category.objects = [x for x in workflows + unused_workflows if not x.category_id]
             else:
-                category_workflows = [
+                category.objects = [
                     x for x in workflows + unused_workflows if x.category_id == str(category.id)
                 ]
-            if category_workflows:
-                if len(categories) > 1:
-                    r += htmltext('<div class="section">')
-                    if category is None:
-                        r += htmltext('<h2>%s</h2>') % _('Uncategorised')
-                    elif category.id == '_default_category':
-                        pass  # no title
-                    else:
-                        r += htmltext('<h2>%s</h2>') % category.name
-                workflow_section(r, category_workflows)
-                if len(categories) > 1:
-                    r += htmltext('</div>')
 
-        return r.getvalue()
+        return {
+            'categories': categories,
+        }
 
     def new(self):
         get_response().breadcrumb.append(('new', _('New')))

wcs/admin/wscalls.py

@@ -19,6 +19,7 @@ from quixote.directory import Directory
 from quixote.html import TemplateIO, htmltext
 
 from wcs.admin import utils
+from wcs.backoffice.applications import ApplicationsDirectory
 from wcs.backoffice.snapshots import SnapshotsDirectory
 from wcs.qommon import _, errors, misc, template
 from wcs.qommon.backoffice.menu import html_top
@@ -181,17 +182,32 @@ class NamedWsCallPage(Directory):
 
 
 class NamedWsCallsDirectory(Directory):
-    _q_exports = ['', 'new', ('import', 'p_import')]
+    _q_exports = ['', 'new', ('import', 'p_import'), ('application', 'applications_dir')]
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.applications_dir = ApplicationsDirectory(NamedWsCall.xml_root_node)
 
     def _q_traverse(self, path):
         get_response().breadcrumb.append(('wscalls/', _('Webservice Calls')))
         return super()._q_traverse(path)
 
     def _q_index(self):
+        from wcs.applications import Application
+
         html_top('wscalls', title=_('Webservice Calls'))
+        get_response().add_javascript(['popup.js'])
+        wscalls = NamedWsCall.select(order_by='name')
+        Application.populate_objects(wscalls)
         return template.QommonTemplateResponse(
             templates=['wcs/backoffice/wscalls.html'],
-            context={'view': self, 'wscalls': NamedWsCall.select(order_by='name')},
+            context={
+                'view': self,
+                'wscalls': wscalls,
+                'applications': Application.select_for_object_type(NamedWsCall.xml_root_node),
+                'has_sidebar': True,
+            },
+            is_django_native=True,
         )
 
     def new(self):

wcs/api.py

@@ -178,8 +178,7 @@ class ApiFormdataPage(FormStatusPage):
             self.formdata.data.update(data)
             self.formdata.store()
 
-            if item.status:
-                self.formdata.jump_status(item.status)
+            if self.formdata.jump_status(item.status):
                 self.formdata.record_workflow_event('api-post-edit-action', action_item_id=item.id)
                 self.formdata.perform_workflow()
             ContentSnapshotPart.take(formdata=self.formdata, old_data=old_data)

wcs/api_export_import.py

@@ -24,6 +24,7 @@ from django.shortcuts import redirect
 from django.urls import reverse
 
 from wcs.api_utils import is_url_signed
+from wcs.applications import Application, ApplicationElement
 from wcs.blocks import BlockDef
 from wcs.carddef import CardDef
 from wcs.categories import (
@@ -39,7 +40,7 @@ from wcs.comment_templates import CommentTemplate
 from wcs.data_sources import NamedDataSource, StubNamedDataSource
 from wcs.formdef import FormDef
 from wcs.mail_templates import MailTemplate
-from wcs.sql import Role
+from wcs.sql import Equal, Role
 from wcs.workflows import Workflow
 from wcs.wscalls import NamedWsCall
 
@@ -247,7 +248,9 @@ class BundleImportJob(AfterJob):
         tar_io = io.BytesIO(self.tar_content)
         with tarfile.open(fileobj=tar_io) as self.tar:
             manifest = json.loads(self.tar.extractfile('manifest.json').read().decode())
-            self.app_name = manifest.get('application')
+            self.application = Application.update_or_create_from_manifest(
+                manifest, self.tar, editable=False, install=False
+            )
 
             # count number of actions
             self.total_count = 0
@@ -260,6 +263,9 @@ class BundleImportJob(AfterJob):
             )
             self.total_count += len([x for x in manifest.get('elements') if x.get('type') in object_types])
 
+            # init cache of application elements, from imported manifest
+            self.application_elements = set()
+
             # first pass on formdef/carddef/blockdef/workflows to create them empty
             # (name and slug); so they can be found for sure in import pass
             for type in ('forms', 'cards', 'blocks', 'workflows'):
@@ -269,6 +275,9 @@ class BundleImportJob(AfterJob):
             for type in object_types:
                 self.install([x for x in manifest.get('elements') if x.get('type') == type])
 
+            # remove obsolete application elements
+            self.unlink_obsolete_objects()
+
     def pre_install(self, elements):
         for element in elements:
             element_klass = klasses[element['type']]
@@ -289,7 +298,8 @@ class BundleImportJob(AfterJob):
             new_object = element_klass()
             new_object.slug = slug
             new_object.name = '[pre-import] %s' % xml_node_text(tree.find('name'))
-            new_object.store(comment=_('Application (%s)') % self.app_name)
+            new_object.store(comment=_('Application (%s)') % self.application.name)
+            self.link_object(new_object)
             self.increment_count()
 
     def install(self, elements):
@@ -304,7 +314,8 @@ class BundleImportJob(AfterJob):
                 if existing_object is None or not hasattr(existing_object, 'id'):
                     raise KeyError()
             except KeyError:
-                new_object.store(comment=_('Application (%s)') % self.app_name)
+                new_object.store(comment=_('Application (%s)') % self.application.name)
+                self.link_object(new_object)
                 self.increment_count()
                 continue
             # replace
@@ -327,9 +338,20 @@ class BundleImportJob(AfterJob):
                     'disabled',
                 ):
                     setattr(new_object, attr, getattr(existing_object, attr))
-            new_object.store(comment=_('Application (%s) update') % self.app_name)
+            new_object.store(comment=_('Application (%s) update') % self.application.name)
+            self.link_object(new_object)
             self.increment_count()
 
+    def link_object(self, obj):
+        element = ApplicationElement.update_or_create_for_object(self.application, obj)
+        self.application_elements.add((element.object_type, element.object_id))
+
+    def unlink_obsolete_objects(self):
+        known_elements = ApplicationElement.select([Equal('application_id', self.application.id)])
+        for element in known_elements:
+            if (element.object_type, element.object_id) not in self.application_elements:
+                ApplicationElement.remove_object(element.id)
+
 
 @signature_required
 def bundle_import(request):
@@ -337,3 +359,59 @@ def bundle_import(request):
     job.store()
     job.run(spool=True)
     return JsonResponse({'err': 0, 'url': job.get_api_status_url()})
+
+
+class BundleDeclareJob(BundleImportJob):
+    def execute(self):
+        object_types = [x for x in klasses if x != 'roles']
+
+        tar_io = io.BytesIO(self.tar_content)
+        with tarfile.open(fileobj=tar_io) as self.tar:
+            manifest = json.loads(self.tar.extractfile('manifest.json').read().decode())
+            self.application = Application.update_or_create_from_manifest(
+                manifest, self.tar, editable=True, install=True
+            )
+
+            # count number of actions
+            self.total_count = len([x for x in manifest.get('elements') if x.get('type') in object_types])
+
+            # init cache of application elements, from manifest
+            self.application_elements = set()
+
+            # declare elements
+            for type in object_types:
+                self.declare([x for x in manifest.get('elements') if x.get('type') == type])
+
+            # remove obsolete application elements
+            self.unlink_obsolete_objects()
+
+    def declare(self, elements):
+        for element in elements:
+            element_klass = klasses[element['type']]
+            element_slug = element['slug']
+            existing_object = element_klass.get_by_slug(element_slug, ignore_errors=True)
+            if existing_object:
+                self.link_object(existing_object)
+            self.increment_count()
+
+
+@signature_required
+def bundle_declare(request):
+    job = BundleDeclareJob(tar_content=request.body)
+    job.store()
+    job.run(spool=True)
+    return JsonResponse({'err': 0, 'url': job.get_api_status_url()})
+
+
+@signature_required
+def unlink(request):
+    if request.method == 'POST' and request.POST.get('application'):
+        applications = Application.select([Equal('slug', request.POST['application'])])
+        if applications:
+            application = applications[0]
+            elements = ApplicationElement.select([Equal('application_id', application.id)])
+            for element in elements:
+                ApplicationElement.remove_object(element.id)
+            Application.remove_object(application.id)
+
+    return JsonResponse({'err': 0})

wcs/applications.py

@@ -0,0 +1,150 @@
+# w.c.s. - web application for online forms
+# Copyright (C) 2005-2023  Entr'ouvert
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, see <http://www.gnu.org/licenses/>.
+
+import collections
+import mimetypes
+
+from quixote import get_publisher
+
+from wcs import sql
+from wcs.qommon.upload_storage import PicklableUpload
+
+
+class Application(sql.Application):
+    id = None
+    slug = None
+    name = None
+    description = None
+    documentation_url = None
+    icon = None
+    version_number = None
+    version_notes = None
+    editable = False
+    visible = True
+    created_at = None
+    updated_at = None
+
+    @classmethod
+    def get_by_slug(cls, slug, ignore_errors=True):
+        objects = cls.select([sql.Equal('slug', slug)])
+        if objects:
+            return objects[0]
+        if ignore_errors:
+            return None
+        raise KeyError(slug)
+
+    @classmethod
+    def update_or_create_from_manifest(cls, manifest, tar, editable=False, install=True):
+        application = cls.get_by_slug(manifest.get('slug'), ignore_errors=True)
+        if application is None:
+            application = cls()
+            application.slug = manifest.get('slug')
+            application.editable = editable
+        application.name = manifest.get('application')
+        application.description = manifest.get('description')
+        application.documentation_url = manifest.get('documentation_url')
+        if manifest.get('icon'):
+            application.icon = PicklableUpload(manifest['icon'], mimetypes.guess_type(manifest['icon'])[0])
+            application.icon.receive([tar.extractfile(manifest['icon']).read()])
+        else:
+            application.icon = None
+        application.version_number = manifest.get('version_number') or 'unknown'
+        application.version_notes = manifest.get('version_notes')
+        if not install:
+            application.editable = editable
+        application.visible = manifest.get('visible', True)
+        application.store()
+        return application
+
+    @classmethod
+    def select_for_object_type(cls, object_type):
+        elements = ApplicationElement.select([sql.Equal('object_type', object_type)])
+        application_ids = [e.application_id for e in elements]
+        return [a for a in cls.get_ids(application_ids, ignore_errors=True, order_by='name') if a.visible]
+
+    @classmethod
+    def populate_objects(cls, objects):
+        object_types = {o.xml_root_node for o in objects}
+        elements = ApplicationElement.select([sql.Contains('object_type', object_types)])
+        elements_by_objects = collections.defaultdict(list)
+        for element in elements:
+            elements_by_objects[(element.object_type, element.object_id)].append(element)
+        application_ids = [e.application_id for e in elements]
+        applications_by_ids = {a.id: a for a in cls.get_ids(application_ids, ignore_errors=True) if a.visible}
+        for obj in objects:
+            applications = []
+            elements = elements_by_objects.get((obj.xml_root_node, obj.id)) or []
+            for element in elements:
+                application = applications_by_ids.get(element.application_id)
+                if application:
+                    applications.append(application)
+            obj._applications = sorted(applications, key=lambda a: a.name)
+
+    @classmethod
+    def load_for_object(cls, obj):
+        elements = ApplicationElement.select(
+            [sql.Equal('object_type', obj.xml_root_node), sql.Equal('object_id', obj.id)]
+        )
+        application_ids = [e.application_id for e in elements]
+        applications_by_ids = {a.id: a for a in cls.get_ids(application_ids, ignore_errors=True) if a.visible}
+        applications = []
+        for element in elements:
+            application = applications_by_ids.get(element.application_id)
+            applications.append(application)
+        obj._applications = sorted(applications, key=lambda a: a.name)
+
+    def get_objects_for_object_type(self, object_type, lightweight=True):
+        elements = ApplicationElement.select(
+            [sql.Equal('application_id', self.id), sql.Equal('object_type', object_type)]
+        )
+        object_ids = [e.object_id for e in elements]
+        select_kwargs = {}
+        if object_type == 'formdef':
+            select_kwargs['lightweight'] = lightweight
+        return (
+            get_publisher()
+            .get_object_class(object_type)
+            .get_ids(object_ids, ignore_errors=True, order_by='name', **select_kwargs)
+        )
+
+
+class ApplicationElement(sql.ApplicationElement):
+    id = None
+    application_id = None
+    object_type = None
+    object_id = None
+    created_at = None
+    updated_at = None
+
+    @classmethod
+    def update_or_create_for_object(cls, application, obj):
+        elements = cls.select(
+            [
+                sql.Equal('application_id', application.id),
+                sql.Equal('object_type', obj.xml_root_node),
+                sql.Equal('object_id', obj.id),
+            ]
+        )
+        if elements:
+            element = elements[0]
+            element.store()
+            return element
+        element = cls()
+        element.application_id = application.id
+        element.object_type = obj.xml_root_node
+        element.object_id = obj.id
+        element.store()
+        return element

wcs/audit.py

@@ -46,6 +46,9 @@ class Audit(sql.Audit):
             user = get_publisher().user_class.get(audit.user_id, ignore_errors=True)
         elif request:
             user = request.get_user()
+            if user and user.is_api_user:
+                # do not audit API calls
+                return
             if user and hasattr(user, 'id'):
                 audit.user_id = user.id
         if request:

wcs/backoffice/applications.py

@@ -0,0 +1,140 @@
+# w.c.s. - web application for online forms
+# Copyright (C) 2005-2023  Entr'ouvert
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, see <http://www.gnu.org/licenses/>.
+
+from quixote import get_response, redirect
+from quixote.directory import Directory
+
+from wcs.qommon import errors, misc, template
+from wcs.qommon.backoffice.menu import html_top
+
+
+class ApplicationsDirectory(Directory):
+    _q_exports = ['']
+
+    def __init__(self, object_type):
+        self.object_type = object_type
+
+    def _q_index(self):
+        return redirect('..')
+
+    def _q_lookup(self, component):
+        from wcs.applications import Application
+
+        application = Application.get_by_slug(component, ignore_errors=True)
+        if not application or not application.visible:
+            raise errors.TraversalError()
+        return ApplicationDirectory(self.object_type, application)
+
+
+class ApplicationDirectory(Directory):
+    _q_exports = ['', 'icon', 'logo']
+
+    formdef_objects_template = 'wcs/backoffice/application_formdefs.html'
+    carddef_objects_template = 'wcs/backoffice/application_formdefs.html'
+    workflow_objects_template = 'wcs/backoffice/application_workflows.html'
+    block_objects_template = 'wcs/backoffice/application_blocks.html'
+    mailtemplate_objects_template = 'wcs/backoffice/application_mailtemplates.html'
+    commenttemplate_objects_template = 'wcs/backoffice/application_commenttemplates.html'
+    datasource_objects_template = 'wcs/backoffice/application_datasources.html'
+    wscall_objects_template = 'wcs/backoffice/application_wscalls.html'
+
+    def __init__(self, object_type, application):
+        self.object_type = object_type
+        self.application = application
+
+    def _q_traverse(self, path):
+        get_response().breadcrumb.append(('application/%s/' % self.application.slug, self.application.name))
+        return super()._q_traverse(path)
+
+    def _q_index(self):
+        html_top('', self.application.name)
+        return template.QommonTemplateResponse(templates=[self.get_template()], context=self.get_context())
+
+    def get_template(self):
+        if hasattr(self, '%s_objects_template' % self.object_type.replace('-', '')):
+            return getattr(self, '%s_objects_template' % self.object_type.replace('-', ''))
+        return 'wcs/backoffice/application_objects.html'
+
+    def get_context(self):
+        context = {
+            'application': self.application,
+        }
+        objects = self.application.get_objects_for_object_type(self.object_type)
+        if hasattr(self, 'get_%s_objects_context' % self.object_type.replace('-', '')):
+            context.update(
+                getattr(self, 'get_%s_objects_context' % self.object_type.replace('-', ''))(objects)
+            )
+        else:
+            context['objects'] = objects
+        return context
+
+    def get_formdef_objects_context(self, objects):
+        from wcs.admin.forms import FormsDirectory
+
+        return FormsDirectory().get_list_context(objects)
+
+    def get_carddef_objects_context(self, objects):
+        from wcs.backoffice.cards import CardsDirectory
+
+        return CardsDirectory().get_list_context(objects)
+
+    def get_workflow_objects_context(self, objects):
+        from wcs.admin.workflows import WorkflowsDirectory
+
+        return WorkflowsDirectory().get_list_context(objects, application=True)
+
+    def get_block_objects_context(self, objects):
+        from wcs.admin.blocks import BlocksDirectory
+
+        return BlocksDirectory(None).get_list_context(objects)
+
+    def get_mailtemplate_objects_context(self, objects):
+        from wcs.admin.mail_templates import MailTemplatesDirectory
+
+        return MailTemplatesDirectory().get_list_context(objects)
+
+    def get_commenttemplate_objects_context(self, objects):
+        from wcs.admin.comment_templates import CommentTemplatesDirectory
+
+        return CommentTemplatesDirectory().get_list_context(objects)
+
+    def get_datasource_objects_context(self, objects):
+        from wcs.admin.data_sources import NamedDataSourcesDirectory
+
+        return NamedDataSourcesDirectory().get_list_context(objects, self.application)
+
+    def icon(self):
+        return self._icon(size=(16, 16))
+
+    def logo(self):
+        return self._icon(size=(64, 64))
+
+    def _icon(self, size):
+        response = get_response()
+
+        if self.application.icon and self.application.icon.can_thumbnail():
+            try:
+                content = misc.get_thumbnail(
+                    self.application.icon.get_fs_filename(),
+                    content_type=self.application.icon.content_type,
+                    size=size,
+                )
+                response.set_content_type('image/png')
+                return content
+            except misc.ThumbnailError:
+                raise errors.TraversalError()
+        else:
+            raise errors.TraversalError()

wcs/backoffice/cards.py

@@ -237,7 +237,7 @@ class CardDefPage(FormDefPage):
 
 
 class CardsDirectory(FormsDirectory):
-    _q_exports = ['', 'new', ('import', 'p_import'), 'categories', 'svg']
+    _q_exports = ['', 'new', ('import', 'p_import'), 'categories', 'svg', ('application', 'applications_dir')]
 
     category_class = CardDefCategory
     categories = CardDefCategoriesDirectory()

wcs/backoffice/data_management.py

@@ -48,7 +48,21 @@ def get_import_csv_fields(carddef):
             data['_user'] = value
 
     # skip non-data fields
-    csv_fields = [x for x in (carddef.fields or []) if isinstance(x, fields.WidgetField)]
+    csv_fields = []
+    for field in carddef.iter_fields(include_block_fields=True, with_backoffice_fields=False):
+        if not isinstance(field, fields.WidgetField):
+            continue
+        if field.key == 'block' and field.max_items == 1:
+            # ignore BlockField if only one item
+            continue
+        block_field = getattr(field, 'block_field', None)
+        if block_field:
+            if block_field.max_items > 1:
+                # ignore fields of BlockField if more than one item
+                continue
+            # complete field label
+            field.label = '%s - %s' % (block_field.label, field.label)
+        csv_fields.append(field)
     if carddef.user_support == 'optional':
         return [UserField()] + csv_fields
     return csv_fields
@@ -301,6 +315,7 @@ class CardPage(FormPage):
             get_response().add_after_job(job)
             if api:
                 return job
+            job.store()
             return redirect(job.get_processing_url())
         else:
             job.execute()
@@ -315,6 +330,7 @@ class CardPage(FormPage):
             get_response().add_after_job(job)
             if api:
                 return job
+            job.store()
             return redirect(job.get_processing_url())
         else:
             job.execute()
@@ -416,6 +432,7 @@ class ImportFromCsvAfterJob(AfterJob):
         for csv_line in self.kwargs['data_lines']:
             data_instance = carddata_class()
             data_instance.data = {}
+            block_data = {}
 
             for i, field in enumerate(carddef_fields):
                 value = csv_line[i].strip()
@@ -425,7 +442,21 @@ class ImportFromCsvAfterJob(AfterJob):
                 # skip unsupported field types
                 if field.convert_value_from_str is None:
                     continue
-                field.set_value(data_instance.data, field.convert_value_from_str(value))
+                block_field = getattr(field, 'block_field', None)
+                if not block_field:
+                    field.set_value(data_instance.data, field.convert_value_from_str(value))
+                    continue
+
+                # field in a BlockField
+                if not block_data.get(block_field.id):
+                    block_data[block_field.id] = {'data': [{}], 'schema': {}, 'block_field': block_field}
+                field.set_value(block_data[block_field.id]['data'][0], field.convert_value_from_str(value))
+                block_data[block_field.id]['schema'][field.id] = field.key
+
+            # fill BlockFields
+            for data in block_data.values():
+                block_field = data.pop('block_field')
+                block_field.set_value(data_instance.data, data)
 
             user_value = data_instance.data.pop('_user', None)
             data_instance.user = self.user_lookup(user_value)

wcs/backoffice/management.py

@@ -251,11 +251,12 @@ class ManagementDirectory(Directory):
     def get_sidebar(self, formdefs):
         r = TemplateIO(html=True)
         r += self.get_lookup_sidebox()
-        r += htmltext('<div class="bo-block">')
-        r += htmltext('<ul id="sidebar-actions">')
-        r += htmltext('<li class="stats"><a href="statistics">%s</a></li>') % _('Global statistics')
-        r += htmltext('</ul>')
-        r += htmltext('</div>')
+        if not get_publisher().has_site_option('disable-internal-statistics'):
+            r += htmltext('<div class="bo-block">')
+            r += htmltext('<ul id="sidebar-actions">')
+            r += htmltext('<li class="stats"><a href="statistics">%s</a></li>') % _('Global statistics')
+            r += htmltext('</ul>')
+            r += htmltext('</div>')
         return r.getvalue()
 
     def lookup(self):
@@ -867,9 +868,13 @@ class FormPage(FormdefDirectoryBase):
             ) % (qs, self.export_data_label)
         if self.formdef.geolocations:
             r += htmltext(' <li><a data-base-href="map" href="map%s">%s</a></li>') % (qs, _('Plot on a Map'))
-        if 'stats' in self._q_exports and (
-            not self.formdef.category
-            or self.formdef.category.has_permission('statistics', get_request().user)
+        if (
+            'stats' in self._q_exports
+            and not get_publisher().has_site_option('disable-internal-statistics')
+            and (
+                not self.formdef.category
+                or self.formdef.category.has_permission('statistics', get_request().user)
+            )
         ):
             r += htmltext(' <li class="stats"><a href="stats">%s</a></li>') % _('Statistics')
 
@@ -1734,9 +1739,7 @@ class FormPage(FormdefDirectoryBase):
 
     def get_criterias_from_query(self, statistics_fields_only=False):
         query_overrides = get_request().form
-        return self.get_view_criterias(
-            query_overrides, request=get_request(), statistics_fields_only=statistics_fields_only
-        )
+        return self.get_view_criterias(query_overrides, statistics_fields_only=statistics_fields_only)
 
     def get_field_allowed_operators(self, field):
         operators = [
@@ -1774,7 +1777,6 @@ class FormPage(FormdefDirectoryBase):
     def get_view_criterias(
         self,
         query_overrides=None,
-        request=None,
         custom_view=None,
         compile_templates=False,
         keep_templates=False,
@@ -1794,6 +1796,8 @@ class FormPage(FormdefDirectoryBase):
         ]
         criterias = []
 
+        request = get_request()
+
         filters_dict = {}
         if self.view:
             filters_dict.update(self.view.get_filters_dict() or {})
@@ -2031,14 +2035,14 @@ class FormPage(FormdefDirectoryBase):
             elif filter_field.type == 'user-id':
                 if filter_field_value == '__current__':
                     context_vars = get_publisher().substitutions.get_context_variables(mode='lazy')
-                    if get_request().is_in_backoffice() and context_vars.get('form'):
+                    if request and request.is_in_backoffice() and context_vars.get('form'):
                         # in case of backoffice submission/edition, take user associated
                         # with the form being submitted/edited, if any.
                         form_user = context_vars.get('form_user')
                         if form_user:
                             filter_field_value = str(form_user.id)
-                    elif isinstance(get_request().user, get_publisher().user_class):
-                        filter_field_value = str(get_request().user.id)
+                    elif request and isinstance(request.user, get_publisher().user_class):
+                        filter_field_value = str(request.user.id)
                     else:
                         filter_field_value = None
                 if filter_field_value in ('__current__', None):
@@ -2048,11 +2052,12 @@ class FormPage(FormdefDirectoryBase):
             elif filter_field.type == 'submission-agent-id':
                 criterias.append(Equal('submission_agent_id', filter_field_value))
             elif filter_field.type == 'user-function':
+                user_object = None
                 if ':' in filter_field_value:
                     filter_field_value, user_id = filter_field_value.split(':', 1)
                     user_object = None if user_id == '__none__' else get_publisher().user_class().get(user_id)
-                else:
-                    user_object = get_request().user
+                elif request:
+                    user_object = request.user
                 criterias.append(
                     ElementIntersects(
                         'workflow_merged_roles_dict',
@@ -2061,8 +2066,8 @@ class FormPage(FormdefDirectoryBase):
                     )
                 )
             elif filter_field.type == 'distance':
-                center_lat = get_request().form.get('center_lat')
-                center_lon = get_request().form.get('center_lon')
+                center_lat = request.form.get('center_lat') if request else None
+                center_lon = request.form.get('center_lon') if request else None
                 if not (center_lat and center_lon):
                     raise RequestError('Distance filter missing a center')
                 center = misc.normalize_geolocation({'lat': center_lat, 'lon': center_lon})
@@ -2488,7 +2493,6 @@ class FormPage(FormdefDirectoryBase):
             offset=offset,
             limit=limit,
         )[0]
-        self.formdef.data_class().load_all_evolutions(items)
         digest_key = 'default'
         if self.view and isinstance(self.formdef, CardDef):
             view_digest_key = 'custom-view:%s' % self.view.get_url_slug()
@@ -2501,6 +2505,8 @@ class FormPage(FormdefDirectoryBase):
         include_submission = get_query_flag('include-submission') or full
         include_workflow = get_query_flag('include-workflow') or full
         include_workflow_data = get_query_flag('include-workflow-data') or full
+        if include_evolution or include_workflow:
+            self.formdef.data_class().load_all_evolutions(items)
         # noqa pylint: disable=too-many-boolean-expressions
         if (
             include_fields
@@ -3456,7 +3462,9 @@ class FormBackOfficeStatusPage(FormStatusPage):
             ('template', '%s / %s' % (_('Template'), _('Django Expression')), 'template'),
             ('html_template', _('HTML Template (WYSIWYG)'), 'html_template'),
         ]
-        if get_publisher().has_site_option('disable-python-expressions'):
+        if get_publisher().has_site_option('disable-python-expressions') or get_publisher().has_site_option(
+            'forbid-python-expressions'
+        ):
             options = [x for x in options if x[0] != 'python-condition']
         form.add(
             RadiobuttonsWidget,
@@ -3475,7 +3483,10 @@ class FormBackOfficeStatusPage(FormStatusPage):
                 'data-dynamic-display-value': 'django-condition',
             },
         )
-        if not get_publisher().has_site_option('disable-python-expressions'):
+        if not (
+            get_publisher().has_site_option('disable-python-expressions')
+            or get_publisher().has_site_option('forbid-python-expressions')
+        ):
             form.add(
                 StringWidget,
                 'python-condition',
@@ -3889,6 +3900,7 @@ class FakeField:
         self.varname = id.replace('-', '_')
         self.contextual_varname = self.varname
         self.store_display_value = None
+        self.store_structured_value = None
         self.addable = addable
         self.include_in_statistics = include_in_statistics
 
@@ -4267,9 +4279,14 @@ class CsvExportAfterJob(AfterJob):
                     elements.append({'field': field, 'value': value, 'native_value': value})
                 continue
             display_value = None
+            structured_value = None
             if field.store_display_value:
                 display_value = data.data.get('%s_display' % field.id) or ''
-            for value in field.get_csv_value(element, display_value=display_value):
+            if field.store_structured_value:
+                structured_value = data.data.get('%s_structured' % field.id)
+            for value in field.get_csv_value(
+                element, display_value=display_value, structured_value=structured_value
+            ):
                 elements.append({'field': field, 'value': value, 'native_value': element})
         return elements
 

wcs/data_sources.py

@@ -415,7 +415,7 @@ def _get_structured_items(data_source, mode=None, raise_on_error=False):
     if data_source.get('type') == 'jsonvalue':
         try:
             value = json.loads(data_source.get('value'))
-        except json.JSONDecodeError:
+        except (json.JSONDecodeError, TypeError):
             get_publisher().record_error(
                 'JSON data source (%r) gave a non usable result' % data_source.get('value'),
                 context='[DATASOURCE]',

wcs/fields.py

@@ -1387,6 +1387,12 @@ class StringField(WidgetField):
             return '%s - %s' % (validation_types.get(validation_type), validation_value)
         return str(validation_types.get(validation_type))
 
+    def i18n_scan(self, base_location):
+        location = '%s%s/' % (base_location, self.id)
+        yield from super().i18n_scan(base_location)
+        if self.validation and self.validation.get('error_message'):
+            yield location, None, self.validation.get('error_message')
+
 
 register_field_class(StringField)
 
@@ -1455,7 +1461,13 @@ class TextField(WidgetField):
         )
 
     def get_admin_attributes(self):
-        return WidgetField.get_admin_attributes(self) + ['cols', 'rows', 'display_mode', 'maxlength']
+        return WidgetField.get_admin_attributes(self) + [
+            'cols',
+            'rows',
+            'display_mode',
+            'maxlength',
+            'anonymise',
+        ]
 
     def convert_value_from_str(self, value):
         return value
@@ -1906,6 +1918,7 @@ class DateField(WidgetField):
             'maximum_date',
             'date_in_the_past',
             'date_can_be_today',
+            'anonymise',
         ]
 
     @classmethod
@@ -2619,7 +2632,7 @@ class ItemsField(WidgetField, ItemFieldMixin):
             self._cached_data_source = [x[:3] for x in data_sources.get_items(self.data_source)]
             return self._cached_data_source[:]
         elif self.items:
-            return [(x, x) for x in self.items]
+            return [(x, get_publisher().translate(x)) for x in self.items]
         else:
             return []
 
@@ -2776,17 +2789,21 @@ class ItemsField(WidgetField, ItemFieldMixin):
         return item_items_stats(self, values)
 
     def get_csv_heading(self):
-        labels = [self.label]
+        if self.data_source:
+            labels = ['%s (%s)' % (self.label, _('identifier')), '%s (%s)' % (self.label, _('label'))]
+            next_columns = [_('(continued)')] * 2
+        else:
+            labels = [self.label]
+            next_columns = [_('(continued)')]
         if self.max_choices:
-            labels.extend([''] * (self.max_choices - 1))
+            labels.extend(next_columns * (self.max_choices - 1))
         elif len(self.get_options()):
-            labels.extend([''] * (len(self.get_options()) - 1))
+            labels.extend(next_columns * (len(self.get_options()) - 1))
         return labels
 
-    def get_csv_value(self, element, **kwargs):
+    def get_csv_value(self, element, structured_value=None, **kwargs):
         values = []
-        for one_value in element:
-            values.append(one_value)
+
         if self.max_choices:
             nb_columns = self.max_choices
         elif len(self.get_options()):
@@ -2794,15 +2811,27 @@ class ItemsField(WidgetField, ItemFieldMixin):
         else:
             nb_columns = 1
 
+        if self.data_source:
+            nb_columns *= 2
+            for one_value in structured_value or []:
+                values.append(one_value.get('id'))
+                values.append(one_value.get('text'))
+        else:
+            for one_value in element:
+                values.append(one_value)
+
         if len(values) > nb_columns:
             # this would happen if max_choices is set after forms were already
             # filled with more values
             values = values[:nb_columns]
         elif len(values) < nb_columns:
             values.extend([''] * (nb_columns - len(values)))
+
         return values
 
     def store_display_value(self, data, field_id, raise_on_error=False):
+        if not data.get(field_id):
+            return ''
         options = self.get_options()
         if not options:
             return ''
@@ -2827,6 +2856,8 @@ class ItemsField(WidgetField, ItemFieldMixin):
         return ', '.join(choices)
 
     def store_structured_value(self, data, field_id, raise_on_error=False):
+        if not data.get(field_id):
+            return
         if not self.data_source:
             return
         try:
@@ -4109,6 +4140,10 @@ class ComputedField(Field):
     def get_real_data_source(self):
         return data_sources.get_real(self.data_source)
 
+    def get_dependencies(self):
+        yield from super().get_dependencies()
+        yield from check_wscalls(self.value_template)
+
 
 register_field_class(ComputedField)
 

wcs/formdata.py

@@ -31,7 +31,6 @@ from quixote.http_request import Upload
 
 from .qommon import _, misc
 from .qommon.evalutils import make_datetime
-from .qommon.publisher import get_cfg
 from .qommon.storage import And, Contains, Intersects, Null, StorableObject
 from .qommon.substitution import CompatibilityNamesDict, Substitutions, invalidate_substitution_cache
 from .qommon.template import Template
@@ -535,9 +534,7 @@ class FormData(StorableObject):
                         # block doesn't exist anymore
                         pass
 
-        users_cfg = get_cfg('users', {})
-        if not self.user_id and users_cfg and users_cfg.get('field_name'):
-            field_name_values = users_cfg.get('field_name')
+        if not self.user_id and get_publisher().has_user_fullname_config():
             form_user_data = {}
             for field in get_all_fields():
                 if not hasattr(field, 'prefill'):
@@ -554,11 +551,11 @@ class FormData(StorableObject):
                                 form_user_data[field.get_prefill_configuration()['value']] = sub_field_data
                     else:
                         form_user_data[field.get_prefill_configuration()['value']] = self.data.get(field.id)
-            user_label = ' '.join(
-                [form_user_data.get(x) for x in field_name_values if isinstance(form_user_data.get(x), str)]
-            )
-            if user_label != self.user_label:
-                self.user_label = user_label
+            user_object = get_publisher().user_class()
+            user_object.form_data = form_user_data
+            user_object.set_attributes_from_formdata(form_user_data)
+            if user_object.name != self.user_label:
+                self.user_label = user_object.name
                 changed = True
 
         if any(fields.values()):
@@ -749,7 +746,9 @@ class FormData(StorableObject):
             return _('Unknown')
         return wf_status.name
 
-    def get_visible_status(self, user):
+    def get_visible_status(self, user=Ellipsis):
+        if user is Ellipsis:
+            user = get_request().user
         if not self.evolution:
             return self.get_status()
         for evo in reversed(self.evolution):
@@ -819,8 +818,13 @@ class FormData(StorableObject):
             if not previous_status:
                 summary = _('Failed to compute previous status')
                 get_publisher().record_error(summary, formdata=self)
-                return
+                return False
             status_id = previous_status.id
+
+        if not self.formdef.workflow.has_status(status_id):
+            # do not jump to undefined or missing status
+            return False
+
         status = 'wf-%s' % status_id
         if not self.evolution:
             self.evolution = []
@@ -834,7 +838,7 @@ class FormData(StorableObject):
             # just update last jump time on last evolution, do not add one
             self.evolution[-1].last_jump_datetime = datetime.datetime.now()
             self.store()
-            return
+            return True
         evo = Evolution(self)
         evo.time = time.localtime()
         evo.status = status
@@ -842,6 +846,7 @@ class FormData(StorableObject):
         self.evolution.append(evo)
         self.status = status
         self.store()
+        return True
 
     def get_url(self, backoffice=False, include_category=False, language=None):
         return '%s%s/' % (
@@ -1288,6 +1293,8 @@ class FormData(StorableObject):
             return self.receipt_time
 
     def set_last_update_time(self, value):
+        if isinstance(value, datetime.datetime):
+            value = value.timetuple()
         self._last_update_time = value
 
     last_update_time = property(get_last_update_time, set_last_update_time)

wcs/formdef.py

@@ -503,7 +503,7 @@ class FormDef(StorableObject):
     def get_all_fields(self):
         return (self.fields or []) + self.workflow.get_backoffice_fields()
 
-    def iter_fields(self, include_block_fields=False):
+    def iter_fields(self, include_block_fields=False, with_backoffice_fields=True):
         def _iter_fields(fields, block_field=None):
             for field in fields:
                 # add contextual_id/contextual_varname attributes
@@ -531,7 +531,11 @@ class FormDef(StorableObject):
                     yield from _iter_fields(field.block.fields, block_field=field)
                     field._block = None  # reset cache
 
-        yield from _iter_fields(self.get_all_fields())
+        if with_backoffice_fields:
+            fields = self.get_all_fields()
+        else:
+            fields = self.fields or []
+        yield from _iter_fields(fields)
 
     def get_widget_fields(self):
         return [field for field in self.fields or [] if isinstance(field, fields.WidgetField)]
@@ -1283,6 +1287,9 @@ class FormDef(StorableObject):
             elif isinstance(option_value, time.struct_time):
                 element.text = time.strftime('%Y-%m-%d', option_value)
                 element.attrib['type'] = 'date'
+            elif isinstance(option_value, bool):
+                element.text = 'true' if option_value else 'false'
+                element.attrib['type'] = 'bool'
             else:
                 pass  # TODO: extend support to other types
 
@@ -1431,6 +1438,8 @@ class FormDef(StorableObject):
             option_value = None
             if option.attrib.get('type') == 'date':
                 option_value = time.strptime(option.text, '%Y-%m-%d')
+            elif option.attrib.get('type') == 'bool':
+                option_value = bool(option.text == 'true')
             elif option.text:
                 option_value = xml_node_text(option)
             elif option.findall('filename'):
@@ -1874,6 +1883,7 @@ class FormDef(StorableObject):
         return odict
 
     def __setstate__(self, dict):
+        super().__setstate__(dict)
         self.__dict__ = dict
         self._workflow = None
         self._start_page = None
@@ -2068,6 +2078,7 @@ def clean_unused_files(publisher, **kwargs):
     known_filenames.update([x for x in glob.glob(os.path.join(publisher.app_dir, 'attachments/*/*'))])
 
     def accumulate_filenames():
+        from wcs.applications import Application
         from wcs.carddef import CardDef
 
         for formdef in FormDef.select(ignore_migration=True) + CardDef.select(ignore_migration=True):
@@ -2085,6 +2096,10 @@ def clean_unused_files(publisher, **kwargs):
                 if is_upload(field_data):
                     yield field_data.get_fs_filename()
 
+        for application in Application.select():
+            if is_upload(application.icon):
+                yield application.icon.get_fs_filename()
+
     used_filenames = set()
     for filename in accumulate_filenames():
         if not filename:  # alternative storage

wcs/forms/common.py

@@ -90,14 +90,18 @@ class FileDirectory(Directory):
     def serve_file(cls, file, thumbnail=False):
         response = get_response()
 
+        if misc.is_svg_filetype(file.content_type) and thumbnail:
+            thumbnail = False
+
         if thumbnail:
             if file.can_thumbnail():
-                try:
-                    content = misc.get_thumbnail(file.get_fs_filename(), content_type=file.content_type)
-                    response.set_content_type('image/png')
-                    return content
-                except misc.ThumbnailError:
-                    raise errors.TraversalError()
+                if file.content_type:
+                    try:
+                        content = misc.get_thumbnail(file.get_fs_filename(), content_type=file.content_type)
+                        response.set_content_type('image/png')
+                        return content
+                    except misc.ThumbnailError:
+                        raise errors.TraversalError()
             else:
                 raise errors.TraversalError()
 
@@ -507,11 +511,10 @@ class FormStatusPage(Directory, FormTemplateMixin):
         )
 
     def check_receiver(self):
-        session = get_session()
-        if not session or not session.user:
+        user = get_request().user
+        if not user:
             if not self.filled.formdef.is_user_allowed_read(None, self.filled):
                 raise errors.AccessUnauthorizedError()
-        user = get_request().user
         if self.filled.formdef is None:
             raise errors.AccessForbiddenError()
         if not self.filled.formdef.is_user_allowed_read(user, self.filled):
@@ -636,8 +639,12 @@ class FormStatusPage(Directory, FormTemplateMixin):
             return
         r = TemplateIO(html=True)
         r += htmltext('<div class="section foldable">')
-        r += htmltext('<h2>%s</h2>') % _('Backoffice Data')
-        r += htmltext('<div class="dataview">')
+        r += htmltext(
+            '<h2><span role="button" aria-expanded="true" '
+            'aria-controls="sect-backoffice-data" '
+            'id="sect-backoffice-data-label">%s</span></h2>'
+        ) % _('Backoffice Data')
+        r += htmltext('<div class="dataview" id="sect-backoffice-data">')
         r += content
         r += htmltext('</div>')
         r += htmltext('</div>')
@@ -1045,7 +1052,7 @@ class FormdefDirectoryBase(Directory):
         if tempfile['charset']:
             response.set_charset(tempfile['charset'])
 
-        if get_request().form.get('thumbnail') == '1':
+        if get_request().form.get('thumbnail') == '1' and not misc.is_svg_filetype(tempfile['content_type']):
             try:
                 thumbnail = misc.get_thumbnail(
                     get_session().get_tempfile_path(t), content_type=tempfile['content_type']

wcs/forms/root.py

@@ -281,8 +281,9 @@ class FormPage(FormdefDirectoryBase, FormTemplateMixin):
     sidebox_templates = ['wcs/front/formdata_sidebox.html', 'wcs/formdata_sidebox.html']
     formdef_class = FormDef
     preview_mode = False
+    edit_mode_submit_label = _('Save Changes')
 
-    def __init__(self, component, parent_category=None):
+    def __init__(self, component, parent_category=None, update_breadcrumbs=True):
         try:
             self.formdef = self.formdef_class.get_by_urlname(component)
         except KeyError:
@@ -300,7 +301,8 @@ class FormPage(FormdefDirectoryBase, FormTemplateMixin):
         self.on_validation_page = False
         self.current_page = None
         self.user = get_request().user
-        get_response().breadcrumb.append((component + '/', get_publisher().translate(self.formdef.name)))
+        if update_breadcrumbs:
+            get_response().breadcrumb.append((component + '/', get_publisher().translate(self.formdef.name)))
 
     def __call__(self):
         # add missing trailing slash.
@@ -616,14 +618,23 @@ class FormPage(FormdefDirectoryBase, FormTemplateMixin):
             session.add_magictoken(magictoken, form_data)
 
         if self.edit_mode and (page is None or page == self.pages[-1]):
-            form.add_submit('submit', _('Save Changes'), css_class='form-save-changes')
+            form.add_submit('submit', self.edit_mode_submit_label, css_class='form-save-changes')
         elif not self.has_confirmation_page() and (page is None or page == self.pages[-1]):
-            form.add_submit('submit', _('Submit'), css_class='form-submit')
+            form.add_submit(
+                'submit', _('Submit'), css_class='form-submit', attrs={'aria-label': _('Submit form')}
+            )
         else:
-            form.add_submit('submit', _('Next'), css_class='form-next')
+            form.add_submit(
+                'submit', _('Next'), css_class='form-next', attrs={'aria-label': _('Go to next page')}
+            )
 
         if self.pages.index(page) > 0:
-            form.add_submit('previous', _('Previous'), css_class='form-previous')
+            form.add_submit(
+                'previous',
+                _('Previous'),
+                css_class='form-previous',
+                attrs={'aria-label': _('Go back to previous page')},
+            )
 
         had_prefill = False
         if page_change or submit_button is True:
@@ -697,11 +708,13 @@ class FormPage(FormdefDirectoryBase, FormTemplateMixin):
 
         if not self.is_popup:
             cancel_label = _('Cancel')
+            aria_label = _('Cancel form')
             css_class = 'cancel'
             if self.has_draft_support() and not (data and data.get('is_recalled_draft')):
                 cancel_label = _('Discard')
+                aria_label = _('Discard form')
                 css_class = 'cancel form-discard'
-            form.add_submit('cancel', cancel_label, css_class=css_class)
+            form.add_submit('cancel', cancel_label, css_class=css_class, attrs={'aria-label': aria_label})
 
         if self.has_draft_support():
             form.add_submit(
@@ -720,12 +733,12 @@ class FormPage(FormdefDirectoryBase, FormTemplateMixin):
             'formdef': LazyFormDef(self.formdef),
             'form_side': self.form_side(data=data, magictoken=magictoken),
             'steps': self.step,
+            'html_form': form,
             # legacy, used in some themes
             'tracking_code_box': lambda: self.tracking_code_box(data, magictoken),
         }
         self.modify_filling_context(context, page, data)
 
-        context['html_form'] = form
         if self.is_popup:
             return template.QommonTemplateResponse(
                 templates=list(self.get_formdef_template_variants(self.popup_filling_templates)),
@@ -1760,9 +1773,9 @@ class FormPage(FormdefDirectoryBase, FormTemplateMixin):
                         user_id = user.id
                 wf_status = item.get_target_status(self.edited_data)
                 if wf_status:
-                    self.edited_data.jump_status(wf_status[0].id, user_id=user_id)
-                    self.edited_data.record_workflow_event('edit-action', action_item_id=item.id)
-                    url = self.edited_data.perform_workflow()
+                    if self.edited_data.jump_status(wf_status[0].id, user_id=user_id):
+                        self.edited_data.record_workflow_event('edit-action', action_item_id=item.id)
+                        url = self.edited_data.perform_workflow()
                 else:
                     # add history entry
                     evo = Evolution()

wcs/locale/fr/LC_MESSAGES/django.po

@@ -4,8 +4,8 @@ msgid ""
 msgstr ""
 "Project-Id-Version: wcs 0\n"
 "Report-Msgid-Bugs-To: \n"
-"POT-Creation-Date: 2023-03-01 10:52+0100\n"
-"PO-Revision-Date: 2023-03-01 10:52+0100\n"
+"POT-Creation-Date: 2023-04-17 18:05+0200\n"
+"PO-Revision-Date: 2023-04-17 18:06+0200\n"
 "Last-Translator: Thomas Noël <tnoel@entrouvert.com>\n"
 "Language-Team: french\n"
 "Language: fr\n"
@@ -14,10 +14,6 @@ msgstr ""
 "Content-Transfer-Encoding: 8bit\n"
 "Plural-Forms: nplurals=2; plural=(n > 1);\n"
 
-#: admin/api_access.py
-msgid "Additional options"
-msgstr "Options supplémentaires"
-
 #: admin/api_access.py admin/blocks.py admin/comment_templates.py
 #: admin/data_sources.py admin/forms.py admin/mail_templates.py admin/tests.py
 #: admin/users.py admin/workflows.py admin/wscalls.py backoffice/management.py
@@ -99,7 +95,6 @@ msgstr "Cette valeur est déjà utilisée."
 #: templates/wcs/backoffice/comment-template.html
 #: templates/wcs/backoffice/data-source.html
 #: templates/wcs/backoffice/mail-template.html
-#: templates/wcs/backoffice/test_sidebar.html
 #: templates/wcs/backoffice/workflow-status.html
 #: templates/wcs/backoffice/wscall.html
 msgid "Edit"
@@ -173,8 +168,9 @@ msgid "Usage"
 msgstr "Utilisation"
 
 #: admin/blocks.py admin/comment_templates.py admin/data_sources.py
-#: admin/fields.py admin/forms.py admin/mail_templates.py admin/workflows.py
-#: qommon/admin/menu.py
+#: admin/fields.py admin/forms.py admin/mail_templates.py admin/tests.py
+#: admin/workflows.py qommon/admin/menu.py
+#: templates/wcs/backoffice/test_sidebar.html
 msgid "Duplicate"
 msgstr "Dupliquer"
 
@@ -224,12 +220,12 @@ msgid "Deleting Block:"
 msgstr "Suppression du bloc :"
 
 #: admin/blocks.py admin/comment_templates.py admin/forms.py
-#: admin/mail_templates.py admin/workflows.py
+#: admin/mail_templates.py admin/tests.py admin/workflows.py
 msgid "(copy)"
 msgstr "(copie)"
 
 #: admin/blocks.py admin/comment_templates.py admin/forms.py
-#: admin/mail_templates.py admin/workflows.py
+#: admin/mail_templates.py admin/tests.py admin/workflows.py
 #, python-format
 msgid "%(name)s (copy %(no)d)"
 msgstr "%(name)s (Copie %(no)d)"
@@ -244,7 +240,7 @@ msgstr "L’identifiant ne peut pas être modifié car le bloc est utilisé."
 
 #: admin/blocks.py admin/comment_templates.py admin/data_sources.py
 #: admin/forms.py admin/mail_templates.py admin/workflows.py
-#: backoffice/cards.py qommon/substitution.py statistics/views.py
+#: backoffice/cards.py qommon/substitution.py
 #: templates/wcs/backoffice/block-inspect.html
 #: templates/wcs/backoffice/formdef-inspect.html
 msgid "Category"
@@ -309,7 +305,8 @@ msgstr "Importer un bloc de champs"
 #: templates/wcs/backoffice/forms.html templates/wcs/backoffice/i18n.html
 #: templates/wcs/backoffice/mail-templates.html
 #: templates/wcs/backoffice/settings/import.html
-#: templates/wcs/backoffice/tests.html templates/wcs/backoffice/wscalls.html
+#: templates/wcs/backoffice/tests.html templates/wcs/backoffice/workflows.html
+#: templates/wcs/backoffice/wscalls.html
 msgid "Import"
 msgstr "Importer"
 
@@ -478,17 +475,18 @@ msgstr "Aucune source de données n’est associée à cette catégorie."
 msgid "Categories are used to sort the different forms."
 msgstr "Ces catégories sont utilisées pour ranger les différents formulaires."
 
-#: admin/categories.py admin/settings.py admin/workflows.py
-#: backoffice/management.py forms/root.py templates/wcs/backoffice/blocks.html
-#: templates/wcs/backoffice/cards.html
+#: admin/categories.py admin/settings.py backoffice/management.py forms/root.py
+#: templates/wcs/backoffice/blocks.html templates/wcs/backoffice/cards.html
+#: templates/wcs/backoffice/categories.html
 #: templates/wcs/backoffice/comment-templates.html
 #: templates/wcs/backoffice/data-sources.html
 #: templates/wcs/backoffice/forms.html
 #: templates/wcs/backoffice/mail-templates.html
+#: templates/wcs/backoffice/workflows.html
 msgid "Categories"
 msgstr "Catégories"
 
-#: admin/categories.py
+#: admin/categories.py templates/wcs/backoffice/categories.html
 msgid "New Category"
 msgstr "Nouvelle catégorie"
 
@@ -522,8 +520,9 @@ msgid "Categories are used to sort the different data sources."
 msgstr ""
 "Ces catégories sont utilisées pour ranger les différentes sources de données."
 
-#: admin/comment_templates.py admin/workflows.py api_export_import.py
+#: admin/comment_templates.py api_export_import.py
 #: templates/wcs/backoffice/comment-templates.html
+#: templates/wcs/backoffice/workflows.html
 msgid "Comment Templates"
 msgstr "Modèles de message"
 
@@ -1044,7 +1043,7 @@ msgstr "Ce formulaire est en lecture seule."
 msgid "This form is currently disabled."
 msgstr "Ce formulaire est actuellement désactivé."
 
-#: admin/forms.py templates/wcs/backoffice/forms.html
+#: admin/forms.py templates/wcs/backoffice/includes/forms.html
 msgid "redirection"
 msgstr "redirection"
 
@@ -1328,6 +1327,7 @@ msgstr "Ouvrir la page du workflow"
 
 #: admin/forms.py admin/roles.py backoffice/cards.py
 #: templates/wcs/backoffice/formdef-inspect.html
+#: templates/wcs/backoffice/test_sidebar.html
 msgid "Options"
 msgstr "Options"
 
@@ -1709,9 +1709,9 @@ msgstr "Exception"
 msgid "Stack trace (most recent call first)"
 msgstr "Trace (appels les plus récents en premier)"
 
-#: admin/logged_errors.py admin/tests.py api_export_import.py
-#: backoffice/management.py formdata.py formdef.py statistics/views.py
-#: wf/create_formdata.py wf/form.py wf/resubmit.py
+#: admin/logged_errors.py api_export_import.py backoffice/management.py
+#: formdata.py formdef.py statistics/views.py wf/create_formdata.py wf/form.py
+#: wf/resubmit.py
 msgid "Form"
 msgstr "Formulaire"
 
@@ -1743,8 +1743,9 @@ msgstr "erreur %(class)s (%(message)s)"
 msgid "Logged Errors"
 msgstr "Erreurs enregistrées"
 
-#: admin/mail_templates.py admin/workflows.py api_export_import.py
+#: admin/mail_templates.py api_export_import.py
 #: templates/wcs/backoffice/mail-templates.html
+#: templates/wcs/backoffice/workflows.html
 msgid "Mail Templates"
 msgstr "Modèles de courriel"
 
@@ -2103,8 +2104,8 @@ msgstr "Configurer les textes qui apparaissent sur certaines pages"
 msgid "Configure known file types"
 msgstr "Configurer les types de fichier connus"
 
-#: admin/settings.py admin/workflows.py data_sources.py
-#: templates/wcs/backoffice/cards.html templates/wcs/backoffice/forms.html
+#: admin/settings.py data_sources.py templates/wcs/backoffice/cards.html
+#: templates/wcs/backoffice/forms.html templates/wcs/backoffice/workflows.html
 msgid "Data sources"
 msgstr "Sources de données"
 
@@ -2124,7 +2125,8 @@ msgstr "Configurer les appels de webservice"
 msgid "Backoffice"
 msgstr "Backoffice"
 
-#: admin/settings.py admin/workflows.py api_export_import.py workflows.py
+#: admin/settings.py admin/workflows.py api_export_import.py
+#: templates/wcs/backoffice/workflows.html workflows.py
 msgid "Workflows"
 msgstr "Workflows"
 
@@ -2331,6 +2333,27 @@ msgstr ""
 "Si complété, envoie tous les courriels à cette adresse au lieu des vrais "
 "destinataires"
 
+#: admin/tests.py
+msgid "Save data"
+msgstr "Enregistrer les données"
+
+#: admin/tests.py
+msgid "Edit data"
+msgstr "Modifier les données"
+
+#: admin/tests.py
+msgid "Mark as failing"
+msgstr "Marquer comme devant échouer"
+
+#: admin/tests.py
+#, python-format
+msgid "This test is expected to fail on error \"%s\"."
+msgstr "Ce test s’attend à échouer avec l’erreur « %s »."
+
+#: admin/tests.py
+msgid "This test is empty."
+msgstr "Ce test est vide."
+
 #: admin/tests.py
 msgid "Deleting Test:"
 msgstr "Suppression du test :"
@@ -2339,9 +2362,9 @@ msgstr "Suppression du test :"
 msgid "Edit test"
 msgstr "Modifier le test"
 
-#: admin/tests.py users.py
-msgid "Unknown User"
-msgstr "Utilisateur inconnu"
+#: admin/tests.py
+msgid "Duplicate test"
+msgstr "Dupliquer le test"
 
 #: admin/tests.py
 msgid "New test"
@@ -2490,6 +2513,7 @@ msgid "Last Modification:"
 msgstr "Dernière modification "
 
 #: admin/utils.py wf/attachment.py wf/comment.py wf/editable.py wf/resubmit.py
+#: workflows.py
 #, python-format
 msgid "by %s"
 msgstr "par %s"
@@ -2894,7 +2918,15 @@ msgstr "Modifier le niveau de criticité"
 msgid "Global Action: %s"
 msgstr "Action globale : %s"
 
-#: admin/workflows.py templates/wcs/backoffice/snapshots.html
+#: admin/workflows.py templates/wcs/backoffice/blocks.html
+#: templates/wcs/backoffice/cards.html templates/wcs/backoffice/categories.html
+#: templates/wcs/backoffice/comment-templates.html
+#: templates/wcs/backoffice/data-sources.html
+#: templates/wcs/backoffice/forms.html
+#: templates/wcs/backoffice/mail-templates.html
+#: templates/wcs/backoffice/snapshots.html templates/wcs/backoffice/tests.html
+#: templates/wcs/backoffice/workflows.html
+#: templates/wcs/backoffice/wscalls.html
 msgid "Actions"
 msgstr "Actions"
 
@@ -2983,7 +3015,7 @@ msgstr "Nouveau déclencheur d’action globale"
 msgid "Automatic"
 msgstr "Automatique"
 
-#: admin/workflows.py
+#: admin/workflows.py workflows.py
 msgid "Manual"
 msgstr "Manuel"
 
@@ -3058,8 +3090,8 @@ msgid "Duplicate Workflow"
 msgstr "Dupliquer le workflow"
 
 #: admin/workflows.py
-msgid "New Workflow"
-msgstr "Nouveau workflow"
+msgid "Uncategorised"
+msgstr "Non-catégorisés"
 
 #: admin/workflows.py
 msgid "Forms and card models"
@@ -3073,9 +3105,9 @@ msgstr "Modèles de fiche"
 msgid "Unused"
 msgstr "Inutilisé"
 
-#: admin/workflows.py
-msgid "Uncategorised"
-msgstr "Non-catégorisés"
+#: admin/workflows.py templates/wcs/backoffice/workflows.html
+msgid "New Workflow"
+msgstr "Nouveau workflow"
 
 #: admin/workflows.py
 msgid "Import Workflow"
@@ -3740,7 +3772,7 @@ msgstr "Date de début"
 msgid "End Date"
 msgstr "Date de fin (non incluse)"
 
-#: backoffice/management.py statistics/views.py
+#: backoffice/management.py
 msgctxt "categories"
 msgid "All"
 msgstr "Toutes"
@@ -4681,11 +4713,11 @@ msgstr "utilisation invalide, les conditions Python ne peuvent pas contenir {{"
 msgid "syntax error: %s"
 msgstr "erreur de syntaxe : %s"
 
-#: data_sources.py templates/wcs/backoffice/data-sources.html
+#: data_sources.py templates/wcs/backoffice/includes/data-sources.html
 msgid "Agendas"
 msgstr "Agendas"
 
-#: data_sources.py templates/wcs/backoffice/data-sources.html
+#: data_sources.py templates/wcs/backoffice/includes/data-sources.html
 msgid "Manually Configured Data Sources"
 msgstr "Sources de données manuellement configurées"
 
@@ -5158,6 +5190,14 @@ msgid "Invalid value for items prefill on field \"%s\""
 msgstr ""
 "Valeur invalide pour le préremplissage du champ « %s » (choix multiple)"
 
+#: fields.py
+msgid "label"
+msgstr "libellé"
+
+#: fields.py
+msgid "(continued)"
+msgstr "(suite)"
+
 #: fields.py
 msgid "Condition"
 msgstr "Condition"
@@ -5674,7 +5714,7 @@ msgstr ""
 "\n"
 "{% if form_status_changed %}\n"
 "Le statut de la demande est passé de « {{ form_previous_status }} » \n"
-"à « {{ form_status }} »).\n"
+"à « {{ form_status }} ».\n"
 "{% endif %}\n"
 "\n"
 "{% if form_comment %}Nouveau commentaire : {{ form_comment }}{% endif %}\n"
@@ -5806,6 +5846,10 @@ msgstr[0] "Pour accéder à la demande, indiquer le contenu du champ ci-dessous.
 msgstr[1] ""
 "Pour accéder à la demande, indiquer le contenu des champs ci-dessous."
 
+#: forms/root.py
+msgid "Save Changes"
+msgstr "Enregistrer les changements"
+
 #: forms/root.py
 msgid "Filling"
 msgstr "Édition"
@@ -5815,20 +5859,36 @@ msgid "Validating"
 msgstr "Validation"
 
 #: forms/root.py
-msgid "Save Changes"
-msgstr "Enregistrer les changements"
+msgid "Submit form"
+msgstr "Valider la saisie"
 
 #: forms/root.py
 msgid "Next"
 msgstr "Suivant"
 
+#: forms/root.py
+msgid "Go to next page"
+msgstr "Aller à la page suivante"
+
 #: forms/root.py
 msgid "Previous"
 msgstr "Précédent"
 
+#: forms/root.py
+msgid "Go back to previous page"
+msgstr "Aller à la page précédente"
+
+#: forms/root.py
+msgid "Cancel form"
+msgstr "Annuler la saisie"
+
+#: forms/root.py
+msgid "Discard form"
+msgstr "Abandonner la saisie"
+
 #: forms/root.py
 msgid "Save Draft"
-msgstr "Sauvegarder en tant que brouillon"
+msgstr "Enregistrer en tant que brouillon"
 
 #: forms/root.py
 msgid "leave this field blank to prove your humanity"
@@ -6248,6 +6308,11 @@ msgstr ""
 msgid "Advanced"
 msgstr "Avancé"
 
+#: qommon/form.py
+#, python-format
+msgid "Too long, value must be at most %d characters."
+msgstr "Trop long, il faut au maximum %d caractères."
+
 #: qommon/form.py
 #, python-format
 msgid "Usable units of time: %s."
@@ -6381,8 +6446,12 @@ msgid "Custom error message"
 msgstr "Message d’erreur personnalisé :"
 
 #: qommon/form.py
-msgid "This message will be be displayed if validation fails."
-msgstr "Message à afficher en cas d’erreur de validation."
+msgid ""
+"This message will be be displayed if validation fails. An empty value will "
+"give the default error message."
+msgstr ""
+"Message à afficher en cas d’erreur de validation. Une valeur vide affichera "
+"la message par défaut."
 
 #: qommon/form.py
 msgid "invalid value"
@@ -8154,8 +8223,8 @@ msgid "Error during upload."
 msgstr "Erreur lors du transfert."
 
 #: qommon/templates/qommon/forms/widgets/file.html
-msgid "Remove this file"
-msgstr "Retirer ce fichier"
+msgid "Remove the file:"
+msgstr "Retirer le fichier :"
 
 #: qommon/templates/qommon/forms/widgets/file.html
 #: templates/wcs/backoffice/tests.html
@@ -8260,10 +8329,6 @@ msgstr "Jour de la semaine"
 msgid "Hour"
 msgstr "Heure"
 
-#: statistics/views.py
-msgid "Category should now be selected using the Form field below."
-msgstr "La catégorie doit être choisie via le champ « Formulaire » ci-dessous."
-
 #: statistics/views.py
 msgid "Cards Count"
 msgstr "Nombre de fiches"
@@ -8285,6 +8350,10 @@ msgstr "Toutes les démarches"
 msgid "All forms of category %s"
 msgstr "Tous les formulaires de la catégorie %s"
 
+#: statistics/views.py
+msgid "Group by"
+msgstr "Regroupement"
+
 #: statistics/views.py
 msgctxt "statistics"
 msgid "Open"
@@ -8295,10 +8364,6 @@ msgctxt "statistics"
 msgid "Done"
 msgstr "Terminés"
 
-#: statistics/views.py
-msgid "Group by"
-msgstr "Regroupement"
-
 #: statistics/views.py
 msgid "Simplified status"
 msgstr "Statut simplifié"
@@ -8312,10 +8377,6 @@ msgstr "Ignorer les demandes/fiches où « %s » est vide."
 msgid "In progress"
 msgstr "En cours"
 
-#: statistics/views.py
-msgid "Time between two statuses"
-msgstr "Durée entre deux statuts"
-
 #: statistics/views.py
 msgid "Start status"
 msgstr "Statut initial"
@@ -8328,6 +8389,20 @@ msgstr "Statut d’arrivée"
 msgid "Any final status"
 msgstr "Tout statut final"
 
+#: statistics/views.py
+#, python-format
+msgid "Time between %(start_status)s and %(end_status)s"
+msgstr "Durée entre %(start_status)s et %(end_status)s"
+
+#: statistics/views.py testdef.py workflows.py
+#, python-format
+msgid "\"%s\""
+msgstr "« %s »"
+
+#: statistics/views.py
+msgid "any final status"
+msgstr "tout statut final"
+
 #: statistics/views.py
 msgid "Minimum time"
 msgstr "Temps minimum"
@@ -8385,9 +8460,14 @@ msgstr "Blocs de champs"
 msgid "New field block"
 msgstr "Nouveau bloc de champs"
 
-#: templates/wcs/backoffice/blocks.html
-msgid "There are no field blocks defined."
-msgstr "Il n’y a pas de blocs de champs configurés."
+#: templates/wcs/backoffice/blocks.html templates/wcs/backoffice/cards.html
+#: templates/wcs/backoffice/comment-templates.html
+#: templates/wcs/backoffice/data-sources.html
+#: templates/wcs/backoffice/forms.html
+#: templates/wcs/backoffice/mail-templates.html
+#: templates/wcs/backoffice/tests.html templates/wcs/backoffice/workflows.html
+msgid "Navigation"
+msgstr "Navigation"
 
 #: templates/wcs/backoffice/card-data-import-form.html
 msgid "You can add data to this card by uploading a JSON file."
@@ -8427,6 +8507,10 @@ msgstr "Blocs de champs"
 msgid "There are no card models defined."
 msgstr "Il n’y a pas de modèles de fiches configurés."
 
+#: templates/wcs/backoffice/categories.html
+msgid "There are no categories defined."
+msgstr "Il n’y a pas de catégories."
+
 #: templates/wcs/backoffice/category.html
 msgid "Permissions"
 msgstr "Permissions"
@@ -8456,10 +8540,6 @@ msgstr "Ce modèle de message doit encore être configuré."
 msgid "New comment template"
 msgstr "Nouveau modèle de message"
 
-#: templates/wcs/backoffice/comment-templates.html
-msgid "There are no comment templates defined."
-msgstr "Il n’y a pas de modèle de message défini."
-
 #: templates/wcs/backoffice/content-snapshot-part.html
 #, python-format
 msgid "changed at %(timestamp)s"
@@ -8554,45 +8634,13 @@ msgstr "Utilisation dans les formulaires"
 msgid "Not configured"
 msgstr "Non configurée"
 
-#: templates/wcs/backoffice/data-sources.html
-msgid "Refresh agendas"
-msgstr "Actualiser les agendas"
-
 #: templates/wcs/backoffice/data-sources.html
 msgid "New User Data Source"
 msgstr "Nouvelle source de données « Utilisateurs »"
 
 #: templates/wcs/backoffice/data-sources.html
-msgid "Users Data Sources"
-msgstr "Sources de données « Utilisateurs »"
-
-#: templates/wcs/backoffice/data-sources.html
-msgid "There are no users data sources defined."
-msgstr "Il n’y a pas de source de données « utilisateurs » configurées."
-
-#: templates/wcs/backoffice/data-sources.html
-msgid "There are no data sources defined."
-msgstr "Il n’y a pas de source de données manuellement configurées."
-
-#: templates/wcs/backoffice/data-sources.html
-msgid "Data Sources from Card Models"
-msgstr "Sources de données issues des modèles de fiches"
-
-#: templates/wcs/backoffice/data-sources.html
-msgid "automatically configured"
-msgstr "automatiquement configurées"
-
-#: templates/wcs/backoffice/data-sources.html
-msgid "There are no data sources from card models."
-msgstr "Il n’y a pas de sources de données définies par des modèles de fiches."
-
-#: templates/wcs/backoffice/data-sources.html
-msgid "not found"
-msgstr "non trouvé"
-
-#: templates/wcs/backoffice/data-sources.html
-msgid "There are no agendas."
-msgstr "Il n’y a pas d’agendas."
+msgid "Refresh agendas"
+msgstr "Actualiser les agendas"
 
 #: templates/wcs/backoffice/deprecations.html
 msgid "Rescan for deprecations"
@@ -8702,6 +8750,54 @@ msgstr "Chercher dans les textes marqués à ne pas traduire"
 msgid "Language:"
 msgstr "Langue :"
 
+#: templates/wcs/backoffice/includes/applications.html
+msgid "Applications"
+msgstr "Applications"
+
+#: templates/wcs/backoffice/includes/blocks.html
+msgid "There are no field blocks defined."
+msgstr "Il n’y a pas de blocs de champs configurés."
+
+#: templates/wcs/backoffice/includes/comment-templates.html
+msgid "There are no comment templates defined."
+msgstr "Il n’y a pas de modèle de message défini."
+
+#: templates/wcs/backoffice/includes/data-sources.html
+msgid "Users Data Sources"
+msgstr "Sources de données « Utilisateurs »"
+
+#: templates/wcs/backoffice/includes/data-sources.html
+msgid "There are no users data sources defined."
+msgstr "Il n’y a pas de source de données « utilisateurs » configurées."
+
+#: templates/wcs/backoffice/includes/data-sources.html
+msgid "There are no data sources defined."
+msgstr "Il n’y a pas de source de données manuellement configurées."
+
+#: templates/wcs/backoffice/includes/data-sources.html
+msgid "Data Sources from Card Models"
+msgstr "Sources de données issues des modèles de fiches"
+
+#: templates/wcs/backoffice/includes/data-sources.html
+msgid "automatically configured"
+msgstr "automatiquement configurées"
+
+#: templates/wcs/backoffice/includes/data-sources.html
+msgid "There are no data sources from card models."
+msgstr "Il n’y a pas de sources de données définies par des modèles de fiches."
+
+#: templates/wcs/backoffice/includes/data-sources.html
+msgid "not found"
+msgstr "non trouvé"
+
+#: templates/wcs/backoffice/includes/data-sources.html
+msgid "There are no agendas."
+msgstr "Il n’y a pas d’agendas."
+
+#: templates/wcs/backoffice/includes/mail-templates.html
+msgid "There are no mail templates defined."
+msgstr "Il n’y a pas de modèle de courriel défini."
+
 #: templates/wcs/backoffice/includes/test-result-fragment.html
 #: wf/display_message.py
 msgid "Success"
@@ -8771,10 +8867,6 @@ msgstr "Ce modèle de courriel doit être configuré."
 msgid "New mail template"
 msgstr "Nouveau modèle de courriel"
 
-#: templates/wcs/backoffice/mail-templates.html
-msgid "There are no mail templates defined."
-msgstr "Il n’y a pas de modèle de courriel défini."
-
 #: templates/wcs/backoffice/popup_response.html
 msgid "Popup closing..."
 msgstr "Fermeture de la fenêtre…"
@@ -9006,11 +9098,16 @@ msgstr "Démarré par :"
 msgid "Date:"
 msgstr "Date :"
 
+#: templates/wcs/backoffice/test-result.html
+msgid "Missing required fields:"
+msgstr "Champs obligatoires manquants :"
+
 #: templates/wcs/backoffice/test-result.html
 msgid "Success!"
 msgstr "Succès !"
 
 #: templates/wcs/backoffice/test-results.html
+#: templates/wcs/backoffice/tests.html
 msgid "Run tests"
 msgstr "Lancer les tests"
 
@@ -9022,9 +9119,35 @@ msgstr "Démarré par"
 msgid "No test results yet."
 msgstr "Pas encore de résultats des tests."
 
-#: templates/wcs/backoffice/test_sidebar.html
-msgid "Edit data"
-msgstr "Modifier les données"
+#: templates/wcs/backoffice/test_edit_sidebar.html
+msgid "Mark test as failing"
+msgstr "Marquer le test comme devant échouer"
+
+#: templates/wcs/backoffice/test_edit_sidebar.html
+#, python-format
+msgid ""
+"This test is expected to fail on error \"%(error)s\". Submitting the form "
+"will mark it as passing again."
+msgstr ""
+"Ce test s’attend à échouer avec l’erreur « %(error)s ». Valider le "
+"formulaire marquera le test comme devant réussir."
+
+#: templates/wcs/backoffice/test_edit_sidebar.html
+#, python-format
+msgid "If test should fail on error \"%(error)s\", click button below."
+msgstr ""
+"Si le test doit échouer avec l’erreur « %(error)s », cliquez sur le bouton "
+"ci-dessous."
+
+#: templates/wcs/backoffice/test_edit_sidebar.html
+msgid "In order to mark test as failing, form must display exactly one error."
+msgstr ""
+"Pour marquer le test comme devant échouer, le formulaire doit afficher une "
+"seule erreur."
+
+#: templates/wcs/backoffice/testdata_filling.html
+msgid "Edit test data"
+msgstr "Modifier les données de test"
 
 #: templates/wcs/backoffice/tests.html
 msgid "Test form data"
@@ -9036,12 +9159,6 @@ msgstr ""
 "Il n’est pas possible de créer de test car le formulaire contient des champs "
 "dépréciés."
 
-#: templates/wcs/backoffice/tests.html
-msgid "Tests cannot be created because there are no completed forms."
-msgstr ""
-"Il n’est pas possible de créer un test car il n’y a pas de demanche/fiche "
-"complétée."
-
 #: templates/wcs/backoffice/tests.html
 msgid "There are no tests yet."
 msgstr "Il n’y a pas encore de tests."
@@ -9240,10 +9357,6 @@ msgstr "Résumé"
 msgid "display form details"
 msgstr "afficher le détail de la demande"
 
-#: templates/wcs/formdata_summary.html
-msgid "Status "
-msgstr "Statut"
-
 #: templates/wcs/includes/drafts-recall.html
 msgid ""
 "You already started to fill this form. You can continue it or submit a new "
@@ -9269,6 +9382,19 @@ msgstr "sur la page %(page_no)s"
 msgid "This site is currently unavailable."
 msgstr "Ce site est pour le moment indisponible."
 
+#: testdef.py
+#, python-format
+msgid ""
+"Expected error \"%(expected_error)s\" but got error \"%(error)s\" instead."
+msgstr ""
+"L’erreur « %(expected_error)s » était attendue mais l’erreur « %(error)s » a "
+"eu lieu."
+
+#: testdef.py
+#, python-format
+msgid "Expected error \"%s\" but test completed with success."
+msgstr "L’erreur « %s » était attendue mais le test a abouti sans erreur."
+
 #: testdef.py
 #, python-format
 msgid ""
@@ -9296,11 +9422,6 @@ msgstr ""
 msgid "Failed to evaluate page %d post condition."
 msgstr "Erreur à l’évaluation de la condition de sortie de la page %d."
 
-#: testdef.py
-#, python-format
-msgid "\"%s\""
-msgstr "« %s »"
-
 #: testdef.py
 #, python-format
 msgid "\"%(subfield)s\" (of field %(field)s)"
@@ -9320,6 +9441,10 @@ msgstr "Valeur vide"
 msgid "%(error)s for field %(label)s: %(details)s."
 msgstr "%(error)s pour le champ %(label)s : %(details)s."
 
+#: users.py
+msgid "Unknown User"
+msgstr "Utilisateur inconnu"
+
 #: users.py
 #, python-format
 msgid "Session User Field: %s"
@@ -9481,6 +9606,10 @@ msgstr "Nouvelles arrivées"
 msgid "Anonymisation"
 msgstr "Anonymisation"
 
+#: wf/anonymise.py
+msgid "only user unlinking"
+msgstr "uniquement la déliaison de l’usager"
+
 #: wf/anonymise.py
 msgid "Only perform form/card user unlinking"
 msgstr "Uniquement délier l’usager associé à la demande/fiche"
@@ -10823,12 +10952,16 @@ msgstr "Rôles inconnus"
 
 #: workflows.py
 #, python-format
-msgid "Manual by %s"
-msgstr "Manuel par %s"
+msgid "from status %s"
+msgstr "depuis le statut %s"
 
 #: workflows.py
-msgid "Manual (not assigned)"
-msgstr "Manuel (non assigné)"
+msgid " or "
+msgstr " ou "
+
+#: workflows.py
+msgid "not assigned"
+msgstr "non assigné"
 
 #: workflows.py
 msgid "Only display to following statuses"

wcs/publisher.py

@@ -100,7 +100,6 @@ class WcsPublisher(QommonPublisher):
     APP_DIR = APP_DIR
     DATA_DIR = DATA_DIR
     ERROR_LOG = ERROR_LOG
-    USE_LONG_TRACES = USE_LONG_TRACES
     missing_appdir_redirect = REDIRECT_ON_UNKNOWN_VHOST
 
     supported_languages = ['fr', 'es', 'de']
@@ -124,8 +123,6 @@ class WcsPublisher(QommonPublisher):
             cls.DATA_DIR = config.get("main", "data_dir")
         if config.has_option("main", "error_log"):
             cls.ERROR_LOG = config.get("main", "error_log")
-        if config.has_option("main", "use_long_traces"):
-            cls.USE_LONG_TRACES = config.getboolean("main", "use_long_traces")
         if config.has_option("main", "missing_appdir_redirect"):
             cls.missing_appdir_redirect = config.get("main", "missing_appdir_redirect")
 
@@ -166,6 +163,10 @@ class WcsPublisher(QommonPublisher):
     def has_postgresql_config(self):
         return bool(self.cfg.get('postgresql', {}))
 
+    def has_user_fullname_config(self):
+        users_cfg = self.cfg.get('users') or {}
+        return bool(users_cfg.get('field_name') or users_cfg.get('fullname_template'))
+
     def set_config(self, request=None, skip_sql=False):
         QommonPublisher.set_config(self, request=request)
         if request:
@@ -403,6 +404,10 @@ class WcsPublisher(QommonPublisher):
         sql.do_tokens_table()
         sql.WorkflowTrace.do_table()
         sql.Audit.do_table()
+        sql.TestDef.do_table()
+        sql.TestResult.do_table()
+        sql.Application.do_table()
+        sql.ApplicationElement.do_table()
         sql.do_meta_table()
         from .carddef import CardDef
         from .formdef import FormDef
@@ -492,7 +497,15 @@ class WcsPublisher(QommonPublisher):
     def get_object_class(self, object_type):
         from wcs.blocks import BlockDef
         from wcs.carddef import CardDef
-        from wcs.categories import BlockCategory, CardDefCategory, Category, WorkflowCategory
+        from wcs.categories import (
+            BlockCategory,
+            CardDefCategory,
+            Category,
+            CommentTemplateCategory,
+            DataSourceCategory,
+            MailTemplateCategory,
+            WorkflowCategory,
+        )
         from wcs.comment_templates import CommentTemplate
         from wcs.data_sources import NamedDataSource
         from wcs.formdef import FormDef
@@ -513,6 +526,9 @@ class WcsPublisher(QommonPublisher):
             CardDefCategory,
             WorkflowCategory,
             BlockCategory,
+            MailTemplateCategory,
+            CommentTemplateCategory,
+            DataSourceCategory,
         ):
             if klass.xml_root_node == object_type:
                 return klass

wcs/qommon/form.py

@@ -24,6 +24,7 @@ import html
 import io
 import itertools
 import json
+import keyword
 import mimetypes
 import os
 import random
@@ -36,6 +37,7 @@ from functools import partial
 import dns
 import dns.exception
 import dns.resolver
+import emoji
 from bleach import Cleaner
 from bleach.linkifier import LinkifyFilter
 from PIL import Image
@@ -207,10 +209,14 @@ class SubmitWidget(quixote.form.widget.SubmitWidget):
     def render_content(self):
         if self.name in ('cancel', 'previous', 'save-draft'):
             self.attrs['formnovalidate'] = 'formnovalidate'
-        value = htmlescape(self.label) if self.label else None
+        label = self.label or ''
+        if label and 'aria-label' not in self.attrs:
+            cleaned_label = emoji.replace_emoji(label, replace='').strip()
+            if cleaned_label and cleaned_label != label:
+                self.attrs['aria-label'] = cleaned_label
         return (
-            htmltag('button', name=self.name, value=value, **self.attrs)
-            + str(self.label)
+            htmltag('button', name=self.name, value=htmlescape(label), **self.attrs)
+            + str(label)
             + htmltext('</button>')
         )
 
@@ -271,10 +277,8 @@ class Form(QuixoteForm):
 
     info = None
     captcha = None
-    advanced_label = '+'
 
     def __init__(self, *args, **kwargs):
-        self.advanced_label = kwargs.pop('advanced_label', self.advanced_label)
         self.use_tabs = kwargs.pop('use_tabs', False)
         QuixoteForm.__init__(self, *args, **kwargs)
         self.attrs['novalidate'] = 'novalidate'
@@ -455,19 +459,8 @@ class Form(QuixoteForm):
                         r += widget.render()
                     r += htmltext('</div>')
             return r.getvalue()
-        advanced_widgets = []
         for widget in self.widgets:
-            if not hasattr(widget, 'advanced') or not widget.advanced:
-                r += widget.render()
-            else:
-                advanced_widgets.append(widget)
-        if advanced_widgets:
-            get_response().add_javascript(['jquery.js', 'qommon.forms.js'])
-            r += htmltext('<fieldset class="form-plus">')
-            r += htmltext('<legend>%s</legend>') % self.advanced_label
-            for widget in advanced_widgets:
-                r += widget.render()
-            r += htmltext('</fieldset>')
+            r += widget.render()
         return r.getvalue()
 
     def add_media(self):
@@ -569,6 +562,7 @@ class StringWidget(QuixoteStringWidget):
             del kwargs['readonly']
         elif 'readonly' in kwargs:
             kwargs['readonly'] = 'readonly'
+        self.maxlength = kwargs.get('maxlength', None)
         self.validation_function = kwargs.pop('validation_function', None)
         super().__init__(name, *args, **kwargs)
 
@@ -576,11 +570,13 @@ class StringWidget(QuixoteStringWidget):
         QuixoteStringWidget._parse(self, request)
         if self.value:
             self.value = self.value.strip()
-        if self.value and self.validation_function:
-            try:
-                self.validation_function(self.value)
-            except ValueError as e:
-                self.error = str(e)
+            if self.maxlength and len(self.value) > self.maxlength:
+                self.error = _('Too long, value must be at most %d characters.') % self.maxlength
+            elif self.validation_function:
+                try:
+                    self.validation_function(self.value)
+                except ValueError as e:
+                    self.error = str(e)
 
     def render_content(self):
         attrs = {'id': 'form_' + self.get_name_for_id()}
@@ -846,6 +842,9 @@ class FileWithPreviewWidget(CompositeWidget):
         if not filetype:
             return False
 
+        if misc.is_svg_filetype(filetype):
+            return True
+
         if filetype == 'application/pdf':
             return HAS_PDFTOPPM
 
@@ -903,6 +902,13 @@ class FileWithPreviewWidget(CompositeWidget):
         elif magic and self.value.fp:
             mime = magic.Magic(mime=True)
             filetype = mime.from_file(self.value.fp.name)
+            if filetype in ('application/octet-stream', 'text/plain'):
+                # second-guess libmagic as we want to accept PDF files
+                # with some garbage at start.
+                with open(self.value.fp.name, 'rb') as fd:
+                    first_bytes = fd.read(1024)
+                    if b'%PDF' in first_bytes:
+                        filetype = 'application/pdf'
         else:
             filetype = getattr(self.value, 'storage_attrs', {}).get('content_type')
             if not filetype:
@@ -1205,7 +1211,7 @@ class ValidationWidget(CompositeWidget):
         if not value:
             value = {}
 
-        options = [(None, _('None'))] + [(x, y['title']) for x, y in self.validation_methods.items()]
+        options = [(None, _('None'), '')] + [(x, y['title'], x) for x, y in self.validation_methods.items()]
 
         self.add(
             SingleSelectWidget,
@@ -1218,7 +1224,6 @@ class ValidationWidget(CompositeWidget):
         if not self.value:
             self.value = {}
 
-        validation_labels = collections.OrderedDict(options)
         self.add(
             RegexStringWidget,
             'value_regex',
@@ -1226,7 +1231,7 @@ class ValidationWidget(CompositeWidget):
             value=value.get('value') if value.get('type') == 'regex' else None,
             attrs={
                 'data-dynamic-display-child-of': 'validation$type',
-                'data-dynamic-display-value': validation_labels.get('regex'),
+                'data-dynamic-display-value': 'regex',
             },
         )
         self.add(
@@ -1236,21 +1241,22 @@ class ValidationWidget(CompositeWidget):
             value=value.get('value') if value.get('type') == 'django' else None,
             attrs={
                 'data-dynamic-display-child-of': 'validation$type',
-                'data-dynamic-display-value': validation_labels.get('django'),
+                'data-dynamic-display-value': 'django',
             },
         )
         self.add(
             StringWidget,
             'error_message',
             size=80,
-            value=value.get('error_message') if value.get('type') in ['regex', 'django'] else None,
+            value=value.get('error_message') if value.get('type') else None,
             title=_('Custom error message'),
-            hint=_('This message will be be displayed if validation fails.'),
+            hint=_(
+                'This message will be be displayed if validation fails. '
+                'An empty value will give the default error message.'
+            ),
             attrs={
                 'data-dynamic-display-child-of': 'validation$type',
-                'data-dynamic-display-value-in': '|'.join(
-                    [str(validation_labels.get('regex')), str(validation_labels.get('django'))]
-                ),
+                'data-dynamic-display-value-in': '|'.join([x[2] for x in options if x[2]]),
             },
         )
         self._parsed = False
@@ -1263,9 +1269,10 @@ class ValidationWidget(CompositeWidget):
             value = self.get('value_%s' % type_)
             if value:
                 values['value'] = value
-            if type_ in ['regex', 'django']:
-                error_message = self.get('error_message')
-                if error_message:
+            error_message = self.get('error_message')
+            if error_message:
+                default_error_message = self.validation_methods[type_].get('error_message')
+                if error_message != default_error_message:
                     values['error_message'] = error_message
         self.value = values or None
 
@@ -1280,6 +1287,15 @@ class ValidationWidget(CompositeWidget):
             r += widget.render_content()
         widget = self.get_widget('error_message')
         r += widget.render()
+        error_messages = {
+            x: str(y.get('error_message'))
+            for x, y in self.validation_methods.items()
+            if y.get('error_message')
+        }
+        r += htmltext(
+            '<script id="validation-error-messages" type="application/json">%s</script>'
+            % json.dumps(error_messages)
+        )
         return r.getvalue()
 
     @classmethod
@@ -1304,11 +1320,8 @@ class ValidationWidget(CompositeWidget):
 
     @classmethod
     def get_validation_error_message(cls, validation):
-        pattern = cls.get_validation_pattern(validation)
-        if validation['type'] == 'regex' and pattern:
-            return validation.get('error_message')
-        if validation['type'] == 'django' and validation.get('value'):
-            return validation.get('error_message')
+        if validation.get('error_message'):
+            return get_publisher().translate(validation.get('error_message'))
         validation_method = cls.validation_methods.get(validation['type'])
         if validation_method and 'error_message' in validation_method:
             return validation_method['error_message']
@@ -1672,47 +1685,8 @@ class VarnameWidget(ValidatedStringWidget):
         # "native" id/text keys in datasources; forbid "status" to avoid status
         # filtering being diverted to a form field.
         # And forbid all reserved Python keywords so varnames can be used in
-        # dotted expressions (form.var.plop). (keyword.kwlist in Python 3.9)
-        if self.value in (
-            'id',
-            'text',
-            'status',
-            'and',
-            'as',
-            'assert',
-            'async',
-            'await',
-            'break',
-            'class',
-            'continue',
-            'def',
-            'del',
-            'elif',
-            'else',
-            'except',
-            'False',
-            'finally',
-            'for',
-            'from',
-            'global',
-            'if',
-            'import',
-            'in',
-            'is',
-            'lambda',
-            'None',
-            'nonlocal',
-            'not',
-            'or',
-            'pass',
-            'raise',
-            'return',
-            'True',
-            'try',
-            'while',
-            'with',
-            'yield',
-        ):
+        # dotted expressions (form.var.plop).
+        if self.value in ('id', 'text', 'status') + tuple(keyword.kwlist):
             self.error = _('this value is reserved for internal use.')
 
 

wcs/qommon/ident/franceconnect.py

@@ -62,7 +62,7 @@ class UserFieldMappingRowWidget(CompositeWidget):
         fields = []
         users_cfg = get_cfg('users', {})
         user_formdef = get_publisher().user_class.get_formdef()
-        if not user_formdef or not users_cfg.get('field_name'):
+        if not user_formdef or not get_publisher().has_user_fullname_config():
             fields.append(('__name', _('Name'), '__name'))
         if not user_formdef or not users_cfg.get('field_email'):
             fields.append(('__email', _('Email'), '__email'))

wcs/qommon/management/commands/cron.py

@@ -70,7 +70,7 @@ class Command(BaseCommand):
                 status, timestamp = sql.get_cron_status()
                 if status == 'running':
                     running += 1
-                    if now() - timestamp > datetime.timedelta(days=1):
+                    if now() - timestamp > datetime.timedelta(hours=6):
                         stalled_tenants.append(hostname)
                         CronJob.log('stalled tenant: %s' % hostname)
                         sql.mark_cron_status('done')

wcs/qommon/misc.py

@@ -616,6 +616,9 @@ class JSONEncoder(json.JSONEncoder):
         if isinstance(o, datetime.date):
             return o.strftime('%Y-%m-%d')
 
+        if isinstance(o, datetime.time):
+            return o.strftime('%H:%M:%S')
+
         if isinstance(o, decimal.Decimal):
             return localize(o)
 
@@ -686,6 +689,10 @@ def file_digest(content, chunk_size=100000):
     return digest.hexdigest()
 
 
+def is_svg_filetype(filetype):
+    return filetype and filetype.split('+')[0] == 'image/svg'
+
+
 def can_thumbnail(content_type):
     if content_type == 'application/pdf':
         return bool(HAS_PDFTOPPM and Image)
@@ -694,7 +701,7 @@ def can_thumbnail(content_type):
     return False
 
 
-def get_thumbnail(filepath, content_type=None):
+def get_thumbnail(filepath, content_type=None, size=None):
     if not filepath or not can_thumbnail(content_type or ''):
         raise ThumbnailError()
 
@@ -704,11 +711,16 @@ def get_thumbnail(filepath, content_type=None):
         os.mkdir(thumbs_dir)
     except FileExistsError:
         pass
-    thumb_filepath = os.path.join(thumbs_dir, hashlib.sha256(force_bytes(filepath)).hexdigest())
+    thumb_filepath = force_bytes(filepath)
+    if size:
+        thumb_filepath = '%s-%s-%s' % (thumb_filepath, *size)
+    thumb_filepath = os.path.join(thumbs_dir, hashlib.sha256(force_bytes(thumb_filepath)).hexdigest())
     if os.path.exists(thumb_filepath):
         with open(thumb_filepath, 'rb') as f:
             return f.read()
 
+    size = size or (500, 300)
+
     # generate thumbnail
     if content_type == 'application/pdf':
         try:
@@ -754,7 +766,7 @@ def get_thumbnail(filepath, content_type=None):
                 image = image.rotate(90, expand=1)
 
         try:
-            image.thumbnail((500, 300))
+            image.thumbnail(size)
         except (ValueError, SyntaxError):
             # PIL can raise syntax error on broken PNG files
             # * File "PIL/PngImagePlugin.py", line 119, in read
@@ -763,10 +775,12 @@ def get_thumbnail(filepath, content_type=None):
             # on broken JPEG files.
             raise OSError
         image_thumb_fp = io.BytesIO()
-        image.convert('RGB').save(image_thumb_fp, 'PNG')
+        image.convert('RGBA').save(image_thumb_fp, 'PNG')
     except OSError:
         # failed to create thumbnail.
         raise ThumbnailError()
+    finally:
+        fp.close()
 
     # store thumbnail
     with open(thumb_filepath, 'wb') as f:
@@ -1181,12 +1195,19 @@ class UnauthorizedPythonUsage(Exception):
 def eval_python(expression, *args, **kwargs):
     # Inherently unsafe, abort if support is forbidden.
     if get_publisher().has_site_option('forbid-python-expressions'):
-        get_publisher().record_error(
-            _('Unauthorized Python Usage'),
-            notify=True,
-            record=True,
-        )
-        raise UnauthorizedPythonUsage()
+        allowed_python_filename = os.path.join(get_publisher().app_dir, 'allowed-python.txt')
+        allowed = False
+        if os.path.exists(allowed_python_filename):
+            with open(allowed_python_filename) as fd:
+                if expression in fd.read().splitlines():
+                    allowed = True
+        if not allowed:
+            get_publisher().record_error(
+                _('Unauthorized Python Usage'),
+                notify=True,
+                record=True,
+            )
+            raise UnauthorizedPythonUsage()
     # noqa pylint: disable=eval-used
     return eval(expression, *args, **kwargs)
 

wcs/qommon/publisher.py

@@ -41,6 +41,7 @@ from django.conf import settings
 from django.http import Http404
 from django.utils import translation
 from django.utils.encoding import force_bytes, force_str
+from django.views.debug import SafeExceptionReporterFilter
 from quixote.publish import Publisher, get_publisher, get_request, get_response
 
 from wcs.qommon.storage import Less
@@ -103,7 +104,6 @@ class QommonPublisher(Publisher):
     APP_DIR = None
     DATA_DIR = None
     ERROR_LOG = None
-    USE_LONG_TRACES = True
 
     root_directory_class = None
     backoffice_directory_class = None
@@ -217,18 +217,13 @@ class QommonPublisher(Publisher):
         return output
 
     def _generate_plaintext_error(self, request, original_response, exc_type, exc_value, tb, limit=None):
-        if not self.USE_LONG_TRACES:
-            if not request:
-                # this happens when an exception is raised by an afterjob
-                request = HTTPRequest(None, {})
-            if not request.form:
-                request.form = {}
-            return Publisher._generate_plaintext_error(
-                self, request, original_response, exc_type, exc_value, tb
-            )
-
         error_file = io.StringIO()
 
+        safe_filter = SafeExceptionReporterFilter()
+        safe_filter.hidden_settings = re.compile(
+            '|'.join(['API', 'TOKEN', 'KEY', 'SECRET', 'PASS', 'SIGNATURE', 'HOST', 'PGCONN']), flags=re.I
+        )
+
         if limit is None:
             if hasattr(sys, 'tracebacklimit'):
                 limit = sys.tracebacklimit
@@ -262,6 +257,7 @@ class QommonPublisher(Publisher):
                 print("  locals: ", file=error_file)
                 for key, value in locals:
                     print("     %s =" % key, end=' ', file=error_file)
+                    value = safe_filter.cleanse_setting(key, value)
                     try:
                         repr_value = repr(value)
                         if len(repr_value) > 10000:

wcs/qommon/static/css/dc2/admin.scss

@@ -1,6 +1,5 @@
 $primary-color: #386ede;
 $secondary-color: #00d6eb;
-$string-color: str-slice($primary-color + '', 2);
 $actions: add, duplicate, edit, jump, remove, copy;
 
 @mixin clearfix {
@@ -149,12 +148,7 @@ ul.biglist li.disabled strong {
 	font-weight: normal;
 }
 
-ul.biglist.sortable li span.handle + strong a {
-	width: calc(100% - 2em - 1ex);
-}
-
 ul.biglist strong {
-	width: 30%;
 	display: inline;
 	color: #202020;
 }
@@ -763,12 +757,15 @@ ul.biglist li.disabled a {
 	color: #888;
 }
 
-div.bo-block ul.biglist.sortable li strong a,
-div.bo-block ul.biglist.sortable li > a,
-ul.biglist.sortable a {
-	display: inline-block;
-	width: calc(100% - 2em);
-	box-sizing: border-box;
+ul.biglist.sortable {
+	li {
+		display: flex;
+		align-items: baseline;
+	}
+	a {
+		flex: 1;
+		box-sizing: border-box;
+	}
 }
 
 #items-list li > a {
@@ -1311,15 +1308,6 @@ div#qrcode img {
 	margin: auto;
 }
 
-fieldset.form-plus legend:after {
-	content: "▼";
-	transition: transform ease 0.1s;
-}
-
-fieldset.form-plus.closed legend:after {
-	transform: rotate(-90deg);
-}
-
 @media screen and (max-width: 760px) {
 	div#main-content.with-sidebar,
 	div#main-content {
@@ -2052,12 +2040,15 @@ a.button.button-paragraph {
 	display: block;
 	max-width: 100%;
 	margin-bottom: 1rem;
+	line-height: 150%;
+	padding-top: 0.8em;
+	padding-bottom: 0.8em;
 }
 
 a.button.button-paragraph p {
 	font-weight: normal;
 	color: #333;
-	margin: 0;
+	margin: 0.5em 0 0 0;
 	line-height: 150%;
 }
 
@@ -2278,9 +2269,10 @@ div#side { // w.c.s. steps in backoffice submission
 		}
 		@each $action in $actions {
 			&.#{$action} a {
-				background: url(/static/css/icons/action-#{$action}.small.#{$string-color}.png) center center no-repeat;
+				background-position: center center;
+				background-repeat: no-repeat;
 				background-size: 20px;
-				background-image: url(/static/css/icons/action-#{$action}.small.#{$string-color}.png),
+				background-image: url(/static/css/icons/action-#{$action}.small.png);
 				&:hover {
 					background-image: url(/static/css/icons/action-#{$action}.hover.png);
 				}
@@ -2375,11 +2367,11 @@ div.timetable-widget {
 		text-indent: -10000px;
 		overflow: hidden;
 		width: 12px;
-		background: white url(/static/css/icons/action-edit.small.#{$string-color}.png) center center no-repeat;
+		background: white url(/static/css/icons/action-edit.small.png) center center no-repeat;
 		background-size: 16px;
 		&:hover {
 			background-color: #386ede;
-			background-image: url(/static/css/icons/action-edit.small.white.png);
+			background-image: url(/static/css/icons/action-edit.hover.png);
 		}
 	}
 }
@@ -2643,3 +2635,7 @@ span.test-failure::before {
       font-family: FontAwesome;
       content: "\f00d";  /* times */
 }
+
+.application-logo, .application-icon {
+	vertical-align: middle;
+}

wcs/qommon/static/css/qommon.scss

@@ -420,38 +420,6 @@ form .widget-hidden {
 	display: none;
 }
 
-fieldset.form-plus {
-	padding: 1ex 0 0 0;
-	border: 0;
-}
-
-fieldset.form-plus > div {
-	display: block;
-}
-
-fieldset.form-plus > .widget-hidden {
-	display: none;
-}
-
-fieldset.form-plus.closed > div {
-	display: none;
-}
-
-fieldset.form-plus legend {
-	display: block;
-	width: 100%;
-	cursor: pointer;
-	padding: 1ex 0 0 0;
-	font-weight: bold;
-	border-bottom: 1px solid #aaa;
-}
-
-fieldset.form-plus legend:after {
-	content: "+";
-	position: absolute;
-	right: 1em;
-}
-
 ul#evolutions {
 	list-style: none;
 	padding: 0;

wcs/qommon/static/css/sofresh.css

@@ -1,332 +0,0 @@
-@import url(/static/xstatic/themes/smoothness/jquery-ui.min.css);
-@import url(qommon.css);
-/* derived from soFresh, a DotClear theme by Maurice Svay (GPL)
- * http://www.svay.com/files/soFresh/theme-sofresh-1.2.zip */
-
-html, body {
-	font-family: sans-serif;
-	text-align: center;
-	background: #eee;
-	color: black;
-}
-
-div#page {
-	width: 800px;
-	margin: 2em auto;
-	text-align: justify;
-	background: white url(img/page.png) repeat-y;
-	color: black;
-}
-
-#top {
-	color: #09F;
-	background: #FFF url(img/top.jpg) no-repeat;
-	height: 100px;
-	margin: 0;
-}
-
-#top h1 {
-	margin: 0;
-	padding-left: 30px;
-	padding-right: 30px;
-	line-height: 100px;
-	height: 100px;
-	overflow: hidden;
-}
-
-#footer {
-	background: #FFF url(img/footer.jpg) no-repeat;
-	color: #999;
-	text-align: center;
-	min-height: 30px;
-}
-
-div#main-content {
-	margin: 0 2em;
-}
-
-div#main-content h1 {
-	color: #09F;
-}
-
-#steps {
-	height: 32px;
-	margin-bottom: 1em;
-}
-
-#steps ol {
-	list-style: none;
-	padding: 0 20px;
-}
-
-#steps li {
-	display: inline;
-	padding-right: 1em;
-	display: block;
-	float: left;
-	width: 20%;
-	list-style: none;
-}
-
-#steps ol ul {
-	border: 1px solid #ffa500;
-	margin-left: 2em;
-	background: #ffdb94;
-	margin-bottom: 1em;
-	padding: 0;
-	width: auto;
-}
-
-#steps li li {
-	display: block;
-	width: auto;
-	float: none;
-	text-align: left;
-	margin-left: 1em;
-	font-size: 90%;
-}
-
-#steps li li.current {
-	color: black;
-}
-
-
-#steps span.marker {
-	font-size: 26px;
-	padding: 2px 9px;
-	font-weight: bold;
-	color: white;
-	text-align: center;
-	background: #ddd;
-	border: 1px solid #ddd;
-	-moz-border-radius: 0.7ex;
-}
-
-#steps li.current span.marker {
-	background: #ffa500;
-	border: 1px solid #ffc400;
-}
-
-#steps span.label {
-	font-size: 90%;
-}
-
-#steps li.current span.label {
-	color: black;
-}
-
-#steps {
-	background: #f0f0f0;
-	color: #aaa;
-}
-
-#steps ol ul {
-	display: none;
-}
-
-#steps ol li.current ul {
-	display: block;
-}
-
-form {
-	clear: both;
-}
-
-p#receiver {
-	margin: 0;
-	margin-left: 2em;
-	margin-top: -0.7em;
-	margin-bottom: 1em;
-	padding: 2px 5px;
-	border: 1px solid #ccc;
-	float: left;
-	background: #ffe;
-}
-
-p#login,
-p#logout {
-	margin-top: 2em;
-	text-align: right;
-}
-
-p#login a, p#logout a {
-	text-decoration: underline;
-	/*
-	text-decoration: none;
-	-moz-border-radius: 2em !important;
-	padding: 1px 6px !important;
-	border: 1px solid #ccc !important;
-	border-bottom: 2px solid #999 !important;*/
-}
-
-p#login a:hover {
-	background-color: #ddeeff;
-}
-
-h2#submitted, h2#done {
-	margin-top: 2em;
-	color: #09F;
-	font-size: 130%;
-}
-
-h2#done {
-	margin-top: 1em;
-}
-
-ul li {
-	list-style-image: url(img/li.png);
-}
-
-h2, h3 {
-	color: #09F;
-	margin-left: 0.5em;
-	margin-bottom: 0;
-}
-
-h3 {
-	margin-left: 1em;
-}
-
-table#listing {
-	margin: 1em 0;
-}
-
-table#listing th {
-	text-align: left;
-	border-bottom: 1px solid #999;
-	background: #eee;
-}
-
-table#listing td {
-	padding-right: 1ex;
-}
-
-table#listing th a {
-	text-decoration: none;
-	color: #666;
-}
-
-table.sortable span.sortarrow {
-	color: black;
-	text-decoration: none;
-}
-
-table#listing tr.status-new {
-	background: #aea;
-}
-
-table#listing tr.status-finished,
-table#listing tr.status-rejected {
-	color: #444;
-}
-
-table#listing tr.status-rejected td {
-	text-decoration: line-through;
-}
-
-div.question p.label {
-	font-weight: bold;
-}
-
-img.bar {
-	border: 1px solid black;
-}
-
-div.question table {
-	margin-left: 10px;
-}
-
-div.question table td {
-}
-
-div.question table td.percent {
-	text-align: right;
-	padding: 0 1em;
-	width: 6em;
-}
-
-div.question table td.label {
-	width: 6em;
-}
-
-span.user {
-	margin-left: 1em;
-	font-style: italic;
-}
-
-a.listing {
-	font-size: 80%;
-	padding-left: 1em;
-}
-
-a {
-	text-decoration: none;
-	color: #113;
-}
-
-a:hover {
-	color: #06C;
-	text-decoration: underline;
-}
-
-
-#prelude {
-	color: #aaa;
-	background: transparent;
-	text-align: right;
-	position: relative;
-	top: -85px;
-	margin: 0;
-}
-
-p#breadcrumb {
-	margin-top: 0;
-}
-
-div.error-page {
-	margin-bottom: 2em;
-}
-
-div.hint {
-	display: inline;
-	padding-left: 1ex;
-	font-style: italic;
-}
-
-a.standalone {
-	background: white url(img/h2.png) top left no-repeat;
-	padding-left: 20px;
-}
-
-div#welcome-message a {
-	text-decoration: underline;
-}
-
-input.cancel {
-	margin-left: 5em;
-}
-
-hr {
-	border: none;
-	border-top: 1px solid #666;
-	height: 1px;
-	width: 80%;
-}
-
-div.buttons {
-	clear: both;
-}
-
-div.note {
-	padding: 6px;
-	border: solid 1px #e5e5e3;
-	background: #f3f3f0 none 5px 15px no-repeat;
-	border-radius: 8px;
-	box-shadow: 0 0 3px #888;
-	margin: 1em 0;
-}
-
-div.icon-important { padding-left: 34px; background-image: url(../images/yelp-note-important.png); }
-div.icon-tip { padding-left: 34px; background-image: url(../images/yelp-note-tip.png); }
-div.icon-warning { padding-left: 34px; background-image: url(../images/yelp-note-warning.png); }
-