fix(sync): don't lose nodes on single download err

`P2pCandidate.dlPromise` promise doesn't resolve on download errors
such as socket timeouts. Each time such an error occurs, the node is lost
for the sync because the 'this.dlPromise.isResolved()' check used for
testing the node's availability never succeeds anymore for this node.

This patch hardens the blocks downloading in two ways:
* Retrying the download up to 5 times when the number of returned blocks is
  different from what was requested
* On failure returns an empty block array so that the dlPromise is resolved
  and the node failure could be handled as expected at the P2PSyncDownloader
  level triggering the `P2pCandidate.addFailure()` function.

app/modules/crawler/lib/sync/p2p/p2p-candidate.ts

@@ -90,21 +90,30 @@ export class P2pCandidate {
     this.dlPromise = querablep(
       (async () => {
         // We try to download the blocks
-        let blocks: BlockDTO[] | null;
-        try {
-          blocks = await (this.api as IRemoteContacter).getBlocks(count, from);
-        } catch (e) {
-          // Unfortunately this can fail
-          blocks = null;
-          error = e;
+        let blocks: BlockDTO[] = [];
+        let tries = 5;
+        while (tries > 0) {
+          try {
+            blocks = await (this.api as IRemoteContacter).getBlocks(
+              count,
+              from
+            );
+          } catch (e) {
+            // Unfortunately this can fail
+            blocks = [];
+            this.logger.error(e);
+          }
+          if (blocks.length != count) {
+            this.logger.error("Wrong number of blocks from %s", this.hostName);
+            tries--;
+          } else {
+            break;
+          }
         }
         this.responseTimes.push(Date.now() - start);
         // Only keep a flow of 5 ttas for the node
         if (this.responseTimes.length > 5) this.responseTimes.shift();
         this.nbSuccess++;
-        if (error) {
-          throw error;
-        }
         return blocks;
       })()
     );

[Benoit Lavenier]

@Pini je vois que tu retentes le docnload aussitôt. Sais tu pourquoi les blocks sont mal récupérés ? Est-ce à cause d'une limitation du nombre de requete sur le noeud requêté ? (ca dépend de l'erreur que tu as dans les logs...) Si oui, il faut peut-être ajouter une tempo, non ?

[pini]

Non, je n'ai pas poussé l'analyse de savoir pourquoi les blocs sont mal récupérés. Ça peut être tout et n'importe quoi. Mais pour les noeuds valides c'est probablement une histoire de charge / temps de réponse. Je suis bien incapable de dire si une tempo serait utile dans ce cas. Tu penses à quoi comme ordre de grandeur ?