Missing prefixes using scan_roas on databae

[fendemann]

I´ve recognized missing prefixes searching the sql with scan_roas. After extracting the information using the rcynic-dump tool it seems, that only roas from the authenticated tree were found by scan_roas in the db. All roas from sha265 tree are missing.

rpki-rp version 1.0.1494995102~xenial

Searching for AS3320 prefixes in db

root@rcynic-03:/tmp# su rpki -c scan_roas | grep 3320
2017-03-31T07:13:10Z 3320 194.127.134.0/24 194.76.30.0/23 194.76.30.0/24 194.76.31.0/24
2017-01-01T00:14:19Z 3320 134.97.3.0/24 134.97.2.0/24 134.97.16.0/24
2017-01-01T01:52:04Z 3320 194.121.46.0/24 194.120.22.0/24 194.120.40.0/24 194.120.249.0/24 194.120.23.0/24 194.121.90.0/24 194.45.148.0/22-24 194.45.48.0/24 194.121.62.0/23-24 193.141.61.0/24 193.141.143.0/24 193.141.23.0/24 194.121.172.0/22-24 194.45.144.0/22-24
2017-01-01T00:38:13Z 3320 195.158.253.0/24 195.158.252.0/24 195.158.252.0/23 195.190.11.0/24 2001:67c:11f8::/48
2017-01-01T00:14:23Z 3320 193.22.4.0/24

Extracting information from db

root@rcynic-03:/tmp# sudo -u rpki /usr/local/sbin/rcynic-dump
root@rcynic-03:/tmp# ll rcynic-data/
total 16
drwxr-xr-x 4 rpki rpki 4096 Jul 12 06:31 ./
drwxrwxrwt 11 root root 4096 Jul 12 06:39 ../
lrwxrwxrwx 1 rpki rpki 34 Jul 12 06:31 authenticated -> authenticated-2017-07-12T05:36:02Z/
drwxr-xr-x 11 rpki rpki 4096 Jul 12 06:29 authenticated-2017-07-12T05:36:02Z/
drwxr-xr-x 258 rpki rpki 4096 Jul 12 06:29 sha256/

Searching for AS3320 prefixes in authenticated tree => matching prefixes from db

root@rcynic-03:/tmp# scan_roas rcynic-data/authenticated | grep 3320
2017-01-01T00:14:23Z 3320 193.22.4.0/24
2017-01-01T01:52:04Z 3320 194.121.46.0/24 194.120.22.0/24 194.120.40.0/24 194.120.249.0/24 194.120.23.0/24 194.121.90.0/24 194.45.148.0/22-24 194.45.48.0/24 194.121.62.0/23-24 193.141.61.0/24 193.141.143.0/24 193.141.23.0/24 194.121.172.0/22-24 194.45.144.0/22-24
2017-01-01T00:38:13Z 3320 195.158.253.0/24 195.158.252.0/24 195.158.252.0/23 195.190.11.0/24 2001:67c:11f8::/48
2017-03-31T07:13:10Z 3320 194.127.134.0/24 194.76.30.0/23 194.76.30.0/24 194.76.31.0/24
2017-01-01T00:14:19Z 3320 134.97.3.0/24 134.97.2.0/24 134.97.16.0/24

Searching for AS3320 prefixes in sha265 tree => matching prefixes + extra prefixes

root@rcynic-03:/tmp# scan_roas rcynic-data/sha256/ | grep 3320
2016-01-01T00:50:52Z 3320 46.183.47.0/24
2016-01-01T01:30:04Z 3320 212.102.160.0/19
2016-01-01T01:47:44Z 3320 195.234.178.0/24 141.6.203.0/24
2017-01-01T00:38:13Z 3320 195.158.253.0/24 195.158.252.0/24 195.158.252.0/23 195.190.11.0/24 2001:67c:11f8::/48
2017-01-01T01:52:04Z 3320 194.121.46.0/24 194.120.22.0/24 194.120.40.0/24 194.120.249.0/24 194.120.23.0/24 194.121.90.0/24 194.45.148.0/22-24 194.45.48.0/24 194.121.62.0/23-24 193.141.61.0/24 193.141.143.0/24 193.141.23.0/24 194.121.172.0/22-24 194.45.144.0/22-24
2016-01-01T01:41:22Z 3320 46.80.0.0/12 91.0.0.0/10 195.145.0.0/16 80.144.0.0/13 79.192.0.0/10 62.153.0.0/16 62.156.0.0/14 62.224.0.0/14 217.224.0.0/11 80.128.0.0/12 212.184.0.0/15 80.157.8.0/21 195.243.0.0/16 87.128.0.0/10 194.25.0.0/16 80.128.0.0/11 217.80.0.0/12 217.0.0.0/13 84.128.0.0/10 80.156.0.0/16 80.152.0.0/14 193.158.0.0/15 93.192.0.0/10 80.157.16.0/20 62.154.0.0/15 80.157.0.0/16 2003::/19 2003::/23 2003:3c0::/28 2003:3e0::/28
2017-01-01T00:14:23Z 3320 193.22.4.0/24
2017-03-31T07:13:10Z 3320 194.127.134.0/24 194.76.30.0/23 194.76.30.0/24 194.76.31.0/24
2016-02-22T21:17:23Z 3320 185.133.12.0/22 2a05:fe80::/29
2017-01-01T00:14:19Z 3320 134.97.3.0/24 134.97.2.0/24 134.97.16.0/24

Searching for AS3320 prefixes in unauthenticated tree => no match

root@rcynic-03:/tmp# scan_roas /var/rcynic/data/unauthenticated/ | grep 3320

[sraustein]

Yes, when pulling from the database, `scan_roas` only reports current authenticated data. The `sha256/` directory in the dump output is *everything*, current or not, valid or not.

[osagga]

I have this script that would run scan_roas every 12 hours and saves the output to a text file, the bash script is as follows:

#! /bin/bash

while true; do
    suffix=$(date +%Y-%m-%dT%H:%M:%SZ)
    filename="authenticated-$suffix"
    sudo -u rpki scan_roas > ROAs/"${filename}.txt"
    sleep $((60 * 60 * 12))
    done

And it seems that when the script runs for some time (more than a week), the number of roas scan_roas finds are lower than expected (it gets ~3K ROAs, when it's expected to have ~7-8K ROAs). Here are the data (I'm using the number of lines to count how many roas given that scan_roas outputs one per line):

ubuntu@rpki-data:~$ wc -l ROAs/*
    8204 ROAs/authenticated-2017-09-11T16:06:13Z.txt
    8280 ROAs/authenticated-2017-09-12T04:06:15Z.txt
    8287 ROAs/authenticated-2017-09-12T16:06:17Z.txt
    8291 ROAs/authenticated-2017-09-13T04:06:19Z.txt
    8296 ROAs/authenticated-2017-09-13T16:06:21Z.txt
    8293 ROAs/authenticated-2017-09-14T04:06:23Z.txt
    8298 ROAs/authenticated-2017-09-14T16:06:25Z.txt
    8306 ROAs/authenticated-2017-09-15T04:06:26Z.txt
    8310 ROAs/authenticated-2017-09-15T16:06:28Z.txt
    8312 ROAs/authenticated-2017-09-16T04:06:30Z.txt
    8315 ROAs/authenticated-2017-09-16T16:06:32Z.txt
    8315 ROAs/authenticated-2017-09-17T04:06:34Z.txt
    8315 ROAs/authenticated-2017-09-17T16:06:36Z.txt
    8322 ROAs/authenticated-2017-09-18T04:06:38Z.txt
    8319 ROAs/authenticated-2017-09-18T16:06:40Z.txt
    8318 ROAs/authenticated-2017-09-19T04:06:42Z.txt
    8322 ROAs/authenticated-2017-09-19T16:06:44Z.txt
    8322 ROAs/authenticated-2017-09-20T04:06:46Z.txt
    8322 ROAs/authenticated-2017-09-20T16:06:48Z.txt
    8323 ROAs/authenticated-2017-09-21T04:06:50Z.txt
    8324 ROAs/authenticated-2017-09-21T16:06:52Z.txt
    8332 ROAs/authenticated-2017-09-22T04:06:54Z.txt
    8341 ROAs/authenticated-2017-09-22T16:06:56Z.txt
    8346 ROAs/authenticated-2017-09-23T04:06:58Z.txt
    8344 ROAs/authenticated-2017-09-23T16:07:00Z.txt
    8344 ROAs/authenticated-2017-09-24T04:07:02Z.txt
    8345 ROAs/authenticated-2017-09-24T16:07:04Z.txt
    8347 ROAs/authenticated-2017-09-25T04:07:06Z.txt
    8352 ROAs/authenticated-2017-09-25T16:07:08Z.txt
    8357 ROAs/authenticated-2017-09-26T04:07:10Z.txt
    8357 ROAs/authenticated-2017-09-26T16:07:12Z.txt
    8359 ROAs/authenticated-2017-09-27T04:07:14Z.txt
    8359 ROAs/authenticated-2017-09-27T16:07:16Z.txt
    8367 ROAs/authenticated-2017-09-28T04:07:18Z.txt
    2984 ROAs/authenticated-2017-09-28T16:07:20Z.txt
    2986 ROAs/authenticated-2017-09-29T04:07:21Z.txt
    2989 ROAs/authenticated-2017-09-29T16:07:22Z.txt
    2997 ROAs/authenticated-2017-09-30T04:07:23Z.txt
    2998 ROAs/authenticated-2017-09-30T16:07:24Z.txt
    2997 ROAs/authenticated-2017-10-01T04:07:25Z.txt
    2998 ROAs/authenticated-2017-10-01T16:07:26Z.txt
    2999 ROAs/authenticated-2017-10-02T04:07:27Z.txt
    2999 ROAs/authenticated-2017-10-02T16:07:28Z.txt
    3077 ROAs/authenticated-2017-10-24T15:22:03Z.txt
    3077 ROAs/authenticated-2017-10-24T15:23:13Z.txt
    3098 ROAs/authenticated-2017-10-25T03:22:11Z.txt
    3098 ROAs/authenticated-2017-10-25T03:23:14Z.txt
    2983 ROAs/authenticated-2017-10-25T15:22:12Z.txt
    2983 ROAs/authenticated-2017-10-25T15:23:15Z.txt
  459283 total

What could be causing the drop of the number of ROAs? Am I using scan_roas in the wrong way?