Custom internet standards report

A lot of requests came in about customizing which urls are included in the internet standards report. By default, it only included all pay level domains ("shadowtrackr.com") and not subdomains ("twilightsparkle.shadowtrackr.com").

The reason for this is that daily checking all urls in ShadowTrackr would costs a lot of CPU and memory and is not sustainable at current price levels. I do very much understand that some pay level domains are not that interesting and some subdomains are. So, something needed to change.

As of this week you can edit the internet standards report. You can remove pay level domains in the edit function under reports, and you can add subdomains by clicking on the action menu (three dots, upper right) in the url page. Just pick "add to internet standards report".

Again, the infrastructure was upgraded. There were too much delays occurring in the database cluster so it has been moved to bigger servers and got some performance tweaks.

Lastly, a number of other code changes were moved into production. Besides bug fixes, the software detection has improved. Most notably for Cisco, vBulletin, Jira and Confluence.

Historical suggestions overview with undo

Quite a few of you requested a historical overview of accepted and rejected suggestions. It's available now through the link (right top) on the suggestions page. It features an "undo" button to undo a previous accept or reject. You can also use the search bar to search suggestions and the undo button will appear in your search results too. Try this query to search for a specific domain:
index=suggestions url=*shadowtrackr.com.

The software detection engine got some updates, and the CVE reject bug (where a submitted and later rejected CVEs were shown in the vulnerabilities overview) is gone. Enjoy the new version :-)

Whitelist ShadowTrackr IPs

Some of you whitelist the IPs of the ShadowTrackr scan nodes in firewalls. If you have a trigger happy firewall, this is a good solution. With the number of nodes now in the dozens and each node doing both IPv4 and IPV6 scans, it has become quite the job maintaining this.

So, by popular request, there is an API endpoint available now to automate this process:
API Nod IP whitelist endpoint

The new DNS scheduler is working well, and this week some additional algorithms will be added. Although every step will reduce load on the database, the full benefits will only appear when all scheduler algorithms are transferred. The nodes will then be able to update much faster.

Also, the have been some minor bug fixes that should reduce timeline noise.