SSL certificate reports page

Besides fixing the bugs that come with a major update, I've also been working on some minor improvements. Nothing really fancy, just more consistent checks and better, more specific messages. For instance if the host name on the certificate does not match, ShadowTrackr not only shows the error message but also the common names the certificate is actually issued for. This is the first thing I want to know when I see this error, and I guess most of you too.

The biggest change has been on the certificates reports page. Up until this weekend there was a separate Bleichenbacher/ROBOT report page. It showed wich certificates on which urls where vulnerable. The overview was nice, but why stop there? You'll also want to know about Ticketbleed, DROWN, and all the other SSL vulnerabilities right? So, the separate Bleichenbacher page is now gone. Any certificates vulnerable to this and other attacks are explicitly shown on the certificate report page. Nothing there? That means you're good :-)

Monitoring is now done separately for each url on each endpoint (instead of just the url like before). So if the same certificate appears on an ipv4 and ipv6 address, it will appear twice in the list with the ip shown next to the url. Most of the vulnerabilities you can have are related to the server anyway and not the certificate itself, so this provides a more accurate overview.

Although these are small steps, I hope you enjoy the improvements.

Major Update

I like to do small incremental changes, test them and put them in production. This is less risky and allows me to focus. Unfortunately, not all changes can be done that way. It was time for a Major Update.

There were some performance tweaks that I wanted to push and I had a solution for a long standing queueing problem. Both required updating the internal data structure and migrating the data, which was done during the weekend.

Some of you might have noted that the number of hosts found kept increasing in the last weeks. Old hosts were indeed not always properly removed, but the big issue here were clouds. If a DNS A record was pointing to Microsoft Exchange Online or Amazon AWS, then the IP returned kept changing. After a while, you'd see a whole group of IPs around the url in the attack graph. ShadowTrackr now recognizes clouds and replaces the group with one black dot, with the name of the cloud next to it. The result is that if you are a cloud user, you'll have less assets now.

Another problem, mostly for the bigger clients, was the clutter on the timeline. If you're scanning the internet, you will encounter servers behaving badly and a lot of weird, unpredictable events. Some modules, the DNS module in particular, didn't handle this properly. This is now fixed.

To further cleanup the timeline, some less interesting messages (like a change in the servername) are no longer visible on the general timeline. They are still there when you need them for a more thorough analysis, but only on the timeline of the asset itself. The timestamps no show the timezone (UTC), the less useful source information is left out, and the messages themselves better explain what is happening.

Also, some new features have begun to slip in (it's just to hard to resist temptation). ShadowTrackr has started gathering BGP prefixes to build up data I need later on, and if you run an FTP server you'll noticed the security settings are checked and the banner is grabbed. There will be more of this in the coming weeks :-)

As with any Major Update, you're always afraid his evil twin Major Error comes along. The weekend went well and so far it's only been minor bugs. I expect there will be some new bugs in the coming period. Please let me know if you find one.

Asset grouping and deleting urls

About 30% of the clients have hundreds of urls in their assets, and some even go beyond 1000. This very long, flat list is not very user friendly. So, time for some UX improvement.

I took a look at several of these long lists, and, as expected, there are many subdomains for the same pay level domains. This is good, since it allows grouping them. The new view under assets lists all your domains as clickable groups with the number of subdomains in front:

+ (3) domain.com

When you click it, it will show all subdomains:

- (3) domain.com
  a.domain.com
  b.domain.com
  c.domain.com

Much better :-) Just as before, the domains are sorted alpabetically on the pay level domain ("b.a-domain.com" will appear before "a.b-domain.com").

Another change is that you can now delete all urls, not just the ones you have added manually. I'm still experimenting with how this should be done, and it's likely to change again someday. There are urls that you legitimately want to delete (since they're no longer yours for instance), so the option should be there.

The problem is that some urls are related to you, and even if you don't like it, they will be discovered and added again. No matter how often you delete them. A delete option for these offers false hope, and I don't like the disappointed that follows later on.

Also, I think it's a good idea to keep monitoring your old urls that expired. These are the ideal candidates for setting up phishing sites. The same holds for those internal urls that should not appear on the internet. I hopw to come up with a better solution one day, but until then: be careful when deleting!