The blacklist counter from hell

The blacklisted page up until now listed all your hits on blacklists. That is, every hit is a separate entry in the table on your screen and is counted as a problem. The number of problems you have screams at you as a number in a red dot in the menu on your screen. Sounds good right?

As one client showed me, some ideas sound good in theory but turn into the blacklist counter from hell in practice. ShadowTrackr at this moment checks your ip addresses and websites against 127 blacklists. A lot of these blacklists overlap and from a security point of view that's just fine. You'd rather be notified twice than not at all.

When an ip gets listed as a source of SPAM on one blacklist, the chances are high that a couple of other blacklists will pick it up too. Since the counter counted the number of blacklist entries, 2 machines getting listed on 4 spamlists resulted in the number 8 screaming at you from the bright red dot. That is not the user experience I intended. In that case you have 2 problems, not 8. The counter is fixed now, and all blacklist entries are sorted per asset now.

What remains is the question on how to handle notifications. For the first time your asset is listed on any blacklist, everyone will want to receive a notification. But how about the second or third blacklist that same asset gets listed on? Do you want to know? I myself would like to get notified of every extra blacklist an asset appears on, so I left it on for now. But if enough users convince me otherwise I'll be happy to turn it off of course. Just let me know!

Red dots on the attack surface map

The attack surface map gives you a good overview of your assets and how they're are related. You can quickly see where most of your servers and websites are, and easily spot the outliers.Wouldn't it be great if it also showed where your problems are? Starting today, it does!

Any ip or url that is on a blacklist somewhere will turn red. Websites with troublesome certificates will be orange, and bad certificates will be red too. Of cource, there's a similar rating for servers. If a server has a troublesome port open it will be orange. The really bad ones (think pownable or DDOS amplifiers) will be red.

I'm quite happy with the result. You'll have an instant view of where most of your problems are and where you need to start improving your security. The thing that does need some work is the layout for really big (3000+ assets) organisations. It still works, but it's just not as beautiful. The attack surface map is built with D3 and it allows for very specific tweaking of the various forces in the force-layout graph that I use, so it should be solvable. I've put it on my todo list and will come back on this later. For now, have fun with the new fancy attack surface map.

Suggestions for new assets

The new algorithms for finding your websites and servers work great. Shadowtrackr is finding and monitoring more than ever. A bit too much actually.

Some clients use shared services, and without any restrictions the other websites and servers on the shared infrastructure were automatically added to assets and used for expanding in turn. Without a proper stop condition, this could end up adding most of the internet. One client using a shared Baidu server ended up with 42 unrelated Baidu machines within a couple of hours. Yes, 42.

I've thought long and hard about a proper stop condition, but there isn't any that I can come up with. If machines are not on a dedicated ip (range) for you but on shared servers, there is no way of reliably determining if all urls pointing to it are really yours. You might be able to relate some of them with Whois information or by analysing links on websites, but this does not solve all cases. Whois data is not always available and larger companies tend to have several different whois contacts anyway.

The most user friendly solution I could come up with is offering suggestions. When a new server or domain is found that somehow relates to one of your assets but is not obviously yours, ShadowTrackr will "suggest" it to you and tell you what existing asset it is related to. You then have the option to reject or accept it. Check out the suggestions page in the menu to see yours.

I'm still thinking of ways to minimising the user interaction needed, like tracking known shared hosting and automatically rejecting suggested assets on it. For large organisations the initial amount of rejections needed can build up to dozens or even more than a hundred suggestions. After the initial load that number stays acceptably low though.