Improved discovery and UX

This weekend ShadowTrackr has undergone a quite noticable update. There are a few extra sources to discover urls and ip addresses. If you see previously unknown assets appear on your timeline, this is the reason.

The messages for these new assets have improved too. Instead of spamming your timeline with multiple messages related to the new asset, we now try to put this all in a "details" box in the message that you can expand or close. This keeps the timeline neat and more relevant.

Since the user interface had to be updated for this anyway, it was time to fix some serious UX problems that have been creeping in as well. With any software system that grows organically over time you always run the risk of loosing some consistency with every new feature. A lot of these issues have been fixed now. I'll mention the most important ones.

On most of the internet, blue links are clickable. In ShadowTrackr this was mostly true too. Except for the timestamp. Somehow this ended up being blue and unclickable. It was also in the wrong location. If you look at the place where most timeline-like systems display the timestamp, you'll notice it's near the left top. That's we're users are trained to expect it. We had it at the top right. Oops. That's fixed now.

Eating your own dog food is always a good idea. This will make you feel the pain users are feeling. One of the majotrpain points for me was not being able to copy ip addresses and urls from the timeline in order to investigate them further in other tools. That is fixed now. Any part of the message can be copy-pasted. The link to view the asset, or suggestion, is now displayed in blue at the right top.

About those links, up until now it was sort of a surprise what would happen if you clicked something on the timeline. Would you stay on site and get more details? Or would you end up on an external site to view the source? The new-style links will show you what is going to happen. If the link leads off site, you will see which site you'll be opening. A keyword hit on pastebin.com will have a link that says "open pastebin.com".

If you have any thoughts on the new user interface, I'd be happy to hear them.

Datadump keyword highlighting and context detection

The detection of keywords in datadumps has improved. First of all, there are more lines around the keyword match included in the snippet you see on the timeline. This allows you to better judge if the datadump is something you should worried about or not.

When the snippets get bigger it's easy to lose track of where your matching keyword is. For this reason keywords are now highlighted in the snippets. If there are several keyword matches, the snippet will consists of multiple matches, with a few lines before and after each match.

A bit of an experimental feature is context detection. The data is now scanned for certain characteristics. For instance if a datadump looks like a password dump, you will see an alert icon next to notification. If it looks like the data contains API keys or password, you'll see a warning for this. If things work well, I'll likely do more work on context detection.

-- edit 26 may --
Unfortunately, the context detection worked much better in test than in production. We're seeing too many false positives. It's off now, and back to the drawingboard.

HTTP/2 support and annoying bugs fixed

The number of ips and urls to monitor keeps increasing. This is good news of course, but it also means needing to pay attention to scaling properly. Last week it was finally time to drop a very inefficient JOIN statement from the frontend code. This has been in the making for more than 6 months, since the legacy code and data requiring the JOIN had to be phased out first.

Another performance boost is that since today ShadowTrackr supports HTTP/2. The new ability to process parallel requests helps speed up the loading of some of the slowest pages.

Some clients had a high volume of annoying messages on their timeline, and that should be fixed now. The most important changes:

• Clear and timely SSL certificate messages
• Websites for ignored urls are now ignored too
• The false positive for doubly issued certificates is gone

Lastly, Amazon Cloudfront users experienced some timeline spam due to a lagging ip range update. This resulted in a lot of new ip messages. As of today, Amazon and Cloudflare ip ranges are updated automatically and this problem is fixed.

If you still have annoying messages that you'd like to get rid of, let me know!