Unraveling many events with animated graphs

What if you could just hit play on your data and see what is happening over time?

The same challenges in investigating events come up across many domains. Whether we are analyzing cybersecurity logs, supply chain interactions, misinformation narratives, web application clickstream analysis, financial transactions, social media mining, or criminal call records, we need to quickly investigate basic time questions like progression and root cause.

This article brings together many of our favorite methods for a frequent question Graphistry users ask us: How should we visualize and investigate time data? Additionally, we peek into the new and much-requested live graph animation mode for event data and how it is already streamlining investigation and reporting even further. Read on for an example of the new experience on a real cybersecurity investigation. Future articles will dig into more advanced methods we use here (and make freely available in pygraphistry) such as from geometric machine learning and AI.

Demo: Hunting the red team hackers at Los Alamos National Labs

As a real example, consider the following purple team cybersecurity exercise where blue team defenders must track red team attackers infiltrating their network – or worse, investigating real attackers doing the same thing.

Most defenders do not have an easy way to clearly map event activity over time. They only have tools like Splunk for searching and listing their logs through an infinite scroll. Such tools come with quite limited dashboarding support for showing bar charts, sankey diagrams, timelines, and the like that only go so far.

In contrast, with Graphistry and Louie.AI, it took only about thirty seconds to ask in natural language, “get the red team (auth) index data and draw a graph connecting computers to one another”. Louie then automatically generated and ran a Splunk query to pull the Windows authentication logs, and automatically generated and ran a PyGraphistry Python call to show the data table as an interactive graph. You can see a sample investigation session using the timebar panel in the video below, where each node is a government computer, and each edge is an internal network login.

The result is we can now scroll across time to easily reveal how the attacks occur as a sequence of discrete sessions over the computers. Likewise, stepping through each attacker session using the playback buttons, we can drill into how they progressed. In the Los Alamos case, this quickly revealed that the same computer is used across most sessions, and only occasionally do we see sessions occur where another computer is used as the launching point. Likewise, we see the attackers are logging in at times different than the regular users of the breached computers.

Six tricks for visualizing time

We frequently use a surprisingly small set of simple techniques to visually investigate over time. In turn, several Graphistry features combine well together to support that, and we have streamlined this flow over the years.

Consider coloring. In the video, we use the Graphistry UI to color the red-team login edges red and normal traffic blue. Coloring by type in this way is often a first good step for a graph visualization. However, as edges are events with timestamps, our first step specific for time visualization is to use the edge timestamps to color the edges from blue (oldest are cold) to red (newest are hot). The result is quickly showing behavioral patterns. An activity burst appears with all the same color, a steady progression appears as a smooth gradient, and an ever-present recurring activity appears as a random rainbow. Applying this to our red team dataset, it is immediately obvious that most of the attack activity is concentrated into a small set of short time bursts.

Importantly, no interaction is needed to drill into these temporal patterns and outliers beyond just zooming around!

1. Shape: Represent events as nodes or as edges — all visual entities of that type will now have timestamps.
2. Enrich: For entities related to events, compute timestamps from aggregates such as first/last seen and duration.
3. Layout: Use data science techniques such as umap to layout the graph such that noise is reduced and patterns in the data stand out.
4. Color: Color by time, so you can statically see spikes, progressions, and recurrences across all time.
5. Explore: Use the new timebar panel play feature as part of your first look, then hover over timebar spans to highlight interesting intervals.
6. Drill: Add histograms for additional attributes, and drill in to combinations of values using the timebar and individual histograms.

Animation: Using an extra dimension for visualizing events

We have been working with analysts to roll out new timebar animation features. Most animation usage comes down to two tasks: analysis and reporting. While much of the timebar animation functionality reuses the existing interactive timebar controls, the automatic animation aspect means analysts get to investigate faster, easier, and with more time spent analyzing vs clicking.

Sidenote: How NOT to animate graphs. To understand how Graphistry’s animation approach optimizes for analysts, it is helpful to understand why we found the animation approaches of other graph tools harmful here. Animations in other graph tools often optimize for neat stylistic effects or on smaller data, which often comes at the expense of misleading and slowing down analysts on real investigations. Less usable tools will show nodes disappearing and appearing over time, and typically with node and edge positions on screen continuously moving in response to the change graph size. It’s hard to tell what is happening as there is little constancy, turning a careful workflow into something more akin to a tricky video game. The result is visually stimulating, but of little or even negative analytic to investigations.

Animation as highlighting. Our new animation controls do not hide nor move the nodes and edges: Positioning remains up to the analyst via the separate graph layout play button. Instead, the only change as time progresses is what parts of the graph get highlighted. Just as cross-filtering enables analysts to manually brush over different histogram bins to see what their progressive values highlight in the graph, the new timebar animation controls let them do this with time values, and with much less work.

The timebar panel has three interaction modes, and we jump between all of them when investigating with the timebar panel. Open the timebar panel using the time icon on the toolbar, pick which time attribute to explore, and go:

• Live animation: Toggle on the play mode and hit play. This starts the new animation mode. The animation will highlight data over time. You can interact with the buttons to more quickly scrub to your desired interval.
• Paused animation: Hit pause to halt the animation while keeping the play mode on. You can more easily and precisely manually scrub back-and-forth over time during a paused animation.
• Non-animation: Toggle off the play mode. You can now use mouse timebar hovers to highlight individual intervals, and drill down by shift-clicking to draw time window filters.

We have also worked with analysts to speed up their reporting workflow using play feature animations by:

• Saving animated graphs: Accelerate the creation of intuitive, easy-to-understand videos showing how the graph structure evolves over time by pairing the play feature with screen-capture.
• Sharing: Share your graphs easily with collaborators via Graphistry web links.
• Embedding: Bring all of our graph visualization capabilities, including the play feature, into your own products and reports with only a single line of code.

We’re continuing to improve the new animation mode, such as ensuring the current time window is clear when in presentation settings.

Get started for free!

Try animating their own datasets containing event data on Graphistry Hub, including with free tier accounts, or on your own self-hosted Graphistry server. For self-hosted users, the new features are available starting version 2.41.0.

As soon as Graphistry detects the presence of your time data, the timebar panel toggle becomes visible. As a reminder, to prepare time data, ensure your arrow/parquet table has a datetime-typed column on the node or edge table. For untyped data formats such as CSV and JSON, send string/numeric data with a column that has date/time/_at in its name.

Alternatively, demo the Bitcoin wallet demo dataset built into Graphistry to check out the play feature.

We are already planning additional updates to the timebar animation controls and welcome feedback, so please stop by our community slack channel.

Get started now! Try on a CSV