AI-Powered Firewall Troubleshooting: GraphRAG, Neo4j, RAG & Network Automation

AI-powered firewall troubleshooting can transform how infrastructure teams investigate complex network security incidents. However, building a reliable troubleshooting platform requires more than simply adding an LLM to existing network data.

This field report explains how I built an AI-assisted network troubleshooting pipeline using NSX-T, VRNI, Neo4j, vector search, reranking, Redis, machine learning, and LLMs.

The goal was straightforward: help infrastructure engineers investigate firewall incidents faster while reducing repetitive manual analysis.

Troubleshooting large firewall environments can involve thousands of rules, network flows, IPAM data, CMDB records, and past incidents.. As a result, engineers often spend significant time reconstructing relationships and searching for similar incidents.

The solution combines GraphRAG and semantic vector search with structured graph relationships. A reranker then improves retrieval accuracy, while Redis reduces repeated computation. Finally, independent network flow and log data help verify the diagnosis before the LLM generates an explanation.

Neo4j GraphRAG architecture combining network relationships with semantic vector search

The following sections explain the technical problems I encountered, the solutions I implemented, and the design decisions behind each component.

If you’ve run a firewall estate at any real scale, you know the drill. A ticket comes in: “traffic from 10.x.x.x to
10.y.y.y on port 443 is being dropped, please advise.” Your environment may have over 500,000 firewall rules across NSX-T, Palo Alto, Fortinet, and Check Point. These rules may also span multiple tenants.

When a problem occurs, engineers must answer several questions. Which subnet is the source? Which rule set applies? Is a broad ALLOW rule being blocked by a specific DROP? Have we seen this traffic pattern before? Most importantly, is the issue new, or did we see and fix it three weeks ago?

That last question often takes the most time. Important knowledge about past incidents is usually scattered across engineers, old tickets, and Slack threads. It is not easy to search or reuse.

This post explains the pipeline I built to solve this problem. It also covers the technical challenges I faced and why I chose each tool.

First, the setup

I had already connected CMDB, RVTools, VRNI, IPAM, and DFW rules into a Neo4j graph. This structure is powerful for data with explicit relationships. For example, VM → subnet → rule → firewall → tenant can be traversed through a single Cypher query.

In addition, the graph handles CIDR containment, rule-priority checks, and root-cause path-finding effectively. Therefore, Neo4j was the right data model for the structured relationships in this environment.

However, the problem was semantic meaning

Two rules can be functionally identical while using completely different descriptions, CIDR ranges, or port definitions. An exact Cypher property match cannot reliably determine that these rules represent the same operational intent.

One rule may say “web tier to DB tier,” while another says “app servers to PostgreSQL.”

The Solution: Embeddings

To address this limitation, I introduced embeddings. The bge-m3 model converts rule descriptions, ticket text, and incident notes into vectors that represent their semantic meaning.

Consequently, Neo4j can combine its existing graph relationships with native vector search. This approach makes it possible to search for semantically similar firewall rules without introducing a separate vector database.

This combination of graph traversal and vector search forms the foundation of the GraphRAG approach.

CALL db.index.vector.queryNodes(‘rule_embeddings’, 20, $query_vector)
YIELD node AS rule, score
MATCH (rule)-[:APPLIES_TO]->(subnet:Subnet)
WHERE subnet.tenant = $tenant
RETURN rule.id, rule.description, score
ORDER BY score DESC
LIMIT 20

Neo4j combining structural graph traversal with semantic vector search in one query

One query, two retrieval modes — structural graph traversal and semantic vector similarity, at once. This is the
pattern generally called GraphRAG, and it’s the single highest-leverage change in this whole pipeline.

Issue #2: Vector Search Alone Is Noisy

The setup. Vector search worked, but only 8 of the top 20 results were truly relevant.Vector similarity measures semantic closeness, but it does not determine whether a document actually answers the user’s question. It doesn’t do the deeper, more expensive read of “does this document actually answer this
specific query.”

The fix: a reranker. GTE-Reranker uses the query and each candidate document to score relevance. It is more accurate than vector similarity but too expensive to run across all 37,000 rules.

The solution is a two-step process:

Vector search: 37,000 → 20 results
Reranker: 20 → top 5 results

This approach is fast and improves result accuracy.

Vector search narrows to 20 candidates, then the reranker refines to the top 5

This two-stage retrieval — cheap-and-broad, then expensive-and-narrow — is standard in production RAG systems
for exactly this reason: neither stage alone gets you both speed and accuracy.

GraphRAG combines the strengths of a knowledge graph and semantic search. In network operations, the graph provides structured relationships between devices, subnets, firewall rules, tenants, and applications. Meanwhile, vector search helps identify information with similar meaning even when the terminology is different.

As a result, engineers can search both what is connected and what is semantically similar. This makes GraphRAG particularly useful for firewall rule analysis, incident investigation, network troubleshooting, and AI-powered network operations.

Issue #3: Every Query Hitting the LLM Gets Expensive, Fast

The setup. Once the LLM (in my case GLM-5.2 or Qwen-based models, tiered by task) was generating the final
diagnosis, every single query — including ones that were near-duplicates of something asked an hour earlier —
triggered a full model call. At troubleshooting-tool scale, with multiple engineers asking variations of the same
handful of recurring issues, that’s a lot of redundant GPU time.

The fix: Redis as a cache layer — in front of everything, not just the LLM. Redis isn’t a database
replacement for Neo4j; it’s a fast, ephemeral, key-value cache that sits in front of both the LLM calls and the
reranker calls.

Deployment: standalone RHEL VM
maxmemory-policy: allkeys-lru ← critical: evict oldest, never crash on full memory
persistence: light RDB, no AOF ← everything here is regenerable from source of truth
ACLs per consumer (LiteLLM, DFW Checker, Jarvis) instead of one shared password

Two cache layers, specifically: 1. LLM response cache (exact-match and semantic near-duplicate) — LiteLLM
supports this natively 2. Reranker result cache, keyed on (query_text, candidate_id_set) — skips the expensive
cross-attention pass entirely on repeat queries.

The setup. This is the trap that’s easy to miss until it bites you. A cached “why is this dropping” answer is only
correct until someone pushes a rule change affecting that exact traffic. TTL-based expiry (just wait N minutes and
hope) means you’re either serving stale, actively wrong answers for a while, or setting TTLs so short the cache
barely helps.

The fix: tie invalidation to the event you already have. Every firewall rule push in my environment already
syncs to Neo4j — that’s an existing trigger, not new plumbing. The fix was making that same sync event also invalidate exactly the Redis keys derived from the changed rule, using a tagging index rather than a guess

#At cache-write time
redis.set(cache_key, result, ex=ttl)
redis.sadd(f”rule:{rule_id}:cache_keys”, cache_key)
#On rule-change sync event (existing hook)
def on_rule_change(rule_id):
keys = redis.smembers(f”rule:{rule_id}:cache_keys”)
if keys:
redis.delete(*keys)
redis.delete(f”rule:{rule_id}:cache_keys”)

This turns the cache from “stale until timeout” into “correct immediately after any relevant change” — the
difference between a cache that saves time and one that quietly erodes trust in the tool.

Issue #5: The Temptation to Reach for a Neural Network When You Don’t
Need One

The setup. Once you’re deep in embeddings, rerankers, and LLMs, everything starts to look like a deep-learning
problem. I hit this directly when building an ALLOW/DROP enforcement predictor: given a historical flow tuple
(source, destination, port, protocol), predict whether it gets dropped, using millions of rows of past VRNI flow
outcomes.

The fix: match model complexity to data shape, not to what’s trendy. Four input features, tabular, with a
clear historical base-rate signal — that’s the textbook case for logistic regression or gradient boosting (XGBoost),
not a neural network. A neural net needs volume and dimensionality to justify its complexity; with this little
structure, a tree-based model matches or beats it on accuracy, trains in seconds on CPU, and — critically for a
troubleshooting tool — stays interpretable: you can see which feature drove a prediction, not just the number.

Neural networks earned their place exactly once in this stack: a separate call-intelligence project turning audio into
structured meaning, where the input is genuinely unstructured. Everywhere else — tabular flow data, graph
relationships — a neural net would have been the wrong tool wearing a trendy hat.

This model runs as the first check in the troubleshooting pipeline, before anything touches the graph, precisely
because it’s the cheapest signal available: milliseconds, no database round-trip, subnet index baked into a selfcontained
pickle file.

Issue #6: A Rule Match Isn’t Proof — You Have to Verify Against What
Actually Happened

The setup. Everything above — the graph, the vectors, the reranker, the ML prior — produces a hypothesis:
“based on rule structure and history, this traffic should behave like X.” That’s not the same as knowing what
actually happened on the wire. Rules describe intent. They don’t guarantee reality. A rule can look correct in the
NSX manager and still not be why traffic is failing — routing, a host-based firewall, an intermediate hop, or simple
rule-priority ordering can all produce a mismatch between “what the rule says” and “what the packet actually did.”

The fix: cross-check the hypothesis against two independent sources of ground truth before trusting a
diagnosis.
VRNI (flow-level truth) — did the traffic actually flow at all, in either direction, at what volume, and when? If
the rule chain says ALLOW but VRNI shows zero flows in either direction, the problem likely isn’t the firewall
rule — it’s routing, or the destination service isn’t listening, or the source never actually attempted the
connection.
VRLI (log-level truth) — what did the enforcement device itself log? This is the actual accept/deny event
from NSX-T or the physical firewall, correlated by timestamp and tuple. If VRLI shows an explicit DENY logged
by the device even though the rule you found in the graph says ALLOW, that’s a strong signal of a rule-priority
conflict — a more specific DROP rule higher in evaluation order is shadowing the ALLOW you matched on.

The other checks that guard against a false diagnosis before it ever reaches VRLI/VRNI: – CIDR-aware
rule matching — confirming the resolved subnet actually falls inside the rule’s declared scope, not just a loose IPstring
match – ALLOW vs. DROP priority ordering — checking evaluation order explicitly, since NSX-T (like most
firewalls) evaluates top-down and a broad ALLOW lower in the list never overrides a narrower DROP above it –

CMDB/IPAM consistency — confirming the resolved owner, tenant, and subnet classification actually agree across
sources before trusting the graph relationship at all; a stale CMDB record pointing to the wrong tenant would
silently poison every downstream check
Only once the rule-based hypothesis, the priority check, and both independent log/flow sources agree does the
pipeline hand a confirmed diagnosis to the LLM to narrate — rather than the LLM confidently explaining something
that was only ever a plausible-looking rule match. This is the difference between an AI tool that sounds right and
one that’s actually checked its own answer against reality before saying anything.

The Full Pipeline, End to End

Full end-to-end pipeline: Redis, ML predictor, Neo4j hybrid retrieval, reranker, VRNI/VRLI verification,
tiered LLM, and cache write-back

Cost and latency stack cheapest-first: a Redis lookup costs ~1ms, ML inference ~10ms, the graph+vector query
~50-100ms, the reranker ~200-500ms, and the LLM call is the expensive tail end — seconds, especially on frontiertier
models. Most repeated or routine questions resolve in the first two stages and never reach the LLM at all.

Why Each Product, Specifically

What I’d Tell Someone Starting This From Scratch
Build in the order the cost curve suggests, not the order that feels most “AI”:

1. Redis first — it’s foundational infrastructure everything else will lean on, and the LLM cache alone pays for
   the effort almost immediately.
2. Vector index on your existing graph — you already have the data; embedding it and adding a vector
   index is additive, not a rebuild.
3. Reranker — only once retrieval is noisy enough to notice; don’t add it prematurely.
4. Cache invalidation tied to your real change events — do this before you trust the cache in production,
   not after a stale answer causes an incident.
5. Classical ML for tabular signals — resist the urge to reach for a neural network until your data actually
   stops being tabular.

The unifying lesson across every one of these decisions: the right tool is determined by the shape of the data and
the cost of being wrong or slow — not by which tool is newest. A graph for relationships, embeddings for meaning, a
reranker for precision, a cache for speed, classical ML for structured tabular signals, and a neural network only
when the input is genuinely unstructured. Once that framework clicks, the specific product names become
swappable details.

AI-powered network troubleshooting is not about replacing every existing network tool with an LLM. Instead, the most reliable architecture combines the right technology with the right type of data.

In this pipeline, Redis provides speed, classical machine learning handles structured signals, Neo4j manages relationships, vector search adds semantic retrieval, reranking improves precision, and the LLM generates the final explanation.

Most importantly, the LLM does not operate as the source of truth. The troubleshooting pipeline verifies its hypotheses against network flows, logs, rule priorities, CMDB data, and IPAM information before presenting a diagnosis.

This approach creates a more reliable foundation for AI-powered firewall troubleshooting, network automation, intelligent network operations, and enterprise network security.

The broader lesson is simple: choose the technology according to the data, the operational requirement, and the cost of being wrong. A graph is valuable for relationships. Vector search is valuable for meaning. Reranking improves precision. Caching reduces latency and cost. Classical ML works well for structured data. LLMs become most valuable when they are given reliable context and verified information.

For infrastructure teams exploring AI-powered network operations, this architecture provides a practical starting point. The individual technologies can change over time; however, the underlying design principles remain useful across enterprise networking environments.

Building an AI-powered network troubleshooting platform requires more than deploying an LLM. It involves data integration, network architecture, security, automation, observability, AI infrastructure, and reliable validation.

If your organization is exploring AI-powered network operations, network automation, enterprise infrastructure, cybersecurity, or intelligent troubleshooting, Trezbon can help you evaluate the right technology approach for your environment.

Explore Trezbon’s technology services:
Visit Trezbon

Have an AI or network infrastructure project in mind? Reach out to Trezbon to discuss your requirements.