Does build-time image scanning replace runtime security tools like Falco?

No. Build-time scanning checks a container image for known vulnerabilities before it ships. Runtime security watches the syscalls a container actually makes after it starts, which catches behaviour that no static scan can predict, such as a process getting an unexpected shell or reading a credential it should not. The two are complementary, not substitutes for each other.

Why would a Kubernetes DaemonSet report a successful rollout while a node has no coverage?

A DaemonSet controller reports success based on how many pods came up, not on whether every intended node is covered. If one pod cannot schedule because of a resource conflict, most tooling still reads as "mostly successful" rather than surfacing the specific node that went unnoticed, which is worth checking directly by comparing ready pod count against total node count.

Why might a Prometheus alert never fire even though the underlying query looks correct?

Two common causes: an aggregation over a completely empty set of series returns an empty result rather than zero, so a comparison against it never evaluates true unless you add an explicit fallback; and a severity or label filter can be scoped far more narrowly than the ruleset actually distributes, matching only a small slice of what you intended to cover. Both pass a casual read of the query and only show up when you trigger a real event and confirm the alert actually arrives.

Runtime Security on AKS: Eight Ways Falco Looked Healthy and Detected Nothing

2026-09-02

We deployed Falco runtime security to AKS via the operator. The deployment was the easy part. The real story is eight ways a healthy-looking deployment, configuration, or detection path can silently lose security coverage.

We run our own AKS infrastructure, and we recently added Falco, the CNCF-graduated runtime security engine, to watch syscalls at the kernel level on every node. The deployment itself was uneventful: the operator model, the CRDs, the eBPF driver, all worked roughly as documented. The problems started after "it's running". Eight separate checks, configurations, and detection paths looked healthy while failing to provide the functionality we expected. This post documents all eight failures, because the same pattern applies beyond Falco: a monitoring system can be more dangerous than no monitoring system when its health signals suggest that the gap is covered.

Why build-time scanning is not enough

We already scan container images before they ship. That answers one question: was the image we built free of known vulnerabilities at build time. It says nothing about what happens after the container starts. Between a deploy and a compromise, a process getting a shell, reading a credential it should not, or writing to a filesystem it should not, produces no application log at all. Runtime security closes that specific gap: it watches the actual syscalls a running container makes, not the image it was built from.

Falco does this with an eBPF probe on every node, comparing every syscall against a rule set, and emitting an event when one matches. No kernel module to build, and no separate security agent to install on every workload. It is also vendor-neutral: no data leaves the cluster unless you choose to ship it somewhere.

Why we picked the operator over the Helm chart, and what that decision cost later

Falco ships two supported install paths: a Helm chart, and a Kubernetes operator with its own set of CRDs. We chose the operator for one property: rule changes go through a CRD, and the operator's sidecar hot-reloads Falco in place. This also avoids a pod restart and the resulting detection gap while a privileged DaemonSet rolls. The trade-off showed up later: the CRDs are still an early API version, and the schema has already changed once upstream. Several of the eight problems below trace back to how that CRD layer merges configuration, which behaves differently from what a values.yaml file would have done.

Eight ways it looked fine and was not

None of these showed up as a crash, a red pipeline, or a failed readiness probe. Every one of them looked, from the outside, like a working deployment.

1. Omitting a field and zeroing it are different things to a structured merge

Our first dev rollout deliberately left a resource request unset on the DaemonSet, reasoning that an omitted key means no request. That reasoning holds for a plain pod spec. The operator behaves differently: it merges the CR's pod template with its own defaults using structured, per-key merging. Leaving a key out does not remove it from the result, it just lets the operator's own default value through underneath. One node in the cluster was running close to its scheduling ceiling, and the default that leaked through was just enough to push a pod onto that node into Pending. The DaemonSet controller still reported a mostly successful rollout because most pods had started. The busiest node in the cluster therefore had no coverage.

The fix needs an explicit zero written into the field. In a quick YAML review they can look equivalent, but the structured merge treats them differently.

2. One changed line, one entire section overwritten without warning

Falco's own configuration layer supports config fragments that combine into one effective config. We changed a single nested setting inside the metrics section, to fix the scrape interval, and left the rest of that section alone, expecting the usual deep merge everyone assumes YAML fragments get. Under the configuration merge behaviour we were using, a fragment replaced the entire top-level section it touched, not just the keys it named. Our one-line change invisibly deleted the flag that turns the metrics endpoint on at all. Every pod stayed Ready. The readiness probe does not check the metrics endpoint, it checks that Falco itself is alive. The endpoint began returning not-found on every single pod in the cluster, and nothing about the deployment's health signals would ever have shown it, short of someone actually hitting the endpoint.

A healthy pod does not prove that a separate feature endpoint is working. Check the endpoint itself.

3. A severity filter that matches almost nothing

We initially built our detection alert around a severity floor, assuming a clean cutoff would separate the events worth paging on from background noise. Looking at the actual events and the rules we had loaded showed a problem: important detections do not all sit at the severity you would intuitively expect. Several of the rules we relied on most, including a flagship interactive-shell detection, sit one level below where a naive severity floor would draw the line. A Warning-or-higher filter would therefore have excluded detections we explicitly wanted to see, with no warning that they were missing, while still technically working for the rare case it did cover.

# Critical-only. Looks reasonable, and misses almost everything we
# actually care about.
sum by (rule, output_fields_k8s_ns_name, output_fields_k8s_pod_name) (
  count_over_time(
    {namespace="falco"}
    | json
    | __error__=""
    | priority =~ "Emergency|Alert|Critical" [5m]
  )
) > 0

# Widened deliberately to include Notice, because some of the
# detections we rely on most live there, not above it.
sum by (rule, output_fields_k8s_ns_name, output_fields_k8s_pod_name) (
  count_over_time(
    {namespace="falco"}
    | json
    | __error__=""
    | priority =~ "Emergency|Alert|Critical|Error|Warning|Notice" [10m]
  )
) > 0

Do not choose a severity floor from the names alone. Look at the priorities of the rules you actually load and the events they actually produce. We ended up including Notice deliberately, because some of the detections we care about most live there.

4. Documentation said a field could not do the one thing that mattered

The CRD field that pins which build of a rule set or plugin to pull was documented, in a comment we trusted, as accepting a tag only, with no way to pin a specific content digest. We believed it and shipped with a version tag. Registry tags are mutable, though, so all a version tag really bought us was convenience, when we thought we had an integrity guarantee. Re-reading the operator's own source rather than the comment above the field showed the field also accepts a digest, a detail the comment never mentioned, and resolves it correctly either way. The field's own inline description said as much; the comment we had trusted did not match the code it sat above. That mismatch cost roughly a week of running on a mutable reference for something that should have been pinned from day one.

A comment describing what a field accepts is somebody's claim about the code, made at some point in the past, and it can drift out of date the moment the code underneath it changes. When the stakes are supply-chain integrity, trace the actual code path that resolves the value.

5. An alert that summed two environments into one meaningless number

Our metrics backend holds more than one environment's series in the same store, distinguished by a cluster label. One alert used an aggregate function without grouping by that label, which is a natural thing to type and an easy thing to miss in review, because the query still runs and still returns a number. The resulting value was not the value intended by the alert: the alert compared one environment's pod count against a completely different environment's node count, summed together as if they were one system. The query returned a plausible value on every evaluation, but it was comparing different populations than intended.

An unscoped aggregate over a shared metrics store is not a bug that throws an error. It is a bug that returns a number, and a wrong number that looks reasonable is far more dangerous than a query that visibly fails.

6. The alert built to catch zero coverage returned nothing itself

We wrote a health alert to catch the case where Falco itself goes missing from an environment entirely, comparing the number of healthy Falco instances against the number of nodes. The logic reads correctly: if the count of running instances is less than the count of nodes, something is down. The problem is that an aggregation over an empty set of series returns no result rather than zero, and a comparison against an empty result does not evaluate to true. As a result, the alert did not fire when Falco was completely absent, which was precisely the condition it was intended to detect. Adding an explicit "or vector(0)" fallback closed the gap.

# Cannot fire in the case it exists to catch: an empty result is not zero.
count(up{job="the-thing-you-are-watching"} == 1) < count(node_info)

# The fix: force the missing case to evaluate as zero, explicitly.
(count(up{job="the-thing-you-are-watching"} == 1) or vector(0)) < count(node_info)

If an alert depends on an aggregation being zero when there are no series at all, test that empty-series case explicitly. In PromQL, an empty result is not the same thing as zero.

7. A test step expecting a rule that was never in the loaded set

Our manual test plan included a step expecting a specific detection to fire for a common attacker action taken inside a container. It never fired, so our first assumption was that the detection engine had failed. It had not. That particular rule lives in an incubating tier of the upstream rule set, one step short of the stable set we actually load, a distinction that is easy to miss when skimming a rule catalogue on a project's website instead of checking the exact file your deployment pulls. The test step had been written against the catalogue instead of the ruleset actually running. We fixed the test plan, not the deployment, once we confirmed which rules were actually loaded by inspecting the live rule set rather than trusting the catalogue or the test plan.

When a detection test fails, first check that the rule exists in the ruleset actually loaded by the running deployment. A rule catalogue is not the same thing as the rules you are running.

8. A dashboard variable with label and value swapped

Our dashboarding tool's custom variables are defined as a display label paired with an underlying value, and it is entirely possible to write the two the wrong way round without any validation catching it, because both are just strings and the editor accepts either order without complaint. One dashboard variable had them transposed: every panel filtered on the human-readable label instead of the actual cluster value used by the Falco series. Every panel matched nothing. The dashboard rendered perfectly, on brand, correctly laid out, and showed No Data everywhere, which is exactly the layout a broken data source would also produce. Nothing about the dashboard itself signalled which of the two failure modes it was.

A perfectly rendered dashboard full of No Data panels and a truly broken Falco data pipeline look identical from a glance. Check the actual query and the labels it filters before assuming the problem is in Falco itself.

The capacity story: full without being busy

Separate from the eight detection gaps, our development cluster had a scheduling problem that will be familiar to anyone who has run a small AKS node pool: a node pinned near the ceiling of its CPU requests, while its actual CPU usage sat in the single digits. Nothing new could schedule there, and the plain read of the metrics said the cluster was almost idle. Both were true at once. Requests are an accounting reservation against the scheduler, and a node can be completely full of those reservations while the services holding them individually use almost none of what they asked for.

A privileged DaemonSet that needs to run on every node, including the busiest one, does not get to wait for that to resolve itself. The fix we shipped for the development environment was to run the agent with no CPU request at all, accepting that it would be the first thing throttled under real contention. We paired that with an alert on falcosecurity_scap_n_drops_total, so a throttled agent dropping syscalls would at least become a visible degraded signal rather than a false sense of full coverage. In production, where there is real headroom, we kept a normal request. The two environments legitimately warranted different answers, and picking one policy for both would have been wrong in one direction or the other.

That distinction is intentional in how we split our alerting: one set of alerts tells us whether the Falco detection system itself is healthy and receiving data, and a separate set tells us that Falco actually observed something worth acting on. Neither one substitutes for the other, and several of the eight failures above were exactly a healthy-looking answer to the first question standing in for the second.

The honest cons

We would still make the same call, but it is worth stating the costs plainly rather than only the wins.

It is a privileged component running on every node. Broad host visibility is exactly what a detection engine needs to do its job, and it is also a meaningfully larger footprint than most of what runs in the cluster. That trade is standard for this category of tool, but it is a real one, not a footnote.
Detections nobody triages are worse than no detections. A rule that fires into a channel nobody watches documents, at audit time, that something was seen and ignored. Shipping this only makes sense with a real owner and a real routing destination for the alerts, not just a running pod.
Even the upstream rule set is not automatically correct for your environment. One of the exceptions the ruleset ships out of the box for a common, well-behaved operator's normal file-handling behaviour turned out to cover only one of the two code paths that operator actually uses, missing the other by one command. "The vendor's default rules already handle this" is a claim worth verifying against what your own workloads actually do, not one worth taking on faith.

The common lesson is broader than Falco: passing surface-level checks such as pod readiness, dashboard rendering, query execution, and pipeline status does not prove that a monitoring or detection system is providing coverage. The only way to know it works is to make it detect something real, on purpose, and watch the alert actually arrive.