stuff I figured out the hard way

Defaults:

  • Run the modern eBPF probe with container collectors enabled.
  • Load the sandbox ruleset but enable only the file-integrity rules.
  • Set the priority floor to notice; route by priority in Falcosidekick.
  • Tune in Git via falco_rules.local.yaml or customRules, never upstream files.
  • Ship JSON alerts to storage retained for at least 12 months.

What Falco is, and isn’t

Falco watches system calls on every node and alerts when one matches a rule: a shell in a production container, a write to /etc, execution of a binary that wasn’t in the image.

It detects; it doesn’t block. Admission policy (Kyverno, Gatekeeper) stops bad workloads being created, Falco catches bad behaviour in running ones. Most compliance programs want both.

flowchart LR
    K["Kernel<br/>system calls"] -->|"modern eBPF probe"| D[Falco DaemonSet<br/>one per node]
    META["Container runtime<br/>pod, namespace, image"] --> D
    D -->|"rules engine:<br/>conditions match?"| OUT[Alerts as JSON]
    OUT --> SK[Falcosidekick]
    SK --> SIEM["SIEM / S3 / Loki<br/>retained as evidence"]
    SK --> PAGE["Slack / PagerDuty<br/>for urgent ones"]

Rules in one minute

A rules file has three building blocks:

  • list - a named set of values: [/etc, /usr/bin]
  • macro - a named, reusable condition: open_write
  • rule - a condition, an output message, a priority and tags
- rule: Terminal shell in container
  condition: >
    spawned_process and container and shell_procs
    and proc.tty != 0 and container_entrypoint
    and not user_expected_terminal_shell_in_container_conditions
  output: A shell was spawned in a container with an attached terminal | user=%user.name command=%proc.cmdline ...
  priority: NOTICE
  tags: [maturity_stable, container, shell, mitre_execution, T1059]

Official rules ship in three maturity levels:

RulesetWhat’s in itLoaded by default
falco_rules.yaml (stable)25 well-tested, low-noise rulesYes
falco-incubating_rules.yaml32 more, e.g. privileged containers, setuid, cron changesNo
falco-sandbox_rules.yaml39 experimental ones, including Write below etc and Write below binary dirNo

The file-integrity rules compliance needs live in sandbox because they’re noisy until tuned. Counts are from rules 5.x/6.x and change between releases.

Mapping to compliance frameworks

Falco produces evidence for these controls. It doesn’t satisfy them on its own, and auditors decide what counts - treat this table as a starting point, not a certification.

FrameworkControlFalco rules that provide evidence
PCI DSS v4.010.2.1.2 - actions by anyone with admin access, including interactive useTerminal shell in container, System user interactive
10.2.1.6 / 10.3 - audit logs stopped, or protected from tamperingClear Log Activities
10.2.1.7 - creation and deletion of system-level objectsWrite below binary dir, Drop and execute new binary in container
11.5.1 - intrusion detectionDrop and execute new binary, Fileless execution via memfd_create, Detect release_agent File Container Escapes
11.5.2 - change detection on critical filesWrite below etc, Write below binary dir, plus your own path rules
SOC 2CC7.2 - monitor system components for anomaliesThe whole ruleset, routed to someone who responds
CC6.8 - detect unauthorized or malicious softwareDrop and execute new binary, Execution from /dev/shm
ISO 27001:2022A.8.16 - monitoring activitiesThe whole ruleset
A.8.15 - loggingJSON alerts shipped to retained storage
NIST 800-53SI-4 system monitoring; SI-7 integrity; AU-2/AU-12 event loggingAs above
CIS Kubernetes BenchmarkPrivileged containersLaunch Privileged Container

Tag custom rules with control IDs so you can filter evidence by control. Some official rules already do: NIST_800-53_AU-10 on Clear Log Activities, PCI_DSS_11.5.1 on Drop and execute new binary, PCI_DSS_10.2.5 on Launch Privileged Container.

Retention matters as much as detection. PCI DSS 10.5.1 wants audit logs kept at least 12 months, the last three immediately available. Falco keeps nothing - retention is up to wherever you send alerts.

A compliance baseline (Helm)

Tested with helm template against chart 9.2.0 (Falco 0.45.0).

driver:
  kind: modern_ebpf          # CO-RE probe: no kernel module to build

collectors:
  enabled: true              # pod, namespace and image metadata on every alert

falcoctl:
  config:
    artifact:
      install:
        refs: [falco-rules:5, falco-incubating-rules:6, falco-sandbox-rules:6]
      follow:
        refs: [falco-rules:5, falco-incubating-rules:6, falco-sandbox-rules:6]

falco:
  rules_files:
    - /etc/falco/falco_rules.yaml
    - /etc/falco/falco-incubating_rules.yaml
    - /etc/falco/falco-sandbox_rules.yaml
    - /etc/falco/falco_rules.local.yaml
    - /etc/falco/rules.d
  rules:
    # Load sandbox, but enable only the file-integrity rules
    - disable:
        tag: maturity_sandbox
    - enable:
        rule: Write below etc
    - enable:
        rule: Write below binary dir
  priority: notice           # default is debug: everything
  json_output: true
  json_include_output_property: true
  json_include_tags_property: true          # filter evidence by compliance tag
  json_include_output_fields_property: true
  buffered_outputs: false
  append_output:
    - match:
        source: syscall
      extra_fields:
        - k8s.ns.name
        - k8s.pod.name
        - container.image.repository
  syscall_event_drops:
    actions: [log, alert]    # a dropped-events alert is itself evidence of a gap
  webserver:
    enabled: true
    prometheus_metrics_enabled: true
  metrics:
    enabled: true
    interval: 1m
    output_rule: false       # metrics to Prometheus, not into your alert stream

falcosidekick:
  enabled: true              # the chart points Falco's http_output at it

Configure Falcosidekick’s outputs (S3, Elasticsearch, Loki, Slack, PagerDuty) in its own values. Set minimumpriority per output so only urgent alerts page anyone.

Tuning without losing evidence

Tune narrowly, in Git, so every exception is reviewed. Noise kills Falco deployments because people stop reading alerts, and the review trail doubles as change-management evidence. Put overrides in falco_rules.local.yaml or the chart’s customRules, never in the upstream files.

Fill in a rule’s built-in hook macro. Many rules end with and not user_... macros that default to never_true:

- macro: user_expected_terminal_shell_in_container_conditions
  condition: (container.image.repository = registry.example.com/platform/debug-tools)
  override:
    condition: replace

Add a narrow exception - by image and path, never by process name alone:

- rule: Write below etc
  exceptions:
    - name: config_reloader
      fields: [container.image.repository, fd.directory]
      comps: [=, =]
      values:
        - [registry.example.com/platform/config-reloader, /etc/app]
  override:
    exceptions: append

Write your own rule for data the framework cares about - here PCI DSS 10.2.1.1, access to cardholder data:

- list: cardholder_data_paths
  items: [/var/lib/payments/cards]

- rule: Unexpected access to cardholder data
  desc: A process other than the payments service read cardholder data files
  condition: >
    open_read
    and fd.name pmatch (cardholder_data_paths)
    and not proc.name in (payments)
  output: >
    Unexpected cardholder data access | file=%fd.name process=%proc.name
    command=%proc.cmdline user=%user.name container=%container.name
    image=%container.image.repository
  priority: WARNING
  tags: [filesystem, pci_dss, PCI_DSS_10.2.1.1]

pmatch matches path prefixes, so /var/lib/payments/cards covers everything below it.

Gotchas

  • Default priority is debug. Unchanged, every informational event goes out. Set a floor and route by priority in Falcosidekick rather than dropping events auditors need.
  • Compliance rules are off by default. File integrity lives in the sandbox ruleset. Load it deliberately and enable only what you’ll tune.
  • Rules update themselves. falcoctl follow pulls new rule versions weekly, which in a controlled environment is an unreviewed production change. Pin versions and update through GitOps if your change process requires it.
  • Dropped events are blind spots. Under heavy load the kernel buffer overflows and events are lost. Keep the syscall_event_drops alert, watch the drop metrics and raise buf_size_preset on busy nodes.
  • Falco is a target. An attacker with root will try to kill it, and a node without the DaemonSet is uncovered. Alert on a missing Falco heartbeat as a security event, and give the DaemonSet tolerations for every taint.
  • Alerts can leak secrets. %proc.cmdline captures arguments, including passwords passed as flags, which land in your SIEM and can pull it into PCI scope. Fix the apps, or mask fields in Falcosidekick.
  • proc.name is truncated. The kernel cuts it to 15 characters, so payments-reconciler becomes payments-reconc. Match on proc.exepath for long names.
  • No metadata, weak evidence. Without the container collectors, alerts show container IDs, not pods, namespaces and images. Enable collectors so auditors and responders get names.
  • Driver requirements. The modern eBPF probe needs kernel 5.8+ with BTF, the legacy eBPF probe has been removed, and serverless nodes like EKS Fargate can’t run Falco at all. Fall back to the kernel module where the probe can’t run.
  • Bad time, bad evidence. Audit evidence is only as good as its timestamps. Keep NTP healthy on every node (PCI DSS 10.6).
  • Detection without response is just logging. An alert with no owner gets no response, which is what SOC 2 CC7.3 and CC7.4 ask about next. Give every alert that matters an owner and a runbook.