Cloudflare Logpush Adds SQL Transformers for Real-Time Data Redaction and Filtering

Cloudflare has integrated SQL-based Transformers into Logpush, enabling teams to filter, reshape, and redact log data before it is exported to external destinations.

Threat Times Systems DeskOctober 12, 20262 min read

Share LinkedIn Reddit

Illustration: A sleek black network appliance sits on a desk, with streams of abstract geometric shapes flowing out from its ports into empty storage containers. — AI-generated illustration: Threat Times

Key points

Logpush now supports Transformers, allowing SQL-based filtering and redaction of logs before export.
The feature uses Apache DataFusion and Apache Arrow to process data within the export pipeline.
Teams can reduce downstream storage costs by discarding unwanted records prior to delivery.

Cloudflare has made its Logpush Transformers feature generally available, allowing security and observability teams to filter, redact, and reshape log streams using SQL queries before the data is exported. According to research published by Cloudflare, this update integrates a transformation engine directly into the Logpush export workflow, eliminating the need for separate processing services that teams previously had to deploy, scale, and maintain independently.

The new capability enables users to define SQL queries that reference specific Logpush dataset fields, determining which records and fields are exported. Logpush applies these queries to each batch of data before it reaches the destination, ensuring that only the necessary information is delivered. Records that do not match the specified WHERE clause are removed entirely, while the SELECT statement output defines the final structure of the exported record.

Technical Implementation and Performance

Cloudflare built the Transformer engine using Apache DataFusion and Apache Arrow to handle high-throughput requirements without introducing additional failure scenarios into the export hot path. The company’s data pipeline ingests more than one billion events per second, with the Bufstream delivery service handling over 110 million Logpush records per second. This infrastructure runs across hundreds of Kubernetes pods, distributing work horizontally to ensure reliable delivery and predictable resource use even for high-volume jobs processing millions of logs each second.

Transformers allow for complex data manipulation, including renaming and nesting fields, redacting sensitive information such as parts of client IP addresses, and adding static provider fields to normalize records from multiple CDNs. The transformed results are exported in NDJSON format, matching the input serialization format. This approach helps teams align log data with team-defined schemas used across different analytics platforms or security information and event management systems.

Operational Impact for Security Teams

Observability teams often pay to export, ingest, and store records that they discard immediately upon arrival at their destination. By filtering logs before delivery, organizations can significantly reduce downstream data volume and associated costs. For example, teams can filter out internal and health-check traffic, which is typically irrelevant for security investigations or dashboard visualization. This pre-export processing ensures that each destination receives data in the exact form it requires, simplifying integration with external analytics platforms and object storage solutions.

Customers can create, preview, and manage Transformers through Transformer Studio in the Cloudflare dashboard or programmatically via the API. Once saved, a Transformer can be attached to a compatible NDJSON Logpush job that uses the same dataset. Each update to a Transformer creates a new version, allowing teams to review how the transformation logic has changed over time. This versioning capability supports auditability and controlled deployment of data processing rules within the organization’s security operations.

What should you do?

Security professionals should review their current log export workflows to identify opportunities for pre-export filtering and redaction. Teams using Logpush should evaluate which sensitive fields or irrelevant traffic types, such as health checks, can be removed before data leaves the Cloudflare environment. Organizations should also assess the potential cost savings from reducing downstream data volume by applying SQL-based transformations directly within the Logpush pipeline.

What to do and how to stay safe: Cloudflare

Audit current log export destinations to identify sensitive data fields that should be redacted before leaving the primary environment.
Evaluate the volume of irrelevant logs, such as health checks or internal traffic, that can be filtered out to reduce storage and ingestion costs.
Test SQL-based transformation queries in a staging environment to ensure they correctly reshape data without dropping critical security events.
Monitor resource usage and throughput metrics after deploying Transformers to ensure predictable performance and reliable delivery to external systems.

General security guidance from the Threat Times newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

Does Logpush Transformers convert data between different serialization formats?

No, both the input and output use NDJSON format, so the feature reshapes and filters data but does not convert between serialization formats.

How do teams manage and update Transformers in Cloudflare?

Teams can create and manage Transformers through Transformer Studio in the dashboard or via the API, with each update creating a new version for tracking changes.