After 20 years of growth and multiple acquisitions created fragmented, noisy security logs, Atlassian rearchitected its security data pipeline. By adopting the Open Cybersecurity Schema Framework (OCSF) and building on Databricks, the company achieved massive performance improvements, streamlined data handling, and enabled real-time detection with a flexible, vendor-neutral platform.
- 44 billion daily security events unified via OCSF schema
- Databricks platform delivers 95% faster queries and cost savings
- Automated, type-safe workflows empower scalable log ingestion and deployment
Infrastructure signal
Atlassian opted to build their security data lakehouse on Databricks, effectively handling a high-volume, multi-team environment with native ACID transactions and auto-scaling Spark clusters. This choice simplified schema migrations and ensured resilience across evolving telemetry streams. The migration to OCSF schema consolidated varied proprietary log formats into a single structured dataset, which enabled more effective partitioning and table indexing techniques like Z-ORDER to gain exceptional query speedups.
The platform improvements resulted in a 95% reduction in query time—from about 45 seconds down to just 2–3 seconds for representative analytic operations. Additional infrastructure tuning, such as activating Graviton compute clusters alongside the Photon engine, further cut runtime and query costs by approximately two-thirds. These efficiencies support sustainable cloud cost management while scaling security log processing globally.
Developer impact
Developers now work within a tightly integrated, type-safe environment where the entire OCSF schema is embedded into Go structs, a strategy dubbed 'TypeGo.' This design enforces schema compliance at compile time, effectively preventing invalid log generation and reducing debugging overhead. The strong typing also curbs AI-induced errors during automated transformations, lowering token consumption and accelerating data pipeline development.
The team transitioned from manual, six-phase data onboarding procedures to an automated, self-service platform. Analysts can request canonical datasets that trigger an automated workflow encompassing schema scanning, DAG generation, local testing, canary deployments, and post-validation. This automation not only reduces human error but also rapidly propagates schema updates, empowering security engineers to focus on investigation and response instead of plumbing data pipelines.
What teams should watch
Security teams and data engineers should watch for advancements in adopting standardized, open data schemas like OCSF to unify fragmented telemetry across diverse sources. The benefits of such standardization include improved cross-source correlation, reduced noise, and better compression, which can significantly decrease storage and processing costs at scale. Teams should also explore integrations that embed schema validation deeper into development workflows for robust data correctness.
Cloud architects and platform teams should note the demonstrated gains from combining ACID-compliant lakehouses with advanced table optimizations and specialized hardware acceleration (e.g., Graviton clusters with Photon). This approach enables cost-effective scaling of security observability data while maintaining query performance. Additionally, shift-left automation around deployments and testing can streamline platform operations and enhance reliability.