Amazon launches OpenSearch integration with Security Lake

Amazon has announced the general availability of its OpenSearch Service zero-ETL integration with Amazon Security Lake.

The new offering is designed to enable organisations to efficiently search, analyse, and gain actionable insights from their security data, which could streamline complex data engineering requirements. It allows for in-place query and analysis of logs in Security Lake, reducing the need for data duplication and operational management of custom data pipelines.

Channy Yun, Principal Developer Advocate at AWS, explained, "With OpenSearch Service zero-ETL integration with Security Lake, you can use the rich analytics capabilities of OpenSearch Dashboards to query and visualise your data in Security Lake. You can also analyse multiple data sources within a single tool and a single schema, the Open Cybersecurity Schema Framework (OCSF) schema to help with threat-hunting and investigation scenarios."

The integration also provides an option to boost query performance by enabling additional accelerations, such as indexed views and dashboards, within Amazon OpenSearch Service. This feature supports time-sensitive investigations and monitoring by offering fast and frequent access to specific data subsets, thus providing complete visibility into Security Lake-stored data irrespective of log volume.

"These capabilities provide complete visibility into all your data stored in Security Lake, regardless of the log volume, to support security investigations, better understanding of your security posture, and gain security-relevant insights," Yun added.

To begin using the service, organisations need to enable Security Lake by creating a Security Lake subscriber and subsequently establishing a data connection in Amazon OpenSearch Service. This process automatically generates an OpenSearch Serverless collection to store direct query results and indices.

Users need to configure permissions and select data sources like Amazon Route 53 DNS queries, AWS CloudTrail logs, Amazon VPC Flow logs, and AWS Security Hub findings. Once the setup is completed, all collected data are ingested into an Amazon S3 bucket within the user's account.

For users accessing Security Lake from a different account, AWS Lake Formation provides cross-account permissions to facilitate data access through AWS Glue tables. It involves setting up an AWS Lake Formation subscriber and accepting sharing via AWS Resource Access Manager.

To create a zero-ETL integration, customers can utilise the OpenSearch Service console to connect to Security Lake as a data source. The setup automatically establishes an OpenSearch Serverless collection and application, alongside offering pre-built dashboards for data visualisation.

In the OpenSearch Dashboards, users can employ the Discover page to locate specific Security Lake tables for querying. The interface supports the use of Piped Processing Language (PPL) or Structured Query Language (SQL) for executing queries, with access to over 200 SQL and PPL queries covering AWS log sources in Security Lake.

Additionally, users may generate on-demand indexed views for multiple queries on the same dataset to enhance the efficiency of security investigations. This functionality allows results to be ingested into an OpenSearch index, enabling low-latency subsequent queries and analysis using OpenSearch analytics features.

Zero-ETL integration with Amazon Security Lake is now accessible across multiple AWS Regions, including US East, US West, Asia Pacific, Europe, South America, and Canada. OpenSearch Service costs involve charges solely for the compute needed to query external data, alongside maintaining indexes in the service.