Collect Amazon CloudWatch Logs

Overview
USM Anywhere Architecture
USM Anywhere Data Security
USM Anywhere Log Data Enhancement
USM Anywhere Quick Start Guides
USM Anywhere Deployment Guide
- Overview
- USM Anywhere Deployment Types and Scalability
- USM Anywhere Sensor Deployments
  - Overview
  - AWS Sensor Deployment
    - Overview
    - Requirements for AWS Sensor Deployment
    - Deploy the AWS Sensor
    - Connect the AWS Sensor to USM Anywhere
    - Complete the AWS Sensor Setup
    - Enable Connections in an AWS VPC
    - VPC Traffic Mirroring with an AWS Sensor
    - Adding an Additional Sensor for Other AWS Accounts
    - AWS Log Discovery and Collection in USM Anywhere
  - Azure Sensor Deployment
  - GCP Sensor Deployment
  - Hyper-V Sensor Deployment
  - VMware Sensor Deployment
  - Deploying Your Sensor in a DHCP Environment
- The AWS Cloud Connector Deployment in USM Anywhere
- Network Setup and Configuration
- Data Sources and Log Collection
- File Integrity Monitoring
- Alarm and Event Notifications
- Troubleshooting and Remote Sensor Support
USM Anywhere User Guides
- Overview
- Getting Started with USM Anywhere
- USM Anywhere Best Practices
- USM Anywhere Dashboards
- Asset Management
- Alarms Management
- System Events Management
- Console User Events on USM Anywhere
- Configuration Issues Management
- User Behaviour Analytics
- Events Management
- USM Anywhere Scheduler
  - Overview
  - USM Anywhere Scheduler Best Practices
  - Managing Jobs in the Scheduler
  - Scheduling Active Directory Scans from the Job Scheduler Page
  - Scheduling Asset Scans from the Job Scheduler Page
  - Scheduling Asset Groups Scans from the Job Scheduler Page
  - Scheduling User Discovery Jobs from the Job Scheduler Page
  - Scheduling Log Collection from the Job Scheduler Page
    - Overview
    - AWS Log Discovery and Collection in USM Anywhere
    - Azure Log Discovery and Collection in USM Anywhere
- Rules Management
- Vulnerability Assessment
- Open Threat Exchange® and USM Anywhere
- USM Anywhere Sensor Management
- The AWS Cloud Connector in USM Anywhere
- Subscription Management
- USM Anywhere Reports
- Machine Learning
- USM Anywhere User Management
- Using USM Anywhere for PCI Compliance
- USM Anywhere Investigations
- System Status within USM Anywhere
USM Anywhere Agents Guide
USM Anywhere BlueApps Guide

Amazon CloudWatch Logs monitors applications and systems using log data, aggregating and storing application logs. CloudWatch Logs is useful because you can easily configure it to process additional metadata with the log files. Visit the AWS documentation to learn more about VPC flow log collection.

Important: If you choose to enable CloudWatch Logs in your Amazon Web Services (AWS) environment, you should make sure that you are not collecting more data than you need because this service incurs AWS costs based upon usage. See the CloudWatch pricing information to plan and configure your usage.

If not already done, install and configure the Amazon CloudWatch agent to collect logs from Amazon Elastic Compute Cloud (EC2) instances. See Amazon documentation for instructions. USM Anywhere provides some CloudWatch log collection jobs out of the box, but they are disabled by default. You can enable them under Settings > Scheduler. When enabled, these jobs monitor certain log groups and collect logs from CloudWatch every five minutes. You must configure your CloudWatch agent to use these log group names and to keep the log types the same within a given log group. USM Anywhere Log Collection Jobs and CloudWatch Log Groups

USM Anywhere Log Collection Job Name	CloudWatch Log Group Name	Default File Path	Date Format
CloudWatch - Apache-Access-Logs	Apache-Access-Logs	/var/log/apache2/access.log	%d/%b/%Y:%H:%M:%S
CloudWatch - Linux-Audit-Logs	Linux-Audit-Logs	/var/log/audit/audit.log	Use the default
CloudWatch - Linux-Auth-Logs	Linux-Auth-Logs	/var/log/auth.log	%b %d %H:%M:%S
CloudWatch - Osquery-Logs	OSQuery-Logs	/var/log/osquery/osqueryd.results.log	Use the default

If you want to collect logs from other log groups, ensure that all streams in the same group are of the same type so that USM Anywhere can use a designated data source to parse the collected raw log data. You can then set up a CloudWatch log collection job for each log group. To create a new CloudWatch log collection job

Go to Settings > Scheduler.
In the left navigation menu, click Log Collection.
Note: You can use the Sensor filter at the top of the list to review the available log collection jobs on your AWS Sensor.
Click Create Log Collection Job.

Note: If you have recently deployed a new USM Anywhere Sensor, it can take up to 20 minutes for USM Anywhere to discover the various log sources. After it discovers the logs, you must manually enable the AWS log collection jobs you want before the system collects the log data.
The Schedule New Job dialog box opens.
Enter the name and description for the job. The description is optional, but it is a best practice to provide this information so that others can easily understand what it does.
Select Sensor as the source for your new job.
In the Action Type drop-down list, select Amazon Web Services.
In the App Action drop-down list, select Monitor CloudWatch.
Enter the Region Name, Group Name, and Stream Name information for your AWS account. Region name can be an asterisk ( * ) to monitor all regions for a given group.
In Source Format, select either of the following log formats:
- Syslog: All messages transmitted to USM Anywhere are processed with the assumption that they are syslog formatted. When you choose syslog as the source format, the data source selection is bypassed and USM Anywhere uses the auto-detect hints from the data sources to match the incoming messages to the correct data source.
- Raw: Use for non-syslog formatted data. If you select this option, you must choose the data source that USM Anywhere will use to parse all of the streams in the group. For example, to collect Amazon Virtual Private Cloud (VPC) flow logs, select the VPC Flow Logs data source.
  
  Important: If a group contains streams of mixed log formats, USM Anywhere parses all of them with the data source that you chose, which produces undesired results. In this case, you need to configure CloudWatch to separate the streams into different groups so that each contains only a single log type that can be mapped to the correct data source.
In the Schedule section, specify when USM Anywhere runs the job: a. Select the increment as Minute, Hour, Day, Week, Month, or Year.
Warning: After a frequency change, monitor the system to check its performance. For example, you can check the system load and CPU. See USM Anywhere System Monitor for more information.
b. Set the interval options for the increment. The selected increment determines the available options. For example, on a weekly increment, you can select the days of the week to run the job.
Or on a monthly increment, you can specify a date or a day of the week that occurs within the month.

Important: USM Anywhere restarts the schedule on the first day of the month if the option “Every x days” is selected.
c. Set the start time. This is the time that the job starts at the specified interval. It uses the time zone configured for your USM Anywhere instance (the default is Coordinated Universal Time [UTC]).
Click Save. USM Anywhere detects any enabled jobs with the same configuration and asks you to confirm before continuing. This is because having two jobs with the same configuration generates duplicate events and alarms.

The following video demonstrates how to configure AWS to capture VPC flow logs and how to configure USM Anywhere to retrieve this information to create events:

USM Anywhere™

USM Central™

Automated Policy Manager

Collect Amazon CloudWatch Logs