Configuring and troubleshooting Linux Syslog in Azure Monitor [Part1]

Syslog is an event logging protocol that is common in Linux. Applications will send messages that may be stored on the local machine or delivered to a Syslog collector.

When the Log Analytics agent for Linux is installed, it configures the local Syslog daemon to forward messages to the agent. The agent then sends the message to Azure Monitor where a corresponding record is created.

The Scenario:

We will be configuring the agent to be a central location for syslog messages and we will configure two VMs to send syslog events to it. one of the machines will be in the same subnet and it will send using an internal ip address, and the other one will be a remote VM and it will send publicly using the public IP Address.

Requirements:

1- Log analytics workspace.
2- Linux VM to be configured as central syslog destination (I've used Redhat 7.4 from Azure marketplace) but feel free to use any supported distro from the here
3- Linux VM that will fowrward syslog messeges to the oms agent (in this blog am also using another VM running redhat 7.4

The Diagram:

In part1, we will focus creating and configuring CSSRedhat01 as a central location for syslog messeges, and CSSRedhat02 to forward syslog messages over UDP to the OMS Agent.

Install the OMS Agent for Linux:

I've already created a workspace and we will connect CSSRedhat01 to the workspace to install the OMSAgentForLinux Extension, to do that:

For azure vms:

1- Open the workspace and go to Virtual machines view

2- Click on the machine and click connect.

after a few minutes the machine should show as connected in the virtual machines view.

For non-azure vms, use the command specified in your workspace or the below

https://github.com/Microsoft/OMS-Agent-for-Linux/blob/master/docs/OMS-Agent-for-Linux.md#steps-to-install-the-oms-agent-for-linux

Enable Syslog Collection in your Workspace:

We will need to enable data collection for the facilities that your syslog is sending to, as an example CSSRedhat02 will send data to the agent on CSSRedhat01 on local5 facility, therefor; data collection from local5 on the workspace should be added.

Configure CSSRedhat02 to forward syslog data to the omsagent:

To do that, we need to edit /etc/rsyslog.conf and configure it to forward syslog data to the agent on CSSRedhat01

To do that:

1- Edit /etc/rsyslog.conf and add the below line (Change the remote host to your DNS Name or IP Address of your OMSAgent server).

*.* @cssredhat01:514

Please note the following:

. --> this will collect all type of data and the severity.
@ --> the @ sign configures forwarding data to use UDP. if you are using tcp this should be @@ sign.

Also, I already have DNS name configured pointing to the local IP address for the agent so I added the DNS name.

The file should look as below:

2- Restart rsyslog service

systemctl restart rsyslog

Configure CSSRedhat01 to accept syslog data coming from outside on 514.

On the agent machine, we will need to configure rsyslog to accept rsyslog data from outside

To do that:

1- Edit /etc/rsyslog.conf and uncomment the below two lines

$ModLoad imudp
$UDPServerRun 514

2- Restart rsyslog

systemctl restart rsyslog

this would be all the initial configurations required, let us test that by checking if we have data in Log analytics ?

So far we have no data in log analytics :), so this will lead us to Part 2 where we are going to trace the data flow and start troubleshooting.

Configuring and troubleshooting Linux Syslog in Azure Monitor [Part2]

if you have any questions, please feel free to drop me a comment below.