How to Send Python Logs to OpenSearch Using Fluent Bit

Introduction

Effective log management is essential for monitoring and maintaining applications in today’s data-driven world. OpenSearch, an open-source search and analytics engine, provides robust log aggregation and analysis capabilities. By combining it with Fluent Bit, a lightweight log forwarder, we can efficiently send Python logs to OpenSearch for real-time monitoring and analysis. This guide will walk you through setting up this powerful combination.

This setup benefits developers and system administrators who need to monitor Python applications in real-time. By sending logs from Python applications to OpenSearch, you can analyze logs, create dashboards, set alerts, and gain valuable insights into your application’s performance and behavior. This approach is scalable and suitable for both small projects and large-scale production environments.

Before you begin, make sure you have the following prerequisites in place:

1. Python Installed: Ensure Python is installed on your server or local machine.
2. OpenSearch Cluster: You should have access to an OpenSearch cluster. You can either create a new cluster or use an existing one.
3. Fluent Bit Installed: Fluent Bit should be installed on the server or machine where your Python application runs.

First, update your repository and install Python. Run the following commands:

sudo apt update sudo apt install python3

To verify the Python installation, run:

python3 –version

Configure your Python application to log messages to a file. Here’s a basic setup:

import logging logging.basicConfig( filename=’app.log’, level=logging.INFO, format=’%(asctime)s – %(levelname)s – %(message)s’ ) logging.info(‘This is a test log message.’)

This configuration writes log messages to app.log with a timestamp, log level, and message format.

If you don’t have an OpenSearch cluster, you can create one using DigitalOcean’s command line tool, doctl. Run the following command:

doctl databases create opensearch-doks –engine opensearch –region your-region –size db-s-1vcpu-2gb –num-nodes 1

Replace your-region with your desired region. Alternatively, you can use the DigitalOcean control panel to create a cluster manually. For details on how to do that, refer to this guide on create OpenSearch clusters through control panel.

To install Fluent Bit, use the following command:

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

Create a configuration file named fluent-bit.conf to define how Fluent Bit should read and forward logs to OpenSearch. Here’s a sample configuration:

[SERVICE] Flush 1 Daemon Off Log_Level info [INPUT] Name tail Path /path/to/your/log/file.log Tag python_logs Refresh_Interval 5 [OUTPUT] Name opensearch Match python_logs Host your-opensearch-host Port 25060 HTTP_User your-username HTTP_Passwd your-password Index your-index-name tls On Suppress_Type_Name On

• Path: Specify the path to your app.log.
• Host: Replace with your OpenSearch host.
• HTTP_User: Your OpenSearch username.
• HTTP_Passwd: Your OpenSearch password.
• Index: The name of the index where logs will be stored.

Run Fluent Bit with the configuration file:

/opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf

To ensure logs are being ingested correctly, check your OpenSearch dashboard. You should see the logs being forwarded from Fluent Bit.

1. Log in to OpenSearch Dashboards with your credentials.
2. Expand the left-side menu and click on Index Management under Management.
3. Click on Create index from the Indexes menu.
4. Enter the index name, configure other settings as needed, and click on Create.

By following these steps, you’ve set up a system to send Python logs to OpenSearch using Fluent Bit. This setup enables you to efficiently manage and analyze logs, helping you maintain and monitor your applications effectively. With OpenSearch and Fluent Bit, you have a powerful solution for real-time log analysis and monitoring, tailored to meet the needs of any Python-based application.