Logstash output to Azure blobstorage

Question

I am currently importing IIS-logs into Logstash using Filebeat, and the Logstash is configured to output the documents into Elasticsearch. Now I also want to output my IIS logs to Azure storage (blob) for longtime-backup purposes, but I cannot find a way to do it.

There is a working Logstash plugin for output to AWS. But I need to use Azure blob storage. I cannot find an output plugin for Azure Blobstorage in the elastic output list, or anywhere else..

Is there a way to output documents from Logstash to Azure blobstorage?

Seems like a valuable feature. I guess it would look like something below.

output {
    Azure_storage {
        account => "test"
        key => "SuperSecret"
        container => "Backup_Documents"
        blobName => "nameofblob"
    }
  }

score 1 · Answer 1 · answered Jul 26 '20 at 16:12

There is an open-source plugin: https://github.com/tuffk/Logstash-output-to-Azure-Blob

Basic configuration:

  output {
       azure {
         storage_account_name => "my-azure-account"    # required
         storage_access_key => "my-super-secret-key"   # required
         container_name => "my-container"              # required
         size_file => 1024*1024*5                      # optional
         time_file => 10                               # optional
         restore => true                               # optional
         temporary_directory => "path/to/directory"    # optional
         prefix => "a_prefix"                          # optional
         upload_queue_size => 2                        # optional
         upload_workers_count => 1                     # optional
         rotation_strategy_val => "size_and_time"      # optional
         tags => []                                    # optional
         encoding => "none"                            # optional
       }
     }

To make the plugin available in your Logstash environment, run the following command:

bin/logstash-plugin install logstash-output-azure

This plugin is unfortunately no longer maintained. – BobTuckerman Oct 15 '20 at 00:03 — BobTuckerman, Oct 15 '20 at 00:03

score 1 · Accepted Answer · answered Jun 12 '17 at 17:12

There isn't one in the official plugins, nor in the community-maintained list of plugins. Someone probably hasn't written one yet. Or if they have, they aren't sharing. This may be a case where you output to a local directory and use a scheduled task to sync the directory to blob-storage. We're using this method for one of our stranger workflows.

Logstash output to Azure blobstorage

2 Answers2