AWS Logging with Elastic Search and Kibana with C#

ELK is very famous for logging functionality. Here is the implementation of ELK (Elastic search + Logstash + Kibana) with .Net framework. I have used serilog with Elastic search instead of logstash, which is widely used in .Net.

Step 1: Create Elastic Search Domain. Follow bellow link.

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg-create-domain.html

Step 2: Open Visual Studio and create new console application in .Net Core.

Step 3: Open NuGet management console and install below packages:

    - Serilog.Sinks.Elasticsearch
    - Elasticsearch.Net.aws
    - Elasticsearch.Net

Step 4: In StartUp() register serilog that sink with Elastic serach.

Log.Logger= new LoggerConfiguration()
               .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("YourElasticEndPoint")))
               {
                   ModifyConnectionSettings = conn =>
                   {
                       var httpConnection = new AwsHttpConnection("YourAWSRegion");
                       var pool = new SingleNodeConnectionPool(new Uri("YourElasticEndPoint"));
                       var conf = new ConnectionConfiguration(pool, httpConnection);
                       return conf;
                   },
                   IndexFormat="YourIndexName-{0:MM-yyyy}"
               })
            .CreateLogger();

You can specify AWS Credentials in httpConnection object.

Step 5: Start using Serilogs. Log your events as below.

Log.Information("Event Occured!");

Log.Error(ex,"Error Occured");//ex is exception object

You can log complete object as well.

var auditlog = new AuditLog()
               {
                   FirstName = "Priyanka",
                   LastName = "Shah",             
                   FieldName = "AppintmentConfirmationWriteBackStatus",
                   PreviousValue = "true",
                   NewValue = "false",
                   LogDate = DateTime.Now
               };
Log.Information("{@auditlog}", auditlog);
Step 6: Check your logs in Kibana. Open Kibana dashboard. Create your Index as below.

Step 7: Click on Discover, apply your filters and see the logs.