AWS Logging with Elastic Search and Kibana with C#

ELK is very famous for logging functionality. Here is the implementation of ELK (Elastic search + Logstash + Kibana) with .Net framework. I have used serilog with Elastic search instead of logstash, which is widely used in .Net.

Step 1: Create Elastic Search Domain. Follow bellow link.

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg-create-domain.html

Step 2: Open Visual Studio and create new console application in .Net Core.

Step 3: Open NuGet management console and install below packages:

    - Serilog.Sinks.Elasticsearch
    - Elasticsearch.Net.aws
    - Elasticsearch.Net

Step 4: In StartUp() register serilog that sink with Elastic serach.

Log.Logger= new LoggerConfiguration()
               .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("YourElasticEndPoint")))
               {
                   ModifyConnectionSettings = conn =>
                   {
                       var httpConnection = new AwsHttpConnection("YourAWSRegion");
                       var pool = new SingleNodeConnectionPool(new Uri("YourElasticEndPoint"));
                       var conf = new ConnectionConfiguration(pool, httpConnection);
                       return conf;
                   },
                   IndexFormat="YourIndexName-{0:MM-yyyy}"
               })
            .CreateLogger();

You can specify AWS Credentials in httpConnection object.

Step 5: Start using Serilogs. Log your events as below.

Log.Information("Event Occured!");

Log.Error(ex,"Error Occured");//ex is exception object

You can log complete object as well.

var auditlog = new AuditLog()
               {
                   FirstName = "Priyanka",
                   LastName = "Shah",             
                   FieldName = "AppintmentConfirmationWriteBackStatus",
                   PreviousValue = "true",
                   NewValue = "false",
                   LogDate = DateTime.Now
               };
Log.Information("{@auditlog}", auditlog);
Step 6: Check your logs in Kibana. Open Kibana dashboard. Create your Index as below.

Step 7: Click on Discover, apply your filters and see the logs.

AWS Lambda Prewarm

Lambdas can stay in RAM for 15mins. After that the request will be served from cold start stage. To avoid this latency, we need to keep lambdas alive. This concept is called as “Prewarming”. Certain containers will always ready to serve the request.

I have implemented Prewarming through cloud watch events. To do so, follow the steps as below.

Step 1: Create a rule in cloud watch. Set Schedule which will trigger lambda every 15mins.

Step 2: Add target to lambda function. Give the name of your lambda function “Pwa-prewarmmer-stage”. This lambda function takes two input parameters: number of containers to be created and whether to prewarm stage or production’s lambda.

This prewarming of lambda will invoke all other lambda’s and create containers for the same. This can be cross checked in cloud watch logs as below.

Code of Prewarm Lambda

[LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]
        public async Task<string> FunctionHandlerAsync(JObject input,
            ILambdaContext lambdaContext)
        {
            int num=Convert.ToInt16( input["NumberofContainer"]);
            bool isProd=Convert.ToBoolean( input["isProd"]);
            try
            {
                Console.WriteLine("Prewarmer Start");
                for (int i = 1; i <= num; i++)
                {
                    using (AmazonLambdaClient client = new AmazonLambdaClient())
                    {
                        JObject ob = new JObject { { "Resource", "WarmingLambda" }, { "Body", i.ToString() } };
                        var request = new InvokeRequest
                        {
                            FunctionName = isProd? "Pwa-lead" : "Pwa-lead-stage",
                            InvocationType = InvocationType.Event,
                            Payload = ob.ToString()
                        };
                        var response = await client.InvokeAsync(request);
                       
                    }
                }
                return "Lead prewarmer done";
            }
            catch (Exception ex)
            {
                return ex.Message + ex.StackTrace;
            }
        }
    }

Publish Lambda from Visual Studio

Here are the steps to publish .Net project as lambda in AWS from visual studio.

Step 1: Edit .csproj file and add below line of code.

<PropertyGroup>

    <AWSProjectType>Lambda</AWSProjectType>

  </PropertyGroup>

Step 2: Add Lambda function handler that will be init for API Gateway request.

public class LambdaFunction : APIGatewayProxyFunction

    {

        protected override void Init(IWebHostBuilder builder)

        {

            builder.UseContentRoot(Directory.GetCurrentDirectory())

                   .UseStartup<Startup>()

                   .UseLambdaServer();

        }

    }

Step 3: Add aws-lambda-tools-defaults.json file that will be used to store the publish settings.

Step 4: Finally, Publish Lambda to AWS from Visual Studio, by right click project and “Publish to AWS Lambda”. A wizard will open as shown below.

Settings to be configured in the above wizard

· Define environment variables (Staging or Production). Based on this variable, appsettings file will be executed.
·   Give PWA_Lambda_access IAM role to lambda function.
·   For .Net application Minimum RAM size is required 256MB.
·   All the lambda must be in VPC with proper security group to communicate with other assets.
·   Lambda stays warm for 15mins which are deployed in VPC. The idle lambda will be disposed after 15 mins and new request will be served from cold start.

With this Lambdas will be published on AWS.

AWS API Gateway configuration

Here are the steps to configure API Gateway for integrated website.

Step 1: Create Rest API shown as below.

Step 2: Create resource as “proxy’.

Step 3: Create method as “Any” type

The method type is “Any”, so all the functions behind the Lambda either get or post will work within API. The invocation of the lambda function is dynamic. Define stage variable in place of lambda function name.

On click of Save, AWS will ask to add permission to Lambda function that can be executable by API gateway. Replace the value of ${stageVariables.functionname} to the lambda function name. Execute below command to CLI.

aws lambda add-permission  --function-name "arn:aws:lambda:ap-south-1:518955882229:function:${stageVariables.functionname}"  --source-arn "arn:aws:execute-api:ap-south-1:518955882229:u5ismynmx3/*/*/*"  --principal apigateway.amazonaws.com  --statement-id c7210776-beb6-4aad-bcfa-fdb20972f52f  --action lambda:InvokeFunction

Step 4: Deploy API

Create stages of API gateway like Stage and Prod.

 

Step 5: Assign lambda function to variable

Select the created stage. In stage variables, add new stage variable with name “functionname” and value “your lambda function”.

Step 6: Create custom domain for your API. (To be done by cloud team)

Step 7: Configure cloud watch Logs and alarms. Add newly created API to cloud watch dashboard (To be done by cloud team).

Connect Remote Active Directory using C# code

Sometimes it is required to get users information from remote server’s Active Directory. You may need to fetch some Groups of Active Directory. Here are sample codes which can help you.

Get all the users from AD specific group and fetch their information using Group Principal

   private static void GroupSearch()

        {

            Console.WriteLine("Start Group Search");

            try

            {

                PrincipalContext principalctx = new PrincipalContext(ContextType.Domain, YourServer/YourDomain, YourUserName, YourPassword);

                GroupPrincipal groupprinciple = GroupPrincipal.FindByIdentity(principalctx, IdentityType.Name, GroupName);

                if (groupprinciple != null)

                {

                    foreach (Principal innerprincipal in groupprinciple.GetMembers(false))

                    {

                        try

                        {

                            Console.WriteLine(((System.DirectoryServices.AccountManagement.UserPrincipal)(innerprincipal)).GivenName);

                            Console.WriteLine(((System.DirectoryServices.AccountManagement.UserPrincipal)(innerprincipal)).Surname);

                            Console.WriteLine(innerprincipal.SamAccountName);

                            Console.WriteLine(((System.DirectoryServices.AccountManagement.UserPrincipal)(innerprincipal)).EmailAddress);

                            Console.WriteLine(((System.DirectoryServices.AccountManagement.AuthenticablePrincipal)(innerprincipal)).Enabled ?? false);

                        }

                        catch (Exception exe)

                        {

                        }

                    }

                    groupprinciple.Dispose();

                    principalctx.Dispose();

                }

            }

            catch (Exception ex)

            {

            

            }

            Console.WriteLine("End of Group Search");

        }

Achieve same using Directory Searcher object with LDAP command

  private static void LDAP()

        {

            try

            {

                string DomainPath = "LDAP://YourIP/dc=yourDomain,dc=yourDomainExtension";

               

                DirectoryEntry searchRoot = new DirectoryEntry(DomainPath, "yourUserName", "yourPassword",AuthenticationTypes.Secure);

               

                DirectorySearcher search = new DirectorySearcher(searchRoot);

                search.Filter = ((memberOf=CN=YourGroupName,OU=YourDirectoryHierarchy, dc=yourDomain,dc=yourDomainExtension))";

                search.PropertiesToLoad.Add("samaccountname");

                search.PropertiesToLoad.Add("mail");

                search.PropertiesToLoad.Add("userAccountControl");

                search.PropertiesToLoad.Add("displayname");//first name

                SearchResult result;

                SearchResultCollection resultCol = search.FindAll();

                if (resultCol != null)

                {

                    for (int counter = 0; counter < resultCol.Count; counter++)

                    {

                        string UserNameEmailString = string.Empty;

                        result = resultCol[counter];

                        if (result.Properties.Contains("displayname"))

                        {

                            Console.WriteLine((String)result.Properties["displayname"][0]);

                        }

                        if (result.Properties.Contains("samaccountname"))

                        {

                            Console.WriteLine((String)result.Properties["samaccountname"][0]);

                        }

                        if (result.Properties.Contains("mail"))

                        {

                            Console.WriteLine((String)result.Properties["mail"][0]);

                        }

                        if (result.Properties.Contains("userAccountControl"))

                        {

                            //Property check status of user

                        }

                       

                    }

                }

            }

            catch (Exception ex)

            {

          

            }

            Console.WriteLine("END Start LDAP");

        }

If you want to search all the users from root AD change bellow code from above lines:

search.Filter = "(&(objectClass=user)(objectCategory=person))";

If you want to get one user information change bellow code from above lines:

search.Filter = "(&(objectClass=user)(objectCategory=person) (sAMAccountName=yourUserName))";

Hope this sample codes help you!

Authenticate WebAPI

You can authenticate your web API in simple two steps.

Step 1 – Create HTTPModule to authenticate request and register

Create file with below code and register this module to web.config

using System;

using System.Collections.Generic;

using System.Linq;

using System.Net.Http.Headers;

using System.Security.Principal;

using System.Text;

using System.Threading;

using System.Web;

using System.Web.Mvc;

using System.Web.SessionState;

namespace ProjectNamespace

{

    public class BasicAuthHttpModule : IHttpModule

    {

        private const string Realm = "Message to display";

        public void Init(HttpApplication context)

        {

            // Register event handlers

            context.AuthenticateRequest += OnApplicationAuthenticateRequest;

            context.EndRequest += OnApplicationEndRequest;

            _trackTraceUserService = new TrackAndTraceUserService();

        }

        private static void SetPrincipal(IPrincipal principal)

        {

            Thread.CurrentPrincipal = principal;

            if (HttpContext.Current != null)

            {

                HttpContext.Current.User = principal;

            }

        }

        // TODO: Here is where you would validate the username and password.

        private static bool CheckPassword(string username, string password)

        {

//Write your code to authenticate user

        }

        private static bool AuthenticateUser(string credentials)

        {

            bool validated = false;

            try

            {

                var encoding = Encoding.GetEncoding("iso-8859-1");

                credentials = encoding.GetString(Convert.FromBase64String(credentials));

                int separator = credentials.IndexOf(':');

                string name = credentials.Substring(0, separator);

                string password = credentials.Substring(separator + 1);

                validated = CheckPassword(name, password);

                if (validated)

                {

                    var identity = new GenericIdentity(name);

                    SetPrincipal(new GenericPrincipal(identity, null));

                }

            }

            catch (FormatException)

            {

                // Credentials were not formatted correctly.

                validated = false;

            }

            return validated;

        }

        private static void OnApplicationAuthenticateRequest(object sender, EventArgs e)

        {

            var request = HttpContext.Current.Request;

            var authHeader = request.Headers["Authorization"];

            if (authHeader != null)

            {

                var authHeaderVal = AuthenticationHeaderValue.Parse(authHeader);

                // RFC 2617 sec 1.2, "scheme" name is case-insensitive

                if (authHeaderVal.Scheme.Equals("basic",

                        StringComparison.OrdinalIgnoreCase) &&

                    authHeaderVal.Parameter != null)

                {

                    AuthenticateUser(authHeaderVal.Parameter);

                }

            }

        }

        // If the request was unauthorized, add the WWW-Authenticate header

        // to the response.

        private static void OnApplicationEndRequest(object sender, EventArgs e)

        {

            var response = HttpContext.Current.Response;

            if (response.StatusCode == 401)

            {

                response.Headers.Add("WWW-Authenticate",

                    string.Format("Basic realm=\"{0}\"", Realm));

            }

        }

        public void Dispose()

        {

        }

    }

}

Add bellow line in web.config

<system.webServer>

  <modules>

      <add name="BasicAuthHttpModule" type="ProjectNamespace.BasicAuthHttpModule, ProjectName" />

    </modules>

</system.webServer>

Step 2 – Create Web API

public class MyController : ApiController

    {

    

        [Authorize]

        public List<MyObject> Get([FromUri] MyClass data)

        {

}

Hope this helps you!!