Pyspark Script to Read Logs From Sumo Logic

Note:

• This tutorial requires access to Oracle Cloud. To sign upwards for a free account, come across Get started with Oracle Deject Infrastructure Free Tier.
• It uses instance values for Oracle Cloud Infrastructure credentials, tenancy, and compartments. When completing your lab, substitute these values with ones specific to your deject environment.

Move logs from Oracle Cloud Infrastructure into Sumo Logic

Introduction

The Oracle Cloud Observability and Manageability platform aims to meet our customers where they are. We understand that they have standardized their operational postures with popular third-party observability tools and we want to be interoperable with those tools so our customers tin proceed using the tools they have invested in with Oracle Cloud Infrastructure (OCI).

In this tutorial, we will walk you through how you can motility logs from OCI into Sumo Logic. Sumo Logic is a popular observability tool that provides monitoring and security services that provide full visibility into your applications.

Our solution architecture at a high level is every bit shown below:

Create a Custom HTTP Source Collector in Sumo Logic

In your Sumo Logic business relationship, you need to create an HTTP custom collector app equally described in the steps below.

1. Click Setup Magician.

   

2. Click Starting time streaming data to Sumo Logic.

   

3. Click Your Custom App.

   

4. Click HTTPS Source.

   

5. Configure your HTTP source as shown below.

   Annotation: The HTTP source is a metadata tag, stored with your ingested logs, and is useful when searching and filtering logs later in Sumo Logic. Each log line we are going ingest will start with a timestamp of its event occurrence so select the option Use time zone from log file.

   

   Equally you move to the next screen, we go the HTTPS endpoint for our logs to upload from OCI, using a Post HTTP call. Please take a annotation of this endpoint every bit we will configure our function to use this endpoint to upload logs to Sumo Logic.

   

Configure the Logs You lot Want to Capture

You can prepare any logs as input for Service Connector Hub and hence ingest into Sumo Logic. For simplifying this tutorial, we will capture Oracle Cloud Infrastructure (OCI) generated logs for write-events to an arbitrary bucket of your choice.

1. In the Oracle Deject Panel, click the navigation menu, select Logging, and so select Log Groups.

   

2. To create a log group, click Create Log Group.

   

3. Select your compartment, add together LogGroupForBucketActivity for the name and add a clarification. Click Create.

   

4. Select Logs from the Logging bill of fare. You will run across a screen like to beneath.

   

5. Click Enable service log and enter the following information:

   • Service: Select Object Storage
   • Resource: Choose an arbitrary saucepan(for case, BucketForSumoLogic) that you lot would similar observed with the logs.
   • Log Category: Select Write Access Events
   • Log Proper name: Enter a name for your log, for case, logForBucketActivity.
   • Log Group: Select the LogGroupForBucketActivity log group for the log that you but created in the previous step

6. Click Enable Log.

   

   At present every time a object is uploaded to the BucketForSumoLogic bucket,a log entry volition exist added to the logForBucketActivity log.

Configure Oracle Functions for Ingesting Logs into Sumo Logic

1. In the Oracle Deject Console, click the navigation menu and select Solution and Platform. Select Functions under the Developer Services menu.

2. Click Create Application and enter a name, for example, SumoLogicFnApp.

   

3. Once you create the application, click your application name and select Getting Started from the Resource carte.

   

4. Launch Cloud Shell.

5. Use the context for your region.

                 fn list context fn use context us-ashburn-i                          

6. Update the context with the function's compartment ID.

                 fn update context oracle.compartment-id <compartment-id>                          

7. Update the context with the location of the registry you lot want to use.

                 fn update context registry iad.ocir.io/<tenancy_name>/[YOUR-OCIR-REPO]                          

   Replace iad with the 3-digit region lawmaking for your region.

8. Assuming you have created the Auth Token already, log in to the registry using the Auth Token as your password.

                 docker login iad.ocir.io                          

   Replace iad with the three-digit region lawmaking for your region.

   Y'all are prompted for the following information:

   • Username: <tenancyname>/<username>
   • Password: Create a password

   Notation: If yous are using Oracle Identity Cloud Service, your username is <tenancyname>/oracleidentitycloudservice/<username>.

   Verify your setup by listing applications in the compartment.

9. Generate a 'how-do-you-do-globe' boilerplate function.

                 fn init --runtime python sumologicfn                          

   The fn init control will generate a folder chosen SumoLogicfn with 3 files inside: func.py, func.yaml, and requirements.txt.

   Open up func.py and supersede the content of the file with the post-obit lawmaking.

   1. Import the necessary Python modules, as shown in the following snippet.

                        import io import json import logging import os  import requests from fdk import response                                  

   2. Ascertain a office to parse the log information and invoke the Sumo Logic API to ingest the logs.

                        # This method is the entrypoint for your Function invokation  # aka the method invoked past the OCI Fn platform # it will receive the listing of log entries from OCI every bit input in the form of bytestream # the method proper noun volition exist defined in func.yml def handler(ctx, data: io.BytesIO = None):     logger = logging.getLogger()     logger.info("function outset")      # Sumologic endpoint URL to upload OCI logs to HTTP custom app.     # this value volition be defined divers in func.yaml     sumologic_endpoint = os.environ['SUMOLOGIC_ENDPOINT']                                  

      For information near the format of the logs generated by the Oracle Cloud Infrastructure Logging service, encounter Logging Format Overview.

   3. Recall the log entries from the Service Connector Hub received past our sumologicfn part as its invocation payload. Loop through these log-entries and log-lines one by one.

                        effort:     logentries = json.loads(data.getvalue()) # deserialize the bytesstream input equally JSON array     if not isinstance(logentries, listing):         logger.error('Invalid connector payload. No log queries detected')         raise      # Optional...log the input to the part equally human readble JSON.      # Non to exist used in product     logger.info("json input from SCH")     logger.info(information.getvalue())       for logEntry in logentries:          logger.info("Extracting/Parse log details from the log entry json")         event_name = logEntry["data"]["requestResourcePath"] + '\t'         time_of_event = logEntry["time"] + '\t'         cmpt_name = logEntry["data"]["compartmentName"] + '\t'         bucket_namespace = logEntry["information"]["namespaceName"] + '\t'         bucket_name = logEntry["information"]["bucketName"] + '\t'         request_action = logEntry["data"]["requestAction"]          log_line = time_of_event + event_name + cmpt_name + \                     bucket_namespace + bucket_name + request_action          # Phone call the Sumologic with the payload and ingest the OCI logs         headers = {'Content-type': 'text/plain'}         response_from_sumologic = requests.mail service(sumologic_endpoint,                                                 information=log_line,                                                 headers=headers)         logging.getLogger().info(response_from_sumologic.text)      logger.info("function finish")     return  except Exception as east:      logger.error("Failure in the function: {}".format(str(e)))      raise                                  

10. Replace func.yml contents as follows. Brand sure you put the value for your SumoLogic_ENDPOINT that nosotros got in the previous step.

                                      schema_version                  :                  20180708                  proper noun                  :                  sumologicfn                  version                  :                  0.0.i                  runtime                  :                  python                  entrypoint                  :                  /python/bin/fdk /function/func.py handler                  retention                  :                  1024                  timeout                  :                  120                  config                  :                  SUMOLOGIC_ENDPOINT                  :                  [                  YOUR SUMOLOGIC API ENDPOINT URL HERE                  ]                              

11. Replace requirements.txt contents as follows.

12. Deploy your function.

                    fn -v deploy --app sumologicFnApp --no-crash-land                              

13. Optionally, you can test your SumoLogicfn part with instance input as follows:

                  curl -O https://raw.githubusercontent.com/mayur-oci/sumologicfn/primary/example.json fn invoke sumologicFnApp sumologicfn < instance.json                          

Create a Service Connector for Reading Logs from Logging and Triggering the Role

1. In the Oracle Deject Console, click the navigation bill of fare, and select Solution and Platform. Select Service Connectors under the Logging menu.

2. Click Create Connector, and from the Source drop-down list, select Logging and from the Functions drop-down list, select Target.

3. On Configure Source Connection, select your compartment name, your LogGroupForBucketActivity log grouping, and your logForBucketActivity logs.

4. If yous want to use inspect logs, click +Another log, choose your compartment and add _Audit for Log Group.

   

5. If prompted to create a policy for writing to Functions, click Create.

The Service Connector is now set and will trigger the part to ingest logs into Sumo Logic every time it finds logs in the Logging service.

Visualize Oracle Cloud Infrastructure Logs in Sumo Logic

1. In Sumo Logic, select the Source - Custom App menu to run across logs ingested from Oracle Cloud Infrastructure (OCI) using our SumoLogicfn part.

   

Troubleshoot

This section shows how y'all can use a simple email alert to monitor the condition of your solution.

For more details, see Overview of Functions.

Create a Topic and a Subscription for the Notification Service

1. In the Oracle Cloud Console, from the navigation menu in the upper-left corner, select Application Integration, and then select Notifications.

2. Click Create Topic and create a topic with the name my_function_status.

3. Cull your topic, click Create Subscription and use the following example:

   • Protocol: Email and add create a subscription with your electronic mail.

4. The subscription will be created in "Pending" condition. You will receive a confirmation email and will need to click the link in the electronic mail to confirm your electronic mail accost.

Cheque Metrics and Create an Alarm Definition from Metrics

1. From the navigation card in the upper-left corner, select Developer Services, and then select Functions.

2. Choose the application and the function that you want to monitor.

3. From the Metrics page, go to the Functions Errors chart, click Options, and then click Create an Warning on this Query.

4. Add a proper noun and under Notification, select Destination service as the notification service, select your_compartment, and and then select Topic equally my_function_status.

Monitor the Status Service Connector Hub

This section shows how yous can utilise a simple email alert to monitor the status of your Service Connector Hub (SCH).

For more details, refer to Service Connector Hub Overview.

Create a Topic and a Subscription for the Notification Service

1. From the navigation carte du jour in the upper-left corner, select Application Integration, and so select Notifications.

2. Click Create Topic and create a topic with my_sch_status name.

3. Choose your topic, click Create Subscription and utilise the post-obit instance:

   • Protocol: Email and add create a subscription with your email

4. The subscription volition be created in "Pending" status. You volition receive a confirmation email and volition need to click the link in the electronic mail to ostend your electronic mail address.

Check Metrics and Create an Alarm Definition from Metrics

1. From the navigation menu in the upper-left corner, select Logging, and and then select Service Connectors.

2. Choose the connector that you want to monitor and from the Resources list in the left navigation panel, select Metrics.

3. From the metrics chart that you want to add the alarm to, for example, "Service Connector Hub Errors", click Options and Create an Alarm on this Query.

4. Add together a name and under Notification, select Destination service every bit the notification service, select your_compartment, and so select Topic equally my_sch_status.

Conclusion

This tutorial showed how Oracle Cloud Infrastructure and Sumo Logic customers tin can configure a highly scalable solution with low overhead for moving logs from Oracle Cloud Infrastructure Logging to Sumo Logic using Service Connector Hub and Oracle Functions.

Acknowledgements

• Writer - Mayur Raleraskar, Solutions Architect

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to go an Oracle Learning Explorer.

For product documentation, visit Oracle Help Eye.

---

Motility logs from Oracle Cloud Infrastructure into Sumo Logic

F41581-03

November 2021

Copyright © 2021, Oracle and/or its affiliates.

reckfectauz.blogspot.com

Source: https://docs.oracle.com/en/learn/blog_sumologic/index.html

Bagikan Artikel ini