chore: add logger at integration level

[sanpj2292]

What are the changes introduced in this PR?

We are adding structured logger in all flows necessary.

Example of structured log

{"destinationId":"destId","level":"warn","logMetadata":{"body":{"query":"SELECT conversion_action.id FROM conversion_action WHERE conversion_action.name = 'Order Completed'"},"method":"post","url":"https://googleads.googleapis.com/v16/customers/CUST_ID/googleAds:searchStream"},"message":"[GOOGLE_ADWORDS_OFFLINE_CONVERSIONS] get conversion action id request","sourceId":"srcId","timestamp":"2024-06-12 20:11:02+05:30","workspaceId":"wspId"}
{"destinationId":"destId","level":"warn","logMetadata":{"body":{"jobId":"8212712223577296681","partialFailureError":{"code":3,"details":[{"@type":"type.googleapis.com/google.ads.googleads.v16.errors.GoogleAdsFailure","errors":[{"errorCode":{"rangeError":"TOO_HIGH"},"location":{"fieldPathElements":[{"fieldName":"conversions","index":0},{"fieldName":"external_attribution_data"},{"fieldName":"external_attribution_credit"}]},"message":"Too high.","trigger":{"doubleValue":10}}],"requestId":"twcxvLag1781RZAHqVMncg"}],"message":"Too high., at conversions[0].external_attribution_data.external_attribution_credit"},"results":[{}]},"headers":{"alt-svc":"h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000","cache-control":"private","content-type":"application/json; charset=UTF-8","date":"Wed, 12 Jun 2024 14:41:05 GMT","request-id":"twcxvLag1781RZAHqVMncg","server":"ESF","transfer-encoding":"chunked","vary":"Origin, X-Origin, Referer","x-content-type-options":"nosniff","x-frame-options":"SAMEORIGIN","x-xss-protection":"0"},"status":200},"message":"[GOOGLE_ADWORDS_OFFLINE_CONVERSIONS] deliver event to destination","sourceId":"srcId","timestamp":"2024-06-12 20:11:05+05:30","workspaceId":"wspId"}

• Standardized Log Levels:
  • Introduced an enumeration for log levels: debug, info, warn, and error.
• Logger initialisation:
  • should be initialised with const logger = structuredLogger()
  • Default: Creates a logger with the default log level(error).
  • There are couple of options we support at the moment
    • Custom Log Level: Allows setting the desired log level during initialization (e.g., structuredLogger({ level: 'debug' })).
    • Custom fillExcept fields: Allows setting of desired fields to come up at the top-level in structured-logger. For example if we need destId field to be at top-level.
      • We would need to

          const logger = structuredLogger({fillExcept:['destId']}); 
          logger.error('error occurred', {destId: '1234i'}); 
         // Output log -> { level:"error", destId:"1234i", .... }

• Request and response log:
  • Currently we are supporting two extra methods called requestLog and responseLog
  • One logs the request sent to external APIs(destination). It logs url, body and method of request
  • One logs the response obtained from external APIs(destination). Logs url, body and headers of response
  • To see these logs, atm we have to set LOG_LEVEL to warn

Note

• When calling logger instance methods, do not pass an ordinary string as 2nd arg(as of now)
  - Like so, logger.error('My error:', "err")
  - As this would generate a log {level: "error", "logMetadata": {0:"e",1:"r",2:"r"}, message:"My error", ...}

What is the related Linear task?

Resolves INT-2102

Please explain the objectives of your changes below

Put down any required details on the broader aspect of your changes. If there are any dependent changes, mandatorily mention them here

Any changes to existing capabilities/behaviour, mention the reason & what are the changes ?

N/A

Any new dependencies introduced with this change?

N/A

Any new generic utility introduced or modified. Please explain the changes.

N/A

Any technical or performance related pointers to consider with the change?

N/A

@coderabbitai review

---

Developer checklist

• My code follows the style guidelines of this project

• No breaking changes are being introduced.

• All related docs linked with the PR?

• All changes manually tested?

• Any documentation changes needed with this change?

• Is the PR limited to 10 file changes?

• Is the PR limited to one linear task?

• Are relevant unit and component test-cases added?

Reviewer checklist

• Is the type of change in the PR title appropriate as per the changes?

• Verified that there are no credentials or confidential data exposed with the changes.

[devops-github-rudderstack]

Test report for this run is available at: https://test-integrations-dev.s3.amazonaws.com/integrations-test-reports/rudder-transformer/3401/test-report.html

[codecov]

Codecov Report

Attention: Patch coverage is 96.13260% with 7 lines in your changes missing coverage. Please review.

Project coverage is 88.03%. Comparing base (3aee494) to head (d5b2282).
Report is 4 commits behind head on develop.

Files	Patch %	Lines
src/util/utils.js	0.00%	2 Missing ⚠️
src/controllers/delivery.ts	83.33%	1 Missing ⚠️
src/services/destination/postTransformation.ts	87.50%	1 Missing ⚠️
src/util/redis/redisConnector.js	83.33%	1 Missing ⚠️
...gle_adwords_enhanced_conversions/networkHandler.js	93.33%	1 Missing ⚠️
...ogle_adwords_offline_conversions/networkHandler.js	97.14%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3401      +/-   ##
===========================================
+ Coverage    88.00%   88.03%   +0.03%     
===========================================
  Files          572      574       +2     
  Lines        30910    31056     +146     
  Branches      7385     7410      +25     
===========================================
+ Hits         27202    27340     +138     
- Misses        3398     3405       +7     
- Partials       310      311       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
69.1% Coverage on New Code
2.8% Duplication on New Code

See analysis details on SonarCloud