fix((INT-360): enhancement: shopify ecom fix and enhancement #2493

anantjain45823 · 2023-08-18T11:27:34Z

Description of the change

We are changing the code base for shopify as there are many layers through which a payload goes to be enhanced
This PR includes the changes regarding the same.
We have used layering concept where we have identifier events Layer, Track event Layer, Identify Events layer and enhancement Layer.
Todo :- Incorporate mapping for each ecommerce event.

Type of change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Related issues

Fix #1

Checklists

Development

Lint rules pass locally
The code changed/added as part of this pull request has been covered with tests
All tests related to the changed code pass in development

Code review

This pull request has a descriptive title and information useful to a reviewer. There may be a screenshot or screencast attached
"Ready for review" label attached to the PR and reviewers mentioned in a comment
Changes have been reviewed by at least one other engineer
Issue from task tracker has a link to this pull request

linear · 2023-08-18T11:31:50Z

INT-153 Research: Shopify Ecomm spec review

Repo: https://github.com/rudderlabs/rudder-shopify-tracker

Analyze whether shopify events are adhering to Rudderstack Ecomm spec

Research Spec

Shopify Ecomm spec review

codecov · 2023-08-18T11:32:48Z

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (8bca3e4) 87.24% compared to head (093825f) 87.10%.

❗ Current head 093825f differs from pull request most recent head cffefdf. Consider uploading reports for the commit cffefdf to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2493      +/-   ##
===========================================
- Coverage    87.24%   87.10%   -0.14%     
===========================================
  Files          641      613      -28     
  Lines        28519    28540      +21     
  Branches      6780     6776       -4     
===========================================
- Hits         24880    24861      -19     
- Misses        3310     3347      +37     
- Partials       329      332       +3

Files	Coverage Δ
src/v0/sources/shopify_v2/commonUtils.js	`100.00% <100.00%> (ø)`
src/v0/sources/shopify_v2/config.js	`100.00% <100.00%> (ø)`
src/v0/sources/shopify_v2/enrichmentLayer.js	`100.00% <100.00%> (ø)`
src/v0/sources/shopify_v2/identifierEventsLayer.js	`100.00% <100.00%> (ø)`
src/v0/sources/shopify_v2/identifyEventsLayer.js	`100.00% <100.00%> (ø)`
...c/v0/sources/shopify_v2/identityResolutionLayer.js	`100.00% <100.00%> (ø)`
src/v0/sources/shopify_v2/trackEventsLayer.js	`100.00% <100.00%> (ø)`
src/v0/sources/shopify_v2/transform.js	`100.00% <100.00%> (ø)`
src/v0/util/index.js	`87.85% <100.00%> (+0.02%)`	⬆️

... and 136 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/v0/sources/shopify V2/commonUtils.js

src/v0/sources/shopify V2/config.js

src/v0/sources/shopify V2/commonUtils.js

src/v0/sources/shopify V2/config.js

src/v0/sources/shopify V2/trackEventsLayer.js

src/v0/sources/shopify V2/config.js

linear · 2023-08-22T06:16:59Z

INT-360 Shopify Ecom Implementation: Draft Implementation

src/v0/sources/shopify V2/commonUtils.js

src/util/redis/testData/shopify_v2_source.json

src/v0/sources/shopify_v2/data/OrderCompletedConfig.json

utsabc · 2023-09-22T05:20:31Z

src/util/redis/redisConnector.test.js

@@ -3,7 +3,7 @@ const path = require('path');
 const version = 'v0';
 const { RedisDB } = require('./redisConnector');
 jest.mock('ioredis', () => require('../../../test/__mocks__/redis'));
-const sourcesList = ['shopify'];
+const sourcesList = ['shopify', 'shopify_v2'];


lets not keep shopify_v2. Lets create a new Shopify dir under v1 dir on the same level as v0

refer this please

utsabc · 2023-09-22T05:42:40Z

src/v0/sources/shopify_v2/identifierEventsLayer.js

+const { RedisDB } = require('../../../util/redis/redisConnector');
+const logger = require('../../../logger');
+
+const identifierEventLayer = {


Suggested change

const identifierEventLayer = {

const IdentifierEventLayer = {

utsabc · 2023-09-22T05:45:09Z

src/v0/sources/shopify_v2/config.js

+  statusCode: 200,
+};
+
+const identifierEvents = ['rudderIdentifier', 'rudderSessionIdentifier'];


Suggested change

const identifierEvents = ['rudderIdentifier', 'rudderSessionIdentifier'];

const IDENTIFIER_EVENTS = ['rudderIdentifier', 'rudderSessionIdentifier'];

utsabc · 2023-09-22T05:47:15Z

src/v0/sources/shopify_v2/identifierEventsLayer.js

+  async processIdentifierEvent(event, metricMetadata) {
+    let value;
+    let field;
+    if (event.event === 'rudderIdentifier') {


Change the argument name to something which provides more clarity event.event doesn't provide much clarity.

utsabc · 2023-09-22T05:49:54Z

src/v0/sources/shopify_v2/commonUtils.js

+      lineItems[element.id] = {
+        quantity: element?.quantity,
+        variant_id: element?.variant_id,
+        key: element?.key,
+        price: element?.price,
+        product_id: element?.product_id,
+        sku: element?.sku,
+        title: element?.title,
+        vendor: element?.vendor,
+      };


Optional chaining is not needed here

utsabc · 2023-09-22T06:21:40Z

src/v0/sources/shopify_v2/transform.js

+      // eslint-disable-next-line no-param-reassign
+      event.userId = String(event.userId);
+    }
+    if (!get(event, 'traits.email')) {


Suggested change

if (!get(event, 'traits.email')) {

if (!get(event, 'context.traits.email')) {

utsabc · 2023-09-22T06:22:01Z

src/v0/sources/shopify_v2/transform.js

+    event.setProperty(`integrations.${INTEGRATION}`, true);
+    event.setProperty('context.library', {
+      name: 'RudderStack Shopify Cloud',
+      version: '1.0.0',


Suggested change

version: '1.0.0',

version: '1.1.0',

Should be a constant this version number based on the implementations appropriate constant should be used

utsabc · 2023-09-22T06:26:47Z

src/v0/sources/shopify_v2/trackEventsLayer.js

+  extractCustomFields,
+} = require('../../util');
+
+const trackLayer = {


Suggested change

const trackLayer = {

const TrackLayer = {

utsabc · 2023-09-22T06:27:49Z

src/v0/sources/shopify_v2/trackEventsLayer.js

+    }
+  },
+
+  async processTrackEvent(event, eventName, dbData, metricMetadata) {


Suggested change

async processTrackEvent(event, eventName, dbData, metricMetadata) {

async processTrackEvent(shopifyEvent, shopifyEventName, dbData, metricMetadata) {

utsabc · 2023-09-22T06:37:55Z

src/v0/sources/shopify_v2/trackEventsLayer.js

+    return updatedCartProperties;
+  },
+
+  async generateProductAddedAndRemovedEvents(event, dbData, metricMetadata) {


Suggested change

async generateProductAddedAndRemovedEvents(event, dbData, metricMetadata) {

async generateProductAddedAndRemovedEvents(shopifyEvent, dbData, metricMetadata) {

utsabc · 2023-09-22T08:10:56Z

src/v0/sources/shopify_v2/trackEventsLayer.js

+    // if no prev cart is found we trigger product added event for every line_item present
+    if (!prevLineItems) {
+      event?.line_items.forEach((product) => {
+        const updatedProduct = this.getUpdatedProductProperties(product, event?.id || event?.token);


Suggested change

const updatedProduct = this.getUpdatedProductProperties(product, event?.id || event?.token);

const updatedProduct = this.getUpdatedProductProperties(product, event.id || event.token);

utsabc · 2023-09-22T08:30:01Z

src/v0/sources/shopify_v2/trackEventsLayer.js

+    const updatedCartProperties = product;
+    if (updatedQuantity) {
+      updatedCartProperties.quantity = updatedQuantity;
+    }
+    updatedCartProperties.cart_id = cart_token;
+    return updatedCartProperties;
+  },


why are we calling updatedCartProperties for this variable we are updating products?

utsabc · 2023-09-22T08:49:31Z

src/v0/sources/shopify_v2/trackEventsLayer.js

+    let properties = this.createPropertiesForEcomEvent(event, shopifyTopic);
+    properties = removeUndefinedAndNullValues(properties);


Suggested change

let properties = this.createPropertiesForEcomEvent(event, shopifyTopic);

properties = removeUndefinedAndNullValues(properties);

const properties = removeUndefinedAndNullValues(this.createPropertiesForEcomEvent(event, shopifyTopic));

utsabc · 2023-09-22T09:46:37Z

src/v0/sources/shopify_v2/trackEventsLayer.js

+      );
+      if (productAddedOrRemovedEvents.length === 0) return [NO_OPERATION_SUCCESS];
+      productAddedOrRemovedEvents = productAddedOrRemovedEvents.map((productEvent) =>
+        this.mapCustomerDetails(productEvent, event, dbData, eventName),


Suggested change

this.mapCustomerDetails(productEvent, event, dbData, eventName),

this.mapCustomerDetails(productEvent, shopifyEvent, dbData, eventName),

utsabc · 2023-09-22T09:47:03Z

src/v0/sources/shopify_v2/trackEventsLayer.js

+    if (!get(payload, 'traits.email')) {
+      const email = extractEmailFromPayload(event);
+      if (email) {
+        updatedPayload.setProperty('traits.email', email);


Suggested change

updatedPayload.setProperty('traits.email', email);

updatedPayload.setProperty('context.traits.email', email);

utsabc · 2023-09-22T09:49:45Z

src/v0/sources/shopify_v2/identityResolutionLayer.js

+    const cartToken = getCartToken(message, shopifyTopic);
+    if (!isDefinedAndNotNull(cartToken)) {


Can't we validate using dbData since we already have that? Why are we validating with cartToken here?

utsabc · 2023-09-22T09:51:48Z

src/v0/sources/shopify_v2/identityResolutionLayer.js

+    );
+    if (isDefinedAndNotNull(anonymousId)) {
+      updatedMessage.setProperty('anonymousId', anonymousId);
+    } else if (!message.userId) {


Suggested change

} else if (!message.userId) {

} else if (!isDefineNotNullandBlank(message.userId)) {

utsabc · 2023-09-22T09:52:09Z

src/v0/sources/shopify_v2/identityResolutionLayer.js

+    } else if (!message.userId) {
+      updatedMessage.setProperty('userId', 'shopify-admin');
+    }
+    if (isDefinedAndNotNull(sessionId)) {


Use not blank as well for empty string

utsabc · 2023-09-22T09:54:21Z

src/v0/sources/shopify_v2/trackEventsLayer.js

+    if (SHOPIFY_TO_RUDDER_ECOM_EVENTS_MAP.CART_UPDATED === eventName) {
+      let productAddedOrRemovedEvents = await this.generateProductAddedAndRemovedEvents(
+        event,
+        dbData,
+        metricMetadata,
+      );
+      if (productAddedOrRemovedEvents.length === 0) return [NO_OPERATION_SUCCESS];
+      productAddedOrRemovedEvents = productAddedOrRemovedEvents.map((productEvent) =>
+        this.mapCustomerDetails(productEvent, event, dbData, eventName),
+      );
+      return productAddedOrRemovedEvents;
+    }
+    // if event is checkout updated we get the updated event name
+    if (SHOPIFY_TO_RUDDER_ECOM_EVENTS_MAP.CHECKOUTS_UPDATE === eventName) {
+      updatedEventName = this.getUpdatedEventNameForCheckoutUpateEvent(event);
+    }
+    if (Object.keys(RUDDER_ECOM_MAP).includes(updatedEventName)) {
+      payload = this.ecomPayloadBuilder(event, updatedEventName);
+    } else if (Object.keys(SHOPIFY_NON_ECOM_TRACK_MAP).includes(updatedEventName)) {
+      payload = this.nonEcomPayloadBuilder(event, updatedEventName);
+    } else {
+      stats.increment('invalid_shopify_event', {
+        event: eventName,
+        ...metricMetadata,
+      });
+      return [NO_OPERATION_SUCCESS];
+    }
+    return [this.mapCustomerDetails(payload, event, dbData, eventName)];
+  },


can we try using switch case here?

utsabc · 2023-09-22T09:56:38Z

src/v0/sources/shopify_v2/commonUtils.js

+  // return msgpack.encode(lineItems);
+};
+
+const getUnhashedLineItems = (lineItems) => {


Change fxn name

utsabc · 2023-09-22T09:57:51Z

src/v0/sources/shopify_v2/trackEventsLayer.js

+      if (prevQuantity) {
+        delete prevLineItems[key];
+      }


Elaborate what are we doing here?

sonarqubecloud · 2023-10-13T05:37:58Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
3 Code Smells

35.2% Coverage
6.2% Duplication

devops-github-rudderstack · 2023-11-03T01:24:08Z

This PR is considered to be stale. It has been open for 20 days with no further activity thus it is going to be closed in 7 days. To avoid such a case please consider removing the stale label manually or add a comment to the PR.

refactor: shopify

2e02c45

anantjain45823 added the Ready for Review label Aug 18, 2023

anantjain45823 requested review from Gauravudia and a team August 18, 2023 11:27

anantjain45823 self-assigned this Aug 18, 2023

anantjain45823 requested a review from a team as a code owner August 18, 2023 11:27

anantjain45823 requested review from krishna2020 and sandeepdsvs August 18, 2023 11:27

Merge branch 'develop' into refactor.shopify

0a2dfcf

anantjain45823 changed the title ~~enhancement: shopify ecom fix and enhancement~~ fix: enhancement: shopify ecom fix and enhancement Aug 18, 2023

anantjain45823 changed the title ~~fix: enhancement: shopify ecom fix and enhancement~~ fix((INT-153): enhancement: shopify ecom fix and enhancement Aug 18, 2023

Gauravudia reviewed Aug 21, 2023

View reviewed changes

anantjain45823 changed the title ~~fix((INT-153): enhancement: shopify ecom fix and enhancement~~ fix((INT-360): enhancement: shopify ecom fix and enhancement Aug 22, 2023

chore: used prettier

19556fa

koladilip reviewed Aug 23, 2023

View reviewed changes

src/v0/sources/shopify V2/commonUtils.js Outdated Show resolved Hide resolved

anantjain45823 and others added 5 commits August 24, 2023 08:55

basic changes

1d5033a

refactor: shopify ecomm (#2510)

0c5a1c5

Merge branch 'develop' into refactor.shopify

6a52c0a

refactor: update line item exlusion fields

7b79ea4

chore: track events flow improvement

7690f9c

anantjain45823 requested a review from a team as a code owner August 24, 2023 08:34

fix: mapping config

94bd394

Gauravudia self-assigned this Aug 24, 2023

Gauravudia and others added 4 commits August 24, 2023 22:36

fix: ecom mapping

caf9113

fix: order completed mapping

5720d7c

feat: add variant name mapping in lineitem

8ae971a

chore: added test cases and some fixes

f716e0d

anantjain45823 and others added 7 commits September 5, 2023 16:25

chore: test cases added for products event

293688b

remove: extra file

c6727f7

chore: resolve sonar issue

7db623a

Merge branch 'develop' into refactor.shopify

db09237

chore: add testcase

6897f52

Merge branch 'develop' into refactor.shopify

ee364f5

small mapping fix

e98bac4

ItsSudip reviewed Sep 22, 2023

View reviewed changes

src/util/redis/testData/shopify_v2_source.json Outdated Show resolved Hide resolved

src/v0/sources/shopify_v2/data/OrderCompletedConfig.json Show resolved Hide resolved

utsabc reviewed Sep 22, 2023

View reviewed changes

Gauravudia and others added 5 commits September 25, 2023 10:26

chore: addressing review comments

50a66ad

Merge branch 'develop' into refactor.shopify

97c562f

Merge branch 'develop' into refactor.shopify

093825f

Merge branch 'develop' into refactor.shopify

263c5ba

Merge branch 'develop' into refactor.shopify

cffefdf

devops-github-rudderstack added the Stale label Nov 3, 2023

devops-github-rudderstack closed this Nov 10, 2023

anantjain45823 deleted the refactor.shopify branch December 13, 2023 14:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix((INT-360): enhancement: shopify ecom fix and enhancement #2493

fix((INT-360): enhancement: shopify ecom fix and enhancement #2493

anantjain45823 commented Aug 18, 2023 •

edited by Gauravudia

Loading

linear bot commented Aug 18, 2023

Research Spec

codecov bot commented Aug 18, 2023 •

edited

Loading

linear bot commented Aug 22, 2023

utsabc Sep 22, 2023

anantjain45823 Sep 25, 2023

utsabc Sep 22, 2023

Gauravudia Sep 25, 2023

utsabc Sep 22, 2023

Gauravudia Sep 25, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

Gauravudia Sep 25, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

utsabc Sep 22, 2023

sonarqubecloud bot commented Oct 13, 2023

devops-github-rudderstack commented Nov 3, 2023

	const identifierEventLayer = {
	const IdentifierEventLayer = {

	const identifierEvents = ['rudderIdentifier', 'rudderSessionIdentifier'];
	const IDENTIFIER_EVENTS = ['rudderIdentifier', 'rudderSessionIdentifier'];

	if (!get(event, 'traits.email')) {
	if (!get(event, 'context.traits.email')) {

	async processTrackEvent(event, eventName, dbData, metricMetadata) {
	async processTrackEvent(shopifyEvent, shopifyEventName, dbData, metricMetadata) {

	async generateProductAddedAndRemovedEvents(event, dbData, metricMetadata) {
	async generateProductAddedAndRemovedEvents(shopifyEvent, dbData, metricMetadata) {

	const updatedProduct = this.getUpdatedProductProperties(product, event?.id \|\| event?.token);
	const updatedProduct = this.getUpdatedProductProperties(product, event.id \|\| event.token);

		let properties = this.createPropertiesForEcomEvent(event, shopifyTopic);
		properties = removeUndefinedAndNullValues(properties);

	let properties = this.createPropertiesForEcomEvent(event, shopifyTopic);
	properties = removeUndefinedAndNullValues(properties);
	const properties = removeUndefinedAndNullValues(this.createPropertiesForEcomEvent(event, shopifyTopic));

	this.mapCustomerDetails(productEvent, event, dbData, eventName),
	this.mapCustomerDetails(productEvent, shopifyEvent, dbData, eventName),

	updatedPayload.setProperty('traits.email', email);
	updatedPayload.setProperty('context.traits.email', email);

		const cartToken = getCartToken(message, shopifyTopic);
		if (!isDefinedAndNotNull(cartToken)) {

	} else if (!message.userId) {
	} else if (!isDefineNotNullandBlank(message.userId)) {

fix((INT-360): enhancement: shopify ecom fix and enhancement #2493

fix((INT-360): enhancement: shopify ecom fix and enhancement #2493

Conversation

anantjain45823 commented Aug 18, 2023 • edited by Gauravudia Loading

Description of the change

Type of change

Related issues

Checklists

Development

Code review

linear bot commented Aug 18, 2023

Research Spec

codecov bot commented Aug 18, 2023 • edited Loading

Codecov Report

linear bot commented Aug 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Oct 13, 2023

devops-github-rudderstack commented Nov 3, 2023

anantjain45823 commented Aug 18, 2023 •

edited by Gauravudia

Loading

codecov bot commented Aug 18, 2023 •

edited

Loading