Task to export calculated numbers of the monthly reports back to HubSpot #6439

marcospri · 2024-07-10T12:56:36Z

For:

Generate organization usage stats from LMS #6159

This PR needs a follow up to schedule the task in h-periodic.

To summarize the process we:

Fetch HubSpot companies with tasks.hubspot.refresh_hubspot_data
We generate monthly usage reports for thos companies with task.organization.schedule_monthly_deal_report

now, with this commit we:

Push the calculated numbers (unique user and teachers) to the corresponding fields in HubSpot companies.

Testing

Describing here the process I followed locally to test this end to end.

Fetch all HubSpot companies:

refresh_hubspot_data.delay()

Modify some of these companies to match your local organizations, eg:

update hubspot_company set lms_organization_id = 'local.lms.org.FFIHD9f1RwyLPfx14zM_7Q' where id = 11;

Generate/Insert some reports that match that organization:

select * from organization_usage_report where organization_id in (1,2);
 id | organization_id |     tag      |                                   key                                   |   since    |   until    | unique_users | report |          created           |          updated           | unique_teachers 
----+-----------------+--------------+-------------------------------------------------------------------------+------------+------------+--------------+--------+----------------------------+----------------------------+-----------------
 11 |               1 | monthly-deal | local.lms.org.LDtcl7EUTeW2UERPJLAVtA-monthly-deal-2023-01-01-2023-07-31 | 2023-01-01 | 2023-07-31 |            0 | []     | 2024-06-18 10:33:22.193294 | 2024-06-18 10:33:22.193294 |               0
  2 |               2 | monthly-deal | local.lms.org.LDtcl7EUTeW2UERPJLAVtA-monthly-deal-2023-01-01-2024-01-31 | 2023-01-01 | 2024-01-31 |            0 | []     | 2024-06-18 10:29:28.260707 | 2024-06-18 10:29:28.260707 |               0

Finally, run the new task:

from lms.tasks.hubspot import export_companies_contract_billables; 
export_companies_contract_billables.delay()

Let's check that the import actually happened, we log into HuBSpot, and go to the imports in: https://app.hubspot.com/import/21081992
And we can see the records we modified locally to match the HubSpot data being updated and the rest of the rows having no value (--) in those columns.

marcospri · 2024-07-10T13:12:29Z

lms/services/hubspot/_client.py

@@ -1,5 +1,22 @@
+import csv


Part of this code is copied and adapted from the reporting system.

I've avoided many of the abstractions there, so here we hard code the columns we are going to send for companies for example.

I don't think we are going to be modifying this very often right now but we can move into abstractions as we apply this to more models and/or columns.

marcospri · 2024-07-10T13:13:25Z

lms/services/hubspot/_client.py

+                }
+            ]
+
+            import_request = {


We name here the import job on HubSpot side.

marcospri · 2024-07-10T13:13:51Z

lms/services/hubspot/service.py

@@ -27,27 +27,26 @@ def get_company(self, organization_id: str) -> HubSpotCompany | None:
            # More than one company pointing to the same org is a data entry error, ignore them.
            return None

-    def get_companies_with_active_deals(self, date_: date):
+    def _companies_with_active_deals_query(self, date_: date):


Just making the existing query a private method to be able to reusing it for the new code.

marcospri · 2024-07-10T13:14:33Z

lms/services/hubspot/service.py

+    def export_companies_contract_billables(self, date_: date):
+        """Export the contract billable numbers to HubSpot."""
+        query = (
+            self._companies_with_active_deals_query(date_)


From the existing companies_with_active_deals_query(date_) query

marcospri · 2024-07-10T13:14:50Z

lms/services/hubspot/service.py

+        """Export the contract billable numbers to HubSpot."""
+        query = (
+            self._companies_with_active_deals_query(date_)
+            .join(


Join with organizations and reports.

marcospri · 2024-07-10T13:15:02Z

lms/services/hubspot/service.py

+                Organization.public_id == HubSpotCompany.lms_organization_id,
+            )
+            .join(OrganizationUsageReport)
+            .distinct(OrganizationUsageReport.organization_id)


Picking just one report per organization.

marcospri · 2024-07-10T13:15:13Z

lms/services/hubspot/service.py

+            .distinct(OrganizationUsageReport.organization_id)
+            .order_by(
+                OrganizationUsageReport.organization_id,
+                OrganizationUsageReport.until.desc(),


Picking the last report.

marcospri · 2024-07-10T13:15:39Z

lms/services/hubspot/service.py

+                OrganizationUsageReport.organization_id,
+                OrganizationUsageReport.until.desc(),
+            )
+        ).with_only_columns(


And only picking the values we need to pass to the client to generate the CSV.

marcospri · 2024-07-10T13:16:19Z

tests/unit/lms/services/hubspot/_client_test.py

+        csv.writer.return_value.writerow.assert_called_once_with(
+            (sentinel.id, sentinel.teachers, sentinel.users)
+        )
+        api_client.crm.imports.core_api.create.assert_called_once_with(


At this level we just deal with the HubSpot API, we assert we are calling it with the expected values.

marcospri · 2024-07-10T13:16:52Z

tests/unit/lms/services/hubspot/service_test.py

+    def test_export_companies_contract_billables(
+        self, svc, hubspot_api_client, db_session
+    ):
+        organization = factories.Organization(public_id="TEST_ORG")


At this level we care about picking the right data, we set up the data in the DB and assert we get the right values to call the client.

seanh

Suggestion for some more concise and consistent naming: export_billables() (Celery task) --calls-> HubSpotService.export_billables(date) --calls-> HubSpotClient.import_billables(billables)

lms/services/hubspot/_client.py

seanh · 2024-07-11T14:36:31Z

lms/services/hubspot/_client.py

+            ]
+
+            import_request = {
+                "name": f"contract_year_import_{date.today().isoformat()}",


I think the dates should probably be part of the contract_year_billables argument, rather than using the current day here. Those numbers of unique teachers and users in contract_year_billables were calculated at to a given date, it'd be better to include that date in the data rather than implicitly assuming that it matches date.today() at the time when this line later runs.

I have included this as a second independent parameter

seanh · 2024-07-11T14:37:17Z

lms/services/hubspot/service.py

@@ -27,27 +27,26 @@ def get_company(self, organization_id: str) -> HubSpotCompany | None:
            # More than one company pointing to the same org is a data entry error, ignore them.
            return None

-    def get_companies_with_active_deals(self, date_: date):
+    def _companies_with_active_deals_query(self, date_: date):


seanh · 2024-07-11T14:39:26Z

lms/services/hubspot/service.py

+
+        # From the point of view of HubSpot we are creating an import
+        self._client.create_company_import(
+            self._db.execute(query).all(),


This is where you should probably include date_ in the arguments to create_company_import()

To summarize the process we: - Fetch HubSpot companies with tasks.hubspot.refresh_hubspot_data - We generate monthly usage reports for thos companies with task.organization.schedule_monthly_deal_report now, with this commit we: - Push the calculated numbers (unique user and teachers) to the corresponding fields in HubSpot companies.

marcospri · 2024-07-12T08:43:27Z

Merged the naming suggestions 👍

Run the task locally to make sure everything lines up. A new import was created here https://app.hubspot.com/import/21081992

sentry-io · 2024-07-23T14:00:23Z

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

‼️ ApiException: (400) lms.tasks.hubspot.export_companies_contract_bil... View Issue

_{Did you find this useful? React with a 👍 or 👎}

marcospri force-pushed the hubpost-export branch from 96258ee to 28b6508 Compare July 10, 2024 13:19

marcospri commented Jul 10, 2024

View reviewed changes

marcospri requested a review from seanh July 10, 2024 15:18

seanh requested changes Jul 11, 2024

View reviewed changes

marcospri force-pushed the hubpost-export branch from 5d9323e to 440d3b2 Compare July 12, 2024 08:33

marcospri requested a review from seanh July 12, 2024 08:43

seanh approved these changes Jul 12, 2024

View reviewed changes

marcospri merged commit 4c6a2a2 into main Jul 23, 2024
9 checks passed

marcospri deleted the hubpost-export branch July 23, 2024 07:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Task to export calculated numbers of the monthly reports back to HubSpot #6439

Task to export calculated numbers of the monthly reports back to HubSpot #6439

marcospri commented Jul 10, 2024 •

edited

Loading

marcospri Jul 10, 2024

marcospri Jul 10, 2024

marcospri Jul 10, 2024

seanh Jul 11, 2024

marcospri Jul 10, 2024

marcospri Jul 10, 2024

marcospri Jul 10, 2024

marcospri Jul 10, 2024

marcospri Jul 10, 2024

marcospri Jul 10, 2024

marcospri Jul 10, 2024

seanh left a comment

seanh Jul 11, 2024

marcospri Jul 12, 2024

seanh Jul 11, 2024

seanh Jul 11, 2024

marcospri Jul 12, 2024

marcospri commented Jul 12, 2024

sentry-io bot commented Jul 23, 2024

Task to export calculated numbers of the monthly reports back to HubSpot #6439

Task to export calculated numbers of the monthly reports back to HubSpot #6439

Conversation

marcospri commented Jul 10, 2024 • edited Loading

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seanh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcospri commented Jul 12, 2024

sentry-io bot commented Jul 23, 2024

Suspect Issues

marcospri commented Jul 10, 2024 •

edited

Loading