CP Auto tagger #2503

cpefatimaabdillahi · 2023-12-15T17:26:19Z

This pull request involves several changes across eight files:

superdesk/publish/formatters/imatrics.py
superdesk/publish/formatters/ninjs_formatter.py: Enhanced formatting of controlled vocabulary items and merged various entities into subjects for a NINJS response.
superdesk/publish/formatters/semaphore.py: Created a new formatter for Semaphore integration.
superdesk/publish/transmitters/imatrics.py
superdesk/publish/transmitters/semaphore.py: Introduced a new transmitter for Semaphore integration.
superdesk/text_checkers/ai/init.py: Revised article validation and processing for AI services.
superdesk/text_checkers/ai/imatrics.py
superdesk/text_checkers/ai/semaphore.py: Expanded image search capabilities and updated concept formatting.

These changes focus on integration enhancements and more refined article transformation logic across different services and formatters in Superdesk.

Variables are stored in our environment file. Please let us know if we should we move these variables to Superdesk's config file.

V1: Added Code for Imatric to use Semaphore API.

V1: Syntax error addressed.

Added More Logging to TroubleShoot.

V2: Added Logging

inconsistency in the use of tabs and spaces

Added More Logging and updated code to transform input before processing.

Added some prints to check

Added Semaphore.py to use for Semaphore API

Updated environment variables

Returning errors to front end

Created Semaphore.py in transmitter

Created Semaphore.py Formatter file

added env variables to code

Updated api_key

Changed much part of the code.

exception bug fix

Bug Fixes

Bug Fix.

Bug Fixes

Trying to run the build

bug fixes

analyzed_data = service.analyze(item, doc.get("tags")) removed tags

bug fix

More Bug Fix

bug fix

Made some changes for methods of transforming xml and jsons.

Bug fixes

bug fixes and more loggings added

Trying to run the working Semaphore Code.

Updated the semaphore api key

Added Headline Tag generation to Semaphore.py

Added Slugline Functionality

Added Code for Slugline and Showing tags with ID coming from Semaphore.

Added code to show Heirarchy in Tags and added backend code for Search Tag Responses.

Updated init to take in searchString.

Added Code to get entities in Ninjs Response.

Added Code to fetch Parent Tags.

petrjasek

can you pls reset the changes done to imatrics?

petrjasek · 2023-12-22T11:59:28Z

superdesk/text_checkers/ai/semaphore.py

+        logger.error(data)
+
+
+        self.output = self.analyze(data)


not sure if this should be called in init, seems like it will be called with the application object and not an article anyhow

petrjasek · 2023-12-22T12:00:08Z

superdesk/text_checkers/ai/semaphore.py

+
+
+
+        self.session = requests.Session()  


seems like this is not used later, so you can avoid that or use it later instead of the session defined outside of the class

petrjasek · 2023-12-22T12:00:46Z

superdesk/text_checkers/ai/semaphore.py

+            query = qcode
+            parent_url = self.get_parent_url+query+frank
+
+            response = requests.get(parent_url, headers=headers)


it's better to use session everywhere

petrjasek · 2023-12-22T12:05:38Z

superdesk/text_checkers/ai/semaphore.py

+
+	# Embed the 'body_html' into the XML template		
+        xml_output = xml_template.format(headline,headline_extended,body_html,slugline)
+        xml_output = clean_html_content(xml_output)


you can use get_text helper to convert html to text

Only change not done is using get_helper to convert from html to text as it did not work with the API request.

petrjasek

in general we plan to move the code to superdesk-cp repo, but we still need to get rid of those changes in the non semaphore files I commented on, some things like changing the text_checkers schema we can keep

petrjasek · 2024-01-10T14:13:23Z

superdesk/text_checkers/ai/__init__.py

@@ -88,7 +90,8 @@ def create(self, docs, **kwargs):
        except KeyError:
            raise SuperdeskApiError.notFoundError("{service} service can't be found".format(service=service))

-        analyzed_data = service.analyze(item, doc.get("tags"))
+        # analyzed_data = service.analyze(item, doc.get("tags"))
+        analyzed_data = service.analyze(item)


would need the previous version here

petrjasek · 2024-01-10T14:15:21Z

superdesk/publish/formatters/ninjs_formatter.py

would be good to do those changes in a custom formatter, either that Semaphore one or some CP specific NINJS.
I think those changes would break some existing integrations like with newshub

Created CP NINJS Formatter

Modified Code with Added Functionality to Run when a Tag is Created in KMM.

Update semaphore.py in Formatters to work with ninjs_formatter_2

Added Names for NinjsV3

Added Code to write back to KMM.

petrjasek · 2024-01-25T09:09:36Z

hi @cpefatimaabdillahi I've pushed the changes from the PR to superdesk/superdesk-cp#189 , can you pls continue there? btw was trying to run the code locally but was getting some 500 error from semaphore api

Added Changed Name and Type to the Formatter

tcp-bhargav and others added 30 commits August 11, 2023 15:44

Update imatrics.py

ca483fe

V1: Added Code for Imatric to use Semaphore API.

Update imatrics.py

8f942a8

V1: Syntax error addressed.

V2: Update imatrics.py

a6d9fd9

Added More Logging to TroubleShoot.

Update imatrics.py

3de0597

V2: Added Logging

Update imatrics.py

6a8ee9c

V2: Added Logging

Update imatrics.py

13af372

inconsistency in the use of tabs and spaces

Update imatrics.py

2396cbe

Added More Logging and updated code to transform input before processing.

Update imatrics.py

fd6618f

Added some prints to check

Create semaphore.py

f73fcc3

Added Semaphore.py to use for Semaphore API

Update semaphore.py

c88c82d

Updated environment variables

Update semaphore.py

7755b40

Returning errors to front end

Create semaphore.py

d388769

Created Semaphore.py in transmitter

Create semaphore.py

ad38e5c

Created Semaphore.py Formatter file

Update semaphore.py

8a3caca

added env variables to code

Update semaphore.py

e94651d

Updated api_key

Update semaphore.py

032d41d

Changed much part of the code.

Update semaphore.py

1dff98c

exception bug fix

Update semaphore.py

eaadbbb

Bug Fixes

Update semaphore.py

171dc12

Bug Fix.

Update semaphore.py

9f0843e

Bug Fixes

Update semaphore.py

6dbe94e

Trying to run the build

Update semaphore.py

0e30fca

bug fixes

Update __init__.py

19aaacf

analyzed_data = service.analyze(item, doc.get("tags")) removed tags

Update semaphore.py

c2f1cad

bug fix

Update semaphore.py

ecd98be

bug fix

Update semaphore.py

ca28e4e

More Bug Fix

Update semaphore.py

5709de1

bug fix

Update semaphore.py

34a8797

Made some changes for methods of transforming xml and jsons.

Update semaphore.py

8008907

Bug fixes

Update semaphore.py

b98efbc

bug fixes and more loggings added

tcp-bhargav added 12 commits November 15, 2023 13:13

Update imatrics.py

1dfb71c

Trying to run the working Semaphore Code.

Update semaphore.py

b8c73ba

Updated the semaphore api key

Update semaphore.py

7affaa6

Added Headline Tag generation to Semaphore.py

Update __init__.py

cf2b806

Added Slugline Functionality

Update semaphore.py

1115db9

Added Code for Slugline and Showing tags with ID coming from Semaphore.

Update semaphore.py

989d35a

Added code to show Heirarchy in Tags and added backend code for Search Tag Responses.

Update __init__.py

5d45786

Updated init to take in searchString.

Update ninjs_formatter.py

a01fbfd

Added Code to get entities in Ninjs Response.

Update semaphore.py

a581b73

Added Code to fetch Parent Tags.

Added Comments for better Reference..

503787c

Added Comments For Better Reference

2e358de

Removed a couple print Statements.

a9c0144

petrjasek reviewed Dec 22, 2023

View reviewed changes

tcp-bhargav added 2 commits December 23, 2023 00:00

imatrics changes reverted.

6b4457d

Updated with modifications asked by Petr.

2a954db

Only change not done is using get_helper to convert from html to text as it did not work with the API request.

petrjasek reviewed Jan 10, 2024

View reviewed changes

tcp-bhargav added 11 commits January 15, 2024 09:29

Create cp_ninjs_formatter

ce532de

Created CP NINJS Formatter

Rename cp_ninjs_formatter to cp_ninjs_formatter.py

90277f4

Update ninjs_formatter.py Reverted back to the Original.

e23ada9

Update __init__.py -- Reverted to Original Code

814ec1f

Update __init__.py with Create Tag in KMM Feature

c481e60

Modified Code with Added Functionality to Run when a Tag is Created in KMM.

Update semaphore.py in Formatters to work with ninjs_formatter_2

7aa421a

Update semaphore.py in Formatters to work with ninjs_formatter_2

Update and rename cp_ninjs_formatter.py to ninjs_formatter_2.py

083e518

Update __init__.py. Changed ninjs_formatter import to ninjs_formatter_2

fbe407a

Update ninjs_ftp_formatter.py to work with our ninjs_formatter_2

1b635c2

Update vocabularies.json

c6e8ff7

Added Names for NinjsV3

Update semaphore.py

86f8f82

Added Code to write back to KMM.

Update ninjs_formatter_2.py

3711753

Added Changed Name and Type to the Formatter

petrjasek closed this Feb 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CP Auto tagger #2503

CP Auto tagger #2503

cpefatimaabdillahi commented Dec 15, 2023

petrjasek left a comment

petrjasek Dec 22, 2023

petrjasek Dec 22, 2023

petrjasek Dec 22, 2023

petrjasek Dec 22, 2023

petrjasek left a comment

petrjasek Jan 10, 2024

petrjasek Jan 10, 2024

petrjasek commented Jan 25, 2024

CP Auto tagger #2503

CP Auto tagger #2503

Conversation

cpefatimaabdillahi commented Dec 15, 2023

petrjasek left a comment

Choose a reason for hiding this comment

petrjasek Dec 22, 2023

Choose a reason for hiding this comment

petrjasek Dec 22, 2023

Choose a reason for hiding this comment

petrjasek Dec 22, 2023

Choose a reason for hiding this comment

petrjasek Dec 22, 2023

Choose a reason for hiding this comment

petrjasek left a comment

Choose a reason for hiding this comment

petrjasek Jan 10, 2024

Choose a reason for hiding this comment

petrjasek Jan 10, 2024

Choose a reason for hiding this comment

petrjasek commented Jan 25, 2024