Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

As a user, I want all registration numbers extracted with their type #30

Open
Tracked by #229
snakedye opened this issue Sep 10, 2024 · 17 comments · May be fixed by #73
Open
Tracked by #229

As a user, I want all registration numbers extracted with their type #30

snakedye opened this issue Sep 10, 2024 · 17 comments · May be fixed by #73
Assignees

Comments

@snakedye
Copy link
Contributor

snakedye commented Sep 10, 2024

Description

Context
Currently, our system extracts only one registration number (whichever it finds). This registration number may represent either the fertilizer’s registration or one of the ingredient’s registration numbers. In some cases, multiple registration numbers are listed on the label, indicating multiple registered ingredients.

Problem Statement
The client has requested that we extract all registration numbers from fertilizer labels and clearly identify whether each number represents the fertilizer or its ingredients.

Acceptance Criteria

  • Extract all registration numbers listed on fertilizer labels.
  • Identify whether each registration number represents the fertilizer or its ingredients.

Additional Information (Optional)

  • If there are multiple registration numbers, they must be ingredient registration numbers as it should not be possible to have both fertilizer registration number and ingredient registration number on the same label.
@k-allagbe
Copy link
Member

Should we differentiate nutrients and ingredients?

Also please, reminder to have a chat with @Francois-Werbrouck and @Endlessflow to agree on required updates to the analysis data format.

@Francois-Werbrouck
Copy link

Francois-Werbrouck commented Sep 11, 2024

does this mean every entries of Guaranteed_Analysis will look like:

{
  "nutrient" : "Name",
  "value" : 0.0,
  "unit" : false,
  "registration_number" : ????
}

Also what is a registration number of a Nutrient? Is there a list of all the known Nutrients and their registration number?
Should we have a naming convention to differentiate the registration_number of an Inspection?

@k-allagbe
Copy link
Member

does this mean every entries of Guaranteed_Analysis will look like:

{
  "nutrient" : "Name",
  "value" : 0.0,
  "unit" : false,
  "registration_number" : ????
}

@snakedye and @Endlessflow should answer this. But that should be in ingredient.

Also what is a registration number of a Nutrient? Is there a list of all the known Nutrients and their registration number? Should we have a naming convention to differentiate the registration_number of an Inspection?

Not nutrient, but ingredient. I can ask for examples of labels with registered ingredients. But the client himself said that if an ingredient is registered, it's registration number has to be listed on the label (next to the ingredient we can assume). And this is completely different from the registration number of the fertilizer itself. As long as the information is saved, the nomenclature matters less.
Please take a look at the discussion with the client about labelling options for more information.

@Francois-Werbrouck
Copy link

I've asked the client for further information because the information we have at the moment is confusing. I suggest we wait until developping this to get a better understanding of the needs and our solution's impact

@k-allagbe
Copy link
Member

Additional information from the client: Registered fertilizer products list

@k-allagbe k-allagbe moved this from Todo to In progress in FertiScan Sep 16, 2024
@k-allagbe k-allagbe moved this from In progress to In review in FertiScan Sep 16, 2024
@snakedye
Copy link
Contributor Author

snakedye commented Sep 17, 2024

@Francois-Werbrouck

Should we have a naming convention to differentiate the registration_number of an Inspection?

I think so because it just uses the registration number of the fertilizer everywhere.

That or maybe add a description.

label1

{
  "company_name": "Cameron Chemicals, Inc.",
  "company_address": "930 Old Dil Road, Suffolk VA 23434, USA",
  "company_website": null,
  "company_phone_number": "(757) 487-0656",
  "manufacturer_name": null,
  "manufacturer_address": null,
  "manufacturer_website": null,
  "manufacturer_phone_number": null,
  "fertiliser_name": "GRANULAR ZINC 20% GRANULAIRE",
  "registration_number": "20150458",
  "lot_number": "31923352",
  "weight": [
    {
      "value": 2000,
      "unit": "lbs"
    },
    {
      "value": 907,
      "unit": "kg"
    }
  ],
  "density": null,
  "volume": null,
  "npk": null,
  "guaranteed_analysis_en": {
    "title": "GUARANTEED ANALYSIS",
    "nutrients": [
      {
        "nutrient": "Sulphur (S)",
        "quantity": {
          "value": 3,
          "unit": "%"
        },
        "registration_number": "20150458"
      },
      {
        "nutrient": "Zinc (Zn)",
        "quantity": {
          "value": 20,
          "unit": "%"
        },
        "registration_number": "20150458"
      }
    ]
  },
  "guaranteed_analysis_fr": {
    "title": "ANALYSE GARANTIE",
    "nutrients": [
      {
        "nutrient": "Soufre (S)",
        "quantity": {
          "value": 3,
          "unit": "%"
        },
        "registration_number": "20150458"
      },
      {
        "nutrient": "Zinc (Zn)",
        "quantity": {
          "value": 20,
          "unit": "%"
        },
        "registration_number": "20150458"
      }
    ]
  },
  "cautions_en": [
    "CAUTION this fertilizer contains Zinc and Sulphur and should be used only as recommended. It may prove harmful when misused. Applications of this product should be based on the results of soil and/or tissue analysis.",
    "TO BE USED ONLY IN BLENDING FERTILIZER FOR SOIL APPLICATIONS",
    "Avoid breathing dust/fume/gas/mist/vapors/spray.",
    "Wash skin thoroughly after handling.",
    "Use only outdoors or in a well-ventilated area.",
    "Wear protective gloves/protective clothing/eye protection/face protection.",
    "Store in a well-ventilated place. Keep container tightly closed.",
    "Store locked up.",
    "Dispose of contents/container to an approved disposal site in accordance with local/regional/national regulations."
  ],
  "cautions_fr": [
    "AVERTISSEMENT Cet engrais renferme du Zinc et Soufre et ne doit etre employe que de la maniere recommandee. Il peut etra nocif s'il est employe mal a propos. L'application de ce produit devrait être basé sur les résultats des analyses du sol eVou tissue.",
    "POUR EMPLOI EN MELANGE D'ENGRAIS POUR APPLICATION AU SOL SEULEMENT",
    "Éviter de respirer los poussières/fumées / gaz / brouillards / vapeurs /aérosols",
    "Laver la peas soigneusement après manipulation:",
    "Ufiser seulement en plein air ou dans un endroit bien ventile",
    "Porter des gants de protection / des vêtements de protection des yeux / protection / du visage",
    "Conserver dans un endroit bien venblé. Conserver le récipient bien ferme.",
    "Garder sous chef.",
    "Birineg ), esoternu / récipient dans un site élimination seconde approuvé à la réglementaolegale/naorale / nemasale."
  ],
  "instructions_en": [
    "DIRECTIONS FOR USE: To be used in blending fertilizer for soil application. Applications should be based on results of soil or leaf analysis.",
    "Information regarding the contents and levels of metals in this product is available on the internet at http://www.aspte.agrietas.hut",
    "CONDITIONS OF SALE AND LIMITATION OF WARRANTY AND LIABILITY BEFORE BUYING OR USING THIS PRODUCT, READ THE DIRECTIONS FOR USE AND THE FOLLOWING CONDITIONS OF SALE AND LIMITATION OF WARRANTY AND LIABILITY By buying or using this product, the buyer or user accepts the following Conditions of Sale and Limitation of Warranty and Liability, which no employee or agent of the manufacturer or the seller is authorized to vary The manufacturer warrants that this product conforms to the chemical description on the label and is reasonably fit for the purposes stated in the Directions for Use, when the product is used in accordance with such Directions for Use under normal conditions of use: THE MANUFACTURER MAKES NO OTHER WARRANTY, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE THE BUYER'S OR USERS EXCLUSIVE REMEDY FOR ANY INJURY, LOSS, OR DAMAGE RESULTING FROM THE HANDLING OR USE OF THIS PRODUCT SHALL BE LIMITED TO ONE OF THE FOLLOWING. AT THE ELECTION OF THE MANUFACTURER OR THE SELLER DIRECT DAMAGES NOT EXCEEDING THE PURCHASE PRICE OF THE PRODUCT OR REPLACEMENT OF THE PRODUCT. THE MANUFACTURER AND THE SELLER SHALL NOT BE LIABLE TO THE BUYER OR USER OF THIS PRODUCT FOR ANY CONSEQUENTIAL, SPECIAL, OR INDIRECT DAMAGES, OR DAMAGES IN THE NATURE OF A PENALTY"
  ],
  "instructions_fr": [
    "CONSEILS D'UTILISATION: Pour un emploi en mélange d'engrais pour application au sol. Les valtments devraient être faits selon les résultats obtenus à la suite d'une analyse du sol ou des feuilles. Des renseignements à propos du contarsa ot des niveaux de métaux présents dans le prodat sont disponibles enligne au htp://www.magiczorgmetals.han",
    "CONDITIONS DE VENTE ET UNITATION DE GARANTIE ET DE RESPONSABILITÉ AVANT D'ACHETER OU UTILIBE CE PRICOUIT, LIRE LES INSTRUCTIONS D'UTILISATION ET LES CONDITIONS SUIVANTESDE VENTE ET LA LIMITATION DE GARANTIE ET DE RESPONSABILITE. En acheter ou en stlisant es prodat. fachelour qu usiloafour accopto les condes avartas de verte et de limitation de garantia es de masiporsalle",
    "sur l'étiquette of est raisonnablement en forme pour los fits trorczes tara los instructions d'emploi, lorsque le produit est utilisé conformément ala Instructors d'utilisation dans des concitoro normales diploalor. LE FABRICANT NE DONNE ALCUNE AUTRE GARANTIE, EXPLICITE OU IMPLICITE, Y COMPRIS MAIS SANS SY LIMITER MARCHANDE OU D'ADEQUATON A UN USAGE PARTICULIER. L'ACHETEUR EST OU LE RECOURS EXCLUSIF DE L'UTILISATEUR DE TOUTE BLESSURE, PERTE OU DOMMAGES RESULTANT DE LA MANIPULATION OU DE L'UTILISATION DE CE PRODUIT EST UNITÉE À L'UNE DES SUNANTES AU CHOOL DE LE FABRICANT OU LE VENDEUR: DOMMAGES DIRECTS NE DÉPASSANT LE PRIX D'ACHAT DE LE PRODUIT OU LE REMPLACEMENT DU PRODUIT LE FABRICANT ET LE VENDEUR NE GERAPAS RESPONSABLE ENVERS L'ACHETEUR OU L'UTILISATEUR DE CE PRODUIT POUR TOUT DOMMAGE SPECIAL OU INDIRECTS OU DOMMAGES DANS LE CARACTERE D'UNE SANCTION"
  ],
  "ingredients_en": [
    {
      "nutrient": "Zinc Sulphate",
      "quantity": null,
      "registration_number": null
    },
    {
      "nutrient": "Zinc Oxide",
      "quantity": null,
      "registration_number": null
    }
  ],
  "ingredients_fr": [
    {
      "nutrient": "sulfate de Zinc",
      "quantity": null,
      "registration_number": null
    },
    {
      "nutrient": "Oxide de zinc",
      "quantity": null,
      "registration_number": null
    }
  ]
}

@Francois-Werbrouck
Copy link

From my understanding of this exemple, there seems to be multiple number for each ingredients (see the blue section)?
image

The red section is the registration number of the fertilizer

@Samuelpelletier3124 Gave us another use case:
image

I see/understand the following based on the other use case:

  • 2022739S is the ingredient registration number as we currently call it. It can also be found in the following list
  • I do not know which ingredient this registration number refers to and wonder how the pipeline would do on this case. I will ask the client for clarification on the data side to better understand what are the stakes of that data and what it implies.

image

I also wonder if it could be beneficial to link the public table shared in the previous list with our database of ingredients. We may need to revisit the structure of those object in the DB Schema's. WDYT @k-allagbe

@k-allagbe
Copy link
Member

I also wonder if it could be beneficial to link the public table shared in the previous list with our database of ingredients. We may need to revisit the structure of those object in the DB Schema's. WDYT @k-allagbe

I think that if linking to the public record can help with the pipeline extraction, we should seriously consider it right away. @snakedye and @Endlessflow should have a better input on this. However if it is to have more complete information in our db, this could be an entirely new feature we could present the client in our next meeting and implement in future iterations. Please explain what your vision is about the subject.

@Francois-Werbrouck
Copy link

Francois-Werbrouck commented Sep 17, 2024

My vision was for a more complete information relationship in our database. I for sure will bring it up with the next client meeting.

I'm not 100% sure I'm following on how the pipeline extraction mixed with this public record would help? But like mentioned I will ask the client about the usage of those public record.

@k-allagbe k-allagbe moved this from In review to Blocked in FertiScan Sep 18, 2024
@SamuelPelletierEvraire
Copy link
Contributor

SamuelPelletierEvraire commented Sep 26, 2024

From my understanding of this exemple, there seems to be multiple number for each ingredients (see the blue section)? image

The sections in blue of the previous image are not ingredient numbers but product codes. We do not manage product codes.

The red section is the registration number of the fertilizer

Yes it is the registration number of the fertilizer.

@Samuelpelletier3124 Gave us another use case: image

I see/understand the following based on the other use case:

  • 2022739S is the ingredient registration number as we currently call it. It can also be found in the following list
  • I do not know which ingredient this registration number refers to and wonder how the pipeline would do on this case. I will ask the client for clarification on the data side to better understand what are the stakes of that data and what it implies.

image

Here is the clarification provided by the client:

"The number you see here is a fertilizer registration number. Let me explain. There are two scenarios.

Scenario 1:

  1. A fertilizer is sold to a farmer.
  2. The farmer then applies the fertilizer in his field.

Scenario 2:

  1. A fertilizer is sold to a company.
  2. The company then uses this fertilizer in its own fertilizer recipe.
  3. The fertilizer used in the recipe then becomes an ""ingredient.""
  4. The company then places the registration number of the used fertilizer on the label."

So, the registration number does not refer to any ingredient in the list identified by the blue quotes; it refers to the fertilizer as an ingredient. In the example shown in the photo below, the company simply decided to add more information than necessary.

@SamuelPelletierEvraire
Copy link
Contributor

My vision was for a more complete information relationship in our database. I for sure will bring it up with the next client meeting.

I'm not 100% sure I'm following on how the pipeline extraction mixed with this public record would help? But like mentioned I will ask the client about the usage of those public record.

For now, it is not necessary to address this because we have set the idea aside. Our mandate is not to detect data but only to extract it, and using this list would not provide any new information to our data as for now.

@Francois-Werbrouck
Copy link

Francois-Werbrouck commented Sep 26, 2024

Can we have an exemple of scenario 2 @SamuelPelletierEvraire ?

@Francois-Werbrouck
Copy link

So there aren't any registration number for ingredients on a label.

If a fertilizer is used as an ingredient for the inspected fertilizer, then it is expected to see the registration number of the used fertilizer.

So, the registration number does not refer to any ingredient in the list identified by the blue quotes; it refers to the fertilizer as an ingredient.

I think this kinda closes the case of adding registration number in the ingredient field for now. We should circle back in our architecture to allow the newly clarified relationship.

Also this is kinda false:

Our mandate is not to detect data but only to extract it,

Yes our mandate is to extract data, but we also have the responsability of storing it in a good way based on the client needs (which includes efficient research within our record).
Linking the digitalized data to real known data can become immensly usefull for the task of fetching records in the long run. This avenue might become the solution for parsing data in #43

@k-allagbe
Copy link
Member

I think this kinda closes the case of adding registration number in the ingredient field for now. We should circle back in our architecture to allow the newly clarified relationship.

Seems to be the case. I still suggest you (everyone) watch the meeting video to clear out uncertainties. Then we'll move on.

@k-allagbe k-allagbe changed the title The registration number field is missing from Nutrient As a user, I want all registration numbers from fertilizer labels extracted with their type Oct 28, 2024
@k-allagbe k-allagbe changed the title As a user, I want all registration numbers from fertilizer labels extracted with their type As a user, I want all registration numbers extracted with their type Oct 28, 2024
@k-allagbe
Copy link
Member

Requirements updated. Please share your thoughts.

@k-allagbe k-allagbe moved this from Paused to Todo in FertiScan Oct 28, 2024
@Francois-Werbrouck
Copy link

Extract all registration numbers listed on fertilizer labels.

Perfectly said, there might be multiple registration number

Identify whether each registration number belongs to the fertilizer or to its ingredients.

It's more where it's located (in the ingredient list or not?)? Or what it represents (this fertilizer, or an ingredient of this fertilizer) ?
I have a hard time putting into word how to define the difference of 2 registration number : [type, location, defines, represents] 

Additional Information (Optional)
If there are multiple registration numbers, they [should all] be ingredient registration numbers as it should not be possible to have both fertilizer registration number and ingredient registration number on the same label.

Technically true, However, I have a hard time thinking of away to put this into JSON?

Example #1 list of Reg Number

{
  ...
  "fertiliser_name": "Name",
  "registration_number": [
 {
 "number": "reg number 1",
"Some classifier key": "Ingredient or Fertilizer/self"
}, {
 "number": "reg number 2",
"Some classifier key": "Ingredient or Fertilizer/self"
},

...

Example #2 Multiple Reg Number

{
  ...
  "fertiliser_name": "Name",
  "registration_number": "Placement for Reg number representing the fertilizer analyzed"
  "lot_number": "something",
  "weight": [
...
    "ingredients_en": [
    {
      "nutrient": "some ingredient",
      "quantity": "some value",
      "registration_number": null
    },
    {
      "nutrient": "registered fertilizer name without a quantity",
      "quantity": null,  
      "registration_number": "a registration number"
    },
    {
      "nutrient": "registered fertilizer name with quantity",
      "quantity": "some value",  
      "registration_number": "another registration number"
    }
  ],
  "ingredients_fr": [
...

In my opinion we should direct ourselves toward the first exemple. It will reduce the amount of work required to implement these changes and wont "need" a classification from the model until it is ready (the classifcation could be set as a default value until we have more dataset with Registration number in the ingredients)

@k-allagbe
Copy link
Member

A compilation of examples should help in the design. @SamuelPelletierEvraire is making sure we get those.

@k-allagbe k-allagbe moved this from Todo to In progress in FertiScan Oct 31, 2024
@snakedye snakedye linked a pull request Nov 4, 2024 that will close this issue
@k-allagbe k-allagbe moved this from In progress to In review in FertiScan Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In review
4 participants