Complex xml structure

[hballesteross]

Hi there! I started using this library because I need to process big xml files, between a few megabytes up to 50GB.
I'm having troubles with one particular xml file that has a rare structure.
What I need to gather is the information about every "CPE". But the structure is grouped in different locations corresponding to "Distrito" then inside could have "concelho" and inside of it "freguesia" and inside "localidade". But the tricky part is that not every CPE is located inside the last node of the tree, because there are some ungrouped CPEs inside "concelho" that don't have freguesia/localidade info. I'll show the structure to visualize this:

<DISTRITO>
     <DISTRITO_NOME>DISTRICT NAME</DISTRITO_NOME>
     <CPE>123</CPE>  
     <CPE>456</CPE>
     <CPE>789</CPE>
         <CONCELHO>
             <CONCELHO_NOME>CONCELHO NAME</CONCELHO_NOME>
             <CPE>111</CPE>
             <CPE>222</CPE>
             <CPE>333</CPE>
                 <FREGESIA>
                     <FREGUESIA_NOME>FREGUESIA NAME</FREGUESIA_NOME>
                     <CPE>321</CPE>
                     <CPE>654</CPE>
                     <CPE>321</CPE>
                         <LOCALIDADE>
                             <LOCALIDADE_NOME>LOCALIDADE NAME</LOCALIDADE_NOME>
                             <CPE>9999</CPE>
                             <CPE>8888</CPE>
                             <CPE>7777</CPE>
                         </LOCALIDADE>
                 </FREGESIA>
         </CONCELHO>
 </DISTRITO>

This is more or less the structure where every child has 1 to N relations so for example one FREGESIA can have multiples LOCALIDADE.

The thing I need to achieve is process every CPE as an item and if they are in the last child like cpe 9999, I could get the parent names for his "distrito","concelho","freguesia","localidade" but for example, the first cpe "123" is located inside distrito but the info for concelho/freguesia/localidade is not available so it's located there and only has distrito info so the rest would be null.

I'm not sure of where to start first and which element to handle first.

Thanks in advance and I hope I was clear enough for you to understand this.

[Rogdham]

Hello, that's an interesting use case!

For a possible solution, let me assume that you wants your results to be returned as a dataclass instance. Let's use the following code to define the dataclass:

from dataclasses import dataclass
from typing import Optional

@dataclass
class Cpe:
    value: str
    distrito: Optional[str] = None
    concelho: Optional[str] = None
    fregesia: Optional[str] = None
    localidade: Optional[str] = None

Note: I'm not sure if the freguesia VS fregesia spellings are typos or not, the code here works with your provided XML where there is FREGESIA and FREGUESIA_NOME tags.

Now what I will do is to create a generic handler that will handle nodes of name DISTRITO/CONCELHO/FREGESIA/LOCALIDADE.

When a CPE subnode is handled, it will yield a Cpe instance with only the value field being set.

And when the “nome” nodes (DISTRITO_NOME, CONCELHO_NOME, FREGUESIA_NOME, LOCALIDADE_NOME) are encountered, they will update a local nome variable. This variable is later used to update the appropriate attribute of the Cpe item.

The code looks like this:

from bigxml import xml_handle_element

@xml_handle_element("CPE")
def handle_cpe(node):
    yield Cpe(node.text)


@xml_handle_element("DISTRITO")
@xml_handle_element("CONCELHO")
@xml_handle_element("FREGESIA")
@xml_handle_element("LOCALIDADE")
def handler(node):
    nome = None

    for item in node.iter_from(
        # the following is a shortcut to say that we want item to be a node of that name
        # https://bigxml.rogdham.net/handlers/#syntactic-sugar
        "FREGUESIA_NOME" if node.name == "FREGESIA" else f"{node.name}_NOME",
        
        # if we see a CPE tag, item will be a Cpe instance
        handle_cpe,
        
        # recursive call in which we only will get Cpe instance for item
        handler,
    ):
        if isinstance(item, Cpe):
            # Cpe instance, coming from whatever level of recursion
            setattr(item, node.name.lower(), nome)
            yield item

        else:
            # "nome" node
            nome = item.text

And finaly, let's parse the XML, for example like so:

from bigxml import Parser

with open("filename.xml", "rb") as f:
    for cpe in Parser(f).iter_from(handler):
        print(cpe)

It outputs the following:

Cpe(value='123', distrito='DISTRICT NAME', concelho=None, fregesia=None, localidade=None)
Cpe(value='456', distrito='DISTRICT NAME', concelho=None, fregesia=None, localidade=None)
Cpe(value='789', distrito='DISTRICT NAME', concelho=None, fregesia=None, localidade=None)
Cpe(value='111', distrito='DISTRICT NAME', concelho='CONCELHO NAME', fregesia=None, localidade=None)
Cpe(value='222', distrito='DISTRICT NAME', concelho='CONCELHO NAME', fregesia=None, localidade=None)
Cpe(value='333', distrito='DISTRICT NAME', concelho='CONCELHO NAME', fregesia=None, localidade=None)
Cpe(value='321', distrito='DISTRICT NAME', concelho='CONCELHO NAME', fregesia='FREGUESIA NAME', localidade=None)
Cpe(value='654', distrito='DISTRICT NAME', concelho='CONCELHO NAME', fregesia='FREGUESIA NAME', localidade=None)
Cpe(value='321', distrito='DISTRICT NAME', concelho='CONCELHO NAME', fregesia='FREGUESIA NAME', localidade=None)
Cpe(value='9999', distrito='DISTRICT NAME', concelho='CONCELHO NAME', fregesia='FREGUESIA NAME', localidade='LOCALIDADE NAME')
Cpe(value='8888', distrito='DISTRICT NAME', concelho='CONCELHO NAME', fregesia='FREGUESIA NAME', localidade='LOCALIDADE NAME')
Cpe(value='7777', distrito='DISTRICT NAME', concelho='CONCELHO NAME', fregesia='FREGUESIA NAME', localidade='LOCALIDADE NAME')

Of course you are then free to do whatever you want with the Cpe instances.

---

Alternatively, we could go with a more descriptive approach using classes, like the following (click below to see the code). There is quite some duplicate pieces of code but sometimes explicit is more easy to understand than clever.

Click here to show alternative code

from bigxml import xml_handle_element, Parser
from dataclasses import dataclass

from typing import Optional


@dataclass
class Cpe:
    value: str
    distrito: Optional[str] = None
    concelho: Optional[str] = None
    fregesia: Optional[str] = None
    localidade: Optional[str] = None


@xml_handle_element("LOCALIDADE")
class Localidade:
    nome = None

    @xml_handle_element("LOCALIDADE_NOME")
    def handle_nome(self, node):
        self.nome = node.text

    @xml_handle_element("CPE")
    def handle_cpe(self, node):
        yield Cpe(node.text)

    def xml_handler(self, iterator):
        for cpe in iterator:
            cpe.localidade = self.nome
            yield cpe


@xml_handle_element("FREGESIA")
class Fregesia:
    nome = None

    @xml_handle_element("FREGUESIA_NOME")
    def handle_nome(self, node):
        self.nome = node.text

    @xml_handle_element("CPE")
    def handle_cpe(self, node):
        yield Cpe(node.text)

    handle_localidade = Localidade

    def xml_handler(self, iterator):
        for cpe in iterator:
            cpe.fregesia = self.nome
            yield cpe


@xml_handle_element("CONCELHO")
class Concelho:
    nome = None

    @xml_handle_element("CONCELHO_NOME")
    def handle_nome(self, node):
        self.nome = node.text

    @xml_handle_element("CPE")
    def handle_cpe(self, node):
        yield Cpe(node.text)

    handle_fregesia = Fregesia

    def xml_handler(self, iterator):
        for cpe in iterator:
            cpe.concelho = self.nome
            yield cpe


@xml_handle_element("DISTRITO")
class Distrito:
    nome = None

    @xml_handle_element("DISTRITO_NOME")
    def handle_nome(self, node):
        self.nome = node.text

    @xml_handle_element("CPE")
    def handle_cpe(self, node):
        yield Cpe(node.text)

    handle_concelho = Concelho

    def xml_handler(self, iterator):
        for cpe in iterator:
            cpe.distrito = self.nome
            yield cpe


with open("filename.xml", "rb") as f:
    for cpe in Parser(f).iter_from(handler):
        print(cpe)

I hope those examples give you the right pointers to move forward, tell me what you think!

[hballesteross]

Thanks for the reply!

First thanks for your time and providing this code with explanations and examples.

About the FREGUESIA vs FREGESIA, yes it was a typo, my bad.

I was trying to make the same aproach as you with clases but I was failing with the iteration part, I don't get it yet how it works.

I tried the first code you shared with my file, but the only value I get is the cpe number inside value, but the names of the distrito/concelho/fregesia/localidade not.

To give you more clarification about the names, these are territory subdivisions in Portugal. So the country is divided in Distritos, where every < DISTRITO > has a tag inside with the <DISTRITO_NOME> and inside that you have different CONCELHO with its name inside in <CONCELHO_NOME> etc. Inside every subdivision, you have a bunch of < CPE> with their values. You got right the idea of the entity called CPE with the data inside the class. That's what I wast trying to do but I was failing with the recursive part.

I tested the code with a 13MB file with 347824 lines of code where most of them are CPE except for the region tags but only worked with 464 cpes so there is something missing with the recursion.

Finally I tried the second part of the code but the Parser(f).item_from(handler) part calls for handler and it's not set above or I'm missing something.

Thanks again and I hope I gave you some clarification with the issue so we can try to find a solution.

[hballesteross]

UPDATE:

I fixed the TYPO (the correct one is FREGUESIA) and now the recursive is fixed and i'm processing the entire file but I'm not getting the names of every distrito/concelho/freguesia/localidade inside the class. It only takes the CPE value.

[hballesteross]

UPDATE2:

ITS WORKING!

I made another typo with the _NOME tags... the tag is NOME_DISTRITO, and not DISTRITO_NOME and the same with the rest regions... that was why it wasn't taking the names of each region! My bad again, I'm burnt with this project.

So I changed the code to get f"NOME_{node.name}" and now it works!

So your code was actually fine since the begining with the XML example I gave you but because of the typos it wasn't working on my original XML.

Thanks you again!