Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_external_url is bork #25

Open
PietroPasotti opened this issue Aug 21, 2024 · 2 comments
Open

_external_url is bork #25

PietroPasotti opened this issue Aug 21, 2024 · 2 comments

Comments

@PietroPasotti
Copy link
Contributor

PietroPasotti commented Aug 21, 2024

Bug Description

tempo coordinator errors out on relation to prometheus if s3 isn't integrated, because of an issue on how the external url is calculated.

To Reproduce

juju deploy cos-lite --channel edge
juju deploy tempo-coordinator-k8s --channel edge tempo
juju relate tempo prometheus

Environment

microk8s on top of openstack on top of multipass

(good luck)

Relevant log output

unit-tempo-0: 12:48:47 WARNING unit.tempo/0.juju-log metrics-endpoint:38: 'source_url' should start with a scheme, such as 'http://'. Assuming 'http:/
/' since none is present.                                                                                                                             
unit-tempo-0: 12:48:47 ERROR unit.tempo/0.juju-log metrics-endpoint:38: Uncaught exception while in charm code:                                       
Traceback (most recent call last):                                                                                                                    
  File "/var/lib/juju/agents/unit-tempo-0/charm/./src/charm.py", line 390, in <module>                                                                
    main(TempoCoordinatorCharm)                                                                                                                       
  File "/var/lib/juju/agents/unit-tempo-0/charm/venv/ops/main.py", line 551, in main                                                                  
    manager.run()                                                                                                                                     
  File "/var/lib/juju/agents/unit-tempo-0/charm/venv/ops/main.py", line 530, in run                                                                   
    self._emit()                                                                                                                                      
  File "/var/lib/juju/agents/unit-tempo-0/charm/venv/ops/main.py", line 519, in _emit                                                                 
    _emit_charm_event(self.charm, self.dispatcher.event_name)                                                                                         
  File "/var/lib/juju/agents/unit-tempo-0/charm/venv/ops/main.py", line 147, in _emit_charm_event                                                     
    event_to_emit.emit(*args, **kwargs)                                                                                                               
  File "/var/lib/juju/agents/unit-tempo-0/charm/venv/ops/framework.py", line 348, in emit                                                             
    framework._emit(event)                                                                                                                            
  File "/var/lib/juju/agents/unit-tempo-0/charm/venv/ops/framework.py", line 860, in _emit                                                            
    self._reemit(event_path)                                                                                                                          
  File "/var/lib/juju/agents/unit-tempo-0/charm/venv/ops/framework.py", line 950, in _reemit                                                          
    custom_handler(event)                                                                                                                             
  File "/var/lib/juju/agents/unit-tempo-0/charm/lib/charms/prometheus_k8s/v0/prometheus_scrape.py", line 1527, in set_scrape_job_spec                 
    self._set_unit_ip()                                                                                                                               
  File "/var/lib/juju/agents/unit-tempo-0/charm/lib/charms/prometheus_k8s/v0/prometheus_scrape.py", line 1572, in _set_unit_ip                        
    relation.data[self._charm.unit]["prometheus_scrape_unit_address"] = unit_address                                                                  
  File "/var/lib/juju/agents/unit-tempo-0/charm/venv/ops/model.py", line 1792, in __setitem__                                                         
    self._validate_write(key, value)                                                                                                                  
  File "/var/lib/juju/agents/unit-tempo-0/charm/venv/ops/model.py", line 1760, in _validate_write                                                     
    raise RelationDataTypeError(f'relation data values must be strings, not {type(value)}')                                                           
ops.model.RelationDataTypeError: relation data values must be strings, not <class 'NoneType'>                                                         
unit-tempo-0: 12:48:47 ERROR juju.worker.uniter.operation hook "metrics-endpoint-relation-joined" (via hook dispatching script: dispatch) failed: exit
 status 1                                                                                                                                             
unit-tempo-0: 12:48:47 INFO juju.worker.uniter awaiting error resolution for "relation-joined" hook

Additional context

No response

@mmkay
Copy link
Contributor

mmkay commented Aug 21, 2024

It's easier to see even without the whole setup-on-setup:

  1. deploy cos-lite first, then deploy tempo-coordinator and tempo-worker
  2. jhack imatrix fill
  3. no profit

It works if you deploy coordinator and worker first and relate them, then add cos-lite and jhack imatrix fill.

It might be an issue with the shared coordinator-worker object.

@PietroPasotti
Copy link
Contributor Author

yes, the underlying issue is that tempo will not set up ingress until s3 is related and configured, and the _external_url property makes some incorrect assumptions about what TraefikRouteRequirer.is_ready means.

This should fix it:

    @property
    def _external_url(self) -> str:
        """Return the external url."""
        # traefik-route's is_ready() doesn't mean that there is data in the databags, hence the explicit checks.
        if self.ingress.is_ready() and self.ingress.scheme and self.ingress.external_host:
            ingress_url = f"{self.ingress.scheme}://{self.ingress.external_host}"

@PietroPasotti PietroPasotti changed the title tempo integration with prometheus is bork _external_url is bork Aug 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants