Add Tinybird query runner #5616

pandomic · 2021-10-09T23:27:20Z

What type of PR is this? (check all applicable)

Description

Tinybird is a fast-growing service providing real-time analytics for large amounts of data. Integrating it with powerful Redash tools will open new opportunities for real-time data analysis and visualizations, especially when mixed with other data sources like Python.

Under the hood, Tinybird uses ClickHouse, which makes it possible to re-use existing query runner with minimal adjustments.

Implementation Details

Connection Testing

Tinybird has few system data sources that should become queriable once the connection is properly configured. We use one of those (tinybird.pipe_stats) to test whether it can be executed or not.

Data Querying

There are different ways how you may run SQL queries against Tinybird. The most flexible one is of course the /sql endpoint which does not enforce you to use a single data source or pipe.

Schema Fetching

Schema for pipes and data sources can not be easily observed through SQL queries. Thus you may notice a little trick here: we are fetching lists of pipes and data sources and then running a simple query on each of them. Result schema is then returned as meta attribute. There is one limitation though, for pipes schema can only be observed if it is exported as an endpoint (because you are restricted to run queries only against shared pipes).

Related Tickets & Documents

Proactive initiative

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

The following Loom recording shows the Tinybird query runner at work
https://www.loom.com/share/8d96d361e2534de09710216df528af44

guidopetri · 2023-08-20T22:36:00Z

@pandomic , thanks for the PR! We've updated a lot of things now that we're Community-driven so - if you're still interested in getting this merged - would you mind rebasing off master to re-run the CI?

We're trying to clean up our PR todo list, so if you're not interested, that's fine - we'll close the PR in about a week if we don't hear back. If you're interested in reopening the PR afterwards, we would also very much welcome that.

pandomic · 2023-08-21T14:38:31Z

Hi @guidopetri, thanks for the update, I will try to align the PR in the next following days 🙌

pandomic · 2023-08-28T17:02:22Z

Hey @guidopetri, I just rebased the PR. Hope it looks good. Looking forward to getting Tinybird support 🙌

justinclift · 2023-08-29T00:30:14Z

@pandomic Looks like it's about 99.95% done, but still not quite finished. It's failing the formatting check lint test:

Run black --check .
would reformat /home/runner/work/redash/redash/redash/query_runner/tinybird.py

We have instructions on how to get that fixed if that's useful?

https://github.com/getredash/redash/wiki/Local-development-setup

It's the make format command there. 😄

codecov · 2023-08-29T19:39:37Z

Codecov Report

Merging #5616 (82c2998) into master (0258dca) will increase coverage by 0.20%.
Report is 19 commits behind head on master.
The diff coverage is 72.53%.

❗ Current head 82c2998 differs from pull request most recent head 4e46071. Consider uploading reports for the commit 4e46071 to get more accurate results

@@            Coverage Diff             @@
##           master    #5616      +/-   ##
==========================================
+ Coverage   60.77%   60.97%   +0.20%     
==========================================
  Files         154      156       +2     
  Lines       12656    12780     +124     
  Branches     1721     1735      +14     
==========================================
+ Hits         7692     7793     +101     
- Misses       4739     4753      +14     
- Partials      225      234       +9

Files Changed	Coverage Δ
redash/handlers/queries.py	`86.17% <0.00%> (ø)`
redash/settings/__init__.py	`98.65% <ø> (ø)`
redash/query_runner/google_spreadsheets.py	`67.64% <65.85%> (+0.50%)`	⬆️
redash/query_runner/tinybird.py	`70.96% <70.96%> (ø)`
redash/destinations/asana.py	`81.48% <81.48%> (ø)`
redash/models/__init__.py	`92.25% <83.33%> (+0.29%)`	⬆️
redash/__init__.py	`88.57% <100.00%> (ø)`
redash/query_runner/elasticsearch2.py	`59.89% <100.00%> (ø)`
redash/query_runner/snowflake.py	`29.06% <100.00%> (+0.83%)`	⬆️

... and 2 files with indirect coverage changes

guidopetri · 2023-08-31T00:29:32Z

@pandomic thanks for your patience and for updating the PR! Any chance you could add some tests to make the Codecov bot happy?

guidopetri

@pandomic thanks for adding these tests. I left a few comments but this looks good to me. I'll approve and merge tomorrow unless you want to change anything in response to the feedback.

guidopetri · 2023-09-04T18:55:57Z

redash/query_runner/tinybird.py

+    def test_connection(self):
+        try:
+            self._send_query("SELECT count() FROM tinybird.pipe_stats LIMIT 1 FORMAT JSON")
+            return True
+        except Exception:
+            return False


I think if we set Tinybird.noop_query this should work automatically and we don't have to override the base class implementation.

guidopetri · 2023-09-04T18:56:43Z

redash/query_runner/tinybird.py

+    @classmethod
+    def name(cls):
+        return "Tinybird"
+
+    @classmethod
+    def type(cls):
+        return "tinybird"


Same here, I don't think we have to override this.

Makes sense 👍 Slightly adjusted the ClickHouse class to re-use the underlying type() method

guidopetri · 2023-09-04T18:58:14Z

redash/query_runner/tinybird.py

+        url = endpoint % self.configuration.get("url", self.DEFAULT_URL)
+        authorization = "Bearer %s" % self.configuration.get("token")


In general, I think we should lean on f-strings or .format() - I believe it's safer than using string interpolation with %s

redash/query_runner/tinybird.py

guidopetri · 2023-09-04T19:07:05Z

tests/query_runner/test_tinybird.py

+
+class TestTinybird(TestCase):
+    @patch("requests.get")
+    def test_get_schema(self, get_request):


Both the test cases here are testing downstream functionality (i.e. get_schema and run_query, that are not directly implemented in the Tinybird runner). One potential improvement would be to test the Tinybird query runner directly, since it uses the base implementation for these methods.

👍 I adjusted the test name to reflect what is happening in get_schema (it collects the schema from available pipes and datasources).

I am not sure what else to test when it comes to query execution though, as it's essentially just a tiny GET request which is already being tested for proper formatting and with mocked response handling 🤔

Co-authored-by: Thomas Rausch <[email protected]> Co-authored-by: Vlad Gramuzov <[email protected]>"

justinclift · 2023-09-06T10:56:59Z

Looks like it's passing all of the tests. Weirdly though, the Code coverage tests don't seem to have automatically-rerun. Not sure why. 🤔

guidopetri

Thanks again for your contribution :) I retriggered the CI but it should be passing.

guidopetri added the 1-week-review label Aug 20, 2023

guidopetri removed the 1-week-review label Aug 21, 2023

pandomic force-pushed the feat/tinybird-query-runner branch from 0be05c9 to 7d8a80b Compare August 28, 2023 15:55

guidopetri self-assigned this Aug 29, 2023

pandomic force-pushed the feat/tinybird-query-runner branch from 7d8a80b to 5f8aaac Compare August 29, 2023 14:47

justinclift added the Team Review Meets PR criteria, ready for full review label Aug 29, 2023

pandomic force-pushed the feat/tinybird-query-runner branch from 3de96c4 to 512ed91 Compare September 4, 2023 11:31

guidopetri approved these changes Sep 4, 2023

View reviewed changes

Add Tinybird query runner

4e46071

Co-authored-by: Thomas Rausch <[email protected]> Co-authored-by: Vlad Gramuzov <[email protected]>"

pandomic force-pushed the feat/tinybird-query-runner branch from 82c2998 to 4e46071 Compare September 6, 2023 09:51

guidopetri approved these changes Sep 6, 2023

View reviewed changes

guidopetri enabled auto-merge (squash) September 6, 2023 12:23

guidopetri merged commit cb4af6d into getredash:master Sep 6, 2023
14 checks passed

pandomic deleted the feat/tinybird-query-runner branch September 6, 2023 14:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Tinybird query runner #5616

Add Tinybird query runner #5616

pandomic commented Oct 9, 2021 •

edited

Loading

guidopetri commented Aug 20, 2023

pandomic commented Aug 21, 2023

pandomic commented Aug 28, 2023

justinclift commented Aug 29, 2023 •

edited

Loading

codecov bot commented Aug 29, 2023 •

edited

Loading

guidopetri commented Aug 31, 2023

guidopetri left a comment

guidopetri Sep 4, 2023

guidopetri Sep 4, 2023

pandomic Sep 6, 2023 •

edited

Loading

guidopetri Sep 4, 2023

guidopetri Sep 4, 2023

pandomic Sep 6, 2023

justinclift commented Sep 6, 2023

guidopetri left a comment

		url = endpoint % self.configuration.get("url", self.DEFAULT_URL)
		authorization = "Bearer %s" % self.configuration.get("token")

Add Tinybird query runner #5616

Add Tinybird query runner #5616

Conversation

pandomic commented Oct 9, 2021 • edited Loading

What type of PR is this? (check all applicable)

Description

Implementation Details

Connection Testing

Data Querying

Schema Fetching

Related Tickets & Documents

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

guidopetri commented Aug 20, 2023

pandomic commented Aug 21, 2023

pandomic commented Aug 28, 2023

justinclift commented Aug 29, 2023 • edited Loading

codecov bot commented Aug 29, 2023 • edited Loading

Codecov Report

guidopetri commented Aug 31, 2023

guidopetri left a comment

Choose a reason for hiding this comment

guidopetri Sep 4, 2023

Choose a reason for hiding this comment

guidopetri Sep 4, 2023

Choose a reason for hiding this comment

pandomic Sep 6, 2023 • edited Loading

Choose a reason for hiding this comment

guidopetri Sep 4, 2023

Choose a reason for hiding this comment

guidopetri Sep 4, 2023

Choose a reason for hiding this comment

pandomic Sep 6, 2023

Choose a reason for hiding this comment

justinclift commented Sep 6, 2023

guidopetri left a comment

Choose a reason for hiding this comment

pandomic commented Oct 9, 2021 •

edited

Loading

justinclift commented Aug 29, 2023 •

edited

Loading

codecov bot commented Aug 29, 2023 •

edited

Loading

pandomic Sep 6, 2023 •

edited

Loading