Merge remote-tracking branch 'upstream/master'

.github/workflows/docstests.yml

@@ -0,0 +1,28 @@
+name: Docs Build
+
+on:
+    push:
+        branches:
+            - master
+        paths:
+            - 'docs_website/**/*'
+    pull_request:
+        paths:
+            - 'docs_website/**/*'
+
+jobs:
+    test-docs-build:
+        name: Test Docs Build
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v3
+            - uses: actions/setup-node@v3
+              with:
+                  node-version: 18
+                  cache: yarn
+            - name: Change to docs_website directory
+              run: cd docs_website
+            - name: Install dependencies
+              run: yarn install --frozen-lockfile
+            - name: Test build website
+              run: yarn build

CONTRIBUTING.md

@@ -1,3 +1,3 @@
 ## Thanks for taking the time to contribute!
 
-Please checkout the [guide on contribution](docs_website/docs/developer_guide/contributing.md) before you start.
+Please checkout the [guide on contribution](docs_website/docs/developer_guide/contributing.mdx) before you start.

README.md

@@ -26,12 +26,12 @@ Please install Docker before trying out Querybook.
 
 Pull this repo and run `make`. Visit https://localhost:10001 when the build completes.
 
-For more details on installation, [click here](docs_website/docs/setup_guide/overview.md)
+For more details on installation, [click here](docs_website/docs/setup_guide/overview.mdx)
 
 ## Configuration
 
-For infrastructure configuration, [click here](docs_website/docs/configurations/infra_config.md)
-For general configuration, [click here](docs_website/docs/configurations/general_config.md)
+For infrastructure configuration, [click here](docs_website/docs/configurations/infra_config.mdx)
+For general configuration, [click here](docs_website/docs/configurations/general_config.mdx)
 
 ## Supported Integrations
 

docs_website/Dockerfile

@@ -1,4 +1,4 @@
-FROM node:16
+FROM node:18
 
 WORKDIR /opt/docs_website
 COPY docs_website/package.json docs_website/yarn.lock /opt/docs_website/

docs_website/docs/changelog/2023-09-20.md → docs_website/docs/changelog/2023-09-20.mdx

@@ -26,11 +26,11 @@ The LLM powered AI assistant can help on
 -   Text to SQL
 -   Query error auto fix
 
-Please check the [guide](../user_guide/ai_assistant.md) for more details.
+Please check the [guide](../user_guide/ai_assistant.mdx) for more details.
 
 ### Vector Table Search
 
-Previously table searching is only keyword based search. Now we introduced [vector store plugin](../integrations/add_ai_assistant.md#vector-store-plugin) and added the support of searching a table by natural language.
+Previously table searching is only keyword based search. Now we introduced [vector store plugin](../integrations/add_ai_assistant.mdx#vector-store-plugin) and added the support of searching a table by natural language.
 ![](/img/user_guide/table_vector_search.png)
 
 ### Data Cell & Table Comment
@@ -67,12 +67,12 @@ Note: it can only be synced from metastore.
 
 ### Stats logging
 
-Add support of stats logging, like number of users, number of api requests and etc. Please check the [plugin](../integrations/add_stats_logger.md) for more details.
+Add support of stats logging, like number of users, number of api requests and etc. Please check the [plugin](../integrations/add_stats_logger.mdx) for more details.
 
 ## Small Feature Improvements/Bug Fixes
 
 -   Add username and password authentication for the trino client [#1315](https://github.com/pinterest/querybook/pull/1315)
--   Add two new plugins: [monkey patch plugin](../integrations/plugins.md#monkey-patch-plugin) and [api plugin](../integrations/plugins.md#api-plugin) [#1266](https://github.com/pinterest/querybook/pull/1266)
+-   Add two new plugins: [monkey patch plugin](../integrations/plugins.mdx#monkey-patch-plugin) and [api plugin](../integrations/plugins.mdx#api-plugin) [#1266](https://github.com/pinterest/querybook/pull/1266)
 -   Fix the display of long table names in search modal [#1246](https://github.com/pinterest/querybook/pull/1246)
 -   Allow data doc deletion from sidebar [#1241](https://github.com/pinterest/querybook/pull/1241)
 -   Ensure meta_info is updated when an exception occurs [#1230](https://github.com/pinterest/querybook/pull/1230)

docs_website/docs/changelog/breaking_change.md → docs_website/docs/changelog/breaking_change.mdx

@@ -7,12 +7,22 @@ slug: /changelog
 
 Here are the list of breaking changes that you should be aware of when updating Querybook:
 
+## v3.29.0
+
+Made below changes for `S3BaseExporter` (csv table uploader feature):
+
+-   Both `s3_path` and `use_schema_location` are optional now
+-   If none is provided, or `use_schema_location=False`, the table will be created as managed table, whose location will be determined by the query engine.
+-   Previously `use_schema_location=True` will create managed table, and now it creates external table.
+
 ## v3.27.0
+
 Updated properties of `QueryValidationResult` object. `line` and `ch` are replaced with `start_line` and `start_ch` respectively.
 
 ## v3.22.0
 
 Updated the charset of table `data_element` to `utf8mb4`. For those mysql db's default charset is not utf8, please run below sql to update it if needed.
+
 ```sql
 ALTER TABLE data_element CONVERT TO CHARACTER SET utf8mb4
 ```
@@ -23,20 +33,20 @@ Added two new abstract methods `notify_recipients` and `notifier_help` for `Base
 
 ## v3.6.0
 
-Added a new search index for list search (or boards search internally). Follow the instructions at [Re-Initialize ElasticSearch](../developer_guide/reinitialize_es.md) to recreate ElasticSearch indices.
+Added a new search index for list search (or boards search internally). Follow the instructions at [Re-Initialize ElasticSearch](../developer_guide/reinitialize_es.mdx) to recreate ElasticSearch indices.
 
 ## v3.5.0
 
 The ElasticSearch index mappings for `query_cells` and `query_executions` were updated to include fields to restrict access to queries based on datadoc access permissions. Query cells and executions on private datadocs will only be shown in query search to users that have access to these datadocs.
 
-Follow the instructions at [Re-Initialize ElasticSearch](../developer_guide/reinitialize_es.md) to recreate ElasticSearch indices.
+Follow the instructions at [Re-Initialize ElasticSearch](../developer_guide/reinitialize_es.mdx) to recreate ElasticSearch indices.
 
 Alternatively, you can run the script `querybook/server/scripts/es_3_5_0_upgrade.py` to update only the added fields in the query indices.
 
 ## v3.4.0
 
 There is a new field `feature_params` added to the `query_engine` table to replace `status_checker`.
-Please run [db migrations](../developer_guide/run_db_migration.md) for query engines to continue to work.
+Please run [db migrations](../developer_guide/run_db_migration.mdx) for query engines to continue to work.
 
 `_perform_check` in EngineStatusChecker is now deprecated. Override `perform_check_with_executor` for custom
 health check logic.
@@ -69,7 +79,7 @@ Query cells and query executions are now added as new indices in ElasticSearch.
 
 Depending on your deployment of ElasticSearch, you might need to manually re-initialize the ElasticSearch to create the new query indices and bulk insert existing query cells & executions.
 
-Follow the instructions at [Re-Initialize ElasticSearch](../developer_guide/reinitialize_es.md) to ensure that ElasticSearch will be re-initialized in your deployment.
+Follow the instructions at [Re-Initialize ElasticSearch](../developer_guide/reinitialize_es.mdx) to ensure that ElasticSearch will be re-initialized in your deployment.
 
 ## v3.0.0
 
@@ -106,11 +116,11 @@ The following integrations will now require custom installation:
     -   AWS Based (requests-aws4auth)
 
 Even though these packages are removed from the default installation, these dependencies are
-quite easy to add back! Checkout the [Infra Installation Guide](../configurations/infra_installation.md) to learn how.
+quite easy to add back! Checkout the [Infra Installation Guide](../configurations/infra_installation.mdx) to learn how.
 
 ### ElasticSearch
 
-Depending on deployment of Querybook, re-initialization of indices in ElasticSearch 7 cluster might be needed. Follow the instructions in [Re-Initialize ElasticSearch](../developer_guide/reinitialize_es.md).
+Depending on deployment of Querybook, re-initialization of indices in ElasticSearch 7 cluster might be needed. Follow the instructions in [Re-Initialize ElasticSearch](../developer_guide/reinitialize_es.mdx).
 
 #### Make and Docker-compose
 

docs_website/docs/changelog/security_advisories.md → docs_website/docs/changelog/security_advisories.mdx

@@ -13,5 +13,5 @@ sidebar_label: Security Advisories
 | Summary           | Attackers could perform reflected XSS if Querybook authentication is set to `oauth_auth` or `okta_oauth`. |
 | CVE URL           | https://cve.report/CVE-2022-46151                                                                         |
 | GitHub URL        | https://github.com/pinterest/querybook/security/advisories/GHSA-mrrw-9wf7-xq6w                            |
-| Affected versions | <3.14.2                                                                                                   |
+| Affected versions | \<3.14.2                                                                                                  |
 | Patched versions  | 3.14.2                                                                                                    |

docs_website/docs/configurations/general_config.md → docs_website/docs/configurations/general_config.mdx

@@ -13,7 +13,7 @@ The first user that installs Querybook gets the "Admin" role. Admins can add oth
 In the next section we will go over different things that can be configured in the Admin tools.
 
 :::info
-Checkout [Sharing & Security](../overview/sharing_and_security.md) to learn how to configure access permission for these entities.
+Checkout [Sharing & Security](../overview/sharing_and_security.mdx) to learn how to configure access permission for these entities.
 :::
 
 ### Environment

docs_website/docs/configurations/infra_config.md → docs_website/docs/configurations/infra_config.mdx

@@ -7,7 +7,7 @@ sidebar_label: Infra Config
 ## Overview
 
 :::caution
-THIS GUIDE IS ONLY FOR INFRASTRUCTURE SETUP, PLEASE READ THE [GENERAL CONFIG](./general_config.md) TO CONFIGURE ENTITIES SUCH AS QUERY ENGINE & ACCESS PERMISSION.
+THIS GUIDE IS ONLY FOR INFRASTRUCTURE SETUP, PLEASE READ THE [GENERAL CONFIG](./general_config.mdx) TO CONFIGURE ENTITIES SUCH AS QUERY ENGINE & ACCESS PERMISSION.
 :::
 
 Eventhrough Querybook can be launched without any configuration, it is absolutely required for more powerful infrastructure/flexible customization. In this guide we will walkthrough different kind of environment settings you can set for Querybook. You can see all possible options and default values in this repo by checking out `querybook/config/querybook_default_config.yaml`.
@@ -90,7 +90,7 @@ The following settings are only relevant if you are using `s3` and your S3 bucke
     - console: This will print the event logs to the console. Could be used for debugging purpose.
     - db: This will save the event logs to table **event_logs** in querybook mysql db.
 
-You can also add addtional loggers in the event logger plugin. See [Add Event Logger guide](../integrations/add_event_logger.md) for more details.
+You can also add addtional loggers in the event logger plugin. See [Add Event Logger guide](../integrations/add_event_logger.mdx) for more details.
 
 ### Stats Logging
 
@@ -99,7 +99,7 @@ You can also add addtional loggers in the event logger plugin. See [Add Event Lo
     - null: This is the default logger, which does nothing and disregards the logs.
     - console: This will print the stats logs to the console. Could be used for debugging purpose.
 
-You need to add your own stats logger plugin to use it. See [Add Stats Logger guide](../integrations/add_stats_logger.md) for more details.
+You need to add your own stats logger plugin to use it. See [Add Stats Logger guide](../integrations/add_stats_logger.mdx) for more details.
 ## Authentication
 
 `AUTH_BACKEND` (optional, defaults to **app.auth.password_auth**): Python path to the authentication file. By default Querybook provides:

docs_website/docs/configurations/infra_installation.md → docs_website/docs/configurations/infra_installation.mdx

@@ -10,7 +10,7 @@ To add more integrations such as query engines, metastores, or authentications,
 install the Python packages yourself.
 
 Querybook comes with a set of integrations that can be auto included once the Python packages are installed.
-It also supports [plugins](../integrations/plugins.md) for you to add additional integrations that are not yet included.
+It also supports [plugins](../integrations/plugins.mdx) for you to add additional integrations that are not yet included.
 
 ## Out-of-box support
 
@@ -54,7 +54,7 @@ If you install the required packages, these integrations will be automatically s
     -   Redshift (via `-r engine/redshift.txt`)
     -   Snowflake (via `-r engine/snowflake.txt`)
     -   Trino (via `-r engine/trino.txt`)
-    -   And [any sqlalchemy supported engines](../setup_guide/connect_to_query_engines.md)
+    -   And [any sqlalchemy supported engines](../setup_guide/connect_to_a_query_engine.mdx)
 -   Metastore:
     -   Hive Metastore (via `-r metastore/hms.txt`)
     -   Hive Metastore with Thrift (install Hive Metastore and Hive)
@@ -126,7 +126,7 @@ docker push ...
 
 ### Install by extending the prod image
 
-This part is very similar to [plugin installation](../integrations/plugins.md#installing-plugins). Follow the steps to create a custom repo.
+This part is very similar to [plugin installation](../integrations/plugins.mdx#installing-plugins). Follow the steps to create a custom repo.
 Now add the requirements file somewhere, in this example we will put the requirements file called 'custom_deps.txt' at the plugins project root.
 In the requirements file, put the following:
 
@@ -146,4 +146,4 @@ RUN pip install -r /opt/querybook/requirements/custom_deps.txt
 
 ## Additional integrations
 
-If you need to integrate with anything that is not listed above, please refer to the [plugins guide](../integrations/plugins.md) to learn how to add them.
+If you need to integrate with anything that is not listed above, please refer to the [plugins guide](../integrations/plugins.mdx) to learn how to add them.

docs_website/docs/developer_guide/contributing.md → docs_website/docs/developer_guide/contributing.mdx

@@ -12,7 +12,7 @@ everyone.
 
 ## Code of Conduct
 
-Please be sure to read and understand our [CODE_OF_CONDUCT.md](https://github.com/pinterest/querybook/blob/master/CODE_OF_CONDUCT.md).
+Please be sure to read and understand our [CODE_OF_CONDUCT.mdx](https://github.com/pinterest/querybook/blob/master/CODE_OF_CONDUCT.mdx).
 We work hard to ensure that our projects are welcoming and inclusive to as many
 people as possible.
 
@@ -91,7 +91,7 @@ Here are all the prefix allowed by Querybook:
 
 ## Help
 
-Start by reading the developer starter guide [this guide](./developer_setup.md) to setup Querybook/
+Start by reading the developer starter guide [this guide](./developer_setup.mdx) to setup Querybook/
 If you're having trouble using this project, please go through the docs and search for solutions in the existing open and closed issues.
 
 You can also reach out to us at [email protected] or on our [Slack](https://join.slack.com/t/querybook/shared_invite/zt-se82lvld-yyzRIqvIASsyYozk7jMCYQ).

docs_website/docs/integrations/add_ai_assistant.md → docs_website/docs/integrations/add_ai_assistant.mdx

@@ -5,7 +5,7 @@ sidebar_label: AI Assistant
 ---
 
 :::info
-Please check the [user guide](../user_guide/ai_assistant.md) of how the AI assistant features look like.
+Please check the [user guide](../user_guide/ai_assistant.mdx) of how the AI assistant features look like.
 :::
 
 The AI assistant plugin is powered by LLM(Large Language Model), like ChatGPT from openai. We're using [Langchain](https://python.langchain.com/docs/get_started/introduction) to build the plugin.

docs_website/docs/integrations/add_auth.md → docs_website/docs/integrations/add_auth.mdx

@@ -17,7 +17,7 @@ The user/password authentication method is used by default but it cannot be inte
 
 ## Using an existing authentication
 
-We will go through how to setup authentication with OAuth and LDAP starting from the [devserver](../setup_guide/quick_setup.md). Before starting, you should checkout the required fields in [Infra Config](../configurations/infra_config.md#authentication). All fields that starts with OAUTH\_ are required for OAuth, and similarily for LDAP. For Google OAuth, only client id and client secrets are needed since other fields are provided by Google. If you need customized behavior for OAuth/LDAP, you can extend the class, see [below](#adding-a-new-authentication-method) for details.
+We will go through how to setup authentication with OAuth and LDAP starting from the [devserver](../setup_guide/quick_setup.mdx). Before starting, you should checkout the required fields in [Infra Config](../configurations/infra_config.mdx#authentication). All fields that starts with OAUTH\_ are required for OAuth, and similarily for LDAP. For Google OAuth, only client id and client secrets are needed since other fields are provided by Google. If you need customized behavior for OAuth/LDAP, you can extend the class, see [below](#adding-a-new-authentication-method) for details.
 
 ### OAuth
 
@@ -103,7 +103,7 @@ Restart the container with `docker restart querybook_web_1` and it should work.
 
 ## Adding A New Authentication Method
 
-To add a new authentication, add a new file under <project_root>/querybook/server/app/auth/ and name it <auth_method>\_auth.py. Fundamentally every auth file should export the following 3 things:
+To add a new authentication, add a new file under \<project_root>/querybook/server/app/auth/ and name it \<auth_method>\_auth.py. Fundamentally every auth file should export the following 3 things:
 
 -   login(request) (**required**): A function that logs in the user. One common implementation would be:
 
@@ -123,8 +123,6 @@ def init_app(app):
 
 Moving on to the login manager itself, you need to use the `QuerybookLoginManager` from auth/utils.py as the login manager as it provides api token access. Use a class that has this login manager as a field (check oauth_auth.py as an example) if you need to initialize resources when app starts or use only functions (check password_auth.py as an example) if nothing needs to be initialized.
 
-<!-- TODO: add examples of adding ldap auth and okta auth as 2 different authentication methods to implement. -->
-
 ## Plugins
 
-If the authentication method is too specific for your org or it has some post-authentication actions (such as adding users to a specific environment based on their permission), it is recommended to add the auth as part of the plugin. Please check the [Plugin Guide](plugins.md) to see how setup plugins. Once it is setup and running, simply put your auth code under auth_plugins/ (or any other folder, it is not an enforced restriction) and set AUTH_BACKEND='auth_plugin.<auth_module_name>' as the environment variable.
+If the authentication method is too specific for your org or it has some post-authentication actions (such as adding users to a specific environment based on their permission), it is recommended to add the auth as part of the plugin. Please check the [Plugin Guide](plugins.mdx) to see how setup plugins. Once it is setup and running, simply put your auth code under auth_plugins/ (or any other folder, it is not an enforced restriction) and set AUTH_BACKEND='auth_plugin.\<auth_module_name>' as the environment variable.