From 1b1557cc9c24951acd27ec5ef3c1bcb56e578c1b Mon Sep 17 00:00:00 2001 From: mecrouch Date: Fri, 22 Nov 2024 15:12:12 -0500 Subject: [PATCH 1/2] Add list of available and recommended packages --- docs/datasets/transform-dataset.md | 144 ++++++++++++++++++++++++++--- mkdocs.yml | 1 + 2 files changed, 134 insertions(+), 11 deletions(-) diff --git a/docs/datasets/transform-dataset.md b/docs/datasets/transform-dataset.md index b31b15a..585b0e3 100644 --- a/docs/datasets/transform-dataset.md +++ b/docs/datasets/transform-dataset.md @@ -56,7 +56,7 @@ In a workflow, the Transform dataset operator takes one or more datasets or simu You can choose any step in your transformation process as the thumbnail preview. -
![Transform dataset that takes two datasets as inputs and displays choropleth comparison maps](../img/data/transform-operator.png)
**How it works**: The Transform dataset operator is an interactive code notebook with a [pandas](https://pandas.pydata.org/) :octicons-link-external-24:{ alt="External link" title="External link" } dataframe. See the [pandas User Guide](https://pandas.pydata.org/docs/user_guide/index.html#user-guide) :octicons-link-external-24:{ alt="External link" title="External link" } for information on using pandas to work with data.
+
![Transform dataset that takes two datasets as inputs and displays choropleth comparison maps](../img/data/transform-operator.png)
**How it works**: The Transform dataset operator is an interactive code notebook with a [pandas](https://pandas.pydata.org/) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } dataframe. See the [pandas User Guide](https://pandas.pydata.org/docs/user_guide/index.html#user-guide) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for information on using pandas to work with data.
@@ -84,7 +84,7 @@ You can choose any step in your transformation process as the thumbnail preview. ## Modify data in the Transform dataset code notebook -Inside the Transform dataset operator is a code notebook in which you can prompt an AI assistant to answer questions about or modify your data. If you're comfortable writing code, you can also edit anything the assistant generates or add your own custom code (in a variety of languages). +Inside the Transform dataset operator is a code notebook. In the notebook, you can prompt an AI assistant to answer questions about or modify your data. If you're comfortable writing code, you can edit anything the assistant creates or add your own custom code. ![Transform dataset code notebook in which the AI assistant creates a new comparison](../img/data/transform-notebook.png) @@ -125,8 +125,11 @@ The Transform dataset AI assistant interprets plain language to answer questions ??? list "Prompt or question the AI assistant" - 1. Use the text box at the top of the page to enter a question or describe the transformation you want to make and then click Submit :octicons-paper-airplane-24:{ style="transform: rotate(-45deg);" title="Submit" aira-labelledby="Submit" }. - 2. Scroll down to the new code cell to inspect the transformation. + 1. Click in the text box at the top of the page and then perform one of the following actions: + - Select one of the suggested prompts and edit it to fit your dataset and the transformation your want to make. + - Enter a question or describe the transformation you want to make. + 2. Click Submit :octicons-paper-airplane-24:{ style="transform: rotate(-45deg);" title="Submit" aira-labelledby="Submit" }. + 3. Scroll down to the new code cell to inspect the transformation. ??? list "Choose where to insert a prompt" @@ -184,7 +187,132 @@ When the response is complete, the code cell may also contain: ### Add or edit code -At any time, you can edit the code generated by the AI assistant or enter your own custom code. +At any time, you can edit the code generated by the AI assistant or enter your own custom code. The notebook environment supports the following languages, each extended with commonly used data manipulation and scientific operation libraries. + +???+ note + + The use of Julia is currently disabled. + +
+ +- :simple-python:{ .lg .middle aria-hidden="true" } __[Python](https://docs.python.org/3.10/) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } libraries__ + + --- + + - [pandas](https://pandas.pydata.org/pandas-docs/version/1.3/user_guide/index.html) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for organizing, cleaning, and analyzing data tables and time series. + - [numpy](https://numpy.org/doc/1.24/user/absolute_beginners.html) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for handling of large arrays of numbers and performing mathematical operations. + - [scipy](https://docs.scipy.org/doc/scipy-1.11.4/index.html) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for performing advanced scientific operations, including optimization, integration, and interpolation. + - [pickle](https://docs.python.org/3/library/pickle.html) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for saving and reloading complex data structures. + +- :simple-julia:{ .lg .middle aria-hidden="true" } __[Julia](https://docs.julialang.org/en/v1.10/) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } libraries__ + + --- + + - [DataFrames](https://dataframes.juliadata.org/stable/man/getting_started/) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for manipulating data tables. + - [CSV](https://csv.juliadata.org/stable/#Overview) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for reading, writing, and processing CSV files. + - [HTTP](https://juliaweb.github.io/HTTP.jl/stable/) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for sending and receiving data over the Internet. + - [JSON3](https://github.com/quinnj/JSON3.jl) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for working with JSON data + - [DisplayAs](https://github.com/tkf/DisplayAs.jl) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for displaying data. + +- :simple-r:{ .lg .middle aria-hidden="true" } __[R](https://cran.r-project.org/doc/manuals/r-release/R-intro.html) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } libraries__ + + --- + + - [data.frame](https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/data.frame) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for manipulating data tables. + + +
+ +???+ tip + + More libraries are available in the code notebook, but you may need to import them before use. + + 1. To list the available packages, click :octicons-plus-24:{ aria-hidden="true" } **Add a cell** and then enter and run: + + === "Python" + + ```bash + pip list + ``` + + === "Julia" + + ```julia + Pkg.installed() + ``` + + === "R" + + ```r + installed.packages() + ``` + + 2. To import a package, click :octicons-plus-24:{ aria-hidden="true" } **Add a cell** and then enter and run: + + === "Python" + + ```bash + import + ``` + + === "Julia" + + ```julia + using + ``` + + === "R" + + ```r + library() + ``` + + ??? tip "Additional libraries that may be useful for data transformations" + +
+ + - __Data manipulation and analysis__ + + --- + + - [dask](https://docs.dask.org/en/stable/) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for handling large datasets and computations efficiently. + - [geopandas](https://geopandas.org/en/v0.13.2/docs.html) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for working with geographic data in tables. + - [xarray](https://docs.xarray.dev/en/v0.19.0/) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for managing and analyzing multidimensional datasets. + + + - __Data visualization__ + + --- + + - [cartopy](https://scitools.org.uk/cartopy/docs/latest/) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for creating maps and visualizing geographic data. + - [matplotlib](https://matplotlib.org/3.7.5/users/index.html) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for creating static, animated, and interactive visualizations. + + - __Machine learning__ + + --- + + - [scikit-learn](https://scikit-learn.org/1.4/user_guide.html) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for creating machine learning models. + - [torch](https://pytorch.org/docs/2.5/) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for building, training, and experimenting with machine learning models. + + - __Image processing__ + + --- + + - [scikit-image](https://scikit-image.org/docs/stable/) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for processing and analyzing images. + + - __Graph and network analysis__ + + --- + + - [networkx](https://networkx.org/documentation/stable/reference/) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for working with networks and graphs. + +
+ +??? list "Change the language of the code notebook" + + The Transform dataset AI assistant writes Python code by default. You can switch between Python, R, or Julia code at any time. + + - Use the **language** drop down above the code cells. ??? list "Make changes to a transformation" @@ -202,12 +330,6 @@ At any time, you can edit the code generated by the AI assistant or enter your o 1. Select the cell above where you want to insert the new code cell. 2. Click :octicons-plus-24:{ aria-hidden="true" } **Add a cell**. -??? list "Change the language of generated code" - - The Transform dataset AI assistant writes Python code by default. You can switch between Python, R, or Julia code at any time. - - - Use the **language** drop down above the code cells. - ## Save transformed data Saved datasets appear in your Resources panel and the output of the Transform dataset operator. diff --git a/mkdocs.yml b/mkdocs.yml index 8699514..e12876e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -15,6 +15,7 @@ theme: - navigation.instant - navigation.footer - content.code.copy + - content.tabs.link logo_dark_mode: img/terarium-logo-dark.svg logo_light_mode: img/terarium-logo-light.svg palette: From 955cc375f3b112af368e67d1dcd9a823cd47ec5a Mon Sep 17 00:00:00 2001 From: mecrouch Date: Fri, 22 Nov 2024 15:18:50 -0500 Subject: [PATCH 2/2] Typo fixes --- docs/datasets/transform-dataset.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/datasets/transform-dataset.md b/docs/datasets/transform-dataset.md index 585b0e3..48dd61b 100644 --- a/docs/datasets/transform-dataset.md +++ b/docs/datasets/transform-dataset.md @@ -126,7 +126,7 @@ The Transform dataset AI assistant interprets plain language to answer questions ??? list "Prompt or question the AI assistant" 1. Click in the text box at the top of the page and then perform one of the following actions: - - Select one of the suggested prompts and edit it to fit your dataset and the transformation your want to make. + - Select one of the suggested prompts and edit it to fit your dataset and the transformation you want to make. - Enter a question or describe the transformation you want to make. 2. Click Submit :octicons-paper-airplane-24:{ style="transform: rotate(-45deg);" title="Submit" aira-labelledby="Submit" }. 3. Scroll down to the new code cell to inspect the transformation. @@ -211,7 +211,7 @@ At any time, you can edit the code generated by the AI assistant or enter your o - [DataFrames](https://dataframes.juliadata.org/stable/man/getting_started/) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for manipulating data tables. - [CSV](https://csv.juliadata.org/stable/#Overview) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for reading, writing, and processing CSV files. - [HTTP](https://juliaweb.github.io/HTTP.jl/stable/) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for sending and receiving data over the Internet. - - [JSON3](https://github.com/quinnj/JSON3.jl) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for working with JSON data + - [JSON3](https://github.com/quinnj/JSON3.jl) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for working with JSON data. - [DisplayAs](https://github.com/tkf/DisplayAs.jl) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } for displaying data. - :simple-r:{ .lg .middle aria-hidden="true" } __[R](https://cran.r-project.org/doc/manuals/r-release/R-intro.html) :octicons-link-external-24:{ aria-hidden="true" alt="External link" title="External link" } libraries__