Skip to content

Commit

Permalink
#132: Fixed outdated documentation (#150)
Browse files Browse the repository at this point in the history
  • Loading branch information
MarleneKress79789 authored Nov 14, 2023
1 parent 5aacbce commit 57f9d33
Show file tree
Hide file tree
Showing 3 changed files with 36 additions and 25 deletions.
2 changes: 1 addition & 1 deletion doc/changes/changes_0.6.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,6 @@ T.B.D

### Documentation

- n/a
- # 132: Fixed outdated information in documentation


34 changes: 22 additions & 12 deletions doc/developer_guide/developer_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ In this developer guide we explain how to build this project and how you can add
new transformer tasks and tests.


## Building the Project
## Installation
There are two ways to install the Transformers Extension Package:

### 1. Build the Python Package
### 1. Build and install the Python Package
This project needs Python 3.8 or above installed on the development machine.
In addition, in order to build Python packages you need to have the [Poetry](https://python-poetry.org/)
(>= 1.1.11) package manager. Then you can install and build the `transformers-extension` as follows:
Expand All @@ -16,21 +17,30 @@ poetry install
poetry build
```

### 2. Install the Project
The latest version of the Python package of this extension can be downloaded
from the Releases in GitHub Repository (see [the latest release](https://github.com/exasol/transformers-extension/releases/latest)).
Please download the built archive `transformers_extension.whl` and install it as follows:
### 2. Download and install the pre-build wheel
Instead of building yourself, the latest version of the Python package of this extension can be downloaded
from the Releases in the GitHub Repository (see [the latest release](https://github.com/exasol/transformers-extension/releases/latest)).
Please download the built archive
`exasol_transformers_extension-<version-number>-py3-none-any.whl`(`transformers_extension.whl` in older versions)
and install it as follows:
```bash
pip install dist/transformers_extension.whl --extra-index-url https://download.pytorch.org/whl/cpu
pip install <path/wheel-filename.whl> --extra-index-url https://download.pytorch.org/whl/cpu
```

### 3. Run All Tests

### Run Tests
All unit and integration tests can be run within the Poetry environment created
for the project as follows:
for the project using nox. See [the nox file](../../noxfile.py) for all tasks run by nox. There are three tasks for tests.

Run unit tests:
```bash
poetry run pytest tests
poetry run nox -s unit_tests
```
Start a test database and run integration tests:
```bash
poetry run nox -s start_database
poetry run nox -s integration_tests
```


## Add Transformer Tasks
In the transformers-extension library, the 8 most popular NLP tasks provided by
Expand All @@ -39,7 +49,7 @@ been defined. We created separate UDF scripts for each NLP task. You can find
these tasks and UDF script usage details in the [User Guide](../user_guide/user_guide.md#prediction-udfs).
This section shows you step by step how to add a new NLP task to this library.

### 1. Add UDF Template
### 1. Add a UDF Template
The new task's UDF template should be added to the `exasol_transformers_extension/resources/templates/`
directory. Please pay attention that the UDF script is uses _"SET UDF"_ and the inputs
are received ordered by pre-determined columns. In addition, the first 4 input
Expand Down
25 changes: 13 additions & 12 deletions doc/user_guide/user_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ use of pre-trained NLP models provided by the [Transformers API](https://hugging

The extension provides two types of UDFs:
- DownloaderUDF : It is responsible to download the specified pre-defined model into the Exasol BucketFS.
- Prediction UDFs: These are a group of UDFs for each supported task. Each of them uses the downloaded pre-trained model and perform prediction. These supported tasks:
- Prediction UDFs: These are a group of UDFs for each supported task. Each of them uses the downloaded pre-trained
model and perform prediction. These are the supported tasks:
1. Sequence Classification for Single Text
2. Sequence Classification for Text Pair
3. Question Answering
Expand Down Expand Up @@ -67,17 +68,18 @@ The extension provides two types of UDFs:
### The Python Package
#### Download The Python Wheel Package
- The latest version of the python package of this extension can be
downloaded from the Releases in GitHub Repository
(see [the latest release](https://github.com/exasol/transformers-extension/releases/latest)).
downloaded from the [GitHUb Release](https://github.com/exasol/transformers-extension/releases/latest).
Please download the following built archive:
```buildoutcfg
transformers_extension.whl
exasol_transformers_extension-<version-number>-py3-none-any.whl
```
If you need to use a version < 0.5.0, the build archive is called `transformers_extension.whl`.


#### Install The Python Wheel Package
- Install the packaged transformers-extension project as follows:
Install the packaged transformers-extension project as follows:
```shell
pip install transformers_extension.whl --extra-index-url https://download.pytorch.org/whl/cpu
pip install <path/wheel-filename.whl> --extra-index-url https://download.pytorch.org/whl/cpu
```

### The Pre-built Language Container
Expand All @@ -87,12 +89,12 @@ extension to run. It can be installed in two ways: Quick and Customized
installations

#### Quick Installation
The desired language container is downloaded and installed by executing the
deployment script below with the desired version. (see GitHub Releases
[the latest release](https://github.com/exasol/transformers-extension/releases).
The language container is downloaded and installed by executing the
deployment script below with the desired version. Make sure the version matches with your installed version of the
Transformers Extension Package. See [the latest release](https://github.com/exasol/transformers-extension/releases) on Github.

```buildoutcfg
python -m exasol_transformers_extension.deploy language-container
python -m exasol_transformers_extension.deploy language-container \
--dsn <DB_HOST:DB_PORT> \
--db-user <DB_USER> \
--db-pass <DB_PASSWORD> \
Expand All @@ -107,8 +109,7 @@ deployment script below with the desired version. (see GitHub Releases
--language-alias <LANGUAGE_ALIAS> \
--version <RELEASE_VERSION> \
--ssl-cert-path <ssl-cert-path> \
--use-ssl-cert-validation \
--no-use-ssl-cert-valiation
--use-ssl-cert-validation
```
The `--ssl-cert-path` is optional if your certificate is not in the OS truststore.
The option `--use-ssl-cert-validation`is the default, you can disable it with `--no-use-ssl-cert-validation`.
Expand Down

0 comments on commit 57f9d33

Please sign in to comment.