Skip to content

Commit

Permalink
"Created by snbot"
Browse files Browse the repository at this point in the history
  • Loading branch information
streamnativebot committed Nov 27, 2024
1 parent 70b2f72 commit 2e12abf
Show file tree
Hide file tree
Showing 4 changed files with 43 additions and 79 deletions.
18 changes: 9 additions & 9 deletions docs/lakehouse-sink.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ To build the Lakehouse sink connector from the source code, follow these steps.

```bash
ls target
pulsar-io-lakehouse-{{connector:version}}.nar
pulsar-io-lakehouse-3.3.2.4.nar
```

# How to configure
Expand Down Expand Up @@ -127,7 +127,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"inputs": [
"test-hudi-pulsar"
],
"archive": "connectors/pulsar-io-hudi-{{connector:version}}.nar",
"archive": "connectors/pulsar-io-hudi-3.3.2.4.nar",
"processingGuarantees": "EFFECTIVELY_ONCE",
"parallelism": 1,
"configs": {
Expand All @@ -151,7 +151,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"inputs": [
"test-hudi-pulsar"
],
"archive": "connectors/pulsar-io-hudi-{{connector:version}}-cloud.nar",
"archive": "connectors/pulsar-io-hudi-3.3.2.4-cloud.nar",
"parallelism": 1,
"processingGuarantees": "EFFECTIVELY_ONCE",
"configs": {
Expand Down Expand Up @@ -179,7 +179,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"inputs": [
"test-iceberg-pulsar"
],
"archive": "connectors/pulsar-io-lakehouse-{{connector:version}}.nar",
"archive": "connectors/pulsar-io-lakehouse-3.3.2.4.nar",
"processingGuarantees":"EFFECTIVELY_ONCE",
"configs":{
"type":"iceberg",
Expand Down Expand Up @@ -207,7 +207,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"inputs": [
"test-iceberg-pulsar"
],
"archive": "connectors/pulsar-io-lakehouse-{{connector:version}}-cloud.nar",
"archive": "connectors/pulsar-io-lakehouse-3.3.2.4-cloud.nar",
"processingGuarantees":"EFFECTIVELY_ONCE",
"configs":{
"type":"iceberg",
Expand Down Expand Up @@ -238,7 +238,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"inputs": [
"test-delta-pulsar"
],
"archive": "connectors/pulsar-io-lakehouse-{{connector:version}}.nar",
"archive": "connectors/pulsar-io-lakehouse-3.3.2.4.nar",
"processingGuarantees":"EFFECTIVELY_ONCE",
"configs":{
"type":"delta",
Expand All @@ -260,7 +260,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"inputs": [
"test-delta-pulsar"
],
"archive": "connectors/pulsar-io-lakehouse-{{connector:version}}-cloud.nar",
"archive": "connectors/pulsar-io-lakehouse-3.3.2.4-cloud.nar",
"processingGuarantees":"EFFECTIVELY_ONCE",
"configs":{
"type":"delta",
Expand Down Expand Up @@ -330,7 +330,7 @@ This example describes how to use the Lakehouse sink connector to fetch data fro
1. Copy the NAR package to the Pulsar connectors directory.
```
cp pulsar-io-lakehouse-{{connector:version}}.nar PULSAR_HOME/connectors/pulsar-io-lakehouse-{{connector:version}}.nar
cp pulsar-io-lakehouse-3.3.2.4.nar PULSAR_HOME/connectors/pulsar-io-lakehouse-3.3.2.4.nar
```
2. Start Pulsar in standalone mode.
Expand Down Expand Up @@ -363,7 +363,7 @@ This example explains how to create a Lakehouse sink connector in an on-premises
1. Copy the NAR package of the Lakehouse sink connector to the Pulsar connectors directory.
```bash
cp pulsar-io-lakehouse-{{connector:version}}.nar $PULSAR_HOME/connectors/pulsar-io-lakehouse-{{connector:version}}.nar
cp pulsar-io-lakehouse-3.3.2.4.nar $PULSAR_HOME/connectors/pulsar-io-lakehouse-3.3.2.4.nar
```
2. Reload all [built-in connectors](https://pulsar.apache.org/docs/en/next/io-connectors/).
Expand Down
10 changes: 5 additions & 5 deletions docs/lakehouse-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ To build the Lakehouse source connector from the source code, follow these steps

```bash
ls target
pulsar-io-lakehouse-{{connector:version}}.nar
pulsar-io-lakehouse-3.3.2.4.nar
```

# How to configure
Expand Down Expand Up @@ -90,7 +90,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"parallelism":1,
"topicName": "delta_source",
"processingGuarantees":"ATLEAST_ONCE",
"archive": "connectors/pulsar-io-lakehouse-{{connector:version}}.nar",
"archive": "connectors/pulsar-io-lakehouse-3.3.2.4.nar",
"configs":{
"type":"delta",
"checkpointInterval": 180,
Expand All @@ -115,7 +115,7 @@ You can create a configuration file (JSON or YAML) to set the properties if you
"parallelism":1,
"topicName": "delta_source",
"processingGuarantees":"ATLEAST_ONCE",
"archive": "connectors/pulsar-io-lakehouse-{{connector:version}}-cloud.nar",
"archive": "connectors/pulsar-io-lakehouse-3.3.2.4-cloud.nar",
"configs":{
"type":"delta",
"checkpointInterval": 180,
Expand Down Expand Up @@ -175,7 +175,7 @@ This example describes how to use the Lakehouse source connector to fetch data f
1. Copy the NAR package to the Pulsar connectors directory.
```
cp pulsar-io-lakehouse-{{connector:version}}.nar PULSAR_HOME/connectors/pulsar-io-lakehouse-{{connector:version}}.nar
cp pulsar-io-lakehouse-3.3.2.4.nar PULSAR_HOME/connectors/pulsar-io-lakehouse-3.3.2.4.nar
```
2. Start Pulsar in standalone mode.
Expand Down Expand Up @@ -206,7 +206,7 @@ This example explains how to create a Lakehouse source connector in an on-premis
1. Copy the NAR package of the Lakehouse source connector to the Pulsar connectors directory.
```
cp pulsar-io-lakehouse-{{connector:version}}.nar $PULSAR_HOME/connectors/pulsar-io-lakehouse-{{connector:version}}.nar
cp pulsar-io-lakehouse-3.3.2.4.nar $PULSAR_HOME/connectors/pulsar-io-lakehouse-3.3.2.4.nar
```
2. Reload all [built-in connectors](https://pulsar.apache.org/docs/en/next/io-connectors/).
Expand Down
2 changes: 1 addition & 1 deletion image/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ ENV JAVA_HOME=/usr
# we need to update the download link once the version is updated
RUN curl -Ls https://github.com/xerial/snappy-java/archive/refs/tags/$SNAPPY_VERSION.tar.gz | tar zxf - && cd snappy-java-$SNAPPY_VERSION && make clean-native native

FROM streamnative/pulsar-functions-java-runner:${PULSAR_VERSION}
FROM snstage/pulsar-functions-java-runner:${PULSAR_VERSION}
USER 0
COPY --from=snappy-java /tmp/libsnappyjava.so /usr/lib/libsnappyjava.so
# need to configure -Dorg.xerial.snappy.use.systemlib=true in the jvm options to use the system lib
Expand Down
Loading

0 comments on commit 2e12abf

Please sign in to comment.