Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add otlplogfile exporter #5408

Open
pellared opened this issue May 23, 2024 · 6 comments · May be fixed by #5743
Open

Add otlplogfile exporter #5408

pellared opened this issue May 23, 2024 · 6 comments · May be fixed by #5743
Assignees
@thomasgouveia
Labels
area:logs Part of OpenTelemetry logs blocked:specification Waiting on clarification of the OpenTelemetry specification before progress can be made enhancement New feature or request pkg:exporter:otlp Related to the OTLP exporter package

Comments

@pellared
Copy link
Member

pellared commented May 23, 2024 •
edited
Loading

Blocked:

  • Awaits until it is stable in the specification.

Implement an OTLP Log File exporter based on https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/file-exporter.md
@pellared pellared added the enhancement New feature or request label May 23, 2024
@pellared pellared added area:logs Part of OpenTelemetry logs pkg:exporter:otlp Related to the OTLP exporter package labels May 23, 2024
@dmathieu dmathieu changed the title Add otlplogfile expoter Add otlplogfile exporter May 23, 2024
@pellared pellared mentioned this issue Jun 6, 2024
Merged
MrAlias added a commit that referenced this issue Jun 11, 2024
@pellared @MrAlias
sdk/log: Remove slice allocation from SimpleProcessor.OnEmit (#5493)
6082e83 
The reason for this improvement (apart that, in general, it is good to
have better performance) is there may be good use case to use a
`SimpleProcessor` to emit logs efficiently to standard output. It could
be one of the most efficient solutions (from application performance
perspective) and thanks to such configuration the user would not lose
any logs if the application suddenly crashes. For instance, a useful
configuration could be a simple processor with an OTLP file exporter
(#5408).

I think we might consider changing the following portion of
`SimpleProcessor` documentation (but I would prefer to do it as separate
PR):

> // This Processor is not recommended for production use. The
synchronous
// nature of this Processor make it good for testing, debugging, or
// showing examples of other features, but it can be slow and have a
high
// computation resource usage overhead. [NewBatchProcessor] is
recommended
// for production use instead.

```
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/otel/sdk/log
cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
                                    │   old.txt   │               new.txt                │
                                    │   sec/op    │    sec/op     vs base                │
Processor/Simple-16                   449.4n ± 7%   156.2n ±  5%  -65.25% (p=0.000 n=10)
Processor/ModifyTimestampSimple-16    468.0n ± 6%   171.3n ± 15%  -63.40% (p=0.000 n=10)
Processor/ModifyAttributesSimple-16   515.8n ± 3%   233.2n ±  8%  -54.77% (p=0.000 n=10)
geomean                               476.9n        184.1n        -61.40%

                                    │   old.txt   │                 new.txt                 │
                                    │    B/op     │    B/op     vs base                     │
Processor/Simple-16                    417.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
Processor/ModifyTimestampSimple-16     417.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
Processor/ModifyAttributesSimple-16   465.00 ± 0%   48.00 ± 0%   -89.68% (p=0.000 n=10)
geomean                                432.4                    ?                       ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean

                                    │  old.txt   │                 new.txt                 │
                                    │ allocs/op  │ allocs/op   vs base                     │
Processor/Simple-16                   1.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
Processor/ModifyTimestampSimple-16    1.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
Processor/ModifyAttributesSimple-16   2.000 ± 0%   1.000 ± 0%   -50.00% (p=0.000 n=10)
geomean                               1.260                    ?                       ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean
```

---------

Co-authored-by: Tyler Yahn <[email protected]>
OrHayat pushed a commit to OrHayat/opentelemetry-go that referenced this issue Jun 23, 2024
@pellared @MrAlias @OrHayat
sdk/log: Remove slice allocation from SimpleProcessor.OnEmit (open-te…
810d25d 
…lemetry#5493)

The reason for this improvement (apart that, in general, it is good to
have better performance) is there may be good use case to use a
`SimpleProcessor` to emit logs efficiently to standard output. It could
be one of the most efficient solutions (from application performance
perspective) and thanks to such configuration the user would not lose
any logs if the application suddenly crashes. For instance, a useful
configuration could be a simple processor with an OTLP file exporter
(open-telemetry#5408).

I think we might consider changing the following portion of
`SimpleProcessor` documentation (but I would prefer to do it as separate
PR):

> // This Processor is not recommended for production use. The
synchronous
// nature of this Processor make it good for testing, debugging, or
// showing examples of other features, but it can be slow and have a
high
// computation resource usage overhead. [NewBatchProcessor] is
recommended
// for production use instead.

```
goos: linux
goarch: amd64
pkg: go.opentelemetry.io/otel/sdk/log
cpu: Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
                                    │   old.txt   │               new.txt                │
                                    │   sec/op    │    sec/op     vs base                │
Processor/Simple-16                   449.4n ± 7%   156.2n ±  5%  -65.25% (p=0.000 n=10)
Processor/ModifyTimestampSimple-16    468.0n ± 6%   171.3n ± 15%  -63.40% (p=0.000 n=10)
Processor/ModifyAttributesSimple-16   515.8n ± 3%   233.2n ±  8%  -54.77% (p=0.000 n=10)
geomean                               476.9n        184.1n        -61.40%

                                    │   old.txt   │                 new.txt                 │
                                    │    B/op     │    B/op     vs base                     │
Processor/Simple-16                    417.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
Processor/ModifyTimestampSimple-16     417.0 ± 0%     0.0 ± 0%  -100.00% (p=0.000 n=10)
Processor/ModifyAttributesSimple-16   465.00 ± 0%   48.00 ± 0%   -89.68% (p=0.000 n=10)
geomean                                432.4                    ?                       ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean

                                    │  old.txt   │                 new.txt                 │
                                    │ allocs/op  │ allocs/op   vs base                     │
Processor/Simple-16                   1.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
Processor/ModifyTimestampSimple-16    1.000 ± 0%   0.000 ± 0%  -100.00% (p=0.000 n=10)
Processor/ModifyAttributesSimple-16   2.000 ± 0%   1.000 ± 0%   -50.00% (p=0.000 n=10)
geomean                               1.260                    ?                       ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean
```

---------

Co-authored-by: Tyler Yahn <[email protected]>
@pellared pellared mentioned this issue Jul 31, 2024
Open
@MrAlias MrAlias moved this to Backlog in Go: Logs (Post-GA) Aug 1, 2024
@thomasgouveia
Copy link

thomasgouveia commented Aug 8, 2024 •
edited
Loading

Hi @pellared!

Before doing this one, as stated in #5500, maybe it will be necessary to introduce something that can serialize the data into the OpenTelemetry JSON format.

Because as you mentioned here, we can add a public function into a otlplogfile package, but is it coherent with what the package is responsible for? I mean that IMHO, the otlplogfile package should be responsible for appending log records to a JSON file.

If we extract the serialization part into another internal (or public, if it makes sense) package such as otlpserde (naming is just an example), we can use a dependency into otlplogfile and stdoutlog, so the format will be consistent across the different implementations.

What do you think about it?

@pellared
Copy link
Member Author

pellared commented Aug 8, 2024

stdoutlog is intended to be used mainly for debugging therefore it aims for human-readability.

otlplogfile is on the other hand MUST serialize telemetry in computer-readable format as defined in the specification.

I suggest not extracting any separate package until necessary.

thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
feat(open-telemetry#5408): add otlplogfile exporter
f575b5f 
This commit adds a new experimental exporter `otlplogfile`, that outputs log records to a JSON line file.
It is based on the following specification: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/file-exporter.md

Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
feat(open-telemetry#5408): add otlplogfile exporter
9b22c0d 
This commit adds a new experimental exporter `otlplogfile`, that outputs log records to a JSON line file.
It is based on the following specification: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/file-exporter.md

Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
feat(open-telemetry#5408): add otlplogfile exporter
1f72ae3 
This commit adds a new experimental exporter `otlplogfile`, that outputs log records to a JSON line file.
It is based on the following specification: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/file-exporter.md

Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
feat(open-telemetry#5408): add otlplogfile exporter
1869179 
This commit adds a new experimental exporter `otlplogfile`, that outputs log records to a JSON line file.
It is based on the following specification: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/file-exporter.md

Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
docs(open-telemetry#5408): add example file
eb79f04 
Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
feat(open-telemetry#5408): add otlplogfile exporter
15c9ef5 
This commit adds a new experimental exporter `otlplogfile`, that outputs log records to a JSON line file.
It is based on the following specification: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/file-exporter.md

Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
docs(open-telemetry#5408): add example file
66151b4 
Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
feat(open-telemetry#5408): add otlplogfile exporter
cf66e2d 
This commit adds a new experimental exporter `otlplogfile`, that outputs log records to a JSON line file.
It is based on the following specification: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/file-exporter.md

Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
docs(open-telemetry#5408): add example file
714fa75 
Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
docs(open-telemetry#5408): update CHANGELOG
360e32d 
Signed-off-by: thomasgouveia <[email protected]>
@thomasgouveia thomasgouveia linked a pull request Aug 27, 2024 that will close this issue
Draft
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
docs(open-telemetry#5408): update CHANGELOG
02d2dab 
Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
feat(open-telemetry#5408): add otlplogfile exporter
6b5c5cd 
This commit adds a new experimental exporter `otlplogfile`, that outputs log records to a JSON line file.
It is based on the following specification: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/file-exporter.md

Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
docs(open-telemetry#5408): add example file
322f7e9 
Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
docs(open-telemetry#5408): update CHANGELOG
27c761d 
Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
docs(open-telemetry#5408): update CHANGELOG
64ee265 
Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
feat(open-telemetry#5408): add otlplogfile exporter
89d1e79 
This commit adds a new experimental exporter `otlplogfile`, that outputs log records to a JSON line file.
It is based on the following specification: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/file-exporter.md

Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
docs(open-telemetry#5408): add example file
7dde8dc 
Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
docs(open-telemetry#5408): update CHANGELOG
0f3e7c8 
Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Aug 27, 2024
@thomasgouveia
docs(open-telemetry#5408): update CHANGELOG
37c4f5e 
Signed-off-by: thomasgouveia <[email protected]>
thomasgouveia added a commit to thomasgouveia/opentelemetry-go that referenced this issue Sep 6, 2024
@thomasgouveia
feat(open-telemetry#5408): use stdout as default output
7244adc 
Signed-off-by: thomasgouveia <[email protected]>
@pellared
Copy link
Member Author

During last SIG meeting (#5743 (comment)) we decided that we do not want to maintain such exporter until it is stable.

@thomasgouveia, it may be worth to move the exporter you have created in #5743 to a separate repository and maintain it there. I can help you in maintaining it.

@pellared pellared closed this as not planned Won't fix, can't repro, duplicate, stale Sep 12, 2024
@github-project-automation github-project-automation bot moved this from Backlog to Done in Go: Logs (Post-GA) Sep 12, 2024
@thomasgouveia
Copy link

@pellared may it be located in the contrib repository? Or the contrib focuses only on stable implementation?

If we can't put it in the contrib repository, I can transfer it to a repository on my account, until the specification is stable.

@pellared
Copy link
Member Author

pellared commented Sep 13, 2024 •
edited
Loading

During SIG it was recommended to not put it to contrib as well yet.

I can transfer it to a repository on my account, until the specification is stable.

Let's do this.

@pellared
Copy link
Member Author

pellared commented Nov 26, 2024 •
edited
Loading

A shout-out to work related to OTLP File exporter:

However, let's wait until it is stable in the specification as we miss velocity to maintain an new experimental module at this point of time.

@pellared pellared reopened this Nov 26, 2024
@pellared pellared moved this from Done to Backlog in Go: Logs (Post-GA) Nov 26, 2024
@pellared pellared moved this from Backlog to Blocked in Go: Logs (Post-GA) Nov 26, 2024
@pellared pellared added the blocked:specification Waiting on clarification of the OpenTelemetry specification before progress can be made label Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thomasgouveia thomasgouveia

Labels
area:logs Part of OpenTelemetry logs blocked:specification Waiting on clarification of the OpenTelemetry specification before progress can be made enhancement New feature or request pkg:exporter:otlp Related to the OTLP exporter package
Projects
Go: Logs (Post-GA)
Status: Blocked
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

exporters: add otlplogfile exporter thomasgouveia/opentelemetry-go
2 participants
@pellared @thomasgouveia