Pool decode buffers when encoding messages using compression

[masih]

Pool the decode buffers and set strict max cap allocation for zstd decompressor.

See Benchamark results below.

Before:

BenchmarkCborEncoding
BenchmarkCborEncoding-12    	  467352	      2376 ns/op	   14680 B/op	      10 allocs/op
BenchmarkCborDecoding
BenchmarkCborDecoding-12    	  104410	     11347 ns/op	   14944 B/op	      27 allocs/op
BenchmarkZstdEncoding
BenchmarkZstdEncoding-12    	  286735	      3897 ns/op	   46748 B/op	      12 allocs/op
BenchmarkZstdDecoding
BenchmarkZstdDecoding-12    	  110794	     10783 ns/op	   28512 B/op	      28 allocs/op

After:

BenchmarkCborEncoding
BenchmarkCborEncoding-12    	  436754	      2383 ns/op	   14680 B/op	      10 allocs/op
BenchmarkCborDecoding
BenchmarkCborDecoding-12    	  106809	     11280 ns/op	   14944 B/op	      27 allocs/op
BenchmarkZstdEncoding
BenchmarkZstdEncoding-12    	  294043	      3918 ns/op	   46746 B/op	      12 allocs/op
BenchmarkZstdDecoding
BenchmarkZstdDecoding-12    	  114854	     10747 ns/op	   18314 B/op	      27 allocs/op

Fixes: #849

[Stebalien]

I'm not seeing the "more performant" part except a slight perf bump on decode, but using the same library makes sense.

I'm surprised we saved an allocation on encode and nothing on decode (where we added the pool). What happens if we remove the pool? Does the datadog version just have an additional allocation internally?

[codecov]

Codecov Report

Attention: Patch coverage is 92.85714% with 1 line in your changes missing coverage. Please review.

Project coverage is 67.14%. Comparing base (bae93e2) to head (6194fee).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/encoding/encoding.go	92.85%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #854      +/-   ##
==========================================
- Coverage   67.21%   67.14%   -0.07%     
==========================================
  Files          83       83              
  Lines        9055     9064       +9     
==========================================
  Hits         6086     6086              
- Misses       2430     2434       +4     
- Partials      539      544       +5     

Files with missing lines	Coverage Δ
internal/encoding/encoding.go	68.62% <92.85%> (+6.72%)	⬆️

... and 7 files with indirect coverage changes

[rvagg]

consistent dependency with the one used in Lotus

Not really. filecoin-project/lotus#12015 kind of relevant, there's some pressure to not introduce more cgo lock-in if we can help it, particularly for downstream dependents of those components. I'd say unless the perf improvement here is really dramatic maybe let's just stuck with klauspost/compress.

[masih]

consistent dependency with the one used in Lotus

Not really.

Consistent dependency to the same libraries for the same functionality IMO is a desirable. Maybe the answer here is to move Lotus off the DataDog dependency. I know that klauspost/compress zstd package has come a long way since its early days in terms of performance.

[masih]

I'm surprised we saved an allocation on encode and nothing on decode (where we added the pool). What happens if we remove the pool? Does the datadog version just have an additional allocation internally?

pooling makes no difference to the encoding part. The difference in allocation comes from the internals of the dependency libraries. Here are the numbers for DataDog/zstd without pooling:

BenchmarkCborEncoding
BenchmarkCborEncoding-12    	  436611	      2375 ns/op	   14680 B/op	      10 allocs/op
BenchmarkCborDecoding
BenchmarkCborDecoding-12    	  106036	     11302 ns/op	   14944 B/op	      27 allocs/op
BenchmarkZstdEncoding
BenchmarkZstdEncoding-12    	  258430	      4643 ns/op	   28248 B/op	      11 allocs/op
BenchmarkZstdDecoding
BenchmarkZstdDecoding-12    	   27158	     43543 ns/op	 1063526 B/op	      28 allocs/op

[masih]

To unblock work I am going to separate pooling from dependency change in this PR until we figure out filecoin-project/lotus#12852.