Skip to content

Commit

Permalink
[Go] SBE implementation in Go using flyweights (#951)
Browse files Browse the repository at this point in the history 
* [Go] SBE implementation in Go using flyweights #765

To generate flyweights instead of structs, set the following flag:

```
sbe.go.generate.generate.flyweights=true
```

* [Go] Checkstyle fix

* [Go] Fix null string return value

* [Go] Remove spurious type definition for choices

* [Go] Fix formatting in WrapForDecode

* [Go] Add returning an array by []byte

* [Go] Fix array not present in version

* [Go] Add ComputeLength methods

* [Go] Fix some compiler warnings and remove unused code

* [Go] Draft of OTF decoder

* [Go] Add OTF Json Printer

* [Go] Fix SbeBlockAndHeaderLength and add constants for composite sizes

* [Go] Add IrDecoder for runtime types

* [Go] Add enum get method

* [Go] Fixes for updating component counts in irdecoder

* [Go] Run goimports

* [Go] Update with latest changes

* [Go] Change the file structure for GOPATH

* [Go] Fix copywrite year

* [Go] Fix tests for otf

This generates flyweights in the sbe-tool build for use in the otf library.

* [Go] Add helpers for otf PrimitiveValue
  • Loading branch information
@ethanf
ethanf authored Oct 29, 2024
1 parent e34fc03 commit da547cc
Show file tree
Hide file tree
Showing 87 changed files with 8,428 additions and 65 deletions.
37 changes: 12 additions & 25 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -65,31 +65,18 @@ cppbuild/Win32

# golang
gocode/pkg
gocode/src/baseline/*.go
!gocode/src/baseline/*_test.go
gocode/src/baseline-bigendian/*.go
!gocode/src/baseline-bigendian/*_test.go
gocode/src/composite/*.go
!gocode/src/composite/*_test.go
gocode/src/composite_elements/*.go
!gocode/src/composite_elements/*_test.go
gocode/src/since-deprecated/*.go
!gocode/src/since-deprecated/*_test.go
gocode/src/group_with_data*/*.go
!gocode/src/group_with_data*/*_test.go
gocode/src/mktdata/*.go
!gocode/src/mktdata/*_test.go
gocode/src/simple/*.go
!gocode/src/simple/*_test.go
gocode/src/issue*/*.go
!gocode/src/issue*/*_test.go
gocode/src/*/*/*.go
!gocode/src/*/*/*_test.go
gocode/src/example-schema/example-schema*
gocode/src/example-schema/cpu.out
gocode/src/example-socket-clientserver/example-socket-clientserver
gocode/src/extension
gocode/src/extension2
gocode/*/pkg
gocode/struct/src/*/*.go
!gocode/struct/src/*/*_test.go
gocode/struct/src/*/*/*.go
!gocode/struct/src/*/*/*_test.go
gocode/struct/src/example-schema/example-schema*
gocode/struct/src/example-socket-clientserver/example-socket-clientserver
gocode/flyweight/src/*/*.go
!gocode/flyweight/src/*/*_test.go
gocode/flyweight/src/*/*/*.go
!gocode/flyweight/src/*/*/*_test.go
gocode/flyweight/src/otf/code-generation-schema.sbeir

# csharp
csharp/*/bin
Expand Down
2 changes: 2 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,8 @@ For convenience on Linux, a gnu Makefile is provided that runs some tests and co
$ cd gocode
# make # test, examples, bench

Go supports both generating Go structs with encode / decode methods, and flyweights like the other languages. Structs are generated by default for compatibility. Set `sbe.go.generate.generate.flyweights=true` to generate flyweights.

Users of golang generated code should see the [user
documentation](https://github.com/real-logic/simple-binary-encoding/wiki/Golang-User-Guide).

Expand Down
100 changes: 94 additions & 6 deletions build.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -700,7 +700,7 @@ tasks.register('generateGolangCodecTestComposite', JavaExec) {
mainClass.set('uk.co.real_logic.sbe.SbeTool')
classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
systemProperties(
'sbe.output.dir': 'gocode/src',
'sbe.output.dir': 'gocode/struct/src',
'sbe.target.language': 'golang')
args = ['sbe-tool/src/test/resources/composite-elements-schema-rc4.xml']
}
Expand All @@ -709,7 +709,7 @@ tasks.register('generateGolangCodecTestBasic', JavaExec) {
mainClass.set('uk.co.real_logic.sbe.SbeTool')
classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
systemProperties(
'sbe.output.dir': 'gocode/src/basic',
'sbe.output.dir': 'gocode/struct/src/basic',
'sbe.target.language': 'golang')
args = ['sbe-tool/src/test/resources/basic-types-schema.xml']
}
Expand All @@ -718,7 +718,7 @@ tasks.register('generateGolangCodecTestGroup', JavaExec) {
mainClass.set('uk.co.real_logic.sbe.SbeTool')
classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
systemProperties(
'sbe.output.dir': 'gocode/src/group',
'sbe.output.dir': 'gocode/struct/src/group',
'sbe.target.language': 'golang')
args = ['sbe-tool/src/test/resources/basic-group-schema.xml']
}
Expand All @@ -727,7 +727,7 @@ tasks.register('generateGolangCodecTestVarData', JavaExec) {
mainClass.set('uk.co.real_logic.sbe.SbeTool')
classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
systemProperties(
'sbe.output.dir': 'gocode/src/vardata',
'sbe.output.dir': 'gocode/struct/src/vardata',
'sbe.target.language': 'golang')
args = ['sbe-tool/src/test/resources/basic-variable-length-schema.xml']
}
Expand All @@ -736,7 +736,7 @@ tasks.register('generateGolangCodecsWithXIncludes', JavaExec) {
mainClass.set('uk.co.real_logic.sbe.SbeTool')
classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
systemProperties(
'sbe.output.dir': 'gocode/src',
'sbe.output.dir': 'gocode/struct/src',
'sbe.target.language': 'golang',
'sbe.xinclude.aware': 'true',
'sbe.validation.xsd': validationXsdPath)
Expand All @@ -748,7 +748,87 @@ tasks.register('generateGolangCodecsWithXSD', JavaExec) {
mainClass.set('uk.co.real_logic.sbe.SbeTool')
classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
systemProperties(
'sbe.output.dir': 'gocode/src',
'sbe.output.dir': 'gocode/struct/src',
'sbe.target.language': 'golang',
'sbe.xinclude.aware': 'true',
'sbe.validation.xsd': validationXsdPath)
args = ['sbe-tool/src/test/resources/group-with-data-schema.xml',
'sbe-tool/src/test/resources/FixBinary.xml',
'sbe-tool/src/test/resources/issue435.xml',
'sbe-tool/src/test/resources/issue472.xml',
'sbe-tool/src/test/resources/issue483.xml',
'sbe-tool/src/test/resources/issue488.xml',
'sbe-tool/src/test/resources/issue560.xml',
'sbe-tool/src/test/resources/issue661.xml',
'sbe-tool/src/test/resources/issue847.xml',
'sbe-tool/src/test/resources/issue848.xml',
'sbe-tool/src/test/resources/issue849.xml',
'sbe-tool/src/test/resources/since-deprecated-test-schema.xml',
'sbe-tool/src/test/resources/example-bigendian-test-schema.xml',
'gocode/resources/example-composite.xml',
'gocode/resources/group-with-data-extension-schema.xml',
'gocode/resources/simple.xml']
}

tasks.register('generateGolangFlyweightCodecTestComposite', JavaExec) {
mainClass.set('uk.co.real_logic.sbe.SbeTool')
classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
systemProperties(
'sbe.output.dir': 'gocode/flyweight/src',
'sbe.go.generate.generate.flyweights': 'true',
'sbe.target.language': 'golang')
args = ['sbe-tool/src/test/resources/composite-elements-schema-rc4.xml']
}

tasks.register('generateGolangFlyweightCodecTestBasic', JavaExec) {
mainClass.set('uk.co.real_logic.sbe.SbeTool')
classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
systemProperties(
'sbe.output.dir': 'gocode/flyweight/src/basic',
'sbe.go.generate.generate.flyweights': 'true',
'sbe.target.language': 'golang')
args = ['sbe-tool/src/test/resources/basic-types-schema.xml']
}

tasks.register('generateGolangFlyweightCodecTestGroup', JavaExec) {
mainClass.set('uk.co.real_logic.sbe.SbeTool')
classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
systemProperties(
'sbe.output.dir': 'gocode/flyweight/src/group',
'sbe.go.generate.generate.flyweights': 'true',
'sbe.target.language': 'golang')
args = ['sbe-tool/src/test/resources/basic-group-schema.xml']
}

tasks.register('generateGolangFlyweightCodecTestVarData', JavaExec) {
mainClass.set('uk.co.real_logic.sbe.SbeTool')
classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
systemProperties(
'sbe.output.dir': 'gocode/flyweight/src/vardata',
'sbe.go.generate.generate.flyweights': 'true',
'sbe.target.language': 'golang')
args = ['sbe-tool/src/test/resources/basic-variable-length-schema.xml']
}

tasks.register('generateGolangFlyweightCodecsWithXIncludes', JavaExec) {
mainClass.set('uk.co.real_logic.sbe.SbeTool')
classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
systemProperties(
'sbe.output.dir': 'gocode/flyweight/src',
'sbe.go.generate.generate.flyweights': 'true',
'sbe.target.language': 'golang',
'sbe.xinclude.aware': 'true',
'sbe.validation.xsd': validationXsdPath)
args = ['sbe-samples/src/main/resources/example-schema.xml',
'sbe-samples/src/main/resources/example-extension-schema.xml']
}

tasks.register('generateGolangFlyweightCodecsWithXSD', JavaExec) {
mainClass.set('uk.co.real_logic.sbe.SbeTool')
classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
systemProperties(
'sbe.output.dir': 'gocode/flyweight/src',
'sbe.go.generate.generate.flyweights': 'true',
'sbe.target.language': 'golang',
'sbe.xinclude.aware': 'true',
'sbe.validation.xsd': validationXsdPath)
Expand Down Expand Up @@ -780,6 +860,13 @@ tasks.register('generateGolangCodecs') {
'generateGolangCodecTestComposite',
'generateGolangCodecsWithXIncludes',
'generateGolangCodecsWithXSD',
'generateGolangFlyweightCodecTestVarData',
'generateGolangFlyweightCodecTestGroup',
'generateGolangFlyweightCodecTestBasic',
'generateGolangFlyweightCodecTestComposite',
'generateGolangFlyweightCodecsWithXIncludes',
'generateGolangFlyweightCodecsWithXSD',
'generateIrCodecs',
':sbe-all:jar'
}

Expand Down Expand Up @@ -868,6 +955,7 @@ tasks.register('generateGolangIrCodecs', JavaExec) {
systemProperties(
'sbe.output.dir': 'sbe-tool/src/main/golang',
'sbe.target.language': 'golang',
'sbe.go.generate.generate.flyweights': true,
'sbe.validation.xsd': validationXsdPath)
args = ['sbe-tool/src/main/resources/sbe-ir.xml']
}
Expand Down
74 changes: 52 additions & 22 deletions gocode/Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,69 +19,99 @@ all: example test # bench
# This target is used to build golang files using parent gradle
# script's invocation of sbe-tool in case it needs to be run again. We
# use this one output file to check that dependency as it's simple
DEP=src/simple/SbeMarshalling.go
DEP=struct/src/simple/SbeMarshalling.go
$(DEP): $(SBE_JAR)
(cd ..; ./gradlew generateGolangCodecs)
(export GOPATH=$(GOPATH) && \
cd src/simple && \
(export GOPATH=$(GOPATH)/struct && \
cd struct/src/simple && \
go build && \
$(SAVE_FORMAT) \
go fmt && \
go test)

# Will regenerate codecs by removing dependencies
clean:
rm -f src/*/SbeMarshalling.go src/*/*/SbeMarshalling.go
rm -f struct/src/*/SbeMarshalling.go struct/src/*/*/SbeMarshalling.go

# Example code and benchmarking
example: src/example-schema/example-schema
src/example-schema/example-schema: $(DEP)
(export GOPATH=$(GOPATH) && \
cd src/example-schema && \
# Example code and benchmarking
example: struct/src/example-schema/example-schema
struct/src/example-schema/example-schema: $(DEP)
(export GOPATH=$(GOPATH)/struct && \
cd struct/src/example-schema && \
go build && \
go fmt && \
./example-schema)

bench: $(DEP)
(export GOPATH=$(GOPATH) && \
cd src/example-schema && \
(export GOPATH=$(GOPATH)/struct && \
cd struct/src/example-schema && \
go test --bench . -cpuprofile=cpu.out && \
go tool pprof -text example-schema.test cpu.out)

src/baseline/SbeMarshalling.go: $(DEP)
struct/src/baseline/SbeMarshalling.go: $(DEP)
$(GOPATH)/sbe-tool -d src -s $(EXAMPLES_SCHEMA)
(export GOPATH=$(GOPATH) && \
cd src/baseline && \
(export GOPATH=$(GOPATH)/struct && \
cd struct/src/baseline && \
go build && \
go fmt && \
go test && \
go install)

# The first set does a make install as there is a test that uses
# multiple packages and needs them in GOPATH The second set work in
# src/foo, and the third need a GOPATH addition as for Java and C++
# struct/src/foo, and the third need a GOPATH addition as for Java and C++
# they generate into the same directory but golang doesn't allow that
test: $(DEP)
(export GOPATH=$(GOPATH) && \
(export GOPATH=$(GOPATH)/struct && \
(for t in baseline extension; do \
export GOPATH=$(GOPATH) && \
cd $(GOPATH)/src/$$t && \
export GOPATH=$(GOPATH)/struct && \
cd $(GOPATH)/struct/src/$$t && \
go build && \
go fmt && \
go test && \
go install \
;done))
(export GOPATH=$(GOPATH) && \
(export GOPATH=$(GOPATH)/struct && \
(for t in baseline-bigendian mktdata group_with_data group_with_data_extension composite_elements composite since-deprecated simple issue435 issue472 issue483 issue488 issue560 issue847 issue848 issue849 issue505 test973; do \
cd $(GOPATH)/src/$$t && \
cd $(GOPATH)/struct/src/$$t && \
go build && \
go fmt && \
go test \
;done))
(for t in vardata group basic; do \
export GOPATH=$(GOPATH)/$$t && \
cd $(GOPATH)/src/$$t/'SBE tests' && \
export GOPATH=$(GOPATH)/struct/$$t && \
cd $(GOPATH)/struct/src/$$t/'SBE tests' && \
go build && \
go fmt && \
go test \
;done)
(export GOPATH=$(GOPATH)/flyweight && \
(for t in baseline extension; do \
export GOPATH=$(GOPATH)/flyweight && \
cd $(GOPATH)/flyweight/src/$$t && \
go build && \
go fmt && \
go test && \
go install \
;done))
(export GOPATH=$(GOPATH)/flyweight && \
(for t in baseline-bigendian mktdata group_with_data group_with_data_extension composite_elements composite since-deprecated simple issue435 issue472 issue483 issue488 issue560 issue847 issue848 issue849; do \
cd $(GOPATH)/flyweight/src/$$t && \
go build && \
go fmt && \
go test \
;done))
(for t in vardata group basic; do \
export GOPATH=$(GOPATH)/flyweight/$$t && \
cd $(GOPATH)/flyweight/src/$$t/'SBE tests' && \
go build && \
go fmt && \
go test \
;done)
(export SBE_JAR=$(SBE_JAR) && \
export GO111MODULE=on && \
(cd $(GOPATH)/flyweight/src/otf && \
go generate && \
go build && \
go fmt && \
go test))
10 changes: 6 additions & 4 deletions gocode/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,13 @@ There is now a
[user guide](https://github.com/real-logic/simple-binary-encoding/wiki/Golang-User-Guide)
and this document is for development of the SBE golang generator.

Go supports both generating Go structs with encode / decode methods, and flyweights like the other languages.
Structs are generated by default for compatibility. Set `sbe.go.generate.generate.flyweights=true` to generate flyweights.

Code Layout
-----------
The Java code that performs the generation of golang code is
[here](https://github.com/real-logic/simple-binary-encoding/tree/master/sbe-tool/src/main/java/uk/co/real_logic/sbe/generation/golang).
[here](https://github.com/real-logic/simple-binary-encoding/tree/master/sbe-tool/src/main/java/uk/co/real_logic/sbe/generation/golang). There is both a struct and a flyweight generator.

Golang code used for testing resides in the top-level
[gocode directory](https://github.com/real-logic/simple-binary-encoding/tree/master/gocode).
Expand All @@ -24,11 +27,12 @@ the build environment, example and test code are structured into a top
level gocode directory hierarchy.

Code is generated into this structure with pre-existing test code in place.
There are tests for both the struct and flyweight styles of generated code.

For the example, the code is generated into a library and the matching
example code lives in it's own directory at the same level. For
example, the example-schema generates the baseline library code into
`gocode/src/baseline` and example code lives in `gocode/src/example-schema`.
`gocode/src/struct/baseline` and example code lives in `gocode/src/example-schema`.

To use this layout you should `set GOPATH=/path/to/gocode` or use the
supplied Makefile which does this for you. For the tests you will need
Expand Down Expand Up @@ -60,5 +64,3 @@ generator processes.
Roadmap
=======
* Windows developer support (currently tested on Linux/MacOS)
* Further Unicode support
* Testing/Bug fixes
Loading

0 comments on commit da547cc

Please sign in to comment.